Ask a language model to fix a specific CVE on a fleet of Ubuntu 20.04 hosts and it will hand back a plausible looking apt command in under two seconds. Whether that command is correct for your exact kernel, your package pinning, and your cluster topology is a separate question, and it is the one that decides whether you let the model touch production or just read its output. This piece looks at where LLM generated remediation is trustworthy, where it quietly breaks, and how to build a review loop that catches the difference before a command runs.
What a model can produce reliably
Large language models are strongest on remediation tasks that are well represented in training data and have low branching: one clear command, one clear syntax, few environmental variables. In practice that covers three categories well.
- Standard configuration directives. Disabling TLS 1.0 and 1.1 in an Apache ssl.conf, adding an HSTS header in Nginx, turning off directory listing, setting PermitRootLogin no in sshd_config. These are documented the same way across thousands of tutorials and vendor docs, so the model has converged on one correct answer.
- Package manager commands for common distributions. apt-get install –only-upgrade openssl, dnf update httpd, pip install –upgrade urllib3==2.2.2. When the target package name matches the CVE advisory and the distribution is mainstream, the model rarely gets the verb or flag wrong.
- Generic hardening steps. Restricting SSH ciphers to a modern set, enabling rp_filter in sysctl, setting a password complexity policy. These map almost one to one onto CIS benchmark language, which the model has effectively memorized.
If your remediation backlog is dominated by these three categories, a first draft from an LLM is a reasonable starting point for a human to sign off on. The problem is that most backlogs are not dominated by these categories once you get past the first few hundred findings.
Where accuracy drops
Accuracy degrades in three predictable places, and all three share a root cause: the correct answer depends on a fact the model was not given.
Version-specific syntax
The flag that disables weak ciphers in OpenSSH 7.4 is not identical to the one in OpenSSH 8.9. Apache 2.2’s mod_ssl directives differ from 2.4’s. A model prompted only with a CVE ID and a product name will often produce syntax for whichever version dominates its training data, which is frequently the latest stable release, not the one actually installed.
Vendor-specific paths and dialects
Config file locations differ across Debian, RHEL, and Alpine even for the same daemon. Network appliance CLIs differ more sharply: a remediation phrased in Cisco IOS syntax is not valid on NX-OS or on a Palo Alto PAN-OS device, even though all three might be described in a finding as “firewall.” Without a device fingerprint, the model has to guess a dialect, and it guesses the most common one in its corpus.
Clustered and stateful systems
A single-host remediation and a clustered one are not the same task. Patching a database node without checking replica lag first, restarting all Kubernetes nodes in a DaemonSet at once instead of a rolling update, or applying a firewall rule change to every node in an HA pair simultaneously can each turn a routine patch into an outage. Models trained mostly on single-host examples default to single-host remediation language unless the prompt explicitly supplies cluster topology.
Why a plausible-looking command can be wrong for your exact platform
The dangerous failure mode is not an obviously broken command, it is one that is syntactically valid, executes without error, and still does not fix the vulnerability or fixes it while breaking something else. A few concrete patterns show up repeatedly:
- The command targets a package name that exists on the system but is not the one actually vulnerable (a common issue with bundled libraries, vendored dependencies, or language-specific package managers shadowing an OS package of the same name).
- The command references a flag that existed in an older or newer version of the binary than the one installed, so it either errors out or silently no-ops.
- The remediation overwrites a config file wholesale instead of patching the relevant directive, discarding local customizations that were never visible to the model.
- The exit code is zero, so an automation pipeline marks the finding closed, but the actual vulnerable code path is untouched because the service was not restarted, a cached binary was not reloaded, or the patch applied to a package that was not in the actual load path.
That last pattern deserves attention because it defeats naive automation completely. A patch command returning success tells you the shell executed without a fatal error, not that the vulnerability is gone. Platforms that automate this phase, such as SITEY, re-test the specific finding instead of trusting the patch command exit code, re-running the original detection logic against the host after the change lands. That is the only way to know the fix worked rather than assuming it did because nothing printed in red.
Grounding techniques that raise accuracy
The fix for version and vendor ambiguity is not a better prompt, it is better input. Three grounding techniques consistently move remediation from plausible to correct.
Retrieved vendor documentation
Instead of asking the model to recall syntax from training data, retrieve the relevant page from the vendor’s current documentation (or an offline mirror of it) and place the exact directive names and version constraints into the prompt. This turns a recall task into an extraction and formatting task, which models handle far more reliably.
Detected OS and package facts
Before generating a command, query the host for its actual OS release, package manager, installed package version, and init system (systemd, OpenRC, SysV). Feeding these facts into the prompt eliminates most of the version-mismatch failures described above, because the model is no longer guessing which dialect to use.
Parameterized templates over free text
For high-volume, well-understood finding types, a remediation template with placeholders (package name, version, config path) filled from detected facts is more reliable than free-form generation every time. Reserve free-text generation for findings that do not fit an existing template, and route everything else through the template. Systems that ground remediation this way, SITEY among them, pull the installed version and package manager from the asset inventory before drafting a command rather than inferring either from the CVE description alone.
Measuring remediation accuracy with a labelled test set instead of vibes
“It looked right” is not a measurement. Build a labelled test set: take a sample of past findings, and for each one have an engineer write the verified-correct remediation independently of the model. Run the model against the same findings and score each output against four categories rather than a single pass or fail:
| Category | What it checks | Why it matters |
|---|---|---|
| Applicability | Does the command target the right package, host, or device at all | Wrong target is worse than wrong syntax |
| Syntactic validity | Does it parse and run without error on the actual installed version | Catches version-specific flag mismatches |
| Functional correctness | Does the finding actually re-test as resolved afterward | The only category that reflects the real goal |
| Blast radius | Does it touch only the intended scope (host, node, config block) | Catches wholesale config overwrites and cluster-wide restarts |
Track these four numbers separately rather than collapsing them into one accuracy score. A remediation engine that is 95 percent syntactically valid but only 60 percent functionally correct is telling you it produces confident-looking commands that frequently do not fix the underlying problem, which is a materially different risk profile than a lower syntactic score with high functional correctness. Re-run this evaluation whenever the underlying model, prompt, or grounding data changes; treat it as a regression suite, not a one-time benchmark.
Review workflow that catches the remaining errors before execution
No grounding technique gets functional correctness to 100 percent, so the review step is not optional. A workable gate looks like this:
- Dry run first. Generate the command and its expected diff (config change, package version delta) without executing it, and show both to the reviewer together.
- Human approval on anything outside a template. Free-text generated remediation for a novel finding type gets a mandatory approval gate; templated remediation for a well-tested finding type can have a lighter one.
- Canary before fleet-wide. Apply to one host or one node in a cluster, wait for the retest to confirm the finding closed, then roll out to the rest.
- Automatic re-test, not exit-code trust. Re-run the original scanner or detection logic against the changed host and only mark the finding closed when that check passes, not when the shell returns zero.
- Rollback path defined before execution. Every generated remediation should ship with the inverse command or a snapshot reference, so a bad canary can be reverted in minutes rather than diagnosed from scratch.
The combination of a labelled accuracy baseline, fact-grounded generation, and a staged approval gate is what turns LLM-drafted remediation from an interesting demo into something you can point at production. None of it removes the need for a human in the loop on anything outside the templated, low-risk category, and that is a feature of the workflow, not a limitation to engineer away.
About SITEY
SITEY is an autonomous vulnerability management platform. It discovers, validates, prioritizes, remediates and re-tests vulnerabilities through an eight-phase automated pipeline, unifying output from 17 integrated scanners. SITEY is self-hosted: it runs in your own infrastructure and your findings are stored there. Outbound connections are limited to licence activation and the optional services you enable, such as an AI provider, CVE enrichment and patch catalogues. Pricing is 599 USD per month or 5,999 USD for a perpetual lifetime license. See pricing or how the platform works.