How to Verify an AI-Written Patch Script Before You Run It

An AI model can draft a patch script in seconds, but the exit code it prints when it “succeeds” tells you nothing about what actually happened on the host. Before an AI generated patch script touches a single production system, it needs to pass through the same discipline you would apply to a script written by a junior engineer you have never worked with: read it, run it without applying it, run it on one machine you can throw away, then sign off on the exact bytes that go to the fleet. This is the sequence that makes that discipline concrete, with the commands and thresholds to apply at each step.

A pre-execution checklist: idempotency, privilege scope, service restarts, file overwrites

Before reading a single line of code, decide what the script is allowed to do and confirm it stays inside that boundary. Four properties matter more than the rest, because they are the ones that turn a routine patch into an incident.

• Idempotency. Run the script twice in a row on a test host. The second run should exit 0 and report no changes, not fail, not reapply the same edit, and not append a duplicate line to a config file. A script that appends to /etc/hosts or a cron file without checking for an existing entry first will corrupt state the third time a scheduler retries it.
• Privilege scope. Map every command to the minimum privilege it needs. A script that starts with sudo su - and runs forty lines as root, when only three of those lines touch a root-owned file, is a script that should be split. Prefer explicit sudo prefixes on individual commands over a blanket privilege escalation at the top.
• Service restarts. Distinguish a reload from a restart. systemctl reload nginx re-reads configuration without dropping connections; systemctl restart nginx does not. An AI-generated fix for a config vulnerability has no reason to restart a service unless the patch also touches a binary or a loaded module, and if it does restart, the script should check for graceful drain support first.
• File overwrites. Confirm the script backs up before it writes: a timestamped copy such as cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.$(date +%s) before it edits the original. If the script overwrites in place with no backup and no diff shown, treat that as a blocking finding, not a style note.

Static review of the script: destructive commands, wildcards, unpinned versions

Static review means reading the script end to end without executing it, specifically hunting for patterns that are individually common but collectively dangerous when generated without a human in the loop. Grep for the obvious ones first:

• grep -nE 'rm -rf|Remove-Item .*-Recurse.*-Force|DROP (TABLE|DATABASE)|chmod 777|chown -R' script.sh surfaces destructive, irreversible operations. Any match needs a manual justification, not a rubber stamp.
• Wildcards attached to deletion or permission commands are the most common way an AI-generated script causes collateral damage. rm -rf /var/log/app/* is fine; rm -rf $LOG_DIR/* is not, if $LOG_DIR can be empty or unset, because an unset variable turns that command into rm -rf /*. Check every variable used inside a path with a wildcard for a default value or an explicit non-empty guard.
• Unpinned package or dependency versions turn a one-time fix into a moving target. apt-get install -y openssl installs whatever is newest in the repository on the day it runs, which may not be the version the AI validated against. Pin it: apt-get install -y openssl=3.0.13-0ubuntu3.4, or the equivalent pinned syntax for yum, pip, or npm.
• Anything that pipes a remote fetch straight into an interpreter, curl https://example.com/fix.sh | bash or iwr https://example.com/fix.ps1 | iex, should be rewritten to download, hash-check, and then execute as two separate steps. There is no scenario where the compressed one-liner is worth the loss of a verification point.

Dry-run and check-mode execution in Ansible, PowerShell and shell

Every major automation surface has a built-in mode that reports intended changes without applying them, and skipping it because “it’s just a small script” is how small scripts cause large outages.

• Ansible. ansible-playbook site.yml --check --diff -vv runs the playbook in check mode and prints a diff of every file it would modify. Modules that do not support check mode (some command or shell tasks) will report as changed unconditionally, so confirm each task either uses a native module or is explicitly marked with check_mode: no and a manual justification.
• PowerShell. Functions written with [CmdletBinding(SupportsShouldProcess)] accept -WhatIf, which prints what the cmdlet would do without doing it: Set-Service -Name W32Time -StartupType Disabled -WhatIf. If the AI-generated script uses raw .NET calls or COM objects instead of cmdlets, -WhatIf has nothing to hook into, and that alone is a reason to ask for a rewrite using native cmdlets.
• Shell. There is no universal check mode for a raw bash script, so build one in: gate every state-changing line behind a DRY_RUN variable, print the command instead of running it, and only remove the gate after the printed plan has been reviewed. A minimal pattern: run() { if [ "$DRY_RUN" = "1" ]; then echo "+ $*"; else "$@"; fi; }, then call every mutating command through run.

Platforms that orchestrate patch rollout across a fleet, SITEY among them, put this dry-run step behind an approval gate by default: the generated remediation is rendered as a diff and a command preview before anyone can promote it to a live run, so the check-mode output is what gets reviewed rather than a description of it.

Sandbox and snapshot testing on a representative host before fleet rollout

Dry-run output tells you what a script intends to do; it does not tell you what happens when that intent meets a real filesystem, a real kernel version, and real load. That requires actually running it, on a host you can discard.

• Pick a host that matches the fleet’s median configuration, not the newest or the most-patched one. Match OS point release, kernel version, and installed package versions as closely as you can; a script that behaves correctly on a host patched last week and incorrectly on one three months behind is exactly the gap you are trying to close.
• Snapshot before you run: an LVM snapshot (lvcreate --size 5G --snapshot --name pre_patch /dev/vg0/root), a ZFS snapshot (zfs snapshot pool/root@pre_patch), or a cloud provider’s volume or AMI snapshot. The rollback plan is only real if you have tested restoring from it at least once, not just created it.
• Define success criteria before you run the script, not after: the target service responds on its health check endpoint within a fixed timeout, the process count matches expectation, the original vulnerability no longer reproduces against the scanner signature that flagged it, and no new error appears in the last five minutes of logs. If any of these is undefined, the run’s outcome will be judged by feel instead of by a threshold.
• Roll out to a canary slice before the full fleet, commonly one host per distinct OS and patch-level combination, or a fixed small percentage such as two to five percent of the affected population, and hold for a soak period (an hour is a reasonable floor for most services) before continuing.

This is also where re-testing the actual finding, not just the exit code, pays for itself. A patch script can exit 0 because the package manager reported success while the vulnerable configuration file was never reloaded, or a config that a previous manual change had already excluded from the automated path. Closing the loop with an independent re-scan against the same finding, the same approach SITEY’s retest phase takes before it closes a ticket, is the only way to confirm the fix actually landed rather than trusting the script’s own report of itself.

Signing and storing the approved script so the reviewed version is the one that runs

A script that passed review is worthless if the version that executes on the fleet is not provably the same one. Close that gap explicitly.

• Hash the approved file the moment review finishes: sha256sum approved_fix.sh > approved_fix.sh.sha256. Before every execution, verify: sha256sum -c approved_fix.sh.sha256, and abort on mismatch.
• For PowerShell, use Authenticode signing rather than a hash file alone: Set-AuthenticodeSignature -FilePath fix.ps1 -Certificate $cert, combined with an execution policy of AllSigned on the hosts that run it, so an unsigned or tampered script fails closed instead of prompting a human to click through a warning.
• Store the approved script by immutable reference: a Git commit SHA or a content-addressed artifact ID, never a mutable path like latest or a branch name that can move after review. Reference the SHA in the execution job, and log which SHA actually ran against which host, alongside who approved it and when.
• Keep the audit trail outside the script itself. An approval workflow that records reviewer identity, timestamp, target host list, and the exact command line used is what lets you answer “who approved this and against what version” six months later, which is the point where most teams discover they cannot.

Red flags that mean a human should rewrite it rather than edit it

Some issues are worth a quick edit before the script proceeds. Others mean the generated script should be discarded and rewritten by a person, because patching the symptom in the script does not fix the underlying gap in what the model understood about the target system.

• The script fetches and executes remote content in the same step it modifies system state, with no separation between download, verification, and execution.
• It disables or bypasses logging, auditing, or a security control (auditd, SELinux, a firewall rule) as a side effect of applying the fix, rather than as the fix’s actual intent.
• It contains base64-encoded or otherwise obfuscated command blocks that need to be decoded before you can read what they do.
• It touches subsystems unrelated to the vulnerability it claims to remediate, for example a script addressing a web server header misconfiguration that also modifies SSH settings or user account permissions.
• It has no error handling: no set -e in bash, no exit code checks in PowerShell, no failure branch in Ansible tasks, so a mid-script failure leaves the host in a half-applied state with no signal that anything went wrong.
• It contains a hardcoded credential, API key, or connection string, which means the generated fix itself introduces a new finding.
• The commands do not semantically match the vulnerability description that prompted the fix, a sign the generation drifted from the actual finding rather than addressing it.

Any one of these is grounds to stop the review and hand the task to a person who writes the fix from scratch, informed by what the AI got wrong, rather than patching the AI’s script line by line. Teams that build a remediation pipeline around this, generation, static review, dry-run, canary, signed storage, retest, tend to converge on the same design: the model proposes, a fixed set of gates decides what qualifies for automatic promotion, and everything else routes to a person. That pattern is what modules like AI-driven remediation planning and approval gate workflows are built to enforce, and it is the same reason patch management and retest and closure are kept as separate, auditable phases instead of one opaque “apply fix” button.

About SITEY

SITEY is an autonomous vulnerability management platform. It discovers, validates, prioritizes, remediates and re-tests vulnerabilities through an eight-phase automated pipeline, unifying output from 17 integrated scanners. SITEY is self-hosted: it runs in your own infrastructure and your findings are stored there. Outbound connections are limited to licence activation and the optional services you enable, such as an AI provider, CVE enrichment and patch catalogues. Pricing is 599 USD per month or 5,999 USD for a perpetual lifetime license. See pricing or how the platform works.