Before a vulnerability management pipeline is trusted to close tickets, push patches, or reassign work without asking, most security teams want evidence that its decisions match what a competent analyst would have made on the same data. Shadow mode is the mechanism for producing that evidence. The automation runs its full decision logic against the live queue, records a verdict for every finding, and the effectful step at the end (closing the ticket, triggering the patch, reassigning the owner) is replaced with a no-op that only logs what would have happened. Nothing reaches production. This article covers how to design that comparison so it actually tells you something, how long to run it, how to read the disagreements, and how to decide, action class by action class, when to flip execution on.
What Shadow Mode Actually Means for a Security Pipeline
Shadow mode is not a demo environment and it is not the automation running against a copy of last quarter’s findings. It has to sit in the same code path as production, consuming the same live feed from your scanners, at the same latency, with the same access to asset context, ticketing state, and prior remediation history. The only difference is a guard at the final action boundary: instead of calling the ticketing API to close a finding or the orchestration layer to push a patch, the system writes a structured record of the decision it would have executed, along with the evidence and reasoning that produced it.
This distinction matters because a shadow mode built as a separate simulation, with its own simplified logic, tells you nothing about the system you are actually going to trust. If the shadow path and the production path diverge in code, you are validating a different product. The guard should be a single choke point right before the write, not a parallel decision tree. Platforms that automate this phase, such as SITEY, implement the switch as a flag on the execution step inside the same autonomous operations pipeline, so what you observe in shadow is exactly what would have fired in production, down to the same API calls that get suppressed rather than sent.
It’s also worth being explicit about what “decides” means here. A shadow-mode verdict should include the full output an analyst would need to agree or disagree with it: severity, exploitability reasoning, affected asset, proposed remediation action, and a confidence score if the model produces one. A bare pass/fail label is not enough to run the disagreement analysis in section four.
Designing the Comparison: Automation Verdict Versus Analyst Verdict
The comparison only works if both sides are judging the identical unit: the same finding, on the same asset, with the same evidence snapshot, at the same point in time. Reprocessing a finding a week after an analyst closed it, once more context has accumulated, will inflate agreement artificially. Capture the automation’s verdict at first sighting, the same moment it would have fired in production, and pair it against the analyst’s verdict from the normal triage queue.
The harder problem is anchoring bias. If the analyst can see the automation’s suggested severity or remediation before making their own call, you are no longer measuring independent judgment, you are measuring how often analysts defer to a visible recommendation. Two practical ways to avoid this: either hide the automation’s verdict from the analyst’s queue entirely during the shadow period, or route a random subset of findings through a queue where the AI verdict field is masked until after the analyst submits theirs. The second approach is cheaper to run and gives you a clean control group without duplicating your triage headcount.
Store each pair as an append-only record rather than overwriting anything: finding ID, source scanner, first-seen timestamp, automation verdict and confidence, analyst verdict and analyst ID, and the timestamps each decision was made. This is the same record shape your AI triage module should already be producing for its own audit trail, so shadow mode is largely a matter of pointing that logging at a comparison table instead of, or in addition to, live action.
How Long to Run Shadow Mode, and What Sample Size Matters
Calendar time is the wrong unit. A finding class that occurs twice a week will not produce a meaningful sample in thirty days no matter how long you wait. Decision count per action class is the unit that matters, and it should be set before you start, not chosen after the fact to justify a result you already like.
As a rule of thumb: to distinguish 90 percent agreement from 95 percent agreement with a confidence interval narrow enough to act on (roughly plus or minus 5 percentage points), you need on the order of 300 to 400 paired decisions for that specific action class. For a common category like duplicate-finding suppression or routine severity labeling, that volume typically arrives within two to four weeks on a mid-sized estate. For rare, high-severity categories, such as a remote code execution finding on an internet-facing asset, you might see five or six of those in a month. Waiting for statistical confidence on that category alone could take most of a year.
Two ways to handle low-frequency classes without stalling the whole rollout. First, set a minimum sample floor (for example, 100 decisions) as a gate, and accept that some action classes simply stay in shadow longer than others rather than blocking the classes that already have enough data. Second, where your environment allows it, inject synthetic findings of the rare category, generated by a controlled internal red-team exercise, purely to raise the shadow-mode sample size for that class. Never use synthetic findings to inflate a live production metric, only to test the automation’s decision quality in shadow.
A reasonable default policy: run shadow mode for a minimum of one full patch cycle, commonly 30 days, and require at least 100 decisions for lower-risk action classes or 300 for classes tied to production changes, whichever condition takes longer to satisfy.
Disagreement Analysis: Where and Why the Automation Gets It Wrong
Once you have enough paired verdicts, build a confusion matrix per action class rather than a single aggregate agreement number. An 88 percent overall agreement rate can hide a category where agreement is 40 percent, and that category is exactly what you need to find before turning execution on for everything at once.
Break disagreements into distinct types instead of lumping them together:
- False positive suppression: automation calls a finding benign, analyst calls it real. This is the type that matters most for auto-close action classes, because a missed real finding disappears from the queue.
- Severity under-call: automation rates something medium that an analyst rates high or critical, usually because exploitability context (public exploit availability, asset exposure) was incomplete.
- Severity over-call: automation escalates something an analyst would triage as low priority, which mostly costs analyst time rather than risk, but at scale it erodes trust in the system’s judgment.
- Wrong owner or asset assignment: the finding is correctly classified but routed to the wrong team, usually a data problem in asset inventory rather than a triage logic problem.
- Wrong remediation action: the classification is right but the proposed patch or configuration change does not match what the analyst would apply.
Look for clustering, not just counting. If disagreement on severity under-calls is twice the baseline rate specifically for findings from one scanner, or specifically for assets tagged as legacy or unmanaged, that is a data quality problem in that source, not a general flaw in the model. Pull the reasoning trace behind each disagreement and check whether the automation lacked context (missing asset criticality tag, no exposure data), misjudged an exploit chain, or landed on a genuinely ambiguous case where two analysts would also have disagreed. That last category should be excluded from the metric entirely, since it is not something the automation can be expected to resolve more consistently than your own team does. Systems that log a full evidence and reasoning trail for every verdict, an approach used in SITEY’s operations log, make this root-causing step tractable because you are reading the actual chain of reasoning behind each decision, not just the label.
Exit Criteria for Turning Execution On, Per Action Class
Different action classes carry different blast radius, so a single agreement threshold applied across the board is the wrong model. A wrong ticket assignment costs a few minutes of a manual reroute. A wrong “safe to auto-close” verdict on a real finding hides risk until someone notices independently. A wrong “safe to patch” verdict on a production system can cause an outage. Set thresholds accordingly.
| Action class | Blast radius if wrong | Suggested agreement threshold | Minimum sample |
|---|---|---|---|
| Priority/severity labeling (no auto-action) | Low, analyst still reviews | 90% | 100 |
| Ticket routing/assignment | Low, cheap to reroute | 85% | 100 |
| Auto-close of duplicate or informational findings | High if a real finding is suppressed | 98%, zero suppressed true positives | 200 |
| Automated patch deployment, non-production | Medium | 95% | 150 |
| Automated patch deployment, production or customer-facing | High, potential outage | 99%, zero verdicts later reversed by retest | 300, staged canary rollout |
For the two highest blast-radius rows, do not rely on a single percentage as the only gate. Add a zero-tolerance criterion tied to a specific failure mode: zero instances where the automation marked a finding as resolved and a subsequent retest, through a module like retest and closure, showed the underlying vulnerability was still present. An automation that is right 99 percent of the time but occasionally claims a patch worked when it didn’t is not ready for production write access on that class, regardless of the aggregate number.
Turn execution on one action class at a time, and only for that class. Do not use a single global switch. A platform that supports per-class gating, such as SITEY’s approval gates, lets you enable automated patch management for internal non-production systems while keeping production patch deployment and ticket auto-closure in shadow, so a failure in one class doesn’t force you to roll back everything.
Keeping a Permanent Shadow Lane
Shadow mode is not a one-time gate you clear and forget. Every time you onboard a new scanner, add a new finding category, introduce a new remediation script, or bring a new asset class into scope (a first batch of cloud container workloads, or OT and ICS assets that were previously out of band), the thresholds you validated for existing categories do not transfer. The new category has to run through shadow again with its own sample size and its own exit criteria, because the automation has no track record on it yet.
It’s also worth keeping a small permanent shadow slice on action classes that are already live, as a canary against silent regression. Route 5 to 10 percent of production decisions through a shadow-only comparison even after go-live, so that a scanner version bump, a rule change, or a model update that quietly degrades accuracy shows up in the comparison data before it shows up as an incident. This complements rather than replaces the human checkpoints your highest-severity action classes should keep permanently; shadow mode tells you whether the automation’s judgment is trustworthy in aggregate, an approval gate is what stops any single bad call on a critical system from executing unsupervised.
The practical output of all this is a living table, not a one-time report: action class, current status (shadow or live), current sample size, current agreement rate, and date of last threshold review. Treat it the same way you’d treat a compliance control, something a security engineer or CISO can point to and say exactly why a given automated action is trusted, and exactly what evidence would revoke that trust.
About SITEY
SITEY is an autonomous vulnerability management platform. It discovers, validates, prioritizes, remediates and re-tests vulnerabilities through an eight-phase automated pipeline, unifying output from 17 integrated scanners. SITEY is self-hosted: it runs in your own infrastructure and your findings are stored there. Outbound connections are limited to licence activation and the optional services you enable, such as an AI provider, CVE enrichment and patch catalogues. Pricing is 599 USD per month or 5,999 USD for a perpetual lifetime license. See pricing or how the platform works.