Evaluating Agentic AI: Benchmarks, Traces and Success Metrics

Every vendor now claims an “agent” that plans, acts and resolves incidents on its own — but almost none of them can show you the trace that proves it, the benchmark that stress-tested it, or the metric that tells you whether it is getting better or worse in your environment. This article is a working engineer’s guide to evaluating agentic AI in IT and security operations: how to instrument the plan-act-verify loop, which benchmarks are worth your time, which success metrics actually predict production outcomes, and how to build the guardrails that keep an autonomous system honest.

Why evaluating an agent is not like evaluating a model

Model evaluation asks a narrow question: given this input, is the output correct? Agent evaluation asks a much harder one: given this goal, did the system choose a reasonable path, execute it safely, recover from the inevitable surprises, and stop at the right point? A large language model that scores well on MMLU or HumanEval tells you almost nothing about whether an autonomous remediation agent will correctly triage a flapping Kubernetes node, decide whether to restart a pod or escalate to a human, and avoid touching a production database credential along the way. The unit of evaluation has shifted from a single input-output pair to a multi-step trajectory across a changing environment, and that changes everything about how you test.

Three properties make agentic systems structurally harder to evaluate than the classifiers and chatbots that preceded them. First, the state space is non-stationary: an agent investigating a security alert changes the very system it is observing every time it queries a log source, quarantines a host, or opens a ticket, so the same test cannot be run twice under identical conditions unless the harness snapshots and restores state. Second, correctness is rarely binary. An incident-response agent that isolates the wrong host for ninety seconds before self-correcting is not simply “wrong” — it is partially correct, with a real but bounded cost. Third, the failure modes compound across steps. A small misclassification in step two (mislabeling a process as benign) can cascade into a materially wrong action in step seven (closing an incident that should have triggered forensic isolation), and a step-by-step accuracy average will hide that cascade completely.

This is why teams evaluating agentic platforms for SOC operations or integrated NOC/SOC environments need a different toolkit: trace-level instrumentation, task-completion benchmarks built from real operational scenarios, and a metrics hierarchy that separates what the model did from what actually happened to the environment. The rest of this article builds that toolkit piece by piece.

Anatomy of an agent trace: plan, act, verify

Before you can evaluate an agent you have to be able to see it think. A trace is the complete, ordered record of everything an agent did to pursue a goal — not just the final answer, but every intermediate reasoning step, every tool call, every observation returned from the environment, and every decision point where the agent chose to continue, retry, or stop. Without a trace, agent evaluation degenerates into judging outcomes with no visibility into cause, which is exactly how brittle automation gets shipped to production and then quietly disabled six weeks later when it does something inexplicable.

The canonical agent loop — plan, act, verify — maps to three distinct classes of events that a trace must capture with equal fidelity:

• Plan events: the decomposition of a goal into sub-tasks, the selection of a strategy among alternatives, and the reasoning that justified that selection. For an alert-triage agent this includes the hypothesis it formed about root cause before it ever touched a tool.
• Act events: every tool invocation, API call, script execution, or state-changing command, along with its exact parameters, the identity/credential context under which it ran, and a timestamp precise enough to correlate against downstream system logs.
• Verify events: the checks the agent ran against its own actions — did the pod actually restart, did the firewall rule actually apply, did the query actually return the expected shape of data — and the confidence or evidence it used to decide the sub-task was complete.

A trace worth evaluating needs six fields at minimum for every step: a monotonic step index, the step type (plan/act/verify/reflect), the input context the agent conditioned on, the raw model output before any parsing, the parsed action and its parameters, and the environment’s observation after the action executed. Teams that skip the raw model output field regret it the first time they need to distinguish a parsing bug in the harness from a genuine reasoning failure in the model — these look identical from the outside but require completely different fixes.

It also matters where you draw trace boundaries. A single user-facing “task” (for example, “investigate this SIEM alert and recommend a disposition”) frequently spans multiple agent invocations, tool round-trips, and even multiple specialized sub-agents handed off in sequence — a triage agent that calls a threat-intel enrichment agent that calls a case-management agent. Your tracing schema needs a parent task ID that threads through every sub-agent invocation, or you will lose the ability to compute end-to-end metrics and be left with disconnected fragments.

A benchmark taxonomy for IT and security agents

Generic agent benchmarks — WebArena-style browsing tasks, SWE-bench style code-fix tasks, general tool-use leaderboards — are useful for judging raw model capability but poor proxies for whether an agent will operate safely on your infrastructure. Operational teams need benchmarks built from the actual shape of IT and security work, organized along four axes: task complexity, environment fidelity, adversarial pressure, and time horizon.

Task complexity tiers

Not all agentic tasks deserve the same evaluation rigor. It helps to bucket them into three tiers before designing tests:

• Tier 1 — single-tool retrieval and classification: “is this alert a known false positive,” “pull the last 24 hours of CPU metrics for host X.” These are close to traditional ML evaluation and can largely be scored with precision/recall against a labeled set.
• Tier 2 — multi-tool investigation: correlating a SIEM alert with EDR telemetry, identity logs, and network flow data to build a root-cause hypothesis. Correctness now depends on the sequence and completeness of evidence gathered, not just a final label.
• Tier 3 — autonomous remediation with side effects: isolating a host, rotating a credential, rolling back a deployment, opening a firewall change. These tasks have real-world blast radius and require the richest evaluation: safety checks, rollback verification, and blast-radius containment testing.

A benchmark suite that only tests Tier 1 tasks will systematically overstate an agent’s readiness for production, because Tier 3 failure modes — wrong target selection, missing rollback, race conditions with concurrent human operators — simply don’t appear in Tier 1 scenarios. Insist on seeing Tier 3 evaluation results before granting any agent write access to production systems.

Environment fidelity

Benchmarks run against static, pre-recorded transcripts (replay-mode evaluation) are cheap and reproducible but cannot catch failures that only emerge from live system dynamics: rate limiting, eventual consistency, flaky APIs, or an environment that changes state mid-task because another process (or another agent) touched it concurrently. Live-environment benchmarks, run against sandboxed but functionally real infrastructure — a disposable Kubernetes cluster, a synthetic Active Directory forest, a scaled-down SIEM with realistic log volume — catch a materially different and generally more severe class of bug. A mature evaluation program runs both: replay benchmarks for fast, cheap regression testing on every model or prompt change, and live-environment benchmarks on a slower cadence (weekly or per-release) to catch integration and timing failures that replay cannot surface.

Adversarial pressure

Security-facing agents need a benchmark category that generic agent evaluation ignores entirely: does the agent hold up when the environment is actively trying to deceive it? This includes prompt injection embedded in log data or ticket text, alert floods designed to exhaust an agent’s context or budget, and adversarial log entries crafted to make a compromised host look benign. This is the same discipline that underlies continuous threat exposure management programs applied to the agent itself — you are running an adversarial validation cycle against your own automation, not just against your network.

Time horizon

Short-horizon benchmarks (a task completes in one session, typically under a few minutes of wall-clock agent time) are the norm today, but a growing share of valuable operational work is long-horizon: a capacity-planning agent that needs to observe a system for a week before recommending a scaling change, or a threat-hunting agent that maintains a standing hypothesis across days of intermittent evidence. Long-horizon evaluation requires the benchmark harness to support pausing, state persistence, and re-invocation of the agent against updated context — a capability most off-the-shelf agent-eval frameworks still lack.

Building an evaluation harness: architecture

An evaluation harness for agentic IT/security systems needs five components working together, and most of the teams that struggle with agent evaluation are missing at least two of them.

The first is a scenario library: a versioned, growing collection of task definitions, each specifying the starting environment state, the goal given to the agent, the ground-truth or acceptable-outcome criteria, and any adversarial elements to inject. Scenario libraries should be sourced from three places in roughly equal measure — synthetic scenarios authored by SMEs to cover known-important edge cases, replayed real incidents from your own environment (anonymized and reduced to their essential structure), and adversarially generated scenarios designed specifically to find the agent’s blind spots. A scenario library that is 100% synthetic will overfit to what your engineers already imagined; one that is 100% replayed history will systematically under-test novel attack patterns and infrastructure changes that haven’t happened yet.

The second component is an environment sandbox capable of instantiating the starting state for each scenario, exposing the same tool interfaces the agent uses in production (ideally the identical API surface, not a mock), and capturing every side effect for later scoring. The temptation to mock tool calls for speed is understandable but dangerous: mocked environments cannot catch the failure modes that live-environment benchmarks exist to find, and a harness that only ever tests against mocks will certify agents that fail on first contact with a real system’s quirks.

The third is the trace collector described in the previous section, wired to capture every plan/act/verify event with enough fidelity to reconstruct the full trajectory after the fact.

The fourth is a scoring layer that combines automated checks (did the expected state change occur, was the correct tool called, did the action stay within policy bounds) with LLM-as-judge scoring for the parts of the trajectory that resist purely mechanical evaluation (was the reasoning sound, was the final explanation to the human operator clear and accurate). LLM-as-judge scoring is powerful but needs its own calibration: run the judge against a held-out set of human-graded traces regularly and track judge-human agreement as its own metric, because judge drift is real and silent.

The fifth is a regression gate integrated into your deployment pipeline: no change to a prompt, model version, tool definition, or retrieval index should reach production without running the full replay benchmark suite and a sample of live-sandbox scenarios, with hard thresholds that block promotion on regression. This is the same discipline as unit and integration testing in conventional software engineering, applied to a system whose behavior is probabilistic rather than deterministic — which means your thresholds need to account for variance across repeated runs, not just a single pass/fail.

Success metrics that actually predict production outcomes

Most agent metrics reported in vendor demos measure the wrong layer of the system. Task-completion rate on a benchmark tells you about the model and prompt; it tells you almost nothing about mean time to resolution in your environment, analyst trust, or the rate of costly autonomous mistakes. A useful metrics hierarchy separates three layers: trajectory-level metrics (did the agent reason and act well, step by step), task-level metrics (did the overall goal get achieved, and at what cost), and operational-level metrics (what happened to the business as a result of running this agent continuously in production).

Trajectory-level metrics

• Step validity rate — the fraction of individual actions that were well-formed, permitted by policy, and directed at a plausible target, independent of whether the overall task succeeded.
• Tool-selection accuracy — whether the agent picked the tool a domain expert would have picked given the same context, scored against a labeled reference set.
• Verification coverage — the fraction of state-changing actions that were followed by an explicit verification step before the agent proceeded or closed the task. This is one of the single best predictors of downstream harm; agents that skip verification after action are the ones that leave systems in a broken or half-changed state.
• Self-correction rate — how often the agent detected its own error mid-trajectory and corrected course versus needing external intervention.

Task-level metrics

• Task success rate, ideally reported separately for fully autonomous completion versus completion with human-in-the-loop approval, since these represent very different risk profiles even when the number looks similar.
• Time-to-resolution, compared against a human-operator baseline on the same task distribution, not against a generic industry average.
• Cost per task, including compute/token cost and the fully loaded cost of any human review the task required — agentic automation that is technically successful but triples token spend per ticket resolved is not obviously a win.
• Escalation precision — when the agent decides a task is beyond its competence and hands off to a human, how often was that the right call? Both over-escalation (erodes trust in autonomy, defeats the purpose) and under-escalation (the dangerous failure mode) need to be tracked as distinct rates.

Operational-level metrics

• Mean time to detect / respond / remediate (MTTD/MTTR), tracked as a rolling trend after agent deployment, not a one-time snapshot.
• False-positive suppression rate without missed true positives — the metric that matters most for alert-triage agents deployed in AI-driven XDR alert triage, since the entire value proposition collapses if suppression comes at the cost of missed detections.
• Analyst trust and override rate — what fraction of the agent’s recommendations do human operators accept versus override, tracked over time. A rising override rate is an early warning sign that model or environment drift is degrading quality well before task-success metrics catch up.
• Blast-radius containment — for Tier 3 remediation actions, the measured scope of impact when something goes wrong, compared against the theoretical maximum blast radius the agent’s permissions would allow. This is a metric about your guardrail design as much as the agent’s behavior.

The discipline that matters most here is refusing to let any single number stand in for “the agent is working.” A dashboard with a single task-success percentage is a marketing artifact, not an evaluation system. Insist on seeing the trajectory, task, and operational layers together, because a system can look excellent at one layer while quietly failing at another — high task-success rate with catastrophically low verification coverage is the classic pattern behind an agent that appears production-ready in a demo and then causes an outage in week three.

Guardrails and verification layers

Evaluation and guardrails are two sides of the same coin: evaluation tells you where an agent is likely to fail, and guardrails are what you build in response. A layered guardrail architecture for operational agents typically has four bands, from least to most restrictive.

The first band is policy-scoped tool access: the agent is only ever given credentials and tool bindings scoped to the minimum needed for its task class, following the same least-privilege discipline that underlies mature privileged access management programs. An agent that investigates alerts should never hold the same credential as one that is authorized to isolate a host, and both should be distinct from whatever credential can modify identity or access policy. This is not a nicety — it is the single most effective bound on worst-case blast radius, because it limits what a misjudging or manipulated agent is even capable of doing regardless of what it decides to do.

The second band is pre-action policy checks: before any state-changing action executes, a deterministic policy engine (not the LLM itself) validates the action against a rule set — is this target host in an approved list, is this the correct blast-radius tier for the current approval level, is this action being taken outside of a defined change window that requires human sign-off. This layer should be boring, auditable, non-probabilistic code, precisely because everything upstream of it is probabilistic.

The third band is post-action verification, which the trajectory metrics above already flagged as one of the strongest predictors of safe operation. Verification should be independent of the agent’s own self-report — query the actual system state, don’t trust the agent’s claim that the action succeeded. A pod-restart agent that asks Kubernetes “did the pod restart” rather than asking the LLM “do you think the pod restarted” is the difference between a verifiable system and a hopeful one.

The fourth band is human-in-the-loop gates calibrated by task tier and confidence. Tier 1 tasks with high model confidence can run fully autonomously; Tier 3 tasks, or any task where the agent’s own confidence signal is below a calibrated threshold, should route to a human approver before the action executes, not after. Confidence-threshold calibration itself needs to be evaluated — a model that reports 95% confidence but is empirically right only 70% of the time at that confidence level is miscalibrated, and you will not catch that without dedicated calibration testing against your scenario library.

A well-designed guardrail stack does not eliminate agent errors; it bounds their consequences and makes them observable. That reframing matters for how you evaluate readiness: the question is never “will this agent ever make a mistake” (it will) but “when it does, is the mistake caught before it causes harm, and is the harm bounded to a scope you already decided was acceptable.”

A failure mode taxonomy worth testing for explicitly

Generic evaluation tends to lump all agent errors into “task failed,” which hides the information you actually need to fix the system. A more useful taxonomy separates failures by mechanism, because each mechanism calls for a different fix.

• Planning failures — the agent decomposed the goal incorrectly or chose a strategy a domain expert would not have chosen. Fixed by better few-shot examples, retrieval of relevant playbooks, or fine-tuning on operational task decompositions.
• Grounding failures — the agent’s plan was reasonable but its factual premises about the environment were wrong, often because retrieved context was stale, incomplete, or from the wrong system of record. Fixed by improving retrieval quality and freshness, not by touching the reasoning process at all.
• Tool-use failures — correct intent, wrong or malformed tool invocation. These are often the easiest to fix and the easiest to catch mechanically, because tool schemas can be validated deterministically before execution.
• Verification failures — the agent acted correctly but failed to confirm the action worked, or confirmed it against the wrong signal. This is the failure mode most correlated with silent, compounding damage, because the agent proceeds as though everything is fine.
• Context-window and memory failures — on long-horizon tasks, the agent loses track of earlier findings, contradicts its own earlier hypothesis, or re-does work it already completed. These require architectural fixes: structured memory stores, summarization checkpoints, or explicit state objects that persist outside the model’s context window.
• Adversarial and manipulation failures — the agent was actively deceived by injected content, spoofed logs, or crafted inputs designed to trigger a specific unsafe action. These require the adversarial benchmark category described earlier and input sanitization layers independent of the model.

Tagging every failure in your evaluation results against this taxonomy, rather than recording a flat pass/fail, turns your evaluation program from a scorecard into a diagnostic tool. It also lets you track whether a given class of failure is actually improving release over release, which a single aggregate success-rate number cannot show you — an agent can hold steady at 85% task success while quietly trading planning failures for verification failures, which is a much worse trade from a risk standpoint even though the headline number looks unchanged.

Human-in-the-loop calibration and the trust curve

The relationship between an operations team and an agentic system follows a predictable trust curve, and evaluation programs need to be designed around where a given agent sits on that curve rather than treated as a one-time certification. Early in deployment, human operators review nearly every recommendation, which is exactly the regime in which you should be collecting the human-graded case reviews that calibrate your LLM-judge scoring and your confidence thresholds. As trust builds and metrics hold up, autonomy should expand task by task, tier by tier — never as a blanket “turn on full autonomy” switch.

The mistake teams make most often is expanding autonomy based on a shrinking override rate alone, without checking whether the override rate dropped because the agent got better or because operators got fatigued and stopped reviewing carefully — a phenomenon sometimes called automation complacency. The way to distinguish these is to keep a small, randomized, mandatory human-review sample even after an agent earns expanded autonomy, sized so you retain statistical power to detect quality regression, and to track review thoroughness (time spent per review, rate of substantive comments) as its own signal, not just the accept/override decision.

Calibration also needs to run in both directions. Just as an agent can be granted too much autonomy too fast, teams sometimes leave an agent permanently pinned to heavy human review long after the evidence supports expanding its scope, which is its own cost: it caps the return on the automation investment and burns analyst time on judgments the agent has demonstrably learned to make well. The evaluation program’s job is to make that expand/contract decision an evidence-based one, reviewed on a fixed cadence, rather than a one-off leap of faith or an indefinite freeze born of initial caution.

Continuous evaluation in production, not just at launch

Pre-launch benchmarking answers “is this agent ready to deploy.” It cannot answer “is this agent still working correctly six months later,” and in practice the answer to that second question drifts for reasons entirely outside the agent’s own code: the underlying model provider ships a silent update, your infrastructure topology changes, a new class of alert appears that wasn’t in the original scenario library, or attacker techniques evolve past what your adversarial benchmarks anticipated. Continuous production evaluation closes this gap through four mechanisms.

First, shadow-mode replay: a fraction of live production tasks are re-run against the current model/prompt version in a sandboxed shadow environment before any change is promoted, and the outputs are diffed against what the currently deployed version would have done, surfacing behavioral drift before it reaches customers or infrastructure. Second, canary deployment with tiered rollback triggers: new agent versions run on a small percentage of real traffic with tighter human-review gates and automatic rollback if trajectory or task metrics fall outside a statistically defined band relative to the incumbent version. Third, periodic scenario-library refresh: every incident that required unusual human intervention, every near-miss, and every genuinely novel alert pattern should be triaged for inclusion in the scenario library, so the benchmark suite itself evolves with the threat and infrastructure landscape rather than calcifying around whatever was true at initial launch. Fourth, metric-drift alerting: the operational metrics described earlier — override rate, escalation precision, verification coverage — should have their own statistical process control, with alerts fired on trend breaks, not just absolute threshold breaches, since a slow six-week drift is exactly the pattern a fixed threshold is least likely to catch.

This continuous-evaluation discipline is also where agentic evaluation converges with broader exposure-management practice: both are fundamentally about running a standing adversarial and drift-detection process against a system whose risk surface changes continuously, which is the same posture organizations already apply to their infrastructure attack surface through programs like continuous threat exposure management. Applying that same posture to the agent stack itself — treating the agent as an asset with its own exposure surface, its own drift risk, and its own continuous validation cycle — is the maturity marker that separates teams running agentic automation safely at scale from teams that deployed it once and hoped.

An adoption maturity model for agentic operations

Organizations evaluating and adopting agentic AI for IT and security operations tend to move through four recognizable stages, and the evaluation rigor required is different at each one.

1. Stage 1 — Assisted investigation. Agents gather and correlate evidence but make no recommendations and take no actions; a human reads the assembled context and decides everything. Evaluation focus: completeness and accuracy of evidence gathering, measured against SME-labeled ground truth.
2. Stage 2 — Recommended action, human executes. The agent proposes a specific remediation or disposition, and a human operator reviews and manually executes it. Evaluation focus shifts to recommendation quality and, critically, to override-rate tracking, since this is the stage where you build the labeled dataset that calibrates confidence thresholds for the next stage.
3. Stage 3 — Tiered autonomous execution. Low-risk, high-confidence task classes execute autonomously behind the guardrail bands described earlier; higher-risk classes still route to human approval. Evaluation focus expands to trajectory metrics, verification coverage, and blast-radius containment, because the agent now has direct write access to production systems.
4. Stage 4 — Continuous autonomous operation with standing oversight. Most task classes run autonomously with the continuous production evaluation program described above as the primary safety mechanism, and human review is concentrated on genuinely novel or high-severity cases plus the randomized quality-audit sample. Evaluation focus is now dominated by drift detection and scenario-library evolution rather than initial certification.

Skipping stages is the most common and most costly mistake in agentic adoption. Teams that go straight from a Stage 1 pilot to Stage 3 production autonomy because a demo looked impressive are, in effect, deploying an agent whose Tier 3 failure modes were never benchmarked against live infrastructure, whose confidence thresholds were never calibrated against real override data, and whose guardrail bands were designed on paper rather than validated against actual near-miss incidents. The maturity model exists precisely to force each of those gaps to be closed with evidence before the next stage begins.

Evaluating vendors and platforms, not just models

Everything above applies whether you build agentic capability in-house or buy it, but buying introduces an additional evaluation question: does the platform itself expose the traces, benchmarks, and metrics you need, or does it hide them behind a black-box “resolution rate” number? When evaluating an agentic platform for IT operations, security operations, or both, insist on direct answers to a short list of architectural questions before any pilot begins.

Can the platform export a full plan-act-verify trace for every task, in a format your own tooling can ingest and score independently of the vendor’s own dashboards? Does it support a live-sandbox evaluation mode against your actual environment topology, or only replay-mode demos against the vendor’s own curated scenarios? Does it expose tiered autonomy controls mapped to the guardrail bands described earlier, with policy checks enforced outside the model rather than inside a prompt? Does it support the deployment model your governance requires — cloud, on-premises, or fully air-gapped, since sovereign and regulated environments frequently cannot accept any architecture that depends on external inference calls for evaluation or operation? And critically, does it let you run your own scenario library and your own scoring criteria against it, rather than only reporting the vendor’s internal benchmark numbers, which are almost always measured on tasks curated to flatter the product?

This is the standard an AI-native operations stack needs to meet to be trustworthy for autonomous execution rather than just assisted investigation: traceability by default, guardrails enforced by deterministic policy rather than model discipline alone, and evaluation surfaces open enough that a customer can independently verify the claims rather than trusting a leaderboard number. Products built across the ITMox, CyberMox, Norra, and MoxDB lines are designed around exactly this discipline — agents that operate across IT operations and security operations workflows, coordinated through the same agentic workforce layer and grounded against the same unified data foundation, so that a trace generated by an alert-triage agent and a trace generated by a remediation agent share one schema, one scoring pipeline, and one audit trail rather than three incompatible vendor dashboards. Whatever platform you evaluate, the bar should be the same: if you cannot independently reproduce the vendor’s evaluation numbers against your own scenarios, you have not evaluated the product — you have read its marketing.

It is also worth pressure-testing platforms specifically on identity and exposure workflows, since these are the task classes where blast radius is highest and evaluation rigor matters most. An agent operating in identity security and privileged access contexts, or one performing autonomous detection and response actions, should be able to show you specifically how its guardrail bands and verification steps differ between a low-privilege investigation task and a high-privilege remediation task — if the vendor describes only one undifferentiated autonomy level across every task type, that is itself a finding worth weighting heavily against adoption.

A worked example: evaluating an alert-triage agent end to end

Concretely, consider evaluating an agent whose job is to triage SIEM alerts for a mid-size security operations team: ingest the alert, correlate against EDR and identity telemetry, form a disposition (true positive requiring escalation, benign, or needs more data), and either close the alert, escalate with a written summary, or request specific additional evidence.

The scenario library for this agent should include at minimum: a set of clear true positives spanning your top attack techniques (labeled against your MITRE ATT&CK coverage map), a set of clear benign alerts including the specific false-positive patterns your environment historically generates, a set of genuinely ambiguous cases that even experienced analysts disagree on (used to test calibrated escalation rather than binary correctness), and an adversarial subset with injected prompt content in log fields, spoofed process names designed to look like known-good software, and alert floods designed to test whether the agent’s per-alert quality degrades under volume pressure.

The trace for each run should capture which telemetry sources the agent queried and in what order, what hypothesis it formed after each query, what evidence it weighted most heavily in its final disposition, and its stated confidence. Scoring combines mechanical checks (did it query all telemetry sources a human analyst would consider mandatory for this alert type, did its final disposition match ground truth) with LLM-judge scoring of the written summary quality for escalated cases, since a technically correct escalation with an unclear or misleading summary still burns analyst time and erodes trust.

Success metrics tracked over the pilot: task success rate broken out by true-positive, benign, and ambiguous categories separately (a single blended number would hide if the agent is, say, excellent at benign suppression but weak at true-positive detection, which is the worst possible profile for a triage agent); escalation precision measured against blinded analyst review of a sample of escalated and non-escalated alerts; false-negative rate on the true-positive set held to a near-zero tolerance given the cost asymmetry of missed detections versus wasted analyst review time; and override rate tracked weekly through the Stage 2 recommend-only period before any autonomous closure of alerts is permitted. Only after several weeks of stable metrics across all of these — not just an aggregate accuracy number — would this agent be a reasonable candidate for Stage 3 autonomous closure of the clear-benign category, with true-positive and ambiguous categories remaining human-gated indefinitely or until a separately calibrated confidence threshold earns them their own promotion.

Common pitfalls in agent evaluation programs

A handful of mistakes recur often enough across evaluation programs to call out explicitly. Teams frequently evaluate against benchmarks built entirely from historical incidents, which systematically under-tests novel scenarios and gives false confidence precisely where confidence is least warranted. Teams frequently collapse multi-step trajectories into a single pass/fail outcome, discarding the trace information that would have told them which mechanism failed and how to fix it. Teams frequently let an LLM judge grade an LLM agent without ever calibrating that judge against human-graded ground truth, which can silently inflate or deflate scores for months before anyone notices the discrepancy. Teams frequently expand autonomy based on a shrinking override rate without checking whether that shrinkage reflects genuine improvement or reviewer fatigue. And teams frequently treat evaluation as a pre-launch gate rather than a continuous process, missing the drift that model updates, infrastructure changes, and evolving attack patterns inevitably introduce after the initial certification is long forgotten.

Each of these pitfalls has the same underlying cause: treating agent evaluation as a simpler problem than it is, because the tooling and habits carried over from evaluating classifiers and chatbots make it feel familiar. It is not the same problem. It requires trace-level visibility, multi-layer metrics, adversarial and live-environment testing, calibrated human oversight, and a continuous rather than one-time evaluation cadence. Teams that build this discipline early spend real engineering effort on it that does not show up in a product demo — and that effort is exactly what separates an agent that is safe to hand a production credential to from one that only looks safe in a sales deck.

Frequently asked questions