Building a Reliable Agent Runtime: Retries, Timeouts and Idempotency

An agent that plans brilliantly but fails silently on a network blip, double-fires a remediation, or hangs forever waiting on a stuck API call is not an autonomous system — it is a liability wearing an autonomy costume. Reliability in agentic AI is not a bolt-on feature; it is the runtime discipline of retries, timeouts and idempotency that turns a clever planner into something a SOC or NOC can actually trust to touch production.

Why reliability, not intelligence, is the hard part

Most of the public conversation about agentic AI focuses on the reasoning layer — how well a large language model decomposes a goal, selects a tool, or reflects on an intermediate result. That work matters, but in production IT and security operations the reasoning layer is rarely what breaks a deployment. What breaks deployments is the boring machinery around the reasoning: an API call to a ticketing system that times out after fourteen seconds, a firewall rule push that succeeds on the device but never returns an acknowledgment, a script that runs twice because a retry fired before the first attempt's side effects were confirmed. These are not edge cases. In any environment with real network variance, rate limits, and third-party APIs with inconsistent SLAs, they are the median case.

The distinction that matters here is between an agent's cognitive loop — plan, act, observe, revise — and its execution substrate — the runtime that actually invokes tools, waits for responses, handles partial failures, and decides whether an action already happened. Vendors and open-source frameworks alike have spent enormous energy on the cognitive loop: chain-of-thought, tool selection, memory, multi-agent orchestration. Comparatively little attention goes to the execution substrate, yet it is the substrate that determines whether an agent can be trusted with a production credential. A planner that occasionally picks a suboptimal tool costs you efficiency. An executor that occasionally double-executes a password reset, a firewall block, or a database rollback costs you an incident review and possibly a customer.

This is the lens through which Algomox designs agent runtimes across ITMox and CyberMox: every autonomous action an agent takes — closing a ticket, isolating a host, rotating a credential, restarting a service — passes through a runtime layer that treats failure as the default expectation, not the exception. The rest of this article is a practical anatomy of that layer: how retries should actually be structured, how timeouts should be chosen and enforced, how idempotency is designed into actions before an agent ever calls them, and how all three combine into a runtime that SREs, SOC analysts and IT operators can reason about and audit.

Anatomy of an agent runtime

Before getting into retries, timeouts and idempotency individually, it helps to lay out where they sit in the overall architecture. An agent runtime for IT and security operations typically has five layers, and reliability mechanisms live primarily in the middle three.

At the top sits the planning layer, where an LLM or a hybrid planner decomposes an operator's goal or a triggering event — an alert, a ticket, a policy violation — into a sequence or graph of candidate actions. Below that is the orchestration layer, which sequences those actions, manages dependencies between them, and decides what happens when one step fails: retry, skip, escalate, or roll back. Below orchestration is the execution layer, the actual tool-calling substrate that issues API calls, CLI commands, or RPA-style UI interactions against the systems of record — a CMDB, an EDR console, a firewall, a directory service. Underneath that is a state and ledger layer that records what was attempted, what succeeded, what is still pending, and what the verified end-state was. And at the base is the connector layer, the actual adapters to ServiceNow, Splunk, CrowdStrike, Palo Alto, Active Directory, Kubernetes, and the hundred other systems that make up a real enterprise estate.

Retries, timeouts and idempotency are not features of any single layer — they are cross-cutting concerns that must be coherent across orchestration, execution and the ledger. A retry policy configured in the orchestration layer is meaningless if the execution layer does not tag each attempt with a unique idempotency key. A timeout enforced at the connector layer is dangerous if the orchestration layer does not know whether "timeout" means "definitely failed" or "unknown, possibly succeeded." This is why runtime reliability has to be designed as a system, not patched in as individual try/catch blocks scattered through connector code.

Retries done right: beyond "try again"

The naive implementation of a retry is a for-loop: call the tool, catch the exception, call it again, up to some maximum count. This is worse than doing nothing in several concrete ways, and it is worth being precise about why.

First, a naive retry conflates transient failures with permanent ones. A 503 from an overloaded API gateway is transient and worth retrying. A 403 because the agent's service account lacks permission to modify a firewall object is permanent, and retrying it five times just delays the escalation the human operator actually needs to see. An agent runtime needs an explicit error taxonomy at the connector layer that classifies every failure into at least three buckets: retryable (network timeout, 429 rate limit, 502/503/504), non-retryable (403 authorization, 400 malformed request, 404 target not found), and ambiguous (connection reset mid-request, where the server may have processed the request before the connection dropped). Only the first bucket should trigger automatic retry. The third bucket is the hardest and is where idempotency becomes load-bearing, discussed below.

Second, naive retries with fixed intervals create synchronized retry storms. If ten thousand agent-driven remediations all hit a rate-limited API at once, fail, and all retry after exactly two seconds, they will collide again. The standard mitigation is exponential backoff with jitter: each retry waits roughly base * 2^attempt, with a random jitter factor (typically full jitter, where the actual wait is a uniform random value between zero and the computed backoff) to de-synchronize concurrent retriers. For an agent runtime orchestrating thousands of concurrent actions across a large estate — patch pushes, credential rotations, alert triage actions — this is not optional; without jitter, backoff simply reproduces the thundering-herd problem at a longer period.

Third, and most overlooked: retries need a budget, not just a count. A count-based retry (say, three attempts) behaves very differently depending on the backoff schedule and the timeout per attempt — three attempts at a five-second timeout with linear backoff might resolve in fifteen seconds, while three attempts at a sixty-second timeout with exponential backoff might take four minutes. For operator-facing agent workflows, especially in a SOC context where mean-time-to-respond is a tracked metric, the retry policy should be expressed as a total time budget for the action (for example, "attempt for up to ninety seconds, then escalate"), with the per-attempt timeout and backoff schedule derived from that budget. This makes the runtime's behavior predictable to the humans who have to reason about SLAs.

Retry policy by action class

Not every action deserves the same retry treatment, and a mature runtime differentiates by the operational cost of retrying versus the operational cost of not retrying quickly enough. A useful way to think about this is to bucket actions into three classes.

• Read-only queries — pulling asset inventory from a CMDB, querying a SIEM for related events, checking a vulnerability scanner's last scan date. These are safe to retry aggressively (five or more attempts, short backoff) because they have no side effects. Failure here should almost never surface to a human; the runtime should just keep trying within budget.
• Reversible mutating actions — adding a comment to a ticket, tagging an asset, opening a case in a case management system. These can be retried moderately (two to four attempts) as long as the action is idempotent (see below), because a duplicate is annoying but not dangerous.
• High-consequence mutating actions — isolating a host from the network, disabling a user account, pushing a firewall rule change, rotating a production credential, restarting a service that carries customer traffic. These deserve a conservative retry policy (one retry, sometimes zero) combined with mandatory human confirmation on ambiguous outcomes, because the cost of a duplicate or out-of-order execution is high and the cost of waiting an extra thirty seconds for a human to glance at a confirmation prompt is low.

This tiering should not be improvised per-agent. It belongs in a policy layer that the orchestration engine consults before deciding whether to retry, and it is exactly the kind of guardrail that platforms like Algomox's AI-native stack encode centrally, so that every agent — whether it is triaging alerts through AI-driven XDR triage or executing a NOC remediation — inherits the same consequence-tiered retry behavior rather than each tool integration reinventing its own ad hoc logic.

Timeouts as a first-class design decision, not a default value

Every HTTP client ships with a default timeout, and every agent framework's tool-calling wrapper inherits one whether or not anyone chose it deliberately. This is one of the most common silent failure sources in agent runtimes: a connector to an EDR API has a thirty-second default timeout baked into the underlying SDK, but the EDR's isolation endpoint can legitimately take ninety seconds to acknowledge under load. The agent times out, the orchestrator marks the step as failed, retries, and now there is a real chance that the first isolation call actually succeeds a few seconds after the timeout fired — and the second call either duplicates the action or, worse, races with a rollback the orchestrator has meanwhile decided to trigger.

Timeouts need to be chosen per action, not per client library, based on the actual observed latency distribution of the downstream system, not on a guess. The practical process is: instrument every connector call with latency histograms in production for at least a few weeks, then set the timeout at roughly the 99th or 99.5th percentile of observed successful-call latency, with a floor that accounts for network variance. Setting a timeout at the median latency guarantees you will time out a meaningful fraction of perfectly healthy calls; setting it too generously (say, five minutes for everything) means a single hung dependency can stall an entire orchestration graph and blow through the SLA the human operator is expecting.

There is also a structural distinction that matters more in agent runtimes than in typical request-response services: the difference between a connection timeout, a per-attempt timeout, and an end-to-end action timeout. The connection timeout bounds how long the runtime waits to establish a TCP/TLS session with the target API — this should be short, typically one to five seconds, because a slow connection almost always indicates a network or DNS problem rather than a busy server. The per-attempt timeout bounds a single call once connected, and should reflect the downstream system's real processing time. The end-to-end action timeout bounds the entire retry sequence for a logical action ("isolate host X") including all attempts and backoff waits, and is the number that should map to the SLA an analyst or IT operator actually cares about.

A frequently missed nuance: for long-running remote operations — a patch deployment, a full disk scan, a large data export — the correct pattern is not a long synchronous timeout at all. It is an asynchronous submit-then-poll pattern, where the agent submits the job, receives a job ID immediately, and polls (with its own bounded, retryable poll loop) or subscribes to a webhook/callback for completion. Treating a five-minute patch job as a single five-minute synchronous HTTP call with a five-minute timeout is fragile: any transient network blip during those five minutes loses all progress and forces a full restart. Submit-then-poll decouples the reliability of the request-issuing call (short, retryable) from the reliability of the long-running job's own execution, which the downstream system is usually much better positioned to track than the agent is.

Timeout tuning is not a one-time exercise. Downstream API performance drifts as vendors change infrastructure, as your own request volume grows, and as network paths change (a firewall migration, a new proxy, a move to a private link). A reliable agent runtime treats timeout thresholds as configuration that is reviewed on the same cadence as capacity planning, and ideally auto-tunes per-attempt timeouts from a rolling latency percentile rather than hardcoding a value that was correct on the day someone wrote the connector.

Idempotency: the real safety net underneath retries

Retries and timeouts are coping mechanisms for an uncertain network. Idempotency is what makes coping mechanisms safe. An idempotent action is one where executing it once has the same observable effect as executing it two, three, or ten times. Without idempotency, every retry is a bet that the first attempt actually failed rather than merely appearing to fail because the response was lost in transit — and in distributed systems, that bet loses often enough to matter.

The starting point is an idempotency key: a unique identifier generated once per logical action (not per HTTP attempt) and carried through every retry of that action. The downstream system — or, if the downstream system does not natively support this, a shim the agent runtime maintains — uses that key to detect duplicate submissions and return the original result instead of re-executing the side effect. Many modern APIs (Stripe's Idempotency-Key header is the canonical example, and increasingly ticketing systems, cloud provider APIs, and SOAR platforms support equivalents) provide this natively. Where the downstream system does not, the agent runtime itself must maintain an idempotency ledger: before issuing a mutating call, check whether a call with this key has already been recorded as completed or in-flight; if completed, return the cached result; if in-flight, either wait or refuse to duplicate; if new, record the attempt and proceed.

Idempotency keys alone are not sufficient, however, because many real-world operations are not naturally idempotent at the semantic level even if you dedupe the request. "Disable this user account" is naturally idempotent — disabling an already-disabled account is a no-op with the same end state. "Increment the failed-login counter by one" is not idempotent — if you cannot tell whether the first increment succeeded, retrying blindly overcounts. "Rotate this credential" is subtly non-idempotent: rotating twice produces two different new credentials, and if the first rotation succeeded but the response was lost, a naive retry rotates again and the credential the operator's automation expected is no longer valid. This is why idempotency has to be designed into the action's semantics, not just its transport.

Three patterns for building idempotent actions

The first pattern is desired-state actions rather than imperative commands. Instead of an agent issuing "add rule X to the firewall," it issues "ensure rule X exists in the firewall with these parameters." The downstream system (or an idempotent wrapper the runtime provides) checks current state before mutating, and a repeated call is a no-op if the desired state already holds. This is the same principle that underlies infrastructure-as-code tools like Terraform and Ansible, and it generalizes cleanly to security and IT operations: "ensure host X is isolated," "ensure user Y is disabled," "ensure ticket Z has status Resolved" are all naturally repeatable.

The second pattern is conditional writes with version checks, borrowed from optimistic concurrency control. The agent reads the current state along with a version token (an ETag, a row version, a last-modified timestamp), and the mutating call includes that token, succeeding only if the state has not changed since the read. If a retry occurs after the first attempt actually succeeded, the second attempt's version check fails harmlessly because the state has already moved — the agent can then treat "version mismatch" as "someone (possibly my earlier attempt) already made this change" and verify rather than blindly re-attempting.

The third pattern, needed for actions that are neither naturally idempotent nor easily wrapped in a desired-state check — credential rotation is the sharpest example — is outcome memoization at the ledger layer. Before executing, the agent runtime records "attempting rotation of credential C with request ID R." On any retry of the same logical action, before calling the downstream system again, it first checks the ledger: if request ID R is already marked completed, it fetches and returns the previously recorded outcome (the new credential's reference, not necessarily the secret itself) instead of rotating again. This requires the runtime, not the downstream system, to own the source of truth for "did this already happen," which is precisely the job of the state and ledger layer described earlier.

It is worth being blunt about the ambiguous-timeout case, because it is the one engineers most often get wrong under pressure. When a request times out, there are exactly three possibilities: the request never reached the server, the request reached the server but the response was lost, or the request reached the server and is still processing. A runtime that treats all three as "definitely failed, safe to retry" is gambling on the second and third cases not mattering. The correct behavior on an ambiguous timeout is to first attempt a reconciliation check — query the downstream system's current state for the target object using a read-only, always-safe call — before deciding whether to retry the mutation, retry the query, or escalate to a human. This single discipline, checking before re-acting, eliminates the majority of duplicate-execution incidents in production agent deployments.

Orchestrating multi-step workflows: sagas, compensations and partial failure

Individual actions being retryable and idempotent solves only part of the problem, because most real IT and security workflows are multi-step: contain a threat, gather forensic evidence, notify stakeholders, open a ticket, apply a fix, verify the fix, close the ticket. If step four fails after steps one through three succeeded, what should the runtime do? This is the classic distributed-transaction problem, and agent runtimes should borrow the answer that has already been proven at scale in distributed systems: the saga pattern.

A saga decomposes a multi-step workflow into a sequence of local actions, each paired with a compensating action that can semantically undo it if a later step fails. For an agent-driven incident response workflow, this might look like: isolate host (compensation: restore network access), snapshot disk for forensics (compensation: delete snapshot), disable compromised account (compensation: re-enable account with forced password reset), notify SOC channel (compensation: post retraction notice). If the workflow fails at the account-disable step, the orchestrator walks backward through the completed steps' compensations in reverse order, restoring the system to a known state rather than leaving it in whatever partial condition the failure happened to produce.

Not every action has a clean compensation, and this is where honest architecture beats aspirational architecture. Sending a notification cannot be un-sent. Deleting a piece of evidence cannot be un-deleted if you did not first archive it. The discipline this forces on agent designers is valuable in itself: for every action added to an agent's toolset, the runtime should require an explicit answer to "what is the compensating action, and if there isn't one, is this action safe to place after the workflow's point of no return?" Actions without compensations belong at the end of a workflow, or behind an explicit human confirmation gate, never in the middle of a chain where an earlier failure might need to unwind past them.

This is also where the distinction between orchestration-layer retries and workflow-layer compensation matters. A step-level retry (retry the API call three times with backoff) should exhaust before the orchestrator escalates to saga-level compensation (undo everything done so far). Conflating the two — triggering a full workflow rollback on the first transient timeout — produces a jumpy, unreliable-feeling system that undoes legitimate work because of a single slow API response. Algomox's approach in the agentic SOC context is to keep these as genuinely separate state machines: a per-step retry policy governs transient failure, and a workflow-level saga controller only engages after the step-level budget is exhausted or a non-retryable error is classified.

Verification: trust the observed state, not the status code

A subtle but critical principle: a 200 OK response is evidence that an action was accepted, not proof that it achieved its intended effect. An EDR isolation API might return success while the actual network isolation takes another ninety seconds to propagate to the endpoint agent. A firewall rule push might succeed on the management plane while failing to commit on a specific device in a high-availability pair. An agent runtime that treats "the API call returned 200" as "the goal is achieved" will confidently report success on actions that have not actually happened yet, and this gap is exactly where security incidents worsen and IT outages extend.

The fix is to make verification a distinct, mandatory step after every consequential action, separate from the action call itself. For isolation, this means querying the endpoint's actual network state, not just the isolation API's response. For a firewall change, this means querying the device's active rule set, not just the management console's push confirmation. For a service restart, this means checking the service's health endpoint or process state, not just the restart command's exit code. This verification step should itself be retryable and time-bounded (the effect may take a few seconds to propagate), but it must exist as a separate, auditable check, and its result — not the original action's response — is what determines whether the workflow proceeds, retries the action, or escalates.

This verify-don't-trust principle compounds with idempotency in a useful way: because verification is itself just a read-only, safe, freely-retryable operation, it is the natural mechanism for resolving ambiguous timeouts described earlier. When a mutating call times out ambiguously, the runtime's first move should always be verification against observed state, and only escalate to a retry of the mutation if verification shows the desired state was not achieved. This single pattern — verify before retrying a mutation — is arguably the single highest-leverage reliability practice available to an agent runtime designer, because it converts every ambiguous failure into a deterministic decision grounded in actual system state rather than a guess about what probably happened.

Human-in-the-loop gates and blast-radius tiering

No amount of retry sophistication substitutes for a well-designed boundary on autonomy itself. A mature agent runtime tiers every action by blast radius and couples that tier to how much autonomous retry and execution latitude the agent gets before a human must confirm. Low blast-radius actions — enriching an alert with threat intelligence, tagging an asset, drafting a ticket comment — can run fully autonomously with aggressive retry. Medium blast-radius actions — opening a change request, quarantining a single low-criticality endpoint, disabling a non-privileged account — can run autonomously but should be logged prominently for near-real-time human review. High blast-radius actions — isolating a production database server, disabling a domain admin account, pushing a change to a core routing device — should require explicit human confirmation before the first attempt, with the runtime's reliability machinery (retries, timeouts, idempotency, verification) fully engaged only after that human gate.

This tiering is not just a safety feature; it is what makes the reliability engineering described above tractable to reason about. If every action in the system had unrestricted autonomous retry with no blast-radius awareness, then a bug in the retry logic for a single connector could, in the worst case, autonomously repeat a high-consequence action across an entire estate before anyone noticed. Blast-radius tiering bounds the worst case of a reliability bug to the tier it occurred in. This is a core reason platforms addressing identity and privileged access management or continuous threat exposure management insist on graduated autonomy: the runtime's technical reliability has to be matched by an authorization model that limits how much damage a reliability failure, however unlikely, can actually cause.

The practical mechanism is a policy engine that every agent action consults before execution, returning one of: auto-execute, auto-execute-with-notification, or require-approval. This policy should be driven by attributes of the action (which system, which operation type, which target's criticality tier in the CMDB) rather than by which agent or workflow happens to be calling it, so that the same "isolate this host" action carries the same approval requirement whether it is invoked by an alert-triage agent or a manually-triggered incident response playbook.

Observability: the metrics that actually predict reliability

You cannot manage what you do not measure, and generic uptime metrics do not capture the failure modes described above. An agent runtime needs its own metric surface, distinct from standard application performance monitoring, focused on the retry/timeout/idempotency layer specifically.

• Retry exhaustion rate — the percentage of actions that exhaust their full retry budget without succeeding. A rising trend here indicates either a genuinely degraded downstream dependency or a timeout/backoff configuration that no longer matches reality.
• Duplicate-suppression rate — how often the idempotency layer detects and blocks a would-be duplicate execution. This should not be zero; a healthy system experiences transient failures and its idempotency layer is catching the resulting retries. A rate that spikes suggests either an unstable network path or a downstream system whose acknowledgment latency has drifted past the configured timeout.
• Ambiguous-outcome rate — how often verification is required because a mutating call's result was unknown (timeout, connection reset). This directly measures how often the runtime is operating in the riskiest zone, and it is the number most worth showing on an SRE dashboard.
• Compensation trigger rate — how often a saga-level rollback fires. Every occurrence deserves a post-incident look, because it means a multi-step workflow got far enough to have side effects and then failed anyway.
• Time-to-verified-completion — not time-to-API-response, but time until the verification step confirms the desired state was actually reached. This is the metric that should feed SLA reporting, because it is the only one that reflects what actually happened in the target system.

These metrics belong in the same operational dashboards used for the rest of the IT and security estate, correlated against the systems the agents act on, so that a spike in ambiguous-outcome rate against a specific EDR connector is visible alongside that EDR vendor's own status page and the broader NOC/SOC view, rather than buried in an agent framework's internal logs that only the platform team ever looks at.

Testing reliability before production: chaos and fault injection for agents

Retry logic, timeout configuration and idempotency guarantees are exactly the kind of code that looks correct in review and fails only under conditions nobody thought to simulate. The standard mitigation from distributed systems engineering — fault injection and chaos testing — applies directly and should be a mandatory pre-production gate for any agent that will take mutating actions.

Concretely, before an agent workflow that touches production systems goes live, it should be exercised against a test harness that deliberately injects: dropped responses after the request has been recorded as received by the target system (to test ambiguous-timeout handling and idempotency), duplicate delivery of the same request (to test idempotency-key deduplication directly), slow responses just past and just under the configured timeout threshold (to test that timeouts trigger at the right boundary and that legitimately slow-but-successful calls are not falsely killed), and out-of-order delivery where a retry's response arrives before the original attempt's response (to test that the runtime does not act on stale data). Running this suite is far cheaper than discovering the same gaps during a live incident, and it should be re-run whenever a connector's timeout values, retry policy, or idempotency implementation changes, not just once at launch.

It is also worth deliberately testing the human-facing side of failure: when an action does exhaust retries and escalates, does the operator receive a clear, actionable notification that explains what was attempted, what state the system was left in, and what decision is needed — or does it surface as an opaque stack trace in a log nobody is watching? Reliability engineering that stops at the machine boundary and does not extend to how failure is communicated to the SOC analyst or IT operator on shift has only solved half the problem.

Adoption guidance: getting from pilot to trusted production runtime

Organizations evaluating agentic AI for IT operations or security typically start with read-only or low-blast-radius pilots — alert enrichment, ticket summarization, log correlation — and the reliability concerns discussed here are almost invisible at that stage, because a failed enrichment call just gets logged and retried without consequence. The risk is that teams then extend the same lightly-engineered runtime to mutating, high-consequence actions without revisiting the retry, timeout and idempotency architecture, on the assumption that if the pilot worked, the production rollout will too. It will not, because the pilot's action set never exercised the failure modes that matter.

A more disciplined adoption path treats the introduction of every new mutating capability as a gate: before an agent is allowed to call a new mutating action for the first time in production, the team should be able to answer, in writing, what the action's idempotency guarantee is, what its compensating action is (or why it does not need one), what its timeout values are and how they were derived, what its blast-radius tier is and what approval gate that implies, and what its verification step checks. This is not bureaucratic overhead for its own sake — it is the minimum documentation needed for an SRE or SOC lead to actually trust the action running unattended at 3 a.m.

It is also worth sequencing capability rollout by reversibility rather than by apparent business value. Teams are often tempted to prioritize the highest-value automation first — automatic account lockout for compromised credentials, automatic firewall rule remediation — because that is where the ROI story is clearest. But these are also the highest blast-radius, hardest-to-compensate actions. A more resilient path builds the reliability muscle on reversible, low-consequence actions first (ticket routing, enrichment, evidence collection), validates the retry/timeout/idempotency machinery there under real production load, and only then extends the same, now-proven runtime to higher-consequence actions, rather than building bespoke reliability handling for the highest-stakes action first under deadline pressure.

Finally, reliability is not a property you achieve once. Downstream APIs change their latency characteristics, rate limits tighten or loosen, and new failure modes emerge as an estate grows. The runtime needs an owner — a team or role responsible for reviewing retry-exhaustion and ambiguous-outcome metrics on a regular cadence, adjusting timeout thresholds as observed latency drifts, and auditing whether blast-radius tiers still match the actual criticality of the systems they gate. Products like Norra, built as an agentic workforce operating continuously across IT and security functions, make this ownership question unavoidable in a useful way: because the agents run continuously rather than in scheduled batch jobs, the reliability of the runtime beneath them is not a one-time launch concern but an ongoing operational responsibility, reviewed with the same rigor as any other production service, including in air-gapped or sovereign deployments where automated telemetry export is restricted and reliability review has to happen against locally retained logs.

Worked example: an idempotent, retryable account-compromise response

To make the preceding principles concrete, consider a common SOC workflow: an identity provider emits a signal indicating a likely compromised account — impossible travel followed by a new MFA device registration. The agent's goal is to contain the account while preserving forensic value and notifying the right people, without acting twice or leaving the account in an inconsistent state if something fails partway through.

The workflow decomposes into five steps, each engineered against the principles above. Step one, gather context: read-only queries against the identity provider, EDR and CMDB to confirm the account's privilege level and associated assets — aggressively retryable, no idempotency concerns since nothing is mutated. Step two, force session revocation: a desired-state action ("ensure all active sessions for this account are revoked") that is naturally idempotent — calling it again when sessions are already revoked is a harmless no-op — with a moderate retry budget and a short per-attempt timeout, since session revocation APIs are typically fast. Step three, require step-up authentication on next login: another desired-state action, idempotent by construction, with its own idempotency key so a retry after an ambiguous timeout does not risk toggling a boolean twice into an unintended state. Step four, open an incident ticket with full context attached: this uses a runtime-generated idempotency key passed to the ticketing system's API (or, if unsupported, checked against the runtime's own ledger) so a retry after a lost response updates the existing ticket rather than creating a duplicate. Step five, notify the on-call analyst: not retried blindly, because a notification is not idempotent in the way that matters to a human (a person paged twice for the same event experiences it as noise, not safety) — instead the runtime checks its ledger for "notification already sent for this incident ID" before sending.

Each step is verified against observed state before the workflow proceeds: session revocation is confirmed by re-querying active session count, not by trusting the revocation API's response code; the ticket's existence is confirmed by its returned ID being resolvable, not merely by receiving a 201. If step two fails after exhausting its retry budget with a non-retryable error — say, the identity provider returns a permission error because the service account's role was recently narrowed — the saga controller triggers the workflow's compensation path: no session-revocation compensation is needed since nothing succeeded, but the workflow escalates directly to human notification with full context on what was attempted and why it stopped, rather than silently failing or blindly retrying against an error class that retry cannot fix. This is the shape — idempotent steps, tiered retry, explicit verification, saga-aware failure handling — that a reliable agent runtime produces for essentially any consequential IT or security workflow, whether the trigger is identity compromise, an exposure finding from continuous threat exposure management, or a routine data platform maintenance task.

Frequently asked questions