SLOs, Error Budgets and AI-Driven Reliability

Every reliability program eventually collides with the same wall: dashboards are green, alerts are firing, and nobody can say with confidence whether the service is actually healthy or how much risk is left to spend before the business feels it. Service level objectives and error budgets give that judgment a number. AI-driven operations give it a nervous system — one that senses drift before customers do, explains why, and in the best implementations, acts.

The reliability contract: SLIs, SLOs, and error budgets defined precisely

Most organizations that say they "do SRE" have monitoring, not a reliability contract. The distinction matters because a contract is enforceable: it defines what good looks like, how much badness is tolerable, and what happens when the tolerance is exceeded. The contract has three layers, and conflating them is the single most common mistake teams make.

A service level indicator (SLI) is a precisely defined, quantitative measure of some aspect of the level of service provided. It is not "is the API healthy" — it is "the proportion of HTTP requests to /checkout that complete in under 300ms with a 2xx or 3xx status, measured at the load balancer, over a rolling 28-day window." Precision is not pedantry here; it is the difference between an SLI you can alert on and a vibe you can argue about in a postmortem.

A service level objective (SLO) is the target value or range for that SLI over a compliance period — for example, 99.9% of checkout requests fast and successful over 30 days. The SLO is a promise made internally (and sometimes externally, as an SLA with contractual penalties) that shapes engineering priorities. Crucially, an SLO is never 100%. Choosing 99.9% instead of 99.99% is not laziness; it is an explicit acknowledgment that the marginal cost of the next nine is usually not justified by the marginal reliability the business needs, and that some failure is required to actually learn how the system behaves under stress.

The error budget is the arithmetic complement of the SLO: 100% minus 99.9% leaves a budget of 0.1% of requests (or minutes, or events, depending on how the SLI is framed) that are allowed to fail before the objective is breached. Over a 30-day window at typical request volumes, that 0.1% translates into a concrete, spendable quantity — roughly 43 minutes of full-outage-equivalent downtime, or a proportionally larger amount of partial degradation. The budget converts an abstract reliability goal into a currency that product and engineering teams can spend on releases, experiments, and infrastructure changes, and that they must stop spending once it runs out.

This is the part organizations skip: an error budget without a policy is just a number on a dashboard. The policy specifies what changes when the budget is exhausted — feature freezes, mandatory postmortems, rerouted on-call priorities, or escalation to a reliability review board. Without a policy, the budget becomes decorative, and teams keep shipping through red because nothing forces the conversation. Get the policy signed off by the same people who approve the roadmap, before the first burn, not during an incident.

Why traditional monitoring breaks down at scale

The instinct when reliability suffers is to add more monitoring: more metrics, more dashboards, more alert rules. This instinct is usually wrong, and it is wrong in a specific, mechanical way. As systems decompose into microservices, event meshes, and multi-cloud footprints, the volume of raw telemetry grows roughly linearly with the number of components, but the number of pairwise interactions that can produce an incident grows combinatorially. Traditional threshold-based monitoring was designed for the linear world; it is structurally unequipped for the combinatorial one.

The visible symptom is alert fatigue. A mid-size platform with a few hundred services routinely generates thousands of threshold breaches a day — CPU spikes, queue depth blips, latency percentile wobbles — the overwhelming majority self-resolving within minutes and correlated to a small number of underlying root causes. SOC analysts and SREs on rotation face an impossible triage problem: every alert looks equally urgent because static thresholds carry no information about blast radius, trend, or business impact. The result is well documented across the industry — analysts habituate to noise, real signals get buried, and mean time to detect (MTTD) creeps upward even as the number of monitoring rules climbs.

There is a second, subtler failure mode: threshold monitoring is topology-blind. A 15% latency increase on an internal caching tier might be irrelevant noise on a quiet Tuesday morning and a five-alarm precursor to customer-facing failure during a marketing campaign, because the caching tier's role in the critical path changes with traffic shape. Static thresholds cannot encode that context; they either under-alert during peak risk or over-alert during normal variance, and teams typically tune them toward whichever failure mode was most painful in the last postmortem, which is itself a form of noise.

The third failure mode is temporal: legacy monitoring evaluates each signal independently, so an incident that manifests as a slow-building correlated drift across twelve metrics over ninety minutes looks, from any single dashboard, like normal jitter. Humans are good at pattern recognition across a handful of correlated series; they are poor at doing it across hundreds of series in real time, which is precisely the workload that machine learning models handle well and cheaply.

This is the actual case for AI in reliability engineering. It is not that AI is fashionable. It is that the noise-to-signal ratio of modern distributed systems telemetry has crossed a threshold where rule-based detection cannot keep MTTD and mean time to resolve (MTTR) inside acceptable bounds without either drowning humans in false positives or missing the incidents that matter.

Instrumentation architecture: building SLIs from telemetry

Before any AI layer adds value, the underlying SLI plumbing has to be right. Garbage telemetry produces garbage SLOs regardless of how sophisticated the anomaly detector on top is. There are four SLI classes worth standardizing across a platform, borrowed from the widely adopted "four golden signals" framing but adapted for how modern services actually fail:

• Availability SLIs — proportion of valid requests served successfully, measured server-side (to capture what the system did) and, where possible, client-side or via synthetic probes (to capture what the user experienced). The gap between the two is itself a diagnostic signal, often revealing CDN, DNS, or last-mile network issues invisible to server-side telemetry.
• Latency SLIs — expressed as percentiles (p50, p95, p99), never averages, because averages hide the tail where user pain actually lives. A service can have a perfectly healthy average latency while 2% of requests time out, and that 2% is frequently concentrated in your highest-value customers because they generate the largest, most complex requests.
• Quality/correctness SLIs — proportion of responses that are not just fast and 200-OK but semantically correct: no silently truncated payloads, no stale cache reads served as fresh, no partial writes. These are the hardest to instrument and the most commonly skipped, and they are exactly the SLIs that catch the incidents that dashboards otherwise miss entirely.
• Freshness/durability SLIs — for data and event pipelines, the proportion of records processed within an acceptable lag window, and the proportion durably persisted without loss. Increasingly relevant as event-driven and streaming architectures replace synchronous request/response.

Architecturally, the pipeline that produces these SLIs from raw telemetry needs to separate concerns cleanly: a collection layer (agents, sidecars, OpenTelemetry instrumentation) that emits metrics, traces, and logs with consistent service and environment tagging; a streaming aggregation layer that computes rolling-window SLI values continuously rather than on a fixed batch cadence, because burn-rate alerting depends on knowing the current rate, not last hour's rate; and a durable time-series store that retains raw and aggregated data long enough to train seasonal baselines — a minimum of thirteen months if you want a model that understands year-over-year retail spikes, tax-season load, or academic-calendar traffic.

A platform such as ITMox typically sits at the aggregation and correlation layer, ingesting from existing metrics and logging stacks rather than replacing them, normalizing service topology into a consistent dependency graph, and computing SLI/SLO burn continuously so that the AI layers described below always operate on current, correctly windowed data rather than stale snapshots.

One implementation detail that determines whether the whole program succeeds or collapses under its own weight: SLI computation must be defined once, centrally, and consumed everywhere — dashboards, alerting, error-budget policy enforcement, and postmortem tooling all need to agree on the exact same number for "success rate last Tuesday at 14:00." Teams that let each tool compute its own version of the SLI end up debating whose number is right during incidents instead of fixing the incident.

Error budget policy and multi-window, multi-burn-rate alerting

The single highest-leverage technique in SLO-based alerting, and one that predates the current AI wave by years but is still under-adopted, is multi-window, multi-burn-rate alerting. The idea: instead of alerting when an SLI crosses a static threshold, alert when the rate at which the error budget is being consumed (the "burn rate") is fast enough that, if it continued, the budget would be exhausted before the compliance window ends.

A burn rate of 1x means the budget is being consumed exactly on pace to hit zero right at the end of the 30-day window — sustainable, expected. A burn rate of 10x means the budget will be gone in three days if the current error rate holds. A burn rate of 60x means it will be gone in about twelve hours. The key insight is that you want different alerting behavior at different burn rates: a 60x burn rate for five minutes deserves an immediate page, because even a short window at that rate represents a meaningful budget bite and likely signals a hard outage; a 2x burn rate sustained for six hours deserves a ticket, not a page, because it is a slow leak worth investigating during business hours, not a 3am wake-up.

Implementing this requires evaluating each burn-rate threshold across two time windows simultaneously — a short window (for fast detection) and a long window (for confirming the burn is real and not a five-minute blip that will self-resolve). Google's SRE workbook popularized a four-tier scheme that remains the sound default:

The short window prevents the classic failure of long-window-only alerting, where a real outage takes an hour to trip the alert because the long window dilutes the spike. The long window prevents the classic failure of short-window-only alerting, where a five-minute blip pages someone for an issue that self-heals before they even open a laptop. Requiring both windows to agree is what makes this scheme dramatically quieter than threshold alerting while being faster to detect genuine incidents — the two properties that threshold-based approaches force you to trade off against each other.

This is also where AI earns its keep incrementally even before any anomaly detection model is involved: dynamically setting the burn-rate thresholds themselves based on learned traffic seasonality (a 10x burn rate on Black Friday traffic volume represents vastly more absolute requests than the same 10x burn rate on a quiet Sunday, and the thresholds should reflect that), and automatically suppressing correlated multi-service burn-rate alerts that all trace back to one shared dependency, so on-call engineers get one actionable page instead of forty.

From reactive to predictive: AI-driven anomaly detection and forecasting

Burn-rate alerting is reactive: it tells you the budget is being spent right now. The next architectural layer is predictive: forecasting whether the budget will be exhausted before the compliance window closes, given the trajectory of the underlying SLIs, and flagging anomalies in the leading indicators that precede budget burn by minutes to hours.

Three model families do most of the useful work here, and it is worth being concrete about them rather than waving at "machine learning" generically:

• Seasonal decomposition and forecasting (STL decomposition, Prophet-style additive models, or seasonal ARIMA variants) separates a metric into trend, daily/weekly seasonal components, and residual. The residual is what should be small and boring; when it widens beyond its historical distribution, that is the anomaly signal, and critically it is a signal that already accounts for the fact that Monday morning traffic looks nothing like Saturday night traffic — the single biggest source of false positives in naive threshold systems.
• Multivariate correlation and clustering (dynamic time warping, correlation-matrix change detection, or embedding-based clustering over service metric vectors) identifies when a group of metrics that normally move together starts moving apart, or when metrics that are normally independent start moving together — both are strong precursors to cascading failure, and both are essentially invisible to single-metric threshold rules.
• Sequence and log-pattern models (clustering of log templates via tools like Drain, combined with sequence models over the resulting event stream) detect when the pattern of log events shifts — a new error signature appearing, a normally-rare warning suddenly repeating, a sequence of events occurring out of the usual order — often the earliest available signal, appearing well before latency or error-rate SLIs move at all.

The forecasting piece deserves particular emphasis because it is what actually converts SLOs from a scorekeeping exercise into a planning tool. Given the current burn trajectory and historical variance, a forecasting model can answer, continuously: "at this rate, the 30-day error budget will be exhausted in 4.2 days, with an 80% confidence interval of 3 to 6 days." That is a materially different, and more useful, statement than "the error budget is currently at 62% consumed," because it gives engineering leadership lead time to decide whether to freeze a risky release, add capacity, or accept the projected breach and plan around it. Feeding that projection into sprint planning — alongside velocity and defect counts — is one of the more mature moves a reliability program can make, and one very few organizations have operationalized despite the underlying math being simple.

None of this replaces domain-informed thresholds; it augments them. The practical architecture keeps deterministic burn-rate alerting as the ground truth for paging (predictable, explainable, auditable) and layers the predictive models as an earlier, lower-confidence signal that feeds a separate "watch list" or triggers automated diagnostic collection ahead of a possible page — so that if the page does fire, the diagnostic context (recent deploys, dependency graph snapshot, correlated metric changes) is already assembled rather than needing to be gathered under pressure.

Root cause analysis and topology-aware correlation

Detection tells you something is wrong. Root cause analysis (RCA) tells you why, and it is where the majority of MTTR is actually spent in most organizations — not in fixing the problem once identified, but in the forensic work of figuring out which of the forty things that changed in the last hour is the one that matters.

Effective AI-driven RCA depends on a dependency graph as a first-class artifact, not an afterthought. The graph encodes which services call which, which infrastructure components underlie which services, which deploys touched which components, and ideally which business transactions traverse which service paths. Without this graph, correlation is just statistics on unrelated time series and will happily surface spurious correlations — two metrics that move together because they both follow the diurnal traffic curve, not because one causes the other.

With the graph in place, RCA becomes a ranking problem: given an anomaly on service A, which upstream and downstream nodes in the dependency graph also show anomalous behavior in a plausible causal time order, weighted by the strength and directness of the dependency, and cross-referenced against the change log (deploys, config changes, feature flag flips, infrastructure scaling events) in the preceding window? This is exactly the kind of high-cardinality, graph-structured correlation problem that traditional dashboards cannot present usefully to a human under time pressure, and where an AIOps correlation engine earns its place — not by being smarter than an experienced SRE in the abstract, but by holding and traversing a graph with thousands of nodes and change events far faster than a human can query it manually across a dozen separate tools during an active incident.

A concrete worked example: a checkout-latency SLO burn-rate alert fires. The topology-aware correlation engine, in the same second, surfaces that a config change was pushed to a shared feature-flag service four minutes prior, that three unrelated services downstream of that flag service show correlated latency increases starting within thirty seconds of each other, and that the flag change's blast radius per the dependency graph covers exactly those three services and no others. The on-call engineer's job collapses from "investigate forty services" to "confirm and roll back one flag change" — the difference between an eighteen-minute MTTR and a ninety-minute one, and that difference compounds across every incident a team handles in a quarter.

This same graph-and-correlation approach generalizes directly to security operations, where the "incident" is a threat rather than a latency spike but the underlying problem — too many disconnected signals, not enough correlated context — is identical. Platforms built for agentic SOC operations and AI-driven alert triage apply the same topology-aware correlation logic to reduce thousands of raw detections into a small number of ranked, explained cases, which is precisely why organizations running converged NOC and SOC functions increasingly build both reliability and security correlation on a shared telemetry and topology substrate rather than maintaining two parallel, disconnected pipelines.

Self-healing and automated remediation workflows

Detection and correlation reduce MTTD and the diagnostic portion of MTTR. Automated remediation attacks the remaining portion: the time between "we know what's wrong" and "it's fixed." This is the highest-leverage and highest-risk layer of the stack, and the design principle that separates programs that succeed from ones that cause a second outage while "fixing" the first is graduated autonomy.

Graduated autonomy means every remediation action is classified by blast radius and reversibility before it is ever allowed to run automatically, and the automation level assigned to each class is earned through a track record, not assumed on day one:

1. Tier 0 — diagnostic only. The system gathers logs, traces, recent-change history, and dependency context and attaches them to the incident automatically. No state changes to production. This tier should be fully automated from day one; there is essentially no downside risk.
2. Tier 1 — suggested action, human-approved. The system proposes a specific remediation (restart pod X, roll back deploy Y, scale service Z from 4 to 12 replicas) with its confidence and reasoning, and a human clicks approve. This builds the trust and audit trail needed to graduate actions to the next tier.
3. Tier 2 — automated with guardrails, for known-safe, fully reversible actions. Restarting a stateless pod, clearing a cache with a documented rebuild path, or scaling within pre-approved bounds can run automatically because the blast radius is bounded and the action is trivially reversible.
4. Tier 3 — automated with human notification, for higher-blast-radius but well-tested playbooks. Automated rollback of a canary deployment that is failing its own SLO checks, or automated failover to a secondary region, executed immediately with a page sent simultaneously rather than requiring prior approval, because the cost of delay exceeds the marginal risk of the automated action once the playbook has a long track record.

Every remediation action, at every tier, needs three properties to be trustworthy in production: it must be idempotent (running it twice does not make things worse), it must be observable (its own effect on the SLI is monitored, so an automated fix that makes things worse triggers its own rollback rather than silently compounding the incident), and it must be logged with full context to an audit trail that satisfies whatever compliance regime the organization operates under — a requirement that becomes non-negotiable the moment remediation touches regulated data paths or air-gapped environments where every action needs to be reconstructable after the fact.

The practical starting point for most teams is not exotic AI-driven remediation but codifying the runbooks that already exist in tribal knowledge into executable playbooks, then instrumenting those playbooks so their execution and outcome are logged, then only after a quarter or two of clean execution history promoting the highest-frequency, lowest-risk ones from Tier 1 to Tier 2. The AI's job in this progression is threefold: proposing the right playbook for a given anomaly signature by matching it against historical incidents with similar fingerprints, estimating confidence so humans know when to trust versus scrutinize a suggestion, and continuously re-evaluating whether a Tier 2 automation's success rate still justifies its autonomy level as the underlying system evolves underneath it.

A reference architecture for AI-driven reliability

Pulling the preceding layers together into a single reference architecture clarifies where each piece lives and, just as importantly, where the boundaries between deterministic and probabilistic components need to be explicit, since conflating the two is how teams end up unable to explain why an alert fired or an action ran.

Two architectural decisions determine whether this reference model survives contact with a real, heterogeneous estate. The first is deployment topology: for organizations running sovereign or air-gapped environments — common across defense, critical infrastructure, and regulated financial services — the AI models, the correlation engine, and the telemetry store all need to run fully within the isolated boundary, with no dependency on an external SaaS control plane, and model updates need to ship as versioned artifacts that can be validated and promoted through the same change-control process as any other production software rather than silently updating from a cloud endpoint. This is a fundamentally different engineering posture than most AIOps tooling assumes, and it is one of the reasons a platform's ability to run the full stack — ingestion, correlation, and remediation — inside a customer-controlled boundary matters as much as its model quality; the broader AI-native platform stack approach of building detection, correlation, and action as integrated layers rather than bolted-together point tools is what makes that portability practical.

The second decision is data foundation. Every layer above depends on having a queryable, consistent store of both current and historical telemetry, dependency topology, and incident/change history that multiple tools can read from without each maintaining its own divergent copy. Organizations that let their metrics store, their log store, their CMDB, and their incident management tool drift into four separate sources of truth are, in effect, rebuilding the noisy-telemetry problem described earlier at the data-architecture layer rather than the alerting layer, and a converged data foundation such as MoxDB exists precisely to give the correlation and forecasting models a single consistent substrate to query instead of stitching results across four disconnected systems at query time, which is slow, fragile, and a common source of the "the numbers don't match" arguments that derail incident reviews.

Where the operating model extends beyond a single team, an agentic layer — exemplified by an approach like Norra — can take the correlation engine's ranked findings and actually execute the Tier 1 and Tier 2 workflow: drafting the incident summary, proposing the remediation with its reasoning and confidence, routing for approval where required, and executing where the autonomy tier allows, closing the loop between detection and action without a human manually operating each tool in the chain.

The metrics that prove impact

None of this architecture is worth building unless it moves measurable numbers, and the numbers need to be agreed upon before the program starts, not retrofitted afterward to justify the spend. The core set:

• MTTD (mean time to detect) — time from the first observable symptom to the first alert or automated flag. This is where multi-window burn-rate alerting and predictive forecasting show up directly; organizations moving from static thresholds to burn-rate alerting commonly report MTTD reductions in the 30–60% range within the first two quarters, largely by eliminating the dilution effect of long-window-only rules.
• MTTR (mean time to resolve), decomposed into detection time, diagnosis time, and remediation time separately — because a program that improves detection but not diagnosis will show a smaller aggregate MTTR win than the detection number alone suggests, and leadership needs to see which segment is actually moving.
• Alert-to-incident ratio — the fraction of alerts that correspond to a genuine, actionable incident versus noise. This is the most honest measure of whether an alerting overhaul actually reduced fatigue rather than just relabeling the same volume of noise with a fancier dashboard.
• Error budget burn predictability — how often the forecasting layer's projected exhaustion date was within its stated confidence interval of the actual outcome, which is the calibration check that keeps the forecasting model honest and prevents it from becoming a source of alert fatigue in its own right if its predictions are routinely wrong.
• Toil hours reclaimed — direct engineering time no longer spent on manual diagnosis and repetitive remediation, measured against a baseline captured before the program starts, since this is usually the line item that funds the next phase of investment.
• Automation coverage and success rate — the proportion of incidents that had an applicable Tier 1–3 playbook, and of those, the proportion where the automated or suggested action resolved the incident without human escalation, tracked separately per tier so a regression in Tier 2 success rate triggers a re-review of that tier's autonomy grant.
• SLO attainment trend — the actual measure of whether all this machinery is producing the outcome it exists to produce: are services meeting their objectives more consistently quarter over quarter, and is the variance in attainment shrinking, which indicates the system is becoming more predictable, not just occasionally lucky.

These figures are directional, drawn from patterns observed across mature SRE and AIOps programs rather than a single controlled study, and any organization adopting this approach should instrument its own baseline before touching the tooling, precisely so the before/after comparison is credible internally rather than asserted.

Governance, change control, and security considerations

An AI-driven reliability stack expands the attack surface and the compliance surface simultaneously, and both need explicit design attention rather than being treated as someone else's problem once the pipeline is live.

On governance: every automated remediation action must be attributable to a specific model version, playbook version, and triggering condition, retained in an immutable audit log, because the first question after any automation-caused incident will be "what exactly ran, on whose authority, and why," and "the AI decided" is not an acceptable answer in a regulated environment or, frankly, in any environment where trust in the automation needs to survive its first mistake. Error budget policies themselves need a change-control process — who can approve a change to an SLO target, and what evidence is required — because SLOs that can be quietly loosened whenever they're inconvenient stop functioning as a contract.

On security: the telemetry pipeline and correlation engine described throughout this piece are themselves high-value targets, since they hold a real-time map of system topology, deploy history, and known weak points — exactly the reconnaissance data an attacker wants. This is why reliability and security telemetry increasingly converge onto the same underlying data foundation with the same access controls, rather than existing as separate systems with separate, inconsistently applied protections; the correlation techniques used for RCA overlap substantially with those used for detection and response and continuous exposure management, and the identity boundary around who can trigger a Tier 2 or Tier 3 remediation action deserves the same rigor as any other privileged access path, governed through the same identity and PAM controls applied to other high-privilege automation, since an automated remediation credential with broad production access is, functionally, a standing privileged account that needs the same lifecycle management, rotation, and monitoring as a human administrator's.

A related and frequently overlooked point: the AI models themselves need governance. Anomaly detection thresholds and forecasting models drift as the underlying system evolves, and a model that was well-calibrated at launch can silently become miscalibrated eighteen months later as traffic patterns, service topology, and deploy cadence change. Scheduling periodic recalibration reviews, with the same rigor as a security control review, prevents the AI layer from becoming exactly the kind of stale, untrusted, ignored system that threshold-based monitoring became in the first place.

Implementation roadmap: a practical sequence

Organizations that succeed with this approach almost never start with the AI layer. The sequence that works, in practice, looks like this:

1. Define SLIs and SLOs for the top five to ten business-critical services first, not all services at once. Precision on a small set beats approximate coverage everywhere, and the first round will surface instrumentation gaps that need fixing before anything downstream can be trusted.
2. Stand up multi-window, multi-burn-rate alerting on those services, retiring the old static-threshold alerts for the same signals rather than running both in parallel indefinitely, which just doubles the noise during the transition.
3. Get the error budget policy signed off by engineering leadership and the product owners of those services before the first budget exhaustion event, including exactly what happens — freeze, escalation, reallocation — so the first real test of the policy is not also the first time anyone has read it.
4. Build the dependency graph covering those same services and their immediate upstream/downstream neighbors, since this is the prerequisite for any correlation or RCA capability and is usually more work than teams estimate, particularly in older estates where the topology exists only in individual engineers' heads.
5. Introduce anomaly detection and forecasting as advisory signals first — visible on dashboards, feeding a watch list, but not yet paging anyone — for at least one full seasonal cycle, so the models' calibration can be checked against real outcomes before they're trusted to influence on-call behavior.
6. Codify existing runbooks into Tier 0 and Tier 1 playbooks, instrumented so every execution and outcome is logged, building the track record needed to justify promotion to Tier 2 for the highest-frequency, lowest-risk cases.
7. Expand service coverage and automation tier by tier, using the metrics from the previous section to decide where to invest next rather than expanding uniformly, since the highest-incident-volume services usually deserve the next round of automation investment before lower-traffic ones.

This sequence deliberately front-loads the boring, deterministic work — SLI definitions, burn-rate math, dependency graphs — before any AI model touches production behavior, because the AI layer's entire value proposition depends on the substrate underneath it being trustworthy. An anomaly detector trained on noisy, badly windowed SLI data will produce noisy, badly calibrated anomalies, and no amount of model sophistication fixes a data foundation problem.

Worked example: a payments checkout incident end to end

To make the architecture concrete, walk through a single incident as it would flow through the full stack. A payments checkout service has an availability SLO of 99.95% and a p95 latency SLO of 400ms, both measured over a rolling 28-day window, with a monthly error budget of roughly 21 minutes of full-equivalent downtime.

At 14:32, the 5-minute/1-hour burn-rate pair for the latency SLI crosses the 14.4x threshold: p95 latency has degraded enough that, sustained, the monthly budget would be gone in under two hours. A page fires. Simultaneously, Tier 0 diagnostic collection triggers automatically: the last thirty minutes of traces for the affected service, the dependency graph neighborhood two hops out, and the change log for the same window are all attached to the incident before the on-call engineer has opened a laptop.

The correlation engine, running against that attached context, identifies that a database connection pool configuration change was deployed to a shared payment-authorization service at 14:29, three minutes before the burn-rate crossing, and that two other services sharing that same authorization dependency show correlated latency increases starting within the same thirty-second window — while a dozen unrelated services with no dependency on that path show no anomaly at all. The ranked root-cause candidate presented to the engineer is the connection pool change, with the dependency graph showing exactly why the blast radius is limited to those three services.

A Tier 1 remediation suggestion accompanies the finding: roll back the connection pool configuration to its prior value, a fully reversible action with a documented history of resolving similar connection-exhaustion patterns in nine of the last ten occurrences. The on-call engineer approves it. The rollback executes, latency recovers within ninety seconds, and the incident closes with a total MTTR of eleven minutes — three minutes to detect, five to correlate and confirm root cause, three to execute the approved rollback.

In the postmortem, two follow-ups emerge directly from the metrics discussed earlier: the connection pool rollback's nine-of-ten historical success rate and now-confirmed tenth success case is enough of a track record to propose promoting it to Tier 2 automated-with-guardrails, since the action is bounded and reversible; and the fact that this is the third incident in two months traced to configuration changes on the shared authorization service prompts a review of that service's change-management gate, a process fix that no amount of remediation automation would have surfaced on its own — a reminder that the AI-driven layer's job is to shrink MTTD and MTTR on individual incidents, while the pattern-across-incidents insight is what actually prevents the fourth one.

Frequently asked questions