Anomaly Detection for Metrics, Logs and Traces

Every production system now emits three distinct languages of telemetry — metrics, logs and traces — and most operations teams are still reading them with tools built for one. The payoff for fusing them into a single, statistically honest anomaly detection layer is not incremental: it is the difference between an operator who discovers an outage from a customer ticket and a platform that predicts, correlates and remediates the same failure before a human ever looks at a dashboard.

The Telemetry Flood and Why Static Thresholds Fail

A mid-sized microservices estate with a few hundred services, a service mesh, a handful of Kubernetes clusters and a modest fleet of databases will typically generate tens of millions of metric data points per minute, hundreds of gigabytes of logs per day and trace spans numbering in the billions once sampling is accounted for. This is not a hypothetical — it is the baseline for any organization that has adopted microservices, containers and event-driven architectures in the last five years. The volume alone breaks the operating model that most enterprises still run: a human sets a static threshold on a handful of metrics, a paging system fires when the threshold is crossed, and an on-call engineer manually correlates that alert against logs and traces to find a root cause.

Static thresholds fail for three structural reasons. First, they cannot express seasonality. CPU utilization on a batch-processing fleet that spikes to 85% every night at 2 a.m. during a scheduled ETL job is normal; the same value at 2 p.m. on a Tuesday might indicate a runaway query. A single threshold cannot hold both truths. Second, thresholds are set once and decay in relevance as the system evolves — a threshold tuned for a service running on three pods becomes meaningless after autoscaling triples the fleet size following a deployment. Third, and most consequentially, thresholds operate on a single metric in isolation. Real incidents are rarely visible in one signal; they are visible in the joint behavior of several signals moving together in a way that no individual threshold would trip.

The practical consequence is alert fatigue. Studies of enterprise NOC and SOC operations consistently show that 40–70% of paged alerts are either duplicates, transient blips that self-resolve, or symptoms of a single root cause fanning out across dozens of downstream services. Engineers stop trusting the paging system, mute channels, and triage reactively from customer complaints — which is precisely the failure mode that anomaly detection, properly implemented, is designed to eliminate. Getting this right requires treating metrics, logs and traces not as three separate monitoring problems but as three projections of the same underlying system state, correlated through time and topology.

The Anatomy of Metrics, Logs and Traces as Anomaly Surfaces

Each telemetry type carries a different statistical shape, and effective anomaly detection has to respect that shape rather than force all three through the same algorithm.

Metrics

Metrics are dense, regularly sampled numeric time series — CPU, memory, request rate, latency percentiles, queue depth, error rate, saturation. They are the easiest signal to model statistically because they arrive on a predictable cadence and typically exhibit strong seasonality (daily, weekly, sometimes monthly cycles tied to business activity). The anomaly detection task on metrics is fundamentally a time-series forecasting problem: build an expectation of what the value should be at time t given its history, and flag deviations that exceed a statistically justified band. The complexity lies in handling trend, seasonality, changepoints (a deployment that permanently shifts baseline latency), and heteroscedasticity (variance that itself changes with time of day or load).

Logs

Logs are semi-structured or unstructured text events, arriving at irregular intervals and in enormous volume and cardinality. A single service might emit hundreds of distinct log line templates, each with variable fields (request IDs, user IDs, timestamps, stack traces). Anomaly detection on logs is not a single problem but at least three: volume anomalies (a service that normally logs 200 lines per minute suddenly logs 40,000), template anomalies (a log line pattern that has never been seen before appears, or a common pattern vanishes), and content anomalies (a known template's field values drift outside historical norms, such as an error code distribution shifting). Because logs are free text, the first engineering step is almost always template mining — converting raw text into a finite vocabulary of structural patterns — before any statistical model can be applied.

Traces

Traces capture the causal structure of a single request as it moves through a distributed system: a tree of spans, each with a service name, operation name, duration, status code and set of tags. Traces are the hardest signal to model because the object of interest is not a scalar value but a graph shape and a duration distribution conditioned on that shape. Anomaly detection on traces operates at two levels: span-level (is this individual span's duration or error status anomalous given its historical distribution for that operation), and trace-level (is the overall shape of this request — which services it touched, in what order, with what fan-out — different from the topology's normal call graph). Traces are also the signal most affected by sampling; head-based sampling that keeps only 1% of traces can silently discard the very outliers anomaly detection needs to see, which is why tail-based sampling that preferentially retains error and high-latency traces is a prerequisite for trace-based anomaly detection at any real scale.

Understanding these differences matters because a common failure mode in building an observability pipeline is to bolt a generic anomaly detector — often just a z-score or a fixed percentile band — onto all three signal types uniformly. It will produce mediocre results everywhere. The right approach treats each telemetry type with the statistical machinery suited to its shape, then fuses the outputs at a correlation layer, which is the architecture described later in this article.

Detection Techniques That Actually Work in Production

There is no single algorithm that solves anomaly detection across metrics, logs and traces. What works in production is a layered toolkit, applied selectively based on the signal's cardinality, seasonality and cost tolerance for false positives.

Metric-focused techniques

For metrics, the workhorse techniques, roughly in order of implementation complexity, are:

• Rolling statistical bands (EWMA, Bollinger-style bands): compute an exponentially weighted moving average and standard deviation, flag points more than k sigma away. Cheap, explainable, works well for stationary or slowly drifting series, but breaks on strong seasonality.
• Seasonal decomposition (STL, seasonal-hybrid ESD): decompose the series into trend, seasonal and residual components, then apply extreme-value statistics to the residual. This is the standard approach for metrics with daily or weekly cycles — request volume, batch job duration, business-hours traffic.
• Forecasting-based detection (Holt-Winters, Prophet-style additive models, ARIMA variants): forecast the expected value with a confidence interval and flag actuals that fall outside it. More expensive to maintain per-series but handles multiple seasonalities and holiday effects well.
• Changepoint detection (CUSUM, Bayesian online changepoint detection): rather than flagging point anomalies, detect a structural shift in the series mean or variance — the signature of a deployment, a configuration change or a capacity event. This is essential for distinguishing "this is a new normal" from "this is an outlier that will revert."
• Multivariate models (PCA-based reconstruction error, isolation forests, autoencoders): when a single metric looks fine in isolation but the joint behavior of a metric vector (CPU, memory, GC pause, thread count) is unusual, multivariate models catch what univariate detectors miss. These are the right tool for detecting silent degradation — a service that is technically within SLO on every individual metric but is behaving in a way that historically preceded a cascading failure.

Log-focused techniques

Log anomaly detection starts with parsing. Template mining algorithms (Drain, Spell, and their production derivatives) convert raw log lines into a stable set of templates by clustering on structural similarity and masking variable tokens. Once logs are templated, three detection modes follow:

• Volume-based detection: apply the same time-series techniques used for metrics to the per-template event rate. A sudden 50x spike in a specific "connection refused" template is a volume anomaly even if total log volume looks normal.
• Novelty detection: maintain a rolling vocabulary of known templates and flag any log line that does not match an existing template within a similarity threshold. New templates are frequently the first evidence of a new failure mode, a new exception type, or a code path that has never executed in production before.
• Sequence and co-occurrence modeling: some failures are only visible in the ordering or co-occurrence of log events, not in any single template's rate. Sequence models (n-gram based, or more advanced LSTM/transformer-based log sequence models) learn the normal ordering of events within a session or transaction and flag sequences that deviate — catching things like a retry storm, a state machine stuck in a loop, or an authentication flow that skips an expected step.

Trace-focused techniques

Trace anomaly detection layers on top of span-level latency and error modeling:

• Per-operation latency baselines: for every unique (service, operation) pair, maintain a rolling distribution (not just mean and standard deviation — latency distributions are heavy-tailed, so percentile-based or histogram-based baselines are far more reliable) and flag spans whose duration falls in an extreme percentile relative to history.
• Critical-path analysis: for a given trace, identify which span is actually responsible for the end-to-end latency (the critical path, not simply the slowest span, since parallel spans can be slow without affecting total duration). This focuses anomaly attribution on the span that matters rather than surfacing every slow-looking span in a large fan-out.
• Graph-shape anomalies: compare the topology of an individual trace (which services were called, fan-out degree, depth) against the learned normal call graph for that entry point. A trace that unexpectedly calls a deprecated service, skips a cache layer, or fans out to twice the normal number of downstream calls is anomalous independent of its duration.
• Error propagation modeling: track how error and status codes propagate up a trace tree, distinguishing a root-cause error (originating in a leaf span) from a cascading symptom (a parent span reporting an error because a child failed). This is the mechanism that lets a detection system point directly at the offending service instead of the dozens of services that merely inherited the failure.

Correlating Across Signals and Topology

Individual anomaly detectors on metrics, logs and traces will each fire independently during a real incident, and if left unfused they simply multiply the alert volume instead of reducing it. The step that actually converts detection into operational value is correlation: grouping anomalies that share a root cause into a single incident, and ranking that incident by business impact.

Correlation works along two axes simultaneously. The first is temporal: anomalies that begin within a tight window of each other (accounting for propagation delay through the dependency graph) are candidates for the same root cause. The second is topological: anomalies on services that are directly connected in the service dependency graph — built from trace data, service mesh telemetry, or a configuration management database — are far more likely to be causally related than anomalies on unrelated services that merely happen to fire at the same time. A robust correlation engine builds a live dependency graph from trace data (which is the most reliable source of ground-truth call relationships, more accurate than static architecture diagrams that go stale), weights edges by call frequency and criticality, and uses that graph to cluster contemporaneous anomalies into candidate incidents.

A second, equally important correlation dimension is cross-signal reinforcement. A latency anomaly on a payment service, a spike in "connection timeout" log templates on the same service, and a trace-level graph-shape anomaly showing that service calling a downstream database twice instead of once are three independent detections of the same underlying event — likely a connection pool exhaustion. When a correlation engine can recognize that all three signals point at the same service within the same time window, it can present the operator with a single incident carrying multi-signal evidence, rather than three separate pages that each look ambiguous in isolation. This is the mechanism that underlies effective triage in platforms like Algomox’s ITMox, where metric, log and trace anomalies are fused against a live topology graph before anything reaches a human, and it is the same principle that agentic SOC operations apply on the security side when correlating detection signals across endpoints, identity and network telemetry.

Root cause ranking within a correlated incident cluster typically uses a combination of graph centrality (the node whose anomaly onset time is earliest and whose position in the dependency graph explains the propagation pattern of the others) and anomaly score magnitude. Techniques borrowed from causal inference — Granger causality tests on the time series of anomaly scores, or more sophisticated causal graph discovery methods — can further refine this ranking, though in practice a well-built dependency graph combined with onset-time ordering gets an operations team most of the value without the computational overhead of full causal discovery.

A Reference Architecture for Predictive Operations

Turning the techniques above into a system that runs continuously in production requires a pipeline architecture with clear separation between ingestion, feature extraction, per-signal detection, correlation and action. The following flow reflects the pattern used across mature AIOps deployments, including the pipeline underpinning ITMox and the data foundation layer, MoxDB, that many of these deployments run on.

Collection should standardize on OpenTelemetry wherever possible — a single SDK and collector model for metrics, logs and traces dramatically simplifies the normalization layer and avoids the historical problem of three disjoint pipelines (a metrics TSDB, a log indexer and a distinct tracing backend) that never share a common schema or set of resource attributes. Where legacy agents or vendor-specific formats persist, the collector layer's job is to normalize them into a common resource model — consistent service name, environment, host and version tags — because correlation downstream depends entirely on being able to join signals on shared attributes.

The normalization stage is also where log template mining, trace sampling decisions and metric downsampling policies live. This is a deceptively important layer: if two services tag their environment field inconsistently ("prod" versus "production"), every downstream correlation and dashboard breaks silently. Enforcing a resource attribute schema at ingestion, with rejection or auto-correction of malformed telemetry, pays for itself many times over in reduced debugging of the observability pipeline itself.

Per-signal detection runs the techniques described in the previous section, typically as a set of independently scalable stream-processing jobs — one class of jobs handling metric time-series models, another handling log template and volume models, a third handling trace span and graph-shape models. Each emits a normalized anomaly event with a common schema: timestamp, resource, signal type, anomaly score, confidence, and a pointer back to the raw evidence. This normalization is what allows a single correlation engine to sit downstream of all three without needing signal-specific logic.

The correlation and action layers are covered in the next two sections, but the architectural principle worth emphasizing here is that detection and correlation must be decoupled. Coupling them — building a monolithic system that tries to detect and correlate in a single pass — makes it impossible to independently tune detection sensitivity per signal type or to swap in improved models for one signal without touching the others. Keep the interface between the two a well-defined event schema, and the whole pipeline becomes independently evolvable.

Worked Examples: From Raw Signal to Verified Anomaly

Concrete examples make the abstractions above tangible. Consider three incidents, one per signal type, and how a properly layered detector catches each.

Metrics: a slow-burn memory leak

A service's heap utilization metric, sampled every 15 seconds, climbs from a normal working range of 40–55% to 90% over six hours following a deployment that introduced a caching bug. A static threshold set at 85% would fire only in the final twenty minutes before the process OOMs and restarts — far too late for any preventive action. A seasonal-hybrid ESD detector running on the residual after removing the daily seasonal pattern catches the anomaly within roughly 30–40 minutes of the trend beginning, because the residual component starts drifting steadily upward in a way that is statistically inconsistent with the historical noise band around the trend. Layering a changepoint detector on top identifies the deployment timestamp as the structural break point, which lets the correlation engine immediately associate the anomaly with a specific release — turning "memory is high" into "memory started climbing 40 minutes after deployment v2.14.3 went live," a root cause an on-call engineer can act on in seconds rather than after twenty minutes of log spelunking.

Logs: a silent authentication regression

An identity provider integration begins silently failing for a subset of users after a certificate rotation is misconfigured on one of three redundant auth nodes. Total request volume and overall error rate barely move because only one of three nodes is affected and load balancing masks the aggregate impact — a pure volume-based metric threshold sees nothing worth alerting on. However, template mining on the auth service's logs surfaces a new log line template — "certificate verification failed: unknown authority" — that has never appeared in the service's history. Novelty detection flags it within the first few occurrences, well before volume accumulates to a level that would move an aggregate error-rate metric. This is precisely the class of incident that log-only or metric-only monitoring misses systematically: partial, node-scoped failures that are diluted at the aggregate level but are unambiguous at the template level. This pattern is directly relevant to identity-focused operations, which is why platforms addressing identity and privileged access risk lean heavily on log template and sequence anomaly detection rather than aggregate error-rate thresholds.

Traces: a cache-layer bypass under load

A product catalog service normally serves 92% of read requests from a Redis cache, with the remaining 8% falling through to the primary database. During a traffic spike, a cache eviction policy misconfiguration causes the cache hit rate to collapse, and traces begin showing a graph-shape anomaly: the normal call graph (service → cache → occasional database fallback) is replaced by a graph where the majority of traces show service → cache miss → database on every request. Average latency increases only modestly at first because the database can still absorb the extra load, so a pure latency-threshold detector is slow to react. But the graph-shape anomaly detector, comparing the distribution of call-graph shapes against the learned baseline, flags the shift within the first few minutes because the proportion of traces taking the "database fallback" path has moved several standard deviations from its historical baseline. This gives operators a fifteen-to-twenty-minute head start before the database queue saturates and latency breaches SLO — the difference between a graceful cache-warming remediation and a customer-facing outage.

Closing the Loop: From Detection to Self-Healing

Detection and correlation only produce operational value once they connect to action. The maturity curve for an operations team typically runs through four stages, and understanding where a given service or incident class sits on that curve determines how much automation is appropriate.

1. Detect and notify: the system identifies an anomaly, correlates it into an incident, and pages a human with the ranked root cause and supporting evidence. This is table stakes and should be the default for any newly onboarded service.
2. Detect and diagnose: beyond notification, the system automatically attaches a diagnostic bundle — the relevant log templates, the anomalous trace exemplars, the metric charts with the anomaly window highlighted, and a natural-language summary of the likely root cause — so the responding engineer starts triage with an answer already 80% assembled instead of starting from a blank dashboard.
3. Detect and recommend: the system proposes a specific remediation — restart this pod, roll back this deployment, scale this resource pool, rotate this credential — based on pattern matching against a runbook library of previously successful remediations for similar incident signatures, but still requires human approval before execution.
4. Detect and remediate autonomously: for well-understood, low-blast-radius incident classes with a proven remediation (a known memory leak that is fixed by a rolling restart, a known cache-stampede pattern fixed by a cache warm-up job, a known certificate expiry fixed by a rotation job), the system executes the remediation automatically, logs the action, and only escalates to a human if the remediation does not resolve the anomaly within an expected window.

The architectural pattern that makes stage four safe is a closed feedback loop: every automated remediation is itself instrumented, and the anomaly detector that triggered the action continues watching the same signal to confirm resolution. If the anomaly score does not fall back within the normal band within a defined SLA, the system automatically escalates rather than silently retrying or, worse, declaring success prematurely. This closed loop is what distinguishes genuine self-healing from "automation that might make things worse" — the detector is not just a trigger, it is also the verifier.

In practice, most enterprise operations teams run a mixed model: a growing library of stage-four automations for well-characterized, high-frequency, low-risk incidents (disk cleanup, connection pool resets, stuck job restarts, known certificate rotations), while everything else stays at stage two or three until enough incident history accumulates to justify promoting a remediation to autonomous status. This is also the model that Algomox's agentic approach to operations follows across both ITMox for IT operations and Norra for broader agentic workforce automation — agents are given increasing autonomy only as their remediation success rate on a given incident signature crosses a defined confidence threshold, with every autonomous action remaining fully auditable and reversible.

Taming Alert Fatigue Without Losing Signal

The instinct when alert fatigue becomes a problem is to raise thresholds or suppress noisy alert sources, and this is almost always the wrong fix — it trades false positives for false negatives and erodes trust in the alerting system from the other direction. The correct fix operates at the correlation and presentation layer, not at the detection sensitivity layer.

Deduplication is the first lever: many alerting systems treat every threshold crossing as a new alert, so a metric that oscillates around a boundary during a genuine incident can generate dozens of pages for what is operationally one event. Alert deduplication should key on (resource, anomaly type, root cause cluster) rather than raw threshold crossings, with a single alert that updates in place as the underlying anomaly evolves rather than spawning new notifications.

Grouping is the second lever, and it is where the topology-aware correlation described earlier pays off directly: instead of forty pages for forty services showing elevated error rates during a single upstream database outage, the operator receives one incident with the database identified as the probable root cause and the forty downstream services listed as impacted-but-secondary. This alone typically reduces page volume by 60–85% in estates with meaningfully deep service dependency graphs, and it is the mechanism behind the alert-volume reductions that platforms addressing AI-driven alert triage report in both IT operations and security operations contexts — the underlying correlation math is nearly identical whether the alerts originate from infrastructure telemetry or from security detections.

Suppression windows are the third lever, applied narrowly: known maintenance windows, expected batch-job resource spikes, and canary deployment traffic patterns should be excluded from baseline calculations entirely (not merely alert-suppressed after the fact) so that the anomaly detector's learned "normal" is not contaminated by legitimate but atypical activity. Failing to do this is a common cause of detectors that either miss real anomalies during maintenance windows (because everything looks abnormal, so the real signal is buried) or become desensitized afterward (because the model incorporated the maintenance spike into its baseline and now under-reacts to a genuinely similar-looking future incident).

Finally, severity scoring should be continuous and business-aware rather than binary. An anomaly on a canary environment serving 1% of traffic and an anomaly of identical statistical magnitude on the primary payment path are not operationally equivalent, and the alerting system should weight anomaly scores by business criticality (derived from the service topology, SLA tier and traffic share) before deciding paging urgency. This single change — moving from statistical significance alone to statistical significance weighted by business impact — is often the highest-leverage improvement available to a team that has already implemented reasonable detection and correlation.

The Metrics That Prove Impact

An anomaly detection program has to justify itself in numbers the same way any other operations investment does. The relevant metrics fall into two categories: detection quality and operational outcome.

These figures should be treated as illustrative ranges drawn from typical enterprise deployment patterns rather than universal constants — the actual numbers for any given estate depend heavily on service topology depth, telemetry coverage completeness, and how long the anomaly models have had to learn seasonal baselines (most statistical models need a minimum of two to four full seasonal cycles, meaning two to four weeks at minimum for daily-seasonal metrics, before their false-positive rate stabilizes at production-acceptable levels). Any team standing up a new anomaly detection program should expect a "burn-in" period where false positive rates are elevated while baselines calibrate, and should communicate that expectation clearly to stakeholders rather than over-promising day-one accuracy.

Beyond the operational metrics above, the business case typically rests on three further numbers worth tracking explicitly: engineer hours reclaimed from manual correlation and triage work (often the single largest line item in a cost-justification model), reduction in customer-visible incident minutes (directly tied to SLA credit exposure and churn risk), and the ratio of proactive to reactive incident discovery (the share of incidents caught by anomaly detection before any customer-facing symptom versus the share first reported by a customer or a downstream team) — a ratio that should climb steadily as detection coverage and correlation maturity improve.

Deployment Models: Cloud, On-Prem and Air-Gapped

Anomaly detection architectures have to accommodate three materially different deployment realities, and the differences are not cosmetic — they change what techniques are even feasible.

In a cloud-native deployment, the primary constraints are cost and cardinality. Cloud telemetry backends typically charge on ingestion volume and cardinality, which pushes teams toward aggressive sampling, downsampling and retention tiering. Anomaly detection in this environment needs to be cardinality-aware — running per-series models efficiently across potentially millions of unique time series (one per pod, per container, per endpoint) requires either model sharing across similar series (clustering services by behavioral profile and sharing a model per cluster rather than per series) or a tiered approach where only high-cardinality but high-value series get dedicated models.

On-premises deployments typically have more predictable, bounded infrastructure and looser cost pressure on telemetry volume, but they introduce integration complexity: legacy monitoring tools, mainframe and midrange systems, and homegrown log formats that require custom parsers before template mining or any statistical model can operate. The anomaly detection layer in these environments needs a flexible ingestion and normalization stage capable of absorbing SNMP traps, syslog, proprietary agent formats and modern OpenTelemetry streams side by side, which is precisely the kind of heterogeneous integration challenge that platforms built around a unified AI-native stack are designed to absorb without forcing a wholesale telemetry migration as a prerequisite.

Air-gapped and sovereign environments — common in defense, critical infrastructure, and regulated financial and government sectors — impose the hardest constraint: no external connectivity for model training, threat intelligence feeds, or cloud-hosted ML inference. Anomaly detection in these environments must run entirely on infrastructure the customer controls, with models trained and updated using only locally collected telemetry, and with any pretrained baseline models shipped and versioned as part of the deployment artifact rather than pulled from a live service. This has real implications for technique selection: computationally expensive deep learning approaches that assume access to elastic cloud compute for training are often impractical, favoring statistically efficient methods (seasonal decomposition, changepoint detection, template mining with lightweight clustering) that can retrain incrementally on modest on-prem hardware. Algomox's architecture supports this mode explicitly across ITMox, CyberMox and MoxDB, because a meaningful share of operators in regulated and defense-adjacent sectors cannot accept any dependency on external connectivity for a system as operationally central as anomaly detection.

Common Pitfalls and Anti-Patterns

Teams building or buying anomaly detection capability repeatedly run into the same set of mistakes, and naming them explicitly is more useful than another abstract best-practices list.

The single-signal tunnel vision pattern is the most common: a team invests heavily in metric anomaly detection because metrics are the easiest signal to model, achieves genuinely good results there, and declares victory — while logs and traces remain on static rules or no anomaly detection at all. The worked examples earlier in this article show why this leaves real gaps: the silent authentication regression and the cache-bypass incident were both invisible to metric-only detection and only surfaced through log and trace anomaly models respectively.

Baseline contamination is subtler and harder to catch. If a genuine incident occurred during the training window and was never excluded, the model learns the incident's signature as part of "normal," which means a recurrence of the same incident will be systematically under-detected. The fix requires a feedback mechanism where confirmed incidents are flagged and excluded from training windows retroactively, which in turn requires the incident management and anomaly detection systems to be integrated rather than siloed.

Wiring every per-signal detector directly to a paging system, bypassing correlation, simply recreates the alert fatigue problem with more sophisticated math behind it — the volume problem was never really about detection accuracy, it was about the absence of aggregation, and a more accurate detector without aggregation still produces one page per anomalous signal rather than one page per incident.

Finally, and most damaging over time, is the absence of a feedback loop from human triage back into the model. Every time an on-call engineer marks an alert as a false positive or confirms a true incident, that judgment is training signal — and systems that do not capture and feed it back into model tuning plateau at whatever accuracy they achieved during initial calibration, while the systems that do capture it improve continuously as incident history accumulates. This feedback loop, and the operational discipline of actually reviewing false positive and false negative rates on a regular cadence rather than only when something goes visibly wrong, is what separates anomaly detection programs that compound in value over years from those that decay into background noise within a few quarters.

Frequently asked questions