On-Prem and Air-Gapped AI: Architecture Patterns

Every enterprise that has tried to run generative or agentic AI inside a regulated boundary has hit the same wall: the fastest models live in someone else’s cloud, and the data that makes them useful cannot leave the building. This article is a working blueprint — hardware, model lifecycle, network topology, and operational workflow — for running trustworthy AI entirely within your own borders and controls.

Why sovereignty is now an architecture requirement

For most of the last three years, “AI architecture” meant an API key to a hosted model and a prompt template. That pattern is fine for marketing copy generation and internal chatbots with no sensitive context. It falls apart the moment the workload touches classified telemetry, patient records, industrial control system data, financial transaction streams, or anything covered by export control, data residency law, or a security clearance boundary. In those environments the question is never “which model is smartest,” it is “which model can I legally and technically run without a network path to a third party.”

Sovereignty is not a single control — it is a stack of decisions that compound. Data sovereignty determines where bytes physically rest. Operational sovereignty determines who can patch, retrain, or roll back a model without a vendor in the loop. Supply-chain sovereignty determines whether you can prove, cryptographically, that the weights running in production are the weights you approved and not a tampered artifact introduced somewhere between a public repository and your inference cluster. Air-gapped and on-prem AI architectures exist to satisfy all three simultaneously, and each one has direct, unavoidable consequences for how you design compute, storage, identity, and update pipelines.

The engineering reality is that sovereignty is expensive in a specific, measurable way: it trades convenience for provable control. A cloud-hosted frontier model updates itself weekly with no action on your part. An air-gapped open-weight model requires you to build the update mechanism, the evaluation harness, and the rollback plan yourself. That is not a reason to avoid the pattern — for regulated workloads it is the only legally defensible option — but it means the architecture has to be treated as a first-class system with its own SRE discipline, not a side project bolted onto an existing MLOps pipeline.

The three deployment postures: on-prem, air-gapped, and sovereign cloud

These terms get used interchangeably in vendor material, but they describe materially different architectures with different threat models and different engineering constraints.

On-prem connected means the inference and training infrastructure lives in your own data center or colocation facility, but it retains a controlled, monitored network path to the outside world — typically for model updates, threat intelligence feeds, telemetry export, or license validation. This is the most common posture for enterprises with strong data residency requirements but no formal air-gap mandate. The architecture challenge here is mostly about segmentation: keeping the AI workload's egress path narrow, authenticated, and auditable while still getting the benefit of external updates.

Air-gapped means there is no routable network path between the AI environment and any external network, full stop. Every artifact — model weights, container images, vulnerability signatures, dependency packages — crosses the boundary through a physical, mediated transfer process: a data diode, a write-once optical medium, or a formally reviewed removable media transfer under two-person control. This posture is standard in defense, intelligence, and increasingly in critical infrastructure and industrial control environments. The architecture challenge shifts from network segmentation to supply-chain integrity and update cadence — how do you keep a model current with emerging threats when it can go months between transfers?

Sovereign cloud is a hybrid: infrastructure operated by a cloud provider (or a national/regional sovereign cloud operator) but with contractual, jurisdictional, and sometimes cryptographic guarantees that data and control never cross a defined boundary — no foreign personnel access, no cross-border replication, dedicated key management under local control. It gives you cloud elasticity with sovereignty guarantees enforced by contract and technical control (confidential computing, customer-held keys) rather than physical isolation. It is the right answer when the requirement is jurisdictional control rather than literal network isolation.

Most organizations end up running all three simultaneously for different workloads: sovereign cloud for elastic, less-sensitive inference; on-prem connected for the bulk of operational AI; and a genuinely air-gapped enclave for the highest-classification work. Designing for that mixed reality from day one — rather than assuming a single posture — saves a painful re-architecture later.

Reference architecture for the air-gapped enclave

A production-grade air-gapped AI enclave has five logical zones, and the discipline of keeping them logically and physically separate is what makes the environment auditable. Collapsing zones for convenience is the single most common mistake teams make when they first build one of these environments.

Zone 1: the transfer gateway

This is the only place artifacts enter or leave. It should be a dedicated, hardened host or appliance — not a shared jump box — running content-disarm-and-reconstruction on any document payloads, cryptographic signature verification on every model artifact and container image, and full-disk write logging. Every file that crosses the gateway gets a manifest entry: SHA-256 hash, source, approver, timestamp, and destination zone. Data diodes (hardware that physically permits one-directional flow) are the gold standard for outbound-only telemetry; inbound artifact transfer should go through a reviewed, logged process even if the medium is removable storage.

Zone 2: the model and artifact registry

Once a model, dataset, or container image clears the gateway, it lands in an internal registry — think a private OCI registry plus a model registry like an internal MLflow or a hardened Hugging Face Hub mirror — that never talks to the internet. This registry is the single source of truth for what is approved to run. Every artifact is content-addressed and signed with an internal key (Sigstore/cosign patterns work well even fully offline, since the key material and verification tooling can be self-hosted).

Zone 3: training and fine-tuning compute

GPU or accelerator clusters used for continued pretraining, fine-tuning, or retrieval-index building. This zone is typically the most expensive and the most idle — most air-gapped enterprises fine-tune infrequently, so this compute is often shared with inference at off-peak times, or explicitly separated to avoid resource contention affecting production inference latency.

Zone 4: inference serving

The production-facing layer: model servers (vLLM, TensorRT-LLM, Triton, or similar), a gateway/router that handles authentication, rate limiting and request logging, and the application layer consuming the models — SOC copilots, IT automation agents, document processing pipelines. This is the zone with the tightest latency and availability SLAs, and it should be architected for horizontal scale independent of the training zone.

Zone 5: evaluation and audit

A logically separate zone that runs continuous evaluation harnesses against every model version, stores immutable logs of inference requests and responses for the retention period your compliance regime demands, and hosts the dashboards SOC and compliance teams use to prove the system is behaving as approved. Keeping this zone separate from inference serving prevents a compromised or misbehaving application from tampering with its own audit trail.

Open-weight model selection and lifecycle management

Air-gapped and on-prem deployments live or die on open-weight models, because a closed API model cannot be transferred across a gap or verified for supply-chain integrity in any meaningful sense — you are trusting a black box you cannot inspect, hash, or re-run offline. The open-weight ecosystem has matured enough that this is no longer a significant capability sacrifice for most operational workloads, though it is still a real one for frontier reasoning tasks.

Model selection for a sovereign deployment should weigh five factors beyond raw benchmark performance:

• License terms — permissive licenses (Apache 2.0, MIT) versus restricted-use licenses that prohibit certain deployment contexts (some open-weight licenses explicitly restrict military or intelligence use, which disqualifies them for defense-sector air-gapped work regardless of technical fit).
• Parameter count versus available accelerator memory — a 70B-class dense model needs roughly 140GB of memory at FP16 for weights alone before accounting for KV cache, which typically means multi-GPU serving; a well-tuned 7B-14B model can run comfortably on a single high-memory GPU with room for long context.
• Quantization tolerance — how much quality degrades under INT8 or INT4 quantization matters enormously for hardware sizing, and this varies significantly by model architecture and training recipe, not just parameter count.
• Provenance and training data documentation — model cards that disclose training data composition, safety evaluation methodology, and known failure modes are increasingly a procurement requirement in regulated sectors, not just a nice-to-have.
• Fine-tuning and distillation friction — how easily the model architecture supports LoRA/QLoRA adaptation to your domain data, since most sovereign deployments need at least light domain adaptation to be useful.

Once selected, a model needs a lifecycle process, not a one-time download. Establish a defined cadence — monthly or quarterly depending on your transfer window — for pulling updated weights, safety patches, and tokenizer fixes from the public ecosystem into a staging environment on the connected side of your boundary. Run your full evaluation harness there before the artifact is ever queued for transfer. This is where most organizations underinvest: they treat the initial model deployment as the finish line, and eighteen months later are running a model with known, publicly disclosed vulnerabilities or behavioral regressions that were patched upstream long ago but never made it across the gap.

Version pinning discipline matters as much here as it does for any other production dependency. Every inference deployment should reference an exact, hashed model version — never “latest” — and every change to that pin should go through the same change-control process as a code deployment, including a documented rollback plan tested before go-live.

Hardware and inference stack sizing

Sizing an on-prem inference stack is a different exercise than sizing cloud capacity, because you are committing capital to a fixed ceiling rather than renting elastic headroom. The core sizing inputs are concurrent request volume, target latency (time to first token and tokens per second), context window requirements, and the quantization level you are willing to accept for the quality trade-off.

A useful rule of thumb for dense transformer models: memory required for weights is roughly 2 bytes per parameter at FP16/BF16, 1 byte at INT8, and 0.5 bytes at INT4. On top of weights, KV cache memory scales with context length, batch size, and the number of attention heads/layers — for long-context workloads (32K+ tokens) with concurrent users, KV cache can exceed the weight memory footprint, which is the single most common sizing mistake teams make when they size only for the model and forget the cache.

Mixture-of-experts architectures change this calculus again: total parameter count determines storage and memory footprint, but only the active parameters per token determine compute cost, so a large MoE model can be surprisingly cheap to serve on a per-token basis relative to its total size, provided you have the memory to hold the full expert set resident.

Serving software choice matters almost as much as hardware. vLLM's paged attention gives strong throughput for high-concurrency, variable-length workloads and has become close to a default choice for open-weight serving. TensorRT-LLM extracts more raw performance per GPU on NVIDIA hardware at the cost of a heavier build and optimization pipeline, which matters when every deployment cycle has to be validated and transferred across an air gap rather than pulled from a package index on demand. Triton Inference Server is a reasonable choice when you need to serve heterogeneous model types — LLMs alongside classical ML and vision models — behind one management plane, which is common in SOC environments doing multi-modal alert triage.

Do not undersize storage. Model weights, checkpoints for fine-tuning experiments, vector indices for retrieval, and audit logs at multi-year retention add up fast — a realistic sovereign AI deployment should budget for petabyte-scale storage growth within two to three years even if the initial footprint looks modest, particularly if the audit and evaluation zone is retaining full request/response pairs.

Air-gapped update and patch workflow

The single hardest operational problem in air-gapped AI is not the initial build, it is keeping the environment current without a routine network connection. Vulnerability signatures, model safety patches, dependency CVE fixes, and threat intelligence all update continuously on the connected side of the world, and every day of lag between a patch being published and it landing in your enclave is measurable risk exposure.

The workflow that works in practice has five stages, run on a fixed cadence rather than ad hoc:

1. Aggregate on the connected side. A staging environment outside the air gap pulls updated model weights, container base images, OS and dependency patches, and threat intelligence feeds on a defined schedule — weekly for security patches, monthly or quarterly for model weight updates, depending on your risk tolerance.
2. Evaluate before transfer. Every candidate artifact runs through an automated evaluation suite — regression tests against your golden prompt set, safety and jailbreak resistance tests, performance benchmarks — before it is ever queued for the gap. Nothing crosses the boundary without passing this gate; it is far cheaper to catch a regression here than inside the enclave.
3. Package and sign. Approved artifacts are bundled into a transfer package with a full manifest: hashes, evaluation results, approver identity, and change ticket reference. The package is signed with a key that only exists on the connected staging side.
4. Mediated transfer. The package crosses the gap through your approved mechanism — write-once media under two-person control, a formally accredited diode, or a cross-domain solution appliance. The transfer gateway verifies the signature and manifest hash before anything is written to the internal registry.
5. Staged rollout inside the enclave. The new artifact deploys first to a canary slice of inference capacity, runs the same evaluation suite a second time against live-shaped internal traffic, and only then rolls out to full production with a tested rollback path held ready for a defined observation window.

This cadence should be written down as a runbook with named owners for each stage, not left as tribal knowledge. Teams that skip the written runbook consistently end up with either dangerous update lag (because no one owns pushing the process forward) or dangerous update velocity (because someone bypasses evaluation under deadline pressure). Both failure modes are common and both are avoidable with a documented, rehearsed process.

Retrieval-augmented generation and data pipelines inside the perimeter

Most operational value in sovereign AI deployments comes not from the base model but from retrieval over internal data — ticket history, runbooks, threat intelligence, configuration management databases, log archives. Designing this pipeline correctly matters as much as model selection, and it has to respect the same isolation boundaries as everything else.

The embedding model used for retrieval should itself be open-weight and deployed inside the perimeter — sending internal document text to an external embedding API defeats the entire purpose of the air gap, a mistake that is more common than it should be because embedding calls feel less consequential than chat completions. Run embedding generation as a batch or streaming pipeline inside Zone 4, writing into a vector store (Qdrant, Milvus, or pgvector on a hardened Postgres instance are all reasonable choices for on-prem deployment) that lives entirely within your network boundary.

Data freshness for retrieval indices is a separate operational concern from model updates and typically needs a tighter cadence — ticket systems, log data, and configuration state change constantly, and a retrieval index that is stale by even a day can materially degrade an agent's investigation quality. Build the ingestion pipeline as a continuously running internal service, not a batch job tied to the air-gap transfer cadence; unlike model weights, this data never needs to cross the boundary at all since it originates inside it.

Access control on the retrieval layer deserves the same rigor as access control on the source systems it indexes. A common and serious mistake is building a single shared vector index across all internal documents without carrying forward the source system's access control lists into retrieval-time filtering — this creates a path for a user with narrow access to a chat interface to retrieve content they were never authorized to see in the source system, entirely bypassing the source system's own controls. Retrieval filtering by identity and entitlement should be enforced at query time, not assumed away because “it's just for internal use.”

For SOC and NOC use cases specifically, the retrieval layer typically needs to span structured telemetry (SIEM events, EDR alerts, asset inventory) and unstructured knowledge (runbooks, past incident reports, vendor advisories) in a single agentic reasoning loop. This is the architecture pattern behind platforms like Algomox's AI-native stack, where retrieval, structured query, and generative reasoning are composed into a single agent context rather than treated as separate tools bolted together after the fact.

Identity, network segmentation, and zero-trust controls

An AI enclave, air-gapped or not, is only as sovereign as its weakest identity control. The temptation in a lot of early deployments is to treat the AI system as a trusted internal service with broad access to whatever data sources it needs for retrieval and tool use — this is precisely backwards, because an LLM-driven agent with broad standing access is a much larger blast radius than a human operator with the same access, since it can be manipulated by adversarial input at machine speed and machine scale.

Every tool call and data access an agent makes should go through the same identity and entitlement enforcement as a human user would, ideally impersonating the requesting user's actual entitlements rather than running under a single powerful service account. This is a significant architecture shift from how most internal automation has traditionally been built, and it is the area where identity security and privileged access management intersect directly with AI architecture — the agent's credential and session lifecycle needs to be managed with the same discipline as a human privileged account, including just-in-time elevation, session recording, and automatic revocation.

Network segmentation inside the enclave should follow standard zero-trust principles applied to a new kind of workload: microsegment the inference serving zone from the training zone, require mutual TLS between every internal service, and treat the model server's outbound connections (to tool APIs, retrieval stores, orchestration services) as a defined, allow-listed set rather than open egress. This matters more for AI workloads than traditional applications because prompt injection attacks specifically try to get a model to take an unintended action through a tool call — if the network and identity layer constrain what that tool call can actually reach and do, a successful injection has a bounded blast radius rather than an open one.

Observability, drift detection, and continuous evaluation

A model deployed in an air-gapped environment does not get the benefit of a vendor silently improving it or a large user base surfacing edge cases through aggregate telemetry the way a cloud API does. You are entirely responsible for knowing whether the model is still performing well, and that responsibility does not end at initial validation — it is a continuous operational function for as long as the model is in production.

Three categories of monitoring matter for a production sovereign AI deployment. Infrastructure observability covers the standard SRE surface — GPU utilization, memory pressure, request latency percentiles, queue depth — and is largely identical to monitoring any other high-throughput service. Model quality observability is specific to AI systems: running a fixed, versioned evaluation set against the production model on a schedule and alerting on score regression, tracking hallucination rate on retrieval-grounded tasks through automated groundedness checks, and sampling live production interactions for human review against a defined rubric. Behavioral drift detection looks for changes in the distribution of inputs the model is seeing versus what it was validated against — new attack patterns in SOC alert data, new ticket categories in IT service data — because a model that scored well on last quarter's evaluation set can degrade silently as the operational data distribution shifts underneath it.

Full request and response logging, retained for the period your compliance regime requires, is non-negotiable for regulated deployments and serves double duty: it is both the audit trail and the raw material for building better evaluation sets over time. This logging needs to happen in the isolated Zone 5 described earlier, with access controls that prevent the production application from modifying its own history — a compromised or misbehaving agent should not be able to cover its tracks.

Build dashboards that a SOC lead or compliance officer can read without needing to understand model internals — accuracy trend lines, cost per resolved ticket or triaged alert, human override rate (how often an operator disagrees with or corrects the model's output), and time-to-detect for known regression patterns. The human override rate in particular is an underused but very high-signal metric: a steadily rising override rate is often the earliest indicator of model drift, well before it shows up in formal evaluation scores.

Applying the pattern to SOC and NOC operations

Security and network operations are where sovereign AI architecture earns its keep fastest, because the data involved — raw packet captures, endpoint telemetry, credential and identity events — is exactly the category of data organizations are least willing to send to an external API, and because the operational cost of a slow or wrong triage decision is measured in dwell time and breach impact, not just user frustration.

An agentic SOC architecture built on the patterns above typically layers three types of AI-driven capability on top of the inference and retrieval infrastructure already described: alert triage agents that correlate signals across EDR, SIEM, and network telemetry to reduce analyst-facing noise, investigation agents that can pull context from asset inventory, threat intelligence, and past incident history to build a case file automatically, and response-recommendation agents that propose remediation steps for human approval rather than acting autonomously on high-impact actions. This maps closely to how agentic SOC architectures are designed in practice — a set of narrow, tool-using agents coordinated by an orchestration layer, each with tightly scoped entitlements, rather than one monolithic model with broad access.

The exposure management side of the house benefits from the same pattern applied to a different data domain — asset and vulnerability data instead of live alerts. A sovereign AI deployment supporting continuous threat exposure management needs retrieval over asset inventories, vulnerability scan results, and business context (which systems are actually critical) to prioritize remediation intelligently, and none of that data should be leaving the perimeter to get that prioritization done. The same holds for AI-driven XDR alert triage, where the volume and sensitivity of raw detection data makes on-prem or air-gapped inference close to mandatory in regulated sectors.

NOC operations follow a parallel logic with different data: network flow telemetry, configuration state, and change history feed a retrieval-augmented agent that can correlate a performance degradation with a recent change or a known failure pattern faster than a human working through dashboards manually. Products like Algomox's ITMox and CyberMox are built around this integrated pattern — and the broader convergence of NOC and SOC operating models, discussed in the integrated NOC-SOC approach, depends on exactly the shared, sovereignly-deployed data and inference layer described in this article. An agentic workforce layer — the pattern behind Norra — sits on top of this infrastructure to orchestrate multi-step tasks across both domains, which only works if every agent in that workforce inherits the same identity, entitlement, and audit controls described above.

Data foundation and storage architecture

None of the preceding architecture works without a data foundation that can support both the transactional load of operational telemetry ingestion and the analytical load of retrieval and model evaluation, inside the same sovereignty boundary. This is a genuinely hard distributed systems problem, separate from the AI-specific concerns already covered, and it is where a surprising number of otherwise well-designed sovereign AI projects run into trouble — they get the model and network architecture right and then discover their storage layer cannot handle the combined write throughput of telemetry ingestion, vector index updates, and audit logging at production scale.

A data foundation purpose-built for this — the role a platform like MoxDB plays in the Algomox stack — needs to handle time-series telemetry, document and log storage, and vector search as coherent, jointly queryable data rather than three separate systems an agent has to reconcile at query time. Separating these into disconnected databases is a common early architecture mistake: it works fine in a proof of concept with a handful of data sources, and then falls over once an agent needs to correlate a vector-similarity match against a runbook with a time-windowed join against raw telemetry, which is exactly the kind of query pattern SOC and NOC investigation agents run constantly.

Retention policy design also belongs in the data foundation layer, not bolted on afterward. Regulated environments typically require multi-year retention for audit-relevant logs but much shorter retention for raw high-volume telemetry once it has been summarized, and the storage architecture should tier automatically — hot storage for the last few days of raw data feeding real-time retrieval, warm storage for recent history feeding trend analysis, and cold, cheaper storage for long-term compliance retention that is rarely queried but must remain retrievable and verifiably unaltered.

Compliance mapping and a decision framework for choosing a posture

Different regulatory regimes map to different points on the sovereignty spectrum described earlier, and understanding that mapping helps avoid both under-building (a compliance gap discovered during audit) and over-building (spending air-gap-level engineering effort on a workload that only needed data residency).

Use a simple decision framework when scoping a new sovereign AI project: first, determine the actual legal or contractual requirement (not the perceived one — teams frequently over-assume air-gap requirements when a data residency contract clause would actually be satisfied by sovereign cloud, and under-build when a genuine physical isolation mandate exists). Second, map the workload's data sensitivity tiers separately, since a single deployment often serves multiple sensitivity levels and should be architected with tiered zones rather than a single uniform posture applied to everything. Third, size the engineering investment against the actual update cadence and evaluation rigor the posture demands — air-gapped environments are not just “on-prem with extra steps,” they carry a permanent operational tax in the form of the transfer and evaluation pipeline described earlier, and that tax needs a named owning team, not an assumption that it will run itself.

Four dimensions are worth tracking as a single scorecard for any sovereign AI initiative: data sovereignty (where bytes physically rest and which jurisdiction's law governs them), operational sovereignty (who can patch, retrain, or roll back without an external dependency), supply-chain sovereignty (cryptographic proof of exactly what weights and code are running), and identity sovereignty (every agent action traceable to an entitled, revocable identity). A deployment that scores well on data sovereignty but poorly on the other three is not meaningfully sovereign — it has simply moved the compliance risk from a network diagram to an audit finding.

Cost and operational trade-offs versus cloud AI

The honest cost conversation about on-prem and air-gapped AI has three components that are frequently conflated in vendor pitches and internal business cases alike: capital expenditure on hardware, the ongoing operational cost of the update and evaluation pipeline, and the opportunity cost of slower access to frontier model capability.

Capital expenditure is the most visible and most commonly modeled cost, and it is genuinely significant — a production-grade GPU cluster capable of serving a 70B-class model to a few hundred concurrent SOC analysts represents a real, multi-year hardware investment, plus the data center power and cooling capacity to run it continuously. Organizations frequently underestimate the power and cooling delta specifically; modern data-center GPUs draw meaningfully more power per rack unit than the general-purpose compute most facilities were built around, and a retrofit can rival the cost of the GPUs themselves.

The operational cost that gets underestimated most consistently is the human cost of running the update and evaluation pipeline described earlier — this is not a part-time responsibility bolted onto an existing MLOps engineer's job, particularly in an air-gapped environment where transfer windows are infrequent and each one carries real risk if rushed. Budget for a dedicated team function, not a shared responsibility, if the deployment is large enough to matter operationally.

The opportunity cost is real but shrinking. Frontier closed models retain an edge on the hardest reasoning and coding benchmarks, but for the operational workloads that dominate SOC, NOC, and IT service management — classification, summarization, retrieval-grounded question answering, structured tool use — open-weight models in the 30B-70B class deployed with good retrieval architecture now perform close enough to frontier closed models that the gap is rarely the limiting factor in actual outcomes. The limiting factor is almost always retrieval quality and workflow integration, not raw model capability, which is good news for sovereign architectures since retrieval and workflow integration are exactly the parts you fully control.

• Model licensing costs drop to near zero for open-weight deployments versus per-token API billing, which matters enormously at high volume (continuous log summarization, for instance, can generate token volumes that make API pricing prohibitive well before hardware amortization does).
• Latency is typically better on-prem for high-throughput internal workloads, since there is no external network hop and batching can be tuned to your exact traffic pattern.
• Elasticity is the clearest loss versus cloud — you cannot burst to ten times capacity for a rare incident spike without pre-provisioned headroom, so capacity planning has to account for peak SOC incident volume, not average load.

Frequently asked questions