Running LLMs in Air-Gapped Environments

A model that phones home is a model you do not control. For defense integrators, regulated banks, utilities, and sovereign government programs, the promise of generative AI collides head-on with a hard constraint: the workload cannot leave the wire, and in many cases there is no wire at all. This is the practical playbook for running large language models entirely inside your own perimeter — from hardware sizing and model selection to retrieval pipelines, evaluation harnesses, and the operational discipline that keeps an air-gapped LLM trustworthy for years after the last update was carried in on a drive.

Why air-gapped LLM deployment is now a mainstream requirement

Eighteen months ago, "air-gapped AI" was a niche conversation reserved for classified networks and nuclear facilities. That has changed. Financial services regulators in multiple jurisdictions now treat outbound calls to third-party inference APIs as a data residency and model-risk event requiring its own control framework. Critical infrastructure operators — power generation, water treatment, rail signaling — run operational technology (OT) networks that are physically or logically isolated from the internet by design, and any AI capability introduced there must respect that isolation or the OT segmentation itself becomes worthless. Defense and intelligence programs have always required this, but the surface area has grown: it is no longer just document classification, it is copilot-style assistance for analysts, SOC triage, and maintenance technicians who now expect a chat interface wherever they work.

The driver is not merely paranoia about a vendor reading your prompts, though that risk is real and well documented — several commercial LLM providers reserve rights to log and use API traffic for abuse monitoring, and even with enterprise contractual protections, the data still transits infrastructure you do not own or audit. The deeper driver is architectural: once inference depends on a live connection to an external provider, your operational resilience is now bounded by that provider's uptime, that provider's terms of service, that provider's export-control posture, and geopolitical conditions entirely outside your control. A hospital system, a submarine, a forward operating base, or a sanctioned-adjacent bank branch cannot accept an AI capability that silently degrades the moment connectivity is cut or a vendor account is suspended.

Running LLMs on-prem and air-gapped is not simply "the cloud deployment, but local." It requires rethinking model selection, hardware provisioning, update logistics, evaluation, and observability from first principles, because none of the conveniences of a managed API — automatic model upgrades, elastic GPU capacity, vendor-side content filtering, vendor-side abuse detection — are available. Everything must be built, validated, and operated by your own team, which is precisely why this is now a core competency for platforms like Algomox's AI-native stack, which is designed to run its detection, triage, and remediation reasoning fully within customer-controlled infrastructure across ITMox and CyberMox deployments.

Defining the boundary: air-gapped, sovereign, and on-prem are not the same thing

Precision matters here because these terms get used interchangeably in vendor marketing and it leads to bad architecture decisions. Three distinct postures exist, and each implies a different engineering approach.

On-premises (connected)

The model and inference stack run on infrastructure you own or lease, inside your data center or private cloud, but the network still has a path to the internet — typically for package updates, telemetry, or occasional model downloads through a controlled proxy. This is the most common posture for enterprises that want data residency and cost control without full isolation. The security burden is largely about egress control: proving that inference requests and model weights never leave, even though the network technically could reach outside.

Sovereign (jurisdictionally bounded)

Sovereign AI is a data-residency and legal-control concept, not necessarily a network-isolation one. A sovereign deployment guarantees that data, model weights, and processing stay within a specific legal jurisdiction, are operated by nationals or vetted personnel, and are not subject to foreign lawful-access regimes such as the U.S. CLOUD Act. A sovereign deployment can still have internet connectivity; what it cannot have is uncontrolled data flow to entities or infrastructure outside the sovereign boundary. National government AI programs and regulated telecom and banking sectors in the EU, Middle East, and Asia-Pacific are the primary drivers of this pattern.

Air-gapped (physically isolated)

This is the strictest posture: no network path to the outside world exists at all, ever, by design. Updates — model weights, container images, threat intelligence, vulnerability feeds — move across the gap only via physical media or a mediated one-way transfer device (a data diode). This is the standard for classified networks, industrial control systems in critical infrastructure, and increasingly for the most sensitive enterprise workloads such as trading algorithms and legal privilege review. Air-gapped is the hardest environment to serve because every dependency your inference stack has — tokenizer files, embedding models, vector database, monitoring agents, even NTP time sync — must be resolved without ever reaching out.

The practical implication: design your architecture for the air-gapped case even if you are initially deploying on-prem-connected. It is far easier to relax isolation later (open a controlled egress path) than to retrofit an architecture that assumes internet reachability. Treat every external dependency as a first-class inventory item from day one.

Model selection for closed environments

You cannot run a closed-weight API model behind an air gap — there is no vendor endpoint to call. The entire calculus of model selection changes: you are choosing among open-weight model families that you can download once, inspect, quantize, fine-tune, and operate indefinitely without further contact with the publisher.

As of mid-2026 the practical shortlist for serious on-prem and air-gapped work spans several families, each with different licensing and operational trade-offs worth evaluating deliberately rather than defaulting to whichever model is trending:

• Llama family (Meta) — broad ecosystem support, strong tool-calling and instruction-following, but the license includes acceptable-use restrictions that matter for defense and certain commercial redistributions; legal review before mission use is mandatory, not optional.
• Mistral / Mixtral family — Apache 2.0 licensed variants give the cleanest legal posture for unrestricted internal and even redistributed use, with strong mixture-of-experts efficiency for a given parameter budget.
• Qwen family (Alibaba) — excellent multilingual and coding performance, permissive licensing on most sizes, but provenance and supply-chain review is a harder conversation in defense and government contexts given country-of-origin scrutiny.
• DeepSeek family — strong reasoning benchmarks at competitive parameter counts, MIT-style licensing on several releases, but subject to the same country-of-origin provenance scrutiny as Qwen for sovereign and defense buyers.
• Phi and Gemma (Microsoft, Google) — smaller, efficient models well suited to edge and constrained-hardware deployment where a 7–9B model that fits on a single GPU or even a ruggedized laptop is more valuable than a larger model that requires a rack.

The selection framework should weigh five factors, in this order of importance for air-gapped operators: license terms and redistribution rights, provenance and supply-chain trust (who trained it, on what data, and can that be attested), parameter count versus available hardware, task-specific benchmark performance on your actual use cases rather than generic leaderboards, and community/ecosystem maturity for quantization and fine-tuning tooling. A 70B model that cannot legally be used for the mission, or whose training provenance cannot survive a security review, is worthless regardless of benchmark score.

Do not chase the largest model you can technically fit. A well-instruction-tuned 8B-14B model with retrieval augmentation and a narrow, well-specified task (SOC alert triage, log summarization, runbook lookup) will consistently outperform a poorly-integrated 70B general model on domain accuracy, while costing a fraction in GPU-hours and enabling deployment on hardware that fits in a rack instead of a data hall. This is precisely the pattern Algomox applies inside agentic SOC workflows: smaller specialized models handling high-volume, well-bounded reasoning tasks, with escalation paths to larger models only when the task genuinely requires broader context.

Hardware architecture and capacity sizing

Sizing an air-gapped inference cluster is an exercise in matching model memory footprint, expected concurrency, and latency SLAs to a fixed hardware budget you cannot elastically scale after the fact — there is no autoscaling group reaching into a public cloud region when demand spikes.

Memory footprint math

The first-order calculation every operator needs: VRAM required is approximately parameter count multiplied by bytes-per-parameter at your chosen precision, plus KV-cache overhead that scales with context length and concurrent sessions. A 70B parameter model at FP16 (2 bytes/parameter) needs roughly 140GB just for weights, before any KV cache; at INT8 that drops to about 70GB, and at INT4 (via GPTQ, AWQ, or GGUF quantization) to roughly 35–40GB. KV cache for long-context, high-concurrency workloads can add another 20–40% on top depending on context window and batch size. This is why quantization is not an optional optimization in air-gapped deployments — it is frequently the difference between fitting on hardware you can procure through your existing acquisition channel versus requiring an exotic multi-node GPU cluster that takes a defense acquisition cycle to approve.

Reference tiers

• Edge/tactical tier — single workstation-class GPU (24–48GB VRAM), running a 7–13B model quantized to INT4/INT8, serving a handful of concurrent users. Appropriate for forward-deployed teams, single-analyst workstations, or ruggedized field kits.
• Departmental tier — 2–4 GPU server (e.g., 4x 80GB data-center GPUs), running a 30–70B model, serving a SOC shift or a regional office with tens of concurrent sessions and retrieval augmentation against a local knowledge base.
• Enterprise/mission tier — multi-node GPU cluster with high-bandwidth interconnect, serving multiple 70B+ models or several specialized fine-tunes simultaneously, supporting hundreds of concurrent users across a large SOC, NOC, or intelligence cell.

Beyond GPUs, the air-gapped bill of materials includes local object storage for model artifacts and datasets (sized for several full copies of every model version you retain, plus checkpoints from fine-tuning runs), a local container registry mirroring every base image, an internal package repository mirroring language runtime dependencies, and a hardware security module or equivalent for signing model artifacts and verifying provenance at import time. None of these can be assumed reachable from a public registry once the system is sealed, and forgetting one of them is the single most common cause of an air-gapped deployment stalling mid-build.

The transfer workflow: getting models and updates across the gap safely

This is the operational core of air-gapped AI and the part most teams underestimate. A model is not a single file — it is a bundle of weight shards, tokenizer configuration, chat template definitions, license and model card metadata, and often an associated evaluation suite. Every one of these must cross the boundary intact, verified, and attributable.

A disciplined transfer workflow looks like this in practice:

1. Stage in a connected, controlled environment. Download the model from the publisher's official repository into a hardened staging host that itself has no direct path to the isolated network — a DMZ transfer station, not a jump box with a live tunnel.
2. Verify cryptographic provenance. Check publisher-signed checksums (SHA-256 manifests, and increasingly SLSA provenance attestations or Sigstore signatures for model artifacts) before the bundle is trusted for transfer. Reject anything without a verifiable hash chain back to the publisher.
3. Scan for embedded threats. Model files in formats like pickle-based PyTorch checkpoints can carry arbitrary code execution via unsafe deserialization. Convert to the safetensors format and run static analysis before transfer; never carry raw `.bin`/pickle checkpoints across the gap if a safetensors equivalent exists.
4. Quantize and re-hash in staging. If quantization is needed, perform it in the staging environment, then generate a new manifest of hashes for the quantized artifact set. The chain of custody now covers both the original and the derived artifact.
5. Transfer via approved one-way mechanism. Physical media (write-once optical, or a formally sanitized removable drive following your data spillage procedures) or a certified data diode for larger, recurring transfers. Every transfer is logged against a change ticket with the manifest hash recorded.
6. Re-verify on the isolated side. Before the model touches the inference cluster, re-run hash verification inside the air gap. Any mismatch halts the import and triggers incident review — do not assume the transfer medium preserved integrity.
7. Import into the internal registry with version tagging. The model enters your internal artifact store with a version tag, the original manifest, the quantization parameters used, and a pointer to the evaluation run that will validate it (next section) before it is promoted to any production traffic.

The same workflow applies to everything else that needs to cross the gap on a cadence: CVE and vulnerability feeds for the OS and container layer, updated retrieval corpora, fine-tuning datasets, and guardrail/classifier updates for prompt injection and content filtering. Treat model updates and security feed updates as the same class of problem — both are external artifacts entering a trust boundary and both need the same rigor.

Retrieval augmentation without an internet-connected index

Nearly every serious LLM deployment needs retrieval-augmented generation (RAG) to ground answers in current, authoritative internal data rather than the model's frozen training knowledge. Inside an air gap, every component of that pipeline — the embedding model, the vector database, the document ingestion pipeline, and the reranker — must run locally with no external calls, which rules out the common pattern of calling a hosted embeddings API.

The architecture is straightforward but every piece needs to be selected for local operation: an open-weight embedding model (BGE, E5, or Nomic Embed families are common choices, all runnable on CPU or a modest GPU for reasonable throughput) converts your internal documents — runbooks, incident post-mortems, configuration baselines, compliance policies, threat intel reports — into vectors stored in a local vector database (Qdrant, Milvus, and pgvector-on-Postgres are the most operationally mature choices for isolated deployment, since all three ship as self-contained binaries or containers with no external dependency at runtime). A local reranker model (a cross-encoder, typically small enough to run on CPU) refines the top-k retrieval results before they are injected into the LLM's context window.

The ingestion pipeline itself needs the same update discipline as model weights: new documents enter through a controlled process, get chunked, embedded, and indexed, and the resulting index is versioned so you can roll back a bad ingestion run (a common failure mode is a corrupted or malformed document poisoning chunks with garbage text that degrades retrieval quality across unrelated queries). For SOC and NOC use cases specifically, this local knowledge base typically includes the organization's own historical incident data, asset inventory, and detection logic — which is exactly the kind of grounding that turns a generic LLM into a system capable of the kind of contextual triage described in AI-driven XDR alert triage, where the model needs to reason over an organization's specific environment, not generic security knowledge.

One subtlety operators frequently miss: retrieval freshness inside an air gap has a built-in lag equal to your transfer cadence. If your one-way diode moves data once per shift, your RAG corpus is at best one-shift stale. For fast-moving operational data — active incident status, live asset state — RAG grounded in a periodically refreshed index is the wrong pattern; that data belongs in a direct, real-time internal API call from the agent, with RAG reserved for the slower-changing knowledge (policies, historical precedent, documentation) where staleness is tolerable.

Fine-tuning, adaptation, and domain specialization

Base open-weight models are generalists. Getting them to reliably speak your organization's operational language — ticket taxonomies, runbook formats, compliance citation styles, the specific shorthand your SOC analysts use — usually requires adaptation beyond prompt engineering and RAG alone.

Parameter-efficient fine-tuning (LoRA/QLoRA)

Low-Rank Adaptation and its quantized variant QLoRA let you fine-tune a model by training a small set of additional low-rank matrices rather than the full parameter set, cutting GPU memory requirements by an order of magnitude and making fine-tuning feasible on the same departmental-tier hardware used for inference. This is the workhorse technique for air-gapped adaptation: you can fine-tune a 13B model on a single 48GB GPU in hours, producing a LoRA adapter file measured in tens to low hundreds of megabytes rather than a multi-gigabyte full checkpoint — which also makes the adapter itself far easier to version, transfer, and roll back than a full model re-train.

Full fine-tuning

Reserved for cases where the domain shift is large enough that low-rank adaptation cannot capture it — a genuinely new language, a radically different task format, or safety-critical behavior changes that need to be baked deeply into the weights rather than layered on top. Full fine-tuning inside an air gap demands the enterprise-tier hardware described earlier and a much heavier evaluation burden, because full fine-tuning can degrade capabilities the base model previously had (catastrophic forgetting) in ways a LoRA adapter, being additive and reversible, generally does not.

Instruction and preference tuning with internal data

The most valuable air-gapped fine-tuning data is often not new domain knowledge but corrected behavior: a curated set of past model responses that analysts flagged as wrong, incomplete, or stylistically off, paired with the corrected answer. This turns your production usage logs into training signal, but it requires a deliberate feedback-capture mechanism in the application layer (a simple thumbs up/down with an optional correction field goes a long way) and a review process to ensure the curated set does not itself encode bad patterns or, worse, sensitive data that should not be baked into model weights that might later be exported for a different classification level.

Every fine-tuning run inside the air gap should be treated as producing a new model version subject to the full evaluation gate before promotion — a LoRA adapter is not exempt just because it is small.

Evaluation, red-teaming, and guardrails with no cloud eval service

Commercial API providers run extensive automated safety evaluation and abuse detection on your behalf, invisibly. In an air-gapped deployment, none of that exists unless you build it. This is the single most consequential gap teams leave open, because it is invisible until an incident makes it visible.

Building an internal evaluation harness

Every model version — base, quantized, fine-tuned, or LoRA-adapted — needs to pass a standing evaluation suite before promotion to production traffic. That suite should combine: task-accuracy benchmarks specific to your use cases (a held-out set of real SOC tickets with known-correct triage decisions, for example), general capability regression tests to catch quantization-induced degradation, and safety/red-team probes covering prompt injection, jailbreak attempts, and data leakage through the model's outputs. Open-source harnesses such as `lm-evaluation-harness` and `promptfoo` are mature enough to run fully offline once their benchmark datasets are mirrored locally, and should be part of the same artifact transfer pipeline as the models themselves.

Guardrails without a vendor moderation API

Content and safety moderation cannot lean on a hosted moderation endpoint. The practical alternative is a locally-run classifier model (Llama Guard and similar open-weight safety classifiers are purpose-built for this and small enough to run alongside the primary model) paired with deterministic rule-based filters for known-bad patterns — PII regex matching, secrets/credential detection, and classification-marking checks that prevent a response from crossing a security boundary it should not. Layer these: a fast deterministic filter catches the obvious cases cheaply, and the classifier model catches the nuanced ones the regex misses.

Continuous drift monitoring

Without a live connection to compare against a vendor's updated model, you have no external signal that your fixed, frozen model is drifting in relevance as your environment changes around it — new attack techniques, new infrastructure, new terminology. The mitigation is a recurring internal evaluation cadence (monthly or quarterly, tied to your transfer window) that re-runs the task-accuracy suite against freshly curated real-world examples, not just the original static benchmark, so degradation in practical usefulness is caught before analysts start quietly distrusting the tool.

Agentic workflows and tool use inside the gap

The industry's center of gravity has moved from single-turn chat to agentic systems — models that plan, call tools, and take multi-step action. This raises the stakes for air-gapped deployment considerably, because an agent that can execute actions (query a database, open a ticket, isolate a host, adjust a firewall rule) is a fundamentally different risk surface than one that can only produce text.

Tool-calling in an offline agent requires every tool the model can invoke to be a local, internal API — there is no calling out to a public search engine, no hosted code-execution sandbox, no external plugin ecosystem. Each tool needs its own least-privilege service account, its own input validation independent of what the model claims it's about to do, and its own audit log, because the model's own reasoning trace is not a substitute for an authoritative record of what action was actually taken and by what authorization. This is the architecture pattern behind agentic capabilities in Norra, Algomox's agentic AI workforce layer: the reasoning model proposes actions, but a separate policy and execution layer — not the model itself — enforces what the agent is actually permitted to do, and that enforcement layer runs entirely within the customer's own control plane regardless of network posture.

For SOC and NOC operations specifically, this pattern matters because agentic remediation — automatically isolating an endpoint, rotating a credential, or adjusting an access policy in response to a detected threat — needs to happen with the same latency and reliability whether or not the environment has external connectivity. An air-gapped agentic SOC deployment supporting integrated NOC/SOC operations cannot have its incident response capability degrade because a cloud dependency is unreachable; the entire value proposition of on-prem agentic AI is that it works exactly the same during a network outage, a deliberate isolation event, or a sustained air-gapped mission as it does on a fully connected day.

Human-in-the-loop checkpoints deserve explicit design here, not an afterthought. Define, in policy before deployment, which agent actions are fully autonomous, which require analyst confirmation, and which are advisory-only regardless of model confidence — and enforce that tiering in the execution layer, not by trusting the model's own stated confidence score, which is not a reliable signal of correctness.

Hardening the stack: supply chain, identity, and network segmentation

An air-gapped LLM deployment is still a software supply chain with all the usual attack surfaces — it is simply that the attack surface must be assessed once, at import time, rather than continuously monitored via live telemetry.

Every component — the inference server (vLLM, Text Generation Inference, or llama.cpp derivatives), the container base images, the Python or Rust runtime dependencies, the vector database, the orchestration framework — needs to enter the environment through the same staged, hash-verified, scanned transfer workflow as the model weights. A shocking number of air-gapped deployments treat the model as the sensitive artifact requiring rigor while pulling the surrounding software stack in more casually; from a security standpoint that is backwards, since the inference server and its dependencies have a much larger and more historically exploited attack surface (deserialization vulnerabilities, dependency confusion, malicious PyPI packages) than the model weights themselves.

Identity and access inside the isolated network still needs a real IAM backbone — air-gapped does not mean unauthenticated. Every user and every service account interacting with the model, the retrieval index, or agent tool endpoints should authenticate through your existing internal identity provider, with role-based access controls distinguishing who can query the model, who can see which retrieval sources, and who can approve agentic actions. This is the same identity discipline that underpins identity and privileged access management everywhere else in the environment; an LLM endpoint that bypasses it because "it's just internal" is a soft spot an internal threat actor or a compromised workstation will find quickly.

Network segmentation inside the air gap matters too — not every internal system needs to reach the inference cluster directly. Place the model-serving layer behind an internal API gateway with its own authentication and rate limiting, segment the vector database and any sensitive retrieval corpora from general network access, and apply the same zero-trust principles internally that you would at a connected perimeter. Air-gapping the whole environment from the internet does not mean flattening the internal network into an implicit trust zone; the continuous threat exposure management discipline of assuming compromise and validating every access path applies just as much inside the gap as outside it.

Day-two operations: monitoring, patching, and lifecycle management

Standing up the air-gapped stack is the easy part relative to operating it for years. A handful of operational patterns separate deployments that stay trustworthy from ones that quietly rot.

• Version everything, immutably. Every model, adapter, embedding model, guardrail classifier, and retrieval corpus version gets an immutable identifier and a manifest recording its provenance, evaluation results, and promotion date. Rollback to a known-good version must be a fast, tested operation, not an emergency improvisation.
• Schedule the transfer cadence deliberately. Decide upfront whether model and security updates cross the gap weekly, monthly, or quarterly, and staff the DMZ staging and verification process accordingly. An ad hoc "whenever someone remembers" cadence is how stale, unpatched inference stacks accumulate silent vulnerabilities.
• Monitor inference telemetry locally. Latency, token throughput, GPU utilization, and error rates all need local dashboards and alerting — there is no vendor status page to check. Feed this into whatever internal observability stack already monitors the rest of your infrastructure rather than standing up a bespoke silo.
• Log every prompt and response for audit, with retention aligned to your compliance regime. This is often the most sensitive data store in the entire deployment and needs its own access controls, separate from general application logs.
• Rehearse the disaster recovery path. If a GPU node fails or storage is corrupted, can the model-serving capability be restored from your internal artifact store within your recovery time objective, without any external dependency? Test this, not just document it.
• Maintain a deprecation policy. Base models age; a model trained on data with a mid-2024 cutoff will increasingly misunderstand references to events, tools, and terminology from 2026 and beyond. Plan for periodic base-model refresh cycles as part of the standing transfer cadence, not as an emergency response to visibly degraded output.

Cost modeling deserves explicit attention here too, because air-gapped GPU capacity is capital expenditure with none of the demand-elasticity of a cloud API. Sizing decisions made at deployment time are effectively locked in for the hardware refresh cycle (typically 3–5 years), so err toward provisioning for growth in concurrent usage and model size rather than the minimum viable footprint — the acquisition lead time to add capacity later, especially in a regulated or defense procurement context, can run into months.

A decision framework for choosing your posture

Not every workload needs full air-gapping, and over-isolating is its own cost — slower iteration, harder troubleshooting, more manual operational burden. Use a structured framework rather than defaulting to the most restrictive option out of caution.

1. Classify the data. What is the highest sensitivity level of data the model will ever see in prompts, retrieval context, or fine-tuning data? This alone often dictates the floor for isolation posture, independent of any other factor.
2. Map regulatory and contractual obligations. Does a specific regulation, contract clause, or classification guide mandate a particular residency or isolation posture, or is this a risk-based internal decision with more flexibility?
3. Assess connectivity reliability requirements. Does the mission require the AI capability to function during a connectivity outage or a deliberate network isolation event (a ship at sea, a forward base, a deliberately segmented OT network)? If so, air-gapped or at minimum fully-functional-offline architecture is mandatory regardless of data sensitivity.
4. Evaluate operational maturity. Does the team have the capability to run the full staged-transfer, evaluation, and guardrail pipeline described above? Under-resourcing this is worse than choosing a slightly less isolated posture that a connected update mechanism can help sustain.
5. Right-size per workload, not per organization. A single organization can and often should run a fully air-gapped instance for its most sensitive workloads alongside an on-prem-connected instance for lower-sensitivity general assistance — forcing every use case into the same posture wastes resources on the low-sensitivity side and under-protects if you compromise on the high-sensitivity side to keep one uniform architecture.

This tiered approach is exactly how Algomox architects deployments across MoxDB-backed data platforms and the broader ITMox/CyberMox stack: the same underlying AI-native architecture supports a spectrum from fully connected SaaS to fully air-gapped sovereign deployment, with the isolation posture a deployment-time configuration decision rather than a different product.

Frequently asked questions