Cost and Performance of Self-Hosted AI

Every board asks the same question once the AI pilot works: what does this cost to run for real, and can we trust it enough to put it near production data? Self-hosted AI answers both questions at once — but only if you engineer it with the same rigor you would apply to a payments platform, not a weekend GPU experiment.

The real question behind self-hosting

Self-hosting a large language model is rarely, at its core, a cost decision. It is a control decision that happens to have cost consequences. Regulated banks, defense contractors, health systems, and critical infrastructure operators do not move inference in-house because a spreadsheet says GPUs are cheaper than API tokens — they move it in-house because a third-party API call means customer data, threat intelligence, or operational telemetry crosses a boundary they are contractually or statutorily required to keep closed. Once that boundary requirement is established, cost and performance stop being optional considerations and become the engineering problem you have to solve, because the alternative — a slow, expensive, unreliable self-hosted stack — is what gets self-hosting initiatives cancelled after eighteen months.

This article is written for the people who have to make self-hosted AI work in production: the SRE who owns the GPU fleet's uptime, the SOC analyst who needs triage latency low enough to matter, the platform engineer sizing a cluster for a data residency mandate, and the IT operator running a facility with no path to the public internet at all. We will walk through the architecture patterns that actually hold up, the unit-economics math that lets you defend a budget, the performance levers that separate a usable deployment from an abandoned one, and the operational discipline — monitoring, patching, model lifecycle — that keeps a self-hosted estate alive past its first year.

Algomox builds agentic AI operations software — ITMox for IT operations, CyberMox for security operations, Norra as the agentic workforce layer, and MoxDB as the data foundation underneath all of it — and every one of those products has to run in customer environments that range from public cloud to fully air-gapped government networks. That range is precisely why the cost and performance engineering below is not theoretical: it is the set of decisions we make repeatedly when a customer says "this has to run inside our four walls."

Why organizations self-host in the first place

Before sizing hardware, be honest about which driver applies to your organization, because the driver determines which trade-offs are non-negotiable and which are just preferences.

• Data residency and sovereignty. National or sectoral regulation (financial services examination guidance, defense classification rules, EU data protection expectations, sector-specific critical infrastructure mandates) requires that certain data classes never leave a jurisdiction or a specific network enclave, full stop. No API terms of service, however generous, satisfy a requirement that the data never transits a third party's infrastructure.
• Air-gap and classified environments. Some networks have no route to the public internet by design. If there is no route out, there is no API to call — self-hosting is the only option, not a preference.
• Cost at sustained volume. API pricing is attractive at low and moderate volume because it converts capital expense into linear operating expense with zero idle cost. Above a certain sustained token volume, the math inverts: owned or reserved GPU capacity amortizes below the marginal cost of continued API consumption. Where that crossover sits is the subject of the next section.
• Latency and determinism. A SOC alert-triage pipeline or an OT control-loop advisory system cannot tolerate the variable queueing delay of a shared multi-tenant API during a public provider's peak-load periods. Local inference gives you a latency budget you control end to end.
• Vendor and model-supply-chain risk. Model deprecations, silent behavior changes after a provider-side update, and rate-limit throttling during incidents are unacceptable when the model sits in a detection-and-response critical path. Pinning an open-weight model version under your own control removes that variable entirely.
• IP and prompt confidentiality. Proprietary detection logic, playbooks, and runbook content encoded in system prompts and retrieval corpora are themselves intellectual property. Some organizations simply do not want that content transiting a vendor's inference stack, logged or not.

Notice that these drivers are not mutually exclusive, and in practice most self-hosting programs are driven by two or three of them simultaneously — a bank citing both data residency and cost at volume, a defense integrator citing air-gap and IP protection together. Map your own drivers explicitly at the start of the program; it will save you from over-building for a constraint you do not actually have (for example, provisioning full air-gap tooling for a workload that only needs regional data residency, which is a materially lighter architecture).

The real unit economics: API tokens versus owned GPUs

The comparison engineers actually need is cost per million tokens processed, fully loaded, not sticker price per GPU. Fully loaded means hardware amortization, power and cooling, networking, storage, the engineering and SRE time to operate the fleet, and — critically — the utilization rate you can realistically sustain, not the theoretical peak throughput a vendor benchmark advertises.

Building the cost model

Start from a concrete example: an 8x H100 (80GB) node, roughly the reference building block for serving a 70B-parameter class model at production quality with reasonable concurrency. As of 2026 pricing, that node runs on the order of $250,000–$300,000 fully configured (GPUs, host, NVLink/NVSwitch fabric, high-speed NICs), or roughly $6–9 per GPU-hour if leased from a specialized GPU cloud instead of purchased outright. Depreciate a purchased node over three years and add 15–20% annually for power, cooling, and data-center floor space, and you land near $110,000–$130,000 per year in pure infrastructure cost, before staff time.

That node, running a well-optimized serving stack (vLLM or TensorRT-LLM with continuous batching, discussed below), can sustain on the order of 15,000–30,000 output tokens per second in aggregate across concurrent requests for a 70B dense model at FP8, depending on sequence length and batch composition. At 25,000 tokens/sec sustained and 70% realistic utilization (accounting for off-peak hours, maintenance windows, and burstiness), that is roughly 550 billion tokens per year of usable throughput. Divide the fully loaded annual cost by that throughput and you get a marginal cost in the range of $0.20–$0.25 per million tokens — and that figure drops further once you factor in that a single node typically serves both prompt and completion tokens across many concurrent applications, amortizing the fixed cost across a much larger blended workload.

Compare that to commercial frontier-model API pricing, which for a comparable-quality proprietary model commonly sits in the $2–$15 per million output tokens range depending on model tier, with input tokens priced lower but still non-trivial at volume. The crossover point — where owned infrastructure becomes cheaper than API consumption — typically falls between 3 and 8 billion tokens per month for most organizations, depending on the specific API tier and the open-weight model quality you are willing to accept as substitute. Below that volume, API pricing wins outright once you account for the engineering time to operate a GPU fleet. Above it, ownership wins by a widening margin.

The utilization trap

The single most common mistake in self-hosting cost cases is modeling 90%+ utilization based on a load test, then discovering real production utilization sits at 25–35% because traffic is bursty, business hours are concentrated, and nobody wants to run a SOC triage model at 2 a.m. queue depth zero while paying for eight idle GPUs. Three mitigations matter here:

1. Multi-tenancy across workloads. Run IT operations triage, security alert enrichment, and internal knowledge-base retrieval on the same GPU pool with request-level routing, rather than dedicating a node per use case. This is precisely the pattern behind Algomox's AI-native stack — a shared inference layer that multiple agentic workflows draw from, so idle capacity in one workload absorbs burst traffic in another.
2. Elastic bursting to a secondary tier. Keep a baseline of owned or reserved capacity sized to median load, and burst overflow to on-demand cloud GPU instances or, for non-sensitive workloads only, a commercial API. This requires your routing layer to classify data sensitivity before deciding where a request goes — never burst classified or regulated data to an external tier.
3. Quantization and smaller specialist models. A well-tuned 8B or 14B model handling classification, summarization, and enrichment tasks can absorb 60–80% of total request volume in an operations pipeline, reserving the expensive 70B+ model for genuinely hard reasoning tasks. This alone can cut effective GPU-hours by half.

Reference architecture patterns for self-hosted inference

There is no single "self-hosted AI architecture" — there are three distinct patterns, and choosing the wrong one for your constraint set is the fastest way to blow both budget and timeline. Match the pattern to the driver you identified earlier.

Pattern one: connected on-prem, cloud-adjacent

Data stays on-premises for inference and storage, but the environment retains outbound connectivity for model updates, threat intelligence feeds, telemetry, and vendor support. This is the right pattern when your driver is data residency or cost at volume, but not a hard air-gap requirement. It is the least operationally burdensome of the three because you can still pull model weight updates, security patches, and CVE feeds automatically, and you can lean on cloud-based CI/CD for model evaluation pipelines that run against synthetic or de-identified data before promoting a model version on-prem.

Pattern two: hybrid with a sovereign inference tier

A subset of workloads — typically the ones touching regulated customer data or classified telemetry — run entirely within a sovereign enclave, while lower-sensitivity workloads (general IT ticket summarization, non-sensitive knowledge base search) continue to use cloud APIs or a shared multi-tenant cloud inference tier. This is the most common pattern for enterprises with mixed data sensitivity, and it requires a data-classification-aware routing layer at the front of your AI pipeline: every request gets tagged with a sensitivity label before an orchestrator decides whether it goes to the sovereign tier or the cloud tier. This is exactly the kind of routing Algomox implements across agentic SOC and integrated NOC/SOC deployments, where alert enrichment involving customer PII or classified indicators is pinned to the sovereign tier while generic vulnerability-description summarization can run wherever capacity is cheapest.

Pattern three: full air-gap

Zero connectivity to any external network, ever. Model weights, container images, vulnerability feeds, and dependency packages must be physically transferred in via a one-way diode or an approved removable-media transfer process, typically with a manual or semi-automated review gate. This pattern carries the heaviest operational overhead: every model update, CVE patch, and threat-intel refresh is a scheduled, audited transfer event rather than an automatic pull. Air-gapped programs succeed when they treat the transfer process itself as a first-class piece of infrastructure — a dedicated transfer cadence (weekly or biweekly is typical), a staging environment that validates artifacts before they cross the boundary, and cryptographic signing of every artifact so the receiving environment can verify provenance without needing network access to a certificate authority.

Choosing open-weight models: quality, size, and licensing

Self-hosting only works if the open-weight model you pick is actually good enough for the job, and "good enough" is a workload-specific judgment, not a leaderboard ranking. The open-weight landscape in 2026 spans several credible families — Llama-derivative dense and mixture-of-experts models, Mistral's dense and MoE lines, Qwen's broad size range, DeepSeek's MoE architectures, and Microsoft's Phi small-model family — and the right choice depends on three variables: task complexity, latency budget, and licensing terms.

Sizing to the task, not the leaderboard

A common and expensive mistake is deploying a single large general-purpose model for every task in a pipeline. In practice, most operational AI pipelines — whether alert triage, ticket classification, or log summarization — decompose into a handful of task types with very different reasoning demands:

• Classification and extraction (severity scoring, entity extraction, log field parsing) — a 3B–8B model, fine-tuned or well-prompted, matches larger-model accuracy at a fraction of the latency and cost.
• Summarization and enrichment (incident narrative generation, alert context assembly) — a 14B–32B model is typically the sweet spot; below this range summaries start losing coherence over longer contexts.
• Multi-step reasoning and planning (root-cause analysis chains, agentic remediation planning, correlation across disparate telemetry) — this is where 70B+ dense models or MoE models with large active-parameter counts earn their cost; smaller models measurably degrade on multi-hop reasoning benchmarks.

Running a tiered model estate — small models handling the 70–80% of high-volume, low-complexity requests, with escalation to a larger model only when a confidence threshold is not met — is the single highest-leverage architecture decision for both cost and latency. It is also directly analogous to how a well-run SOC triages: junior analysts handle volume, senior analysts handle escalations, and the routing logic between them is the actual engineering.

Licensing discipline

Open-weight does not always mean unrestricted-use. Read the actual license text, not the marketing summary, for three things: field-of-use restrictions (some licenses restrict use above a monthly active user threshold or in specific commercial contexts), redistribution terms if you plan to fine-tune and redistribute derivative weights internally across business units, and attribution or naming requirements. For regulated industries, get your legal and compliance team to sign off on the specific license version, not just the model family, since license terms have changed across major version releases for more than one popular model family in the last two years.

Fine-tuning versus retrieval augmentation

Do not default to fine-tuning when retrieval-augmented generation solves the problem more cheaply and more auditably. Fine-tuning is justified when you need the model to internalize a stable behavior pattern, tone, or structured output format across thousands of examples, and when you can commit to re-validating and re-certifying that fine-tuned model on every base-model update. Retrieval augmentation — grounding the model at inference time against a curated, versioned knowledge corpus stored in a system like MoxDB — is preferable when the underlying facts change frequently (vulnerability databases, runbook content, asset inventories) and when you need per-response citation back to a source document for audit purposes. In most operations and security use cases, a well-built retrieval layer outperforms fine-tuning on factual accuracy and is dramatically cheaper to keep current, because updating a document corpus is a data-pipeline problem while updating fine-tuned weights is a full retraining-and-revalidation problem.

Performance engineering: the serving stack

Model choice determines your ceiling; the serving stack determines whether you actually reach it. This is where most homegrown self-hosting projects lose the most avoidable ground — teams deploy a model behind a naive single-request HTTP wrapper and then wonder why throughput is a fraction of published benchmarks.

Continuous batching and paged attention

The two mechanisms that separate production-grade serving from a demo script are continuous batching and paged attention (KV-cache paging), both implemented in serving frameworks such as vLLM, NVIDIA's TensorRT-LLM, and Hugging Face's TGI. Continuous batching allows new requests to join an in-flight batch as soon as GPU capacity frees up, rather than waiting for the whole batch to complete — this alone typically improves throughput by 3–10x over naive static batching because request lengths vary widely and static batches waste capacity waiting on the longest sequence. Paged attention manages the KV cache (the per-token attention memory each request accumulates) in fixed-size blocks rather than contiguous pre-allocated memory, which eliminates the fragmentation that otherwise limits how many concurrent sequences a given amount of GPU memory can hold. Together, these two mechanisms are usually the single largest performance lever available to you, and they cost nothing beyond the engineering time to adopt a mature serving framework instead of a hand-rolled inference loop.

Quantization

Precision reduction is the second major lever. FP8 quantization on Hopper- and Blackwell-class GPUs (H100, H200, B200) typically costs 1–2% relative accuracy on standard benchmarks while roughly doubling throughput and halving memory footprint compared to FP16/BF16, making it close to a strict win for most production workloads on supported hardware. INT4 quantization (via GPTQ, AWQ, or newer per-channel schemes) pushes memory savings further — a 70B model that needs roughly 140GB in FP16 fits in around 35–40GB in INT4, meaning it can run on a single 80GB GPU instead of requiring a multi-GPU tensor-parallel split — but accuracy degradation is more workload-dependent and must be validated against your specific task, not just a general leaderboard, before committing to it for anything touching security decisions or compliance output.

Speculative decoding

Speculative decoding pairs a small, fast draft model with the target model: the draft model proposes several tokens ahead, and the target model verifies them in a single forward pass, accepting the ones that match what it would have generated itself. Because verification is parallelizable across the speculated tokens, well-tuned speculative decoding delivers 1.5–2.5x latency improvement on the same hardware for tasks with predictable token patterns (structured output, code, templated summaries), with negligible quality impact because the target model still governs every accepted token.

Parallelism strategy

For models too large for a single GPU, you choose between tensor parallelism (splitting individual layers across GPUs, requiring fast interconnect like NVLink) and pipeline parallelism (splitting the model's layers across GPUs sequentially, more tolerant of slower interconnect but adding latency from inter-stage handoffs). For most on-prem deployments with a single 8-GPU NVLink-connected node, tensor parallelism across 2, 4, or 8 GPUs is the right default for a 70B+ model; pipeline parallelism becomes relevant only when you must span multiple physical nodes without a high-bandwidth fabric between them, which is a materially harder networking problem for an on-prem team to get right and should be avoided unless model size genuinely forces it.

Capacity planning: sizing the fleet to real workload

Sizing a GPU fleet correctly requires working backward from concurrency and latency requirements, not forward from a budget number. Start by characterizing your workload along four axes: requests per second at peak, average input and output token lengths, acceptable p50/p95/p99 latency, and the acceptable queueing behavior during burst.

For a SOC alert-triage pipeline handling, say, 200 alerts per minute at peak with an average 1,500 input tokens (alert context plus retrieved playbook fragments) and 300 output tokens (triage verdict plus reasoning), and a p95 latency target of 3 seconds, work through the math as follows: 200 requests/minute is roughly 3.3 requests/second. At 300 output tokens per response and a 3-second p95 target, you need each request's decode phase to complete in well under 3 seconds even under contention, which at typical decode speeds for a well-quantized 32B model on a single H100 (roughly 80–120 tokens/sec per concurrent stream under moderate batching) means you need enough concurrent capacity to hold your peak in-flight request count without queueing past your latency budget. In practice, this workload comfortably fits on a single 4-GPU tensor-parallel deployment of a 32B model with headroom for burst to 2–3x peak, whereas the naive approach of provisioning a 70B model for the same task would require double the GPU count for a task that a properly tiered 32B (with escalation for hard cases) handles just as well.

Build in headroom deliberately: production capacity planning should target 50–60% steady-state utilization at expected peak load, not 90%, because AI inference workloads are bursty in ways that traditional web traffic is not — a single major security incident can spike alert-triage volume 5–10x within minutes, and a fleet sized to steady-state peak with no headroom will queue or shed load exactly when the triage capability matters most. This is the same principle as sizing a SOC's human analyst bench for surge capacity, applied to GPU capacity instead.

Storage and networking deserve equal attention. Model weights for a 70B model in FP16 are roughly 140GB; loading that from slow network storage on every pod restart or autoscale event adds minutes of cold-start latency. Use local NVMe caching of model artifacts on each inference node, and warm standby replicas for latency-critical workloads rather than relying on cold autoscaling. For retrieval-augmented pipelines, the vector database and document store sit on the same latency-critical path as the model itself — a slow retrieval call adds directly to end-to-end response time, so co-locate retrieval infrastructure on the same high-bandwidth network segment as the inference cluster, not across a WAN link to a separate data center.

Monitoring and operating the fleet like production infrastructure

A self-hosted AI fleet is production infrastructure and needs the same operational discipline as any other Tier-1 system: golden signals, capacity alerting, and a defined on-call rotation. The metrics that matter for AI inference specifically, beyond the standard CPU/memory/network signals, include GPU utilization and memory occupancy per device, KV-cache occupancy (a leading indicator of queueing before latency actually degrades), time-to-first-token and inter-token latency separately (they have different causes — time-to-first-token is dominated by prompt processing and queue depth, inter-token latency by decode throughput), request queue depth, and batch composition (average batch size and sequence-length variance, which tells you whether your traffic mix is causing padding waste).

Set alerting thresholds on KV-cache occupancy before it hits 100%, because once the cache fills, the serving framework starts preempting or rejecting requests, which shows up to users as sudden latency spikes or outright failures rather than graceful degradation. A cache occupancy alert at 75–80% gives you a window to trigger horizontal scale-out or shed non-critical traffic before users notice.

Model-level quality drift is a second, easily neglected monitoring dimension. Unlike a stateless microservice, a model's output quality can degrade for reasons unrelated to infrastructure health — a prompt template change, a retrieval corpus that has grown stale, or an upstream data schema change that silently breaks how context gets constructed. Instrument a continuous evaluation harness that runs a fixed set of representative test cases against production model endpoints on a schedule (daily is reasonable for high-stakes pipelines), scoring outputs against expected patterns or a judge model, and alert when scores drift outside a control band. This is the AI-specific equivalent of synthetic transaction monitoring, and it is the single most common gap in self-hosted deployments that otherwise have excellent infrastructure observability.

Patch and vulnerability management for the serving stack itself matters more in self-hosted environments than API consumers ever have to think about, because you now own the full software supply chain: the base container image, the CUDA and driver stack, the serving framework, and every Python dependency in between. Treat the inference container image with the same CVE-scanning and patch cadence discipline you would apply to any internet-facing service, and track it explicitly against your continuous threat exposure management program rather than treating it as a special AI exception exempt from normal vulnerability management.

Security and access control for the inference layer

A self-hosted model endpoint is a new privileged service in your environment, and it needs to be treated with the same access-control rigor as a database holding sensitive records, because in practice its context window often contains exactly that kind of data. Three controls matter most.

First, authenticate and authorize every call to the inference endpoint individually rather than treating it as an open internal service reachable by anything on the same network segment — this is a direct extension of identity and privileged access management principles applied to a new class of internal service. Service-to-service calls into the model endpoint should carry scoped, short-lived credentials, and you should log which upstream service and, where traceable, which human-initiated action generated each inference request, so a security investigation can reconstruct exactly what data reached the model and why.

Second, treat prompt injection and data exfiltration through model output as a real attack surface, not a theoretical one, particularly for any pipeline that ingests untrusted content (email bodies, ticket descriptions, web-scraped threat intelligence, log lines containing attacker-controlled strings) into a prompt. Apply input sanitization and output filtering consistent with your broader AI security posture, and never grant a model-serving pipeline direct write access to production systems without a human or deterministic policy-engine approval gate in between, regardless of how well-tested the agentic workflow appears in staging.

Third, encrypt model weights and the KV cache at rest and, where hardware supports it, use confidential computing (GPU-based trusted execution environments now shipping on H100 and newer NVIDIA data-center GPUs) for the highest-sensitivity workloads, so that even a compromised hypervisor or a malicious co-tenant on shared infrastructure cannot read model weights or in-flight inference data from memory. This is a meaningful differentiator for organizations self-hosting specifically because of data sensitivity concerns, and it is worth the modest performance overhead (typically single-digit percentage points) for the workloads that genuinely need it.

Worked example: sizing an air-gapped SOC triage deployment

Consider a concrete scenario that pulls the preceding sections together: a defense-adjacent organization needs an air-gapped agentic SOC capable of triaging roughly 15,000 alerts per day across three shifts, with a p95 triage latency target of 5 seconds and zero external connectivity.

Workload characterization first. 15,000 alerts per day averages to about 10 per minute, but SOC traffic is heavily front-loaded during business-hours shifts and spikes during active incidents, so plan for a peak of 60–80 alerts per minute (roughly 1–1.3 per second) sustained, with burst tolerance to 3–4x that during a major incident. Each triage pass involves retrieval of relevant playbook and asset-context documents (roughly 2,000–3,000 input tokens after retrieval) and generates a structured verdict with reasoning (roughly 250–400 output tokens).

Model tier selection next. A three-tier estate fits this workload well: an 8B model handles initial severity classification and deduplication for the roughly 70% of alerts that are routine or previously-seen patterns; a 32B model handles enrichment and narrative generation for the remaining alerts requiring genuine synthesis across multiple telemetry sources; and a 70B model, invoked only when the 32B model's confidence score falls below a defined threshold or when correlation spans more than a defined number of distinct telemetry sources, handles the hardest multi-hop root-cause cases — typically under 5% of total volume.

Hardware sizing follows from that tiering. The 8B and 32B tiers, quantized to FP8, comfortably run concurrently on a single 4-GPU node with tensor parallelism for the 32B model and the 8B model sharing remaining capacity; the 70B tier, invoked rarely enough that low utilization is acceptable, can run on a second, smaller 2-GPU node using INT4 quantization to fit within tighter memory, accepting the modest additional latency because it serves under 5% of requests. Total fleet: roughly 6 GPUs, a fraction of what a single-tier "always use the biggest model" architecture would require, and well within the footprint that fits a single rack in a hardened facility.

Air-gap operations layer last. Model weights, the retrieval corpus (playbooks, asset inventory snapshots, prior incident write-ups), and container images are staged externally, scanned, cryptographically signed, and transferred via an approved one-way transfer process on a biweekly cadence, with an expedited emergency transfer path reserved for critical CVE patches to the serving stack itself. Internally, a private model registry and artifact store track every version deployed, with rollback capability to the prior validated version if a new model or corpus update degrades the continuous evaluation harness's scores.

This pattern — tiered models, right-sized hardware, disciplined transfer cadence, and continuous evaluation — is directly transferable to AI-driven XDR alert triage and broader detection and response workloads outside the air-gap context as well; the air-gap simply adds the transfer-cadence discipline on top of an architecture that is sound regardless of connectivity.

Common failure modes and how to avoid them

Most self-hosted AI programs that fail do so for a small, recurring set of reasons, and recognizing them early is cheaper than discovering them in a post-mortem.

• Provisioning for the demo, not the distribution. Teams size hardware based on a proof-of-concept's happy-path latency and then discover real production traffic has a long tail of large contexts and concurrent bursts the demo never exercised. Always capacity-plan from a realistic traffic distribution, including the 95th and 99th percentile request shapes, not the median.
• Treating the serving framework as an afterthought. As covered above, the difference between a naive wrapper and a mature serving stack (vLLM, TensorRT-LLM) is often 5–10x throughput on identical hardware. Budget real engineering time for this layer; it is not a solved problem you can bolt on in a week.
• No continuous evaluation, only a one-time acceptance test. Models and retrieval corpora drift. Without a scheduled evaluation harness, quality regressions surface first as user complaints or, worse, as missed detections, rather than as an alert your team saw coming.
• Underestimating the transfer-cadence burden in air-gapped environments. Organizations frequently plan for the compute but not for the recurring labor of scanning, signing, and manually transferring every model, corpus, and patch update. Staff this as an ongoing operational function with a named owner, not a one-time migration project.
• Ignoring licensing terms on open-weight models. A model that looked free during the pilot can carry field-of-use restrictions that make production deployment at scale a legal problem. Get this reviewed before, not after, the architecture is finalized.
• Single-tier model estates. Running one large model for every task inflates GPU spend for no accuracy benefit on the majority of low-complexity requests. Tiering is the highest-leverage cost lever available and is frequently skipped simply because it requires more upfront workload analysis than standing up one endpoint.

A decision framework for your own deployment

Bring the preceding sections together into a sequence of questions you can walk through with your own architecture team. First, identify your actual driver — sovereignty, air-gap, cost at volume, latency determinism, or IP protection — because it determines which of the three architecture patterns (connected on-prem, hybrid sovereign tier, or full air-gap) you need, and over-building for a constraint you do not have wastes both budget and time. Second, characterize your workload's request-complexity distribution honestly, because that distribution determines whether a tiered model estate is viable and how many GPUs you actually need, not the peak-case single-large-model estimate. Third, commit to a mature serving stack from day one rather than a custom wrapper, because the throughput difference is not a marginal optimization — it is frequently the difference between meeting your latency SLA and not. Fourth, build the operational muscle — monitoring, continuous evaluation, patch cadence, and for air-gapped environments, a disciplined transfer process — before go-live, not as a follow-up project, because these are the functions that keep a self-hosted estate trustworthy past its first quarter in production. Finally, revisit the API-versus-owned math on a fixed cadence, ideally quarterly, because both open-weight model quality and GPU pricing move quickly enough that a crossover analysis done at project kickoff can be stale within two quarters.

Frequently asked questions