Self-hosting an open-weight model is a control and privacy decision that occasionally also saves money at very high, steady volume. It is not a default cost optimization. Every few months, a release like Kimi K3, Moonshot AI's 2.8-trillion-parameter model matching the capabilities of some closed frontier labs, restarts the same fragmented discussions about self-hosting, residency, and compliance, and most of them stall at the same point: the per-token discount gets treated as the whole ledger.
This guide reads the full ledger. It sets three costs side by side (what the closed labs charge per token, what the open-weight platforms charge per token, what a GPU costs per hour) and names the utilization line that decides which one wins. It weighs the four ways to run a model, gives you a two-question decision tree, and closes on the number no leaderboard can supply: how the model performs on your own workload. Most figures are August 2026 readings with the shelf life of milk, so re-check them before a decision rests on them.
The short answer
Where each option fits, by sustained daily token volume
Below a few million tokens a day, or whenever traffic is bursty. This is the default, and for most teams it stays the default.
Per-token billing, several times cheaper than the closed tiers, and moving between providers is usually a base-URL change.
Sustained, at 50 to 60% utilization, or when a residency requirement rules out shared endpoints.
Justified by residency, auditability, fine-tuning ownership, or independence from a provider's roadmap. Savings are the bonus, not the reason.
What is an open-weight model? Open weights vs. open source vs. closed weights
An open-weight model is a large language model whose trained weights are publicly released, so anyone with the right tools can download it, run it, modify it, and build on it. A closed-weight model stays behind its provider's API, so inference sends your data to the provider's endpoint, your compliance posture is only as strong as theirs, and tuning happens under the provider's terms or not at all.
Open weights are not open source. Open-weight providers release the weights but often withhold what would let you recreate the model: training methods, datasets, pipeline, and optimization details. Open source distributes all of it, and providers often do both: Alibaba ships Qwen3.8-27B under Apache 2.0 while the 2.4-trillion-parameter Qwen3.8-Max carries a custom license.
If all you require is inference, fine-tuning, or deployment, open-weight is enough. If you must audit or reproduce the training, you need open source. Read the license file before you build.
Closed weights vs. open weights vs. open source
Closed weights
Behind the provider's API- Weights
- Not released
- Where inference runs
- The provider's endpoint
- Fine-tuning
- Under the provider's terms, or not at all
- Compliance
- Only as strong as the provider's
Open weights
Download, run, modify, build on- Weights
- Publicly released
- Where inference runs
- Anywhere you choose
- Training recipe
- Often withheld: methods, datasets, pipeline
- Enough for
- Inference, fine-tuning, deployment
Open source
Everything needed to recreate it- Weights
- Publicly released
- Training recipe
- Published: methods, datasets, pipeline, optimization
- Enough for
- Auditing or reproducing the training
- Before you build
- Read the license file
How much GPU memory do you need to run an LLM?
Whether you rent the GPU or own it, the same arithmetic decides what runs. Most frontier open-weight models in 2026 are mixture-of-experts architectures: a huge total parameter count, a slim active slice per token. The active count governs speed. The total count governs whether the model fits in memory at all.
Budget about 2 GB per billion parameters at FP16, about 0.5 GB per billion at 4-bit, plus a 15 to 20% surcharge for the KV cache, activations, and framework overhead. An 8B model at 4-bit slips into 4 to 5 GB, a 70B wants close to 40 GB, and the 1.6T DeepSeek V4 Pro needs 800 to 900 GB. One caveat: the KV cache grows with every token you feed it, so once the model loads, the memory left over is your context budget.
Will it fit? Parameters × precision + overhead
The same arithmetic sizes the rented box. gpt-oss-120b (117B total, about 5B active) fits one H100. GLM-5.2 (744B total, 40B active) carries 744 GB of FP8 weights and a practical floor near 850 GB, which overruns eight H100s at 640 GB and lands you on 8× H200. Kimi K3 (2.8T total, 104B active) wants 1.5 to 1.7 TB at full precision, which is multi-node territory.
Read the active-versus-total split, do the arithmetic, and choose the smallest box that holds the weights plus your context budget.
Four ways to run an LLM: closed API, serverless, dedicated GPU, or your own hardware
| Closed API | Serverless open-weight | Dedicated cloud GPU | Your own hardware | |
|---|---|---|---|---|
| Billing | Per token, premium rates | Per token | Per GPU-hour | Purchase price plus power |
| Spin-up | Instant | Instant | Provisioned | Purchase and build |
| Where your data goes | The provider's endpoint | The provider's endpoint | An instance you control | Stays on-premise |
| Best for | Low or bursty volume, frontier reasoning | Prototypes, sustained volume up to ~100M tokens a day | Sustained volume past several hundred million tokens a day | Residency, ownership, independence |
| Ops burden | None | None | An MLOps owner; total cost runs 3 to 5x raw rental | The same, plus the hardware market |
Closed APIs: the tier you are competing against
The closed labs set the price the rest of the ledger is measured against. As of August 2026, Claude Opus 5 lists at $5.00 per million input tokens and $25.00 per million output; Claude Sonnet 5 at $2.00 and $10.00, an introductory rate made permanent on August 10, 2026; GPT-5.5 at $5.00 and $30.00; and GPT-5 Mini at $0.125 and $1.00 direct from OpenAI, roughly double through resellers. Cache reads run a tenth of fresh input. Batch endpoints halve the bill.
When to use a closed API
Volume sits below a few million tokens a day, traffic is bursty, or the work needs frontier reasoning and nothing in your compliance posture forbids sending data to the endpoint.
Serverless open-weight endpoints: cheaper per token, with a precision caveat
Open-weight platforms serve the same model families at a fraction of the closed rates. DeepInfra lists DeepSeek V4 Pro at $1.30 input and $2.60 output, and GLM-5.1 at $1.05 and $3.50. Qwen3-30B-A3B runs $0.12 and $0.50. DeepSeek V4 Flash runs about $0.14 and $0.28.
Kimi K3 is the exception that shows the tier structure. On OpenRouter, it lists at $2.60 and $13.00, more than Sonnet 5, because frontier-scale open weights command frontier-adjacent serving prices. Open weights are not automatically cheap. Small active counts are cheap.
The discount comes with a precision caveat. Most serverless providers serve in FP8 rather than the reference BF16. The largest published study on the topic (Kurtic et al., 2024), over 500,000 evaluations across the Llama-3.1 family, finds FP8 effectively lossless at every model scale and well-tuned INT8 costing only one to three points. The damage lives further down, in aggressive 4-bit and in activation or KV-cache quantization, and it lands hardest on multi-step math, scientific reasoning, and small models.
Ask what precision an endpoint runs, then measure it on your own set.
When to use serverless endpoints
You are prototyping, or sustained volume sits under roughly 100 million tokens a day, and your eval set confirms the served precision holds on your task.
Dedicated cloud GPUs: what self-hosting an LLM really costs
Two camps rent GPUs. Hyperscalers (AWS, GCP, Azure) sell enterprise SLAs, regions, and compliance at a premium: on-demand, an H100 runs $6.88 per GPU-hour on AWS, about $11 on GCP and $12.29 on Azure, and reserved or capacity-block terms bring AWS and Azure to roughly $5 to 8. GPU-specialist neoclouds (CoreWeave, Lambda, RunPod, Together, Fireworks, Modal and kin) price at $1.49 to 4.00 per H100 GPU-hour. GetDeploying's August 2026 index, tracking 322 H100 listings across 53 providers, records a $3.62 per hour on-demand average.
A rented GPU bills the same whether saturated or asleep, so the per-token cost of self-hosting sizes inversely with load. Practitioners want 50 to 60% sustained utilization before the numbers prove out, and bursty solo workloads rarely clear it. With that lens, the break-evens for a single reserved GPU sort into tiers: roughly 2 to 5 million tokens a day against premium closed APIs on raw rental math, and only against the priciest tiers; hundreds of millions a day against the Mini and Flash tiers; often out of reach entirely against the $0.14 to $0.50 open-weight tier, since a provider spreads the same hardware across thousands of tenants; and a clear win for rentals near 500 million tokens a day, roughly fifteen billion a month, which puts a frontier API bill well into six figures monthly.
Two RunPod H100s at roughly $4,400 a month all-in push about 2.2 billion tokens a month of a 70B-class model at 70% utilization, tokens that would cost five figures monthly at frontier output rates.
The sticker price is not the price. Total cost runs three to five times raw GPU rental once you count MLOps time, model-update cycles, monitoring, storage, and idle headroom. Every break-even above moves out by that multiplier.
When to rent dedicated GPUs
Sustained volume clears the break-even at 50 to 60% utilization, someone on the team owns MLOps, and either the volume or a residency requirement rules out the serverless tier. Prototype on serverless, then move to dedicated when you run the model most hours of most days. The two generally share an OpenAI-compatible API, so the migration is often a base-URL change.
Self-hosting an LLM on your own hardware
Models under roughly 100 billion parameters can run on a personal rig, and 4-bit quantization is the lever that makes them fit. The hardware market is the complication.
The RTX 5090 (32 GB, 1,792 GB/s, 575 W) is the consumer ceiling, and it fails the moment a model goes past 32 GB. Its $1,999 MSRP now sells for at least $5,000 as of September 2026, when it is in stock at all. A Mac Studio with 128 to 256 GB of unified memory is often the only sub-$5,000 desktop that loads a 70B in one shot, at fewer tokens per second.
J.P. Morgan estimates DRAM prices will have risen more than 400% from the start of 2024 to the end of 2026, with no relief before 2028, as AI data centers absorb the memory supply. The 96 GB RTX PRO 6000 Blackwell went from about $8,565 in March 2025 to $16,000 in August 2026.
The sweet spot on one desk is Qwen3.8-27B: a ~17 GB 4-bit GGUF on an RTX 4090 or 5090, with room for a slice of its 262K-token context, at around 65 tokens per second on community rigs. Kimi K3 sits above every consumer machine. Those throughput figures, like the launch benchmarks, are vendor- and enthusiast-reported until independent numbers land.
Run the ledger on the rig. At Anthropic's own $13 per developer per active day, a $5,000 RTX 5090 running Qwen3.8-27B needs about 385 active days to pay off against Opus-class API spend, a year and a half for a developer who codes every working day, or a few months of the heavy multi-agent kind. It costs you 575 watts at load, a dollar or two a day at average US residential rates. It requires you to verify that the smaller model actually covers your use cases, and it never covers its cost against a $0.14-per-million serverless endpoint.
When to buy your own hardware
Residency, auditability, fine-tuning ownership, or independence from a provider's roadmap justifies the box on its own terms. If your only reason is the bill, read the ledger twice.
Should you self-host an LLM? Decide in two questions
Two questions place most workloads
Does HIPAA, PCI DSS, ITAR, or a residency clause make endpoint inference a legal problem?
What is your sustained daily volume?
Four checks then decide where a self-hosted model lives.
Model size against hardware. A 27 to 32B dense model or a 30B-A3B mixture-of-experts at 4-bit fits a single 24 to 32 GB card. gpt-oss-120b wants one 80 GB GPU or a big unified-memory Mac. The 700B to 2.8T giants want 8× H200 and up, where serverless is almost always the saner spend.
Latency and concurrency. Interactive use needs 30-plus tokens per second single-stream. Serving a crowd needs batched inference on vLLM or SGLang on datacenter cards. If your rig cannot deliver the first, no economics can rescue the experience.
Team capacity. Without someone who owns MLOps, the three-to-five-times multiplier applies in full. Serverless exists so that this person can be optional.
Fine-tuning. If you need custom weights you can port, choose open-weight from the start. Fine-tuning is the third resort after prompting and retrieval, and it earns its cost when the problem is form rather than facts: a house style, a rigid output schema, a domain dialect. The event you are hunting is a tuned 27B beating a prompted frontier model on a narrow task, and the literature says it happens: the LoRA Land study (Zhao et al., 2024) found adapters on small bases beating GPT-4 on narrowly scoped tasks while GPT-4 keeps the broader ones. A QLoRA run on Qwen3.8-27B fits one 80 GB card and finishes in hours, and the adapter it produces, measured in megabytes, is the asset the closed labs cannot sell you.
Self-hosting examples: agentic coding and a HIPAA extraction pipeline
Agentic coding: route rather than choose
No workload stresses the ledger like agentic coding. Cursor and Claude Code loop through reading files, planning, writing code, running tests, and iterating, and every turn is a full API round trip carrying the entire conversation. That is 10 to 100 times the appetite of the chat window, 400K to 2M input tokens per task, and about $13 per developer per active day by Anthropic's own documentation.
Both tools can be pointed at a self-hosted backend. Cursor talks natively to any OpenAI-compatible server. Claude Code needs a translation layer to the Anthropic Messages format: claude-code-router for a solo developer, LiteLLM for a team that needs load balancing and audit logs.
The practical backends are Qwen3-Coder-Next (80B total, 3B active, 70.6% on SWE-bench Verified under Qwen's own scaffold), the large mixture-of-experts models, and Qwen3.8-27B on a single 24 GB card. Small dense models generally fail at reliable tool-calling, and the loop cannot run without it.
Three assumptions break when you substitute your own model. First, the tool-call dialect: harnesses parse function calls strictly, open models emit their own variants (a different JSON shape, an XML-ish block, prose leaking into the arguments), and a call the harness cannot parse looks exactly like a model that cannot use tools. vLLM ships per-model parsers for this, and choosing the right one is often the whole fix.
Second, the KV cache grows across the loop rather than resetting between messages, so a configuration that fits at turn five can exhaust VRAM by turn forty; prefix caching and the harness's compaction behavior matter more than the advertised context window. Third, latency compounds across the loop.
Hence the pattern production teams arrive at: route rather than choose. Send file reads and background turns to the local model and escalate the hard ones to a frontier model. The model is the easy swap. The loop's tolerance for a weaker one is the real constraint.
Route rather than choose
Local model
Routine turns: file reads and background work.
Frontier model
Hard turns escalate here.
A regulated extraction pipeline: question one decides it
Take a claims-extraction pipeline in a HIPAA shop. Question one answers yes, so the volume question is void, and the checklist decides where the model lives. The work is batch rather than interactive, so the 30-tokens-per-second floor does not apply, and a 27B dense model at 4-bit fits a single 24 to 32 GB card, so the box is small.
The task is extraction, the pattern where a tuned small model most often beats a prompted frontier one. That makes the case for a dedicated GPU or an on-premises card running a fine-tuned 27B, with the private eval set gating the swap from whatever frontier model the team prototyped on. The residency clause chose the option. The ledger only chose the box.
How to evaluate a self-hosted LLM on your own workload
Leaderboards are marketing artifacts by the time you read them. Every figure above prices a model its vendor benchmarked, and none of them says what that model does on your extraction set, your support transcripts, or your codebase. The ledger tells you what a token costs. It cannot tell you whether the cheaper token does the job.
Build a private eval set. Pull fifty to two hundred examples from your real workload. Grade mechanically where the task allows: exact match, unit tests, schema validation. Use a top frontier model as judge for the rest, and spot-check a sample by hand, because judges carry documented position, verbosity and self-preference biases.
Then measure the quantization bias. Run the same set against three targets: the BF16 reference weights, the FP8 endpoint you plan to buy, and the 4-bit GGUF on your own card. The published deltas are your prior (FP8 near-lossless, INT8 a point or three, 4-bit more competitive than folklore suggests), but a prior is not your workload. If the 4-bit build drops two points on your extraction set, that is the number that belongs in the ledger next to the money it saves.
Re-run at every seam. Treat every model swap, provider migration, and adapter update as a release, and gate it on the set. For the workload you care about, you are the independent judge.
That measurement is where we start with clients: the private eval set, the ledger priced against real volume rather than a rate card, and a routing design that sends each turn to the cheapest model that passes. Choosing where to run a model is arithmetic. Knowing whether the model you chose does your job is a measurement, and it is the one line in the ledger you have to write yourself.
Frequently asked questions
Is it cheaper to self-host an LLM?
Usually not. Self-hosting is a control and privacy decision that occasionally also saves money at very high, steady volume. A single reserved GPU breaks even against premium closed APIs around 2 to 5 million tokens a day on raw rental math, but total cost runs three to five times raw GPU rental once MLOps time, monitoring and idle headroom are counted. Rentals win clearly near 500 million tokens a day.
How much GPU memory do I need to run a 70B model?
Close to 40 GB at 4-bit. Budget about 0.5 GB per billion parameters at 4-bit, or 2 GB per billion at FP16, plus a 15 to 20% surcharge for the KV cache, activations and framework overhead. Once the model loads, the memory left over is your context budget.
What is the difference between open-weight and open-source models?
An open-weight model releases its trained weights, so you can download, run, modify and fine-tune it. An open-source model also publishes what it takes to recreate it: training methods, datasets, and pipeline and optimization details. If all you need is inference, fine-tuning or deployment, open-weight is enough.
Does quantization hurt LLM accuracy?
Less than most people expect. The largest published study, over 500,000 evaluations across the Llama-3.1 family, found FP8 effectively lossless and well-tuned INT8 costing one to three points. The damage shows up in aggressive 4-bit and KV-cache quantization, especially on multi-step math, scientific reasoning and small models. Measure it on your own eval set.
What GPU do I need to self-host an LLM?
It depends on the model. A 27 to 32B dense model or a 30B-A3B mixture-of-experts at 4-bit fits a single 24 to 32 GB card. gpt-oss-120b wants one 80 GB GPU or a big unified-memory Mac. The 700B to 2.8T giants want 8× H200 and up, where serverless is almost always the saner spend.
Can I use Claude Code or Cursor with a local model?
Yes. Cursor talks natively to any OpenAI-compatible server. Claude Code needs a translation layer to the Anthropic Messages format: claude-code-router for a solo developer, or LiteLLM for a team that needs load balancing and audit logs. The model also needs reliable tool-calling, which small dense models generally lack.
What is the difference between serverless and dedicated GPU hosting?
Serverless endpoints bill per token and spin up instantly, which suits prototypes and sustained volume up to roughly 100 million tokens a day. Dedicated cloud GPUs bill per GPU-hour and pay off once you run the model most hours of most days. The two generally share an OpenAI-compatible API, so the migration is often a base-URL change.
Sources
Model specifications: official model cards for Kimi K3, DeepSeek V4 Pro and Flash, GLM-5.2, gpt-oss-120b, Qwen3.8-27B and Qwen3-Coder-Next; independent positioning from Artificial Analysis; quantized build sizes from Unsloth.
Pricing: Anthropic platform pricing and the August 10, 2026 Sonnet 5 announcement; OpenAI pricing; DeepInfra and OpenRouter model pages; GetDeploying GPU Rental Price Index (August 2026); AWS, Azure and GCP instance pricing (verified September 16, 2026); RunPod pricing.
Hardware: Tom's Hardware and VideoCardz price histories; Apple pricing; J.P. Morgan Global Research, "The AI-Driven Memory Shortage."
Studies: Kurtic et al., "'Give Me BF16 or Give Me Death'? Accuracy-Performance Trade-Offs in LLM Quantization" (2024); Zhao et al., "LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4" (2024).
Agentic coding: Anthropic's Claude Code cost documentation; Qwen3-Coder-Next model card.
