AI Inference vs Training vs Fine Tuning | What’s the Difference?

	Training	Fine-tuning	Inference
What it does	Builds a model's general capability from scratch	Adapts an existing model to a specific task or domain	Runs the finished model to produce an answer
Typical cost	Up to roughly $500M for the largest known runs (Epoch AI, Nov 2025)	$4 to a few thousand dollars	Fractions of a cent per request, but continuous
Data needed	Trillions of tokens of general text	Hundreds to tens of thousands of curated examples	None — just the user's prompt
Who does it	A handful of frontier labs	Almost any engineering team	Everyone shipping an AI product
When you do it	Almost never; you buy this	When prompting and retrieval have genuinely failed	Every single request, forever

Those three words get used interchangeably in vendor decks, and the confusion is expensive. This guide separates them properly, gives you the 2026 numbers for speed and cost with sources attached, walks through a real fine-tuning job that cost about $10, and adds the fourth option that barely existed when we first published this piece: distillation and reinforcement fine-tuning. If you are a technical leader deciding where to spend your AI budget, this is the decision framework.

The short version

• Inference is now the majority of AI infrastructure spend. Gartner forecast in August 2026 that inference will take $23.3 billion of the $42 billion AI-optimized IaaS market this year — 55% — against $19 billion for training.
• What counts as fast now depends on the size of the model. On a 120B-parameter model, Cerebras reported in November 2025 that its WSE-3 runs GPT-OSS-120B at over 3,000 tokens per second, roughly 5x an NVIDIA Blackwell GB200 result of 650 tokens per second on the same model. At trillion-parameter scale the leading edge is closer to 1,000: Artificial Analysis measured Cerebras at 981 tokens per second on Kimi K2.6 in May 2026, 6.7x the next-fastest GPU cloud.
• Fine-tuning a small open-weight model is genuinely cheap: Together AI charges $0.48 per million training tokens for LoRA on models up to 16B parameters, with a $4 minimum per job, as listed on its pricing page in August 2026.
• Reinforcement fine-tuning costs 100x to 700x more than supervised fine-tuning and does not reliably win. TensorZero's September 2025 head-to-head found RFT cost $65.07 against SFT's $0.09 on a data extraction task, and actually degraded performance on a customer service task.
• On-policy distillation is the best price-performance story in the category. Thinking Machines Lab reported in October 2025 that on-policy distillation beat a reinforcement learning baseline on AIME'24 — the Qwen3 technical report's 74.4% against RL's 67.6% — using roughly 1,800 GPU hours instead of 17,920, which the post describes as one-tenth the cost of RL. Against off-policy distillation the saving is 9x to 30x, depending on whether you already have the SFT dataset.
• OpenAI is retiring its self-serve fine-tuning platform, announced May 7, 2026: organizations without recent fine-tuning activity lost the ability to create new jobs on July 2, 2026, and all remaining customers lose it on January 6, 2027.

Training: how a model gets built in the first place

Training is the process of taking a neural network with essentially random parameters and exposing it to a vast general dataset until those parameters encode the statistical structure of language. Before training, ask the model to complete "the students opened their…" and it produces noise. After training, it produces "laptops."

The scale is the point. Epoch AI's training cost dataset, last updated in November 2025, tracks spending on the largest runs growing at 2.4x per year, from AlphaGo Zero at about $600,000 in 2017 to Grok 4 at an estimated $500 million. Stanford's 2026 AI Index, as reported by IEEE Spectrum in April 2026, put Grok 4's training emissions above 72,000 tons of CO₂ equivalent.

This is why almost nobody reading this should train a foundation model. Treat the number as a benchmark for what you get essentially for free when you call an API. If you are weighing build-versus-buy at the model layer, our guide to large language models covers the architecture decisions that follow.

Training is a capital expense a few dozen companies in the world can afford. Inference is an operating expense you will pay every day for the life of your product. Budget accordingly.

Fine-tuning: teaching a trained model your specific job

Fine-tuning starts with a model that already understands language and nudges it toward a narrower behavior using a much smaller, curated dataset. Take the same "the students opened their…" prompt and fine-tune on fantasy novels, and the completion becomes "spellbook." You have not taught the model English. You have taught it your context.

The technique that made this affordable is LoRA — Low-Rank Adaptation — which freezes the original weights and trains a small set of new parameters alongside them. In the walkthrough below, that means retraining about 16% of the model rather than all of it.

Fine-tuning is the right tool when you need a consistent output format, a specific tone, or a behavior the base model keeps getting wrong. It is the wrong tool when what you need is current information — retrieval beats fine-tuning there almost every time, which is why our RAG practice is a separate service line. A model that does not know your Q3 pricing needs a retrieval pipeline, not a training run.

Inference: the part your users actually touch

Inference is the model doing its job. Parameters are fixed, nothing is learning, and a prompt goes in and tokens come out. It is the least glamorous of the three and by far the most consequential to your unit economics, because you pay for it on every request forever.

The market has caught up to that reality. At CES in January 2026, Computerworld reported Lenovo CEO Yuanqing Yang describing a flip from roughly 80% of AI spend going to training today to 80% going to inference in future, with Deloitte estimating inference accounted for half of all AI compute in 2025 and two-thirds in 2026. Gartner's August 10, 2026 forecast put hard numbers on it: $23.3 billion for inference against $19 billion for training within a $42 billion AI-optimized IaaS market growing 96.4% year over year.

How fast is inference in 2026?

When we first published this article in September 2024, the honest state of the art was roughly 100 tokens per second from a mainstream hosted chat model and roughly 1,000 tokens per second from Groq's custom silicon — a 10x gap that felt enormous. Both halves of that comparison have since been overtaken, and the shape of the question changed with them.

Here is where the numbers stand, with dates attached, because figures like these go stale within months:

Measurement	Figure	Hardware / provider	Source and date
GPT-OSS-120B, single stream	over 3,000 tokens/sec	Cerebras WSE-3	Cerebras, November 2025
GPT-OSS-120B, single stream	650 tokens/sec	NVIDIA Blackwell GB200 via Baseten	Cerebras, November 2025
Kimi K2.6 (trillion-parameter), single stream	981 tokens/sec	Cerebras	Artificial Analysis via Cerebras, May 2026
GPT-5.3-Codex-Spark, single stream	~1,000 tokens/sec	Cerebras	Forbes, February 2026
Claude Opus 4.6, single stream	~65 tokens/sec	Anthropic API	Forbes, February 2026
DeepSeek-R1, server scenario	8,064 tokens/sec per GPU	NVIDIA GB300 NVL72	MLPerf Inference v6.0, April 2026
DeepSeek-R1, offline scenario	2,494,310 tokens/sec total (9,821 tokens/sec per GPU)	4x NVIDIA GB300 NVL72 (288 GPUs)	MLPerf Inference v6.0, April 2026

Two things in that table deserve unpacking.

Single-stream speed and system throughput are different numbers

The most common mistake in inference benchmarking — one the original version of this article made — is comparing a single-user speed figure to an aggregate throughput figure as if they measure the same thing.

When Cerebras reports 981 tokens per second on Kimi K2.6, that is how fast one response streams back to one user. When MLPerf Inference v6.0 reports 2,494,310 tokens per second for four GB300 NVL72 racks — 288 GPUs, or 9,821 tokens per second per GPU — in April 2026, that is total tokens across thousands of concurrent requests. The first number tells you whether your product feels instant. The second tells you your cost per million tokens at scale. Optimize for the wrong one and you either ship something sluggish or overspend badly on hardware.

Custom silicon leads on speed, and that lead is real but narrow

Cerebras has the strongest claim to raw speed right now, with commercial validation to match: Forbes reported in February 2026 that it raised a $1 billion Series H at a $23 billion valuation and signed a $10 billion deal with OpenAI in January 2026 for 750 megawatts of wafer-scale systems, with OpenAI's GPT-5.3-Codex-Spark running on Cerebras at about 1,000 tokens per second.

But the counter-case matters. Cerebras's own November 2025 comparison priced its 3,000 tokens per second at $0.75 per million tokens against Baseten's Blackwell result of 650 tokens per second at $0.50 per million — faster, but not cheaper. It is worth noting where these numbers come from: the 3,000 and 650 figures are Cerebras' own published benchmarks, and of the speed figures above only the 981 tokens per second on Kimi K2.6 was measured by an independent third party, Artificial Analysis. On the NVIDIA side, StorageReview reported that Blackwell Ultra systems delivered the highest throughput across all the models and scenarios NVIDIA submitted to MLPerf Inference v6.0 in April 2026 — a claim about NVIDIA's own submission set, in a round with 14 other submitting partners — improving per-GPU DeepSeek-R1 server throughput 2.77x over the prior round. And the fastest silicon in the world does not help if the model you need is not hosted on it. For most teams, model availability and price per token decide the provider long before tokens per second does.

What inference and fine-tuning actually cost in 2026

Two cost curves are moving in opposite directions, and conflating them is how AI budgets blow up.

Per-token prices are collapsing. An MIT team — Gundlach, Lynch, Mertens and Thompson — posted "The Price of Progress: Price Performance and the Future of AI" to arXiv in November 2025 and revised it in March 2026, finding that the price for any given level of benchmark performance fell roughly 5x to 10x per year, with about 3x of that attributable to algorithmic efficiency alone. Epoch AI's March 2025 analysis found declines from 9x to 900x per year depending on the capability threshold.

And yet frontier-level workloads got more expensive. The same MIT paper found the cost of running at the frontier rose 3x to 18x per year, because reasoning models emit vastly more tokens and models keep getting larger. Cheap tokens plus enormous token counts is not a cheap bill.

Current list prices give you the floor. OpenAI's pricing page in August 2026 shows gpt-5.6-sol at $2.50 per million input tokens and $15.00 output, dropping to $0.10/$0.60 for gpt-5.6-luna. Google Cloud's Vertex AI pricing shows Gemini 3.1 Pro Preview at $2/$12 and Gemini 3.1 Flash-Lite at $0.25/$1.50, with cached input discounted 90%. The spread across a single vendor's lineup is wider than the spread between vendors, so model choice moves your bill more than provider choice does.

Fine-tuning is cheaper than most people assume. Together AI's pricing page, as of August 2026, lists LoRA supervised fine-tuning at $0.48 per million training tokens for models up to 16B parameters, $1.50 for 17–69B, and $2.90 for 70–100B, with a $4 minimum per job. Full fine-tuning runs only slightly higher — $0.54, $1.65 and $3.20 respectively. Dedicated compute, if you would rather rent the machine, was listed at $5.49 per hour for an HGX H100 and $8.99 per hour for an HGX B200.

Two caveats worth stating plainly. Larger and specialized models cost much more per million tokens, and carry higher job minimums — Together AI lists $10.00 for DeepSeek LoRA SFT with a $20 minimum, and $40.00 for GLM-5 with a $60 minimum, against the $4 minimum on the standard bands. And serving a fine-tuned model can carry a premium: Google prices tuned Gemini 3 endpoints at 1.5x the base model rate, though tuned Gemini 2.5 endpoints match base pricing.

Not sure whether your use case needs fine-tuning, retrieval, or just better prompting? Our AI Strategy & Roadmap engagement pressure-tests that decision before you commit budget to the wrong one.

How to fine-tune an LLM: a real example

This walkthrough is unchanged from the original version of this article, because the economics of small-scale fine-tuning have not moved much and the approach still holds up. The whole job cost about $10.

Step 1: Define the objective

The goal is a model that summarizes conversations accurately. Out of the box, given a dialogue where Mark promises to send George a tracking number, the base model produces something technically true but useless: "Mark will send George the tracking number." It drops the context that makes a summary worth reading.

Step 2: Choose a pre-trained model

Flan-T5 — specifically google/flan-t5-base from the Transformers library. It is already trained to handle conversational input, which means the fine-tuning job only has to teach the summarization behavior, not language itself.

Step 3: Set up the environment

Google Colab, with a Colab Pro subscription at about $10 a month for GPU access. You are renting GPU capability rather than buying it. The contrast with foundation model training is stark: that scale of work uses something like 8,100 GPUs for ten days and costs thousands of times more than the $10 being spent here.

Step 4: Collect and prepare the dataset

SAMSum — thousands of dialogue examples, each paired with a human-written summary. Loaded as JSON with conversation IDs, dialogues and reference summaries. The paired structure is what lets the model learn the mapping from conversation to good summary. On a real project this step is the expensive one, and it is why data engineering work usually precedes any fine-tuning work.

Step 5: Fine-tune with LoRA

LoRA generates a small set of new parameters to train while the original weights stay frozen, which works out to retraining about 16% of the model. Kick off trainer.train() and watch the loss values fall. On the consumer-grade GPU in Colab, the run takes about 17 hours.

The output afterwards captures what actually happened in the conversation. Same model, same prompt, small targeted dataset — different behavior. If you want this running in production rather than in a notebook, our walkthrough on how to deploy an LLM covers the serving side.

The fourth option: reinforcement fine-tuning and distillation

When we first wrote this piece, the choice was training, fine-tuning, or inference. In 2026 there is a fourth category that sits between fine-tuning and full training, and it is where the most interesting work is happening.

Reinforcement fine-tuning (RFT)

RFT optimizes a model against a grader that scores outputs, rather than against labeled examples. It is genuinely useful when you can define "good" but cannot write down the correct answer. It also inherits every failure mode of optimizing against a proxy — reward hacking chief among them, which we cover in our piece on AI model misbehavior.

It is also very expensive and does not reliably beat plain supervised fine-tuning. TensorZero's September 2025 head-to-head is the most useful public evidence: on a data extraction task, RFT cost $65.07 against SFT's $0.09 — a 723x difference — and SFT on a larger dataset produced better results anyway. On agentic coding, RFT cost $168.57 against $0.70 and was the one clear win, improving performance where SFT failed. On customer service, RFT cost $201.85 against $1.80 and actively degraded performance.

OpenAI prices RFT on o4-mini-2025-04-16 at $100 per hour of active training time, billed prorated to the second, with grader model tokens charged separately. Worth knowing before you plan around it: OpenAI announced on May 7, 2026 that it is winding down its self-serve fine-tuning platform entirely. Organizations without recent fine-tuning inference lost the ability to create new jobs on July 2, 2026, and all remaining customers lose it on January 6, 2027. Inference on already-fine-tuned models continues until the base models are deprecated. If your roadmap assumed OpenAI fine-tuning would be there, it will not be.

Distillation

Distillation trains a small "student" model to imitate a large "teacher." You get most of the capability at a fraction of the serving cost — which, given that inference is the recurring expense, is where the money actually is.

The strongest recent result is on-policy distillation, where the student generates its own outputs and the teacher grades them token by token. Thinking Machines Lab reported in October 2025 that distilling Qwen3-32B into Qwen3-8B-Base reached 74.4% on AIME'24 — the score the Qwen3 technical report gives for that run, against 67.6% for a reinforcement learning baseline — using roughly 1,800 GPU hours against RL's 17,920, which the post describes as one-tenth the cost of RL. Thinking Machines' own replication, starting from a 400k-example SFT checkpoint, reached 70% in about 150 steps. The 9x to 30x figure often quoted alongside this is a different comparison: on-policy against off-policy distillation, and the multiple depends on whether you already have the SFT dataset and whether you have to pay for the teacher's outputs. In a separate run starting from Qwen3-8B-Base with no additional SFT, on-policy distillation recovered the teacher's AIME score in under 10 gradient steps where RL took 70 — roughly 7–10x faster — which, combined with shorter training context and smaller batch sizes, the authors put at a cumulative 50–100x reduction in compute.

There is theory behind it. An arXiv survey by Song and Zheng of Tencent, first posted in April 2026 and revised through June 2026, explains that off-policy training accumulates error at O(εT²) over sequence length while on-policy methods reduce that to O(εT) — a widening advantage as reasoning chains get longer. It also cites the canonical production example: DeepSeek-R1 distilled a 671B mixture-of-experts teacher into dense students from 1.5B to 70B parameters with long chain-of-thought reasoning largely intact.

The honest caveat: distillation needs a teacher you are permitted to distill from. Check your provider's terms before building a roadmap on it, and expect open-weight teachers to be the safer foundation.

Which one do you actually need?

Most teams asking "should we fine-tune?" have a prompting problem or a retrieval problem. Work the list in order and stop at the first thing that works.

1. Better prompting and structured outputs. Free, instant, solves more than people expect.
2. Retrieval (RAG). Correct whenever the gap is knowledge rather than behavior, and the only option that stays current without retraining.
3. Fine-tuning with LoRA. For consistent formats, tone, or a behavior the base model keeps missing. From $4 per job on Together AI's standard bands; specialized models carry higher minimums.
4. Distillation. When a big model already does the job and you need it cheaper or faster in production.
5. RFT. When success is scoreable but not writable, and you can absorb 100x+ the cost of SFT with no guarantee.
6. Training from scratch. Effectively never. Buy this.

Where teams go wrong

• Fine-tuning to fix a knowledge gap. Retrieval solves this; training does not, and a fine-tuned model still will not know yesterday's data.
• Benchmarking single-stream speed and budgeting for aggregate throughput. Cerebras' 981 tokens per second streaming to one user on Kimi K2.6 and MLPerf's 2.49 million tokens per second across four NVIDIA GB300 NVL72 racks answer completely different questions.
• Hardcoding a vendor speed claim into a strategy document. The figures in this piece are dated for exactly this reason. Re-check anything older than six months.
• Assuming per-token price cuts mean lower bills. The MIT price-performance study's finding that frontier-level running costs rose 3x to 18x per year while per-token prices fell is the trap.
• Building on a provider's fine-tuning API without an exit. OpenAI's May 2026 wind-down notice is the cautionary example.
• Ignoring evaluation. Without a held-out test set you cannot tell whether fine-tuning helped, and in TensorZero's customer service test it made things worse.

Frequently asked questions

What is the difference between training and fine-tuning?

Training builds a model's general capability from scratch on trillions of tokens, at frontier costs Epoch AI tracks up to around $500 million. Fine-tuning takes that finished model and adapts it to a narrow task with a small curated dataset, often for less than the price of a dinner. Training teaches language; fine-tuning teaches your job.

Is inference more expensive than training?

For almost every company, yes — in total. Any single training run costs far more than any single inference call, but you pay for inference on every request forever, which is why Gartner forecast in August 2026 that inference would take 55% of the $42 billion AI-optimized IaaS market against training's 45%.

How many tokens per second is fast in 2026?

For a frontier-scale model, anything near 1,000 tokens per second to a single user is at the leading edge — Artificial Analysis measured Cerebras at 981 tokens per second on the trillion-parameter Kimi K2.6 in May 2026. Mainstream hosted reasoning models are much slower; Forbes reported Claude Opus 4.6 at roughly 65 tokens per second in February 2026.

How much does it cost to fine-tune a model?

Less than most people expect for open-weight models. Together AI listed LoRA supervised fine-tuning at $0.48 per million training tokens for models up to 16B parameters in August 2026, with a $4 job minimum, and the walkthrough in this article cost about $10 of Colab Pro. Reinforcement fine-tuning is a different category: OpenAI charges $100 per hour of active training.

Should I fine-tune or use RAG?

Use RAG when the model lacks information and fine-tuning when it lacks behavior. If your problem is that the model does not know your current pricing, documents or policies, retrieval is cheaper, faster to update, and easier to audit. Fine-tune when you need a consistent format or tone that prompting cannot hold.

Deciding between fine-tuning, retrieval and distillation is an architecture question with a large bill attached. HatchWorks' AI-powered software development teams and forward deployed engineers build and ship the version that fits your constraints — and our AI training for teams gets your engineers fluent in the tradeoffs.

Sources

• Cerebras, "OpenAI GPT-OSS 120B Benchmarked – NVIDIA Blackwell vs. Cerebras", November 2025, https://www.cerebras.ai/blog/blackwell-vs-cerebras
• Cerebras, "Cerebras Brings Trillion Parameter Inference to Enterprises with Kimi K2.6", May 2026, https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise
• Forbes, "Cerebras Pockets $1 Billion To Challenge Nvidia In AI With 20x Faster Chip", February 2026, https://www.forbes.com/sites/johnkoetsier/2026/02/05/cerebras-pockets-1-billion-to-challenge-nvidia-in-ai-with-20x-faster-chip/
• StorageReview, "NVIDIA Sets MLPerf Inference v6.0 Records with Blackwell Ultra Platform", April 2026, https://www.storagereview.com/news/nvidia-sets-mlperf-inference-v6-0-records-with-blackwell-ultra-platform
• Gartner, "Gartner Forecasts Worldwide AI-Optimized IaaS Spending to Grow 96% in 2026", August 2026, https://www.gartner.com/en/newsroom/press-releases/2026-08-10-gartner-forecasts-worldwide-artificial-intelligence-optimized-iaas-spending-to-grow-96-percent-in-2026
• Computerworld, "CES 2026: AI compute sees a shift from training to inference", January 2026, https://www.computerworld.com/article/4114579/ces-2026-ai-compute-sees-a-shift-from-training-to-inference.html
• Gundlach, Lynch, Mertens and Thompson (MIT), "The Price of Progress: Price Performance and the Future of AI", arXiv preprint, November 2025, revised March 2026, https://arxiv.org/abs/2511.23455
• Epoch AI, "LLM inference prices have fallen rapidly but unequally across tasks", March 2025, https://epoch.ai/data-insights/llm-inference-price-trends
• Epoch AI, "Training compute costs are doubling every eight months for the largest AI models", updated November 2025, https://epoch.ai/data-insights/cost-trend-large-scale
• IEEE Spectrum, "Stanford's AI Index for 2026 Shows the State of AI", April 2026, https://spectrum.ieee.org/state-of-ai-index-2026
• Together AI, "Pricing", accessed August 2026, https://www.together.ai/pricing
• OpenAI, "Pricing", accessed August 2026, https://developers.openai.com/api/docs/pricing
• OpenAI, "Deprecations", announcement dated May 2026, accessed August 2026, https://developers.openai.com/api/docs/deprecations
• OpenAI, "Billing guide for the Reinforcement Fine Tuning API", accessed August 2026, https://help.openai.com/en/articles/11323177-billing-guide-for-the-reinforcement-fine-tuning-api
• Google Cloud, "Vertex AI generative AI pricing", accessed August 2026, https://cloud.google.com/vertex-ai/generative-ai/pricing
• TensorZero, "Is OpenAI's Reinforcement Fine-Tuning (RFT) Worth It?", September 2025, https://www.tensorzero.com/blog/is-openai-reinforcement-fine-tuning-rft-worth-it/
• Thinking Machines Lab, "On-Policy Distillation", October 2025, https://thinkingmachines.ai/blog/on-policy-distillation/
• Song and Zheng (Tencent), "A Survey of On-Policy Distillation for Large Language Models", arXiv preprint, April 2026, revised June 2026, https://arxiv.org/abs/2604.00626