AI Model Selection: The Cost of Model FOMO

Why do teams default to the most capable model?

Because nobody chose. The first model that worked became the model, and no one has revisited it since.

That is how a daily summary, a ticket classification and a genuinely hard reasoning problem all end up on the same expensive tier. Not one of those was a decision. The first was a default, the second copied the first, and the third is the only one that needed it.

Wade Foster, co-founder and CEO of Zapier, put the mismatch plainly on our podcast:

“they might be doing pretty mundane tasks where it's like, ‘Oh, you know, build my daily brief,’ but you're sicking [a frontier model] on it. And it's like, whoa, that's just way overkill. You don't need that type of model for the job.”

Wade Foster, Co-founder and CEO, Zapier

We call the reason for that model FOMO: the suspicion that a cheaper tier would probably do, held alongside the worry that you might miss something by finding out. It is a decision made on nerve rather than measurement, and the tell is that the suspicion is usually already there. What is missing is the evidence to act on it. You pay for that gap every month, on every routine task that never needed the top tier.

Every major vendor ships a tiered ladder, so this is a within-ladder decision rather than a vendor decision. The behaviour is universal, and swapping brands does not solve it.

Does a cheaper model always mean worse output?

Not reliably, and the assumption that it does is the thing worth attacking.

Capability is not one number. A frontier model is general, and general is expensive to run and often unnecessary. On a narrow, well-specified task with a clear right answer, a smaller or purpose-built model frequently matches the frontier and sometimes beats it, because the task rewards precision on one thing rather than breadth across everything. Document classification, entity extraction, routing and structured parsing all sit in that category.

So the trade-off model FOMO assumes is not always there. Sometimes you are paying a premium for a worse answer, and the only way to find out is to test the tiers against your own work.

What does model FOMO actually cost?

We print no per-token prices here. They move monthly, and a page that hard-codes this quarter’s rate card is wrong by the next one. Do the arithmetic with your own numbers instead.

(tasks per month times tokens per task times your frontier rate) minus (the same volume at your lower-tier rate) equals your monthly premium.

Then divide the cost of building an evaluation set by that figure. If two days of work pays back inside one billing cycle, the eval is the cheapest line item in the decision, and the argument is over.

Coinbase reached the same place from the other direction. In a public post, CEO Brian Armstrong writes that better defaults, routing, caching, leaner context and visible usage together cut the company’s AI spend nearly in half while token usage kept growing. His routing line is the one aimed straight at model FOMO: humans shouldn't be choosing models, AI can automate this task. We take that story apart properly in our AI model strategy guide.

How do you prove a cheaper model is good enough?

You build an evaluation set you own, and you keep it private.

Private matters for validity rather than secrecy. Public benchmarks leak into training data, and the leakage is not marginal. A May 2026 preprint from Penn State surveying the problem reports contamination reaching up to 45% on commonly used benchmarks, and finds that rebuilding a clean mirror of one popular maths benchmark cut measured accuracy by up to 13% on some models. As the authors put it, a high score “often reflects the model’s capacity to effectively recall data encountered during pretraining, thereby inflating accuracy.”

That paper is a preprint and has not been peer reviewed. We cite it for the direction and the mechanism rather than for the precise percentages, which vary by benchmark and by model.

Which is why a leaderboard cannot answer your question. It is not that the numbers are fabricated. It is that they may be measuring recall of the test rather than capability on your task, and your task was never in the test.

A useful eval has three properties. It is drawn from work that is easy for a person and hard for a model, because that is where a real capability gap shows up rather than a formatting difference. It is small enough to build in days, because an eval that takes a quarter will not be maintained. And it shrinks as well as grows: failures get added, and cases that every tier passes get removed, because an eval both tiers ace decides nothing.

What should you measure before dropping a tier?

Six numbers, and the last one is a veto rather than a measure.

Measure	Why it decides	Failure it prevents
Pass rate on your private eval set	The only number scoped to your actual work	Trusting a 99% public benchmark that never tested your task
Human-easy, model-hard ratio	Confirms the eval probes a real capability gap	An eval both tiers ace, which decides nothing
Escalation rate to the higher tier	Says whether routing saves anything net	A cheap route that pays twice, once low and once high
Cost per successful task	Retries are where token savings die	Winning on unit price, losing on total spend
Variance across reruns	Consistency, not peak quality, is the product	Passing on average, failing unpredictably
Blast radius of a wrong answer	Some tasks are not eligible at any price	Optimising a workflow that should not have been automated

The one people skip is variance. A model that passes four times in five and fails unpredictably on the fifth is worse in production than a model that is slightly less capable and always the same, because the first one costs you a review process and the second one does not.

Where does this sit in how we work?

The evaluation set is the deliverable, not the model recommendation.

When we map workloads for a client, the first artefact is a small set of representative tasks with a defined pass condition, because until that exists every tier conversation is a matter of opinion. It usually takes days rather than weeks, it exposes where a cheaper model already clears the bar, and it keeps working after the next model release, which a one-off comparison does not.

That is the same sequence as the wider portfolio work: evaluate, specialize, orchestrate, optimize. Model selection is the evaluate step, done once properly, so the other three have something to stand on.

The short version

Uber’s four months are not a cautionary tale about AI being expensive. They are a story about not knowing which requests deserved the expensive model, and caps are what you reach for when you cannot answer that.

Model FOMO is the individual version of the same gap. The suspicion that a cheaper tier would do is usually correct and almost never tested, because testing it requires an eval nobody has built yet.

Build the eval. Make it small, private, and drawn from work that is easy for a person and hard for a model. Then let it decide, and re-run it when the next model ships.

Common questions

Related: AI model strategy for the four decisions this one sits inside, and AI Inference, Training, and Fine-Tuning for what to do once a cheaper tier is not enough.