Savings as Code for AI: right-modeling
Savings as Code for AI. For each task, the smallest model that meets the quality bar, shipped as a change you approve.
In 30 seconds
Right-modeling is rightsizing for AI, also called model right-sizing.
- For each task, use the smallest model that meets the quality bar.
- Measure quality and cost per request.
- Check again when models or prices change.
Why it matters
Teams pick the strongest model during a prototype, and every task inherits it. Many tasks do just as well on a smaller model. The gap is large: per input token, the most expensive model on Anthropic's price list costs 100 times the cheapest, and the same gap holds on OpenAI's (Anthropic, OpenAI, checked November 7, 2026).
Model right-sizing, step by step
- List the routes: every place your application calls a model.
- Write the quality bar: real examples of a good answer, per route.
- Try smaller models against the bar and keep the cheapest that passes.
- Ship it as a change you approve, with the expected saving. This is Savings as Code for AI.
- Re-check when new models or prices arrive.
The price per token is not the whole cost. A larger model is usually slower as well, and every request on that route waits for it.
What a quality bar looks like
A quality bar is a set of real requests with the answer you would accept, written by the people who own the feature. Start with a few dozen per route, and keep the hard cases you have seen fail. Every model gets the same requests and the same prompt, and you count how many answers pass.
| Route | Quality bar | Model kept | Cost per 1,000 requests |
|---|---|---|---|
| Support ticket summary | 48 of 50 summaries correct and complete | Small | $0.40 |
| Contract clause check | 50 of 50 clauses found | Large | $9.00 |
| Product description draft | 45 of 50 accepted by the editor | Medium | $1.60 |
Illustrative example.
Common mistakes
- Changing the prompt and the model at once. You no longer know which one moved the quality. Change one thing per pull request.
- Judging on a handful of easy examples. A model that passes five easy requests can fail the sixth, hard one. Keep the hard cases in the bar.
- Looking only at the price per token. A cheaper model that needs a longer prompt or a second try can cost more per request. Compare the cost per request, and check the response time too.
- Setting it once. New models and new prices arrive often. Run the bar again when they do.
The other AI cost levers
- Prompt caching: a cached prompt read costs 10% of the normal input price on most Claude models (Anthropic).
- Batch processing: half the standard price for work that can wait, on Anthropic, OpenAI and the Gemini API (OpenAI, Google).
- Output limits and context trimming: fewer tokens per request.
All five, with what to measure: Five LLM cost levers. Figures checked November 7, 2026; prices change often.
Questions
Does a smaller model mean lower quality? Not for every task, and the quality bar is what decides, route by route. Tasks that need careful reasoning keep the larger model. Extraction, classification and short summaries can pass on a smaller one.
How often should we check again? When a provider releases a new model or changes its prices, and at least once a quarter for the routes that cost the most. The bar is already written, so a new run is cheap.
Who writes the quality bar? The people who own the feature, because they know what a good answer is. Engineering runs the comparison and proposes the change; the owners approve it.
Keeping it right
The allowed model per route and its token limit belong in the same file as your cloud budgets. That is Spend as Code, coming soon.