Managing Model Costs as Usage Grows

By Brian Tr · Updated Oct 6, 2026

Model costs behave differently from most software costs. A traditional SaaS product serves its ten-thousandth customer for almost nothing extra. An AI product pays for compute every time a customer asks it to do something, so the bill grows with usage, and usage tends to grow faster than revenue when customers love the product. That's the situation founders and COOs are managing: not a one-time optimization project, but a cost line that needs an owner, a dashboard and a set of levers.

This guide is about those levers at the company level. It's deliberately vendor-neutral. It names no model prices, because they change too often to be useful in a reference article, and it describes classes of technique instead of products. If you want the per-agent engineering view of caching, routing and call budgets, read AI agent cost optimization. If you want the accounting side, how these costs are classified and how they flow into margin, see AI company COGS and AI inference costs and gross margin. Here we stay on the question a growth-stage leadership team actually faces: what can we do about it, in what order, and who owns it?

Start with visibility, not optimization

Almost every cost-cutting effort that fails starts in the wrong place. Teams pick a clever technique before they know where the money goes. The first lever isn't a technique at all. It's measurement.

The invoice from a model provider is a single number for the whole company. That number can't tell you whether a particular customer is profitable, whether a new feature is a margin leak, or whether a spike came from real growth or from one runaway job. To get answers you need cost attributed to the unit your business sells.

Pick the unit that matches how you charge and how customers think:

• Cost per task or outcome: one document processed, one ticket resolved, one report generated.
• Cost per customer per month: total model spend tagged to an account.
• Cost per feature: which parts of the product consume the most compute.
• Cost per active user: useful as a sanity check, but it hides the heavy-versus-light split.

Practically, that means tagging every model call with a customer ID, a feature name and a task type at the point it's made, and storing those tags with token counts and latency. It's unglamorous plumbing, and it's the foundation for everything below. Once it's in place, plot the distribution, not just the average. In many products a small share of customers drives a large share of cost, and the average conceals it. The same discipline applies in any business where cost varies by customer, and our explainer on unit economics covers the general framing.

Two views are worth building early. The first is cost per unit against price per unit, by customer segment, so you can see which segments are underwater. The second is cost per unit over time, so you can tell whether product changes or model changes are helping or hurting.

The lever table

Once you can see cost at the unit level, the levers become choices instead of guesses. The table summarizes each one: what it is, the usual trade-off, and when it tends to fit. The sections after it give the detail.

Routing and cascades: small model first

The idea behind a cascade is simple. Most requests are easy, and a cheap model handles them fine. A minority are hard and need a more capable, more expensive one. If you can tell which is which, or find out cheaply, you pay the high price only when it earns its keep.

There are two common shapes. In a cascade, you send the request to the cheaper model first, check whether the answer is good enough, and escalate to a stronger model only if it isn't. In routing, a classifier decides up front which model should handle the request, without trying the cheap one first.

The research here is worth knowing, and it's worth reading the numbers carefully. The FrugalGPT paper proposes cascading across models and reports that the approach can match the performance of the best individual model with up to 98 percent cost reduction, or improve accuracy over that model by 4 percent at the same cost. Those are the authors' results on their benchmarks, with "up to" doing real work in the first claim. Your workload will differ. The RouteLLM paper trains routers on human preference data and reports that its approach reduces costs by over 2 times in certain cases without compromising response quality, and that the routers kept working when the underlying models changed at test time.

Treat both as evidence that the lever is real, not as a savings forecast. The honest trade-offs are these:

• Misrouting costs quality. If the router sends a hard request to the weak model and nothing catches it, a customer gets a bad answer. You need a way to measure answer quality on your own tasks before trusting a router.
• Cascades add latency. A request that fails the cheap attempt pays for the cheap call, the check, and then the expensive call. For interactive products, that delay can matter.
• The checker has a cost too. Deciding whether an answer is good enough isn't free. Some teams use simple rules, some use a small model, some use confidence signals. Each has failure modes.
• It needs an evaluation set. Without a set of representative tasks with known good answers, you can't tell whether a routing change helped or quietly hurt.

At company level, the practical move is to start where the cost is concentrated. Use your unit-level data to find the one or two task types that account for most spend, and test routing there first.

Prompt and context trimming

Cost scales with the amount of text going in and out, so the cheapest tokens are the ones you never send. This lever tends to be underrated because it feels mundane, and it often pays quickly.

Common places to look:

• Bloated instructions. System prompts accumulate rules over months. Many can be shortened or removed without changing behavior.
• Over-retrieval. Pulling ten document chunks into context when three would do raises cost on every call.
• Conversation history. Re-sending a long history on each turn can dominate cost. Summarizing older turns trades a little fidelity for a lot of savings.
• Verbose outputs. Asking for structured, concise responses where the product allows it reduces output spend.

The trade-off is that cutting context can cut quality in ways that are hard to see. Treat every trimming change as an experiment against your evaluation set, not a clean-up chore. A prompt that's 30 percent shorter and 5 percent worse is only a win if your customers can't tell and your margin needed the help.

Caching: don't pay twice for the same work

Caching comes in two broad kinds.

Prompt (prefix) caching lets a provider or serving layer reuse the computation for a long, stable beginning of a prompt, such as instructions and reference documents, so repeated calls with the same prefix cost less. It rewards prompts that are structured with the stable part first and the changing part last. Terms and discounts vary by provider and change over time, so check current documentation instead of trusting a number in an article.

Response (semantic) caching stores answers and serves them again when a new request is identical or close enough. It can eliminate the model call entirely. The risk is serving a stale or subtly wrong answer to a question that only looked similar, which is a real problem for anything personalized, time-sensitive or high-stakes. It suits narrow, repetitive questions such as common support queries or standard classifications, and suits open-ended generation poorly.

Caching fits best where your usage data shows heavy repetition. If your tagged logs show the same large prefix going out thousands of times a day, caching deserves attention before anything fancier does.

Batching: trade speed for price

Not every request needs an instant answer. Nightly enrichment, report generation, bulk classification and back-office clean-up can run asynchronously in batches. Many providers price deferred, batch-style processing below real-time processing, and batching also smooths load on your own infrastructure. The trade-off is that results arrive later. The decision is a product one: which jobs would customers genuinely not notice running overnight? List them, and move what you can.

Distillation and fine-tuning: a smaller model for a narrow job

Distillation trains a smaller model to imitate a larger one on a specific task. Fine-tuning adapts an existing model to your data. Both aim at the same outcome: a cheaper model that's good enough at the one thing you need.

The research supports the idea within limits. The "Distilling Step-by-Step" paper reports that a 770M-parameter T5 model outperformed a few-shot prompted 540B-parameter PaLM model on a benchmark while using only 80 percent of the available training data, by using the larger model's rationales as additional training signal. That's a result on one benchmark, but it illustrates the principle: for a defined task, a small specialized model can compete with a general giant.

The trade-offs for a company:

• Upfront and ongoing effort. You need training data, an evaluation set, engineering time and a plan to retrain when the task or the data drifts.
• Narrower capability. The small model does one job well and may fail on inputs outside it. You need a fallback path.
• A moving target. General models keep improving and getting cheaper, so a distilled model's advantage can shrink. Re-check the comparison periodically.
• Terms of use. Some providers restrict using their model outputs to train other models. Check the terms before you build a pipeline on that assumption.

It fits best when a task is stable, high in volume, well defined and easy to evaluate. It fits badly on tasks that change weekly or that need broad reasoning.

A related technique worth knowing about is speculative decoding, where a small model drafts tokens that a large model verifies. The original paper reports a 2X to 3X acceleration over the standard implementation with identical outputs. It's mostly something your serving provider or infrastructure team implements, rather than a product decision, but it explains why self-hosted serving costs keep falling.

Open-weight and self-hosting versus API

At some volume, founders ask whether to stop renting models through an API and run open-weight models on their own infrastructure. It's a legitimate question, and the answer is rarely a clean yes or no.

What self-hosting can offer: more predictable cost at high, steady volume, control over data location and handling, freedom to customize, and insulation from a provider's price or policy changes.

What it costs: you take on capacity planning, GPU procurement or rental commitments, serving engineering, monitoring, security and upgrades. Idle capacity is pure waste, and demand spikes can overwhelm fixed capacity. You also own the quality gap, if any, between the open model and the frontier one.

A useful way to frame it: an API converts a fixed-capacity problem into a variable cost, and self-hosting converts it back. If your demand is spiky or still unknown, the variable cost is a feature. If your demand is steady and large, and the task fits an open model, the fixed cost may come out ahead, but only after you count the engineers. Many companies end up hybrid: self-host the high-volume, well-understood workloads, and use APIs for the long tail and for the hardest requests. How much of the stack you own is also a strategic choice, not just a cost one, and application layer vs model layer covers that side.

Rate limits and usage caps

Technical limits protect you from the two worst cost surprises: abuse and accident. Free tiers attract people who will push the product as hard as it allows. A bug in a customer's integration can loop and burn thousands of calls in an hour.

Sensible controls include per-user and per-account rate limits, daily or monthly spend ceilings that trigger a review before they trigger more spending, and hard caps on free and trial accounts. Design them with the customer in mind. A limit that surprises a paying customer is a support ticket and a churn risk. Warn at 70 or 80 percent, explain what happens at the limit, and give admins a way to raise it.

Pricing guardrails: let price carry some of the cost

Engineering levers reduce your cost. Pricing levers change who bears it. Both belong in the same conversation, because a margin problem can often be fixed on the revenue side faster than on the cost side.

The usual guardrails are:

• Fair-use policies on plans that say "unlimited," defining what unlimited means in practice.
• Included allowances with overage, so heavy usage generates revenue instead of just cost. Our guide to hybrid pricing for AI products walks through sizing the allowance and the overage rate.
• Credits, which turn messy variable cost into a single currency customers can budget. See credit-based pricing.
• Meters tied to value, so revenue moves with what customers actually get from the product rather than with raw activity.

The trade-off is sales friction and billing complexity. Customers like predictable bills, and every extra rule is something to explain. A good test is whether your most expensive customer is also a profitable one at current prices. If the unit-level data says no, that's a pricing problem before it's an engineering problem.

Governance: who owns the number

Levers don't pull themselves. Companies that control model spend usually have the same few things in place.

A named owner. Model cost sits across engineering, product, finance and sometimes customer success. When it belongs to everyone, it belongs to no one. Pick one accountable person, often someone in engineering leadership with a finance partner, and give them the dashboard.

Budgets by product area. Allocate a model-spend budget to each feature or team, expressed in a unit that makes sense, such as cost per task or share of revenue. Budgets change behavior because people see the consequences of their design choices.

Alerts, not just reports. A monthly review finds a runaway cost thirty days late. Set alerts on daily spend, on cost per task, and on unusual customer-level consumption.

A review before launch. Any new AI feature gets a cost estimate before release: expected calls per task, expected tokens, expected cost per unit, and expected price. It's a short checklist, and it prevents the "we shipped it and the margin disappeared" story.

A regular cadence. Revisit routing rules, caching and model choices quarterly. The options keep changing, and last year's best configuration is rarely this year's.

A sensible order of operations

If you're starting from scratch, a rough sequence keeps effort proportional to payoff:

1. Instrument cost per task, customer and feature.
2. Find where spend concentrates and which customers are underwater.
3. Trim prompts and context, and add caching where repetition is obvious.
4. Move non-urgent work to batch processing.
5. Add routing or cascades for the highest-spend task types, backed by an evaluation set.
6. Set caps, alerts and fair-use terms, and fix pricing where the data shows a gap.
7. Consider distillation or self-hosting only for stable, high-volume workloads, and only after the cheaper levers are exhausted.

That order puts the cheap, reversible moves first and the heavy, hard-to-reverse ones last.

Key Facts

• Measure cost at the unit you sell (task, customer, feature) before optimizing. A single provider invoice can't tell you who is unprofitable.
• Cascades try a cheaper model first and escalate. The FrugalGPT authors report matching the best individual model with up to 98 percent cost reduction, or improving accuracy by 4 percent at the same cost.
• Learned routers can also cut cost. The RouteLLM authors report cost reductions of over 2 times in certain cases without compromising response quality.
• Distillation can produce small task-specific models. One study reports a 770M-parameter model outperforming a few-shot prompted 540B-parameter model on a benchmark, using 80 percent of the available training data.
• Speculative decoding speeds up large-model serving. Its authors report a 2X to 3X acceleration with identical outputs.
• Research results come from specific benchmarks. Validate every lever against your own evaluation set before counting on savings.
• Pricing levers (allowances, overage, credits, fair use) move cost to the party creating it, and are often faster to deploy than engineering changes.
• Every cost line needs a named owner, a budget and alerts.

Related Reading

• AI Agent Cost Optimization
• AI Company COGS
• AI Inference Costs and Gross Margin
• Hybrid Pricing Models for AI Products
• Credit-Based Pricing
• Why AI Gross Margins Differ From SaaS