Inference Costs and Gross Margin in AI Companies

By Brian Tr · Updated Oct 6, 2026

Traditional software has a happy quirk: once the product is built, serving one more customer costs almost nothing. That's why software gross margins became the thing investors admired. AI products break the quirk. Every time a customer asks the product to do something, a model runs, and running a model costs real money. That cost is called inference, and it's the single biggest reason an AI company's gross margin can look nothing like a classic software company's.

This article explains the mechanics. What inference is, why it behaves like a variable cost, how a single request's cost turns into a margin figure, why a handful of heavy users can wreck the average, and how your pricing model decides who absorbs the risk. It's written for founders and owner-CEOs of B2B companies with a team. The worked example uses round, hypothetical numbers and doesn't reflect any real model's price.

For how these costs are classified on the income statement, see how AI companies account for COGS. For ways to reduce them, see AI model cost management. This piece sticks to how the costs work and how they reach margin.

Training versus inference

Two different activities consume compute in an AI business, and they behave very differently.

Training is building or adapting a model. It happens before (or periodically alongside) serving customers. It's expensive, lumpy, and largely independent of how many people use your product. If you build your own model, a training run is closer to a development project than to a cost of delivering a sale.

Inference is running a trained model to produce an answer. It happens each time a customer takes an action. If you fine-tune a model or call someone else's, the same split applies: building the capability is one cost, and serving it is another.

The distinction matters because the two scale on different axes. Training cost scales with how ambitious the model is. Inference cost scales with how much your customers use it. A startup that sells an AI product to more customers, or gets existing customers to use it more, doesn't grow its training bill. It grows its inference bill, almost one for one.

That's why inference is the cost that shows up in gross margin. It's incurred in the act of delivering the product, and it rises when delivery rises.

Why inference behaves like a variable cost

In a classic software product, the marginal cost of a request is a rounding error. You pay for servers in chunks, and traffic fills the chunks. In an AI product, each request does a chunk of computation of its own, and the size of that chunk depends on what the request is. Several things set it.

Tokens in and tokens out. Language models read and write text in small units called tokens. A short question with a short answer is cheap. A long document with a long summary is not. Providers and self-hosted setups both tend to treat the reading side (the input) and the writing side (the output) differently, and long outputs generally cost more to produce than the same amount of input costs to read.

Context length. Many products stuff extra material into each request: the conversation so far, retrieved documents, instructions, examples. That context is re-read every time. A chat that has gone twenty turns deep costs more per message than the first message did, because the whole history rides along.

Model size. A larger, more capable model costs more per token than a smaller one. Picking one model for every task is the simplest design and often the most expensive.

Reasoning steps. Some models spend extra tokens "thinking" before they answer. Those thinking tokens are usually part of what you pay for, even if the customer never sees them. A hard question can generate many times more tokens than the visible answer suggests.

Agent loops. An AI agent doesn't answer once. It plans, calls a tool, reads the result, decides what to do next, and repeats. Each pass is another model call, and each call re-reads a growing context. Anthropic's engineering team, describing its own multi-agent research system, reports that agents typically use about 4 times more tokens than chat interactions, and multi-agent systems about 15 times more. That's one company's data on one kind of workload, so treat the ratios as an illustration of how steeply cost can climb when you move from answering to acting, not as a universal multiplier.

Retries and failures. When an output fails a check, gets rejected by a validator, or a tool call errors out, the product often tries again. Every retry costs the same as the original attempt. A product with a 10% retry rate carries about 10% more inference spend than its "happy path" suggests.

Put these together and you see why two actions that look identical in your product's interface, say "run an analysis," can differ in cost by a factor of ten or a hundred. For a deeper look at cost control in agent systems specifically, see AI agent cost optimization.

How per-request cost becomes gross margin

Gross margin is revenue minus the direct cost of delivering the product, divided by revenue. For an AI company, inference is the biggest direct cost in that subtraction, usually joined by hosting, monitoring, and customer support. Our reference on gross margin covers the general formula. Here we only need the shape of the relationship.

If your revenue per customer is fixed and your inference cost per customer varies with usage, then gross margin varies by customer. Margin isn't a property of the product. It's a property of the product multiplied by how each customer uses it, under a given price.

That has a practical consequence. You can't read the blended margin number and know whether the business is healthy. A 70% blended margin could mean most customers sit at 80% and a few sit at 20%. Or it could mean everyone sits near 70%. The two companies have very different risks, and only the first has a heavy-user problem.

A worked example with illustrative numbers

Let's build one. Every number here is hypothetical and round. None of it is a real price.

Suppose you sell an AI assistant at a flat $30 per seat per month. Assume one request costs you 1 cent in inference. You have 100 seats, and usage is uneven, which it almost always is:

Revenue is 100 seats times $30, which is $3,000. Inference is $710. Ignoring every other cost, gross margin is (3,000 minus 710) divided by 3,000, or about 76%.

Looks healthy. Now look inside it.

• The 60 light seats each yield about $29.50 of margin.
• The 9 heavy seats each lose $10, because $40 of inference exceeds the $30 they pay.
• The 1 extreme seat loses $170.
• Those 10 heavy and extreme seats (10% of customers) produce $560 of the $710 cost, which is about 79% of all inference spend.

The blended number hides a cross-subsidy. Light users pay for heavy ones. That's fine as long as the mix stays stable, and fragile the moment it shifts.

What happens when usage per task rises

Now suppose you ship an upgrade: the assistant becomes an agent that takes several steps per request. Say that multiplies the cost of each request by five, so a request costs 5 cents. Customers don't change how many requests they send. Cost is now $3,550 against $3,000 of revenue, so gross margin is about negative 18%.

You made the product better, usage didn't change, and you went from a software-like margin to losing money on every seat. A flat price can't absorb a change in cost per request, because the price doesn't know about requests.

The heavy-user tail

The example shows the pattern called the heavy-user tail. Usage across customers is rarely a bell curve. It tends to be skewed, with most customers using a modest amount and a small group using vastly more. Under flat pricing, the revenue from each customer is capped, but the cost isn't.

Three things make the tail worse in AI than in older software.

1. The best customers are the most expensive ones. The people who love the product use it most. They renew, they expand, and they tell colleagues. But their inference bills grow with their enthusiasm.
2. Agents raise the ceiling. A human can only send so many requests an hour. A scheduled agent can send thousands. Automation removes the natural limit on usage per seat.
3. The tail grows as the product improves. Better outputs mean customers trust the product with larger tasks, longer documents, and more autonomous work. Each of those raises cost per task.

None of this makes flat pricing wrong. It means flat pricing places the cost risk on you, and you should know that's the bargain you've made.

How pricing model choice decides who carries the risk

The cleanest way to think about pricing for an AI product is to ask a single question: when cost per customer goes up, who pays?

Flat seats are simple to sell and easy to forecast for the buyer. They are also the model most exposed to the tail. Our article on per-seat pricing in the AI era goes through why the seat unit weakens when work is done by software rather than people.

Usage-based pricing ties revenue to the same driver that drives cost. If every request earns 3 cents and costs 1 cent, you keep a 67% margin on each request no matter how many arrive. The price you pay in return is buyer unpredictability.

Credits are usage pricing with a wrapper. If the rate card is built from real costs, margin is protected per action. If cost per action changes and the rate card doesn't, you've recreated the flat-price problem inside a credit system. We cover the design choices in credit-based pricing explained.

Hybrids pair a predictable base with an allowance and then meter beyond it. They give the buyer a number to budget around and give you a cap on how far the tail can run. See hybrid pricing for AI products.

Back in the example, the fix isn't mysterious. If you moved to $10 per seat plus 3 cents per request beyond the first 300 included, the extreme seat would pay roughly $600 instead of $30, and its cost of $200 would be covered. The light seats would pay their $10 base and stay inside the allowance. You'd have to test whether the lower sticker price still converts, which is a real tradeoff. But the structure matches revenue to cost where the cost actually arises.

The long-run trend: cheaper per unit, more units per task

Founders often hear that inference is getting cheaper, so margin problems will solve themselves. There's truth in it, and there's a catch.

The truth is well documented. Andreessen Horowitz's analysis of this trend, which it calls LLMflation, says that for an LLM of equivalent performance, the cost is decreasing by 10x every year. Epoch AI, which examined state-of-the-art models on six benchmarks over about three years, finds that the rate of decline varies dramatically depending on the performance milestone, ranging from 9x to 900x per year. It adds that the fastest drops have come in the most recent year, so it's less clear they will persist. Both statements are about the price of reaching a fixed level of capability. They don't say what you'll pay for the best model available next year.

The catch is the second half of the equation. Cost per task equals cost per unit of capability times the amount of capability the task consumes. The first term has been falling quickly. The second tends to rise, for the reasons earlier in this article:

• Customers want the most capable model, not last year's, so they keep buying at the frontier.
• Reasoning steps add tokens to each answer.
• Agent loops turn one request into many calls.
• Longer contexts and larger documents become practical, so people use them.

The Anthropic figures above show the scale of the usage side: a move from chat to agents multiplies tokens per task several times over. A 10x annual fall in price per unit of capability can be consumed by a product that uses far more units per task. The net effect on your margin depends on which force wins for your workload, and that's an empirical question you have to answer from your own data.

The planning stance that follows is conservative. Don't price today's product on the assumption that costs will fall far enough to rescue it. Price it so each customer's margin holds at today's cost, and treat falling unit costs as upside you can share with customers or keep.

What to measure

Mechanics only help if you can see them in your own numbers. Track these:

• Inference cost per active customer, per month. The average is less useful than the distribution. Look at the median, the 90th percentile, and the top ten accounts.
• Gross margin by customer, or by cohort of usage. Find out how many customers sit below your target.
• Cost per completed task, not per request. If retries and agent steps are hidden inside "one task," per-request numbers will understate the cost.
• Share of cost from the top 10% of users. In the example it was 79%. If yours is similarly concentrated, pricing is where to look first.
• Cost per request over time at constant behavior. This separates a falling unit price from a rising usage footprint.

If you can't compute these today, building the logging to do so is worth more than most pricing experiments, because every pricing decision rests on knowing where cost comes from.

Key Facts: Inference Costs and Gross Margin

• Training builds a model and is largely independent of customer volume. Inference runs the model for each customer action and scales with usage, which is why it lands in gross margin.
• Per-request cost depends on tokens in and out, context length, model size, reasoning steps, agent loops, and retries.
• Agents typically use about 4x more tokens than chat interactions, and multi-agent systems about 15x (Anthropic engineering, on its own research system).
• For an LLM of equivalent performance, cost is decreasing by 10x every year (a16z).
• Epoch AI finds price declines ranging from 9x to 900x per year depending on the performance milestone, with the fastest drops in the most recent year.
• Under flat pricing, revenue per customer is capped and inference cost is not, so heavy users can run at negative margin while the blended number looks healthy.
• In the illustrative example in this article, 10% of seats generate about 79% of inference cost, and a 5x increase in cost per request turns a 76% margin into a loss.
• Usage, credit, and hybrid pricing move more of the cost risk to the customer than flat seats do.

The takeaway is short. Inference turns the cost of delivering software from a rounding error into a line that moves with every request, and your pricing model decides whether that movement lands on the customer or on your margin. Know the distribution of usage across your accounts, and price against the cost you have now.

Related Reading

• How AI Companies Account for COGS
• AI Model Cost Management
• AI vs SaaS Gross Margins
• Credit-Based Pricing Explained
• Hybrid Pricing for AI Products
• Gross Margin