Test-time compute, also called inference-time compute, is the extra computation a model spends reasoning through a problem after it receives a query, instead of relying only on what got baked in during training. Rather than answering instantly, the model generates intermediate reasoning steps, tries multiple approaches, and checks its own work, spending more compute per query to raise accuracy on hard problems.

For most of the history of large language models, getting a smarter answer meant training a bigger model. Test-time compute broke that assumption. A model can now get measurably smarter on a specific question just by being given more time and more computation to work through it, without changing a single one of its trained parameters.

From Bigger Models to Longer Thinking

For years, AI progress followed a simple pattern: add more parameters, more training data, more training compute, and the model gets better. This is train-time scaling, and it defined the field from the first GPT models through the mid-2020s frontier releases. Every gain came from a bigger, more expensive training run, and the resulting model then answered every query in roughly the same amount of time, no matter how hard the question was.

OpenAI's o1, released in September 2024, broke that pattern. Instead of scaling the model, o1 scaled how long the model was allowed to "think" before responding, generating an internal chain of reasoning that the model works through before it commits to a final answer. The results made clear that thinking time was its own lever, separate from model size.

According to the Stanford AI Index 2025 report, o1 scored 74.4% on an International Mathematical Olympiad qualifying exam, compared to GPT-4o's 9.3%, on the same questions. The tradeoff was real too: the report notes that o1 is nearly six times more expensive and 30 times slower than GPT-4o. That tradeoff, more accuracy for more cost and more time, is the entire premise of test-time compute.

How "Thinking Longer" Actually Improves Answers

A standard large language model generates its response token by token, in a single forward pass, with no mechanism to stop, reconsider, or check its own logic before the answer is complete. That's why it can state a wrong answer with the same fluent confidence as a right one.

Reasoning models built for test-time compute add three things a standard model doesn't do by default:

Extended reasoning traces. The model generates a long internal sequence of intermediate steps, working through sub-problems, checking constraints, and narrating its own logic before producing a final answer. This is the mechanism behind chain-of-thought reasoning, scaled up and often run outside what the user sees.

Sampling and selection. Instead of generating one answer, the model can generate several candidate solutions and pick the best one, either through majority vote (the answer most candidates agree on) or a learned scoring function that ranks candidates by likely correctness.

Self-verification. The model checks its own reasoning against the problem's constraints and can backtrack to try a different approach when a step doesn't hold up, closer to how a person double-checks their work before submitting it.

OpenAI's own results on the 2024 AIME math exam show how much each of these levers is worth on its own. According to OpenAI's release notes for o1, the model averaged 74% accuracy with a single sample per problem, climbed to 83% when using consensus across 64 samples, and reached 93% when re-ranking 1,000 samples with a learned scoring function, all using the exact same underlying model. GPT-4o, without test-time reasoning, solved only 12% of the same problems. Nothing about the model's parameters changed between those three scores. Only the amount of test-time compute did.

The New Scaling Law

Before 2024, "scaling law" meant one thing in AI: performance as a function of training compute, model parameters, and training data. Test-time compute added a second axis. A 2024 paper from researchers at UC Berkeley and Google DeepMind, Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters, formalized this: allocating test-time compute in a "compute-optimal" way, matching the reasoning effort to the difficulty of the specific prompt rather than spending a flat amount on every query, improved efficiency by more than 4x compared to a naive best-of-N sampling baseline. In a FLOPs-matched comparison, the paper found that test-time compute let a smaller model outperform a model 14 times larger.

That finding reframes the build-versus-buy question for frontier labs and, downstream, for the businesses buying their models. A lab facing a fixed compute budget now has two places to spend it: a bigger, more expensive training run that improves every future response by a small margin, or a reasoning system that lets an existing model spend more compute on the specific queries that need it.

The tradeoff doesn't disappear at scale, it just gets more visible. OpenAI's o3 model, evaluated on the ARC-AGI-Pub benchmark, shows the curve directly: a high-efficiency configuration scored 75.7% on the semi-private evaluation at a compute cost of $26 per task, while a low-efficiency configuration, using roughly 172 times more compute, scored 87.5% at $4,560 per task. Twelve more points of accuracy cost over 170 times the compute. That's the shape of test-time scaling in practice: real gains, at a steeply rising price for each additional point.

Test-Time Compute vs. Train-Time Compute

Dimension	Train-Time Compute	Test-Time Compute
When it happens	Once, during model development, before deployment	Every time the model answers a query, in production
What it changes	The model's parameters (weights)	Nothing about the model, only how much reasoning effort is spent on one query
Cost shape	Large, front-loaded, project-style spend	Per-query, usage-based, scales with volume and with how "hard" a question is treated
What it buys	A generally more capable model, for every future query	A better answer to this specific query, at higher latency and cost for that query alone
Primary levers	More parameters, more training data, longer training runs	Longer reasoning chains, more sampled candidates, self-verification passes
Failure mode if overspent	An expensive model that's still mediocre if the recipe was wrong	A slow, costly answer to a question that didn't need deep reasoning

Both axes are still scaling laws in the technical sense: performance improves as a roughly predictable function of compute spent. The difference is when you pay and what you get for it. Train-time compute is a one-time investment in the model itself. Test-time compute is a per-query decision that a business, or the application calling the model, has to make every single time.

Cost and Latency Tradeoffs

Test-time compute is not free, and it shows up in two places at once: money and time.

Latency. A standard model responds in a fraction of a second to a few seconds. A reasoning model working through an extended chain of thought can take anywhere from several seconds to multiple minutes, depending on how much internal reasoning it generates before producing a visible answer. For interactive use cases, that difference is the whole ballgame: a support agent waiting 20 seconds instead of 2 for an answer will find a workaround, or stop using the tool. See latency for how response time shapes whether a feature actually gets adopted.

Cost. Reasoning tokens aren't free either, and most of them are invisible. According to Artificial Analysis, OpenAI's o3 is priced at $2.00 per million input tokens and $8.00 per million output tokens, and internal reasoning tokens, the ones generated during the model's "thinking" process, are billed as output tokens even though the user never sees them in the response. A short visible answer can sit on top of a much larger, billed reasoning trace. That's a structurally different cost profile from a standard model, where the token count you see is close to the token count you pay for.

The practical consequence is that test-time compute turns "how hard should the model think about this" into a cost-control decision, not just a quality one. Inference optimization techniques, like capping reasoning length, routing easy queries to a fast model and only escalating hard ones to a reasoning model, and caching repeated queries, all become more valuable once a meaningful share of queries carry a variable, sometimes large, reasoning-token bill.

Business Implications: When Reasoning Model Pricing Makes Sense

Not every AI feature needs test-time compute, and treating every query as if it does is the fastest way to turn a promising pilot into an expensive one. The decision comes down to a fairly simple question: how costly is a wrong answer, compared to how costly is a slow or expensive one?

Use a standard, fast model when: the task is high-volume, the answer is either obviously right or easy to check, and speed matters to the person waiting on it. Customer FAQ responses, basic classification, first-draft content generation, and routine data extraction all fit this pattern. Running every one of these through a reasoning model burns budget on accuracy the task didn't need.

Reserve test-time compute for: multi-step analysis where a mistake is expensive to catch after the fact, tasks with a large solution space where the model benefits from exploring alternatives, and any workflow where a person will act on the output without independently re-deriving it, like financial modeling, contract analysis, or technical debugging. These are the cases where the Stanford AI Index's "six times more expensive, 30 times slower" tradeoff is worth paying.

The budgeting implication for a leader sponsoring AI work: per-query cost for a reasoning-model feature is not a fixed number the way it often was with standard models. It varies with how hard each individual query turns out to be, since the model itself decides how much it needs to reason. That makes usage monitoring and routing logic, sending only the queries that need deep reasoning to the expensive model, part of the cost-control plan from day one, not an afterthought once the bill arrives. AI agents that chain multiple model calls together compound this further, since a multi-step agent workflow can invoke test-time reasoning several times in a single task.

Key Facts

OpenAI's o1 scored 74.4% on an International Mathematical Olympiad qualifying exam, compared to GPT-4o's 9.3%, but o1 is nearly six times more expensive and 30 times slower than GPT-4o, per the Stanford AI Index 2025 report.
On the 2024 AIME math exams, o1 averaged 74% accuracy with a single sample per problem, 83% with consensus across 64 samples, and 93% when re-ranking 1,000 samples, compared to 12% for GPT-4o on the same problems, according to OpenAI's official o1 release notes.
A compute-optimal test-time scaling strategy can improve efficiency by more than 4x compared to a naive best-of-N baseline, and in FLOPs-matched comparisons, test-time compute let a smaller model outperform one 14 times larger, per the Berkeley and DeepMind paper Scaling LLM Test-Time Compute Optimally.
On the ARC-AGI-Pub benchmark, a high-efficiency configuration of OpenAI's o3 scored 75.7% at $26 per task, while a low-efficiency configuration scored 87.5% at $4,560 per task, using roughly 172 times more compute for a 12-point accuracy gain, per ARC Prize's official results.
OpenAI's o3 is priced at $2.00 per million input tokens and $8.00 per million output tokens, and internal reasoning tokens are billed as output tokens even though they never appear in the visible response, according to Artificial Analysis.

Reasoning Models - The model category built specifically to use test-time compute
Inference - The broader production phase that test-time compute is a variable-cost form of
Chain-of-Thought - The step-by-step reasoning technique test-time compute scales up
Large Language Models - The base model category that reasoning models extend
Inference Optimization - Techniques for controlling cost and latency at inference
Latency - Why the extra thinking time in test-time compute affects adoption
AI Agents - Multi-step systems where test-time reasoning cost can compound
Foundation Models - The base models that reasoning capability gets built on top of

External Resources

Stanford AI Index 2025, Technical Performance - Independent benchmark data on o1's accuracy, cost, and latency versus GPT-4o
OpenAI: Learning to Reason with LLMs - OpenAI's own release notes and benchmark data for o1
Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters - The Berkeley/DeepMind paper formalizing test-time compute as a scaling law
ARC Prize: o3 ARC-AGI-Pub Results - Official cost-per-task data across compute configurations

Frequently Asked Questions about Test-Time Compute

What is test-time compute?

Test-time compute, also called inference-time compute, is the extra computation a model spends reasoning through a problem when it receives a query, rather than relying only on what was learned during training. It includes generating extended reasoning chains, sampling multiple candidate answers, and self-checking before responding.

How is test-time compute different from training compute?

Training compute is spent once, before deployment, to set the model's parameters. Test-time compute is spent every time the model answers a query, and it doesn't change the model at all, it changes how much reasoning effort goes into that one specific answer.

Why did OpenAI's o1 make test-time compute a big deal?

o1, released in September 2024, showed that giving a model more time to reason at inference could produce large accuracy gains on hard problems without any change to model size or training. Per the Stanford AI Index 2025 report, it scored 74.4% on an International Mathematical Olympiad qualifying exam versus GPT-4o's 9.3%, at roughly six times the cost and 30 times the latency.

Does more test-time compute always mean a better answer?

It improves accuracy on problems that benefit from multi-step reasoning, but the gains shrink as compute increases. OpenAI's o3 results on ARC-AGI-Pub show a 12-point accuracy gain for roughly 172 times more compute between the efficient and high-compute configurations. On simple, well-defined tasks, extra reasoning often adds cost and latency without a meaningful accuracy improvement.

How does test-time compute affect API costs?

Reasoning models bill internal "thinking" tokens as output tokens, even though those tokens never appear in the visible response. That means the same visible answer can carry a much larger token bill than it would from a standard model, and the total cost varies by query since the model decides how much it needs to reason.

Which tasks benefit most from test-time compute?

Multi-step analysis, tasks with a large solution space, and workflows where a wrong answer is expensive to catch later, like financial modeling, contract analysis, or technical debugging, benefit most. High-volume, low-stakes tasks like FAQ responses or basic classification usually don't need it.

Is test-time compute the same as chain-of-thought prompting?

They're related but not identical. Chain-of-thought is the technique of generating step-by-step reasoning before an answer. Test-time compute is the broader category that includes chain-of-thought plus other techniques like sampling multiple candidates and self-verification, all applied at inference time.

Will test-time scaling replace training bigger models?

No, they're complementary, not a replacement for one another. Research like the Scaling LLM Test-Time Compute Optimally paper shows test-time compute can let a smaller model match or beat a much larger one on specific tasks, but frontier labs continue to invest in both bigger training runs and better test-time reasoning, since each buys something different.

Part of the AI Terms Collection. Last updated: 2026-07-16

About the author

Victor Hoang

Co-Founder, Rework.com

Victor Hoang is Co-Founder and CMO of Rework. He spent 12+ years scaling B2B SaaS growth, building a lead engine that generated over 1 million leads and $10M+ in annual recurring revenue. Today he builds AI agents and MCP servers into Rework's products to empower customers across growth and operations. He writes about what actually works.

View full profile LinkedIn