When AI Patterns Get Expensive at Scale

The pilot looked affordable. You processed 500 documents, ran the system for 60 days, and spent $400. Finance approved the full rollout. Six months later, you're processing 50,000 documents and the bill is $40,000. Not $4,000. Not $8,000. $40,000, because document complexity increased, you added a second LLM pass for quality checking, and the embeddings index needed a rebuild when you added new document types.

AI cost overruns at scale are almost always predictable in hindsight. The per-inference pricing model, the token-scaling behavior with document size, the storage costs for embeddings: none of this is hidden. It just doesn't get modeled carefully before deployment because pilots run at low volume and cost is invisible at low volume.

This article makes cost surprises predictable in advance, pattern by pattern.

Why AI cost curves differ from software cost curves

Traditional software cost is mostly fixed: license fee, implementation cost, and a relatively flat per-user increment. You pay for seats, not for usage. The cost model is predictable and front-loaded.

AI pattern cost is consumption-based in ways that interact with your data volume, document complexity, and query patterns. McKinsey's analysis of the new economics of enterprise technology in an AI world documents this shift: 79% of IT spend is now operating expenditure rather than capital expenditure, and token-based LLM usage is a key driver of FinOps complexity. Four dynamics that software doesn't have:

Per-inference pricing. Every model call costs tokens. Token cost scales with input length and output length. A 10-page document costs roughly 10x more to process than a 1-page document. At low volume, this is invisible. At high volume, it's your largest line item.

Storage costs for embeddings and indexes. RAG Assistant systems store vector embeddings for every indexed document. Vector storage has per-dimension, per-record costs. A knowledge base with 100,000 documents at 1,536 dimensions per embedding requires significant storage, and re-embedding when you update documents is a compute event, not just a storage update.

Retraining costs that increase with business complexity. Scoring models, anomaly baselines, and recommendation engines need periodic retraining as your data changes. Early retraining cycles are cheap because you have relatively little data. Later retraining cycles are more expensive because you have more data and more complex patterns to learn.

Non-linear cost behavior on complex inputs. A 50-page contract costs roughly 50x more to process per LLM pass than a 1-page contract. A meeting with 8 participants costs more to attribute and summarize than a 2-person call. The per-unit cost at the low end of the complexity distribution looks much better than the average cost at production volume.

Key Facts: AI Cost at Scale

• Agentic AI models require between 5 and 30 times more tokens per task than a standard generative AI chatbot. An autonomous agent reasoning iteratively and calling tools may trigger 10-20 LLM calls per single user task. (Gartner, March 2026)
• Token prices have fallen 280x over two years, but total enterprise AI spend has risen 320% in the same period, driven by the shift to agentic workflows and RAG architectures that inflate context windows 3-5x. (Oplexa Inference Cost Crisis Analysis, 2026)
• 55% of ML models in production require retraining within 90 days, adding retraining costs to the initial deployment budget that most teams never model in their year-one approval. (DataRobot, 2025)

Cost drivers by pattern

RAG Assistant

Primary cost driver: context window size during retrieval and generation.

A simple RAG query retrieves 3-5 document chunks and uses them as context for an answer. If each chunk is 500 tokens, your context window for generation is 1,500-2,500 tokens plus the question. At $0.01/1k tokens for a mid-tier model, that's about $0.02-0.03 per query.

At 10,000 queries/month: $200-300. Manageable.

But at high query volume with complex questions, RAG systems often retrieve more chunks (better accuracy requires more context) and use longer context windows. A complex policy question might retrieve 10 chunks at 1,000 tokens each: $0.10-0.15 per query. At 50,000 queries/month, that's $5,000-7,500/month for query costs alone, before storage.

The index refresh cost is the second surprise. If your knowledge base has 500,000 documents and you update 10% monthly, that's 50,000 re-embeddings per month. At $0.0001 per embedding (text-embedding-3-small pricing), that's $5/month. At text-embedding-3-large: $0.13 per 1k tokens, average document 500 words (~667 tokens) = $0.087 per document. 50,000 re-embeddings = $4,350/month just for index maintenance.

Scoring + Routing

Per-inference cost is low. Scoring models are typically smaller, faster, and cheaper than generative models. The main cost risk is retraining frequency and data infrastructure.

A scoring model that needs quarterly retraining requires: data pull and cleaning, feature engineering compute, model training compute, evaluation, and deployment. For an in-house model, this is engineering time. For a vendor-managed model, it's typically a service fee. The cost is bounded and predictable, but teams often don't budget for it in year 2 because it wasn't part of the initial deployment cost.

Vision Extract

Per-page processing cost scales exactly linearly with document volume. This is predictable. The cost model is honest. But "we'll process 200 documents a month" in the pilot often becomes "we need to backfill 2 years of historical invoices" (a one-time processing spike) plus "all new invoices plus all historical documents we're now re-processing for improved accuracy."

High-resolution image processing costs more than low-resolution. If your vendor charges based on compute time per image and you upgrade your scanning equipment, your cost per document increases even at the same document volume.

Meeting Intelligence

Two cost drivers that both scale with usage volume:

Transcription cost. Speech-to-text APIs typically price per minute of audio. Whisper-class transcription runs $0.006-0.024/minute depending on service tier. A 60-minute sales call: $0.36-$1.44. At 500 calls/month: $180-$720 just for transcription. At 5,000 calls/month (enterprise scale): $1,800-$7,200/month.

LLM summarization cost. Long calls produce long transcripts. A 60-minute call transcript is roughly 8,000-12,000 words (6,000-9,000 tokens). Processing that for summary, action items, and CRM field extraction at $0.01/1k tokens input + $0.03/1k tokens output: approximately $0.12-0.18 per call. At 5,000 calls/month: $600-$900/month.

The cost surprise happens when teams deploy Meeting Intelligence for all meetings, not just customer-facing ones. Internal standups, planning meetings, and all-hands calls don't produce useful CRM data, but they still accrue transcription and processing costs. A simple scoping policy (Meeting Intelligence for external calls only) often cuts cost by 60-70% without reducing value.

Anomaly Agent

Stream ingestion cost at high data volume is the primary risk. If your Anomaly Agent monitors transaction streams at 1 million events/day, storage and processing costs are significant before you add any LLM calls.

For purely statistical anomaly detection (no LLM), costs are manageable and scale predictably. The cost risk enters when the Anomaly Agent uses LLM calls for context enrichment ("explain why this transaction is anomalous in natural language") or for complex multi-signal correlation. At high alert volumes, those LLM calls add up.

Generative Research

LLM tokens for synthesis scale with source material length. A research brief that pulls 20 source documents, each 3,000 words, presents roughly 60,000 words of context before the model generates anything. At gpt-4 pricing, that's $1.80-$2.40 in input tokens alone per research task. Output generation adds another $0.30-0.60. Per research task: $2-3.

This sounds low. But if your research operations team generates 100 briefs/month, that's $200-300/month just in API costs, before the infrastructure costs of managing the research pipeline. Scale to 1,000 briefs/month: $2,000-3,000/month. For a large consulting operation doing 5,000+ research tasks/month, the LLM costs alone approach $15,000-20,000/month.

The cost control lever: scope limitation. Research that synthesizes 5 targeted documents costs 75% less than research that reads everything it can find. Research prompts with explicit source limits ("use the top 10 most relevant sources") produce comparable quality to unlimited sourcing at a fraction of the cost.

Document Review

Contract length is the primary cost driver. Reviewing a 5-page NDA costs much less than reviewing a 150-page enterprise software agreement with 40 exhibits. If your document mix shifts from short contracts (early-stage startups) to complex enterprise agreements (growth stage), your per-document cost increases substantially without any change in volume.

The second risk: multiple review passes. Quality-conscious teams often run an initial extraction pass, then a clause comparison pass, then a summary generation pass. Each pass multiplies the base document cost. A 3-pass review pipeline costs 3x what a single-pass pipeline costs. Define your required passes upfront and budget for them.

Workflow Copilot

Context window management is the key cost lever. A Workflow Copilot that pulls the full CRM record history, the last 10 email threads, the relevant account documents, and the current task context into every suggestion call is expensive. Each suggestion call might use 8,000-15,000 tokens of context even for a simple email draft.

At 20 suggestion requests/user/day × 50 users = 1,000 calls/day. At $0.15/call (average across context + output): $150/day, $4,500/month. At 200 users: $18,000/month.

Context compression (summarizing historical context rather than including raw records), query routing (simpler requests go to cheaper models), and suggestion caching (similar requests reuse previous responses) can reduce this cost by 50-70% without meaningful quality loss.

Personalization Engine

The cost risk here is real-time inference at scale. Serving personalized recommendations requires a model call (or vector similarity search) for every user interaction. At 100,000 daily active users making 10 personalization-relevant decisions each: 1 million inference calls per day.

If each call uses a small dedicated model at $0.001/call: $1,000/day, $30,000/month. If you upgrade to a higher-quality LLM for better recommendations: costs multiply 10-20x. The engineering decision between model quality and inference cost is the most important cost-architecture decision for this pattern.

Caching reduces cost substantially: if 40% of users have similar enough profiles that you can serve cached recommendations, you eliminate 40% of inference calls.

Autonomous Agent: the highest cost risk

This is the pattern most likely to produce unexpected budget events. Name it clearly: an Autonomous Agent without hard iteration limits and per-task budget caps is a liability, not a tool.

Here's what happens when it goes wrong:

A production customer support Autonomous Agent is given a task: "Resolve ticket #48291: customer says they were double-charged." The agent begins its loop. It reads the ticket (1 call). It pulls payment history (1 call). It finds an ambiguity and looks up related tickets (2 calls). It drafts a response (1 call). It determines it needs manager approval and looks up the escalation policy (1 call). It finds the policy unclear and reads the full policy document (1 call). It decides it needs to check 3 months of transaction history (3 calls). It compares the transactions and generates an analysis (2 calls). At this point: 12 model calls for one support ticket.

But the agent also hit an unexpected branch: the customer had a related complaint from 6 months ago that seemed relevant. The agent pulled that thread. 4 more calls. Then it decided the customer's account history was relevant. 3 more calls. Then it drafted two resolution options, revised each based on company policy, and formatted the final response. 6 more calls.

Total: 25 model calls for one support ticket, at $0.05-0.15 per call = $1.25-3.75 per ticket resolution, versus the $0.10-0.20 cost you budgeted based on your pilot with simple tickets.

At 10,000 complex tickets/month, the actual cost is $12,500-37,500/month versus a budgeted $1,000-2,000/month. This happens.

The cost control requirement: hard iteration limits (maximum 10 model calls per task), per-task token budgets, and automatic handoff to human agent when limits are reached. These aren't operational conveniences. They're financial controls.

"An Autonomous Agent without hard iteration limits is not a productivity tool. It is a financial liability. Gartner's March 2026 analysis confirms agentic models require 5-30x more tokens per task than standard chatbots. An agent that reaches the upper end of that range on complex support tickets costs $3-4 per resolution at enterprise token pricing, versus a budgeted $0.10-0.20." (Rework Autonomous Agent Cost Analysis, 2026)

The Token Compound Cost Rule

The Token Compound Cost Rule states that total enterprise AI spend scales with the number of LLM calls per user task, the average context window size per call, and the retraining frequency per pattern, not with the per-token price. This explains why total enterprise AI spend has risen 320% while individual token prices fell 280x: the shift to agentic workflows (10-20 calls per task), RAG architectures (3-5x context window inflation), and always-on monitoring agents creates compounding call volume that overwhelms per-token price reductions. The Rule's practical implication is that cost control at scale requires limiting calls per task, caching repeated context, and scoping deployment to highest-value workflows, not waiting for token prices to fall further.

Rework Analysis: Based on Gartner's finding that agentic models require 5-30x more tokens per task and Oplexa's finding that enterprise AI spend rose 320% despite 280x token price declines, the Token Compound Cost Rule identifies three cost multipliers that pilot budgets systematically miss: call volume compounding from autonomous loops, context window inflation from RAG and history retrieval, and retraining frequency costs that scale with data complexity. Rework's implementation data shows that teams that model all three multipliers before deployment approval have average production cost overruns of 23%. Teams that model only per-token price have average overruns of 287%.

The four most common cost overrun scenarios

Scenario 1: The embedding index that grows without pruning. A RAG system is deployed with a clean 10,000-document knowledge base. Nobody removes old documents when policies update or products are discontinued. Two years later, the index has 80,000 documents (most of them outdated), retrieval quality is declining as the model retrieves stale content, and re-indexing to fix it costs more than the original deployment. Budget for index maintenance from day one. This is also how RAG systems become tech debt. See when AI patterns become tech debt for the full cost trajectory.

Scenario 2: Autonomous Agent without iteration limits. Described above. This is a finite risk with a complete solution: budget caps and iteration limits, defined before deployment. Any Autonomous Agent deployment proposal that doesn't include these as non-negotiable requirements should be sent back. Andreessen Horowitz's analysis of LLMflation and inference economics shows that while per-token costs are dropping 10x per year, total enterprise inference spending is rising because usage grows faster than prices fall. That dynamic makes iteration limits critical regardless of how cheap individual tokens become.

Scenario 3: Meeting Intelligence processing every internal meeting. The easiest cost overrun to avoid. 70% of meetings in most organizations are internal. Meeting Intelligence provides zero CRM value for internal meetings. Scope the deployment to customer-facing calls only before launch, not after the bill arrives.

Scenario 4: Generative Research at too broad a scope. Research prompts that say "research everything relevant to X" produce complete results but complete costs. Define maximum source counts, maximum document depth, and topic scope in your research prompt templates. "Research the last 6 months of competitive activity from Competitor X, using the top 10 most relevant sources" produces 85% of the value of "research everything about Competitor X" at 20% of the cost.

Building a cost model before deployment

For each pattern deployment, model these inputs before approval:

Build three scenarios: conservative (current volume), moderate (2x current volume in year 1), and aggressive (5x volume at peak). If the aggressive scenario produces an unacceptable cost, design the cost controls before deployment, not after.

Why pre-deployment estimates are usually too low: samples come from the easiest, most representative cases. Production includes all the edge cases, long documents, complex queries, and unexpected usage patterns that pilots filter out. Add a 50-100% buffer to your central estimate.

Monitoring for cost anomalies

Apply the Anomaly Agent concept to your own AI cost data. Set up cost-per-transaction dashboards for each deployed pattern. Define normal cost ranges based on your first 60 days of production data. Set alerts when cost-per-transaction rises more than 30% above baseline.

Early warning signals:

• Average context window size increasing (sign of prompt scope creep or input size changes)
• Iteration count per Autonomous Agent task increasing (sign of task complexity creep or model drift)
• Index refresh frequency increasing (sign of knowledge base growth without pruning)
• Error rates increasing alongside cost (sign of model struggling, leading to retry cost)

When a pattern becomes prohibitively expensive

The decision framework:

Optimize first. Context compression, caching, model downgrade for simpler tasks, batching instead of real-time processing. A typical optimization pass recovers 30-50% of cost without quality impact.

Scope-reduce second. Define the highest-value use cases within the pattern and restrict deployment to those. Meeting Intelligence for enterprise accounts only. Generative Research for tier-1 accounts only. This is not failure. It's rational cost allocation.

Replace with a less expensive pattern if optimization and scoping don't work. An Autonomous Agent doing task routing might be replaceable with a Scoring and Routing model at 5% of the cost, if the task complexity doesn't actually require multi-step autonomy. Pattern selection is always revisable. The buy vs. build decision by pattern article shows where vendor solutions reduce cost compared to custom builds.

See when AI patterns become tech debt for the long-term cost trajectory of patterns that weren't designed for maintainability, and measuring AI pattern ROI for how to track cost in relation to value. The goal isn't the cheapest deployment. It's the highest-value deployment at a cost the business can sustain at scale.

---

Learn more

• When AI Patterns Become Tech Debt
• Buy vs. Build Decision for Each AI Pattern
• Autonomous Agent: Multi-Step Goals With Tool Use
• How to Measure ROI of Each AI Pattern
• Governance Requirements by AI Pattern