About Our Contributing Expert
Brij Mojan Singh | Head of AI and Data Science
Brij Mohan is the Head of AI at The Modern Data Company, where he leads the company’s AI and data science initiatives, helping enterprises build AI-ready data foundations and intelligent data platforms. He brings decades of experience spanning machine learning, distributed systems, and engineering leadership.
Prior to Modern, Brij held senior engineering leadership roles at Amazon, where he led machine learning and AI engineering teams, and at 24[7].ai and EngageClick, where he helped build large-scale AI-driven customer engagement platforms.
Brij’s interests extend beyond building models to understanding the architectural principles that enable AI at enterprise scale. His writing explores topics such as AI cost governance, agentic AI, software architecture, Conway’s Law, data products, and the evolving relationship between data infrastructure and intelligent systems. Known for applying first-principles thinking to complex technical problems, he bridges deep engineering with practical strategy for organizations preparing for the AI era.
We’re thrilled to feature his insights on Modern Data 101.
We actively collaborate with data experts to bring the best resources to a 20,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com
*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise!
Let’s Dive In
The cost of running an AI system has always reduced to one equation:
Cost = Price per token × Tokens per task.
Every “AI just got cheaper” headline is reporting on the first variable. Per-token prices really have collapsed: running a frontier model cost roughly $60 per million output tokens in 2023.
By 2026, capable frontier models cost a fraction of that, and mid-tier models handle most enterprise work for under a dollar per million tokens. That part of the story is true and complete on its own terms.
But an equation with two variables can’t be understood by watching only one of them. If tokens-per-task is rising faster than price-per-token is falling, the bill goes up.
And that’s exactly what’s happening in most production AI systems today. The architecture that decides how many tokens a task consumes is where the real cost story is, and it’s very clearly a data and systems architecture question.
A Simple Interaction Versus a System
A 2024-style AI interaction was a single hop: prompt in, response out, maybe two thousand tokens, a couple of cents. That’s the baseline most cost estimates were built on, and it’s no longer what’s running in production.
A 2026 agentic workflow is not one hop, but an entire pipeline. An orchestrator decomposes the task, selects tools, calls a retrieval system, hands context to a sub-agent, validates the output, retries on failure, escalates to a different model, and finally assembles a response.
Every one of those steps is a separate inference call, and every inference call re-reads context and re-emits tokens.
Research from Microsoft and Stanford’s Digital Economy Lab found that agentic tasks consume roughly 1,000 times more tokens than a standard chat exchange. Cut the token price by 75% and multiply consumption by 250, and the arithmetic isn’t ambiguous. The invoice clearly grows.
This is the same move as replacing a single database query with a distributed system of microservices: each hop is individually cheap and collectively expensive. The unit cost went down, and the number of units went up faster.
Where the Tokens Actually Go
Let’s break the agent stack into its layers to remove the mystery out of spending:
Orchestration.
A coordination layer sits above most production agents, decomposing tasks and routing work between components. Anthropic’s own research reports that multi-agent systems use roughly fifteen times the tokens of a simple chat call.
A three-agent pipeline can burn about 29,000 tokens to do what a single agent handles in 10,000, and the overhead is the coordination cost instead of reasoning spend. Coordination happens through state passed between agents and shared context re-synchronized at every handoff.
In a five-agent system with fifty reasoning steps built on an 8,192-token planning document, naive context broadcasting alone generates over two million tokens of overhead. Almost all of it is unchanged context, retransmitted at every synchronization point because nothing was deduplicating it.
Context accumulation.
Agents are stateful. Tool outputs, intermediate reasoning, and prior turns all travel forward in the context window at every subsequent call. A twenty-step reasoning trace can hit 50,000 tokens of accumulated history, past which the model is spending more tokens re-reading its own past than reasoning about the current step.
Forwarding full context across a multi-agent pipeline compounds this: a 50-message thread with four handoffs means the fifth agent processes roughly 200 messages, and cost scales quadratically with the number of handoffs, not linearly.
Microsoft’s research adds a specific warning here: token usage on identical tasks varied by up to 30x run to run, and higher token usage didn’t reliably produce higher accuracy. This is unmanaged state, not intelligence.
Retries and failure.
Production systems fail in these ways: hallucinated tool schemas, reasoning loops, context overflow, misclassified severity. Each failure triggers a retry, and each retry re-prompts with a now-larger context.
Each ambiguous result may trigger another validation pass. None of this appears in initial cost estimates, because none of it was part of the design intent.
Researchers call this the Unreliability Tax: the compute spent compensating for the probabilistic uncertainty that agentic systems introduced into what used to be deterministic pipelines.
An agent that works 80% of the time looks impressive in a demo. Pushing it to 95% reliability for production is mostly a token cost and doesn’t imply a capability upgrade.
Tool description overhead.
Every tool an agent can call has to be described in the system prompt, so the model knows when and how to use it. Research notes accuracy degrades noticeably past fifteen tools and collapses past fifty, while the token cost of describing all those tools grows linearly with tool count regardless.
Some production systems now burn tens of thousands of tokens on the system prompt alone, before a single word of the actual user request is processed.
Model misallocation.
The most expensive problem is indiscrimination. Default to a frontier model for every task regardless of what the task requires, and a standard FAQ answer gets priced like a legal analysis.
Enterprise data suggests that in early 2025, roughly 73% of enterprise token volume was routed to the two most expensive model tiers. A response that could run on a model priced at five cents per million tokens instead ran on one priced at thirty dollars per million: a 600-fold cost multiplier for identical output quality. The problem is that it’s missing a routing layer.
The Pattern of this Equation is Called Jevons’ Paradox, and It’s 160 Years Old
None of this is unique to AI. In 1865, William Stanley Jevons observed that more efficient coal engines didn’t reduce total coal consumption, but increased it, by making coal-burning economical for applications that had previously been too expensive to justify.
Efficiency didn’t shrink demand. It expanded it faster than the efficiency gain shrank per-unit cost. The token market is following that curve precisely. When output tokens cost $60 per million, organizations were deliberate: AI got deployed only where the return was obvious.
As the price fell, use cases that had been uneconomical became viable, then routine, then default. Total enterprise AI spending rose 320% in 2025 while per-token prices kept falling. Inference costs fell roughly 1,000-fold over the period; demand rose roughly 10,000-fold.
The lower unit price didn’t reduce spend, but authorized more consumption than it saved. Goldman Sachs projects agentic AI will multiply consumer-side token consumption 24-fold by 2030, with combined demand reaching roughly 120 quadrillion tokens processed monthly. Satya Nadella named the dynamic directly when DeepSeek shipped a cheap model: Jevons Paradox, striking again.
The Precedent that We’ve Already Survived
Data architects have already lived through one version of this: cloud computing. AWS made compute dramatically cheaper per unit, and the promise was that organizations would pay less overall.
Both halves of that promise were individually true, and total compute spend still went up because cheap, accessible compute made architectural complexity viable that had previously been impractical.
Monoliths became microservices, batch jobs became real-time streaming. A simple website became a globally distributed system with dozens of dependent services. Per-server-hour cost fell while server-hours consumed multiplied. Budgets rose for exactly the organizations that adopted the technology fastest.
The AI token economy is the same curve wearing a different unit. Cheaper tokens didn’t shrink the bill but funded more architecture, and the architecture is what’s actually being billed.
Building the Fix From the Same Foundation
If the cost problem is architectural, the fix has to be architectural too instead of a pricing negotiation.
Route by task.
The single highest-leverage lever is matching model capability to task complexity instead of defaulting to the frontier model for everything. Production data suggests roughly 85% of enterprise queries can be handled by budget-tier models with no measurable quality loss.
Reserve frontier models for the 5-15% of tasks that genuinely require deep reasoning, and route the rest to smaller models: a change that alone can cut AI spend by 60-80% with no user-facing impact.
The spread between the cheapest production models (around $0.04 per million tokens) and frontier reasoning models (upward of $180 per million) is roughly 4,500-fold. Defaulting to the expensive end for routine work is a design decision with a daily-compounding bill attached.
Measure at the workflow level.
Cost-per-model tells you what you spent. Cost-per-workflow tells you where it went and whether it was worth it, which is the only version of that number a data architect can actually act on.
Most organizations currently discover wasted frontier compute in a dashboard, after the spend has already happened. What’s missing is authorization before the spend: a governance gate that requires justification for routing to a frontier model, rather than a report that documents the damage afterward.
Build in fewer unnecessary hops.
Not every problem needs an agent, not every agent needs five sub-agents, and not every workflow needs a validation pass, a retry loop, and a reflection cycle stacked on top of each other.
The most expensive system in production is the one making the most unnecessary calls. An orchestration layer that costs more than the work it orchestrates is an architecture defect, and it should be modeled before deployment, not discovered in the invoice.
Tokens as a Design Variable
What’s emerging unevenly is a discipline that treats token consumption the way cloud architects learned to treat latency and data transfer: as a first-class design variable, budgeted and attributed per workflow.
That requires instrumentation at the call level, cost attribution per team and per workflow, deliberate model-selection policy, and the organizational willingness to treat this as an engineering concern rather than an invoice-reconciliation task.
Current pricing is still partly underwritten by provider investment runways, and post-IPO economics may reprice the stack in ways that are hard to predict. But the structural dynamic underneath the pricing (cheaper tokens enabling more complex architecture that consumes more tokens) isn’t a temporary artifact of this moment. It’s the trajectory of a technology getting more capable and more embedded at the same time.
The price per token dropped while the bill went up anyway. Clearly, the second variable in the equation matters exactly as much as the first. We must build systems that are capable of keeping this variable affordable.
The concepts shared in this article has also been discussed at length alongside solutions by the author here ↗️
MD101 Support ☎️
If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡
Author Connect 💬
Got questions? Find Brij on LinkedIn or drop a comment below. 💬
From the Modern Data 101 Team ❤
Just Launched: State of Data Products 2026 Q2
State of Data Products is a quarterly read for senior data and AI decision-makers that reports the industry reality of enterprise data and AI. This edition covers ontology as AI infrastructure, the agentic AI deployment gap, observability blind spots, and why lean AI is the economic reality check the industry needed.
Congratulations on an article that clearly illustrates the dangers of process fragmentation at the architectural and design levels. If you have decided that your code will be generated by your primary AI engine, a necessary step before you commit to deployment is to review the structure of the generated code for physical overheads as well as logical shortcomings.
Thanks for this. The unreliability tax you price in doesn't stop at the token ledger — once retries and reliability engineering have done what they can, what's left over shows up downstream as human verification time instead. A survey of 2,275 finance professionals found teams now spend 13 hours a week reconstructing and validating what their AI produced, clawing back roughly a quarter of the productivity gain the deployment was bought to deliver. That's a third term your equation doesn't carry: unlike price-per-token, human verification cost doesn't fall as compute gets cheaper — it's sticky exactly where the token cost is deflating.