Back to writing

The AI Boom Is a Backend Problem

16 min read

Two numbers that don't agree

In September, Gartner raised its forecast for worldwide AI spending in 2026 to about $2.7 trillion, up 49.5% on last year. That's the third time this year the number has gone up. January said $2.52 trillion. May said $2.59 trillion.

A few weeks earlier, McKinsey published The State of AI in 2026. It surveyed 1,719 people in 97 countries between May and June. About two in ten said their organisation had reached the point of scaling AI agents. And the share saying AI contributes anything at all to their EBIT sat at 37%, essentially where it was a year ago.

Put those side by side. The money is growing by half every year. The part where AI does paid work inside a company is growing much more slowly, and the part where it shows up in profit has barely moved.

I don't run a data centre and I'm not an analyst. I'm a student who builds backends and the occasional agent. But reading through this autumn's reports, the gap between those two numbers kept looking less like a market problem and more like a list of engineering problems, most of which I've run into myself at a much smaller scale. So this is the AI market as I read it from the backend: where the money is going, why agents stall between the demo and production, and what I'm changing in how I build because of it.


Where the money is going

Most of that $2.7 trillion isn't software. The biggest slice is infrastructure: chips, servers, data centres, power. On October 14, Gartner put numbers on the uncomfortable half of that story. AI providers are expected to spend nearly $1 trillion on AI infrastructure this year, and about $6 trillion more between 2027 and 2030.

The question with any capex is whether revenue shows up to pay for it. Gartner did that maths:

revenue AI data centre owners need, cumulative to 2030     $8.5T
  (announced capex + financing + a 12% return on capital)
 
AI revenue Gartner expects those data centres to earn       $3.6T
 
gap                                                         $4.9T

Hence the headline: more than $3.5 trillion of AI infrastructure spending could be exposed to impairment by the end of 2030. Impairment is accountant for "we built it, and it's now worth less than we paid." Gartner also expects more than 80% of neoclouds, the GPU rental companies one tier below the hyperscalers, to stop operating independently by 2030.

The hyperscalers aren't slowing down either. After their July earnings, Amazon, Alphabet, Microsoft and Meta were guiding to somewhere above $700 billion of combined capex for 2026, depending on how you count leases. They report Q3 over the next week or so, and I'd be surprised if that number goes down.

Revenue is growing too, and fast. Anthropic told investors its annualised run rate hit $65 billion in July, against roughly $10 billion of revenue in all of 2025. OpenAI's run rate was reported at around $40 billion over the summer. Those are absurd growth curves for any company, in any industry.

They're still nowhere near $8.5 trillion. Here's how I read that gap. It doesn't close because more people subscribe to a chatbot. It closes if AI moves from answering questions to doing work: an agent that resolves the ticket, reconciles the invoice, books the shipment, reviews the pull request, and gets paid for the outcome. The capex bet rests on agents becoming reliable, cheap and safe enough to trust with real workflows.

Mostly, that isn't a model problem anymore. It's a backend problem.


"Adopting agents" and "running agents" are different sentences

If you want a number about agent adoption, you can find almost any number you like.

survey                              what it asked                          result
PwC AI Agent Survey (2025)          are you using AI agents?                 79%
PwC, same survey                    integrated into almost all workflows?    17%
LangChain, State of Agent Eng.      do you have an agent in production?      57%
McKinsey, State of AI 2026          have you reached scaling with agents?   ~20%

None of these are wrong. They answer different questions. "Using agents" can mean someone in marketing has a copilot licence. "In production" can mean one internal agent that summarises support tickets for eleven people. "At scale" means the agent is load-bearing: if it goes down, work stops. Look at PwC's own numbers. The same executives who said 79% were using agents said 17% had them integrated into almost all their workflows.

Back in June 2025, Gartner predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear business value and weak risk controls. In the same release it estimated that only about 130 of the thousands of vendors selling "agentic AI" are selling agents at all. The rest is what Gartner calls agent washing: chatbots and RPA with a new label.

The distance between the top and the bottom of that table is where projects go to die. From the inside, it doesn't look like strategy. It looks like arithmetic.


The maths a demo hides

A demo is one run. Production is a distribution.

When I built AI Tutor Orchestrator at a hackathon, the whole thing was one hop: read the message, pick a tool, fill in its parameters, call it. One hop is easy to demo. What a demo can't show you is the thousandth message, or a task that takes twenty steps instead of one.

Every step in an agent loop is a model call that's right most of the time. And steps multiply:

const taskSuccess = (pStep: number, steps: number) => pStep ** steps;
 
taskSuccess(0.95, 1);   // 0.950
taskSuccess(0.95, 10);  // 0.599
taskSuccess(0.95, 20);  // 0.358
taskSuccess(0.99, 20);  // 0.818

A step that works 95% of the time sounds fine. Chain twenty of them and the task finishes about a third of the time. Even at 99% per step, a twenty-step task fails almost one run in five. That's the gap between the demo and production in four lines. Your demo was one of the good runs.

The way out is retries:

// a step you can retry k times, if you can tell it failed
const withRetries = (pStep: number, k: number) => 1 - (1 - pStep) ** (k + 1);
 
taskSuccess(withRetries(0.95, 1), 20);  // 0.951

One retry per step takes the twenty-step task from 36% to 95%. That's the whole trick, and it has two conditions hiding in that comment.

You have to know the step failed. A retry only fires on a failure you can detect. A malformed JSON blob is easy to catch. A well-formed answer that's quietly wrong is not. It's the same lesson as my quote and bill bug: every call returned a 200 and the bug shipped anyway. Silent failures never get retried.

Retries aren't independent. The maths assumes each attempt is a fresh coin flip, and it isn't. If the model misunderstood the task, it will often misunderstand it the same way twice. A retry with the same prompt, the same context and the same temperature is a coin you've already flipped. A useful retry changes something: it feeds back what was wrong, adds the missing context, or escalates to a bigger model. I learned the cost of losing that signal in I Tried My Own Context Engineering Advice, where trimming a tool result threw away the one line that contained the error.

So every step needs a detector. In the orchestrator, that was a Pydantic model: the extractor's output either parsed into ExtractedParameters or it didn't. Here's the same idea in TypeScript, with the validation error fed back into the next attempt:

import { z } from "zod";
 
async function step<T>(
  schema: z.ZodType<T>,
  call: (feedback?: string) => Promise<string>,
  maxAttempts = 3,
): Promise<T> {
  let feedback: string | undefined;
 
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const raw = await call(feedback);
    const parsed = schema.safeParse(safeJson(raw)); // undefined on bad JSON, never throws
    if (parsed.success) return parsed.data;
 
    // Don't retry blind. Tell the model exactly what was wrong last time.
    feedback = parsed.error.issues
      .map((issue) => `${issue.path.join(".")}: ${issue.message}`)
      .join("\n");
  }
 
  throw new StepFailed(feedback);
}

A schema catches a broken shape, not wrong content. For content you need checks that know your domain. Does this order ID exist? Is the date in range? Is the refund larger than the order total? Those checks are plain code, and if I had one day to make an agent more reliable, that's where I'd spend it.


Tokens got cheaper. Tasks didn't.

The other reason pilots get cancelled is the bill, which sounds backwards, because token prices have been in free fall. Andreessen Horowitz called it LLMflation: at the same level of capability, inference cost has been falling about 10x a year. Stanford's AI Index measured a drop of more than 280x for GPT-3.5-level performance in about two years.

And yet McKinsey found that AI operating costs are starting to limit AI use at about one in five organisations. Both are true, because nobody buys tokens. You buy tasks, and agents changed how many tokens a task takes, in three ways.

The transcript gets re-sent every turn. LLM APIs are stateless. On turn fifteen of an agent loop you send the system prompt, the tool definitions and everything from turns one to fourteen, again. Context grows linearly per turn, so the total input you pay for grows quadratically.

Reasoning tokens are output tokens. Models that think before they answer bill that thinking, usually at output rates, which cost several times more than input. You don't see those tokens in the response. You see them on the invoice.

Retries multiply everything. The fix from the last section isn't free. A retried step is another full call with the whole transcript attached.

Here's a back-of-envelope model. The prices are made up and round on purpose, so the shape is easy to see. Plug in your own provider's numbers.

const PRICE = { inputPerM: 1.0, outputPerM: 5.0 }; // illustrative, USD per million tokens
 
function agentTaskCost({
  basePrompt = 3_000,     // system prompt + tool definitions
  perTurn = 800,          // tool result + model message appended each turn
  outputPerTurn = 1_500,  // visible output + reasoning
  turns = 15,
} = {}) {
  let input = 0;
  for (let t = 0; t < turns; t++) input += basePrompt + perTurn * t;
  const output = outputPerTurn * turns;
 
  return (input * PRICE.inputPerM + output * PRICE.outputPerM) / 1_000_000;
}
single chat call     3,000 in  +    300 out   →  $0.0045
15-turn agent      129,000 in  + 22,500 out   →  $0.2415   (about 54x)

Same feature, same model, same prices: 54 times the cost per request, before a single retry. If you priced your product on the chat version and shipped the agent version, the margin you planned on is gone. That's one concrete way a pilot "lacks clear business value". The value can be real while the unit economics aren't.

Three levers move that number.

Prompt caching. The major APIs now discount input tokens that repeat a prefix they've seen recently, and in an agent loop almost everything is a repeated prefix: turn fifteen is turn fourteen plus 800 new tokens. If cached reads cost a tenth of normal input, the input side of that task drops from $0.129 to about $0.026, ignoring the small surcharge some providers add for writing to the cache. But caching only works if the prompt is ordered from stable to volatile: system prompt, then tool definitions, then history, then the new message. Put a timestamp or a request ID at the top and every single call is a cache miss. In Context Engineering I wrote that position in the context isn't neutral for quality. It isn't neutral for cost either.

Model routing. Not every step needs the frontier model. Picking a tool, extracting parameters and checking a schema are narrow jobs. In the orchestrator, extraction ran on GPT-4o-mini at temperature 0.1 for exactly that reason. Route the narrow steps to a small model and keep the expensive one for the step that needs judgement.

Effort budgets. Most reasoning models let you choose how hard they think. Start at the lowest setting, and only raise it for a step when an eval shows it actually helps.

With caching in place, output is now $0.11 of the remaining $0.14. The expensive part of a modern agent isn't reading, it's thinking, and routing and effort budgets are the two levers that control how much thinking you pay for.


The plumbing got standardised

Not long ago, connecting a model to your systems meant a custom integration for every tool in every framework. That part of the stack quietly settled.

In December 2025, Anthropic donated the Model Context Protocol to the new Agentic AI Foundation, a fund under the Linux Foundation that it co-founded with Block and OpenAI, with Google, Microsoft, AWS, Cloudflare and Bloomberg backing it. MCP joined Block's goose and OpenAI's AGENTS.md as founding projects. By then there were over 10,000 active public MCP servers, and ChatGPT, Cursor, Gemini, Microsoft Copilot and VS Code all supported it. Through 2026 it has become the default way agents talk to tools.

Standard plumbing is great. It also moves the hard problem. When any agent can call any tool, the integration is no longer your bottleneck. The tool boundary is now your security perimeter, and two things follow from that.

Every tool result is untrusted input. A web page, an email, a support ticket, a file served by an MCP server: anything that lands in the context can contain text that reads like an instruction. The model can't reliably tell "data I fetched" apart from "instructions I was given", because to the model both are just tokens. This is indirect prompt injection, and no system prompt fixes it. You fix it in code, outside the model.

A tool server is a dependency. Installing a third-party MCP server is the same trust decision as installing an npm package that runs with your agent's permissions. It deserves the same suspicion.

Here's the shape of the gate I'd put between the model asking for a tool call and the call actually running:

type Risk = "read" | "write" | "irreversible";
 
const TOOL_RISK: Record<string, Risk> = {
  search_docs: "read",
  get_order: "read",
  update_address: "write",
  issue_refund: "irreversible",
};
 
function gate(call: ToolCall, run: RunState): Decision {
  const risk = TOOL_RISK[call.name];
 
  if (!risk) return deny("unknown tool");                       // default closed
  if (!run.grantedTools.has(call.name)) return deny("not granted for this task");
  if (risk === "irreversible") return askHuman(call);           // a person signs off
  if (risk === "write" && run.tainted) return askHuman(call);   // untrusted text in context
 
  return allow();
}

The tainted flag is the important part. The moment anything from outside enters the context (a fetched page, an email body, a result from a server I don't control), the run is marked tainted, and every write after that needs a human to approve it. Reads stay free. It's crude, but the model can't talk its way around it, because the model never sees it.


The model race, seen from the backend

The model news this autumn moved as fast as ever. Late September and early October brought new frontier models from Google, OpenAI, Anthropic and Mistral, close enough together that the launch posts blurred into one. On October 12, Meta released Muse Wonder, the first model it's pitching as frontier-class. And the release trackers list more than a dozen other models this month alone, most of them small, fast or open-weight.

Here's how I read model news now. For most backend work, the model has become the most swappable part of the stack. When several labs ship frontier models within a few weeks of each other, nobody's moat is "we have the model". The parts that don't swap out are your data, your tools, your evals and your guardrails.

So I write code that assumes the model will change:

type ModelRole = "router" | "extractor" | "worker" | "judge";
 
// One place decides which model does which job. Pinned versions, never "latest".
const MODELS: Record<ModelRole, ModelConfig> = {
  router:    { model: env.SMALL_MODEL, effort: "low",    maxOutput: 200 },
  extractor: { model: env.SMALL_MODEL, effort: "low",    maxOutput: 500 },
  worker:    { model: env.LARGE_MODEL, effort: "medium", maxOutput: 4_000 },
  judge:     { model: env.LARGE_MODEL, effort: "high",   maxOutput: 300 },
};

Application code asks for a role, not a model name. Versions are pinned, because a model upgrade is a behaviour change and should ship like one. And nothing gets swapped until the evals have run. Which brings me to the part I think matters most.


Evals are the new unit tests

You can't ship a distribution without measuring it. A unit test checks that a function returns the right thing. An eval checks that a non-deterministic system returns the right thing often enough, at a cost you can afford.

This is about the smallest eval harness that's still useful:

type EvalCase = { input: string; check: (out: AgentResult) => boolean };
 
async function runEvals(cases: EvalCase[], runsPerCase = 5) {
  const results: { pass: boolean; costUsd: number; ms: number }[] = [];
 
  for (const c of cases) {
    for (let i = 0; i < runsPerCase; i++) {
      const out = await runAgent(c.input);
      results.push({ pass: c.check(out), costUsd: out.costUsd, ms: out.latencyMs });
    }
  }
 
  const passes = results.filter((r) => r.pass).length;
  return {
    passRate: passes / results.length,
    costPerSuccess: sum(results.map((r) => r.costUsd)) / Math.max(1, passes),
    p95Ms: percentile(results.map((r) => r.ms), 95),
  };
}

Three choices in there I'd defend.

Every case runs several times. One run is an anecdote. A case that passes three times out of five is a flaky step, which is exactly the 0.95-per-step problem from earlier, caught before a user finds it.

Cost per successful task, not cost per call. A cheap model that fails half the time is often more expensive than a pricier one that passes, once you count the retries and the failed runs you paid for anyway. This is the number I'd put on a dashboard, because it's the one that decides whether a feature makes money.

Checks are code wherever possible. Did it call issue_refund with the right order ID? Is the amount below the order total? Using a model to grade another model is fine for fuzzy things like tone, but it's a second distribution stacked on top of the first. Deterministic checks first, model judges for whatever's left.

Where do the cases come from? Start with twenty, written by hand from the tasks the agent is supposed to do. Then every failure in production becomes a new case. A year later, that set will be worth more than any prompt you've written.


What I'm changing

None of this is news to people who run agents at real companies. But reading this autumn's numbers next to my own small mistakes made a few rules click, and they're the rules I'm building by from now on.

  1. Budget per task, not per call. Every run gets a step limit, a token limit and a spend limit. When it hits one, it stops and hands back what it has. An agent that loops forever isn't a bug report, it's a bill.
  2. Every step has a detector. A schema at minimum, domain checks where it matters. A failed check is a retry with feedback, never a crash and never a silent pass.
  3. Stable prefix first. System prompt, tools, history, new input, in that order, so caching can work.
  4. Small models for narrow jobs. Route, extract and validate on the cheap model. Spend money on the step that needs judgement.
  5. Tools are closed by default and labelled by risk. Irreversible actions always need a person. Writes need a person once untrusted text is in the context.
  6. Every run is a trace. Inputs, tool calls, tokens, cost, outcome. You can't fix the thousandth run if you can't replay it.
  7. Cost per successful task is the metric. Not tokens, not calls, and definitely not "it worked when I tried it".

Where this leaves me

Gartner's gap is $4.9 trillion, and obviously no backend developer closes that. But the bet underneath all that capex is specific. It assumes agents become reliable enough to trust, cheap enough per task to make money, and safe enough to let near real systems. Each of those words maps onto something unglamorous:

reliable   →  detectors, retries with feedback, evals that run many times
cheap      →  caching, routing, effort budgets and spend limits
safe       →  default-closed tools, taint tracking, a human on irreversible actions

None of that is research. It's backend engineering: the same validation, budgets, permissions and observability we've always needed, applied to a component that's wrong some percentage of the time.

That's good news if you're where I am. As a student it's easy to feel like the AI boom is happening somewhere else, in labs training models none of us will ever train. The models are being built somewhere else, sure. But whether they earn back $8.5 trillion gets decided in ordinary codebases, one detector and one budget at a time. Anyone who can write a good retry loop can work on that part.


Sources