Why Your AI Cost Per Query Doubled This Quarter (and What It Means for Your Roadmap)

If you ship AI-powered features in production, your cost-per-query line item this quarter is materially different from last quarter. For most teams we work with, it’s roughly doubled. For some, it’s tripled. For a few that did the work, it’s down 40%.

The volatility is structural and it will keep happening. The roadmap that survives this environment is built differently from the one that doesn’t.

What’s driving the volatility

Four forces, all compounding:

1. Model upgrades are not free upgrades. Each new generation — Claude Sonnet 4 → 5, GPT-4 → 5 — comes with quality improvements that justify (from the vendor’s side) per-token price increases. The pricing pages haven’t all moved up yet, but the effective cost per query goes up because: - Outputs get longer (the model thinks more) - Thinking modes add token overhead - Context windows grow, and so does context use

2. Your usage shifted from completion to agent mode. If you migrated even one workflow from “model answers a question” to “model uses tools to accomplish a task”, your token count per workflow run went up 5–15×. Each tool call, each result, each agent step is more tokens.

3. Provider pricing isn’t stable. The major providers raised effective prices on at least one SKU between Q4 2024 and Q1 2026. Not catastrophically, but reliably. Pricing pages are still updated as much as the rest of the company.

4. You’re using better models for more cases. A team that started with Haiku for everything ended up using Sonnet for the cases that need quality, which is most cases. Average cost per call went up because the model mix shifted upward.

The roadmap implication: your cost-per-feature is now a moving target, not a fixed estimate. Plan accordingly.

The roadmap that works

Three architectural decisions made early that pay off as the cost environment evolves.

Decision 1: Decouple model selection from product code

Your application code should never have the model name hard-coded. Wrap every LLM call in a thin adapter that takes a “task” identifier (e.g. classify_lead, draft_outreach, summarise_meeting) and an input, and returns the output. The adapter handles model selection internally based on configuration.

When the cost or quality of a model shifts, you change the config, not the code. When you want to A/B test a new model for one task, you flip the config for 10% of calls. When a provider has an outage, you fall back.

This is 80 lines of code. It costs you nothing to add. The teams that didn’t add it are the teams whose CFO is asking why the bill spiked and whose engineers are quoting two-week refactors.

Decision 2: Track cost per task, not cost per provider

Most teams instrument their LLM calls to track total spend. That’s the wrong granularity.

The instrumentation that matters: cost per task, measured as cost-per-completed-workflow-run for each task identifier. This tells you which features are economic on AI and which aren’t.

Common pattern: feature A is genuinely useful and costs $0.04/run. Feature B is cosmetic and costs $0.31/run. The roadmap conversation becomes “kill feature B and put the $35K/month into improving feature A.” Without per-task instrumentation, the conversation is “AI is expensive, can we use a cheaper model everywhere?” The wrong question.

Decision 3: Build the eval harness before the optimisation

When you’re tempted to switch a task to a cheaper model, the right move is to run both through your eval harness and compare quality. The temptation is to skip this and assume “the cheaper model will be 90% as good.” Empirically, sometimes it’s 95%, sometimes it’s 60%. The cost-quality frontier is non-linear and not generalisable across tasks.

The eval harness is the same architectural element we covered in our AI pilot graveyard piece — 200–800 representative test cases, codified expected outputs, automated scoring. Building it costs $15K–$40K of engineering time. Not building it costs you the ability to do informed model-tier optimisation, which over a year is worth orders of magnitude more.

The optimisation playbook

Once decisions 1, 2, and 3 are in place, the optimisation playbook is mechanical:

Quarterly: review cost-per-task report. Identify the tasks where cost has grown faster than usage. These are the optimisation candidates.

Per candidate: run a 3-model bake-off. Through the eval harness, test the current model + two alternatives (one tier down, one tier up). Output: quality score + cost-per-run for each.

Decide: optimise, leave, or rethink. - If a lower tier delivers equivalent quality at half the cost, switch. - If the lower tier degrades quality below the threshold for the use case, leave it alone. - If even the current tier is poor quality, the task may not be a good LLM fit at all. Reconsider whether the workflow benefits from AI here.

This loop run quarterly typically reclaims 30–50% of LLM spend over 12 months without any product-side disruption.

What this means for your AI roadmap

The roadmap that worked in 2023 was “build the AI thing, get it in production, measure usage.” The roadmap that works in 2026 has an additional element: a quarterly cost-quality review where the model mix is treated as a separate concern from the product feature.

The teams that do this quarterly review have stable AI economics. The teams that don’t have a CFO conversation every quarter.

What we ship

AI Pod engagements that ship the architecture above as part of the original build — adapter pattern, per-task instrumentation, eval harness — so the quarterly optimisation loop runs without further engineering investment. The AI API cost calculator linked below estimates your honest provider mix cost across major providers, with optimisation suggestions for your specific workload.

If your AI bill doubled this quarter and you’re wondering what to do: run the calculator first. It usually clarifies whether the conversation is “switch providers” or “rebuild the architecture.”

Read more: /ai-pod/ · /upstream/ · /calculators/ai-api-cost