Is Your AI Doing Useful Work or Just Spending Tokens? AI FinOps for Production SaaS
Caching, routing, and metrics to understand AI cost per useful task, with a transparent simulation and a practical evaluation plan for SaaS.
Voices depend on your browser and device; some require internet access. Code blocks are not read aloud.
Estimated reading time: 10 minutes. References and prices checked on October 26, 2026.
You launch an AI feature, your first users like it, and usage starts growing. Great news. Then the bill arrives, and you discover that the friendly little button also has an impressive appetite for your budget.
The usual response is to look for a cheaper model or a tool promising spectacular savings. I would start one step earlier: how much does it cost to deliver a task the user can actually use?
My work with marketplace integrations and Oombe keeps volume, repetition, and failure handling close to any architecture discussion. The same perspective is useful for AI. An application must remain useful when it moves beyond a demonstration and starts handling hundreds or thousands of operations.
That is where AI FinOps belongs: organizing decisions about cost, quality, and operations so the feature has sustainable economics.
The useful unit is the outcome
The State of FinOps 2026 surveyed 1,192 members of the FinOps community; 98% of respondents reported managing AI spending. That describes this sample, not every company in the world.
The FinOps Foundation connects technical spending with business outcomes, moving beyond cost per token toward units such as assistance provided or actions completed.
For a product, I would express that as:
Cost per accepted task = costs attributed to attempts and rework ÷ number of tasks that pass acceptance criteria.
Imagine a product attribute extractor. One model costs less per call but confuses measurements and needs frequent review. Another charges more yet produces more usable records. Looking only at input pricing can reward the option that creates additional work for the team.
Keep two views: model cost and the full operating cost. The first helps you investigate calls; the second includes tools, infrastructure, and human review. Failed attempts still belong in the numerator. Removing them from the report does not improve your margin.
A hosted API and your own GPUs need different calculations
An API customer can adjust model selection, call volume, context size, output, batch processing, and repetition. Operating your own infrastructure adds reserved capacity, memory, queues, maintenance, and idle resources to the calculation.
These are related problems with different budget mechanisms. An application receiving little traffic might pay dearly for a GPU waiting for work. Another, with steady demand and specific requirements, might justify investigating dedicated infrastructure.
I would start with the actual workload: daily volume, peaks, acceptable latency, input sizes, and quality requirements. Then I would compare scenarios. I would also separate capacity savings from an immediate reduction in the invoice: using an already contracted machine more efficiently does not automatically cancel the financial commitment.
Three different caches enter the discussion
Caching sounds straightforward until everyone uses the word for something different.
An application response cache stores an answer already produced. If the same request returns, reuse may be possible. Its design needs expiration rules, authorization, and keys scoped to the customer. Inventory information can become stale quickly; transforming an immutable document allows a different strategy.
A prefix cache reuses context processing. Anthropic's documentation requires exact matches for reused segments and distinguishes writes from reads. This does not remove output generation. Keeping stable instructions first and variable material afterward can support reuse.
KV cache sharing belongs to inference infrastructure. LMCache demonstrates reuse across compatible serving instances. This does not mean freely transferring one model's internal state to another provider.
I would begin with the technique that fits the existing workload. A distributed caching system in a small application may add more operational pieces than user benefit. A high-volume operation with repeated context deserves careful measurement.
I would also examine interactions between the choices. Rewriting the entire context for every request may undermine a stable prefix; switching providers may fragment reuse. These are architectural hypotheses to test, not promises about the behavior of every service.
A calculation you can reproduce, without inventing a case study
Here is a hypothetical simulation, not a measured Oombe result.
Assume 1,000 requests, each with an 8,000-token stable prefix, 2,000 variable input tokens, and 500 output tokens. In the cached scenario, there are 100 writes and 900 valid reads within the retention window.
The pricing table consulted lists Claude Sonnet 4.6 at these rates per million tokens: $3 for input, $15 for output, $3.75 for a five-minute cache write, and $0.30 for a cache read. All amounts are USD.
| Component | Without caching | With caching |
|---|---|---|
| Prefix: 8 million tokens | $24.00 | $5.16 |
| Variable input: 2 million | $6.00 | $6.00 |
| Output: 500,000 | $7.50 | $7.50 |
| Model total | $37.50 | $18.66 |
The $5.16 prefix cost comprises 800,000 written tokens costing $3.00 and 7.2 million read tokens costing $2.16. The total difference is $18.84, or 50.24%.
That percentage belongs to the assumptions in this example. A lower hit rate changes the calculation. So does a shorter or more variable context. These amounts exclude tools, additional infrastructure, taxes, currency conversion, and human work.
Notice the detail: the prefix discount did not magically spread across the entire invoice. Variable input and output still cost the same. Before announcing savings in your own product, replace every assumption with observed data.
A cheaper model must meet the same acceptance criteria
Routing can help. An inexpensive model might handle a simple classification, while an ambiguous case deserves escalation. OmniRoute is an example project documenting routing, fallback, and compression. That establishes proposed functionality, not savings validated in this article.
I would design routing around the task. Which fields must be correct? When can a result proceed automatically? Which failures require review? How many attempts are allowed?
A fallback must also respect the providers and conditions approved for the data involved. If the primary provider becomes unavailable, sending the content to any cheap alternative changes a decision that involves more than price.
There is an elegant trap here: spending several calls deciding which call would be the most economical. The router itself belongs in the calculation. For some tasks, a simple rule handles the initial classification with more predictable operating costs.
An experiment you can repeat
To turn the discussion into engineering, I would build a fixed collection of representative examples. For a catalog workflow, these could be synthetic records with different units, missing fields, and ambiguous descriptions.
First, define acceptance: valid JSON, preserved identifiers, correct units, no invented values, and required fields filled when evidence exists. Then hold back part of the dataset for final evaluation, without tuning the prompt against those cases.
Compare four configurations: the current workflow, a stable prefix, an economical model with escalation, and batch processing when the task's deadline permits it. Change one variable at a time so the results remain explainable.
The report should include cost per accepted task, success rate, human interventions, attempts, cache hit rate, and median and p95 latency. The last measure helps reveal slow operations that an average may hide.
Record model versions, prompts, rules, and the dataset as well. If the winning configuration loses quality the following week, you need a way to investigate what changed. A good-looking chart without reproducible conditions is presentation decoration.
Observability without logging everything by habit
Each operation should carry a task identifier, a customer or project reference, the feature involved, and the workflow version. That makes it possible to attribute a growing bill to a concrete part of the product.
Measuring cost does not require indiscriminately storing sensitive content. Counts, timings, error codes, and suitable identifiers already answer many questions. Content needed for evaluation deserves defined access and retention.
Caching is part of this architecture too. The vLLM security documentation states that a shared process does not provide full tenant isolation; cache_salt reduces specific risks without replacing an isolation boundary.
The practical decision is who may share what. Saving computation by crossing customer permissions would create a much more expensive problem. That question should accompany the application's design from the start.
What useful work can you offer customers?
I see a concrete service in diagnosis, instrumentation, and improvements followed over time. The first deliverable would be a map: where the application spends money, which tasks fail, how much rework exists, and which interventions have the best chance of helping.
The second would be a verifiable change, such as reducing repeated context, capping retries, separating tasks by complexity, or introducing budgets per customer. The third would track whether the savings persist without damaging the experience.
This follows the idea I explored in AI for repetitive administrative work: useful automation frees time and produces something that fits into the actual workflow. Sustaining that usefulness also means fitting within the budget.
I would rather present a modest, measured, repeatable improvement than promise spectacular savings before inspecting the data. A customer needs to understand what they receive, what it costs, and which limitations remain. That is how an exciting feature starts becoming a sustainable product.