Minimizing Tokens in AI Models: Techniques and Best Practices
The prototype cost almost nothing. At ten requests a minute nobody looked at the bill, and responses felt instant. At a thousand requests a minute, latency crept up, the GPU memory graph turned red, and the monthly invoice became a line item someone had to explain.
The model didn't change. The prompts did. They had quietly grown: a longer system prompt here, ten retrieved chunks there, a chat history nobody ever trimmed. Tokens are what you pay for and what you wait for, so tokens are where the savings are.
Measure tokens per part of the request, give each part a budget, then cut where it pays most: retrieved context first, chat history second, system prompt third, output last. Check quality after every cut, because a cheaper wrong answer isn't a saving.
Why tokens cost you twice
They cost money, because providers bill per token, and output tokens usually cost more than input tokens. They also cost time and capacity. A longer prompt delays the first token and takes up more GPU memory for the KV cache, which limits how many requests a GPU can serve at once. Long prompts also make latency less predictable, because one oversized request can slow others sharing the same hardware. (If KV cache is new to you, see LLMs explained simply.)
Measure before you cut
A request is made of parts: system prompt, tool or schema definitions, chat history, retrieved context, the user's message, and the answer. Log tokens for each part, per endpoint and per tenant, using the usage figures your provider returns or the model's own tokenizer. Then look at the 95th-percentile requests first. A few heavy paths nearly always dominate both spend and tail latency. Optimising without that baseline is how teams make things worse while feeling productive.
Give every part a budget
Treat tokens like money. Reserve headroom for the answer, set a cap per section, and trim the least valuable material first when you go over. A minimal version:
def count(text): # stand-in: swap for your model's real tokenizer
return len(text.split())
def fit(system, history, context, user, limit=80, reserve=10):
"""Keep system + question, then add context chunks and recent history while they fit."""
budget = limit - reserve - count(system) - count(user)
if budget < 0:
raise ValueError("system prompt plus question already exceed the budget")
kept_context = []
for chunk in context: # best-ranked first
if count(chunk) <= budget:
kept_context.append(chunk)
budget -= count(chunk)
kept_history = []
for turn in reversed(history): # newest turns first
if count(turn) > budget:
break
kept_history.insert(0, turn)
budget -= count(turn)
return kept_context, kept_history
words = lambda n: " ".join(["w"] * n)
ctx, hist = fit(system=words(10), user=words(5),
context=[words(15), words(15), words(30)],
history=[words(12)] * 4)
print(f"kept {len(ctx)} context chunks and {len(hist)} history turns")
The word count stands in for a real tokenizer. The logic is the useful bit: protect the system prompt and the question, fill the rest with the best-ranked context, keep the most recent turns of history, and drop what doesn't fit. Count how often you have to cut. If it is more than a few percent of requests, an upstream stage is over-producing.
Where the savings are, in order
1. Retrieved context
This is where the bloat hides. Use smaller, structure-aware chunks, retrieve generously, then rerank and send only the top two or three. If that is still too much, compress what remains by extracting the sentences that matter. Cache the compressed snippets so you don't pay for the same work twice.
2. Chat history
History grows with every turn and gets resent every time. Keep the last handful of turns verbatim and replace everything older with a short rolling summary that captures decisions, preferences and open questions. Summaries can drift, so re-ground from the source of truth now and then.
3. The system prompt
These grow like internal wikis. Turn paragraphs into short, atomic rules; drop duplicate examples; and load rarely needed rules only when a request needs them. Say once, in a schema, that the output must be JSON instead of reminding the model in three places.
4. The output
Output tokens are slower and pricier. Ask for structured output, cap max_tokens, use stop sequences where your provider supports them, and ask for what you need ("three bullet points") instead of "explain". Make sure a cap can't cut a JSON object off halfway, and validate what comes back.
5. Caching and routing
Many providers cache a stable prompt prefix, which can cut cost and time to first token. Put the big, unchanging parts (instructions, tool definitions) first and the volatile parts (history, retrieved text, the question) last. Send easy jobs such as classification to a small model and keep the large one for the hard cases.
Do the sums
Savings are easy to claim and easy to overstate. Put numbers on them:
def monthly_cost(requests, in_tokens, out_tokens, price_in, price_out):
"""Prices are per million tokens."""
return requests * (in_tokens * price_in + out_tokens * price_out) / 1_000_000
PRICE_IN, PRICE_OUT = 1.00, 4.00 # example numbers only; use your provider's price list
before = monthly_cost(1_000_000, 3_500, 400, PRICE_IN, PRICE_OUT)
after = monthly_cost(1_000_000, 1_600, 250, PRICE_IN, PRICE_OUT)
print(f"before: ${before:,.0f}/month after: ${after:,.0f}/month saved: {1 - after / before:.0%}")
The prices are examples only, so plug in your provider's. The point is the shape: trimming a request from 3,500 input tokens to 1,600 and from 400 output tokens to 250 roughly halves the bill at the same traffic.
Short prompts must keep their safety rules, so move policy into compact, testable rules and structural checks. A naive compressor can carry instructions hidden in retrieved text straight through, so mark retrieved content as data and filter instruction-like patterns. And token logs are data logs: redact secrets and personal information before prompts leave your application.
The solution architect's view
Token spend is an architectural property of the system, and it deserves the same attention as latency and availability.
Think in unit economics
Work out cost per request, per feature and per tenant, and compare it with the value each one produces. A feature that costs more to answer than it earns, or a single tenant sending huge prompts, only becomes visible if you measure it. Set per-tenant budgets and quotas, and reflect them in rate limits and, where relevant, in contracts.
Tier the work
Route by difficulty: small models for classification, extraction and compression, and larger ones for final synthesis. Put the routing rule under evaluation and under ownership. It is a design decision with cost and quality consequences, not a one-off optimisation.
Design prompts to be cacheable
Stable content first, volatile content last, and keep the stable part truly stable. A timestamp or user id near the top of a prompt can quietly defeat prefix caching.
Gate every saving on quality
Run the evaluation set before and after each change. A compression step that saves 30% of tokens and loses a key fact has just moved the cost to your support team. Record tokens saved and answer quality side by side.
Plan the failure modes
If the summariser or compressor times out, the request still has to succeed: fall back to a smaller context, log the breach, and move on. Avoid "repair" retries that keep appending to the prompt and inflate cost on exactly the requests that are already struggling.
Make it visible
Put tokens per request, tokens by section, cost per tenant, time to first token, truncation rate and compression savings on the same dashboard as latency and errors. Alert on sudden jumps.
Questions for the design review
- What is the token budget per section, and who owns it?
- Which 5% of requests cause most of the spend and the tail latency?
- How do we know a cut didn't reduce answer quality?
- What happens when compression or summarisation fails?
- Can we explain the cost of one request to finance?
Key Takeaways
- Tokens cost money, time and GPU capacity, so trimming helps on all three.
- Measure tokens per part of the request and start with the heaviest paths.
- Budget each section; cut retrieved context first, then history, then the system prompt.
- Cap and structure the output; it is the slowest and priciest part.
- Put stable content first to benefit from prefix caching, and route easy work to small models.
- Every saving must pass your evaluation set, or it isn't a saving.
Comments
Loading comments…