TECHNIQUE
Serving & Inference
Token-cost engineering is practiced mainly as context-budget control: operators prune schemas, prompts, tools, retrieval scope, duplicate compute, or track per-run token spend rather than simply sending more context.
Prune or narrow input context before the LLM call.
3 of 8 operators with cited token-cost evidence in this pool.Reduce tool/context surface area exposed to the main model.
3 of 8 operators with cited token-cost evidence in this pool.Break large LLM tasks into smaller prompts, agents, or subagents to conserve context and focus the model.
3 of 8 operators with cited token-cost evidence in this pool.Use serving/runtime optimizations to cut repeated computation or resource cost.
2 of 8 operators with cited token-cost evidence in this pool.Track or monitor token usage, per-run cost, or cost-related quality signals after deployment.
2 of 8 operators with cited token-cost evidence in this pool.Choose model/context-window tradeoffs explicitly when token volume is part of the workload.
2 of 8 operators with cited token-cost evidence in this pool.Across the cited deployments, token-cost engineering is tied to production constraints around context size, duplicate compute, latency, or per-run spend—not treated as an offline-only prompt cleanup exercise.
Operators apply token-cost controls at different layers of the stack.
APPROACH 01
Input/context layer: strip schemas, reduce prompt word count, split large tables, prune columns, or narrow RAG search radius.
APPROACH 02
Orchestration layer: reduce exposed tools, route through subagents/meta-tools, or skip retrieval/tool calls when not needed.
APPROACH 03
Serving/infrastructure layer: reduce repeated LLM compute or retrieval resource cost with KV caching, fused kernels, quantization, chunking, and asynchronous serving.
APPROACH 04
Accounting/monitoring layer: track token usage, per-run cost, and model/metric variants over time.
Operators differ on whether to pay for larger context windows or aggressively shrink the prompt.
APPROACH 01
Use a large-context model because the workload benefits from a wide window, while still citing speed/quality tradeoffs.
APPROACH 02
Shrink prompts or schemas so the model has less irrelevant context to process.
Context bloat degrades agent behavior: operators reported that too many tools, retrieval options, or issue records cannot simply be stuffed into the LLM context.
Large inputs can distract the model from the actual task; Grab reported model capacity being consumed by unnecessary detail, and Airbnb strips unused schema fields before prompting.
Token-cost controls do not remove the need for validation: operators still use human checks, schema validation, rule validators, or monitoring to catch misclassifications, invalid outputs, hallucinations, and drift.
| Name | Kind | When | Maturity |
|---|---|---|---|
| Context pruning with summarization budgets | pattern | context grows unbounded; compaction keeps cost linear | established |
| Per-feature output budgeting | pattern | cost control by capping output tokens per product surface | commodity |