TECHNIQUE
Serving & Inference
Caching is used by 6 of 7 operators, but at different layers: LLM KV/prefix state, metadata/configuration, intermediate pipeline states, precomputed embeddings, AI responses, and agent context costs.
Use caching or precomputation to reuse work already present in the AI path, including fixed embeddings, overlapping context, conversion states, LLM KV/prefix state, metadata/configuration, and AI responses.
6 of 7 operators; no caching evidence was provided for Grab in this pool.Cache LLM serving computation: LinkedIn reports KV caching, prefix caching, and in-batch prefix caching to reduce duplicate work and reuse query-prefix KV.
1 of 7 operators.Put multi-level caches around inference metadata/configuration: Salesforce uses a local cache in the AIMS Client / AI Gateway and an L2 service-level cache in AIMS.
1 of 7 operators.Cache intermediate document-processing states so later LLM summarization and Q&A steps can reuse prior conversions.
1 of 7 operators.Precompute fixed embedding sets used at inference: Canva precomputes keyword embeddings for all languages for the top 1000 keywords and serves the 10 highest-probability keywords during inference.
1 of 7 operators.Use response caching inside an AI workflow/orchestration layer.
1 of 7 operators.Use caching to manage the token-consumption trade-off from overlapping context in multi-agent review workflows.
1 of 7 operators.Across the six operators with caching evidence, the cached artifact is workload-specific rather than uniform: embeddings, overlapping context, conversion states, LLM KV/prefix state, metadata/configuration, or AI responses.
Operators differ on what layer they cache.
APPROACH 01
LLM serving-engine state: KV caching, prefix caching, and in-batch prefix KV reuse.
APPROACH 02
Application metadata/configuration: local AIMS Client / AI Gateway cache plus service-level L2 cache in AIMS.
APPROACH 03
Document pipeline state: cached plugin conversions and intermediate pipeline states.
APPROACH 04
Fixed candidate embeddings: precomputed keyword embeddings for all languages and top keywords.
APPROACH 05
AI interaction responses inside a workflow framework.
APPROACH 06
Overlapping multi-agent context in code review workflows, managed with caching strategies to control token consumption.
| Name | Kind | When | Maturity |
|---|---|---|---|
| Provider prompt caching | service | repeated system prompts and contexts dominate token spend | commodity |
| Semantic response cache | pattern | near-duplicate queries can safely share answers | emerging |