TECHNIQUE
Serving & Inference
Across the deployed pool, model routing and cascades are implemented as application/platform orchestration: operators combine pre-filters, stage-specific models, fallback paths, gateways, and human review rather than relying on one model call.
Use thresholded cascades so cheaper or simpler checks handle clear cases and LLMs handle only the uncertain or suspicious band.
2 of 4 operatorsRoute low-confidence or suspicious outputs to human review instead of letting the model decision stand automatically.
2 of 4 operatorsAssign different models or model classes to different stages or roles in the workflow.
2 of 4 operatorsPut multiple providers or model endpoints behind a central platform abstraction or gateway.
2 of 4 operatorsUse automatic fallback, spillover, or real-time rerouting when a model or endpoint is degraded, limited, or over reserved capacity.
1 of 4 operatorsLog or expose routing/cascade execution details for audit, debugging, cost tracking, or future correction datasets.
3 of 4 operatorsUse structured outputs or schemas between cascade stages so downstream stages receive constrained, machine-readable results.
2 of 4 operatorsAll four observed operators implement routing or cascading in application/platform architecture: staged grading, a provider gateway, chained investigation agents/model roles, model hierarchy/fallback, or a results resolver.
Routing objective differs by deployment.
APPROACH 01
Reduce expensive LLM usage by letting earlier filters or embeddings pass easy cases and sending only uncertain/suspicious cases onward.
APPROACH 02
Improve reliability and capacity by falling back to different models, regions, endpoints, or on-demand capacity.
APPROACH 03
Centralize access, authentication, authorization, and provider integration behind a gateway.
Model-selection strategy differs.
APPROACH 01
Fixed stage-specific model choices: Claude 3.5 Sonnet for rubric extraction, GPT-4o for scoring, and text-embedding-3-small for semantic screening.
APPROACH 02
Cost-tiered model roles: low-, medium-, and high-cost models for expert, critic, and director functions.
APPROACH 03
User- or use-case-level model choice through a gateway exposing multiple LLM providers and APIs.
APPROACH 04
Blend rule-based checks, a legacy CNN pre-filter, and a fine-tuned LLM for the harder classification cases.
Operators split on fine-tuning versus calibration/prompted orchestration.
APPROACH 01
Avoid fine-tuning and maintain a calibration loop instead.
APPROACH 02
Fine-tune an LLM on the operator’s specific dataset.
The terminal action of the cascade differs.
APPROACH 01
Send unresolved or suspicious items to human review.
APPROACH 02
Reroute the request to another healthy model endpoint or capacity pool.
APPROACH 03
Return provider responses through the gateway with no to minimal processing after authentication, translation, and proxying.
Cost, throughput, or capacity limits are a recurring driver for routing: operators limit LLM calls to uncertain cases, spill excess traffic to on-demand endpoints, or report cost/showback from gateway usage.
Human review remains part of deployed cascades where confidence or policy risk is insufficient for automatic handling.
Operators report needing safeguards for low confidence, hallucinations, variability, degraded models, or endpoint limits.
Routing across providers or infrastructure requires security, compliance, authentication, authorization, and governance work before traffic moves.
Calibration or labeled data remains operational work: deployments cited manual grading samples, validation samples, corrections datasets, or expanded training datasets.
| Name | Kind | When | Maturity |
|---|---|---|---|
| Tiered model policy | pattern | a fast default handles most traffic; hard cases escalate by rule | established |
| OpenRouter | service | multi-provider routing and fallback without per-provider plumbing | emerging |