TECHNIQUE
Evaluation
Golden-set offline eval practice in this pool is concrete but narrowly evidenced: Dropbox uses golden/curated suites as CI and staging regression gates, while Pinterest uses SME-labeled gold sets to monitor LLM labeling drift.
Run curated or golden offline suites before release changes, including end-to-end sweeps for regression detection.
1 of 2 operators with cited golden-set evidence in this pool.Put a fast canonical-query eval subset into pull-request and merge automation, with red-line misses blocking merges.
1 of 2 operators with cited golden-set evidence in this pool.Score eval examples using the query, model answer, source context, and sometimes a hidden reference answer; use judge models to check factuality, citation support, formatting, and tone.
1 of 2 operators with cited golden-set evidence in this pool.Build evaluation sets from internal production or dogfood logs, then reuse the same scoring logic across offline suites and production sampling.
1 of 2 operators with cited golden-set evidence in this pool.Periodically compare LLM-and-prompt quality against SME-labeled gold sets to detect drift.
1 of 2 operators with cited golden-set evidence in this pool.Keep human or SME review in the evaluation loop through manual spot-checks, human validation queues, or SME-reviewed prompts/gold sets.
2 of 2 operators with cited golden-set evidence in this pool.Operationalize eval outputs with dashboards, pass/fail metrics, trend views, diagnostics, or lineage for auditability.
2 of 2 operators with cited golden-set evidence in this pool.Both observed operators anchor evaluator quality to gold or reference data rather than relying only on ad-hoc checks.
Both observed operators keep human expertise attached to the eval loop.
Where the golden set sits in operations differs.
APPROACH 01
Use golden/curated evals as release-regression infrastructure in PR automation, staging, and merge gates.
APPROACH 02
Use SME-labeled gold sets as ground truth for periodic LLM + prompt drift checks in an AI labeling/prevalence pipeline.
The evaluated artifact differs by product context.
APPROACH 01
Evaluate a multi-stage conversational answer pipeline spanning retrieval, ranking, prompt construction, model inference, and safety filtering.
APPROACH 02
Evaluate multimodal LLM labels for policy-defined content categories, with gold-set drift checks and human validation.
Regression and drift are treated as live risks: Dropbox warns that a tweak anywhere in the pipeline can turn a prior good answer into a hallucination, and Pinterest checks LLM + prompt quality against gold sets to detect model drift.
Automated judging or labeling is not left unchecked: both observed operators use human review mechanisms around sampled outputs or labels.
Cost and latency are explicit evaluation concerns: Dropbox includes latency and cost smoke checks in merge automation, while Pinterest tracks token usage and per-run cost by model or metric variant.
| Name | Kind | When | Maturity |
|---|---|---|---|
| promptfoo | library | declarative golden-set evals wired into CI | established |
| Ragas | library | RAG-specific metrics: faithfulness, context precision/recall | established |
| pytest-style eval harness | pattern | evals as code in the existing test runner and review flow | commodity |