TECHNIQUE
Evaluation
Prompt regression suites in this pool are mostly dataset-backed eval loops: operators curate examples from logs/feedback, run offline or CI regressions before changes, and increasingly reuse scorers in production monitoring.
Operators moved from ad hoc or vibe-based checks toward formal regression/evaluation workflows.
4 of 9 operators explicitly described replacing ad hoc, vibe, spreadsheet, or fragmented manual checks.Operators maintain curated datasets, golden examples, labeled examples, or test sets as the backbone of regression testing.
8 of 9 operators showed dataset- or example-backed regression suites.Operators run offline, staging, experiment, or pre-production evaluations to detect regressions before deployment or before deciding whether to ship an update.
8 of 9 operators showed pre-ship, offline, experiment, staging, CI, or pre-production regression evaluation.Some operators integrate regression suites directly into CI or pull-request workflows, with explicit threshold-based merge blocking.
2 of 9 operators explicitly described CI/PR-integrated regression gates that block merges or require thresholds.Operators feed production logs, traces, user feedback, or sampled live traffic back into evaluation sets and monitoring.
6 of 9 operators showed production/log/trace/feedback data being used for evaluation or regression-suite upkeep.Operators use LLM-based judges or evaluators to score qualitative, factual, or similarity dimensions in regression tests.
6 of 9 operators showed LLM-as-judge, LLM self-evaluation, or LLM-based similarity/evaluator scoring.Operators supplement automated scoring with human review, manual labels, or human-in-the-loop evaluation inputs.
6 of 9 operators showed human review, labeling, manual golden-set construction, or human-in-the-loop inputs alongside automated evaluation.Operators reuse evaluation scorers or feedback charts in production monitoring, not only offline tests.
4 of 9 operators explicitly showed online or production monitoring tied to scorers, feedback charts, or sampled live traffic.Across the deployed LLM-application operators with direct prompt-regression-suite evidence, the suite is anchored in reusable examples, datasets, logs, traces, or labeled cases rather than only one-off inspection.
Across those same deployed LLM-application operators, regression evaluation is tied to iteration on prompts, models, retrieval, tools, or product behavior rather than treated as a one-time launch checklist.
Operators differ on where regression suites run in the delivery lifecycle.
APPROACH 01
CI or PR-integrated regression gates with explicit merge blocking or threshold requirements.
APPROACH 02
Offline, staging, experiment, or pre-production evaluations used to compare changes and detect regressions before deployment or release.
APPROACH 03
Production or online monitoring where live traffic, feedback charts, or real-time scorers continue to evaluate behavior after release.
Operators differ on scoring method: some rely on LLM judges, some use deterministic/task metrics or statistics, and several keep human labeling in the loop.
APPROACH 01
LLM judge, LLM self-evaluation, or LLM-generated similarity/evaluator scores.
APPROACH 02
Heuristic, deterministic, statistical, or task-specific metrics.
APPROACH 03
Manual review, manual labels, or manually curated golden mappings used to ground the suite.
Operators differ on how regression sets are populated and refreshed.
APPROACH 01
Production logs, traces, live traffic, customer feedback, or user interactions are mined for evaluation examples.
APPROACH 02
Synthetic, manually curated, or domain-specific golden datasets are built to cover known scenarios and edge cases.
Ad hoc, vibe, spreadsheet, or fragmented manual evaluation processes were reported as insufficient once AI work scaled.
Multi-stage and agentic systems create regression risk beyond a single prompt: operators reported unpredictable pipeline ripple effects, many LLM calls per interaction, and agents that adjust based on their own results.
Regression suites require ongoing curation work: operators described manual golden-set construction, data specialists labeling failures, labeled memory examples, and continuously expanding test datasets with new scenarios and edge cases.
Quality regression is not the only monitored failure mode: operators also track latency, cost, error rates, real-time deviations, or model expense/slowness.
| Name | Kind | When | Maturity |
|---|---|---|---|
| promptfoo CI gates | library | prompt changes blocked on regression suites like any other code change | established |
| Golden-transcript snapshot diffs | pattern | output drift surfaces as reviewable diffs, not failed scores | commodity |