LLM SQL and Knowledge Base Quality Evaluation

LLM-assisted evaluation workflow for measuring generated SQL quality and source document quality, detecting regressions and recurring failure patterns, identifying component-level weaknesses, and guiding QueryGPT algorithm changes or Genie knowledge base documentation improvements.

Business Blueprint

GROUNDED

LLM-assisted quality evaluation for SQL generation and internal knowledge assistants, turning user logs, golden answers, feedback, and LLM judging into release and documentation improvement signals.

The Problem

Teams running LLM SQL generation and internal knowledge assistants need a repeatable way to measure answer quality, spot regressions, and distinguish recurring component weaknesses from one-off failures before changing algorithms or documentation.

Owners of LLM SQL products

Without standardized evaluation, they struggle to tell repeated shortcomings from anomalous failures and to locate component-level weaknesses in the SQL-generation flow.

Engineers, operations managers, and data scientists using generated SQL

SQL work depends on understanding syntax and internal business-data models; low-quality generated SQL pushes users back into manual dataset search and query authoring.

On-call support and knowledge-base owners

High internal question volumes and fragmented documentation create long waits, repeated back-and-forth, and productivity loss when answers cannot be trusted or found quickly.

Cost of Inaction

Repeated SQL and knowledge-answer failures remain hard to separate from anomalies, while users continue spending time on manual query authoring, support-channel waits, and back-and-forth resolution.

Process Fit

Software development & delivery

As-Is

LLM product teams rely on production prompts, logs, support feedback, curated SQL examples, and source documents to understand where generated SQL or knowledge answers fail, with some manual upfront work to create golden question-to-answer references.

To-Be

Quality evaluation becomes a standard checkpoint in the LLM product-change cycle: real user questions and feedback are scored, regressions are visualized over time, and component-level weak spots guide algorithm changes, prompt changes, or documentation fixes.

Human Checkpoints

  • Approve and maintain golden question-to-SQL answer mappings before they are used as evaluation references. — Domain expert or analytics owner
  • Review selected tables or correct them when the SQL assistant proposes an input set for generation. — SQL user or data analyst

Systems Touched

QueryGPTGenieSlack support channelsEngwikiUber internal Stack OverflowSQL samplestable schemasQueryGPT logsHive feedback and metrics tablesKafka metrics streamLLM evaluator / LLM-as-judge workflow

Business Cycle

Upstream

  • A supply of real user questions from production logs or support channels to make evaluation representative of actual demand.
  • Curated reference material, including golden question-to-SQL mappings, SQL samples, table schemas, and domain workspaces.
  • Knowledge-base content and feedback streams from internal documentation, internal Q&A sources, Slack reactions, and stored response metadata.

Downstream

  • Teams can visualize quality over time to identify regressions and recurring patterns that reveal improvement areas.
  • Teams can isolate component-level weaknesses instead of judging only the final end-to-end answer.
  • Quality findings become an input to prioritizing SQL-generation changes and knowledge-base documentation improvements.

Value Evidence

  • Regression and failure-pattern visibilityIMPROVED
  • Component-level evaluation capabilityIMPROVED
  • LLM-judge scoring for generated SQL similarityIMPROVED

    similarity score between 0 and 1

  • Internal knowledge-assistant footprint available for feedback and evaluationIMPROVED

    154 Slack channels and has answered over 70,000 questions .

  • Scale of internal analytics demand affected by SQL qualityIMPROVED

    approximately 1.2 million interactive queries each month

Adoption Journey

  1. LEVEL 2 — STANDARD

    Gate: Prove value as a production release checkpoint for prompt, retrieval, SQL-generation, or documentation changes.

    Outcome: Teams can catch regressions before or soon after release and compare quality trends over time.

  2. LEVEL 3 — ADVANCED

    Gate: Prove value across multiple business domains, Slack channels, data sources, and failure categories.

    Outcome: Quality management scales from single-assistant checks to portfolio-level views of recurring weak spots by domain, component, and source content.

Detailed per-level builds in the solution spectrum below

Risk & Governance

  • Evaluation quality depends on the quality and maintenance of golden reference sets.

    Posture: Treat golden mappings as owned business assets: assign domain owners, review updates, and refresh them with real production questions.

  • LLM-as-judge scores may be mistaken for final truth rather than a decision aid.

    Posture: Use LLM scores to triage patterns, but require human review for release-blocking regressions, high-impact documentation changes, or ambiguous failures.

  • Feedback and evaluation data can expose sensitive internal questions, metadata, or operational patterns.

    Posture: Keep evaluation data in controlled analytics stores, preserve audit trails, and limit access to teams responsible for the assistant and source systems.

  • Quality scores can hide domain-specific gaps if all business areas are evaluated together.

    Posture: Segment evaluation by workspace, business domain, channel, and source-document family so recurring failures are assigned to the right owner.

Operating Intelligence

How it works

Humans set constraints. AI generates options.

Humans choose what moves forward.

Selections improve future generation quality.

Confidence86%
ArchetypeGenerate & Evaluate
Shape6-step branching
Human gates2
Autonomy
50%AI controls 3 of 6 steps

Who is in control at each step

Each column marks the operating owner for that step. AI-led actions sit above the divider, human decisions and feedback loops sit below it.

Loop shapebranching

Step 1

Define Constraints

Step 2

Generate

Step 3

Evaluate

Step 4

Select & Refine

Step 5

Deliver

Step 6

Feedback

AI lead

Autonomous execution

2AI
3AI
5AI
gate
gate

Human lead

Approval, override, feedback

1Human
4Human
6↺ Loop
AI-led step
Human-controlled step
Feedback loop
TL;DR

Humans define the constraints. AI generates and evaluates options. Humans select what ships. Outcomes train the next generation cycle.

The Loop

6 steps

1 operating angles mapped

Operational Depth

Technologies

Technologies commonly used in LLM SQL and Knowledge Base Quality Evaluation implementations:

Intent accuracy metricOther
4 mentions
Query execution checksOther
4 mentions
Table overlap scoreOther
4 mentions
Golden question-to-SQL mappingsOther
3 mentions
Staging and production evaluation flowsOther
3 mentions
+2 more technologies(sign up to see all)

Key Players

Companies actively working on LLM SQL and Knowledge Base Quality Evaluation solutions:

Not specified in source

Real-World Use Cases

LLM-Based Document Quality Evaluation for Genie Knowledge Base

Uber asks an LLM to review each internal help document and say whether it is clear and useful, then suggest how to improve it so Genie can answer better.

Document critique and scoring using LLM-as-judge.implemented as a document evaluation app within genie’s broader evaluation workflow.
10.0

LLM-assisted evaluation of generated SQL quality

Uber tests QueryGPT by comparing its generated SQL against verified answers and using another LLM to judge how similar the generated query is to the correct one.

LLM-as-judge evaluation plus rule-based and execution-based quality checks over a golden question-to-SQL dataset.operational evaluation framework used to track querygpt performance over time, though affected by llm non-determinism and incomplete test coverage.
10.0

LLM-assisted evaluation pipeline for QueryGPT SQL quality

Uber tests QueryGPT by comparing its generated SQL against hand-checked correct answers and uses another LLM to score how similar the generated query is to the expected one.

LLM-as-judge evaluation combined with execution-based and retrieval-quality metrics.operational evaluation process used to track querygpt improvements across production and staging flows.
9.5

Free access to this report