Evaluating LLM Systems

A developer tweaks the prompt to fix one embarrassing answer. It works. Nobody checks the other three hundred questions the assistant handles, and a week later support notices that refund questions have started getting wrong answers. The fix for one case quietly broke another.

In ordinary software you'd have caught that with a test suite. GenAI systems need one too, but the tests look different because the outputs vary and "correct" is often a matter of degree.

What to measure

• Format validity: does the output parse and match the schema? Cheap and fully automatic.
• Retrieval quality: for RAG, did the right source land in the top results? Measured with recall@k and MRR.
• Answer quality: is it correct, complete and grounded in the sources?
• Abstention: does it say "I can't find that" when it should, and not when it shouldn't?
• Safety and policy: does it refuse or escalate what it must?
• Cost and latency: tokens per request, time to first token, 95th-percentile latency.

Start with retrieval

If the right chunk isn't retrieved, no prompt will rescue the answer. Retrieval is also easy to score without any model judging. Label a few dozen questions with the source that should answer each, then measure:

LABELLED = [   # question -> the source that should be retrieved
                          ("refund window", "S1"),
                          ("can I refund a download", "S2"),
                          ("when is support open", "S3"),
                      ]
                      
                      def retriever(question):                       # swap in your real retriever; returns ranked ids
                          return {"refund window": ["S1", "S2", "S3"],
                                  "can I refund a download": ["S1", "S2", "S3"],
                                  "when is support open": ["S3", "S1", "S2"]}[question]
                      
                      def recall_at_k(k):
                          hits = sum(1 for q, gold in LABELLED if gold in retriever(q)[:k])
                          return hits / len(LABELLED)
                      
                      def mrr():
                          total = 0
                          for q, gold in LABELLED:
                              ranked = retriever(q)
                              total += 1 / (ranked.index(gold) + 1) if gold in ranked else 0
                          return total / len(LABELLED)
                      
                      print(f"recall@1 = {recall_at_k(1):.2f}   recall@3 = {recall_at_k(3):.2f}   MRR = {mrr():.2f}")
                      

Recall@1 of 0.67 means the right source is first two times in three, and recall@3 of 1.00 means it is always somewhere in the top three. That tells you a reranker, or sending three chunks instead of one, would help. MRR rewards putting the right source higher in the list.

Judging answers

For answers, use a mix. Deterministic checks (schema, required facts, valid citations) come first. Human review of a sample catches what automation misses. Many teams also use a second LLM as a judge to score large volumes against a rubric. That is useful and has well-known weaknesses: judges can favour longer answers, their own style or the first option shown. Calibrate any judge against human labels on a sample before trusting it, and keep rubrics specific ("states the 30-day window and cites S1") rather than vague ("is helpful").

Make it a gate

Measurement only changes behaviour when it can stop a release. Compare each candidate with the current baseline and fail the build when a key metric gets worse:

import sys
                      
                      BASELINE = {"answer_accuracy": 0.86, "invented_answer_rate": 0.03, "p95_latency_s": 2.4}
                      CANDIDATE = {"answer_accuracy": 0.88, "invented_answer_rate": 0.05, "p95_latency_s": 2.1}
                      
                      RULES = {   # metric: (direction, allowed slack against baseline)
                          "answer_accuracy": ("higher", 0.01),
                          "invented_answer_rate": ("lower", 0.00),
                          "p95_latency_s": ("lower", 0.20),
                      }
                      
                      failures = []
                      for metric, (direction, slack) in RULES.items():
                          base, new = BASELINE[metric], CANDIDATE[metric]
                          ok = new >= base - slack if direction == "higher" else new <= base + slack
                          print(("PASS " if ok else "FAIL ") + f"{metric}: {base} -> {new}")
                          if not ok:
                              failures.append(metric)
                      
                      print("release blocked:" if failures else "release allowed", ", ".join(failures))
                      sys.exit(1 if failures else 0)
                      

Here accuracy and latency improved, but the invented-answer rate rose from 3% to 5%, so the release is blocked. Without the gate, the headline improvement would have shipped alongside a regression in the thing you care about most.

The solution architect's view

Evaluation is a requirement you write on day one, not a phase near the end.

Turn quality into acceptance criteria

Agree numbers before building: minimum accuracy, maximum invented-answer rate, minimum correct-abstention rate, latency and cost ceilings. These become the definition of done and the basis for go-live and rollback decisions.

Build the dataset like a product

Seed it from real questions and real failures, include unanswerable and adversarial cases, version it, review it with domain experts, and keep a held-out portion you don't tune against. Add every production incident as a new test.

Evaluate at every layer

Keep separate suites for retrieval, generation, tool use and whole conversations or agent runs. When something degrades, the layered view tells you where to look.

Continue after launch

Sample live traffic for review, track user feedback, and watch drift in inputs, retrieval scores, abstention rate and cost. Models, content and users all change underneath you, so a clean test run today is not a permanent guarantee.

Control change

Model upgrades, prompt edits, chunking changes and embedding changes all go through the same gate. Use shadow runs or canary releases for risky changes, and keep a fast rollback.

Questions for the design review

1. What are the acceptance numbers, and who agreed to them?
2. Where does the evaluation set come from, and how does it grow?
3. Do we evaluate retrieval, generation and end-to-end separately?
4. What blocks a release, and how fast can we roll back?
5. How do we detect quality drift in production?