Build an LLM Eval Harness That Runs in CI

Short answer: an eval harness is a test suite for output that is not deterministic. It has four parts: a fixed set of cases, a runner that calls your AI feature, one or more scorers, and a threshold that fails the build. You do not need a platform to start. A JSON file, forty lines of Python, and a GitHub Actions step will catch the regressions that actually hurt — a prompt change that quietly broke JSON output, or a model swap that dropped a required disclaimer.

An eval harness is the regression test for an AI feature. It runs a fixed case set against the current prompt and model, scores each output with deterministic checks where possible and a judged rubric where not, then compares the pass rate against a stored baseline and fails the build if it drops.

Why ordinary tests do not cover this

Unit tests assert on behaviour you can predict. An LLM feature has three properties that break that assumption:

• The output varies. The same input can produce different wording on two runs.
• Correctness is graded, not boolean. A summary can be partly right.
• Failures are silent. A prompt edit does not raise an exception. It just makes answers worse in a way nobody notices for three weeks.

The consequence is familiar to anyone who has shipped one of these features: prompt changes get reviewed by reading the diff, which tells you nothing about behaviour on real inputs.

The four parts

Part	What it holds	Common mistake
Case set	Inputs plus expected properties	Twenty cases that all look like the demo
Runner	Calls the feature, captures output and cost	Calling the model directly instead of the real code path
Scorers	Deterministic checks first, judged rubric second	Using an LLM judge for things a regex can check
Threshold	Minimum pass rate, compared to a baseline	An absolute number with no baseline to regress against

The case file

Keep cases in version control next to the code. Every production bug becomes a case; that is how the set earns its value.

[
  {
    "id": "refund-json",
    "input": "Customer was charged twice for order 8812. What should we do?",
    "checks": {
      "must_contain": ["refund", "8812"],
      "must_not_contain": ["I cannot", "as an AI"],
      "valid_json": true,
      "max_words": 180
    }
  },
  {
    "id": "escalation",
    "input": "I want to speak to a human about a legal dispute.",
    "checks": { "must_contain": ["escalat"], "tone": "calm" }
  }
]

Note what is not here: exact expected output. Asserting on exact strings is the fastest way to build a suite that fails constantly and gets ignored.

The scorer

Deterministic checks first. They are free, instant, and unambiguous.

import json, re

def score(output: str, checks: dict) -> tuple[bool, list[str]]:
    failures = []
    low = output.lower()

    for needle in checks.get("must_contain", []):
        if needle.lower() not in low:
            failures.append(f"missing: {needle}")

    for needle in checks.get("must_not_contain", []):
        if needle.lower() in low:
            failures.append(f"forbidden: {needle}")

    if checks.get("valid_json"):
        try:
            json.loads(output)
        except json.JSONDecodeError:
            failures.append("output is not valid JSON")

    if "max_words" in checks and len(output.split()) > checks["max_words"]:
        failures.append(f"too long: {len(output.split())} words")

    if "must_match" in checks:
        if not re.search(checks["must_match"], output):
            failures.append(f"pattern not matched: {checks['must_match']}")

    return (not failures, failures)

That covers most real regressions. A judged rubric is for the residue: tone, faithfulness to a source document, whether a summary omits the one thing that mattered.

Handling non-determinism without flaky builds

Run each case several times and score the pass rate, not a single run.

RUNS = 3

def run_case(case, feature, runs=RUNS):
    results = [score(feature(case["input"]), case["checks"]) for _ in range(runs)]
    passed = sum(1 for ok, _ in results if ok)
    return {
        "id": case["id"],
        "pass_rate": passed / runs,
        "failures": sorted({f for ok, fs in results if not ok for f in fs}),
    }

Three runs per case is usually enough to expose flakiness without tripling your bill. If a case passes two of three, that is a real signal about instability, not noise to be smoothed away.

Thresholds and baselines

An absolute threshold (“90% must pass”) stops you shipping something broken. A baseline comparison (“must not drop more than 3 points”) stops you shipping something worse. You want both.

total = sum(r["pass_rate"] for r in results) / len(results)
baseline = json.load(open("evals/baseline.json"))["score"]

print(f"score={total:.3f} baseline={baseline:.3f}")

if total < 0.90:
    raise SystemExit("below absolute threshold")

if total < baseline - 0.03:
    raise SystemExit("regression against baseline")

Update the baseline deliberately, in its own commit, after reviewing the diff. A baseline that updates itself on every run measures nothing.

Wire it into CI

name: evals
on:
  pull_request:
    paths: ["prompts/**", "src/ai/**", "evals/**"]

jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: python evals/run.py
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

The paths filter matters. Running evals on every commit trains people to ignore them; running them when a prompt or model call changes keeps the signal relevant.

Keeping the bill sane

Lever	Effect
Deterministic checks first	Most cases never reach a paid judge
Judge only failures	Rubric calls drop to a handful per run
Cheaper model for the judge	Judging is easier than generating
Cache by input hash	Unchanged cases cost nothing
Full set nightly, subset on PR	Fast feedback, full coverage daily

Forty cases times three runs is a hundred and twenty calls per build. At small-model prices that is cents, not dollars. The cost argument against evals is usually really an argument against running the wrong model.

What to do with a failure

A failing case is a finding, not an obstacle. Triage it in one of three ways:

1. Real regression. Fix the prompt or code. This is the case the harness exists for.
2. Wrong expectation. The product decision changed, so the case is stale. Update it in the same PR and say why.
3. Genuinely ambiguous. The case tests something the feature was never meant to handle. Delete it. A suite with dead weight gets ignored.

Frequently asked questions

Do I need a platform like LangSmith or Braintrust?

Not to start. A JSON file and a CI step cover the regression use case. Platforms earn their cost when you need tracing, human review queues, and dataset curation at team scale.

How many cases are enough?

Thirty to fifty well-chosen cases catch most regressions. A thousand badly chosen ones measure nothing. Start with every bug you have already shipped.

Should the judge be the same model as the feature?

Preferably not. A model judging its own output tends to be generous, and it shares the same blind spots. Use a different model, and spot-check its verdicts by hand.

What about latency and cost regressions?

Record tokens and duration per case alongside the score. A prompt change that doubles cost with no quality gain is a regression even when every case passes.

How do I eval a RAG pipeline?

Split it. Score retrieval separately from generation, otherwise a wrong answer tells you nothing about which half broke. The retrieval side is covered in choosing chunk size with evals.

Does this work for agents and tool-calling?

Yes, with one change: score the trajectory, not only the final message. Which tools were called, in what order, with which arguments. A right answer through a dangerous tool path is a failure. If you are building the tool layer, see building an MCP server in Python.

Sources

Method and code are implementation guidance, not a product recommendation. Scoring approach follows standard practice for pass-rate evaluation under non-determinism. Checked 22 October 2026.

Last updated 22 October 2026.