Work / 06 / assaygate
The LLM regression a reviewer would approve, blocked in CI.
Every prompt edit or model swap can quietly break an LLM feature. assaygate runs a test suite on each pull request, compares it item by item with the main-branch run, and fails the build only when the statistics say quality, cost or latency really got worse. It also measures how far to trust the LLM judge before it gates anything.
The problem
Once an LLM feature ships, every change is a bet. Teams usually guard it one of two ways. Manual spot checks miss regressions. Score thresholds fire on run-to-run noise, because the same prompt scores a point or two differently each time, so people learn to ignore them and then switch them off.
assaygate treats a change like any other test. A suite (a dataset, the system under test and its scorers) runs on the pull request. The gate pairs each item with the same item from the main-branch run and asks one question: is the drop bigger than the noise? If yes, the job fails and the PR gets a comment that shows exactly which items broke.
Catching a regression that looks like a win
The demo is a Banking77 intent classifier: a small FastAPI app that sends each customer message to Claude Haiku and gets back one of 77 labels. Version 2 of its prompt is a “simplification”. It is shorter, and it cuts cost per item almost in half. A reviewer skimming the diff would likely approve it.
Both versions ran live on the same 300-utterance test sample. The gate also ran v1 against a re-run of itself, to check it stays quiet when nothing changed.
| Comparison | Δ accuracy | Accuracy | Lost / gained | $ / item | p95 ms |
|---|---|---|---|---|---|
| v1 re-run vs v1 · pass | −0.7 pt[−2.3, +1.0] | 0.857 → 0.850 | 4 / 2p = 0.69 | unchanged | 1,148 → 1,051 |
| v2 “simplified prompt” vs v1 · fail | −6.0 pt[−10.0, −2.0] | 0.857 → 0.797 | 28 / 10p = 0.005 | ×0.52 | 1,148 → 1,071 |
Banking77 (CC BY 4.0), seeded 300-utterance test sample, live API runs committed to the repo. Intervals are 95% paired bootstrap; p is McNemar’s exact test on items lost vs gained. Cost per item: $0.000061 for v1 and its re-run, $0.000032 for v2. On the dev split (154 utterances) the same change showed −9.1 pt [−14.3, −3.9]. The three demo runs cost $0.046 in total.
The whole interval for v2 sits below zero, so the gate fails the build. Cost and latency pass, because neither got confidently worse. This is the summary the Action posts on the pull request:
### assaygate: FAILED
candidate-v2 vs baseline-v1 · 300 paired items · 95% bootstrap CIs
accuracy 0.857 → 0.797 -0.060 [-0.100, -0.020] FAIL
28 items lost, 10 gained (McNemar p=0.0051)
cost_per_item_usd $0.00006 → $0.00003 x0.52 [x0.51, x0.53] pass
latency_p95_ms 1,148 → 1,071 ms x0.93 [x0.86, x1.09] pass
How the gate decides
The key choice is to compare items in pairs. Some utterances are hard for every prompt and some are easy. Looking at the per-item difference removes that variation, so the remaining noise is only what the change itself did.
On the unchanged re-run, the paired 95% interval is −2.3 to +1.0 points. An unpaired bootstrap of the same two runs gives −6.3 to +5.0: 3.4 times wider, and wide enough to hide the real −6.0 point regression. A unit test simulates 40 re-runs of an unchanged, non-deterministic system and allows at most 3 false alarms.
Can the judge be trusted?
Many suites score open-ended answers with an LLM judge. A judge that disagrees with people makes the whole gate meaningless, so assaygate measures it first. Each judging technique is a switch. Every variant judges every pair in both orders. The data is split by question: choices were made on 24 dev questions, and the 56 test questions were run once.
The data is the MT-Bench human judgments: 3,355 expert votes turned into 1,814 pairs with a majority label (214 pairs without a majority were dropped). The test sample is 400 pairs (168 A, 153 B, 79 tie). The judge is Claude Haiku 5.5 with thinking off.
| Judge (Haiku 5.5, N = 400) | Agreement | Kappa | Non-tie | Ties | Flips | Longer | $ / pair* |
|---|---|---|---|---|---|---|---|
| 1 naive | 72.8%[68.2, 76.9] | 0.551[0.485, 0.618] | 86.5% | 4.8% | 11.0% | 68.9% | $0.00028 |
| 2 +rubric | 72.2% | 0.542 | 86.2% | 4.5% | 11.0% | 69.9% | $0.00035 |
| 3 +CoT field | 73.8% | 0.567 | 88.3% | 5.2% | 10.3% | 70.7% | $0.00061 |
| 4 +reference | 74.5% | 0.581 | 89.5% | 6.5% | 12.7% | 72.1% | $0.00065 |
| 5 +swap (conservative) | 75.2% | 0.607 | 91.4% | 16.2% | 12.7% | 74.1% | $0.00065 |
| 6 +few-shot (final) | 75.2%[70.8, 79.2] | 0.606[0.543, 0.667] | 90.1% | 15.8% | 11.5% | 73.1% | $0.00053 |
| 6a average-swap policy | 73.5% | 0.570 | 89.8% | 9.5% | 11.5% | 73.1% | $0.00053 |
| 6b no swap | 76.0% | 0.608 | 88.6% | 7.8% | 11.5% | 70.6% | $0.00053 |
| 6c native thinking, no CoT field | 74.0% | 0.586 | 91.0% | 15.8% | 11.0% | 73.8% | $0.00043 |
| Humans (reference) | 64.2%two experts | – | 80.6% | 19.8% | – | 74.0% | – |
Data: lmsys/mt_bench_human_judgments, human split, CC BY 4.0. GPT-4 reference answers from FastChat (Apache-2.0) for math, reasoning and coding. Brackets are 95% CIs (Wilson for agreement, bootstrap for kappa). “Non-tie” is agreement on pairs where neither side said tie; “Longer” is how often the longer answer was picked. *Both orders are counted for every variant; a single-order judge costs half. “Flips” is how often one single call changes its verdict when the answers swap places. The human row is agreement between two individual experts; the labels are majority votes, which are less noisy, so a judge can score above it without being “better than humans”.
The honest result
| Slice (test) | Final − naive | N | Agreement | Kappa |
|---|---|---|---|---|
| All categories · not significant (McNemar p = 0.24) | +2.5 pt[−1.2, +6.2] | 400 | 72.8% → 75.2% | 0.551 → 0.606 |
| Math, reasoning, coding (post hoc) | +12.1 pt[+5.0, +19.3] | 140 | 64.3% → 76.4% | 0.450 → 0.640 |
| Open-ended categories (post hoc) | −2.7 pt[−6.9, +1.5] | 260 | 77.3% → 74.6% | – |
The category split is where the stack earns its keep. On verifiable tasks (math, reasoning, coding), agreement rose from 64.3% to 76.4%, and the direction matches dev (+26.9 pt on 52 pairs). On open-ended writing it did nothing. I chose this split after seeing the test results, so it should be confirmed on fresh data. The practical advice still holds: choose judge settings by task type rather than one judge for everything.
- Position bias. A single judge call flips its verdict on about 11% of pairs when the two answers swap places. Judging both orders and combining them makes the final verdict independent of order. The conservative policy (orders disagree → tie) beat averaging on both splits (+4.2 pt dev, +1.7 pt test) and moved the judge’s tie rate from 4.8% to 15.8%, closer to the humans’ 19.8%.
- Verbosity bias. None found. Haiku picked the longer answer 68.9% of the time with the naive prompt; the humans did 74.0%.
- Bigger model. On the first 100 test pairs, Sonnet 5.5 cost about 20 times more per pair with no clear gain in agreement (naive 71.0% vs Haiku’s 73.0%, final 73.0% vs 69.0%; intervals about ±9 pt). Its single calls were far more stable under swapping: 3% flips vs 12%.
Safe to run on every pull request
A hard spend cap under concurrency
Before each model call, the client reserves the worst-case cost (prompt estimate plus max tokens) in a shared ledger under a lock, then settles to the real cost from the response, cache tokens included. A test with 20 concurrent calls against a tiny cap never overshoots. The whole project made 7,839 billed model calls and spent $3.16 of its $5.00 cap: $1.14 on the MT-Bench test ladder, $1.63 on the Sonnet comparison, $0.05 on the Banking77 runs, the rest on dev work and smoke tests. A sqlite cache served 2,412 of 7,200 judge calls on the test run.
A shared API key in the real world
The first full calibration run died on a 429: other workloads on the same organisation key had used up the concurrency limit. The client now caps in-flight calls and backs off with jitter, honouring Retry-After. A structured output cut off by the token limit is re-asked once with double the limit, instead of repeating the same failing request. A judge call that still fails counts as a tie and is reported (zero in the final test run) rather than crashing a 4,800-call run. Sonnet 5.5 also rejects the “thinking off” setting Haiku accepts, so that is now mapped per model, with a test.
Quality
- 68 tests, all offline with a fake SDK. The statistics are checked against scipy, scikit-learn and textbook values: bootstrap percentile and BCa, permutation, McNemar, Wilson, kappa.
- CI green on Python 3.12 and 3.13. A gate-demo workflow runs the Action itself: it passes the re-run and fails the regression.
- One Docker image; Compose brings up the report service and the demo classifier.
- Five architecture decision records, including why the gate uses paired intervals and why the client uses the Anthropic SDK directly.
Limitations
- One public judge dataset. MT-Bench is 2023-era chat answers from a small expert pool. Trusting a judge in your own domain needs your own labels;
assaygate calibrateis built for that. - The overall judge improvement is not significant at N = 400, and the category result was chosen after seeing test.
- The demo regression is a deliberately written prompt change, run live. Banking77 labels contain some noise.
- Demo latency includes queueing on a shared key, so the p95 intervals are wide. No latency regression was shown; only quality failed.
- One provider (Anthropic) is wired into the judge client. Installs from GitHub, not PyPI.
Stack
Fits more than classifiers
The same setup gates any LLM feature with a test set: RAG answers, extraction, agents, support replies. Paired statistics so noise doesn’t block merges, and a judge you have measured before you let it decide.