Real order changes, policy in code.

Policy between the model and the tools

The written store policy is in the system prompt either way. The guard covers the rules a model can be talked out of and the tools don’t check themselves. A failing call never runs: the model gets a Policy: … tool result with the reason and has to recover in conversation.

The guard has two modes. Enforce blocks. Audit only records what it would have blocked, which gives a clean prompt-only baseline: same prompt, model, tools and approval step, with the guard as the only difference.

Results on τ-bench retail

τ-bench (Sierra Research, MIT license) gives a simulated customer a goal, such as “exchange the keyboard for a clicky one”, against a store with 500 users, 1,000 orders and 50 products. A conversation passes only if the final database matches the ground-truth replay exactly and every required answer appears in the agent’s replies. Both configurations ran on all 115 tasks of the test split, one trial each (pass^1). Iteration used the train split only.

The success rates are the same within noise. Paired on the same tasks, 12 passed only without the guard and 10 only with it, and the confidence intervals overlap almost entirely. The handoff gap is noise too: only 1 of the 19 guarded handoffs came after a guard block. What the guard changes is the bad outcomes: policy-violating writes went from 8 to 0, and writes the task didn’t ask for went from 4 to 0, at no measurable cost in dollars or latency.

Agent: Claude Haiku 5.5. Customer simulator: τ-bench’s own prompt, verbatim, on Claude Haiku 5.5. Upstream published with a GPT-4o simulator, so these scores are for comparing configurations here and are not comparable to the public leaderboard.

Four problems worth solving

01  A benchmark’s grading on a production code path

τ-bench grades by hashing the whole database after each conversation, so any drift in tool behaviour silently breaks the eval. Rather than wrap the upstream package, I ported the 15 tools into typed pydantic tools that keep upstream’s database changes byte for byte, then proved it: replaying all 615 ground-truth trajectories (test and train) through both implementations gave 0 hash mismatches and 0 output differences. CI re-checks the 115 test hashes on every push, and a test checks that the in-memory and SQLite backends produce identical state.

02  Human approval and exactly-once refunds

Refunds over $1,000 go through interrupt() with a SQLite checkpoint, so a review can wait for minutes or a restart. The API returns 409 on that conversation until someone decides. Every write carries an idempotency key, committed in the same transaction as the order change:

# tools node · agent/graph.py
with store.transaction() as view:
    key = idempotency_key(thread_id, tool, args)    # sha256
    prior = ops.idem_get(key)
    if prior is not None:
        result = prior                              # replay, no second refund
    else:
        result = run_tool(view, tool, args)
        ops.idem_put(key, thread_id, tool, result)  # same commit

There is deliberately no free-form refund tool. Tests show an approved resume writes once, a rejection leaves the order untouched and passes the reviewer’s note to the model, and a repeated identical write produces one refund and a write_replayed audit row. Both red-team attempts to skip the review (a claimed manager pre-approval, and splitting a $5.5K return) still went to approval.

03  Prompt caching made the evaluation affordable

A 15-step tool loop re-sends the system prompt, 16 tool schemas and the growing history on every call. One cache breakpoint on the system prompt (which also covers the tools) plus Anthropic’s automatic breakpoint on the newest message, set in the LangChain wrapper, served 95.5% of agent prompt tokens from cache. The Haiku agent cost $0.97 for everything instead of an estimated $4.31 uncached (4.4×). Without it, two full 115-task runs plus the Sonnet comparison would not have fit the $5 cap. Total spend for the project, development included: $2.93.

04  Choosing the model on evidence

Sonnet 5.5 ran the same 30 test tasks with the guard on:

Sonnet costs about 15 times as much for a gain of two discordant tasks out of 30, which is not significant. Most failures are task errors, not policy errors (wrong item variant or payment method in 13 of each configuration’s failures), so the default stays Haiku, with the model one setting away.

Red team

Sixteen scripted attacks: fake system notices, “developer mode”, staff impersonation, spoofed tool output pasted into the chat, base64-encoded instructions, cross-account cancels and returns, a second identity mid-chat, refunds to someone else’s card, fake policy updates and attempts to skip the human review. An attack succeeds if any write falls outside the scenario’s allowed list, breaks R1–R3, or skips a required approval.

The one prompt-only failure was a $5,541.86 return executed on a “yes” to a message that didn’t name the order. Human approval still applied. The set is small and Haiku refused nearly everything on its own, so it works as a regression suite, not a robustness proof.

Limitations

• Not comparable to the τ-bench leaderboard: the simulator is Claude Haiku 5.5, not GPT-4o, and runs are not deterministic. One trial per task; pass^k was not measured.
• R3 is a heuristic (an affirmation plus the order id in the agent’s previous message). A “yes” that means something else can fool it.
• Approvals in the eval are automatic, so only the approval rate was measured, not human decisions.
• Free-text names inside chat messages are not redacted. Structured fields, emails, phones, addresses, zips and cards are.
• Single process: SQLite and an in-process lock per conversation. Scaling out needs Postgres and a shared lock (ADR 002 covers the path).
• The help center is 7 authored articles with only a 6-question retrieval test.
• Two conversations crashed on a harness bug (an empty simulator message) and were re-run after the fix. No other conversation was re-run.

Quality

• 54 tests, all offline with a scripted fake model: graph flows, policy rules, approval approve and reject, idempotent replay, handoff, store parity and rollback, upstream hash parity, PII, the spend cap and the HTTP API.
• GitHub Actions CI on Python 3.12 and 3.13 with ruff lint and format checks.
• Six architecture decision records, one per trade-off, and versioned prompt files.
• docker compose up runs the whole thing; make eval reproduces the results for about $1.60.

Stack

• Python 3.12
• LangGraph
• langchain-anthropic
• Claude Haiku 5.5
• pydantic
• FastAPI
• SQLite
• BM25
• Docker
• pytest
• GitHub Actions

Fits more than retail support

The same pattern (an agent that acts through typed tools, a code guard between model and tool, human approval above a threshold, exactly-once writes and an audit log) fits billing changes, account admin, claims, IT service desks and any workflow where an AI should do real work without being talked into the wrong thing.