The Receipts Are the Feature: Pi Harness and NIST's Four Functions

For a decade I have built platforms for a bank, a fintech, and a healthcare company. Different uniforms, same exam. The question that determines whether anything ships is rarely “does it work?” It’s “show me.” Show me the tests. Show me the run. Show me the numbers and the exact configuration that produced them. In regulated industries, “trust me” is not a control. It’s a risk you’re asking someone else to own.

So when I built Pi Harness — an open-source coding agent harness that runs a real agent against a versioned, reproducible eval contract and produces a pass-rate-and-cost scorecard anyone can verify — I wasn’t trying to be principled. I was being lazy, the way engineers are lazy: I didn’t want to re-explain myself, so I made the evidence explain itself.

Then I read NIST’s AI Risk Management Framework (NIST AI 100-1), and I had the uncomfortable experience of being described by a government document. The framework organizes AI risk management into four functions — Govern, Map, Measure, Manage — and each one turned out to be a feature of something I had already built. Not because I’m clever. Because the framework is a careful description of what it means to be honest about a system. Let me show you what I mean.

I. Governance Is a Paper Trail

NIST puts Govern first and calls it cross-cutting, which is a polite way of saying: if the paperwork lies, nothing else can be trusted. My version is smaller and dumber. Every scorecard carries provenance stamps — dataset version, agent model, judge model, harness version — and the baseline is committed to the repo. There is a test in CI, test_docs_drift.py, whose entire job is to fail if the documentation stops matching the code. The project cannot lie about itself and pass its own pipeline.

I like that. I like that a lot.

And here is the detail I am prouder of than I should be: fourteen cases in the baseline are recorded at pass rates below 1.0, and they are published. Not hidden. Not averaged away. A scorecard that only shows wins is not a scorecard. It’s a brochure.

II. Know What You’re Measuring

Before you measure, NIST says, establish context. Define intended use. Categorize the system. You cannot measure “agent quality” in the abstract. You can only measure this task, this model tier, this prompt configuration, against this reference. If you haven’t written down what success looks like, you haven’t decided anything — you’ve only hoped.

Pi Harness encodes that refusal to skip context in a versioned tasks.json: fifty-five cases across seven categories, each with a committed reference answer, and zero unbaselined cases — nothing gets measured against a benchmark that doesn’t exist. If we haven’t defined success, we don’t claim to have measured it. That sentence does more work than it looks like.

III. TEVV Is the Whole Game

The framework’s word for what I actually built lives in the Measure function: objective, repeatable, or scalable test, evaluation, verification, and validation processes — TEVV, if you like acronyms. I built a machine that does TEVV to itself, every night.

Fifty-five cases, five runs each, judged by a majority-of-three panel, gated against the committed baseline with a 0.05 pass-rate tolerance and a flake-aware cost gate. Thirty-eight keyless hermetic smoke tests, so the harness can verify itself without spending a cent. Real cost measured from actual session files, not price tables. Seventeen providers, from OpenAI to local Ollama, with no silent fallback — because a measurement that swaps models underneath you mid-run is not a measurement. It’s a confession.

Why does this matter? Because the vendors’ evals are welded to their agents. They measure their model on their terms, and the number is not reproducible by anyone outside the building. NIST’s insistence on objective, repeatable, documented measurement is the same instinct regulated industries beat into me: if you cannot re-run it, it is not a measurement. It is a claim. And claims are what audits eat.

IV. The Loop Closes

Measurement is only useful if it changes what ships. That’s the Manage function: prioritize, respond, recover, improve.

The scorecard gate blocks regressions in CI — exit code 8, if you care. A budget cap stops a runaway night before it becomes a line item — exit 6. A watchdog kills hung runs, SIGTERM then a grace period then SIGKILL, and recovers the working tree, so a wedged run resolves itself or escalates with evidence instead of quietly corrupting the baseline — exit 9. And every change to the harness reports its own scorecard delta. Improve the agent, improve the harness, or watch the number move in the wrong direction and be asked about it.

The loop, in four words: measure, diagnose, improve, re-measure.

Here is the latest committed scorecard, because the receipts live in the repo: 88.7% pass rate across 41 of 55 cases, at $0.72 for the whole night, on the 2026-08-15 run. It’s a dated demo. The mechanism is the promise.

The Honest Version

Let me be clear about what I am not claiming. I am not a standards body, and Pi Harness does not implement NIST AI 100-1. It lives in the space the framework describes. There is no moat here and no new science — the techniques are commodity, and every major agent vendor has an internal eval team that could build this in a week. What they don’t ship is a neutral, cross-provider, verifiable seam for your configuration. That’s the whole offering: the discipline, made external.

I did not set out to implement a framework. I set out to build a tool I could trust, and the framework gave me words for what trust requires: accountability, context, measurement, response. NIST’s seven trustworthy characteristics — valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, fair with harmful bias managed — are not a checklist to wallpaper a slide deck. In a regulated industry, they’re the contract.

Pi Harness is MIT-licensed and public at github.com/forrestbthomas/pi-harness, and every claim in this post is verifiable by running the commands yourself. If you read to the end of an essay about measurement, you already know why that matters.

The framework calls it risk management. I call it keeping receipts.