The Receipts Are the Feature: Pi Harness and NIST's Four Functions

This year, the compliance audit asked for our incident response policy. We had one —
the runbook, the severity tiers (sub-1, sub-2, sub-3), the SLAs, the documentation
requirements, the timelines, the retrospectives. “Here it is,” we said. “Here’s the
document.” The auditor read it, looked up, and said, “Great. Show me.”

We gathered something like fifty incidents from the first half of the year and
produced evidence for each one: timelines, documentation, retrospectives — proof it
had followed the policy to the letter. Every incident came with receipts.

For a decade I have built platforms for a bank, a fintech, an observability company,
and a healthcare company — Capital One, Paxos, Splunk, Garner Health. Four
of those years were at Splunk, where I learned what it looks like when a regulated
industry demands to see what its systems actually did. The question that determines
whether anything ships is “show me.” Show me the tests. Show me the run. Show me the
numbers and the exact configuration that produced them. In regulated industries,
“trust me” means you’re asking someone else to own the risk.

So when I built Pi Harness — an
open-source coding agent harness that runs a real agent against a versioned,
reproducible eval contract and produces a pass-rate-and-cost scorecard anyone can
verify — I wasn’t trying to be principled. I was being lazy, the way engineers are
lazy: I didn’t want to re-explain myself, so I made the evidence explain itself.

Then I read NIST’s AI Risk Management Framework (NIST AI 100-1), and I had the
uncomfortable experience of being described by a government document. The framework
organizes AI risk management into four functions — Govern, Map, Measure, Manage — and
each one turned out to be a feature of something I had already built. Not because I’m
clever. Because the framework is a careful description of what it means to be honest
about a system. Let me show you what I mean.

I. Governance Is a Paper Trail

NIST calls this Govern — cross-cutting, because if the paperwork lies, nothing else
can be trusted. My version is smaller and dumber. Every scorecard carries provenance
stamps — dataset version, agent model, judge model, harness version — and the baseline
is committed to the repo. There is a test in CI, test_docs_drift.py, whose entire job
is to fail if the documentation stops matching the code. The project cannot lie about
itself and pass its own pipeline.

If you’ve ever had to show your work to an auditor, none of this needs explaining.

I like that. I like that a lot.

And here is the detail I am prouder of than I should be: fourteen cases in the
baseline are recorded at pass rates below 1.0, and they are published. Not hidden.
Not averaged away. A scorecard that only shows wins is a brochure.

II. Know What You’re Measuring

That’s Map: before you measure, establish context. Define intended use. Categorize the
system. You cannot measure “agent quality” in the abstract. You can only measure this
task, this model tier, this prompt configuration, against this reference. If you
haven’t written down what success looks like, you haven’t decided anything — you’ve
only hoped.

Pi Harness encodes that refusal to skip context in a versioned tasks.json: fifty-five
cases across seven categories, each with a committed reference answer, and zero
unbaselined cases. If we haven’t defined success, we don’t claim to have measured it.

III. TEVV Is the Whole Game

The framework’s word for what I actually built lives in the Measure function:
objective, repeatable, or scalable test, evaluation, verification, and validation
processes — TEVV, if you like acronyms. I built a machine that does TEVV to itself,
every night.

Fifty-five cases, five runs each, judged by a majority-of-three panel, gated against
the baseline with a 0.05 pass-rate tolerance and a flake-aware cost gate. Thirty-eight
keyless hermetic smoke tests, so the harness can verify itself without spending a
cent. Real cost measured from actual session files, not price tables. Seventeen
providers, from OpenAI to local Ollama, with no silent fallback — because a
measurement that swaps models underneath you mid-run is a confession.

Why does this matter? Because the vendors’ evals are welded to their agents. They
measure their model on their terms, and the number is not reproducible by anyone
outside the building. NIST’s insistence on objective, repeatable, documented
measurement is the same instinct regulated industries beat into me: what you cannot
re-run, you can only claim. And claims are what audits eat.

IV. The Loop Closes

Measurement is only useful if it changes what ships. That’s the Manage function:
prioritize, respond, recover, improve.

The scorecard gate blocks regressions in CI — exit code 8, if you care. A budget cap
stops a runaway night before it becomes a line item — exit 6. A watchdog kills hung
runs, SIGTERM then a grace period then SIGKILL, and recovers the working tree, so a
wedged run resolves itself or escalates with evidence instead of quietly corrupting
the baseline — exit 9. And every change to the harness reports its own scorecard
delta. Improve the agent, improve the harness, or watch the number move in the wrong
direction and be asked about it.

The loop: measure, diagnose, improve, re-measure.

Here is the scorecard the README publishes, because the receipts live in the repo — the
2026-08-15 run: 88.7% pass rate, 41 of 55 cases at or above their committed baselines,
$0.72 for the whole night. It’s a dated demo.

The Honest Version

Let me be clear about what I am not claiming. I am not a standards body, and Pi
Harness does not implement NIST AI 100-1. It lives in the space the framework
describes. There is no moat here and no new science — the techniques are commodity, and
every major agent vendor has an internal eval team that could build this in a week.
What they don’t ship is a neutral, cross-provider, verifiable seam for your
configuration. That’s the whole offering: the discipline, made external.

NIST has kept building since AI 100-1 — the Generative AI Profile and the cross-sector
profiles — but this post is about the core framework, because that is where the four
functions live.

I don’t know what NIST does next, but neither does anybody else. We are in a nascent
space with AI, and everyone is moving real fast. Almost no one is slowing down to ask
if we’re doing what we’re supposed to be doing and if it’s safe.

I did not set out to implement a framework. I set out to build a tool I could trust,
and the framework gave me words for what trust requires: accountability, context,
measurement, response. NIST’s seven trustworthy characteristics — valid and reliable,
safe, secure and resilient, accountable and transparent, explainable and interpretable,
privacy-enhanced, fair with harmful bias managed. In a regulated industry, that list is the contract — not a checklist to wallpaper a slide deck.

There is a real business pressure underneath all of this. AI has leveled the playing
field, and when anyone can point a model at your product and build a clone of it end
to end for a couple hundred dollars, the pressure to adopt AI fast is enormous. In an
ordinary market, that risk is worth taking. In healthcare, finance, or the public
sector — anything that handles people’s information — I have a problem with it.
People’s personal information should be treated as sacred. I think we all need to slow
down just a little bit.

Pi Harness is MIT-licensed and public,
and every mechanism in this post is verifiable keyless: run pi-run eval --quick and
watch thirty-eight hermetic tests pass without spending a cent — install instructions
are in the README. The scorecard
numbers above are the README’s dated demo, a snapshot you can re-run against the
committed contract yourself.

If you read to the end of an essay about measurement, you already know why that
matters.

The framework calls it risk management. I call it keeping receipts.