Two AI agents and one audit trail

The situation

On a financial aid migration, an AI agent worked with me to build the migration engine and to prepare each run. The engine itself is ordinary code: it makes the same decision about the same record every time. What remained was the question every careful person asks about AI-assisted work: who checks it?

The traditional answer is a person doing spot checks. Spot checks find the errors you were already expecting, on the records you happened to open. For federal financial aid data, “we looked at some of them” is not a standard anyone wants to defend.

What I built

A checking setup that puts a second, independent agent between the work and production.

• A separate agent, a separate job. The verifier does not build or migrate anything. Its only task is to establish, record by record, whether the result is correct.
• Checked against the sources. It works from the source systems and the written rules, not from the first agent’s account of what it did.
• Fixed checks, not opinions. The comparisons are written down in advance, so the same record gets the same check on every run. The agent’s part is to run them, read the results, and explain each difference in plain words.
• Disagreement is the point. When the two do not agree, the record is flagged for a person with both findings side by side, and the fix goes back through the build, not straight to production.
• The audit trail is a product, not a by-product. Every check is recorded, so the evidence exists whether or not anyone ever asks for it.

Safeguards

Neither agent applies a change to production: the tools they are given for a live system are read tools. A person reads the evidence and applies the change. The rules that enforce this are part of the tooling, so they do not depend on anyone remembering them.

The verifier is also not the only check. Every production run is checked before and after in three ways: through the system’s own interface, in the database, and by a set of validator reports that financial aid staff can run for themselves inside the student system. The AI verifier sits on top of all three.

The result

AI-assisted work on a compliance-critical migration came with a recorded check for every record in place of spot checks, produced by checks that were fixed in advance.

There is a small recorded example of a two-agent check, scored against an answer key on made-up records, on the approach page.

What I would tell another school

Do not accept “the AI checked it” as an answer. Ask what checked it, against what, and whether you can read the record. If nobody can show you, it was not checked.