Every run said 'Passed'. Nothing had been checked.

Every real run in the MyPromptVerse app finished with a green "Passed". During an audit of the AI layer in October we went looking for the code that earned it.

This is what we found, on the success path of a real model call:

validationSnapshot: { status: "Passed", issues: [] }

Hard-coded. Alongside it, the run log recorded an event saying the reply had "passed the first-run structural check". No check had run. Any reply that came back without an error got the same badge, whether it followed the playbook or ignored it.

How it got that way

The awkward part is that a real validator already existed. validateRunOutput checks required sections, table and JSON shape, length and missing inputs. It was used by the local mock path and by the evaluations page. The real path never called it.

The real path also had nothing to give it. A playbook declares requiredSections, but the model's reply was stored as one undifferentiated block called "Response", whatever the playbook promised. There were no sections to check against, and the status field was simply set to the value that let the run complete cleanly.

That was a trust problem. The Evaluations and quality views read "Passed" for runs nobody had validated, and nothing on screen said so.

The decision

The owner's answer to "show 'Not checked' instead of an unearned 'Passed'?" was yes, with a known side effect written into the plan: some green labels would turn grey, which is correct and visible.

We split the work in two.

First: stop claiming

On 11 October, real runs started recording "Not checked" instead of "Passed". We made sure "Not checked" could still be saved without a warning, because a plain chat reply has no contract to fail. Backend thread turns also stopped showing "No validation snapshot", which had been the other misleading state.

Then: check what can be checked

On 12 October we wired a reply check into the real path. A finished reply is now compared with its playbook:

• each required section must appear as a heading, with something under it
• a playbook that promises a table must contain something table-shaped
• a playbook that promises JSON must contain JSON that parses (a code fence around it is fine)

To give that check a fair chance, the prompt now asks the model for each required section as a Markdown heading.

If there is no contract to check against, because the run used the General subject or a playbook with no required sections or shape, the status is Not checked. The Run inspector says so in plain words: "This reply had no output contract, so nothing was checked." The save panel reads "This reply had no output contract to check, so it was saved as written."

A correction we made within hours

The first version marked a reply with a missing section as Failed. In our app, Failed outputs cannot be saved to the Library. That broke our own signed-in end-to-end proof, which runs against a fake provider and saves its result.

That made us think about what Failed should mean here. The reply is the model's words and it belongs to the user. Blocking the save because a heading is missing punishes the person for the model's habits. So gaps now count as warnings: the run shows "Warnings", the inspector lists the missing sections, and the user can still save.

The statuses a real run can show today:

What it still does not do

The check reads headings and shapes. It does not judge whether the content under a heading is any good, and nothing scores a reply against the playbook's rubric. The plan also includes one bounded repair attempt when a contract fails and the budget allows. That is not built. A reply with warnings stays as the model wrote it, and the user decides.

The next piece we have planned is an evaluation harness: recorded responses replayed offline, so we can test the parser and the validator against real model output instead of examples we wrote ourselves.