We had 10,000 lines of multi-agent code. None of it called a model.
Our app had a planner, scheduler, worker queue and tool registry for teams of agents. An audit found all of it ran as a local simulation. Here is what we kept and what we fixed first.
- Engineering & Data
- Strategy & Operations
- Kind
- Build log
- Min read
- 04
- Sections
- 05
On this page5 sections
The MyPromptVerse composer used to offer a Deep route described as a "Team workflow". Pick it, and the app built a plan with specialists, a critic and a synthesis step. The thread showed workers, an evaluator and a progress line. It looked like several agents had worked on your request.
In October we audited what actually ran. Every real run was one model call, with one operator and one playbook.
What was there
The folder lib/orchestration/ holds 39 files and 10,249 lines (we counted with wc -l). It covers a planner with five modes (single, sequential, parallel specialists, critic, optimisation loop), a graph, a scheduler, run groups, typed step outputs, worker packets and a queue, a tool registry with permission levels and approvals, an event log, transaction and commit layers, and retry and partial-synthesis policies. Separately, there are prompt contracts for synthesis and evaluation.
It is careful work. It is also, at runtime, entirely fake.
What it did
- The executor runs as
local_mockand the worker controller aslocal_mock_worker. - Step outputs are canned text, for example: "This is a local deterministic dry run; no external model or tool execution occurred."
- Its consumers were the browser workbench, the Work Queue and Control Room panels, and the composer's plan preview. No real-provider path imported the executor, scheduler, queue or worker.
- Its one server route wrote to an in-memory store and refused to run outside local development.
- The server accepted a team request, checked that every chosen operator and playbook existed, then used only the first of each.
The plan did reach the model, as prose. The client-built plan was printed into the system prompt, with lines like "N child run(s)" and "Synthesis: Combine specialist work", for a single call that had no children. That costs tokens, and it invites the model to write as if a team had contributed.
Meanwhile a spend check against a child-call cap of 2 was real code that could never trigger, because the real path hard-coded one child call.
Keep the contracts, rebuild the execution
Deleting all of it was tempting. We decided against it because the hard design decisions in that folder are good ones: typed step inputs and outputs instead of raw transcripts, a context slice per step, a partial-synthesis policy for when a step fails, a critic that never reviews its own work, explicit budgets.
The audit sorted the folder into two piles.
Keep: the step types, graph validation and artifact hand-off; the partial-synthesis and retry policies (retries stay at zero); the unused synthesis and evaluator prompt builders, which are the right starting point; and the display helpers.
Retire after the real path ships: the local executor and simulator, the worker packet queue, the transaction log and its API, the in-memory writers, three orchestration tables that nothing writes, and the QA trace fixtures.
There was a second reason to be careful. The safety design around real calls assumes one call per run in many places: database check constraints, unique indexes, fingerprints, the dispatch capability and our audit scripts. That assumption is deliberate and it is why real calls are safe. Multi-call work has to extend it on purpose, one pinned contract at a time.
Honesty fixes first
Before building anything that spends money, we fixed the screens that implied a team had run. That was phase 0, done on 12 October:
| Before | After |
|---|---|
| Backend turns showed "0/N complete · Queued" forever, and "No provider spend" after a paid call | No plan line or drawer on backend turns; simulated threads are labelled "Simulated" |
| Capability previews promised "additional operators join as specialists" | Previews show what runs today, and list the rest under "Not run yet" |
| One-call prompts described child runs and synthesis | That prose is gone; acceptance criteria stay |
| Control Room said "Paid AI: Locked" even with real runs on | Status is read live |
| Automations offered "Run depth: Deep" | Renamed "Task priority", which is all it ever changed |
| The simulator's answer described a completed multi-step plan | It says no model was called |
We also removed a dead end where some Structured messages asked for a Deep confirmation the user could not reach. A new test suite, test:team-honesty, pins all of this with 26 checks.
The Deep route itself stays hidden behind one switch, TEAM_EXECUTION_AVAILABLE, which is set to false.
What we built next, and what we have not
Phase 1 was also done on 12 October. Each run now writes its model calls to a new run_steps table, and Inspect shows them under "Model calls". Today every run has exactly one step. The point is that a single call becomes a plan with one step, so a team becomes the same thing with more rows.
Team runs are not built. The agreed design is one run with up to three specialist steps and one synthesis step, with the whole worst case reserved and shown as a cost range before you confirm. That is phase 2, behind a server flag that defaults to off, and it starts with a live proof the owner runs under a capped budget.
Read next
- 01The Quick button did nothingOur composer offered Quick, Structured and Deep. In a real run, all three changed only the wording of the prompt. This is how we found out and what replaced them.
- 02Every run said 'Passed'. Nothing had been checked.Real runs in our app were stamped 'Passed' by a hard-coded line. Here is how we found it, what the app shows now, and what it still does not check.
- 03Five words, 5,325 tokens: how our app turned 'say hello' into a product specA five-word greeting took about 30 seconds and 5,325 tokens in our closed test. We traced why, and most of the cause was our own defaults.
Newsletter
The next note, by email.
Guides, analysis and build log entries as they are published, plus new Library records. Occasional, practical, easy to leave.