Five words, 5,325 tokens: how our app turned 'say hello' into a product spec

During a closed test of the MyPromptVerse app, we typed "say hello in five words" into a new thread and pressed Send. The reply took about 30 seconds. The provider's generation details said the call had produced 5,325 tokens.

Five words should cost a few dozen tokens. We went looking for where the other five thousand came from.

What the numbers said

All of this comes from one real run, so treat it as an anecdote that started an audit, not a benchmark.

The provider was not slow. 177 tokens a second is normal. The model simply generated a lot, and almost none of it reached the screen. Our adapter read the reasoning token count but nothing in the thread showed it, so we could not say exactly how the 5,325 split between hidden thinking and visible text. The empty reply on the earlier run fits the same picture: a reasoning model that spent its whole 1,000-token allowance thinking and had nothing left to say.

The app picked a job the user never asked for

The thread screenshot showed something we had not chosen. The run had a Product Spec Operator and a Feature Spec Builder playbook attached, plus a "Product Spec PRD" knowledge record.

We never picked them. When nothing was selected, the composer fell back to the first startable playbook in the list, and the server independently fell back to the workspace's first Active operator and playbook. Neither told the user. There was no way to choose "none".

So the model received a greeting wrapped in an instruction to "turn the user's notes into a concise product feature specification", with a list of required inputs. A reasoning model given that brief will think hard about a product spec, even when the notes are five words long.

The prompt was smaller than we expected

Our first draft of the audit blamed prompt size. Then we measured it with the app's own prompt assembler, without calling a model. For the 23-character question, the system prompt was 2,678 characters, roughly 670 tokens. That is wasteful for a greeting, and it carried things that did nothing:

• empty rows such as "Constraints: Not provided"
• the knowledge record listed by name, with none of its content
• an execution contract and a tool policy describing behaviour that did not exist
• the user's text, which was then sent a second time as the user message

But 670 tokens is roughly one-eighth of 5,325. The bulk was the model reasoning, steered by a spec-writing brief, with nothing in our request telling it how hard to think. We had no reasoning-effort setting at all, one global output cap for every run, and a single 60-second timer over the whole call. A model that is thinking quietly and a model that is stuck looked identical.

We corrected the draft before acting on it. If we had trimmed the prompt first, we would have saved about an eighth and still shipped the bug.

What we changed

The owner approved the thread fixes on 11 October. In order of impact:

"None" is now real. An empty selection resolves to a General subject defined on the server, with fixed ids. There is no "first Active record" guess anywhere, in the composer preview, the send path or the server. Keyword suggestions still appear, but only as one-tap suggestions.

A lean prompt for plain asks. General runs, and runs with only an operator or only a playbook, now compile a short system prompt: one rule, the chosen persona or playbook if there is one, and context and history when they exist. No empty rows, no execution or tool boilerplate, and the user's text is sent once.

Timing on every run. We now record queue wait, time to response, time to first words and total time, and show them in Inspect next to the reasoning level.

Reasoning effort and two timeouts. These landed in a later pass, and they get their own post.

What we still don't know

We have one real run of evidence and a set of fixes that should make that run cheap. We have not yet published a like-for-like rerun of the same greeting under the new defaults. That is the next thing we will do with the timing data, and if the numbers disappoint, we will write that up too.