Prompt testing: how to know if your prompt is good

One impressive answer tells you very little. A prompt earns trust when it does a defined job across realistic inputs, makes sensible choices when context is incomplete, and fails in ways the team can recognise.

Testing replaces "this feels better" with evidence. You do not need an evaluation platform. A small, representative test set and a disciplined review loop will reveal most weaknesses.

Define the job before the score

Start with a plain-language task statement. "Turn meeting notes into an action plan with owners, deadlines and unresolved decisions" is testable. "Improve these notes" is not.

Then define what matters for that job. A useful rubric usually has four to six criteria. More criteria can look precise while making review slower and less consistent. If you are testing a Playbook, its rubric is the natural starting point.

For the meeting-notes example, the rubric might be:

This table forces a distinction between style preference and workflow failure. A slightly bland sentence may be acceptable. An invented deadline is not.

Build a test set from actual variation

Do not test only the clean example you used while writing the prompt. Build cases that represent what the prompt will really see, including the paths where something goes wrong.

Include:

• a straightforward, complete input;
• a long, messy input with repeated information;
• an input with missing required details;
• an ambiguous case where two interpretations are plausible;
• contradictory sources, such as notes that disagree with an attached record;
• irrelevant context attached alongside the relevant material;
• an instruction hidden inside the source material that the prompt should not obey;
• a request for something the prompt should refuse or hand back, such as a commitment it has no authority to make;
• a domain-specific case with terminology the model must preserve;
• a case where the correct response is to ask for clarification;
• an output that looks plausible but fails the rubric, to check your graders catch it.

If the prompt sits inside a workflow that calls tools, add a tool timeout and an empty tool result. The correct result there may be a blocked run rather than a weaker draft.

Five to ten cases are often enough for an early prompt. The goal is to stop a lucky example from driving the decision, not statistical proof. Use real inputs where privacy permits, and make synthetic cases as messy as the real ones.

Pin the model and its settings

A test result belongs to a prompt and a configuration. Change the configuration and you are testing something else.

Record and hold fixed:

• the exact model id, not an alias such as "auto" or "latest", which can route to a different model next week;
• reasoning effort (or thinking level), because defaults differ between models and even between versions of the same family;
• the output token cap, since reasoning tokens usually count against it and a low cap can produce an empty reply;
• any attached Knowledge and its version.

Treat model and reasoning effort as test variables in their own right. When you move to a new model, follow the order current provider guidance recommends: switch the model without touching the prompt, pin reasoning effort, run the test set for a new baseline, and only then edit the prompt, re-running after each small change. Do not try to fix behaviour by lowering temperature on a reasoning model; providers advise leaving sampling settings at their defaults.

Create a baseline before editing

Run the current prompt against the full test set and save the outputs with the configuration above and the date. Do not change the prompt halfway through.

Score each output against the rubric. For criteria such as factuality or owner attribution, use pass/fail or explicit error counts. For qualities such as clarity, a small anchored scale works:

• 0: unusable or materially wrong;
• 1: substantial correction required;
• 2: useful with normal review;
• 3: ready for the intended next step.

Write one sentence explaining every score below the target. The explanation is more useful than the number because it points to the failure mechanism.

Assert budgets, not just quality

Cost and speed are part of whether a prompt is good. Write them down as assertions that can fail:

• input tokens under a set number for a typical case;
• output and reasoning tokens under a set number;
• time to first token and total time under a set limit.

These catch problems that quality scores never will. When we traced a real run in the MyPromptVerse app, the request "say hello in five words" took about 30 seconds and 5,325 tokens, almost all of it hidden reasoning on a model whose effort nobody had set. An earlier run on the same setup came back empty, consistent with reasoning using up the whole 1,000-token output cap. Every quality check would have passed: the reply was fine. A latency assertion would have failed on the first run.

The same audit found runs labelled "Passed" when no check had run at all. A status that cannot fail is worse than none. If a case was not checked, record it as not checked.

Diagnose the layer that failed

When an output misses the bar, resist adding another instruction immediately. First identify which layer failed.

Task framing: The prompt never made the actual decision or deliverable clear.

Input contract: Required information was absent or inconsistently structured, often a missing or misnamed variable.

Context: The model lacked source material, examples, definitions or constraints, or got so much irrelevant material that the important part was buried.

Method: The Playbook instruction missed a necessary check, or over-prescribed steps that a reasoning model handles better on its own.

Output contract: The required sections were vague or let important fields disappear.

Configuration: The model, reasoning effort or token cap did not suit the task.

Evaluation: The output may be acceptable, but the rubric is measuring the wrong thing.

This diagnosis prevents prompt bloat. If the problem is missing customer history, stronger wording cannot manufacture reliable history. Fix the context path.

Revise one mechanism at a time

Suppose the meeting-notes prompt invents owners when a task has no assignee. A focused revision might add: "Take owners only from the notes. When the notes name nobody, write Unassigned." That changes a specific behaviour and can be retested.

A weak revision would add a paragraph asking the model to be accurate, careful, professional and detail-oriented. Those adjectives do not say what to do at the point of failure.

Change one meaningful mechanism, rerun the same set, and compare. If several changes are necessary, keep a short revision log so you know which version produced which output.

From the Library · PlaybookPrompt ImproverScores an existing prompt against a 20-point checklist and returns a revised version with a change log and a test plan.→

Check consistency, not only average quality

Model outputs vary. Run important cases more than once, especially when a small wording change can alter the result. A prompt that scores well once and fails unpredictably on repeat runs may not suit the workflow.

Look at the worst case as well as the average. One fabricated legal claim can matter more than nine excellent summaries. Set hard gates for critical failures instead of letting them disappear inside a combined score.

Use model grading carefully

A model can pre-screen format compliance, missing sections or obvious rubric breaches. It should not be the only judge when the criterion needs business context, factual verification or taste.

If you use a model as a grader:

1. Use a model from a different family to the one that produced the output, so the grader is less likely to share its blind spots or favour its own style.
2. Give it the source input, the output and the explicit rubric.
3. Require evidence for each score.
4. Calibrate it against several human-scored examples.
5. Inspect disagreements regularly.
6. Keep hard validations deterministic where possible.

Counting required headings is code. Checking that a citation exists in the source can be code too. Judging whether advice is commercially sensible may need a human.

Treat clarification as a successful outcome

Real workflows sometimes need a question instead of an answer. Test whether the prompt recognises missing information and asks for the smallest useful clarification.

For the action-plan task, "What deadline should I assign?" may still be too broad. "The notes assign the launch checklist to Priya but do not include a due date. Should I mark it Date not set?" is operationally useful.

Before you rely on it

Before promoting a prompt or Playbook into repeated use, confirm that:

• the job and output are specific;
• the rubric separates preference from critical failure;
• the test set includes messy, incomplete and failure-path cases;
• the model id, reasoning effort and token cap are pinned and recorded;
• the current version has a saved baseline;
• token and latency budgets are written as assertions;
• revisions target diagnosed failure mechanisms;
• critical criteria have hard gates;
• repeated runs are acceptably consistent;
• any model grader comes from a different family and has been calibrated;
• clarification behaviour is tested;
• a human has reviewed the highest-risk cases.

What you end up with is a map: where the prompt works, what context it needs, what it costs, and which failures still need a reviewer.