Prompt Testing: How to Know If Your Prompt Is Good

A prompt is not good because it produced one impressive answer. It is good when it performs a defined job across realistic inputs, makes sensible choices when context is incomplete, and fails in ways the team can recognise.

Testing replaces “this feels better” with evidence. It does not need a large evaluation platform. A small, representative test set and a disciplined review loop are enough to reveal most prompt weaknesses.

Define the job before the score

Start with a plain-language task statement. “Turn meeting notes into an action plan with owners, deadlines, and unresolved decisions” is testable. “Improve these notes” is not.

Then define what matters for that job. A useful rubric usually has four to six criteria. More criteria can create the appearance of precision while making review slower and less consistent.

For the meeting-notes example, the rubric might be:

| Criterion | Pass condition | Serious failure | | --- | --- | --- | | Coverage | Every explicit action is captured | A committed action is omitted | | Attribution | Owners match the source notes | An owner is invented | | Dates | Dates are copied or clearly marked unknown | A deadline is fabricated | | Decisions | Confirmed decisions are separate from open questions | Discussion is presented as agreement | | Format | Output follows the requested sections | Key information is buried in prose |

This table forces a distinction between style preference and workflow failure. A slightly bland sentence may be acceptable. An invented deadline is not.

Build a test set from actual variation

Do not test only the clean example used while writing the prompt. Build cases that represent the distribution the prompt will see.

Include:

• a straightforward complete input;
• a long, messy input with repeated information;
• an input with missing required details;
• an ambiguous case where two interpretations are plausible;
• an adversarial or irrelevant instruction inside the source material;
• a domain-specific case with terminology the model must preserve;
• a case where the correct response is to ask for clarification.

Five to ten cases are often enough for an early prompt. The goal is not statistical proof. It is to stop a lucky example from driving the decision.

Use real inputs where privacy permits. Synthetic cases should model real messiness rather than read like textbook exercises.

Create a baseline before editing

Run the current prompt against the full test set and save the outputs. Do not change the prompt halfway through. Record model, settings, date, and any attached context so the run can be reproduced.

Score each output against the rubric. For criteria such as factuality or owner attribution, use pass/fail or explicit error counts. For qualities such as clarity, a small anchored scale works:

• 0: unusable or materially wrong;
• 1: substantial correction required;
• 2: useful with normal review;
• 3: ready for the intended next step.

Write one sentence explaining every score below the target. The explanation is more useful than the number because it points to the failure mechanism.

Diagnose the layer that failed

When an output misses the bar, resist adding another instruction immediately. First identify which layer failed.

Task framing: The prompt never made the actual decision or deliverable clear.

Input contract: Required information was absent or inconsistently structured.

Context: The model lacked source material, examples, definitions, or constraints.

Method: The prompt skipped a necessary reasoning or checking step.

Output contract: The requested structure was vague or allowed important fields to disappear.

Evaluation: The output may be acceptable, but the rubric is measuring the wrong thing.

This diagnosis prevents prompt bloat. If the problem is missing customer history, stronger wording cannot manufacture reliable history. Fix the context path.

Revise one mechanism at a time

Suppose the meeting-notes prompt invents owners when a task has no assignee. A focused revision might add: “Never infer an owner. Use Unassigned when the source does not name one.” That changes a specific behaviour and can be retested.

A weak revision would add a paragraph asking the model to be accurate, careful, professional, thoughtful, and detail-oriented. Those adjectives do not define what to do at the point of failure.

Change one meaningful mechanism, rerun the same set, and compare. If several changes are necessary, keep a short revision log so you know which version produced which output.

Check consistency, not only average quality

Model outputs vary. Run important cases more than once, especially when a small change in wording can alter the result. A prompt that scores well once and fails unpredictably on repeat runs may not be suitable for the workflow.

Look at the worst case as well as the average. One fabricated legal claim can matter more than nine excellent summaries. Set non-negotiable gates for critical failures instead of allowing them to disappear inside a combined score.

Use model grading carefully

A model can help pre-screen format compliance, missing sections, or obvious rubric breaches. It should not be the only judge when the criterion requires business context, factual verification, or taste.

If using a model as a grader:

1. Give it the source input, output, and explicit rubric.
2. Require evidence for each score.
3. Calibrate it against several human-scored examples.
4. Periodically inspect disagreements.
5. Keep hard validations deterministic where possible.

Counting required headings is code. Verifying a source citation may use retrieval. Judging whether advice is commercially sensible may need a human.

Treat clarification as a successful outcome

Prompts are often rewarded for always producing an answer. Real workflows sometimes need a question instead.

Test whether the prompt recognises missing information and asks for the smallest useful clarification. For the action-plan task, “What deadline should I assign?” may still be too broad. “The notes assign the launch checklist to Priya but do not include a due date. Should I mark it Date not set?” is operationally useful.

A practical release checklist

Before promoting a prompt into repeated use, confirm that:

• the job and output are specific;
• the rubric separates preference from critical failure;
• the test set includes messy and incomplete cases;
• the current version has a saved baseline;
• revisions target diagnosed failure mechanisms;
• critical criteria have hard gates;
• repeated runs are acceptably consistent;
• clarification behaviour is tested;
• a human has reviewed the highest-risk cases.

Prompt testing is not about proving a prompt is universally good. It establishes where the prompt is useful, what context it requires, and which failures still need review. That is enough to turn a promising instruction into a controlled working pattern.