2 · Specification — Prompting as a Deliverable¶

A weak request gets a weak result, and the weakness often stays hidden until the wrong code is already written and under review. The cost lands late, after the agent has spent its run on the wrong target. The problem is that a casual prompt leaves too much unstated: what a finished task looks like, the order of the work, and the parts of the system the agent should leave alone. When those are unstated, the agent fills them in by guessing, and each guess surfaces only during review.

The remedy is to treat the thing you hand the agent as a deliverable in its own right, a specification rather than a wish. A specification does three jobs a casual prompt skips: it states what a finished task looks like, it breaks the work into steps, and it marks the parts the agent must not change. It then asks for a plan before any edit, so a wrong direction can be caught while it is still a paragraph rather than a diff. That is the subject of this module.

A good spec settles three things before the agent starts: what done looks like (acceptance criteria), the steps (decomposition), and what not to touch (constraints). Plan first; edit second.

Concept¶

Four moves turn a prompt into a spec. We take them in the order they belong in the spec itself.

Acceptance criteria — define done. State the checkable condition that means the task is finished: "every classifier response includes a confidence field between 0 and 1, and the existing tests pass." This is the bridge to verification (Module 3): a definition of done that a machine can check is a definition the agent can self-correct toward.
Decomposition — name the steps. Break the work into the two or three moves it actually takes. The agent can decompose on its own, but when you name the steps you remove the most expensive failure, the agent solving a different problem than the one you meant.
Constraints — draw the lines. Say what the agent must not touch: "don't change the label logic; keep prompt text in prompts.py; don't add a dependency." Constraints are where the never-delegate boundary from Module 3 appears at the level of a single task.
Plan first. Ask for a plan and approve it before any edit. A wrong plan costs one paragraph to fix; a wrong implementation costs a full review and a redo. This is also the cheapest lever on the north-star metric: a sharp spec lets the agent run the whole task and return something checkable, instead of stopping to ask what you meant.

Bad / Good / Great — a feature spec¶

Here is the same task in Triage, add a confidence score to the classifier's output, written three ways. The distance between Good and Great is the lesson.

BadGoodGreat

Add a confidence score to the classifier.

Why it fails: no definition of done, no range or format, no location, nothing to check. The agent guesses — at the field name, the file, the scale — and you discover each guess by reviewing wrong code.

Add a `confidence` field (float, 0.0–1.0) to the classifier's JSON output in
app/triage/classify.py.
Done when: every response includes it, values are in range, and `pytest` passes.

Why it's solid: done is checkable and the file is named. The agent can verify itself instead of handing you something to inspect. This is already most of the value.

Goal: add a `confidence` field (float, 0.0–1.0) to the classifier output.

Steps:
  1. Compute confidence in app/triage/classify.py.
  2. Thread it through the response schema.
  3. Add a test asserting the field exists and is in range.

Constraints:
  - Don't change the label logic.
  - Prompt text stays in app/llm/prompts.py.
  - No new dependencies.

Done when: the new test passes, `pytest` is green, and `python -m eval.run`
holds its score.

Show me the plan before you edit anything.

Why it's great: the steps stop it solving the wrong problem, the constraints fence off what matters, verification is baked into "done," and the plan-first line lets you correct course before a single line changes.

Guided Lab¶

You will run the same feature request twice: once at the Good bar, once at the Great bar. The feature is a real build — add a confidence field to the classifier output — and a single test grades whether the field is both present and informative. The point you are measuring is how much reviewing each run costs you.

On the Practice Repo, work through the following.

Clone Triage, build the offline venv, and confirm the suite is green before you start:
```
git clone https://github.com/mrfelixwong/agentic-engineering-triage triage && cd triage
python3 -m venv .venv && .venv/bin/pip install -e . && .venv/bin/pip install -e '.[dev]'
.venv/bin/pytest -q
```
Expected: one test is red by design — test_billing_ticket_is_labeled_billing (that is the Module 3 bug, not this lab); the rest pass. Leave it as is. This lab adds a new field and a new test; it does not touch the routing bug.
The Good attempt. Start the agent in the repo root and give it this request verbatim:
```
claude
```
Add a confidence field (float, 0.0–1.0) to the classifier's output in app/triage/classify.py. Done when: every response includes it, values are in range, and pytest passes.

Expected: the agent names the field but has to guess the rest — where the score is computed, whether the stub or the LLM path produces it, whether Classification in app/models.py needs the field. It edits, you read the diff, and you find at least one guess to correct (a constant 1.0 everywhere, or the field added to classify.py but not to the Classification model). Note how many turns and how many correction rounds that took.
Clear the session so the Good run's context does not bias the next one:
```
/clear
```
The Great attempt. Give the agent this spec verbatim:
Goal: add a confidence field (float, 0.0–1.0) to the classifier output.

Acceptance criteria:
- every Classification includes confidence, with 0.0 <= confidence <= 1.0
- a clearly on-topic ticket (a billing keyword) scores higher than one that falls through to general
- existing tests still pass; the eval holds its 0.80 baseline
Steps:
1. add confidence: float to Classification in app/models.py, constrained 0–1
2. compute it in the offline stub in app/llm/client.py: return (category, confidence)
3. thread it through classify_ticket in app/triage/classify.py
Constraints:
- don't change the routing / _ALIASES logic
- keep it deterministic offline (no API key needed)
Show me the plan before editing anything.
Expected: the agent returns a plan first (the three steps, in order, touching exactly those three files). You approve or correct it as a paragraph, before any edit. It then adds the field to Classification, makes the stub return a higher confidence on a keyword match than on the general fall-through, and threads the value through — without touching _ALIASES.
Grade the feature with the named test. The acceptance criterion "a clear ticket scores higher than an ambiguous one" is the one a machine can check; the answer key supplies that test. Add a new file in the tests/ directory named test_confidence.py with this content:
```
from app.models import Ticket
from app.triage.classify import classify_ticket

def test_confidence_is_in_range_and_informative():
    strong = classify_ticket(Ticket(id="T-1", subject="Refund", body="I was charged twice"))
    weak   = classify_ticket(Ticket(id="T-2", subject="Hi", body="just saying hello"))
    assert 0.0 <= strong.confidence <= 1.0
    assert 0.0 <= weak.confidence <= 1.0
    assert strong.confidence > weak.confidence
```
Then run that one test by name:
```
.venv/bin/pytest -q -k test_confidence_is_in_range_and_informative
```
Expected: 1 passed. The field is in range and informative — a clear billing ticket outscores an ambiguous one. If strong.confidence > weak.confidence fails, the agent returned a constant; that is the acceptance criterion the Good bar left unstated, now caught by a test instead of by your eye. The graded ground truth lives in answer-keys/module-2.md.
Compare the two runs. The Good attempt cost you a review-and-correct round on guesses the request left open. The Great attempt cost you one plan approval up front, then a single test run that either passes or names the exact missing property. That difference — a correction round on a finished diff versus a paragraph approved before any edit — is the spec paying for itself.

On your own codebase, take a task you would normally one-line to an agent and write it as a spec. Write the acceptance criteria before the steps. If you cannot state what done looks like, the task is not yet ready to delegate.

Keep the artifact¶

The artifact is a reusable spec template, the skeleton you paste at the start of every non-trivial task: goal, acceptance criteria, steps, constraints, and the plan-first line. Save it as a snippet or a custom command so a real spec is the default rather than a discipline to remember each time.

Self-check¶

You did it right if the agent's plan matches your intent on the first pass, and the result meets every acceptance criterion you wrote without you adding one mid-stream. If the plan came back aimed at the wrong thing, the gap was in your decomposition or your constraints, so tighten those before reaching for a different model.

Recall¶

Before moving on: what three things must a spec settle before the agent makes its first edit — and why does asking for a plan come before any code?

Answer

Acceptance criteria, decomposition, and constraints. The plan comes first because a wrong direction is a paragraph to fix before an edit and a full redo after one.