Skip to content

4 · Shaping the Agent Harness — Tools, Skills, and Permissions

The model is only one part of an agent. The useful work happens through the harness around it: the files and instructions it can see, the tools it can call, the permissions it has, the workflows it can reuse, and the checks that stop or steer the loop. When an agent stalls, the fix is often a harness change rather than a longer prompt.

This module teaches the operator-facing version of harness work. You are shaping the harness your coding agent already runs inside: Claude Code, Cursor, Codex, or another tool. You are deciding what capabilities to expose, what workflows to package, what permissions to grant, and what verification path must run before the agent's work counts as done.

The governing rule is to extend the harness only when the agent repeatedly needs a capability or workflow it lacks. A skill that captures a repeated verify sequence earns its keep. A tool wired in for one imagined future task adds maintenance, trust risk, and one more way for the loop to go wrong.

Concept

  1. Prompt, context, harness. These three were named in the agent loop; this module is the harness in depth. A prompt is what you ask in this turn. Context is what the agent can see. Harness is the operating environment that lets the agent act: tools, permissions, reusable workflows, external systems, and checks. The prompt points the agent; the harness determines what the loop can actually do.

  2. Tools — the agent's hands. A tool is a capability the agent can call and get a result back from: running tests, reading a file, searching a codebase, opening a browser, or hitting an API. Tool calling turns a planner into an actor. Each tool also expands the blast radius, so expose the smallest capability that lets the task close its loop.

  3. MCP — one plug for external systems. The Model Context Protocol is a standard interface for connecting an agent to systems it did not ship with: an issue tracker, a database, a browser, logs, or internal docs. The value of a standard plug is reuse across compatible agents. The cost is trust: every external system can return stale, irrelevant, or adversarial text. Treat tool output as data the agent must inspect before it acts.

  4. Skills — packaged workflows. A skill is a repeated operator move written down so the agent can run it in one invocation. Good skills package more than commands; they encode the decision at the end. For example, "run the eval" is a command wrapper. "Run tests, run the eval, compare to the baseline, and refuse to pass if the score regresses" is a harness improvement.

    Skills are also the right home for context that matters, but not on every task. Keep the always-needed map in CLAUDE.md; move the repeated, occasional recipe into a skill so it is loaded when the workflow calls for it.

    A Claude Code skill is a Markdown file the agent reads on demand. It lives at .claude/skills/<name>/SKILL.md, where <name> is the skill's name. The file opens with front matter: name, a one-line description, and user_invocable: true. The body gives the steps, commands, and final decision rule. The description matters because it is how the agent decides when the skill applies.

  5. Permissions and boundaries. A harness also includes what the agent is allowed to do without asking. Safe permission is task-specific: read-only exploration can be broad; writes, network calls, deploys, database access, and destructive commands need tighter boundaries. A broader harness is useful only when the verification path is strong enough to catch the mistakes it makes possible.

  6. Harness diagnosis. When the agent fails, diagnose the harness before adding words to the prompt:

    Failure Harness question
    It guesses where code lives. Did the context map point it to the right files?
    It cannot check its work. Does it have the test, eval, or command it needs?
    It repeats a manual ritual. Should that ritual become a skill?
    It needs data outside the repo. Is an MCP server or other tool justified by repeated use?
    It is touching too much. Are permissions and guardrails too broad for this task?

Bad / Good / Great — a harness change

The same Triage workflow can be handled three ways. The gap is in how much of the harness is shaped and how much judgment it encodes.

Add a browser tool, issue-tracker MCP server, and database connector because they might be useful later.
Keep the verify workflow in your head.

Why it fails: the harness got broader without a repeated need or a stronger check. The agent gained new ways to act, but the recurring verification decision still depends on you.

Skill "run-eval": run `python -m eval.run` and print the score.

Why it is solid: the repeated command is now one invocation. The harness saves typing, but the pass/fail judgment still sits outside the skill.

Skill "check-change":
  1. Run `pytest`.
  2. Run `python -m eval.run`.
  3. Confirm `eval.run` holds the committed baseline.
  4. If the score dropped, print the regressed categories and stop.
  5. If tests and eval pass, say the change is ready to review.

Why it is great: the harness now contains the workflow and the decision rule. The agent can run farther because the check that used to live in your head is part of the environment it operates inside.

Guided Lab

Triage seeds a workflow you run by hand after every change to the classifier: run the tests, run the eval, and check the score against the committed baseline. You will map the current harness, capture that repeated workflow as a skill, and make the skill encode the pass/fail call. The graded answer is in answer-keys/module-4.md — write your own before you open it.

On the Practice Repo, work through the following.

  1. Clone Triage, build the offline venv, and run the verification workflow by hand once:

    git clone https://github.com/mrfelixwong/agentic-engineering-triage triage && cd triage
python3 -m venv .venv && .venv/bin/pip install -e . && .venv/bin/pip install -e '.[dev]'
.venv/bin/pytest -q
.venv/bin/python -m eval.run

    Expected: tests pass and the eval prints Triage eval: 10/10 correct = 1.00 (baseline 0.80) followed by PASS. That sequence is the workflow the harness should make repeatable.

  2. Write a harness map before adding anything. Create toolkit/harness-map.md in the cloned repo:

    mkdir -p toolkit
printf '%s\n' '# Harness map' '' '- Context: CLAUDE.md and the classifier files it points to.' '- Tools: shell commands for pytest and eval.run.' '- Skill to add: check-change, because verification repeats after every classifier change.' '- Permission boundary: no deploys, database writes, or network calls for this task.' '- Check: tests pass and eval holds the baseline.' > toolkit/harness-map.md

    Expected: toolkit/harness-map.md names what the agent can see, what it can call, what workflow should become reusable, and what it may not do for this task.

  3. Capture the workflow as a skill. Create .claude/skills/check-change/SKILL.md:

    mkdir -p .claude/skills/check-change
cat > .claude/skills/check-change/SKILL.md <<'EOF'
---
name: check-change
description: After any change to Triage, verify it by running tests and eval, then fail if the eval drops below the baseline.
user_invocable: true
---

Run, in order, and report the result of each:

1. `.venv/bin/pytest -q` — every test must pass. If any test fails, stop and show the failure.
2. `.venv/bin/python -m eval.run` — report the score and whether it holds the committed baseline.

Conclude "ready to review" only if tests pass and the eval holds. Otherwise, say exactly what regressed.
EOF

    Expected: the file exists at .claude/skills/check-change/SKILL.md. It encodes the commands and the final decision.

  4. Invoke the skill on the clean repo. Start the agent in the repo root and run:

    claude

    /check-change

    Expected: the agent runs .venv/bin/pytest -q and .venv/bin/python -m eval.run, reports the eval at 10/10 correct = 1.00 against the 0.80 baseline, and concludes "ready to review." The harness now contains the repeated verification workflow.

  5. Break the classifier on purpose, then invoke the skill again to confirm it refuses to bless the change. Edit a keyword rule in app/llm/client.py so a category is misrouted, then in the session run:

    /check-change

    Expected: the eval drops below the 0.80 baseline and eval.run exits non-zero; the skill reports the regression and the failing categories. Restore the rule when you are done.

  6. Log one restraint decision in the harness map:

    printf '%s\n' '' 'Declined: no issue-tracker MCP server yet, because this workflow only needs local tests and eval.' >> toolkit/harness-map.md

    Expected: the map records one capability you declined and the reason it failed the repeated-need test. This is part of shaping the harness: what you leave out matters as much as what you add.

▶ Coached mode

Want this walked through live? Coached mode walks the lab with you. The /lesson-4-skills instructor skill (it ships in practice-repo/triage/.claude/skills/lesson-4-skills/) coaches you step by step from a second terminal — it has you name the recurring workflow, capture it as a Good skill, then push it to Great by encoding the pass/fail call. It points you to answer-keys/module-4.md only after you've written your own.

On your own codebase, draw the same harness map for one workflow you have walked an agent through more than twice this week. Capture the repeated part as a skill at .claude/skills/<name>/SKILL.md, invoked with /<name>. Have it encode the decision you make at the end of that workflow. Add only the tools or external systems the workflow repeatedly needs.

Keep the artifact

Keep two artifacts: a harness map for the workflow, and the custom skill that makes the repeated verification move one invocation. Add the repeated-need rule to your project instructions so future tool additions are judged by the same standard.

Self-check

You did it right if your harness map names context, tools, permissions, reusable workflow, and check; your skill runs the workflow end to end on its own; and every external connection you added has a one-sentence repeated-need justification. If the justification is "it might be handy," remove it.

Recall

Before moving on: what is the difference between a prompt, context, and the harness?

Answer

The prompt is what you ask in this turn, context is what the agent can see, and the harness is the environment that lets the agent act and stay bounded: tools, permissions, skills, external systems, and checks.