Harness and Loop Engineering Demo

A small, runnable repo that demonstrates harness engineering and loop engineering for AI-assisted software engineering.

The companion article explains the idea using a payments-style transfer API bug: duplicate transfer retries with the same idempotency key must return the original result, not create a second transfer or deduct balance again.

What this repo demonstrates

This repo is intentionally small. The interesting part is not the business domain. The interesting part is the engineering system around the change.

It demonstrates:

• A small transfer API with idempotency behavior.
• Regression tests that prove the bug and the fix.
• A task file that defines the agent assignment.
• A context builder that compiles relevant context for a coding agent.
• A verifier that runs deterministic checks.
• A lightweight reviewer that checks the patch against the requirement.
• A loop script that creates an episode report instead of pretending the agent is magically done.

Mental model

Prompt  = instruction
Harness = controlled runtime around the agent
Loop    = repeatable workflow that keeps checking and deciding what happens next

A prompt says: “Fix the bug.”

A harness says: “Here is the task, context, boundaries, allowed paths, verification commands, and definition of done.”

A loop says: “Prepare context, run checks, feed failures back, retry within limits, review, record evidence, and stop at the human gate.”

Repo structure

harness-loop-engineering-demo/
├── app/
│   ├── api.py
│   ├── buggy_transfer_service.py
│   ├── models.py
│   ├── repository.py
│   └── transfer_service.py
├── tests/
│   ├── conftest.py
│   ├── test_idempotency.py
│   ├── test_transfer_limits.py
│   └── test_transfer_success.py
├── agent_harness/
│   ├── task.yaml
│   ├── context_builder.py
│   ├── verifier.py
│   ├── reviewer.py
│   └── episode_logger.py
├── loop/
│   └── run_loop.py
├── docs/
│   ├── architecture.md
│   ├── coding-agent-instructions.md
│   └── requirement-transfer-idempotency.md
├── examples/
│   └── episode_report.md
├── scripts/
│   └── run_demo.sh
├── .github/workflows/tests.yml
├── Makefile
├── pyproject.toml
└── requirements.txt

Quick start

Create and activate a virtual environment:

python -m venv .venv
source .venv/bin/activate

On Windows PowerShell:

python -m venv .venv
.venv\Scripts\Activate.ps1

Install dependencies:

pip install -r requirements.txt

Run the focused idempotency tests:

pytest tests/test_idempotency.py

Run the full test suite:

pytest tests/

Run the harness loop demo:

python loop/run_loop.py

This writes two generated files:

agent-context.txt
agent-episode-report.md

They are intentionally ignored by Git.

Run the API

uvicorn app.api:app --reload

Then send a transfer request:

curl -X POST http://127.0.0.1:8000/transfers \
  -H "Content-Type: application/json" \
  -d '{
    "fromAccount": "A100",
    "toAccount": "B200",
    "amount": "700",
    "idempotencyKey": "REQ-123"
  }'

Send the same request again. The response should contain the same transferId.

The bug being demonstrated

The correct transfer flow is:

1. Check idempotency key.
2. If already processed, return the original result.
3. Otherwise validate the request.
4. Process and save the transfer result.

The intentionally buggy version in app/buggy_transfer_service.py does this instead:

1. Validate the request.
2. Check idempotency key.
3. Process and save the transfer result.

That order is wrong because a retry is not a new business request. It should not be revalidated against the latest account state.

The test test_buggy_service_demonstrates_why_validation_order_matters shows the failure mode.

Harness engineering in this repo

The harness is represented by files under agent_harness/.

task.yaml

Defines:

• Task ID and title
• Risk level
• Required context
• Allowed paths
• Verification commands
• Definition of done
• Review checks
• Maximum attempts
• Human gate

context_builder.py

Compiles selected context into a single prompt-like artifact for a coding agent.

This is deliberate context selection, not repo dumping.

verifier.py

Runs the commands defined by the task file.

reviewer.py

Performs simple deterministic checks:

• Idempotency lookup happens before validation.
• Regression tests prevent double debit and duplicate transfer records.
• Public API fields remain stable.

episode_logger.py

Writes an evidence report showing what passed, what failed, and what still needs human review.

Loop engineering in this repo

The loop is represented by loop/run_loop.py.

It does not call an LLM by default. That is intentional. The repo is designed to be safe, runnable, and understandable without API keys.

The loop demonstrates the workflow shape:

1. Read the task file.
2. Build task-specific context.
3. Run verification commands.
4. Run reviewer checks.
5. Write an episode report.
6. Stop at the human gate.

In a real implementation, the LLM call would sit between context preparation and verification:

Build context -> Agent modifies code -> Verify -> Review -> Human gate

Why this matters

AI-assisted development is not only about code generation.

The surrounding engineering system decides whether AI-generated code is safe to absorb:

• Context selection
• Boundaries
• Tests
• Review checks
• Retry limits
• Evidence
• Human decision points

That is the real lesson behind harness and loop engineering.

Suggested article title

The repo was created for an article tentatively titled:

The Next AI Coding Skill Is Not Prompting. It Is Designing the Loop.

License

MIT