Skip to content

AGENTS.md

Latest commit

 

History

History
85 lines (64 loc) · 3.27 KB

AGENTS.md

File metadata and controls

85 lines (64 loc) · 3.27 KB
Raw

Repository instructions

Purpose

This repository contains fact-only tools for coding-agent evaluations. The tools extract evidence from run artifacts. They do not replace a human evaluation.

Documentation requirements

  • Write short and direct sentences.
  • Use active voice and present tense.
  • Give only one instruction in each sentence.
  • Use the same term for the same item.
  • Define an uncommon technical term before you use it.
  • Put a condition before the action that depends on it.
  • Use must for a requirement.
  • Use can for a capability.
  • Use do not for a prohibition.
  • Do not use idioms, slang, contractions, or decorative language.
  • Keep a procedural sentence at 20 words or fewer when possible.
  • Keep a descriptive sentence at 25 words or fewer when possible.
  • Use lists for sequences and sets of related facts.

Required behavior

  • Preserve source references for each reported event.
  • Use an explicit status field to count a failed or successful result.
  • Keep raw result counts separate from human judgments.
  • State when a source does not provide a timestamp, status, cost, token count, or duration.
  • Prefer ATIF or OTLP over a vendor-specific format when both are available.
  • Add a deterministic fixture for every adapter change.
  • Keep generated reports and raw trajectories out of Git.
  • Remove credentials and private data from fixtures and examples.

Human decision boundary

Do not make these decisions for a human evaluator:

  • Do not assign a model rating or severity.
  • Do not select a winning model.
  • Do not select a failure mode.
  • Do not write a submission-ready preference explanation.
  • Do not infer private chain-of-thought.

You can count observable events. You can group facts under an evaluation axis. You can check a human claim against a cited event.

Public task boundary

The task at examples/harbor-toy-task/ is a teaching fixture. Its instruction and verifier are public. Do not present it as a valid benchmark.

Do not add active evaluation material to this repository. This material includes prompts, private tests, hidden verifiers, expected patches, and raw private trajectories.

Keep active evaluation cases in a separate private repository.

Development checks

Run all checks before a commit:

uv sync --all-groups
uv run python -m unittest discover -s tests -v
uv run ruff check .
uv run ruff format --check .
uv run mypy

Repository structure

  • src/trajectory_facts/: adapters, normalization, analysis, reports, and the dashboard.
  • tests/: deterministic unit and public-example tests.
  • .codex/skills/trajectory-facts/: reusable fact-analysis skill.
  • docs/: architecture, evidence, format, and safety documentation.
  • examples/trajectories/: small synthetic input files.
  • examples/sample-harbor-job/: a synthetic job for offline dashboard use.
  • examples/harbor-toy-task/: a public non-benchmark Harbor task.
  • scripts/: task preparation and end-to-end example runners.
  • design/: dashboard interface rules and tokens.

File rules

  • Use UTF-8 text.
  • Use LF line endings except for PowerShell files.
  • Use Python 3.11 or later.
  • Keep runtime dependencies in the Python standard library when possible.
  • Use pathlib.Path for paths.
  • Do not log a token, password, private key, or authentication file content.