Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
106 lines (79 loc) · 4.64 KB

CONTRIBUTING.md

File metadata and controls

106 lines (79 loc) · 4.64 KB
Raw

Contributing to aakit

This is a measurement instrument, not an application. That changes what a good contribution looks like. The most valuable thing you can send is evidence that a number it produced is wrong.

The one rule

Do not change the instrument in the middle of a run.

If you edit aakit/extract.py's SYSTEM prompt, the trace truncation caps in aakit/transcripts.py, or any threshold in aakit/config.py, then every assumption adjudicated before your change was produced by a different instrument than the ones adjudicated after. The combined number is not a measurement of anything.

If you need to change one of those, say so in the PR title with [INSTRUMENT] and state in the body what has to be re-run. Runs in progress get discarded, not patched.

Everything else is ordinary code and can be changed freely: CLI ergonomics, reporting, docs, new task fixtures.

The highest-value contribution is instrument bugs

The first version of this kit truncated tool outputs head-only. Tool results are where the evidence for an inference lives, and it usually lives at the end: the last line of a file, the error at the bottom of a log, the final row of a query. Head-only truncation deleted that evidence, so the extractor saw well-supported inferences with no visible support and labelled them invented. The instrument was manufacturing the errors it was built to measure.

That class of bug is the thing most worth finding, and it is only findable by reading traces by hand. If you find one:

  1. Open an issue titled [INSTRUMENT BUG] <one line>.
  2. Include the trace excerpt that shows the problem, redacted as needed.
  3. Say which direction it biases the number, inflating or deflating the load-bearing-wrong rate, and roughly by how much if you can tell.

A report that a published number is too high is more welcome than a feature.

"Someone already built this"

The README claims no shipped AI product maintains a machine-checkable, automatically-revisable record of what an agent assumed. That is a claim about the state of the market, made after a search, and searches miss things.

If you know of a counterexample, open an issue titled [PRIOR ART] <product> with a link and a sentence on which part it implements: the record, the revision loop, or both. This is a wanted issue, not a nuisance one. Being correctly told the claim is wrong is a better outcome than the claim standing because nobody checked.

The same applies to the academic prior art in the README. If the lineage is mis-attributed, say so.

Adding task fixtures

Fixtures for the ask-policy harness live in tasks/*.jsonl. A good fixture is an underspecified task where:

  • the missing information is genuinely missing, not merely unstated but obvious
  • there is a spec-bound oracle answer, so a grader can score without taste
  • the number of hidden facts is recorded

That last one matters. The central finding so far is that a one-question budget fails on multi-fact gaps, so say how many facts your task hides. A fixture that hides exactly one is not representative of real underspecified requests and will flatter the gate.

Also check that your ambiguous tasks actually become solvable once the missing fact is supplied. In the first real run, two ambiguous tasks failed even under always with a full oracle. Those are capability- or rubric-limited rather than information-limited, and they carry no signal for the experiment. Cap rubrics at criteria a single-shot artifact can satisfy.

Running the tests

python3 -m venv .venv && source .venv/bin/activate
pip install -e .
python3 tests/smoke.py

Requires Python 3.10+. The smoke test must end with all checks passed. It covers the Wilson interval, store round-trips, transcript parsing and redaction, the kill-criterion branches, and the full pipeline through report rendering.

Code conventions

  • Standard library only. No runtime dependencies. If you need a package to do it, it probably belongs outside the kit.
  • Comments explain why, especially where a choice is non-obvious and was made because the alternative produced a wrong number. Those comments are the instrument's changelog. Keep them.
  • Match the surrounding style instead of introducing your own.

What is out of scope

  • Anything that grades assumptions automatically and reports the result as a finding. auto-review exists for smoke tests only. If a model both extracts and adjudicates, the output is a model artifact, and reporting it as a measurement is the exact failure this kit was built to detect.
  • Vendor integrations that send transcripts anywhere by default. A human reviews redaction before any trace leaves the machine, and that stays true.