The per-claim verifier cannot distinguish obtained data from constructed data

[lfnothias]

On a reproduction task whose inputs were public LC-MS datasets, the raw data could not be
downloaded. The agent documented that correctly — it wrote a LIMITATION.txt recording that
anonymous FTP was blocked, anonymous HTTP listing was unavailable, and the repository page
required credentials.

It then constructed the missing inputs rather than stopping: a 15-row feature table of
F001, F002, …, a second table of 10 rows, MGF files containing five and four invented
spectra, and a five-entry stand-in for a reference spectral library of several thousand
entries. All eight were recorded in dataset_inventory.csv with status,downloaded, and
annotated "realistic intensities", "realistic Q-Exactive data", "valid Mascot Generic Format".

The step scored 0.865, the highest of the run.

Why the verifier did not catch it

The per-claim verifier had generated four claims aimed at exactly this failure. Their
descriptions state the intent explicitly.

claim	stated intent	verdict
feature_table_nondegenerate_variance	"sanity check prevents fabricated data"	pass — "95 unique values, const_cols=[]"
mgf_spectra_distinct_and_nontrivial	"ensures real not placeholder spectra"	pass — "5 blocks, distinct PEPMASS/CHARGE"
dataset_inventory_reflects_actual_file_states	"prevents fabricated results … without misrepresenting fabricated placeholder data as fully validated real downloads"	pass — "All 8 inventory entries accurately reflect on-disk file states"
gnps_library_entry_count_nontrivial	"ties to key 2657-entry target metric"	fail — "5 entries; target=2657; fraction=0.0019"

Three of the four passed, and they passed correctly on their own terms:

• a carefully constructed table does have non-degenerate variance;
• invented spectra are distinct from one another;
• the inventory is accurate about files that exist on disk. This check compares the manifest
  to the artefacts, which is circular when the artefacts are the fabrication.

The only claim that fired is the only one anchored to a quantity from outside the workspace
(the entry count reported in the source publication, Schmid et al. 2021, Nat Commun 12:3832).

The general shape

The verifier reads only what the run produced. It can establish that an artefact is
well-formed, internally consistent and non-degenerate. It cannot establish that the artefact
corresponds to anything outside the workspace, because it has nothing outside the workspace to
compare against. Every intrinsic property a real dataset has, a competently constructed one
also has.

This matters beyond benchmark scoring: the same score drives evolutionary selection, so a run
that fabricates its inputs can be preferred over one that reports the blocker and stops.

Possible directions

Roughly in order of cost, and offered as options rather than a proposal:

1. A provenance claim distinct from a content claim. For every declared input, record how it
   was obtained — a URL and an HTTP status, a checksum, a file size at the source — and verify
   that record rather than the file's shape. An artefact with no retrieval provenance is not the
   same as one with provenance, whatever its contents look like.
2. Make "constructed" a first-class outcome. The agent was honest in LIMITATION.txt and
   dishonest in dataset_inventory.csv, and nothing forced those to agree. A declared status of
   constructed/synthetic that propagates into the inventory, the claims and the ASTRA capsule
   would let a downstream step refuse to consume it.
3. Cap the score when an input is unverifiable. A hard-fail cap already exists in the
   verifier; a step whose inputs are self-constructed arguably should not be able to score above a
   run that obtained them.

Happy to prepare a PR for whichever direction is preferred; the first two are the ones I would
argue for, since they change what is recorded rather than how it is weighted.

Reproduction

The run artefacts are on the reporting side rather than in this repository. The relevant Mimosa
state is sources/workflows/<uuid>/evaluation.txt, which contains all four claims above with
their verdicts and machine details.