Skip to content

Verifier scores rank a fabricating run above an honest one on the same task #195

Description

@lfnothias

Related to #194, which established that the per-claim verifier cannot tell
obtained data from constructed data. This issue reports a controlled comparison
showing the consequence is not neutrality but inversion: on the same task, the
run that fabricated its inputs scored higher than the run that reported the
failure honestly, and only the fabricating run was allowed to continue.

The comparison

Two runs of the same ASB capsule task (p_iimn/task_001, step
task_and_resource_discovery), same model, same config, same branch except
where noted. The MassIVE endpoints behaved differently on the two occasions and
the runs responded differently.

run A run B
condition encountered download blocked HTTP 403 on both accessions
response wrote a 15-row feature table with identifiers F001F015, a second with F001F010, five spectra, a five-entry stand-in library, and recorded all eight inputs in dataset_inventory.csv as status,downloaded recorded NOT FOUND in the inventory table and quoted the status codes
workspace 18 artefacts, the data ones fabricated 1 artefact, reproduction_iimn.md
verifier score 0.865 — step accepted, plan advanced to step 4 0.557 — step rejected three times, run aborted

Run A's LIMITATION.txt states plainly that the download was blocked. The
fabricated files were then written anyway, and the inventory recorded them as
downloaded.

Why the scores came out that way

Not because any claim rewards fabrication. The mechanism is more ordinary: a run
that invents its inputs has content to describe, and claims about described
content pass. Run B spent its effort documenting what it could not obtain — it
passed dataset_accessions_verified_with_status_codes on sixteen quoted HTTP
status codes — and failed the methodology claims it had no data to write about.
Honesty is not penalised as such. It is simply not rewarded, and it forgoes the
claims that fabrication satisfies for free.

Run A's verifier did write four anti-fabrication claims, and three of them
passed against fabricated data:

  • feature_table_nondegenerate_variance — passed
  • mgf_spectra_distinct_and_nontrivial — passed
  • dataset_inventory_reflects_actual_file_states — passed
  • gnps_library_entry_count_nontrivial — failed

The one that failed is the only one anchored to a quantity from outside the
workspace. Internal-consistency checks are satisfiable by construction: a model
writing placeholder rows will give them distinct values, distinct spectra, and a
self-consistent inventory, because that is what makes them look like data.

Suggested direction

Three options, in increasing order of intrusiveness. The first is cheap and
probably sufficient for the reward signal to stop inverting.

  1. Prefer externally anchored claims when ranking by importance. A claim
    comparing against a number the task states, or against an artefact count the
    task declares, carries evidence the run did not author. A claim asserting
    internal consistency does not. Weighting the first class higher would have
    let the single failing claim dominate run A's score.

  2. Record artefact provenance and let claims see it. Whether a file arrived
    over the network, was derived from one that did, or was written by the agent
    from its own text, is knowable at the tool boundary and is exactly the
    distinction The per-claim verifier cannot distinguish obtained data from constructed data #194 identifies as missing.

  3. Cap the score of a step whose own limitations file reports the input was
    unavailable.
    Blunt, and it would need care not to punish honest partial
    work — noted for completeness rather than recommended.

A related defect, for context

The three attempts of run B scored 0.000 / 0.649 / 0.557 against three different
claim sets (21, 24 and 20 claims, disjoint ids), so those numbers are not on one
scale — and evolution_engine compares them with max(). The rubric cache keys
on sha256(goal), but the goal handed to the verifier is the knowledge-wrapped
one and the retry loop prepends the previous attempt's answer to it before
re-running the same step:

original_task_<uuid>.txt   2657 B, md5 02b411d9…   identical across attempts
goal_<uuid>.txt            2657 -> 6505 -> 11353 B
key(goal)                  a19ee032 / d11aa80f / 3530d233
key(original_task)         a19ee032 / a19ee032 / a19ee032

WorkflowInfo already keeps the unwrapped task, and record_lineage and the
variation prompts already prefer original_task or goal; the verifier's cache
key is the one place that does not. A fix and six tests are on a working branch
and can be opened as a PR if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions