In sources/benchmark_evaluation/csv_mode.py, _generate_task_default interpolates the URLS column verbatim into the agent prompt ("Url to paper: {url}"). The column is treated as opaque text, so when a benchmark CSV references files on disk rather than web URLs, machine-specific absolute paths are embedded in every prompt and preserved in the run records. Such records neither replay nor compare across machines, and the prompts disclose the local directory layout.
The repository already exhibits the case: all 25 rows of datasets/asb_metabo_v1.csv carry absolute paths of the form /Users/<user>/git/... in the URLS column.
Rewriting local paths to a workspace-relative form at load time — or copying the referenced file into the task workspace and recording the original location separately — would keep prompts and run records portable across machines.
In
sources/benchmark_evaluation/csv_mode.py,_generate_task_defaultinterpolates the URLS column verbatim into the agent prompt ("Url to paper: {url}"). The column is treated as opaque text, so when a benchmark CSV references files on disk rather than web URLs, machine-specific absolute paths are embedded in every prompt and preserved in the run records. Such records neither replay nor compare across machines, and the prompts disclose the local directory layout.The repository already exhibits the case: all 25 rows of
datasets/asb_metabo_v1.csvcarry absolute paths of the form/Users/<user>/git/...in the URLS column.Rewriting local paths to a workspace-relative form at load time — or copying the referenced file into the task workspace and recording the original location separately — would keep prompts and run records portable across machines.