Triage of the 2026-08-20 full run: 12 failed lisi tasks, and 26% of the score grid is silently NaN

[rcannood]

Triage of the full de.NBI run 31fiy5fioKYoQx (results dir run_2026-08-18_09-42-55, tasks executed 2026-08-20 07:55 → 17:42). Nextflow reported SUCCEEDED, but exitStrat() in labels_denbi.config ignores non-zero exits after retrying, so 12 tasks actually failed.

The bigger problem is not in the trace at all: on human_blood_mass_cytometry, four of the nine metric values are NaN for every single method, produced by tasks that exited 0. Counting both, 118 of the 450 cells in the score grid (26%) carry no number.

Where the data is

• Published results: https://openproblems.bio/benchmarks/cyto_batch_integration/v0.0.1-rc2/ — this run, released as v0.0.1-rc2. Its quality-control panel reproduces everything below from the results alone: human_blood_mass_cytometry 47% missing, iLisi 62%, emd_mean_ct_vert and both functional_marker_preservation_* 50%, the six cycombine_* methods 39%, and metric component lisi 24% failed.
• Seqera run: https://cloud.seqera.io/orgs/openproblems-bio/workspaces/openproblems-bio/watch/31fiy5fioKYoQx
• Results: s3://openproblems-data/resources/task_cyto_batch_integration/results/run_2026-08-18_09-42-55/ with trace.txt, score_uns.yaml, report/ and logs/ (all .command.* and .exitcode, 695/695 tasks present, logs/_index.tsv maps work-dir hash → task)
• Fetched off the cluster with common/scripts/fetch_denbi_run --logs --upload 31fiy5fioKYoQx. Full work dirs are not synced — those only live on the cluster's /vol/scratch, which is deleted when the cluster is terminated, so grab anything you still need before teardown.

What ran

25 methods (20 + 5 controls) × 2 datasets × 6 metrics. All 50 method × dataset combinations are present in score_uns.yaml; no method produced nothing.

exit	trace rows	distinct tasks	meaning
1	25	13	12 × lisi (permanent; exit 1 is retried once → 24 rows) + 1 × batchadjust_one_control (retry succeeded)

Must fix: non-method components

• lisi: 12/50 tasks failed, and the metric is missing for all six cycombine_* methods on both datasets. An h5ad encoding-version skew, not a logic bug. cycombine_*/script.R records its cyCombine call in uns, including "covar" = NULL, "anchor" = NULL. R NULL → reticulate → Python None → anndata 0.12 writes it as encoding_type='null'. But metrics/lisi/config.vsh.yaml:113 pins anndata~=0.11.0, and 0.11 has no reader for that spec:

  anndata._io.specs.registry.IORegistryError: No read method registered for
  IOSpec(encoding_type='null', encoding_version='0.1.0') from <class 'h5py._hl.dataset.Dataset'>.
  Error raised while reading key 'anchor' of <class 'h5py._hl.dataset.Dataset'> from /uns/parameters/correct_data
  

  Verified against the exact images from the run — methods/cycombine_no_controls_to_goal:build_main has anndata 0.12.19, metrics/lisi:build_main has 0.11.4 — and reproduced end to end: writing {'anchor': None} in the first image and reading it in the second raises the same IORegistryError. Every other metric sits on openproblems/base_python:1 (anndata 0.12.19), which is why only lisi failed.

  Fix is one line: anndata~=0.11.0 → anndata~=0.12.0. Checked that this does not drag scib-metrics along — pip install "scib-metrics==0.5.7" "anndata~=0.12.0" resolves cleanly in the lisi image, and read_h5ad then returns {'anchor': None, 'covar': None, 'method': 'ComBat'}. When re-running, re-run all 50 lisi tasks rather than only the 12; mixing anndata versions within one metric column is not worth the saved compute.

• bras carries the identical anndata~=0.11.0 pin at metrics/bras/config.vsh.yaml:80. It is not currently wired into the workflow (see Housekeeping) so it did not fail, but it will the moment it is switched back on. Same one-line fix.

• emd_mean_ct_vert, functional_marker_preservation_wilcoxon, functional_marker_preservation_cohens_d and iLisi are NaN for all 25 methods on human_blood_mass_cytometry — 94 empty cells, from tasks that exited 0 and are invisible in the trace. One shared root cause: after control removal and the split, that dataset has exactly one sample per (group × batch).

  Donor1 is marked is_control and Donor2 is the benchmark data, so the non-control samples are Donor2{Stim,Unstim}_Batch{1,2}, and the split sends one sample of each group to each side:

  split 1: Donor2Stim_Batch2, Donor2Unstim_Batch1
  split 2: Donor2Stim_Batch1, Donor2Unstim_Batch2
  

  Each metric then hits its own guard, all of them behaving correctly:

  metric	guard	why it trips
  emd_mean_ct_vert	emd/helper.py:113 — needs ≥2 samples per group to form a pair	1 sample per group per split
  iLisi	lisi/helper.py:28 — NaN when any group lives in a single batch	stim is only in batch 2, unstim only in batch 1 → batch is perfectly confounded with group
  functional_marker_preservation_*	needs pairs significant in both unintegrated batches; helper.py:122 drops pairs with <2 samples per group	each batch yields 0 tested pairs, hence 0 significant

  For contrast, mouse_spleen_flow_cytometry has 3 WT and 3 KO non-control samples per split and all three metrics compute normally.

  The guards are the right behaviour — they degrade instead of crashing. The problem is that a metric which is structurally undefined for a dataset is indistinguishable, in score_uns.yaml, from one that failed. Two decisions needed:

  1. Is a 1-donor design what we intend to ship for this dataset? If the raw VanGassen_dataset.h5ad has more usable donors, promoting some out of is_control fixes all four metric values at once. If not, the dataset genuinely cannot support them.
  2. If it cannot, these should exit_non_applicable() (exit 99) rather than write NaN, so the omission is explicit in the trace and the report renders "not applicable" instead of a blank that reads as a failure.

Not a bug

• batchadjust_one_control on mouse_spleen_flow_cytometry: exit 1 on attempt 1, COMPLETED on attempt 2 (26 m → 5 m). Transient NFS write error while writing the output h5ad (errno = 5, 'Input/output error', offset 6920). Single occurrence across all 695 tasks; the retry policy did its job. Nothing to fix, but worth remembering the shape if /vol/scratch starts misbehaving more often.

• ratio_consistent_peaks skipped 5750 of 76350 peak comparisons (7.5%) with WARNING! Inconsistent peaks detected in unintegrated data (case NGT). The skip is decided on the unintegrated data, so the same comparisons are dropped for every method on a given dataset and the ranking stays fair. Noted only so the number is on the record.

Housekeeping

• src/metrics/bras and src/metrics/n_inconsistent_peaks still exist but were dropped from the workflow's dependencies in bc8e0af (Setup run hpc #119). Either re-enable or delete them — right now the repo advertises 8 metrics and the benchmark computes 6.

Suggested order

lisi's anndata pin first — it is one line, it is broken right now, and it is the only thing in this run that produced a hard failure. Then bras while the fix is in hand. Then the human_blood_mass_cytometry question, which is the one that actually decides whether half the benchmark is interpretable, and needs a call rather than a patch.

[rcannood]

@LuLeom @ghar1821 Could you take a look at the results? Once any fixes are in I can do a rerun.

[ghar1821]

The 3 metrics (ilisi, emd vertical, functional marker preservation) were purposely not calculated for the cytonorm data (human blood mass cytometry data) because the dataset design confound batch and group. After removing control samples, we end up with just 1 sample per group in each batch, which makes it not possible to compute the metric. Thus they return NaN.

We are keeping it as NaN so in the final heatmap, it is treated as NaN rather than the metric errored when we ran the benchmark (if we change it to return error code 99 as suggested). Hence not fixing this.

But to make it explicit in the code that these metrics cannot be calculated due to the dataset design, we will create a generic function that will first do the check before the calculation is done. Fix will be implemented in the future - see issue #138.

[ghar1821]

@rcannood, there is a dataset that is in dataset_raw folder in the s3 bucket, but has not been processed. It is called lille_spectral_flow_cytometry. Need to run process_raw_datasets.sh script before doing another run.

[ghar1821]

@rcannood we have updated the anndata version in ilisi to 0.12 which support NULL to be written the uns of a method output. Also did some cleanup. See merged pull request #139

[rcannood]

Thanks @ghar1821 and @LuLeom ! Reprocessing the raw datasets and will trigger a run afterwards !