Update ilisi and add functional marker based metrics - #126
Merged
Conversation
- iLISI is now computed per group; groups where batch is fully confounded by group are skipped and return NaN - Refactored cLISI calculation into helper.py - Added compute_ilisi_per_group and compute_clisi to helper.py - Updated CHANGELOG.md
- Switch Docker image to base_pytorch_nvidia:1.0.0 - Pin harmonypy>=0.2.0 and add torch dependency - Pass device=None to run_harmony for auto GPU detection - Remove epsilon workaround as harmonypy v0.2.0 handles numerical stability internally via clamping and L2 normalisation - Update Nextflow label to lowcpu, gpu - Update CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Summary
New metric
metrics/functional_marker_preservation: evaluates whether batch integration preserves biologically meaningful differences in functional marker expression between two groups (e.g. WT vs KO):Updated
metrics/lisi: iLISI is still computed globally across all cells per split (not per group), but is now skipped and returns NaN for a split if batch is confounded by group in any group — i.e. a group whose cells all come from a single batch, so cross-batch mixing can't be assessed for it. The iLISI/cLISI computation was moved out ofscript.pyinto a newhelper.py(compute_ilisi,compute_clisi,_check_batch_group_confounding), and the outputunswas expanded to include per-cell iLISI/cLISI values and cell IDs for each split.Updated
methods/harmonypy: switched to the PyTorch GPU backend (harmonypy≥0.2.0) using new openproblems/base_pytorch_nvidia:1.0.0 image, automatic GPU detection, removed the manual epsilon workaround, and updated the Nextflow label to lowcpu, gpu.Updated
metrics/flowsom_mapping_similarity: mapping similarity is now computed bidirectionally (split1→split2 and split2→split1), and per-donor cluster×cell_type absolute-difference matrices are exported in the output uns.Added
scripts/adhoc_runs/run_metric_adhoc.py+.shwhich re-runs a single metric against existing pipeline output by injecting a VIASH block and running it as a subprocess, discovering (dataset, method) pairs automatically, with a skip-existing option. Lets you iterate on metric code without re-running the full Nextflow pipeline.Added
scripts/fetch_intermediate_files.pywhich consolidate the previous two-step log-parsing/file-copying process into one script that auto-detects SLURM vs AWS Batch runs and organizes output under<dataset>/method_out/ and <dataset>/metric_out/.This replaces the old
find_intermediate_files.pyanddownload_intermediate_files.pyandcopy_intermediate_files.py.Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!