Skip to content

Add denbi script #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rcannood merged 3 commits into from
Sep 2, 2026
Merged

Add denbi script #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
41faf9e
add denbi raw dataset processing script
rcannood Sep 1, 2026
34ca815
bump common for denbi run fetch script
rcannood Sep 1, 2026
35d6714
update submodule
rcannood Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion common
Open in desktop
Submodule common updated 1 files
+44 −51 TASK_INSTRUCTIONS.md
25 changes: 25 additions & 0 deletions scripts/create_resources/process_raw_datasets_denbi.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash

# de.NBI variant of process_raw_datasets.sh: runs the process_datasets workflow on the
# BiBiGrid SLURM cluster and publishes straight to S3. The S3 credentials are the `tower`
# service user's, provided on the cluster via /vol/scratch/cluster-env.sh
# (AWS_SHARED_CREDENTIALS_FILE) — no credentials need to be passed at launch.

cat > /tmp/params.yaml << 'HERE'
input_states: s3://openproblems-data/resources/task_cyto_batch_integration/datasets_raw/**/state.yaml
rename_keys: 'input:output_dataset'
output_state: '$id/state.yaml'
settings: '{"output_unintegrated": "$id/unintegrated.h5ad", "output_censored_split1": "$id/censored_split1.h5ad", "output_censored_split2": "$id/censored_split2.h5ad"}'
publish_dir: s3://openproblems-data/resources/task_cyto_batch_integration/datasets/
HERE

tw launch https://github.com/openproblems-bio/task_cyto_batch_integration.git \
--revision build/main \
--pull-latest \
--main-script target/nextflow/workflows/process_datasets/main.nf \
--workspace 53907369739130 \
--compute-env 3qstFmP9lNwdzutSNuJq7c \
--params-file /tmp/params.yaml \
--entry-name auto \
--config common/nextflow_helpers/labels_denbi.config \
--labels task_cyto_batch_integration,process_datasets,denbi
Loading