Skip to content

Commit 1e2d508

Browse files
ghar1821ghar1821
committed
reorganising scripts for hpc
1 parent 0e11692 commit 1e2d508

6 files changed

Lines changed: 126 additions & 128 deletions

File tree

‎scripts/run_benchmark/run_warmup.sh‎

Lines changed: 0 additions & 127 deletions
This file was deleted.

‎scripts/run_benchmark/wehi_hpc/README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Running the benchmark on WEHI hpc
2+
3+
This directory contains all the scripts needed to submit the benchmark job on WEHI hpc.
4+
5+
The pipeline will use the config file in `scripts/labels_tw_wehi.config`.
6+
7+
Few things to note if you want to replicate this on different slurm system:
8+
9+
1. Before submitting the job, make sure to run `scripts/run_benchmark/wehi_hpc/build_apptainer_images.sh` script first. See sections below for more info.
10+
2. Make sure you have setup seqera tower's compute environment to point to the HPC cluster.
11+
You must select Slurm as the as environment. Have a look at the existing one and clone the relevant settings.
12+
3. Change the workspace ID to point to the new environment created in step 2.
13+
4. Create all the necessary temp folders. See the config file.
14+
5. Run tower agent in the background, using tmux or something.
15+
16+
17+
## Apptainr image build
18+
19+
The `scripts/run_benchmark/wehi_hpc/build_apptainer_images.sh` will pull the docker images
20+
from url in `scripts/run_benchmark/wehi_hpc/images.txt` and build them in parallel (one job = 1 image).
21+
It will automatically rename the images to something that the pipeline can understand.
22+
So something like: `ghcr.io-openproblems-bio-openproblems-utils-extract_uns_metadata-build_main.img`.
23+
24+
This script must be run before you run the full benchmark as otherwise the head job will be overwhelmed with
25+
the number of images it has to build and just die.
26+
27+
Why not get each worker to build its own image? It is because the apptainer cache directory
28+
is shared between different jobs.
29+
So there will be locking contention.
30+
This may be fixable by setting each job to use its own cache, but i don't have time to test it.
31+
32+
If there is a new method or metric, make sure that is added to the `scripts/run_benchmark/wehi_hpc/images.txt` file.
33+
If we have remove a method or metric, make sure you remove it from the file as well.
34+
35+
If you only update one or two methods or metrics, there is no needed to pull and rebuild everything.
36+
Just build those that were updated.
37+
Create a new text file with the url of the images you need to build and run the script.
38+
Make sure you change the filename of the `scripts/run_benchmark/wehi_hpc/images.txt`.
39+
40+
The apptainer images must be stored directly in the `APPTAINER_CACHE` directory.
41+
If you want to test whether the job uses the images properly, run the `scripts/run_benchmark/wehi_hpc/run_subset_hpc.sh`
42+
script which by default will only run one metric and one method.
43+
If you see `cache` folder or `*.lock` file in the `APPTAINER_CACHE` directory, then you know
44+
the head job is repulling and rebuilding the images.
45+

‎scripts/run_benchmark/wehi_hpc/build_apptainer_images.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
# filepath: /vast/scratch/users/putri.g/submit_apptainer_jobs.sh
3+
4+
#SBATCH --job-name=apptainer-pull_array
5+
#SBATCH --ntasks=1
6+
#SBATCH --mem=16G
7+
#SBATCH --partition=regular
8+
#SBATCH --time=01:00:00
9+
#SBATCH --array=1-31
10+
#SBATCH --output=/vast/scratch/users/putri.g/slurm_log/apptainer/pull_%a.out
11+
#SBATCH --error=/vast/scratch/users/putri.g/slurm_log/apptainer/pull_%a.err
12+
13+
echo "Running Array Job ID="$SLURM_ARRAY_JOB_ID", Array Index="$SLURM_ARRAY_TASK_ID
14+
15+
module load apptainer
16+
17+
IMAGES_FILE="/vast/scratch/users/putri.g/cytobenchmark/apptainer_images/images.txt"
18+
WORK_DIR="/vast/scratch/users/putri.g/nextflow/apptainer_cache"
19+
20+
# Each job gets its own isolated cache
21+
export APPTAINER_CACHEDIR="$WORK_DIR/cache_$SLURM_ARRAY_JOB_ID_${SLURM_ARRAY_TASK_ID}"
22+
mkdir -p "$APPTAINER_CACHEDIR"
23+
24+
# Read image for this task
25+
image_url=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$IMAGES_FILE")
26+
if [ -z "$image_url" ]; then
27+
echo "Error: No image at line $SLURM_ARRAY_TASK_ID in $IMAGES_FILE" >&2
28+
exit 1
29+
fi
30+
31+
echo "Job $SLURM_ARRAY_TASK_ID: Building $image_url"
32+
echo "Cache directory: $APPTAINER_CACHEDIR"
33+
34+
final_dir="$WORK_DIR/images"
35+
mkdir -p "$final_dir"
36+
37+
# Extract full name from URL: docker://ghcr.io/path/to/image:tag -> ghcr.io-path-to-image-tag
38+
image_name=$(echo "$image_url" | sed 's|docker://||' | sed 's|/|-|g' | sed 's|:|-|g')
39+
output_file="$final_dir/${image_name}.img"
40+
41+
echo "Output file will be: $output_file"
42+
43+
# Build directly to .img format
44+
if apptainer build "$output_file" "$image_url"; then
45+
echo "Successfully built: $output_file"
46+
else
47+
echo "Error: Failed to build $image_url" >&2
48+
exit 1
49+
fi

‎scripts/run_benchmark/wehi_hpc/images.txt‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
docker://ghcr.io/openproblems-bio/openproblems/utils/extract_uns_metadata:build_main
2+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/control_methods/no_integration:build_setup_run_hpc
3+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/control_methods/perfect_integration:build_setup_run_hpc
4+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/control_methods/shuffle_integration_globally:build_setup_run_hpc
5+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/control_methods/shuffle_integration_within_batch:build_setup_run_hpc
6+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/control_methods/shuffle_integration_within_cell_type:build_setup_run_hpc
7+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/batchadjust_all_controls:build_setup_run_hpc
8+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/batchadjust_one_control:build_setup_run_hpc
9+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/combat:build_setup_run_hpc
10+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cycombine_all_controls_to_goal:build_setup_run_hpc
11+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cycombine_all_controls_to_mid:build_setup_run_hpc
12+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cycombine_no_controls_to_goal:build_setup_run_hpc
13+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cycombine_no_controls_to_mid:build_setup_run_hpc
14+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cycombine_one_control_to_goal:build_setup_run_hpc
15+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cycombine_one_control_to_mid:build_setup_run_hpc
16+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cytonorm_all_controls_to_goal:build_setup_run_hpc
17+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cytonorm_all_controls_to_mid:build_setup_run_hpc
18+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cytonorm_no_controls_to_goal:build_setup_run_hpc
19+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cytonorm_no_controls_to_mid:build_setup_run_hpc
20+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cytonorm_one_control_to_goal:build_setup_run_hpc
21+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/cytonorm_one_control_to_mid:build_setup_run_hpc
22+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/gaussnorm:build_setup_run_hpc
23+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/harmonypy:build_setup_run_hpc
24+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/limma_remove_batch_effect:build_setup_run_hpc
25+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/rpca_to_goal:build_setup_run_hpc
26+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/methods/rpca_to_mid:build_setup_run_hpc
27+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/metrics/average_batch_r2:build_setup_run_hpc
28+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/metrics/emd:build_setup_run_hpc
29+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/metrics/flowsom_mapping_similarity:build_setup_run_hpc
30+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/metrics/lisi:build_setup_run_hpc
31+
docker://ghcr.io/openproblems-bio/task_cyto_batch_integration/metrics/ratio_inconsistent_peaks:build_setup_run_hpc

scripts/run_benchmark/run_full_hpc.sh renamed to scripts/run_benchmark/wehi_hpc/run_full_hpc.sh

File renamed without changes.

scripts/run_benchmark/run_subset_hpc.sh renamed to scripts/run_benchmark/wehi_hpc/run_subset_hpc.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ cat > /tmp/params.yaml << HERE
1919
input_states: /vast/scratch/users/putri.g/cytobenchmark/benchmark_out_hpc/datasets/**/state.yaml
2020
rename_keys: 'input_censored_split1:output_censored_split1;input_censored_split2:output_censored_split2;input_unintegrated:output_unintegrated'
2121
output_state: "state.yaml"
22-
settings: '{"metrics_include": ["lisi"], "methods_include": ["combat"]}'
22+
settings: '{"metrics_include": ["emd"], "methods_include": ["combat"]}'
2323
publish_dir: "$publish_dir"
2424
HERE
2525

0 commit comments

Comments
 (0)