It's non-trivial to 'dry-run' a pipeline. The main.py --dry-run functionality is very limited, and just prints the stages in the workflow, +/- some first/last/skipping through config
What I really want to test locally is the code in queue_jobs and expected_outputs, as well as the interconnection between stages. This requires me to run the workflow locally, without the --dry_run parameter (so that the Stage code, expected_outputs, queue_jobs actually runs), but with dry run communicated directly to Hail here through config, to prevent launching of the assembled jobs. To do this (AFAIK) I need to run the pipeline normally (without dry-run), and with a substitute config block like this:
dataset_gcp_project = "test-analysis-dataset-1234"
sequencing_type = "genome"
path_scheme = "local"
driver_image = "<stub>"
[hail]
billing_project = "test-billing-project"
delete_scratch_on_exit = true
dry_run = true
backend = "local"
It would be nice if this was a behaviour I could trigger with just a CLI setting.
Another glitch here is that even providing these config settings, some methods write directly to GCP during workflow setup (e.g. here or here - the latter responds to workflow.dry_run in config, which is a third different place we can add dry_run, and affects a different subset of behaviours.)
It's non-trivial to 'dry-run' a pipeline. The
main.py --dry-runfunctionality is very limited, and just prints the stages in the workflow, +/- some first/last/skipping through configWhat I really want to test locally is the code in queue_jobs and expected_outputs, as well as the interconnection between stages. This requires me to run the workflow locally, without the
--dry_runparameter (so that the Stage code, expected_outputs, queue_jobs actually runs), but with dry run communicated directly to Hail here through config, to prevent launching of the assembled jobs. To do this (AFAIK) I need to run the pipeline normally (without dry-run), and with a substitute config block like this:It would be nice if this was a behaviour I could trigger with just a CLI setting.
Another glitch here is that even providing these config settings, some methods write directly to GCP during workflow setup (e.g. here or here - the latter responds to
workflow.dry_runin config, which is a third different place we can adddry_run, and affects a different subset of behaviours.)