Skip to content

Enable 'pipeliner' tasks to be forced to run even if preceeding tasks have failed - #948

Open
pjbriggs wants to merge 3 commits into
develfrom
pipeliner-tasks-always-run
Open

pjbriggs wants to merge 3 commits into
develfrom
pipeliner-tasks-always-run

Conversation

@pjbriggs

Copy link
Copy Markdown
Member

Adds new functionality to the pipeliner module, to enable tasks to be flagged as always_run when being added to a pipeline. If the pipeline fails in "deferred" failure mode then tasks that are flagged in this way will not be removed by the clean up operations even if they are dependent either directly or indirectly on the failed tasks.

For example for a simple notional pipeline:

ppl.add_task(first_task)
ppl.add_task(second_task,requires=(first_task,))
ppl.add_task(final_task,requires=(second_task,),always_run=True)

In this case if either of first_task or second_task fails then final_task will still be executed by the pipeline.

Note that any failed tasks will result in the pipeline being marked as "failed" (i.e. returning a non-zero exit code).

The expected application for this is reporting within the QC pipeline, where it would be useful to produce a report even if the pipeline has failed.

@pjbriggs
pjbriggs force-pushed the pipeliner-tasks-always-run branch from 950e2da to cec5dce Compare February 26, 2024 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant