Job finish path re-raises FileNotFoundError on missing tool_stdout, masking the real cause of the failure

[afgane]

Summary

JobRunner._finish_or_resubmit_job opens outputs/tool_stdout unguarded. If a job dies before that file is written, the resulting FileNotFoundError is re-raised, the caller turns it into job_wrapper.fail("Unable to finish job", exception=True), and everything the runner did know about the failure is discarded. The job lands with exit_code = NULL, no tool_stderr, no job_stderr and no job_messages.

The code anticipates this and asks for a report — lib/galaxy/jobs/runners/__init__.py#L653-L669 on dev:

try:
    with open(tool_stdout_path, "rb") as stdout_file:
        tool_stdout = self._job_io_for_db(stdout_file)
    with open(tool_stderr_path, "rb") as stderr_file:
        tool_stderr = self._job_io_for_db(stderr_file)
except FileNotFoundError:
    if job.state in (model.Job.states.DELETING, model.Job.states.DELETED):
        # We killed the job, so we may not even have the tool stdout / tool stderr
        tool_stdout = ""
        tool_stderr = "Job cancelled"
    else:
        # Should we instead just move on ?
        # In the end the only consequence here is that we won't be able to determine
        # if the job failed for known tool reasons (check_tool_output).
        # OTOH I don't know if this can even be reached
        # Deal with it if we ever get reports about this.
        raise

This is that report. The branch is reachable, and not rarely.

Observed

A single ~1 hour window on one deployment (2163 jobs) produced 14 occurrences, on two different runners:

runner	tool	count	exit_code IS NULL	no stderr at all
k8s	mgnify_seqprep	6	6	6
local	__DATA_FETCH__	8	8	8

Every one logs the same thing:

galaxy.jobs.runners ERROR (227/25156) Job wrapper finish method failed
Traceback (most recent call last):
  File "/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 655, in _finish_or_resubmit_job
    with open(tool_stdout_path, "rb") as stdout_file:
FileNotFoundError: [Errno 2] No such file or directory:
  '/galaxy/server/database/jobs_directory/000/227/outputs/tool_stdout'

and every one is stored as:

info       = "Unable to finish job"
exit_code  = NULL
tool_stderr / job_stderr / job_messages = empty

Why this is worth changing

The two groups above have completely different underlying causes, and are indistinguishable in Galaxy:

• mgnify_seqprep (k8s). job_state_history is new → queued → error; the job never reaches running. The container never starts, because the image it requests has no /bin/bash while the runner sets "command": [ajs.job_wrapper.shell]. The real reason is sitting in the Kubernetes event:

  Error: failed to create containerd task: ... runc create failed:
  unable to start container process: error during container init:
  exec: "/bin/bash": stat /bin/bash: no such file or directory
  

  Recoverable via kubectl get events, not from Galaxy.

• __DATA_FETCH__ (local). These do reach running and run to completion. outputs/tool_stdout is created empty at job start by the plain shell redirect (stream_stdout_stderr is off), so it existed and then went away. Cause still unknown — but the raise guarantees no diagnostic survives to help.

So an infrastructure problem, a bad tool container and (presumably) a filesystem problem all render as the same opaque Unable to finish job. That is what makes this expensive: the message is identical no matter what actually went wrong, so triage has to start from the handler log, and only if it hasn't rotated.

Worth noting retry_job_output_collection does not help — the failing read happens in _finish_or_resubmit_job, before job_wrapper.finish(), so the retry never wraps it.

Suggested fix

The alternative the comment itself proposes — "just move on" — would be a large improvement. Treating a missing tool_stdout as empty and continuing lets check_tool_output run against the exit code and the runner's own state, so the job fails with whatever the runner actually knew instead of with a swallowed FileNotFoundError. If the file's absence is itself worth surfacing, recording it in job_messages would keep the signal without destroying the rest.

Environment

• Galaxy 26.1.1 (quay.io/galaxyproject/galaxy-min:26.1-auto), deployed via galaxy-helm on Kubernetes
• Job working directories on NFS
• Reproduced across local and kubernetes runners; the same shape was present in an earlier run a day before

Related

• k8s runner not collecting logs for failed job #8146 — older, k8s-specific report of the same user-visible symptom ("Unable to finish job", logs not collected). Quite possibly the same root cause seen through one runner; this issue is the general form, since the local runner hits it too.