LocalProcessBackend crashes on Windows due to bash PATH resolution conflict (WSL relay vs Git Bash)

[kingabdulsalamsas111-lgtm]

What happened?

When running a training job locally via LocalProcessBackend on Windows, the job crashes with a misleading error unrelated to the actual training code:

The RPC call contains a handle that differs from the declared handle type.
Error code: Bash/Service/0x8007072c

Root Cause

LocalProcessBackend` invokes the training command via subprocess using the bare string "bash":

https://github.com/kubeflow/sdk/blob/main/kubeflow/trainer/backends/localprocess/utils.py (get_local_train_job_script):

return "bash", "-c", command

On Windows, "bash" resolves to whichever bash.exe is first in PATH. It is common for Windows systems to have multiple bash.exe binaries:

• C:\Windows\System32\bash.exe — a WSL relay launcher
• C:\Program Files\Git\usr\bin\bash.exe — Git Bash

If the WSL relay is earlier in PATH (a common default) and the WSL distro is not in a valid/running state, the subprocess call fails immediately with the cryptic RPC error above — even if a fully working Git Bash install is present on the same machine.

Reproduction Steps

1. On Windows, ensure both WSL (in any state) and Git Bash are installed.

2. Confirm PATH order causes bash to resolve to the WSL relay:
   (powershell)
   where.exe bash

   C:\Windows\System32\bash.exe <- resolves first

   C:\Program Files\Git\usr\bin\bash.exe

3. Follow the official examples/pytorch/image-classification/mnist.ipynb notebook using LocalProcessBackend:

   from kubeflow.trainer import CustomTrainer, TrainerClient, LocalProcessBackendConfig
   
   client = TrainerClient(backend_config=LocalProcessBackendConfig())
   for runtime in client.list_runtimes():
       if runtime.name == "torch-distributed":
           torch_runtime = runtime
           break
   
   job_name = client.train(trainer=CustomTrainer(func=train_fn), runtime=torch_runtime)
   for log in client.get_job_logs(job_name, follow=True):
       print(log, end="")

4. Observe the job fails immediately with:

   The RPC call contains a handle that differs from the declared handle type.
   Error code: Bash/Service/0x8007072c
   

Verification

Confirmed the two bash.exe binaries behave differently when invoked directly:

# WSL relay - fails
> bash --version
<3>WSL (10 - Relay) ERROR: CreateProcessCommon:818: execvpe(/bin/bash) failed: No such file or directory

# Git Bash - works
> & "C:\Program Files\Git\usr\bin\bash.exe" --version
GNU bash, version 5.3.9(1)-release (x86_64-pc-cygwin)


## Expected Behavior

`LocalProcessBackend` should resolve a working shell deterministically rather than relying on unqualified PATH lookup — for example, using `shutil.which("bash")` with validation, or falling back through known-good bash locations on Windows.

## Environment

- OS: Windows
- Kubeflow SDK version: 0.4.1
- Backend: LocalProcessBackend
- Tested against: kubeflow/trainer main branch (current with v2.3.0 timeframe)

## Impact

Any Windows user following the official example notebooks with `LocalProcessBackend` who has multiple `bash.exe` installations on PATH (common with Git for Windows installed) will hit this crash, with an error message that gives no indication the actual cause is bash resolution.


### What did you expect to happen?

LocalProcessBackend should successfully start the training process on Windows when a valid Bash installation is available. If multiple Bash installations are present, it should resolve a usable Bash executable reliably or provide a clear error indicating the required Bash configuration, rather than failing with the misleading Bash/Service/0x8007072c error

### Environment

Kubernetes version:
```bash
$ kubectl version

Kubeflow Trainer version:

$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

Kubeflow Python SDK version:

$ pip show kubeflow

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[kingabdulsalamsas111-lgtm]

@robert-bell Sir, could you please review the issue and, if you consider it valid, assign it to me? I’d be happy to investigate it further and work on a fix.

[robert-bell]

Thanks @kingabdulsalamsas111-lgtm please do investigate and feel free to propose as fix.

/assign @kingabdulsalamsas111-lgtm

[kingabdulsalamsas111-lgtm]

Thank you! @robert-bell I thoroughly investigated both issues, worked on the fixes, and also addressed a couple of smaller related issues along the way.

The fixes are now ready, and I’m just waiting for your review. Please take a look when you get a chance. Thank you for your guidance and review!