Instrument the client where the hang actually reproduces

The three-corner comparison put this in the certificate path: forty attempts of
the same five hostile modes reproduced nothing on upstream and nothing on this
branch with public-key targets, while the certificate suite reproduces on the
first iteration.

So trace the client here rather than in the probe that hits once in sixty. The
gateway's log already shows it refusing the target and closing the session
within a second; what is missing is which message the client never got.

Fork-only: this edits the suite's connect() and is not for PR warp-tech#2397.

Author: janisdombr

.github/workflows/hostile-hunt.yml

@@ -62,7 +62,7 @@ jobs:
             echo "::group::iteration $i"
             TIMEOUT=120 poetry run pytest \
               test_vault_hostile_target.py test_vault_hostile_certs.py \
-              -q --tb=short -p no:randomly || { echo "reproduced on iteration $i"; exit 1; }
+              -q --tb=short -s -p no:randomly || { echo "reproduced on iteration $i"; exit 1; }
             echo "::endgroup::"
           done
           echo "twenty iterations, no reproduction"

tests/test_ssh_target_cert_auth.py

@@ -14,6 +14,8 @@
 from uuid import uuid4
 
 import psutil
+import subprocess
+
 import pytest
 import yaml
 
@@ -149,8 +151,20 @@ def start(processes: ProcessManager, wg: WarpgateProcess, user, target, *extra):
 
 
 def connect(processes: ProcessManager, wg: WarpgateProcess, user, target, timeout, *extra):
-    client = start(processes, wg, user, target, *extra)
-    stdout = client.communicate(timeout=timeout)[0]
+    client = start(processes, wg, user, target, "-v", *extra)
+    try:
+        stdout = client.communicate(timeout=timeout)[0]
+    except subprocess.TimeoutExpired:
+        # Fork-only instrumentation. The gateway's log already shows it closing
+        # the session in the same second; this is the other half — the stage the
+        # client was still waiting at when its patience ran out.
+        client.kill()
+        out, err = client.communicate(timeout=10)
+        print("---- ssh stdout ----")
+        print(out.decode(errors="replace")[-2000:])
+        print("---- ssh stderr ----")
+        print("\n".join(err.decode(errors="replace").splitlines()[-80:]))
+        raise
     return client.returncode, stdout