/ports saturates the default asyncio thread pool on hosts with many processes, wedging /execute and leaking zombie children

[mhlas7]

Summary

GET /ports resolves each listening socket to a PID by walking every /proc/<pid>/fd/ entry, and repeats that entire walk once per listening socket. On a host with many processes and many listening sockets this takes minutes of wall-clock time, and it runs on asyncio's default ThreadPoolExecutor.

Open WebUI polls GET /ports roughly every 15 seconds. Because each call takes far longer than the poll interval, calls accumulate, each holding one pool thread. The default pool is min(32, cpu_count + 4) threads, so it saturates within a few minutes. Once it does, the entire server wedges: POST /execute never returns, spawned children are never reaped and pile up as zombies, and process log files stay empty at 0 bytes.

The failure looks like a bug in command execution, but /execute is only the victim. It shares the default executor with /ports.

Environment

• open-terminal 0.11.34 (pipx, PyPI)
• Python 3.13.5, Linux 6.x, single-user mode, running as root under systemd
• 6 CPUs, so the default asyncio executor is min(32, 6 + 4) = 10 threads
• Open WebUI 0.11.1, terminal server configured as an admin/system connection so the backend proxies and polls it

Root cause

open_terminal/utils/port.py, _pid_from_inode scans all of /proc looking for one socket inode:

def _pid_from_inode(inode: str) -> int | None:
    """Resolve a socket inode to a PID by scanning /proc/*/fd/."""
    target = f"socket:[{inode}]"
    for pid_dir in os.listdir("/proc"):
        if not pid_dir.isdigit():
            continue
        fd_dir = f"/proc/{pid_dir}/fd"
        try:
            for fd in os.listdir(fd_dir):
                try:
                    link = os.readlink(f"{fd_dir}/{fd}")
                    if link == target:
                        return int(pid_dir)
                except (OSError, ValueError):
                    continue
        except PermissionError:
            continue
    return None

_parse_proc_net_tcp then calls it inside the per-socket loop:

inode = parts[9] if len(parts) > 9 else ""
pid = _pid_from_inode(inode) if inode else None

So the cost is O(listening_sockets × total_fds) rather than O(total_fds). The scan does short-circuit via return int(pid_dir) once a match is found, so the worst case is not reached, but the repeated walk is still expensive. Measured on the affected host:

processes=792  total_fds=14017  listening_sockets=111

detect_listening_ports(), single call, otherwise-idle server:
  original:  3.8s    (56 ports resolved)
  patched:   0.092s  (56 ports resolved, identical pid/process values)

A ~41x reduction. Over HTTP the patched GET /ports returns in 0.14s.

3.8s per call does not by itself exhaust a 10-thread pool at a 15s poll interval. What makes it fatal is concurrency: the thread dump below caught at least five detect_listening_ports calls in flight simultaneously. This is a pure-Python loop issuing millions of syscalls, so overlapping calls contend heavily on the GIL and each one slows down as more pile up — which is the feedback loop into saturation. I have not instrumented that amplification precisely; what is certain is the five concurrent calls in the dump, and that fixing the per-socket rescan eliminates the failure entirely (see Verification).

Evidence

Started with PYTHONFAULTHANDLER=1, reproduced the hang, then sent SIGABRT to dump every thread stack. Five worker threads were in the same place:

Thread 0x00007ffa427316c0 (most recent call first):
  File ".../open_terminal/utils/port.py", line 55 in _pid_from_inode
  File ".../open_terminal/utils/port.py", line 34 in _parse_proc_net_tcp
  File ".../open_terminal/utils/port.py", line 144 in detect_listening_ports
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59 in run
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 93 in _worker
  File "/usr/lib/python3.13/threading.py", line 994 in run
  File "/usr/lib/python3.13/threading.py", line 1043 in _bootstrap_inner
  File "/usr/lib/python3.13/threading.py", line 1014 in _bootstrap

Notably, no thread was blocked in a read syscall. /proc/<pid>/task/*/syscall showed the workers in futex (202), waiting for pool capacity — the PTY read had not started yet.

Cascade

Once the pool is saturated:

1. PtyRunner.read_output calls loop.run_in_executor(None, os.read, self._master_fd, 4096) on the same default pool. It never gets a thread, so the PTY master is never drained.
2. The child sh finishes and exits, but runner.wait() is only reached after read_output returns, so the child is never reaped. Zombies accumulate one per /execute, along with one leaked PTY master FD each.
3. aiofiles also uses the default executor, so the per-process .jsonl log never receives even its "start" record and stays 0 bytes.
4. read_log needs a pool thread too, so POST /execute never returns at all — execute_timeout does not bound it, because the timeout only wraps the log task, not the response path.

Observed on the affected host: 7 zombie [sh] <defunct> children and 7 leaked /dev/ptmx FDs held by the server process, with every endpoint hanging. The process also stopped responding to SIGTERM, so systemctl restart blocked until the 90s TimeoutStopSec elapsed and systemd sent SIGKILL.

Reproduction

1. Run open-terminal on a host with a large process/FD count and many listening sockets — a Docker host works well. Roughly 800 processes, 14k FDs, and 100+ listening sockets reproduces it reliably.
2. Connect it to Open WebUI as an admin/system terminal server so the backend polls GET /ports on its own.
3. Wait 2–3 minutes.
4. POST /execute with any command, for example {"command": "echo hello"}. It never returns.
5. ps --ppid <server_pid> shows [sh] <defunct> accumulating, one per attempt.

Early in the window the degradation is visible before the hard hang — the first /execute took 29 seconds and returned status: "running" with empty output.

An instance on a port Open WebUI is not polling handles the identical commands correctly (status: "done", exit_code: 0, output present, no zombies), which isolates /ports as the trigger.

Suggested fix

Build the inode → PID map once per call instead of once per socket. This turns ~1.5M readlink calls into ~14k on the host above:

def _build_inode_map() -> dict[str, int]:
    """Map socket inode -> pid in a single pass over /proc."""
    mapping: dict[str, int] = {}
    for pid_dir in os.listdir("/proc"):
        if not pid_dir.isdigit():
            continue
        fd_dir = f"/proc/{pid_dir}/fd"
        try:
            for fd in os.listdir(fd_dir):
                try:
                    link = os.readlink(f"{fd_dir}/{fd}")
                except (OSError, ValueError):
                    continue
                if link.startswith("socket:["):
                    mapping[link[8:-1]] = int(pid_dir)
        except (PermissionError, FileNotFoundError):
            continue
    return mapping

_parse_proc_net_tcp builds it once and does dict lookups in the loop.

Verification

I applied exactly this change to a local 0.11.34 install and restarted the server, leaving Open WebUI polling normally throughout.

• detect_listening_ports() went from 3.8s to 0.092s, returning the same 56 ports with identical pid/process values.
• The command that previously hung every time — {"command": "ls", "cwd": "/opt/docker/apps"} — now returns status: "done", exit_code: 0, with full output.
• 16 consecutive /execute calls over ~4.5 minutes: all status: "done", zero zombies, zero leaked PTY master FDs.
• Server thread count settled at 7, down from 17, since the pool no longer needs to grow.

Before the patch, the same server wedged after a single /execute and never recovered.

Two further hardening suggestions, independent of the above:

• Run PTY reads on a dedicated executor rather than the default one, so a slow endpoint can never starve command execution. The current coupling means any long-running run_in_executor(None, ...) call can wedge the whole server.
• Consider a short TTL cache on /ports, since clients poll it on a fixed interval and the answer rarely changes between polls.