Skip to content

fix(macos): macOS session-user agent silently dies during self-upgrade #204

Description

@impolitepanda

Description

On macOS, in session-user installation mode, the agent can permanently stop running after an upgrade cycle, with no application error log, no crash report, and no automatic restart. The service simply disappears from launchd and never comes back until the user manually reinstalls it.

Environment

  1. OS: macos
  2. Agent version at time of repro: 2.3.5
  3. Install dir: ~/.local/openaev-agent-session
  4. launchd label: io.filigran.openaev-agent-session (LaunchAgent, gui/<uid> domain)

Reproducible steps

Steps to create the smallest reproducible scenario:

  1. install a brand new session agent on a mac
  2. ps aux | grep openaev-agent should list it as running
  3. wait for 32 secs and ps aux | grep openaev-agent again
  4. Agent is now down and will never come back

Expected output

Upgrade should continue and restart the agent

Actual output

Agent is now down and will never come back

Additional information

Root cause

The upgrade job is executed like any other agent job: the agent spawns the upgrade script as a child process via Command::new("bash").spawn() (src/process/agent_exec.rs), with stdout/stderr redirected to /dev/null. This means the upgrade script (and everything it spawns, e.g. curl) inherits the same process group as the agent itself, which is the process supervised by launchd under the LaunchAgent label.

The generated upgrade script (agent-upgrade-session-user.sh, materialized at runtime as runtimes/execution-<job-id>/execution.sh) ends with:

echo "03. Starting agent service"
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/io.filigran.${session_name}.plist || (echo "Fail restarting io.filigran.${session_name}" >&2 && exit 1)
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.filigran.${session_name}.plist

Per launchd.plist(5):

AbandonProcessGroup: If false, the job's process group is sent a SIGTERM followed by a SIGKILL when the job is stopped/removed. If true, the process group is left alone. Default is false.

The generated .plist does not set AbandonProcessGroup, so it defaults to false. As a consequence, when the upgrade script calls launchctl bootout on its own job label, launchd tears down the entire process group — including the very script that issued the bootout command, before it can reach the launchctl bootstrap line right after it.

Result:

  • The service is cleanly removed from launchd (launchd: service inactive: ... / removing service: ... visible in the unified log).
  • The bootstrap call is never reached → the agent never comes back.
  • The script's stdout/stderr were already redirected to /dev/null by the parent agent process, and the kill is an external SIGTERM/SIGKILL (not an application error), so nothing is logged anywhere — no entry in openaev-agent.log, no crash report in DiagnosticReports, empty runner.log (StandardOutPath/StandardErrorPath).

Suggested fixes (any of these would address the root cause)

  1. Set AbandonProcessGroup to true in the generated LaunchAgent/LaunchDaemon .plist templates, so that bootout/bootstrap (or any job-triggered stop) does not tear down descendant processes such as the upgrade script.
  2. Fully detach the upgrade script from the agent's process group before it reaches the bootout step (e.g. setsid/double-fork/nohup at the very start of the generated script), so it survives the parent job being stopped.
  3. Avoid the stop/start round-trip entirely: replace launchctl bootout + launchctl bootstrap with launchctl kickstart -k gui/$(id -u)/io.filigran.${session_name}, which restarts the job in place without ever removing it from launchd's registry, closing the window where the service can vanish and never come back.

Any of the above should also apply to the equivalent LaunchDaemon (system-wide, sudo) upgrade script if it uses the same stop/start pattern (agent-upgrade-daemon.sh or similar) — that variant wasn't repro'd here but shares the same structural risk if it also self-boots-out from within its own process group.

Diagnostic done the following way:

  1. Agent running normally, openaev-agent.log shows a healthy startup and job polling:
{"timestamp":"2026-07-09T13:58:11.691077Z","level":"INFO","fields":{"message":"Starting OpenAEV agent 2.3.5 (production)", ...}}
{"timestamp":"2026-07-09T13:58:11.831818Z","level":"INFO","fields":{"message":"Starting ping thread", ...}}
{"timestamp":"2026-07-09T13:58:11.953229Z","level":"INFO","fields":{"message":"Starting listening jobs thread", ...}}
{"timestamp":"2026-07-09T13:58:12.040431Z","level":"INFO","fields":{"message":"Starting cleanup thread", ...}}
  1. ~32s later (next job poll cycle), an upgrade job is received and dispatched:
{"timestamp":"2026-07-09T13:58:43.438991Z","level":"INFO","fields":{"message":"Start handling inject: None", ...}}
{"timestamp":"2026-07-09T13:58:43.439029Z","level":"INFO","fields":{"message":"Cleaning job: \"30a742f7-2248-427a-9146-655a621c8891\"", ...}}
{"timestamp":"2026-07-09T13:58:43.572804Z","level":"INFO","fields":{"message":"Invoking execution", ...}}
{"timestamp":"2026-07-09T13:58:43.574544Z","level":"INFO","fields":{"message":"Revoking execution", ...}}
{"timestamp":"2026-07-09T13:58:43.574628Z","level":"INFO","fields":{"message":"Done handling inject: None", ...}}

No further log line ever appears — the log file ends here. "Invoking"→"Revoking" happens in ~1.7ms, consistent with a fire-and-forget spawn (the agent does not wait for the child script to finish before marking the job as done).

  1. log show/log stream (unified log) at the corresponding timestamp shows launchd tearing down the job:
launchd: [gui/501 [100023]:] service inactive: io.filigran.openaev-agent-session
launchd: [gui/501 [100023]:] removing service: io.filigran.openaev-agent-session

No subsequent launchd line ever shows the service being re-registered/bootstrapped.

  1. launchctl print gui/<uid>/io.filigran.openaev-agent-session afterwards returns:
Bad request.
Could not find service "io.filigran.openaev-agent-session" in domain for user gui: 501

confirming the job was fully removed from launchd and never came back.

  1. The actual generated execution.sh used at repro time (paths/tenant/token redacted where relevant):
#!/bin/sh
set -e

base_url=http://localhost:3001
architecture=$(uname -m)

install_dir="/Users/yann/.local/openaev-agent-session"
session_name="openaev-agent-session"
tenant_id="b50f11c2-30f8-4055-b181-31df7740996a"

os=$(uname | tr '[:upper:]' '[:lower:]')
if [ "${os}" = "darwin" ]; then
  os="macos"
fi

echo "Starting upgrade script for ${os} | ${architecture}"

openaev_dir=$(printf %s "${install_dir}" | sed 's/openbas/openaev/g')
if [ -d "$openaev_dir" ]; then
  echo "01. Downloading OpenAEV Agent into ${install_dir}..."
  curl -sSfL ${base_url}/api/tenants/${tenant_id}/agent/executable/openaev/${os}/${architecture} -o ${install_dir}/openaev-agent_upgrade
  mv ${install_dir}/openaev-agent_upgrade ${install_dir}/openaev-agent
  chmod +x ${install_dir}/openaev-agent

  echo "02. Updating OpenAEV configuration file"
  cat > ${install_dir}/openaev-agent-config.toml <<EOF
  ...
EOF

  echo "03. Starting agent service"
  launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/io.filigran.${session_name}.plist || (echo "Fail restarting io.filigran.${session_name}" >&2 && exit 1)
  launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.filigran.${session_name}.plist
else
  ...
fi

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions