Skip to content

Integration tests are killed with exit 137 when the branch name contains a daemon name #832

Description

@anromerom

Description

Any Linux, manager or macOS IT module that stops a daemon individually dies mid-run when the branch under test is named after a Wazuh daemon. The pytest session receives SIGKILL, so the job ends with exit code 137, no failed assertion, no traceback and no results.html, and the run reads as if the change under review broke the tests.

The branch fix/38456-wazuh-agentd-memory-footprint-regression-in-500 (wazuh/wazuh#38489) reproduced it on every one of its 6 runs. Renaming the branch to fix/38456-agentd-memory-footprint-regression-in-500, with the same commit, is the whole workaround.

Evidence

Job Progress reached Result
IT Linux - agentd (tier-0-1) 5/19 tests passed (26 %) Killed, exit 137
IT Linux - enrollment (tier-0-1) 30 collected, 0 reported Killed, exit 137
2026-08-21T16:45:03Z /home/runner/work/_temp/c85549f5-....sh: line 8:  3808 Killed  sudo GITHUB_SHA="bec3c4e9..." GITHUB_REF_NAME="${BRANCH_NAME}" ... python -m pytest --tier 0 --tier 1 test_agentd/ --html=results.html --self-contained-html
2026-08-21T16:45:08Z ##[error]Process completed with exit code 137
2026-08-21T16:45:09Z ##[warning]No files found for path './tests/integration/results.html'; skipping upload of artifact
  • Runs: 32500969840, 32499584257, 32496282817, 32493881933.
  • The kill point is deterministic. Enrollment dies inside its first test, where the module-scoped fixture stops AGENT_DAEMON. Agentd passes the five test_reconnection files, which use the service-level control_service('stop'), and dies on the sixth test, the first one that stops daemons one by one.
  • The Windows IT jobs pass on the same commits.

Probable cause

On its Unix path, control_service(action, daemon=...) in src/wazuh_testing/utils/services.py picks its victims by substring over the whole command line:

elif daemon in proc.name() or daemon in ' '.join(proc.cmdline()):
    processes.append(proc)
...
for proc in processes:
    proc.terminate()
_, alive = psutil.wait_procs(processes, timeout=5)
for proc in alive:
    proc.kill()

The IT workflows launch pytest through sudo with the branch name as an environment assignment, so the branch name is part of sudo's argv and matches that substring test. sudo is then terminated and SIGKILLed, and pytest, its child, goes with it. The Windows path of control_service is unaffected because it matches the WazuhSvc process name only.

The same matcher will select any other process that merely mentions a daemon on its command line, for instance a tail -f on the daemon's log or an editor with wazuh-agentd.c open, and the suite runs as root.

Acceptance criteria

  • Stopping a daemon affects only processes that are that daemon, never a process that merely names it.
  • The pytest session cannot be killed by its own daemon handling, whatever the branch is called.
  • An IT run on a branch named after a daemon (e.g. test/wazuh-agentd-naming) completes and reports test results.
  • Per-daemon stop behaviour is unchanged for every daemon currently supported, including those resolved through their pidfile (wazuh-manager-clusterd, wazuh-manager-apid).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions