Skip to content

ci(experimental): GraalPy workers exhaust their heap in the long shards #2713

Closed

Description

@JarryShaw

On main at 57a6c366d, the GraalPy experimental leg loses xdist workers: 8 crashes ("node down: Not properly terminated"). It is non-blocking, so the job stays green with a warning. Every complete run since 16ccc4373 has lost workers, and a25eb0e39 lost none.

  • Cause (inferred): each GraalPy process runs with --vm.Xmx2g --vm.XX:+ExitOnOutOfMemoryError. Every worker that died was in one of the four long shards (tests/foundation, tests/protocols/internet, tests/protocols/[a-m]*, tests/protocols), at 2.7–2.9 GiB RSS. Those shards have grown since test: sort the remaining GraalPy-only failures into defects and gates #1963 sized them. There was no kernel or scope OOM (oom_kills=0).
  • The crash-attributed tests are victims. All of them pass alone and in earlier GraalPy JUnit artifacts.
  • Fix: split those shards in the SHARDS env of the Run unit tests step, keeping tests/project/test_graalpy_shards_1963.py in step. Re-measure the test: sort the remaining GraalPy-only failures into defects and gates #1963 numbers in workflows.rst. The leg took 4510 s against its 6600 s timeout.

Activity

  1. added this to the PyPCAPKit v1.6 milestone on Nov 3, 2026
  2. added
    ciPull requests that change CI or workflow configuration (ci: subject prefix)
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Nov 3, 2026
  3. JarryShaw commented on Nov 3, 2026

    @JarryShaw
    OwnerAuthor

    The PR is #2720.

  4. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciPull requests that change CI or workflow configuration (ci: subject prefix)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions