Skip to content

Test run leaves orphaned processes holding pipes after tests pass #322

Description

@ChrisonSimtian

The test run leaves child processes alive well past Test Run Successful. #321 works around this so CI is green, but the workaround treats the symptom.

How it surfaced

Pinning Fallout to 10.4.0 made CI fail with exit 143 roughly 30 seconds after every test had already passed (41 passed, 5 skipped, Test Run Successful), and with no Fallout target-summary table — so the build never got past the test target.

Fallout waits for the child's output streams to reach EOF. MSBuild's reusable worker nodes and build server outlive dotnet test while still holding the inherited stdout handle, so EOF never arrives, Fallout waits out its grace period, kills the process and surfaces 143.

Evidence

Observation
dotnet test invoked directly, Fallout out of the picture exits on its own — the tests are not what hangs
Same command via ./build.sh TestNoUi on Fallout 10.4.0 hangs until killed
After a killed local run ~45 orphaned dotnet processes plus 2 testhost
One of those testhost processes reached 20.8 GB working set, and held a lock on .nuke/temp/build.log for ~13 minutes before exiting on its own
With MSBUILDDISABLENODEREUSE=1 + DOTNET_CLI_USE_MSBUILD_SERVER=0 CI green, 2m55s

Why the workaround isn't the fix

Disabling the MSBuild daemons stops them holding the pipe, so the build completes. It does not stop processes being left behind — it just stops them mattering to the build. A future tool that also waits on process exit hits the same wall, and the 20 GB figure suggests something worth understanding on its own.

Fallout 11.x tolerated this, which is why it went unnoticed: the repo had been silently resolving unlisted 11.0.8 for months (see #321). The behaviour was always there; only 10.4 surfaced it.

Worth finding out

  • Which test project leaves processes behind? Running each project's dotnet test separately and watching for survivors should isolate it. Satisfactory.Presentation.Api.Tests is the prime suspect — it's last to finish and its shutdown log shows Wolverine and TickerQ still working (Application stopping signal received, TimeTickers UPDATE statements) right up to the end.
  • Are the Wolverine / TickerQ / Aspire hosts in the API test fixtures actually disposed, or does the fixture leave a runtime running between classes?
  • Is the 20 GB testhost a genuine leak, or an artefact of a dev machine? Caveat worth checking first: that measurement is from a machine with both Satisfactory and Captain of Industry installed, so the real-fixture tests actually execute there and CI skips them. Satisfactory.Infrastructure.Tests took 3m39s locally for 53 tests. The number may not reproduce on CI.
  • Once fixed, does the guard in build: Fallout 10.4 across the repo group, pinned exactly #321 become unnecessary? Keeping it is harmless, but the comment explaining it should be updated rather than left describing a solved problem.

Not urgent

CI is green and the workaround is stable. This is worth doing because a build that can't tell "finished" from "still holding a pipe" will bite again somewhere else — not because anything is broken today.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions