[Bug]Scheduler hang: all tasks asleep after fork, CPU stuck in scheduler path (__schedule_inner / switch_finish_hook, 7/8 lmbench runs)

[mistcoversmyeyes]

Scheduler hang: all tasks asleep after fork, CPU stuck in scheduler path

Symptom

Running lmbench (full suite or single test) in a QEMU+KVM guest, user-space fork-related operations hang: the serial goes silent, no output at all, all tasks fall asleep (gdb confirms CPU0 idle, CPU1 stuck in the scheduler), and QEMU keeps consuming CPU. The hang location is random — the ENOUGH-precompute fork, pipe-test fork, semaphore-test fork, and lat_fs fork all triggered it — with very high probability.

Reproduction (7/8 runs, same kernel 671b758 / clean vfat image)

Run	Scenario	Hang point	Behavior
R1	full e2e	ENOUGH precompute after init (timeout ... enough fork)	600s silent, 0 metrics
R2	full e2e	same	600s silent, 0 metrics
R3	full e2e	pipe_lat (lat_pipe benchmp fork)	3 samples all rc=124 (fork worker hung)
R4	--only ramfs	ENOUGH fork after init	20min silent, 0 metrics
R5	--only ramfs (ENOUGH preset)	lat_fs startup fork (inside run_with_timeout)	silent after RUN log, no sample-timeout logs
R6	pressure (init + bw_mem/...)	first pressure-test fork	6min silent; gdb: switch_finish_hook
R7	--only ramfs	ENOUGH fork	20min silent, 0 metrics

(Plus one gdb diagnostic run with an ENOUGH deadlock and one pipe_lat hang, not counted.)

gdb snapshots (QEMU -gdb attach, while hung)

Snapshot 1 (R1/R2):

Thread 2 (CPU#1 [running]):
#0 current_cpu_id
#4 core::ptr::drop_in_place<SpinLockGuard>
#8 dragonos_kernel::sched::__schedule_inner
Thread 1 (CPU#0 [running]):
#0 current_cpu_id   (idle/scheduler entry)

CPU1 stuck in the scheduler __schedule_inner lock operation (SpinLockGuard lifetime), CPU0 idle. All user-space tasks asleep (the fork caller never returns).

Snapshot 2 (R6):

Thread 2 (CPU#1 [halted]):
#0 arch_idle_func
#9 sched::completion::Completion::complete
Thread 1 (CPU#0 [running]):
#0 current_cpu_id
#1 ProcessManager::switch_finish_hook

CPU0 stuck in the process-switch finish hook switch_finish_hook; CPU1 waiting on a completion. Same scheduler subsystem, different function — the hang point drifts, the trigger (fork) is consistent.

Initial analysis

• Trigger is fork (sys_fork → scheduler enqueue → __schedule_inner / switch_finish_hook)
• Location random: ENOUGH precompute / pipe / semaphore / lat_fs / process_fork all triggered — any user-space fork can hit it
• Related code: kernel/src/sched/ (__schedule_inner, scheduler locks + SpinLockGuard); kernel/src/arch/x86_64/process/idle.rs:14-31 (idle loop); ProcessManager::switch_finish_hook
• Was misattributed to vfat corruption (power-loss damage on the old image amplified it to "always hangs"); after the vfat fix the issue exists independently (clean vfat, 7/8 runs)
• Probability observation: the first run after an image rebuild can complete 17-32 metrics before hanging; later runs often hang on the first fork — some state-accumulation effect

Relation to tmpfs/lat_fs slowness

• lat_fs run directly (fresh boot) is normal: 0k/1k/4k/10k all produce values (8724/8842/7727/8818 ops)
• lat_fs run 3x consecutively (fresh boot) is normal: all 3 complete (2nd run shows 1k dropping 9128→3874 — early sign of state accumulation)
• After a full e2e of 32 metrics (see 调度器死锁：fork 后所有任务睡眠、CPU0 卡 idle、CPU1 卡 pick_next_task（lmbench 全量 5/5 复现） #2167): ramfs 0k 3x timeout (lat_fs slow) + 10k hang — the only tmpfs-related stall observed

Conclusion: the tmpfs/lat_fs slowness is not an independent bug — it is the milder manifestation of the same kernel state-accumulation problem (fork hang is the severe one). Under stress (long runtime + heavy fork/memory/file activity) FS operations slow down first, then fork hangs. Because the fork-hang probability is very high (7/8 runs), the tmpfs slowness cannot be reproduced 3x independently.

Impact

The lmbench full suite cannot complete reliably (every run hangs at some fork point); any fork-dependent workload (shell scripts, test suites) is at risk. The vfat conclusion in #2167 needs correction: vfat corruption is an aggravating factor; this issue is the independent root cause.

Environment

• QEMU + KVM (WSL2 / Linux 6.6 host), 2 vCPU (q35, hpet=off), 2G RAM, virtio-blk, user-mode NAT
• DragonOS feat/lmbench @ 671b758