Skip to content

Commit dfa69f3

Browse files
MDBF-1194: Reduce the memory pressure of hz-bbw8 and hz-bbw9
During the MTR phase of amd64-msan-clang-20-debug, the "MTR normal" step intermittently reports success even when errors are present in the stdio output. Observed symptoms: - truncated MTR logs in stdio - step completes with exit code 0 despite failures System logs show evidence of OOM killer activity during these runs. In particular, addr2line processes spawned while generating backtraces are being killed: ``` Mar 18 02:55:49 hz-bbw8 kernel: checkpoint_bg invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 Mar 18 02:55:49 hz-bbw8 kernel: oom_kill_process.cold+0xb/0x10 Mar 18 02:55:49 hz-bbw8 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Mar 18 02:55:50 hz-bbw8 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-15b53e20d1e4f7dba6166ebce83a54eaa35e13a2460cd0647ef632639ad6a95b.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-15b53e20d1e4f7dba6166ebce83a54eaa35e13a2460cd0647ef632639ad6a95b.scope,task=addr2line,pid=212899,uid=1000 Mar 18 02:55:50 hz-bbw8 kernel: Out of memory: Killed process 212899 (addr2line) total-vm:14357312kB, anon-rss:14320664kB, file-rss:3840kB, shmem-rss:0kB, UID:1000 pgtables:28136kB oom_score_adj:0 Mar 18 02:55:52 hz-bbw8 kernel: oom_reaper: reaped process 212899 (addr2line), now anon-rss:0kB, file-rss:3840kB, shmem-rss:0kB ``` The fact that most of the runs end while attempting a backtrace is not a coincidence. ``` Retrying test innodb_fts.innodb_fts_large_records, attempt(2/3)... ***Warnings generated in error logs during shutdown after running tests: innodb_fts.innodb_fts_large_records 260318 10:24:04 [ERROR] /home/buildbot/bld/sql/mariadbd got signal 6 ; Attempting backtrace. Include this in the bug report. ``` This suggests that MTR does not properly propagate failures when child processes are terminated by the OOM killer, resulting in the main perl process exiting with code 0. This change documents the issue and is an attempt at avoiding the issue by reducing the number of parallel workers at runtime.
1 parent 4f61aaf commit dfa69f3

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

master-migration/master.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ def msan_builder(name: str, debug: bool) -> GenericBuilder:
193193
jobs = 12
194194
if debug:
195195
tag_msan = tags_msan + ("debug",)
196-
jobs = 20
196+
jobs = 15
197197

198198
return GenericBuilder(
199199
name=name,

0 commit comments

Comments
 (0)