fix: raise fuzz-smoke rss_limit_mb 512->2048 (mcb_libfuzzer OOM)

The Clang ASan+UBSan CI job's mcb_libfuzzer hit libFuzzer's OOM guard
(550MB against a 512MB -rss_limit_mb) during the 20s corpus-seeded
smoke run -- newly reached now that the tinyobjloader ODR violation no
longer aborts the job earlier.

Not a per-input security bug: replaying the exact CI-reported "oom-"
artifact (a 172-byte MCB file) 2000x in a single process found no leak
and no crash. Root cause is ordinary coverage-guided fuzzing growth --
libFuzzer retains every corpus entry that finds new coverage in memory
for the whole run, and ASan adds its own instrumentation overhead on
top, so RSS climbs steadily over a run with zero malicious input
involved. Confirmed via local repro with the identical seed/params:
peaked at 505MB and 822MB across two seeds in 20s runs, 668MB over a
longer 40s run, all with zero crashes. mc3_xml_libfuzzer (365MB) and
mc3_json_libfuzzer (282MB) in the same CI run were comfortably under
512MB already, so the shared ceiling only needed raising, not a
per-target split.

Raised the shared --rss-mib ceiling (run_fuzz_smoke.py's own default,
and its explicit CI invocation) from 512 to 2048 -- comfortably above
every local measurement, costs nothing during normal operation (an
abort ceiling, not a reservation). Same fix pattern as
mc3_json_document_budget's CTest TIMEOUT: raise the ceiling to match
legitimate behavior rather than treating a calibration mismatch as a
security finding.

Author: robertvokac

.github/workflows/ci.yml

@@ -214,7 +214,7 @@ jobs:
           build-sanitize/mc3togltf/mc3togltf \
             mc3togltf/test/golden/basic_scene.mc3.xml \
             build-sanitize/fuzz-corpus/glb/basic_scene.glb
-          python3 test/run_fuzz_smoke.py --seconds 20 --rss-mib 512 \
+          python3 test/run_fuzz_smoke.py --seconds 20 --rss-mib 2048 \
             --work-dir build-sanitize/fuzz-work \
             --target build-sanitize/mc3/mc3_xml_libfuzzer mc3/test/fuzz/corpus/xml \
             --target build-sanitize/mc3/mc3_json_libfuzzer mc3/test/fuzz/corpus/json \

test/run_fuzz_smoke.py

@@ -15,7 +15,19 @@ def main() -> int:
     parser = argparse.ArgumentParser()
     parser.add_argument("--seconds", type=int, default=20,
                         help="maximum libFuzzer run time per target")
-    parser.add_argument("--rss-mib", type=int, default=512,
+    # 512 (the original default) is too tight: a normal, non-malicious ASan
+    # run legitimately grows RSS well past it just from libFuzzer's own
+    # retained in-memory corpus (it keeps every input that finds new
+    # coverage) plus ASan's own instrumentation overhead -- confirmed via
+    # local repro (peaked at 505MB/20s and 822MB/20s across two seeds, 668MB
+    # over a longer 40s run, zero crashes/leaks in any of them), matching a
+    # CI run that hit 550MB and got killed by the old 512MB ceiling. Not a
+    # per-input security bug (replaying the exact CI-reported "oom-" input
+    # 2000x in one process found no leak). 2048 leaves real headroom above
+    # the highest local measurement rather than just clearing one observed
+    # number -- raise the ceiling, don't shrink real fuzzing coverage, same
+    # fix pattern as mc3_json_document_budget's CTest TIMEOUT.
+    parser.add_argument("--rss-mib", type=int, default=2048,
                         help="libFuzzer resident-memory ceiling per target")
     parser.add_argument("--work-dir", type=Path, required=True,
                         help="generated working corpora and crash artifacts; never a source corpus")