Skip to content

Make a failed priority-queue growth fatal instead of corrupting the heap - #334

Merged
1a1a11a merged 1 commit into
developfrom
claude/pqueue-growth-failure-guard
Sep 15, 2026
Merged

1a1a11a merged 1 commit into
developfrom
claude/pqueue-growth-failure-guard

Conversation

@1a1a11a

@1a1a11a 1a1a11a commented Sep 15, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Follow-up to #333, which merged without this guard — the review that found the problem landed while the merge was in flight, so develop now has the geometric growth policy and not the check that makes it safe.

pqueue_insert returns 1 when its realloc fails, and the header documents "0 on success", but no caller in the tree checks it — Size.c:181, Belady.c:201, PG.c:344 and PG.c:399 all discard the return. They could not act on it if they did:

  • Size_insert calls cache_insert_base first, so the object is already in the cache.
  • It then allocates the node with my_malloc — plain malloc, not zeroed.
  • It assigns cached_obj->Size.pq_node = node whether or not the insert took.
  • bubble_up never ran, so node->pos holds whatever was in that heap cell.
  • Size_remove_obj sees a non-NULL pq_node and calls pqueue_remove:
size_t posn = q->getpos(d);
q->d[posn] = q->d[--q->size];

An unbounded write at that uninitialised index. Not a wrong miss ratio — a heap out-of-bounds write.

The path predates #333, but eight million entries of slack stood in front of it. Growth is how the queue reaches its working size now, so it becomes reachable from roughly 65K objects onward, and specifically under the virtual-memory limits the smaller reservation was meant to fit inside.

This terminates instead, which is how create_chained_hashtable_v2 already handles a failed table allocation, and which covers PG's two call sites by the same guard. Checking at the callers was the alternative and does not work: by the time the insert is attempted the object is in the cache, so there is no atomic recovery short of unwinding cache_insert_base.

Found by Codex review on #333.

Type of change

  • Bug fix
  • New eviction / admission / prefetch algorithm
  • New trace reader or trace format
  • Performance improvement
  • Documentation
  • Build / CI
  • Other:

How was it tested?

cd _build
ctest --output-on-failure      # 10/10
bash ../test/test_cli.sh       # 185 passed, 0 failed

The failure path cannot be reached with the sample traces — cloudPhysicsIO's working set is ~49K objects, which fits inside the 65536-entry reservation, so the queue never grows. I verified it by fault injection instead, forcing the third growth's realloc to return NULL:

[ERROR] pqueue.c:168 cannot grow priority queue to 136 entries (1088 bytes)
rc=134

instead of continuing into the corrupting path. The injection was reverted before committing.

  • ctest --test-dir _build --output-on-failure passes
  • Build is warning-free (CI uses -Wall -Wextra -Werror)

Results

No change to memory or miss ratios — this only replaces an unhandled error return with a diagnostic. The size and belady figures from #333 are unaffected.

Checklist

  • Formatted with clang-format (or the pre-commit hook from scripts/setup_hooks.sh)
  • Added or updated tests — the path needs an allocation failure, which test_cli.sh cannot induce portably; fault injection is described above
  • Added or updated documentation — no user-facing interface changes
  • New algorithms are registered in the CLI and listed in the README — not applicable

🤖 Generated with Claude Code

https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc


Generated by Claude Code

…he heap

#333 changed pqueue to grow geometrically so Size and Belady could stop
reserving 64MB apiece. It merged without this guard -- the review that found
the problem landed while the merge was in flight -- so develop now has the
growth policy and not the check.

pqueue_insert returns 1 when the realloc fails, and the header documents "0
on success", but no caller in the tree checks it: Size.c:181, Belady.c:201,
PG.c:344 and PG.c:399 all discard the return. They could not act on it if
they did. Size_insert calls cache_insert_base first, so the object is already
in the cache, then allocates the node with my_malloc -- plain malloc, not
zeroed -- and assigns cached_obj->Size.pq_node = node whether or not the
insert took. bubble_up never ran, so node->pos holds whatever was in that
heap cell. Size_remove_obj sees a non-NULL pq_node and calls pqueue_remove:

    size_t posn = q->getpos(d);
    q->d[posn] = q->d[--q->size];

an unbounded write at that uninitialised index. Not a wrong miss ratio -- a
heap out-of-bounds write.

The path predates #333, but 8 million entries of slack stood in front of it.
Growth is how the queue reaches its working size now, so it becomes reachable
from about 65K objects onward, and specifically under the virtual-memory
limits the smaller reservation was meant to fit inside.

Terminate instead, which is how create_chained_hashtable_v2 already handles a
failed table allocation, and which covers PG's two call sites by the same
guard. Checking at the callers was the alternative and does not work: by the
time the insert is attempted the object is in the cache, so there is no
atomic recovery short of unwinding cache_insert_base.

Verified by injecting a failed realloc on the third growth: the run aborts
with "cannot grow priority queue to 136 entries (1088 bytes)" and rc=134
rather than continuing into the corrupting path. Suite passes 10/10 with 185
CLI checks, and neither memory nor miss ratios move.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Copilot AI lite review requested due to automatic review settings September 15, 2026 01:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-15T01:45:29.572202Z 00e3e5f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@1a1a11a
1a1a11a merged commit 9fc2edd into develop Sep 15, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants