Skip to content

fix: don't crash when the span cache can't write its temp file - #2804

Open
George Shostakovych (gshost1) wants to merge 1 commit into
braintrustdata:mainfrom
gshost1:fix/span-cache-flush-errors
Open

George Shostakovych (gshost1) wants to merge 1 commit into
braintrustdata:mainfrom
gshost1:fix/span-cache-flush-errors

Conversation

@gshost1

Copy link
Copy Markdown

Fixes #2803

Summary

SpanCache.queueWrite() kicks off this.flushWriteBuffer() in the background and stores the promise on this.flushPromise without a .catch() handler. During an experiment-backed Eval(), nothing awaits flushPromise, so if opening or appending to the temp file fails (ENOENT on a missing TMPDIR, read-only container rootfs, ENOSPC/EIO), Node terminates the process with an unhandled promise rejection.

This catches initialization and append errors inside flushWriteBuffer(), logs via debugLogger.warn, resets this.initPromise = null when initialization fails so later writes don't stay permanently stuck on a cached rejected promise, and leaves unwritten spans in writeBuffer so getByRootSpanId() continues serving them from memory.

Verification

  • Added js/src/span-cache-flush-failure.test.ts (2 tests covering missing TMPDIR unhandled-rejection suppression + in-memory read fallback; fails on main, passes with fix)
  • cd js && pnpm exec vp test run across span-cache, trace, framework, framework2, and agent-assertions: 209 passed, 0 failed
  • cd js && pnpm exec tsc --noEmit: clean
  • Includes patch changeset (.changeset/span-cache-flush-errors.md)

(Note on interaction with #2767: if merged alongside #2767's serialized drain continuation, the .then() continuation there should avoid immediately re-invoking flushWriteBuffer() when a flush returns early on disk error so unwritten records don't trigger a tight retry loop.)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SpanCache background flush errors are unhandled, crashing Eval() when the temp file cannot be written

1 participant