Skip to content

fix(reader): fail ReadNext once the reader's session is retired - #278

Open
tinswzy wants to merge 1 commit into
masterfrom
fix/reader-read-after-close-277
Open

fix(reader): fail ReadNext once the reader's session is retired#278
tinswzy wants to merge 1 commit into
masterfrom
fix/reader-read-after-close-277

Conversation

@tinswzy

@tinswzy tinswzy commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Closes #277.

Problem

ReadNext kept serving entries after the reader was closed:

reader.Close(ctx)                  // session retired, temp info key deleted
msg, err := reader.ReadNext(ctx)   // err == nil, msg is a real message

logBatchReaderImpl carries no closed state and nothing on the read path consulted the session it holds, so a use-after-close read on silently — including draining whatever was already in the cached batch. The only code that noticed was the position report, whose refusal was swallowed into a Warn.

That is worse than a lost report. A retired reader is gone from GetAllReaderTempInfoForLog, so the writer's cleanup no longer protects the segments it is still reading. If a writer elsewhere truncates and cleans, findNextReadableSegment silently advances past the cleaned range and the application loses entries with no error at all.

Why failing fast is the right response

A retired session is terminal. entry.closed is set in exactly one place, retireReaderTempSession, reached from DeleteReaderTempInfo (the reader closing itself) and from provider Close(). Nothing ever clears it. Reopening is not a recovery path either — when the cause is provider Close(), CreateReaderTempInfo refuses.

So ReadNext now returns werr.ErrLogReaderClosed, matching what the equivalent APIs do (os.File.ReadErrClosed, sql.DBErrConnDone). That error already existed at common/werr/errors.go:81, was unused anywhere in the repo, and is already marked non-retryable — the semantics line up exactly.

The check sits after the context check and ahead of the cached-batch fast path, so a dead reader cannot drain what it prefetched. IsActive() is used rather than a reader-local closed flag because it covers both terminal causes with one check, and this is the purpose it was added for — it had no production caller until now.

Milvus is unaffected

Verified end to end. Milvus closes strictly inside-out:

openerAdaptorImpl.Close()
├─ walInstances.Range → wal.Close()
│    └─ walAdaptorImpl.Close()
│         ├─ scanners.Range → scanner.Close()
│         │    ├─ ScannerHelper.Close()   blocks until executeConsumer exits
│         │    └─ reader.Close()          no ReadNext in flight here
│         └─ rwWALImpls.Close()           writer + log handle
└─ openerCache → opener.Close()           only now: woodpecker client → provider

ScannerHelper.Close() blocks on BlockAndGetResult() until the consuming goroutine calls Finish, so no ReadNext can be running against a retired session, and the provider is only closed once every reader and writer already is. Milvus therefore never observes the old behaviour and will not observe the new error.

The point is that this ordering is a discipline the caller keeps, not something the API enforced or documented.

Contract change

ReadNext gains an error it has never returned. Callers that treat any non-context error from it as terminal — Milvus's scannerImpl.executeConsumer calls s.Finish(err) — will surface it as a scanner failure rather than reading on silently. That is the intended behaviour, and per the ordering above it cannot fire in Milvus today.

Test literals

Tests in this package build logBatchReaderImpl directly and left readerTempSession nil, which was invisible while nothing dereferenced it. Production readers always have one — NewLogBatchReader rejects nil — so the 34 literals now set one, rather than the read path tolerating nil.

Testing

  • Both new tests verified failing first. TestLogReader_ReadNext_RetiredSessionDoesNotDrainCachedBatch returned a real LogMessage before the fix, which is the bug in one line of output.
  • go test -race -short ./woodpecker/log/ ./meta/ — pass
  • golangci-lint run ./woodpecker/log/... — 0 issues

Closes #277.

ReadNext kept serving entries after the reader was closed. logBatchReaderImpl
carries no closed state, and nothing on the read path consulted the session it
holds, so a use-after-close read on silently - including draining whatever was
already in the cached batch. The only code that noticed was the position
report, and UpdateReaderTempInfo's refusal was swallowed into a Warn.

That is worse than a lost report. A retired reader is gone from
GetAllReaderTempInfoForLog, so the writer's cleanup no longer protects the
segments it is still reading; if a writer elsewhere truncates and cleans,
findNextReadableSegment silently advances past the cleaned range and the
application loses entries with no error at all.

A retired session is terminal. entry.closed is set in one place,
retireReaderTempSession, reached from DeleteReaderTempInfo (the reader closing)
and from provider Close(); nothing ever clears it. Reopening is not a recovery
either - when the cause is provider Close(), CreateReaderTempInfo refuses.

So fail fast, the way os.File returns ErrClosed and sql.DB returns ErrConnDone.
werr.ErrLogReaderClosed already existed for exactly this, unused, and is
already marked non-retryable. The check sits after the context check and ahead
of the cached-batch fast path, so a dead reader cannot drain what it prefetched.
IsActive() is used rather than a reader-local flag because it covers both
terminal causes at once, and this is the purpose it was added for.

Milvus does not reach this path: it closes strictly inside-out - scanners (whose
ScannerHelper.Close blocks until the consuming goroutine has exited) then the
writer, and only then the woodpecker client and its provider - so no ReadNext is
ever in flight against a retired session. The ordering is a discipline the
caller keeps, though, not something this API enforced.

Test literals in the package built logBatchReaderImpl directly and left
readerTempSession nil, which was invisible while nothing dereferenced it.
Production readers always have one, enforced by NewLogBatchReader, so the
literals now set one too rather than the read path tolerating nil.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReadNext keeps serving data after the reader is closed

1 participant