fix(reader): fail ReadNext once the reader's session is retired - #278
Open
tinswzy wants to merge 1 commit into
Open
fix(reader): fail ReadNext once the reader's session is retired#278tinswzy wants to merge 1 commit into
tinswzy wants to merge 1 commit into
Conversation
Closes #277. ReadNext kept serving entries after the reader was closed. logBatchReaderImpl carries no closed state, and nothing on the read path consulted the session it holds, so a use-after-close read on silently - including draining whatever was already in the cached batch. The only code that noticed was the position report, and UpdateReaderTempInfo's refusal was swallowed into a Warn. That is worse than a lost report. A retired reader is gone from GetAllReaderTempInfoForLog, so the writer's cleanup no longer protects the segments it is still reading; if a writer elsewhere truncates and cleans, findNextReadableSegment silently advances past the cleaned range and the application loses entries with no error at all. A retired session is terminal. entry.closed is set in one place, retireReaderTempSession, reached from DeleteReaderTempInfo (the reader closing) and from provider Close(); nothing ever clears it. Reopening is not a recovery either - when the cause is provider Close(), CreateReaderTempInfo refuses. So fail fast, the way os.File returns ErrClosed and sql.DB returns ErrConnDone. werr.ErrLogReaderClosed already existed for exactly this, unused, and is already marked non-retryable. The check sits after the context check and ahead of the cached-batch fast path, so a dead reader cannot drain what it prefetched. IsActive() is used rather than a reader-local flag because it covers both terminal causes at once, and this is the purpose it was added for. Milvus does not reach this path: it closes strictly inside-out - scanners (whose ScannerHelper.Close blocks until the consuming goroutine has exited) then the writer, and only then the woodpecker client and its provider - so no ReadNext is ever in flight against a retired session. The ordering is a discipline the caller keeps, though, not something this API enforced. Test literals in the package built logBatchReaderImpl directly and left readerTempSession nil, which was invisible while nothing dereferenced it. Production readers always have one, enforced by NewLogBatchReader, so the literals now set one too rather than the read path tolerating nil. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #277.
Problem
ReadNextkept serving entries after the reader was closed:logBatchReaderImplcarries no closed state and nothing on the read path consulted the session it holds, so a use-after-close read on silently — including draining whatever was already in the cached batch. The only code that noticed was the position report, whose refusal was swallowed into a Warn.That is worse than a lost report. A retired reader is gone from
GetAllReaderTempInfoForLog, so the writer's cleanup no longer protects the segments it is still reading. If a writer elsewhere truncates and cleans,findNextReadableSegmentsilently advances past the cleaned range and the application loses entries with no error at all.Why failing fast is the right response
A retired session is terminal.
entry.closedis set in exactly one place,retireReaderTempSession, reached fromDeleteReaderTempInfo(the reader closing itself) and from providerClose(). Nothing ever clears it. Reopening is not a recovery path either — when the cause is providerClose(),CreateReaderTempInforefuses.So
ReadNextnow returnswerr.ErrLogReaderClosed, matching what the equivalent APIs do (os.File.Read→ErrClosed,sql.DB→ErrConnDone). That error already existed atcommon/werr/errors.go:81, was unused anywhere in the repo, and is already marked non-retryable — the semantics line up exactly.The check sits after the context check and ahead of the cached-batch fast path, so a dead reader cannot drain what it prefetched.
IsActive()is used rather than a reader-local closed flag because it covers both terminal causes with one check, and this is the purpose it was added for — it had no production caller until now.Milvus is unaffected
Verified end to end. Milvus closes strictly inside-out:
ScannerHelper.Close()blocks onBlockAndGetResult()until the consuming goroutine callsFinish, so noReadNextcan be running against a retired session, and the provider is only closed once every reader and writer already is. Milvus therefore never observes the old behaviour and will not observe the new error.The point is that this ordering is a discipline the caller keeps, not something the API enforced or documented.
Contract change
ReadNextgains an error it has never returned. Callers that treat any non-context error from it as terminal — Milvus'sscannerImpl.executeConsumercallss.Finish(err)— will surface it as a scanner failure rather than reading on silently. That is the intended behaviour, and per the ordering above it cannot fire in Milvus today.Test literals
Tests in this package build
logBatchReaderImpldirectly and leftreaderTempSessionnil, which was invisible while nothing dereferenced it. Production readers always have one —NewLogBatchReaderrejects nil — so the 34 literals now set one, rather than the read path tolerating nil.Testing
TestLogReader_ReadNext_RetiredSessionDoesNotDrainCachedBatchreturned a realLogMessagebefore the fix, which is the bug in one line of output.go test -race -short ./woodpecker/log/ ./meta/— passgolangci-lint run ./woodpecker/log/...— 0 issues