Skip to content

Commit 4edefc5

Browse files
authored
fix(spanner): synchronize stream lifecycle and retry buffering in ResumableStreamIterator (#14770)
Asynchronous result sets interact with ResumableStreamIterator from multiple threads: the caller thread initiating or closing the stream, gRPC callback threads notifying that messages or errors have arrived, and consumer threads polling for readiness and reading rows. Previously, stream lifecycle state and the retry buffer were not synchronized across these interactions, which caused several issues: 1. Duplicate or prematurely consumed streams during startup: When a gRPC callback fired while the initial stream was still being started and prefetched, the consumer thread could observe a partially initialized stream or start a second stream concurrently. Conversely, if a retryable error occurred immediately and the consumer thread reset the stream to retry, the finishing startup call could overwrite the reset state with the failed stream. 2. Uncancelled streams and deadlocks on concurrent close: Closing a result set while a stream was being created or retried could leave the newly created stream open, close it multiple times, or hold an internal lock while invoking external close callbacks. In addition, transaction metadata flags (whether the stream included an inlined BeginTransaction or was the last statement) were lost when the active stream reference was cleared on close or during retry backoff, preventing callers from properly cleaning up transactions and sessions. 3. Premature readiness signals for unresumable chunks: ResumableStreamIterator buffers chunks that do not carry a resume token so that a transient failure can be retried transparently without yielding duplicate data. However, non-blocking readiness checks delegated directly to the underlying gRPC stream and reported data as available as soon as a single tokenless chunk arrived. This caused asynchronous consumers to wake up and block on network I/O waiting for a resume token or buffer limit. How this change fixes these problems: - Coordinates stream startup so only one thread starts a stream at a time while concurrent callers wait for startup to complete, and tracks an in-flight stream separately from the published stream. A stream is only exposed to consumers and readiness checks after startup and initial prefetching have completed, while still allowing concurrent close calls to cancel an in-flight stream immediately and tearing down streams outside the lock. - Preserves stream transaction flags across stream resets, startup failures, and concurrent closes so upper layers always know whether an inlined transaction or final statement was in progress, and ensures the tracing span is always ended exactly once. - Drains available non-error chunks into the internal retry buffer during non-blocking readiness checks and only reports data as ready to poll when the buffer can actually be flushed (a resume token has been seen, the buffer is full, or the stream has reached the end), or when an error or closure needs to be surfaced.
1 parent dcb29b0 commit 4edefc5

3 files changed

Lines changed: 2632 additions & 95 deletions

File tree

‎java-spanner/google-cloud-spanner/src/main/java/com/google/cloud/spanner/AbstractResultSet.java‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,8 @@ interface CloseableIterator<T> extends Iterator<T> {
158158

159159
/**
160160
* @param streamMessageListener A class object which implements StreamMessageListener
161-
* @return true if streaming is supported by the iterator, otherwise false
161+
* @return true if streaming is supported by the iterator, otherwise false. Also returns false
162+
* if the iterator has already been closed, in which case no stream is started.
162163
*/
163164
default boolean initiateStreaming(AsyncResultSet.StreamMessageListener streamMessageListener) {
164165
return false;
@@ -170,6 +171,15 @@ default void requestPrefetchChunks() {}
170171
/**
171172
* Returns true if data (a chunk, row, EOF, or error) is available to read immediately without
172173
* blocking the calling thread on network I/O.
174+
*
175+
* <p><strong>Single-Consumer Threading Contract:</strong> This method drains underlying stream
176+
* state into internal buffers and MUST only be called by the designated consuming thread. It
177+
* must NEVER be invoked concurrently with {@link #hasNext()} or {@link #next()}.
178+
*
179+
* <p><strong>Note on Pending Retryable Errors:</strong> If a retryable stream error is pending,
180+
* this method returns {@code true} so the consumer is prompted to call {@link #next()} to
181+
* handle backoff and reconnect. In that specific scenario, the subsequent call to {@link
182+
* #next()} may block for backoff sleep and stream re-establishment.
173183
*/
174184
default boolean isDataAvailable() {
175185
return true;
@@ -179,6 +189,10 @@ default boolean isDataAvailable() {
179189
* Returns true if the iterator can be polled immediately (via {@link #hasNext()} or {@link
180190
* #next()}) to yield a chunk or EOF without blocking on network I/O and without throwing a
181191
* stream error.
192+
*
193+
* <p><strong>Single-Consumer Threading Contract:</strong> This method drains underlying stream
194+
* state and MUST only be called by the designated consuming thread. It must NEVER be invoked
195+
* concurrently with {@link #hasNext()} or {@link #next()}.
182196
*/
183197
default boolean canPollWithoutErrorOrBlocking() {
184198
return false;

0 commit comments

Comments
 (0)