Repository navigation
Commit 4edefc5
authored
fix(spanner): synchronize stream lifecycle and retry buffering in ResumableStreamIterator (#14770)
Asynchronous result sets interact with ResumableStreamIterator from
multiple threads: the caller thread initiating or closing the stream,
gRPC callback threads notifying that messages or errors have arrived,
and consumer threads polling for readiness and reading rows. Previously,
stream lifecycle state and the retry buffer were not synchronized across
these interactions, which caused several issues:
1. Duplicate or prematurely consumed streams during startup: When a gRPC
callback fired while the initial stream was still being started and
prefetched, the consumer thread could observe a partially initialized
stream or start a second stream concurrently. Conversely, if a retryable
error occurred immediately and the consumer thread reset the stream to
retry, the finishing startup call could overwrite the reset state with
the failed stream.
2. Uncancelled streams and deadlocks on concurrent close: Closing a
result set while a stream was being created or retried could leave the
newly created stream open, close it multiple times, or hold an internal
lock while invoking external close callbacks. In addition, transaction
metadata flags (whether the stream included an inlined BeginTransaction
or was the last statement) were lost when the active stream reference
was cleared on close or during retry backoff, preventing callers from
properly cleaning up transactions and sessions.
3. Premature readiness signals for unresumable chunks:
ResumableStreamIterator buffers chunks that do not carry a resume token
so that a transient failure can be retried transparently without
yielding duplicate data. However, non-blocking readiness checks
delegated directly to the underlying gRPC stream and reported data as
available as soon as a single tokenless chunk arrived. This caused
asynchronous consumers to wake up and block on network I/O waiting for a
resume token or buffer limit.
How this change fixes these problems:
- Coordinates stream startup so only one thread starts a stream at a
time while concurrent callers wait for startup to complete, and tracks
an in-flight stream separately from the published stream. A stream is
only exposed to consumers and readiness checks after startup and initial
prefetching have completed, while still allowing concurrent close calls
to cancel an in-flight stream immediately and tearing down streams
outside the lock.
- Preserves stream transaction flags across stream resets, startup
failures, and concurrent closes so upper layers always know whether an
inlined transaction or final statement was in progress, and ensures the
tracing span is always ended exactly once.
- Drains available non-error chunks into the internal retry buffer
during non-blocking readiness checks and only reports data as ready to
poll when the buffer can actually be flushed (a resume token has been
seen, the buffer is full, or the stream has reached the end), or when an
error or closure needs to be surfaced.1 parent dcb29b0 commit 4edefc5
3 files changed
Lines changed: 2632 additions & 95 deletions
File tree
- java-spanner/google-cloud-spanner/src
- main/java/com/google/cloud/spanner
- test/java/com/google/cloud/spanner
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
161 | | - | |
| 161 | + | |
| 162 | + | |
162 | 163 | | |
163 | 164 | | |
164 | 165 | | |
| |||
170 | 171 | | |
171 | 172 | | |
172 | 173 | | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
173 | 183 | | |
174 | 184 | | |
175 | 185 | | |
| |||
179 | 189 | | |
180 | 190 | | |
181 | 191 | | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
182 | 196 | | |
183 | 197 | | |
184 | 198 | | |
| |||
0 commit comments