Skip to content

fix(Spanner): retry and resume BatchWrite streams on UNAVAILABLE - #9758

Merged
alkatrivedi merged 3 commits into
mainfrom
spanner-batchwrite-retry
Nov 19, 2026
Merged

alkatrivedi merged 3 commits into
mainfrom
spanner-batchwrite-retry

Conversation

@alkatrivedi

Copy link
Copy Markdown
Contributor

batchWriteAtLeastOnce previously failed immediately on any stream error including transient UNAVAILABLE errors and treated streams that closed prematurely as complete. This PR introduce automatic retries and resumes interrupted BatchWrite streams for only the remaining unacknowledged mutation groups, while translating response indexes back to their original positions.

@product-auto-label product-auto-label Bot added the api: spanner Issues related to the Spanner API. label Oct 28, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a resilient batch write stream mechanism (batchWriteStream) for Cloud Spanner, allowing batchWriteAtLeastOnce to transparently retry and resume on transient failures (such as UNAVAILABLE or premature stream termination) for only the unacknowledged mutation groups while correctly translating response indexes. It also updates metrics tracing and request stream coordination to support deferred operation completion. The feedback recommends avoiding the override of the private _destroy method on instantiated Node.js streams in both batch-write-stream.ts and database.ts, suggesting instead the use of standard public stream events like 'close' and 'error' for cleanup.

Comment thread handwritten/spanner/src/database.ts Resolved
@alkatrivedi
alkatrivedi force-pushed the spanner-batchwrite-retry branch 5 times, most recently from f8ace3c to d9d9733 Compare October 29, 2026 09:19
@alkatrivedi
alkatrivedi marked this pull request as ready for review October 29, 2026 09:22
@alkatrivedi
alkatrivedi requested review from a team as code owners October 29, 2026 09:22
@alkatrivedi
alkatrivedi force-pushed the spanner-batchwrite-retry branch from d9d9733 to 04d2439 Compare October 29, 2026 09:32

@olavloite olavloite left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we consider adding these tests as well:

  • Trailing gRPC error after all mutation groups are acknowledged (remaining === 0)

    • Scenario: Attempt 1 streams responses acknowledging all mutation groups (0..N-1), but the underlying gRPC stream terminates with an error (e.g. UNAVAILABLE or connection reset) rather than a clean EOF / end.
    • Expected behavior: The stream should finalize with grpc.status.OK and emit end on userStream without error or attempting another retry (batch-write-stream.ts:L335-L340).
  • Total timeout / deadline exceeded during retries (DeadlineError)

    • Scenario: An attempt fails with UNAVAILABLE, but elapsed + delayMs >= timeout (or elapsed time already exceeds timeout).
    • Expected behavior: The client should short-circuit the backoff timer, finalize with grpc.status.DEADLINE_EXCEEDED, and destroy userStream with a DeadlineError wrapping the underlying error (batch-write-stream.ts:L222-L227). Currently, no tests assert DeadlineError.
  • Fatal error after a partial retry

    • Scenario: Attempt 1 acknowledges group 0, but fails with UNAVAILABLE before acknowledging group 1. Attempt 2 is issued with [group1], but fails with a non-retryable error (e.g. PERMISSION_DENIED).
    • Expected behavior: userStream emits the PERMISSION_DENIED error, final status is PERMISSION_DENIED, and no further retry is attempted. (Currently, every retry test in test/batch-write-stream.ts eventually succeeds; testing eventual failure after partial progress is missing).
  • Exhaustion of maxRetries

    • Scenario: Configure backoffSettings.maxRetries: 2 and fail consecutive attempts with UNAVAILABLE.
    • Expected behavior: Retries cease after the 2nd retry attempt, and the user stream emits the final UNAVAILABLE error (batch-write-stream.ts:L216).
  • Corrupted RetryInfo metadata decode fallback

    • Scenario: Unit test getNextRetryDelayMs with malformed binary payload in google.rpc.retryinfo-bin.
    • Expected behavior: getNextRetryDelayMs gracefully catches the decode exception and falls back to jittered exponential backoff rather than throwing an unhandled exception (batch-write-stream.ts:L73-L75).

});
});

it('should fail withINTERNAL error and not retry when server returns an out-of-bounds index', done => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add space with INTERNAL

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

import {DeadlineError, isRetryableInternalError} from './transaction-runner';
import {getRetryInfo} from './protos';

const RETRY_INFO_BIN = 'google.rpc.retryinfo-bin';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This constant seems to be duplicated throughout the client in multiple places. Can we rather export it from one of the 'central' util files like src/proto.ts or src/common.tsx and import it here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

* Extracts server-supplied retry delay (in ms) from `google.rpc.retryinfo-bin`
* trailing metadata if present, otherwise computes jittered exponential backoff.
*/
export function getNextRetryDelayMs(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can we move this function to common.ts? In follow-up PRs, we can update other places that implement similar functionality (but in a less correct way) to reuse this method.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, done

Comment on lines +202 to +203
(retryableCodes.includes(grpc.status.UNAVAILABLE) &&
isRetryableInternalError(err))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I had to read over this a few times to understand why exactly we are doing this here. Maybe rewrite to this to make it easier to understand:

const allowsNetworkRetries = retryableCodes.includes(grpc.status.UNAVAILABLE);
return retryableCodes.includes(err.code) || (allowsNetworkRetries && isRetryableInternalError(err));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

* retries and resumes on transient stream failures (`UNAVAILABLE`, retryable `INTERNAL`,
* and premature stream `end`) for only the unresolved mutation groups.
*/
export function batchWriteStream(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we split this function into smaller parts? It is almost 300 lines long, and therefore hard to read.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines +289 to +326
// Validate all response indexes before mutating state.
for (let i = 0; i < respIndexes.length; i++) {
const idx = Number(respIndexes[i]);
if (
!Number.isInteger(idx) ||
idx < 0 ||
idx >= streamIndexes.length
) {
cleanupActiveRequestStream();
const msg = `BatchWrite index ${idx} out of bounds for current request size ${streamIndexes.length}`;
const internalErr = Object.assign(new Error(msg), {
code: grpc.status.INTERNAL,
details: msg,
metadata: new grpc.Metadata(),
} as grpc.ServiceError);
finalize(grpc.status.INTERNAL);
userStream.destroy(internalErr);
return;
}
}

// Translate retry-local indexes back to original caller indexes in-place
// and mark each acknowledged group as resolved (nulling reference for GC).
for (let i = 0; i < respIndexes.length; i++) {
const retryIdx = Number(respIndexes[i]);
const origIdx = streamIndexes[retryIdx];
respIndexes[i] = origIdx;
if (pending[origIdx] !== null) {
pending[origIdx] = null;
remaining--;
}
}
}

// Forward the response with translated indexes and respect downstream backpressure.
if (!userStream.write(response)) {
currentStream.pause();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we for example extract this into a separate method?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines +279 to +285
if (typeof response !== 'object' || response === null) {
sawNonObjectChunk = true;
if (!userStream.write(response)) {
currentStream.pause();
}
return;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this is only needed for a test case, and will never be used in real life, right? If so, can we get rid of this by modifying the test case to send a real object into the stream?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done and updated the tests accordingly.

? [...mutationGroups]
: [];
let remaining = totalGroups;
let sawNonObjectChunk = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this is only used to handle specific test cases. Can we get rid of it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

* });
* ```
*/
batchWriteAtLeastOnce(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we try to split up this method also? It is relatively long and heavily indented, making it very hard to read and understand what is going on.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, now we have extracted the helper methods to simplify batchWriteAtLeastOnce.

Comment thread handwritten/spanner/src/index.ts Outdated
headers: {[k: string]: string};
metricsTracer?: MetricsTracer;
metricsTracer?: MetricsTracer | null;
deferOperationCompletion?: boolean;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that deferOperationCompletion was added here for multi-attempt streaming RPCs like BatchWrite, we don't want RequestStreamCoordinator to prematurely finalize the operation metric when Attempt 1 fails with UNAVAILABLE, because the overall operation is still ongoing and will be retried.

However, adding deferOperationCompletion?: boolean to the generic RequestConfig feels like a leaky abstraction:

  1. It leaks an internal coordination detail of the OpenTelemetry metrics subsystem into the generic gRPC request configuration interface (RequestConfig), even though it is only used in this one place.
  2. In database.ts, it leads to a non-obvious side-effect dance: Attempt 1 passes metricsTracer: undefined, waits for requestStream to mutate config.metricsTracer in-place, and then captures it for Attempt 2+. The only reason Attempt 1 doesn't finalize the operation metric is because deferOperationCompletion: true is passed alongside metricsTracer: undefined.

Proposed Alternative: Ownership by Injection

We can drop deferOperationCompletion completely by establishing a clean ownership rule:

If the caller provides config.metricsTracer, the caller owns the operation lifecycle.

Instead of letting requestStream lazily create the tracer during Attempt 1:

  1. database.ts initializes metricsTracer upfront at operation start (via this._getSpanner()._initMetricsTracer(...)) and passes it into config.metricsTracer for all attempts (1, 2, ... N).
  2. In requestStream, we check if config.metricsTracer !== undefined. If so, we pass isExternalTracer = true to RequestStreamCoordinator.
  3. In RequestStreamCoordinator.prototype.completeOperation:
completeOperation(): void {
  if (this.operationCompleted || this.isExternalTracer) {
    return;
  }
  this.operationCompleted = true;
  this.metricsTracer?.recordOperationCompletion();
}

This achieves the exact same multi-attempt behavior without needing deferOperationCompletion on RequestConfig, eliminates the in-place mutation of config, and establishes a clean pattern that streaming queries/reads (partial-result-stream.ts) can also adopt in the future when retrying with resume tokens.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. We have removed deferOperationCompletion and adopted the ownership by injection approach within the implementation.

@alkatrivedi
alkatrivedi force-pushed the spanner-batchwrite-retry branch 4 times, most recently from cc84631 to 028e822 Compare November 4, 2026 08:51
@alkatrivedi
alkatrivedi force-pushed the spanner-batchwrite-retry branch 4 times, most recently from bc4b5f7 to 2841077 Compare November 16, 2026 08:07
@alkatrivedi

Copy link
Copy Markdown
Contributor Author

Can we consider adding these tests as well:

  • Trailing gRPC error after all mutation groups are acknowledged (remaining === 0)

    • Scenario: Attempt 1 streams responses acknowledging all mutation groups (0..N-1), but the underlying gRPC stream terminates with an error (e.g. UNAVAILABLE or connection reset) rather than a clean EOF / end.
    • Expected behavior: The stream should finalize with grpc.status.OK and emit end on userStream without error or attempting another retry (batch-write-stream.ts:L335-L340).
  • Total timeout / deadline exceeded during retries (DeadlineError)

    • Scenario: An attempt fails with UNAVAILABLE, but elapsed + delayMs >= timeout (or elapsed time already exceeds timeout).
    • Expected behavior: The client should short-circuit the backoff timer, finalize with grpc.status.DEADLINE_EXCEEDED, and destroy userStream with a DeadlineError wrapping the underlying error (batch-write-stream.ts:L222-L227). Currently, no tests assert DeadlineError.
  • Fatal error after a partial retry

    • Scenario: Attempt 1 acknowledges group 0, but fails with UNAVAILABLE before acknowledging group 1. Attempt 2 is issued with [group1], but fails with a non-retryable error (e.g. PERMISSION_DENIED).
    • Expected behavior: userStream emits the PERMISSION_DENIED error, final status is PERMISSION_DENIED, and no further retry is attempted. (Currently, every retry test in test/batch-write-stream.ts eventually succeeds; testing eventual failure after partial progress is missing).
  • Exhaustion of maxRetries

    • Scenario: Configure backoffSettings.maxRetries: 2 and fail consecutive attempts with UNAVAILABLE.
    • Expected behavior: Retries cease after the 2nd retry attempt, and the user stream emits the final UNAVAILABLE error (batch-write-stream.ts:L216).
  • Corrupted RetryInfo metadata decode fallback

    • Scenario: Unit test getNextRetryDelayMs with malformed binary payload in google.rpc.retryinfo-bin.
    • Expected behavior: getNextRetryDelayMs gracefully catches the decode exception and falls back to jittered exponential backoff rather than throwing an unhandled exception (batch-write-stream.ts:L73-L75).

Done

@alkatrivedi
alkatrivedi force-pushed the spanner-batchwrite-retry branch 4 times, most recently from 8ca69f3 to c320a0f Compare November 16, 2026 11:35
@alkatrivedi
alkatrivedi force-pushed the spanner-batchwrite-retry branch from c320a0f to c757c9f Compare November 18, 2026 05:32
backoffSettings?.totalTimeoutMillis ??
DEFAULT_TOTAL_TIMEOUT_MS;
const maxRetries = backoffSettings?.maxRetries ?? Infinity;
const retryable = isRetryableError(err, gaxOptions?.retry?.retryCodes);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gaxOptions.retry === null does not disable retries here. Should we consider doing that? That is normally what gax does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

gaxOptions?.timeout ??
backoffSettings?.totalTimeoutMillis ??
DEFAULT_TOTAL_TIMEOUT_MS;
const maxRetries = backoffSettings?.maxRetries ?? Infinity;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider including gaxOptions.maxRetries here? E.g. like this:

gaxOptions?.maxRetries ?? backoffSettings?.maxRetries ?? Infinity

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines +253 to +255
if (!state.userStream.write(response)) {
currentStream.pause();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have any test coverage for this. Can we add that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added unit tests covering downstream backpressure both on a single attempt and across stream retries.

Comment thread handwritten/spanner/src/common.ts Outdated
NonNullable<NonNullable<CallOptions['retry']>['backoffSettings']>
>,
): number {
const retryInfoValues = err?.metadata && err.metadata.get(RETRY_INFO_BIN);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could in theory trigger an unexpected error if metadata is undefined or does not have a get(..) method. So maybe we should consider moving the try block so it also includes this? Or do a check before calling metadata.get(..)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// Caller destroys stream while retry timer is waiting
stream.destroy();

setTimeout(() => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we try to get rid of the setTimeout(..) here? For example with sandbox.useFakeTimers() and clock.tick(...)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread handwritten/spanner/test/database.ts Outdated
Comment on lines +997 to +1000
setTimeout(() => {
stream2.write({indexes: [0], status: {code: grpc.status.OK}});
stream2.end();
}, 15);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here also, can we try to get rid of the setTimeout(..), both to keep the tests as fast as possible, but also because this could potentially be flaky.

Can we replace setTimeout(..., 15) with requestStreamStub.onCall(1).callsFake(() => { setImmediate(() => { stream2.write(...); stream2.end(); }); return stream2; })?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread handwritten/spanner/test/database.ts Outdated
Comment on lines +1059 to +1062
setTimeout(() => {
stream2.write({indexes: [0], status: {code: grpc.status.OK}});
stream2.end();
}, 15);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as above regarding the setTimeout(..)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Updated.

@alkatrivedi
alkatrivedi merged commit b86d909 into main Nov 19, 2026
52 checks passed
@alkatrivedi
alkatrivedi deleted the spanner-batchwrite-retry branch November 19, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the Spanner API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants