Repository navigation
Send heartbeats for open DMS deployment versions - #7342
Merged
Merged
Conversation
DMS marks a version in progress at CreateVersion. If the CLI dies before CompleteVersion, nothing refreshes the version and the UI shows it as deploying forever. The service will expose a per-version last update time, refreshed by UpdateOperation and Heartbeat, and the UI will show a version as interrupted once that time is over two minutes old. The CLI now keeps that time fresh while a version is open. A heartbeat is sent only after 30s with no successful UpdateOperation, so a fast deploy sends none. Heartbeats are best effort: a failed one is retried quickly, then reported as a warning, then tried again at the next interval. It never fails or stops the deploy. Co-authored-by: Isaac <no-reply@databricks.com>
Stop is a context cancel instead of a stop channel, a done channel and a sync.OnceFunc guard. The quick retries after a failed heartbeat are gone: the SDK already retries transient errors inside each call, and the next attempt is at most 30s away against a 2-minute UI threshold. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
marked this pull request as ready for review
November 5, 2026 15:38
Sankalp-Mittal
requested review from
anton-107,
denik and
shreyas-goenka
and removed request for
anton-107
November 5, 2026 15:38
Collaborator
✅ Integration test reportCommit: 421bd8a
Top 3 slowest tests (at least 2 minutes):
|
shreyas-goenka
approved these changes
Nov 9, 2026
denik
reviewed
Nov 9, 2026
A failed heartbeat is not config-related and needs no user action, so it is logged with log.Infof instead of a warning diagnostic. Co-authored-by: Isaac <no-reply@databricks.com>
DeploymentState held both the buffer and the heartbeat for the open version and had to wire them together. The buffer now starts the heartbeat itself and stops it in Stop, so DeploymentState keeps one field. The heartbeat still runs past Drain until CompleteVersion. Co-authored-by: Isaac <no-reply@databricks.com>
Both UpdateOperation and Heartbeat update the version row, so running them at the same time can conflict on the server. Renewing when a write starts, instead of when it succeeds, means a heartbeat only overlaps a write that has been in flight for a full interval. Co-authored-by: Isaac <no-reply@databricks.com>
Resolves conflicts with #7232: write takes a bufferedUpdate and keeps renewing the heartbeat as it starts, and Stop sits next to the new Stats. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
❌ Integration test reportCommit: 943d3af
40 interesting tests: 40 FAIL
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DMS marks a version
IN_PROGRESSatCreateVersion. If the CLI dies beforeCompleteVersion, nothing refreshes the version and the UI shows it as deploying forever.Design (meeting notes: Nov 2, Nov 3):
UpdateOperationand byHeartbeat, and exposes it asVersion.update_time.CreateVersionstill rejects while the previous version's 2-minute lease (last write or heartbeat + 2 min) is live and closes an expired one asLEASE_EXPIRED, so writes and heartbeats both extend that lease. No auto-close;deploy.lockstays for now and will be removed in a later update.update_timeis over 2 minutes old. The state is recoverable.What
libs/dms/heartbeat.go: a goroutine per open version. It callsPOST .../versions/{v}/heartbeatonly after 30s with noUpdateOperationstarted, so a fast deploy sends none. A failed heartbeat is logged at info level and tried again at the next interval; the SDK already retries transient errors inside each call. It never fails or stops the deploy.libs/dms/operation_buffer.go: the buffer owns the heartbeat. Each write resets the timer as it starts, not when it succeeds:UpdateOperationandHeartbeatboth update the version row, so this way a heartbeat only overlaps a write that has been in flight a full interval.Stop()ends the heartbeat; it keeps running pastDrainso a post-deploy script is still covered.bundle/direct/dstate/state.go: the buffer (and with it the heartbeat) starts right afterCreateVersion, so deploy and destroy are both covered, andCompleteVersioncallsbuf.Stop()before the RPC.HeartbeatResponse.expire_timeis ignored.Dependencies
Version.update_timeis exposed. Until it lands, a deploy that writes more often than every 30s sends no heartbeat. Nothing reads the field yet, so this is not a regression.Tests
libs/dms/heartbeat_test.go:testing/synctestwith the real 30s constants: beats while idle, a write skips the beat, a failure tries again next interval,Stop().acceptance/bundle/dmsis unchanged: fixtures finish in under 30s, so no heartbeat is recorded. The fake server already serves the endpoint..nextchangesfragment: DMS is experimental and the change is invisible until the UX PR lands.This pull request and its description were written by Isaac.