Repository navigation
fix(socket-mode): observe reconnect failures and stop connecting after disconnect() - #2820
Conversation
…r disconnect() delayReconnectAttempt chained the reconnect with .then(res) only, and the close handler dropped the returned promise, so a failed reconnect became an unhandled rejection that crashes Node by default. - Propagate the reconnect result (resolve and reject) and log failures from the close-triggered reconnect. - When disconnect() wins the race, reject with the new SMDisconnectedWhileConnectingError, which the close handler swallows, and check for it in start() right after apps.connections.open returns, so no WebSocket is opened after the user disconnected. Fixes #2656. Supersedes #2657. Co-authored-by: w3lld1 <42353747+w3lld1@users.noreply.github.com> Co-authored-by: Claude <svc-devxp-claude@slack-corp.com>
🦋 Changeset detectedLatest commit: e1ce02b The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2820 +/- ##
==========================================
+ Coverage 93.93% 94.28% +0.35%
==========================================
Files 66 67 +1
Lines 10581 10671 +90
Branches 611 680 +69
==========================================
+ Hits 9939 10061 +122
+ Misses 608 582 -26
+ Partials 34 28 -6
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
….then delayReconnectAttempt is now async, so start() runs a microtask after the timer fires. The at-most-one-reconnect test awaits the clock, faking only setTimeout so tickAsync does not stall the node:test runner. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…t phase disconnect() during the WebSocket handshake rejected start() with undefined, so a stopped reconnect logged a spurious error. disconnect() during a WSS URL retry wait cleared the timer without settling the delay, so start() never settled. start() now rejects with SMDisconnectedWhileConnectingError whenever it is shutting down, and disconnect() ends the pending reconnect delay. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…MConnectAbortedError A shorter name that fits every path that throws it, the internal reconnect included, and matches the State.Connecting vocabulary. The error code becomes slack_socket_mode_connect_aborted_error. Unreleased, so no compatibility shim. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
A reconnect that fails for good (invalid_auth after a token revoke) was only logged, leaving the process up with no socket and nothing for the app to listen to. Emit State.Disconnected with the cause so apps and Bolt can react, as they could when the rejection crashed the process. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
When disconnect() landed while apps.connections.open was in flight and that call then failed recoverably, a fresh backoff was armed after disconnect() had already ended the pending one, so start() only rejected after clientPingTimeout * n ms. Bail before arming the wait. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
The handshake tests stubbed SlackWebSocket.connect, so disconnect() took the already-disconnected cleanup branch instead of closing a CONNECTING socket. Point them at a local TCP server that never upgrades. Closing a CONNECTING socket logs a WebSocket error (also on main), so the reconnect test now asserts only that the reconnect failure is not logged. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
A reconnect that fails for good used to crash with exit code 1, which restart-on-failure supervisors relied on. Now it logs, emits disconnected, and exits 0 when nothing else holds the event loop, or keeps running without a connection. Spell that out in the changeset with the opt-in to keep crashing. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
clearTimeout never settles the promise awaiting the timer, so disconnect() needed a stored resolver to end the wait. The abortable setTimeout from node:timers/promises does that directly and replaces both the timer handle and the resolver. A new test pins that disconnect() clears the pending delay right away, which keeps the close listener from ignoring the close event disconnect() itself triggers. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
… an alias Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
mwbrooks
left a comment
There was a problem hiding this comment.
✅ Thanks @WilliamBergamin! Small one: the PR description still names SMDisconnectedWhileConnectingError, but the code ships SMConnectAbortedError.
Co-authored-by: Michael Brooks <mbrooks@slack-corp.com>
…es loud The close handler now ignores only SMConnectAbortedError, the disconnect() race from #2656. A reconnect that fails for good rejects with its cause, as on main, instead of being logged and emitted as `disconnected`. Bolt never listens for that event, so a default app exited 0 and on-failure supervisors stopped restarting it, and nested reconnects emitted `disconnected` twice. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
A SlackWebSocket nulls its inner socket before emitting close, so the current connection is never active when its own close arrives, and the identity check already ignores every other connection. The changeset no longer promises a `disconnected` error now that #2820 keeps a failed reconnect loud. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
The close handler now ignores a close from any connection other than the current one, by identity, instead of asking whether the current one is still active. Same outcome for the stale-close case from #2709, and it lets the next commits tell connections apart. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…ts URL A count of connects in flight covers the backoff wait and the apps.connections.open fetch, and drops once the new socket is assigned, so a failed handshake still reconnects through the close handler as before. A close during start()'s URL fetch no longer schedules a second connection. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
disconnect() aborts the session's AbortController and start() opens a new one, so a wait or URL fetch from before the restart sees its own session was stopped instead of reading a flag the restart already reset. This fixes disconnect(); start() during the reconnect wait or its URL fetch, which opened a second live connection. The per-wait reconnectDelay controller is gone. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…l start() The test no longer stubs start(): a reconnect whose URL fetch fails for good makes the close handler reject with that cause. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Against the mock servers, disconnect() and start() during the reconnect wait leave one connection open. The 409 test no longer swallows unhandled rejections: a reconnect stopped by disconnect() now settles quietly. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Each wait or URL fetch adds its own promise and removes exactly that one when it settles, so the busy check reads as "is any connect pending" and a holder can't release another's slot. Same behavior as the count. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
The wait uses try/catch instead of .catch(() => {}), and rethrows anything
that isn't the abort.
Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Matches python-slack-sdk's connect_operation_lock, the sibling SDK's guard for the same job. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Without --enable-source-maps the coverage report gives line numbers of the compiled output, so codecov's patch check reads the wrong lines. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…tart A URL fetch still in flight from a session that disconnect() aborted kept connectOperations non-empty, so after start() the close handler skipped every reconnect of the new session. The stale fetch throws on its own aborted signal before it can connect, so the new session starts with an empty set. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
The two tests that held the WSS URL now use createClient's holdNextURL and a real connection drop instead of a second URL stub and a close event for a socket that never existed. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
SlackWebSocket's own tests already pin that disconnect() while CONNECTING emits close, so the client tests only need a socket that never says hello. The local TCP server that never upgraded, its socket bookkeeping, and the node:net import go away. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…name is gone - The waiting-reconnect test now asserts after the aborted wait settles, so it fails if the wait reconnects anyway. - The stale-close test is named for the identity check and ticks asynchronously, so it fails if a non-current close reconnects. - The multiple-close test closes a named current socket instead of an undefined one, and the fails-for-good test drops an unused readyState line. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
…old one on refresh (#2821) Co-authored-by: hippi345 <13539685+hippi345@users.noreply.github.com> Co-authored-by: Claude <svc-devxp-claude@slack-corp.com>
zimeg
left a comment
There was a problem hiding this comment.
@WilliamBergamin Thanks for chasing these races down! Moving the reconnect wait onto an AbortSignal and acking each envelope on the connection it arrived on reads so much clearer 🔌 ✨ I left a question on a disconnect(); start() case and a thought on surfacing a reconnect that gives up, and neither is blocking.
📝 Before merge: #2821 landed on this branch, so I think the description and title might want to mention the #2767 refresh change too, since the squash commit carries both.
zimeg
left a comment
There was a problem hiding this comment.
@WilliamBergamin LGTM! Thanks for chasing these races down. Moving the reconnect wait onto an AbortSignal and acking each envelope on the connection it arrived on reads so much clearer 🔌 ✨ The notes are follow-ups for later only.
📝 Before merge: #2821 landed on this branch, so the description might mention the #2767 refresh change too, since the squash commit carries both.
| } else { | ||
| if (socket === this.retiringWebsocket) this.retiringWebsocket = undefined; | ||
| if (socket !== this.websocket) return; | ||
| if (this.shuttingDown || !this.autoReconnectEnabled) { |
There was a problem hiding this comment.
📝 note (non-blocking): A follow-up for later, since main behaves the same: an old socket's close is checked against the newest session, so disconnect(); start() without an await in between doesn't emit disconnected for the stop. The awaited path this PR fixes works well.
| await this.delayReconnectAttempt(() => this.start(), this.lifecycle.signal); | ||
| } catch (error) { | ||
| if (!(error instanceof SMConnectAbortedError)) { | ||
| throw error; |
There was a problem hiding this comment.
🧪 note (non-blocking): Rejecting with the cause when a reconnect fails for good matches main, so this is good to land. Later we might also emit disconnected with the error for apps that watch that event.
| return; | ||
| } | ||
| if (socket && socket !== this.websocket) return; | ||
| if (this.autoReconnectEnabled && !this.shuttingDown && event.reason !== 'link_disabled') { |
There was a problem hiding this comment.
question: Any disconnect reason other than link_disabled now opens a replacement right away with no backoff, so a reason we don't expect could loop fast on apps.connections.open. Would it be safer to skip the backoff only for warning and refresh_requested and send other reasons through the normal reconnect?
There was a problem hiding this comment.
I incorporated this into the changes 💯 thanks for calling it out
Only `warning` and `refresh_requested` open a replacement connection right away. Any other reason, like `too_many_websockets`, closes the connection and reconnects after the usual delay, as on `main`, so an unexpected reason can't loop fast on `apps.connections.open`. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Summary
Fixes two Socket Mode connection bugs:
disconnect()during a reconnect could crash the process. The reconnect rejected withundefinedand nothing caught it. Now a reconnect stopped bydisconnect()is ignored. A reconnect that fails for good still rejects with its cause.disconnect(refresh_requested) leaves the app without a connection for 9–12 s #2767: apps went offline for 5 to 12s on every connection refresh. On a Slackdisconnectmessage, the client now opens the new connection first and keeps the old one open until Slack closes it, so no events are dropped. (From fix(socket-mode): open the replacement connection before closing the old one on refresh #2821, merged into this branch.)Related fixes:
disconnect()whilestart()is still connecting makesstart()reject with the newSMConnectAbortedError, and no connection opens.start()right afterdisconnect()opens one connection, not two.test:coverageuses--enable-source-maps, so coverage points at TypeScript lines.Supersedes #2657 and #2785; their authors are credited as co-authors.
Testing
main, plus integration tests with the real WebSocket client.stop()during a reconnect exits 0. A reconnect that hitsinvalid_authexits 1 with theWebAPIPlatformError, same asmain.Promise.reject(undefined). A realdisconnect()doesn't.Requirements
🤖 Generated with Claude Code