diff --git a/.changeset/socket-mode-reconnect-rejections.md b/.changeset/socket-mode-reconnect-rejections.md new file mode 100644 index 000000000..115c253ac --- /dev/null +++ b/.changeset/socket-mode-reconnect-rejections.md @@ -0,0 +1,9 @@ +--- +"@slack/socket-mode": minor +--- + +Fix an unhandled promise rejection of `undefined` that could crash the process when `disconnect()` raced a pending automatic reconnect ([#2656](https://github.com/slackapi/node-slack-sdk/issues/2656)). A reconnect that fails for good still rejects as before, with its cause. + +Also, `disconnect()` called before `start()` finishes connecting (while fetching the WebSocket URL, waiting to retry it, or during the WebSocket handshake) no longer leaves a connection opening afterwards: `start()` rejects with the new `SMConnectAbortedError` instead of hanging or rejecting with `undefined`. + +Calling `start()` right after `disconnect()`, while a reconnect is waiting or fetching its URL, now opens one connection instead of two. diff --git a/.changeset/socket-mode-refresh-overlap.md b/.changeset/socket-mode-refresh-overlap.md new file mode 100644 index 000000000..6909ad4ca --- /dev/null +++ b/.changeset/socket-mode-refresh-overlap.md @@ -0,0 +1,7 @@ +--- +"@slack/socket-mode": patch +--- + +Fix a 5 to 12 second gap without a connection each time Slack refreshes a Socket Mode connection ([#2767](https://github.com/slackapi/node-slack-sdk/issues/2767)). On a `disconnect` message with the `warning` or `refresh_requested` reason, the client now opens the replacement connection right away, without a reconnect backoff, and leaves the old connection open until Slack sends it a `disconnect` message too, so payloads Slack still sends on it during the switch are received instead of dropped. Each payload is acknowledged on the connection it arrived on. While the replacement is being opened, the current connection keeps serving, and overlapping `disconnect` messages open one replacement. A replacement that fails for good rejects with its cause, as a failed reconnect does. Any other reason, such as `link_disabled`, still closes the connection first and reconnects after the usual delay. + +**Note:** during each refresh, a client now holds two connections for about 60 seconds, until Slack sends the old one its own `disconnect` (`refresh_requested`, about a minute after the `warning`). Both count toward Slack's limit of 10 open Socket Mode connections per app, so apps that run many instances at once should leave room for one extra connection per instance while they refresh. diff --git a/packages/socket-mode/README.md b/packages/socket-mode/README.md index 6463d578b..42bc71073 100644 --- a/packages/socket-mode/README.md +++ b/packages/socket-mode/README.md @@ -127,13 +127,13 @@ The client's connection to Slack has a lifecycle. This means the client can be s In the table below, the client's states are listed, which are also the names of the events you can use to observe the transition to that state. The table also includes descriptions for the states and arguments that a listener would receive. -| Event Name | Arguments | Description | -| --------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `connecting` | | The client is in the process of connecting to the platform. | -| `connected` | | The client is connected to the platform and incoming events will start being emitted. | -| `disconnecting` | | The client is no longer connected to the platform and cleaning up its resources. It will soon transition to `disconnected`. | -| `reconnecting` | | The client is no longer connected to the platform and cleaning up its resources. It will soon transition to `connecting`. | -| `disconnected` | `(error)` | The client is not connected to the platform. This is a steady state - no attempt to connect is occurring. The `error` argument will be `undefined` when the client initiated the disconnect (normal). | +| Event Name | Arguments | Description | +| --------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `connecting` | | The client is in the process of connecting to the platform. | +| `connected` | | The client is connected to the platform and incoming events will start being emitted. | +| `disconnecting` | | The client is no longer connected to the platform and cleaning up its resources. It will soon transition to `disconnected`. | +| `reconnecting` | | The client is replacing its connection. On a Slack refresh, the current connection keeps serving after the new one is `connected`, until Slack sends it `disconnect`. It will soon transition to `connecting`. | +| `disconnected` | `(error)` | The client is not connected to the platform. This is a steady state - no attempt to connect is occurring. The `error` argument will be `undefined` when the client initiated the disconnect (normal). | The client also emits events that are part of its lifecycle, but aren't states. Instead, they represent specific moments that might be helpful to your app. The following table lists these events, their description, and includes the arguments that a listener would receive. diff --git a/packages/socket-mode/package.json b/packages/socket-mode/package.json index f3481cd53..5d5624495 100644 --- a/packages/socket-mode/package.json +++ b/packages/socket-mode/package.json @@ -44,7 +44,7 @@ "docs": "npx typedoc --plugin typedoc-plugin-markdown", "prepack": "npm run build", "test": "npm run test:unit && npm run test:integration", - "test:coverage": "npm run build && bash -c 'node --experimental-test-coverage --test-reporter=spec --test-reporter-destination=stdout --test-reporter=lcov --test-reporter-destination=lcov.info --test-reporter=junit --test-reporter-destination=test-results.xml --import tsx --test src/*.test.ts test/integrations/*.test.js'", + "test:coverage": "npm run build && bash -c 'node --enable-source-maps --experimental-test-coverage --test-reporter=spec --test-reporter-destination=stdout --test-reporter=lcov --test-reporter-destination=lcov.info --test-reporter=junit --test-reporter-destination=test-results.xml --import tsx --test src/*.test.ts test/integrations/*.test.js'", "test:integration": "npm run build && bash -c 'node --import tsx --test test/integrations/*.test.js'", "test:types": "tsd", "test:unit": "npm run build && bash -c 'node --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=test-results.xml --import tsx --test src/*.test.ts'", diff --git a/packages/socket-mode/src/SlackWebSocket.test.ts b/packages/socket-mode/src/SlackWebSocket.test.ts index 92d18e77c..ceae6a8c9 100644 --- a/packages/socket-mode/src/SlackWebSocket.test.ts +++ b/packages/socket-mode/src/SlackWebSocket.test.ts @@ -93,6 +93,36 @@ describe('SlackWebSocket', () => { ws.dispatchEvent(new ErrorEvent('error', { error: new Error('boom'), message: 'boom' })); sinon.assert.calledOnce(discStub); }); + + it('should emit "ws_message" with the receiving SlackWebSocket, so the client can tell connections apart', () => { + const ws = new WSMock(); + SlackWebSocket = proxyquire.load('./SlackWebSocket', { + undici: { + WebSocket: class Fake { + constructor() { + // biome-ignore lint/correctness/noConstructorReturn: for test mocking purposes + return ws; + } + }, + CloseEvent, + ErrorEvent, + MessageEvent, + ping: () => {}, + }, + }).SlackWebSocket; + const client = new EventEmitter(); + const received = sandbox.spy(); + client.on('ws_message', received); + const sws = new SlackWebSocket({ + url: 'whatevs', + client, + clientPingTimeoutMS: 1, + serverPingTimeoutMS: 1, + }); + sws.connect(); + ws.dispatchEvent(new MessageEvent('message', { data: '{"type":"hello"}' })); + sinon.assert.calledOnceWithExactly(received, '{"type":"hello"}', false, sws); + }); }); describe('disconnect() with an unresponsive peer', () => { @@ -358,6 +388,24 @@ describe('SlackWebSocket', () => { }); }); + describe('close notification', () => { + it('should emit "close" with the closed SlackWebSocket, so the client can tell connections apart', () => { + const client = new EventEmitter(); + const closed = sandbox.spy(); + client.on('close', closed); + const sws = new SlackWebSocket({ + url: 'ws://127.0.0.1/', + client, + clientPingTimeoutMS: 1, + serverPingTimeoutMS: 1, + }); + + sws.disconnect(); + + sinon.assert.calledOnceWithExactly(closed, sws); + }); + }); + describe('buildDefaultDispatcher() connect hook', () => { type ConnectCb = (err: Error | null, socket: unknown) => void; type ConnectFn = (opts: unknown, cb: ConnectCb) => void; diff --git a/packages/socket-mode/src/SlackWebSocket.ts b/packages/socket-mode/src/SlackWebSocket.ts index a8b01cbf9..8e40452b2 100644 --- a/packages/socket-mode/src/SlackWebSocket.ts +++ b/packages/socket-mode/src/SlackWebSocket.ts @@ -179,7 +179,7 @@ export class SlackWebSocket { return; } const isBinary = typeof event.data !== 'string'; - this.options.client.emit('ws_message', event.data, isBinary); + this.options.client.emit('ws_message', event.data, isBinary, this); }; this.websocket.addEventListener('message', this.messageHandler); @@ -296,7 +296,7 @@ export class SlackWebSocket { clearTimeout(this.closeHandshakeTimeout); // Emit event back to client letting it know connection has closed (in case it needs to reconnect if // reconnecting is enabled) - this.options.client.emit('close'); + this.options.client.emit('close', this); } /** diff --git a/packages/socket-mode/src/SocketModeClient.test.ts b/packages/socket-mode/src/SocketModeClient.test.ts index eed6574b0..6bdda9ab9 100644 --- a/packages/socket-mode/src/SocketModeClient.test.ts +++ b/packages/socket-mode/src/SocketModeClient.test.ts @@ -1,7 +1,7 @@ import assert from 'node:assert/strict'; import { afterEach, beforeEach, describe, it } from 'node:test'; import { ConsoleLogger } from '@slack/logger'; -import type { FetchFunction } from '@slack/web-api'; +import type { FetchFunction, WebClient } from '@slack/web-api'; import proxyquire from 'proxyquire'; import sinon from 'sinon'; @@ -15,6 +15,83 @@ describe('SocketModeClient', () => { afterEach(() => { sandbox.restore(); }); + + const CONNECTING = 0; + const OPEN = 1; + + class FakeSlackWebSocket { + static instances: FakeSlackWebSocket[] = []; + readyState: number | undefined = CONNECTING; + sent: string[] = []; + disconnectCalls = 0; + private client: { emit: (event: string, ...args: unknown[]) => boolean }; + constructor({ client }: { client: { emit: (event: string, ...args: unknown[]) => boolean } }) { + this.client = client; + FakeSlackWebSocket.instances.push(this); + } + connect() {} + isActive() { + return this.readyState === OPEN; + } + open() { + this.readyState = OPEN; + return this; + } + receive(message: Record) { + this.client.emit('ws_message', JSON.stringify(message), false, this); + } + hello() { + this.receive({ type: 'hello' }); + } + drop() { + this.readyState = undefined; + this.client.emit('close', this); + } + disconnect() { + this.disconnectCalls += 1; + this.drop(); + } + send(data: string, cb: (err: Error | undefined) => void) { + this.sent.push(data); + cb(undefined); + } + } + + const sockets = FakeSlackWebSocket.instances; + + function createClient(options: { clientPingTimeout?: number; autoReconnectEnabled?: boolean } = {}) { + sockets.length = 0; + const ProxiedSocketModeClient: typeof SocketModeClient = proxyquire('./SocketModeClient', { + './SlackWebSocket': { SlackWebSocket: FakeSlackWebSocket }, + }).SocketModeClient; + const logger = new ConsoleLogger(); + const error = sandbox.stub(logger, 'error'); + sandbox.stub(logger, 'debug'); + const client = new ProxiedSocketModeClient({ appToken: 'xapp-', clientPingTimeout: 60_000, logger, ...options }); + const urls: Array<() => Promise> = []; + const fetches = sandbox + .stub(client as unknown as { retrieveWSSURL: () => Promise }, 'retrieveWSSURL') + .callsFake(() => urls.shift()?.() ?? Promise.resolve('wss://fake')); + const holdNextURL = () => { + let release: () => void = () => {}; + const held = new Promise((resolve) => (release = () => resolve('wss://fake'))); + urls.push(() => held); + return () => release(); + }; + const failNextURL = (cause: Error) => { + urls.push(() => Promise.reject(cause)); + }; + client.on('interactive', ({ ack }) => ack()); + return { client, error, fetches, holdNextURL, failNextURL }; + } + + async function connect(client: SocketModeClient) { + const started = client.start(); + await sleep(0); + sockets[sockets.length - 1].open().hello(); + await started; + return sockets[sockets.length - 1]; + } describe('constructor', () => { let logFactory: sinon.SinonStub; beforeEach(() => { @@ -234,7 +311,7 @@ describe('SocketModeClient', () => { assert.strictEqual(args.type, 'events_api'); assert.deepEqual(args.body, body); await args.ack({}); - sinon.assert.calledOnceWithExactly(send, 'malformed', {}); + sinon.assert.calledOnceWithExactly(send, 'malformed', {}, undefined); } }); @@ -487,30 +564,532 @@ describe('SocketModeClient', () => { }); describe("reconnection on 'close' (issue #2709)", () => { - it('should not reconnect on a stale close event while the current connection is still active', () => { + it('should not reconnect on a close event from a connection that is no longer current', async () => { const client = new SocketModeClient({ appToken: 'xapp-', logLevel: LogLevel.ERROR }); - client.websocket = { isActive: () => true } as unknown as SlackWebSocket; + client.websocket = {} as SlackWebSocket; const startStub = sandbox.stub(client, 'start').resolves(undefined as never); - const clock = sandbox.useFakeTimers(); + const clock = sandbox.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); - client.emit('close'); - clock.tick(1_000_000); + client.emit('close', {} as SlackWebSocket); + await clock.tickAsync(1_000_000); sinon.assert.notCalled(startStub); }); - it('should schedule at most one reconnect when multiple close events fire', () => { + it('should schedule at most one reconnect when multiple close events fire', async () => { const client = new SocketModeClient({ appToken: 'xapp-', logLevel: LogLevel.ERROR }); const startStub = sandbox.stub(client, 'start').resolves(undefined as never); - const clock = sandbox.useFakeTimers(); + const clock = sandbox.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + const current = {} as SlackWebSocket; + client.websocket = current; - client.emit('close'); - client.emit('close'); - clock.tick(1_000_000); + client.emit('close', current); + client.emit('close', current); + await clock.tickAsync(1_000_000); sinon.assert.calledOnce(startStub); }); }); + + describe('reconnect failures (issue #2656)', () => { + let unhandledRejection: sinon.SinonSpy; + beforeEach(() => { + unhandledRejection = sandbox.spy(); + process.on('unhandledRejection', unhandledRejection); + }); + afterEach(() => { + process.off('unhandledRejection', unhandledRejection); + }); + + function clientWithLogger() { + const logger = new ConsoleLogger(); + sandbox.stub(logger, 'error'); + sandbox.stub(logger, 'debug'); + return new SocketModeClient({ appToken: 'xapp-', clientPingTimeout: 60_000, logger }); + } + + function failWSSURL(client: SocketModeClient) { + const { webClient } = client as unknown as { webClient: WebClient }; + sandbox.stub(webClient.apps.connections, 'open').rejects(new Error('internal_error')); + } + + function holdThenFailWSSURL(client: SocketModeClient) { + const { webClient } = client as unknown as { webClient: WebClient }; + let rejectURL: (error: Error) => void = () => {}; + sandbox.stub(webClient.apps.connections, 'open').returns(new Promise((_resolve, reject) => (rejectURL = reject))); + return (error: Error) => rejectURL(error); + } + + it('should reject with the cause when a reconnect fails for good', async () => { + const { client, failNextURL } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + const cause = new Error('invalid_auth'); + failNextURL(cause); + const [onClose] = client.listeners('close'); + + await assert.rejects(async () => onClose(old), cause); + }); + + it('should not open a connection when disconnect() wins the race with a reconnect', async () => { + const { client, error, holdNextURL } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + const releaseURL = holdNextURL(); + + old.drop(); + await sleep(10); + await client.disconnect(); + releaseURL(); + await sleep(10); + + assert.strictEqual(sockets.length, 1); + sinon.assert.notCalled(error); + sinon.assert.notCalled(unhandledRejection); + }); + + it('should not reconnect when disconnect() is called while a reconnect is waiting', async () => { + const { client, fetches } = createClient(); + const old = await connect(client); + + old.drop(); + await client.disconnect(); + await sleep(10); + + sinon.assert.calledOnce(fetches); + sinon.assert.notCalled(unhandledRejection); + }); + + it('should open one connection when disconnect() and start() run during the reconnect wait', async () => { + const { client, fetches } = createClient({ clientPingTimeout: 20 }); + const old = await connect(client); + + old.drop(); + await client.disconnect(); + const restarted = client.start(); + await sleep(0); + sockets[1].open().hello(); + await restarted; + await sleep(50); + + assert.strictEqual(sockets.length, 2); + sinon.assert.calledTwice(fetches); + }); + + it('should reconnect after a drop while an aborted session is still fetching its URL', async () => { + const { client, holdNextURL } = createClient({ clientPingTimeout: 1 }); + holdNextURL(); + client.start().catch(() => {}); + await client.disconnect(); + const restarted = client.start(); + await sleep(0); + sockets[0].open().hello(); + await restarted; + + sockets[0].drop(); + await sleep(20); + + assert.strictEqual(sockets.length, 2); + }); + + it('should drop a reconnect URL that arrives after disconnect() and start()', async () => { + const { client, holdNextURL } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + const releaseURL = holdNextURL(); + const disconnected = sandbox.spy(); + client.on('disconnected', disconnected); + + old.drop(); + await sleep(10); + await client.disconnect(); + const restarted = client.start(); + await sleep(0); + sockets[1].open().hello(); + await restarted; + releaseURL(); + await sleep(10); + + assert.strictEqual(sockets.length, 2); + assert.strictEqual(client.websocket, sockets[1]); + sinon.assert.calledOnce(disconnected); + }); + + it('should not schedule a reconnect while start() is fetching its URL', async () => { + const { client, holdNextURL } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + const releaseURL = holdNextURL(); + + const restarted = client.start(); + old.drop(); + await sleep(10); + releaseURL(); + await sleep(0); + sockets[1].open().hello(); + await restarted; + await sleep(10); + + assert.strictEqual(sockets.length, 2); + }); + + it('should reject start() when disconnect() is called while it is connecting', async () => { + const { client, holdNextURL } = createClient(); + const releaseURL = holdNextURL(); + + const starting = client.start(); + await client.disconnect(); + releaseURL(); + + await assert.rejects(starting, { name: 'SMConnectAbortedError' }); + assert.strictEqual(sockets.length, 0); + }); + + it('should reject start() when disconnect() is called during the WebSocket handshake', async () => { + const { client } = createClient(); + + const starting = client.start(); + await sleep(0); + await client.disconnect(); + + await assert.rejects(starting, { name: 'SMConnectAbortedError' }); + }); + + it('should stop a reconnect during the WebSocket handshake without an unhandled rejection', async () => { + const { client } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + const disconnected = sandbox.spy(); + client.on('disconnected', disconnected); + + old.drop(); + await sleep(10); + await client.disconnect(); + await sleep(10); + + assert.strictEqual(sockets.length, 2); + sinon.assert.notCalled(unhandledRejection); + sinon.assert.calledOnce(disconnected); + }); + + it('should reject start() when disconnect() is called while waiting to retry the WSS URL', async () => { + const client = clientWithLogger(); + failWSSURL(client); + + const starting = client.start(); + await sleep(10); + await client.disconnect(); + + await assert.rejects(starting, { name: 'SMConnectAbortedError' }); + }); + + it('should reject start() without waiting to retry when the WSS URL fails after disconnect()', async () => { + const client = clientWithLogger(); + const failURL = holdThenFailWSSURL(client); + + const starting = client.start(); + await client.disconnect(); + failURL(new Error('internal_error')); + + await assert.rejects(starting, { name: 'SMConnectAbortedError' }); + }); + }); + + describe('server-requested disconnect (issue #2767)', () => { + const refreshRequested = { type: 'disconnect', reason: 'refresh_requested' }; + const envelope = { type: 'interactive', envelope_id: 'e-1', payload: {} }; + + it('should open the replacement right away and leave the old connection open after it says hello', async () => { + const { client, error } = createClient(); + const old = await connect(client); + const disconnected = sandbox.spy(); + client.on('disconnected', disconnected); + + old.receive(refreshRequested); + await sleep(0); + + assert.strictEqual(sockets.length, 2, 'the replacement should open without waiting for a backoff'); + assert.strictEqual(old.disconnectCalls, 0); + + sockets[1].open().hello(); + await sleep(10); + + assert.strictEqual(old.disconnectCalls, 0, 'Slack may still deliver on the old connection'); + assert.strictEqual(client.websocket, sockets[1]); + old.drop(); + await sleep(10); + assert.strictEqual(sockets.length, 2, "the old connection's close must not trigger another reconnect"); + sinon.assert.notCalled(disconnected); + sinon.assert.notCalled(error); + }); + + it('should acknowledge each envelope on the connection it arrived on', async () => { + const { client } = createClient(); + const old = await connect(client); + + old.receive(refreshRequested); + await sleep(0); + old.receive(envelope); + sockets[1].open(); + old.receive(envelope); + await sleep(0); + + assert.strictEqual(old.sent.length, 2); + assert.strictEqual(sockets[1].sent.length, 0); + + sockets[1].hello(); + sockets[1].receive(envelope); + old.receive(envelope); + await sleep(0); + + assert.strictEqual(sockets[1].sent.length, 1); + assert.strictEqual(old.sent.length, 3); + }); + + it('should acknowledge on the current connection when the arrival connection has closed', async () => { + const { client } = createClient(); + const old = await connect(client); + const acks: Array<() => Promise> = []; + client.on('slash_commands', ({ ack }) => acks.push(ack)); + + old.receive(refreshRequested); + await sleep(0); + sockets[1].open().hello(); + old.receive({ type: 'slash_commands', envelope_id: 'e-2', payload: {} }); + old.drop(); + await acks[0](); + + assert.strictEqual(old.sent.length, 0); + assert.strictEqual(sockets[1].sent.length, 1); + }); + + it('should close the old connection at the next refresh or on disconnect()', async () => { + const { client } = createClient(); + const old = await connect(client); + const disconnected = sandbox.spy(); + client.on('disconnected', disconnected); + + old.receive(refreshRequested); + await sleep(0); + sockets[1].open().hello(); + assert.strictEqual(old.disconnectCalls, 0); + sockets[1].receive(refreshRequested); + await sleep(0); + + assert.strictEqual(old.disconnectCalls, 1); + assert.strictEqual(sockets[1].disconnectCalls, 0); + + await client.disconnect(); + + assert.strictEqual(sockets[1].disconnectCalls, 1); + assert.strictEqual(sockets[2].disconnectCalls, 1); + sinon.assert.calledOnce(disconnected); + }); + + it('should not close an old connection again after Slack closed it', async () => { + const { client } = createClient(); + const old = await connect(client); + + old.receive(refreshRequested); + await sleep(0); + sockets[1].open().hello(); + old.drop(); + await client.disconnect(); + + assert.strictEqual(old.disconnectCalls, 0); + }); + + it('should back off and keep the old connection serving when the replacement fails before hello', async () => { + const { client } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + + old.receive(refreshRequested); + await sleep(0); + sockets[1].drop(); + await sleep(10); + + assert.strictEqual(sockets.length, 3, 'the failed replacement should be retried with the usual backoff'); + assert.strictEqual(old.disconnectCalls, 0); + old.receive(envelope); + await sleep(0); + assert.strictEqual(old.sent.length, 1); + + sockets[2].open().hello(); + await sleep(0); + + assert.strictEqual(old.disconnectCalls, 0); + assert.strictEqual(client.websocket, sockets[2]); + }); + + it('should open only the replacement when the old connection closes while its URL is fetched', async () => { + const { client, holdNextURL } = createClient({ clientPingTimeout: 1 }); + const old = await connect(client); + const releaseURL = holdNextURL(); + const events = { connected: sandbox.spy(), reconnecting: sandbox.spy() }; + for (const [name, spy] of Object.entries(events)) client.on(name, spy); + + old.receive(refreshRequested); + await sleep(0); + old.drop(); + await sleep(10); + releaseURL(); + await sleep(0); + sockets[1].open().hello(); + await sleep(10); + + assert.deepStrictEqual( + sockets.map((socket) => socket.isActive()), + [false, true], + ); + assert.strictEqual(client.websocket, sockets[1]); + sinon.assert.calledOnce(events.connected); + sinon.assert.calledOnce(events.reconnecting); + }); + + it('should open one replacement when disconnect messages arrive back to back', async () => { + const { client } = createClient(); + const old = await connect(client); + const reconnecting = sandbox.spy(); + client.on('reconnecting', reconnecting); + + old.receive({ type: 'disconnect', reason: 'warning' }); + old.receive(refreshRequested); + await sleep(0); + sockets[1].open().hello(); + await sleep(10); + + assert.strictEqual(sockets.length, 2); + assert.strictEqual(client.websocket, sockets[1]); + sinon.assert.calledOnce(reconnecting); + }); + + it('should close the retiring connection, not open another, on its disconnect message while the replacement handshakes', async () => { + const { client } = createClient(); + const old = await connect(client); + + old.receive({ type: 'disconnect', reason: 'warning' }); + await sleep(0); + const replacement = sockets[1].open(); + old.receive(refreshRequested); + await sleep(0); + + assert.strictEqual(sockets.length, 2); + assert.strictEqual(old.disconnectCalls, 1); + + replacement.hello(); + replacement.receive(envelope); + await sleep(0); + + assert.strictEqual(replacement.disconnectCalls, 0); + assert.strictEqual(replacement.sent.length, 1); + assert.strictEqual(client.websocket, replacement); + }); + + it('should reject with the cause when the replacement fails for good', async () => { + const { client, failNextURL } = createClient(); + const old = await connect(client); + const cause = new Error('invalid_auth'); + failNextURL(cause); + const [onMessage] = client.listeners('ws_message'); + + await assert.rejects(async () => onMessage(JSON.stringify(refreshRequested), false, old), cause); + }); + + it('should close a replacement that never opened when start() replaces it', async () => { + const { client } = createClient(); + const old = await connect(client); + + old.receive(refreshRequested); + await sleep(0); + const restarted = client.start(); + await sleep(0); + sockets[2].open().hello(); + await restarted; + + assert.strictEqual(sockets[1].disconnectCalls, 1); + assert.strictEqual(old.disconnectCalls, 0); + assert.strictEqual(client.websocket, sockets[2]); + }); + + it('should close both connections and emit disconnected once when disconnect() runs mid-refresh', async () => { + const { client, error } = createClient(); + const old = await connect(client); + const disconnected = sandbox.spy(); + client.on('disconnected', disconnected); + + old.receive(refreshRequested); + await sleep(0); + await client.disconnect(); + await sleep(10); + + assert.strictEqual(old.disconnectCalls, 1); + assert.strictEqual(sockets[1].disconnectCalls, 1); + sinon.assert.calledOnce(disconnected); + sinon.assert.notCalled(error); + }); + + it('should close without reconnecting when auto-reconnect is disabled', async () => { + const { client } = createClient({ autoReconnectEnabled: false }); + const old = await connect(client); + const disconnected = sandbox.spy(); + client.on('disconnected', disconnected); + + old.receive(refreshRequested); + await sleep(10); + + assert.strictEqual(old.disconnectCalls, 1); + assert.strictEqual(sockets.length, 1); + sinon.assert.calledOnce(disconnected); + }); + + it('should close the connection instead of opening a replacement when the link is disabled', async () => { + const { client } = createClient(); + const old = await connect(client); + + old.receive({ type: 'disconnect', reason: 'link_disabled' }); + await sleep(0); + + assert.strictEqual(sockets.length, 1); + assert.strictEqual(old.disconnectCalls, 1); + await client.disconnect(); + }); + + it('should close the connection and reconnect after the backoff on an unexpected reason', async () => { + const { client } = createClient({ clientPingTimeout: 20 }); + const old = await connect(client); + + old.receive({ type: 'disconnect', reason: 'too_many_websockets' }); + await sleep(0); + + assert.strictEqual(sockets.length, 1); + assert.strictEqual(old.disconnectCalls, 1); + await sleep(40); + assert.strictEqual(sockets.length, 2); + await client.disconnect(); + }); + + it('should ignore a disconnect message that arrives after disconnect()', async () => { + const { client } = createClient(); + const old = await connect(client); + + await client.disconnect(); + old.receive(refreshRequested); + await sleep(10); + + assert.strictEqual(sockets.length, 1); + }); + + it('should ignore a late disconnect message from a connection a later refresh closed', async () => { + const { client } = createClient(); + const old = await connect(client); + + old.receive(refreshRequested); + await sleep(0); + sockets[1].open().hello(); + sockets[1].receive(refreshRequested); + await sleep(0); + sockets[2].open().hello(); + old.receive(refreshRequested); + await sleep(10); + + assert.strictEqual(sockets.length, 3); + }); + }); }); async function sleep(ms: number): Promise { diff --git a/packages/socket-mode/src/SocketModeClient.ts b/packages/socket-mode/src/SocketModeClient.ts index 5d69a67c1..04ce1e5c5 100644 --- a/packages/socket-mode/src/SocketModeClient.ts +++ b/packages/socket-mode/src/SocketModeClient.ts @@ -1,3 +1,4 @@ +import { setTimeout } from 'node:timers/promises'; import { type AppsConnectionsOpenResponse, addAppMetadata, @@ -12,7 +13,12 @@ import { EventEmitter } from 'eventemitter3'; import { type RequestInit, fetch as undiciFetch } from 'undici'; import packageJson from '../package.json'; -import { SMSendWhileDisconnectedError, SMSendWhileNotReadyError, SMWebsocketError } from './errors'; +import { + SMConnectAbortedError, + SMSendWhileDisconnectedError, + SMSendWhileNotReadyError, + SMWebsocketError, +} from './errors'; import log, { type Logger, LogLevel } from './logger'; import { SlackWebSocket, WS_READY_STATES } from './SlackWebSocket'; import type { SocketModeDispatcher, SocketModeOptions } from './SocketModeOptions'; @@ -71,6 +77,8 @@ export class SocketModeClient extends EventEmitter { public websocket?: SlackWebSocket; + private retiringWebsocket?: SlackWebSocket; + /** * Enables ping-pong detailed logging if true */ @@ -93,16 +101,17 @@ export class SocketModeClient extends EventEmitter { private customLoggerProvided = false; + private connectOperations = new Set>(); + + private lifecycle = new AbortController(); + /** * Sentinel tracking if user invoked `disconnect()`; for enforcing shutting down of client * even if `autoReconnectEnabled` is `true`. */ - private shuttingDown = false; - - /** - * Timer handle for scheduling automatic reconnection attempts after a disconnect. - */ - private reconnectionTimer: ReturnType | undefined; + private get shuttingDown(): boolean { + return this.lifecycle.signal.aborted; + } public constructor( { @@ -155,15 +164,22 @@ export class SocketModeClient extends EventEmitter { this.on('error', (err) => { this.logger.error(`WebSocket error! ${err}`); }); - this.on('close', () => { + this.on('close', async (socket: SlackWebSocket) => { // Underlying WebSocket connection was closed, possibly reconnect. - if (this.websocket?.isActive()) return; - if (this.reconnectionTimer) return; - if (!this.shuttingDown && this.autoReconnectEnabled) { - this.delayReconnectAttempt(this.start); - } else { + if (socket === this.retiringWebsocket) this.retiringWebsocket = undefined; + if (socket !== this.websocket) return; + if (this.shuttingDown || !this.autoReconnectEnabled) { // If reconnect is disabled or user explicitly called `disconnect()`, emit a disconnected state. this.emit(State.Disconnected); + return; + } + if (this.connectOperations.size > 0) return; + try { + await this.delayReconnectAttempt(() => this.start(), this.lifecycle.signal); + } catch (error) { + if (!(error instanceof SMConnectAbortedError)) { + throw error; + } } }); this.on('ws_message', this.onWebSocketMessage.bind(this)); @@ -179,11 +195,20 @@ export class SocketModeClient extends EventEmitter { */ public async start(): Promise { // python equiv: SocketModeClient.connect - this.shuttingDown = false; + if (this.shuttingDown) { + this.lifecycle = new AbortController(); + this.connectOperations = new Set(); + } + const { signal } = this.lifecycle; this.logger.debug('Starting Socket Mode session ...'); + const url = await this.withConnectOperation(this.retrieveWSSURL(signal)); + if (signal.aborted) { + throw new SMConnectAbortedError(); + } + const previous = this.websocket; // create a socket connection using SlackWebSocket this.websocket = new SlackWebSocket({ - url: await this.retrieveWSSURL(), + url, // web socket events relevant to this client will be emitted into the instance of this class // see bottom of constructor for where we bind to these events client: this, @@ -194,6 +219,7 @@ export class SocketModeClient extends EventEmitter { serverPingTimeoutMS: this.serverPingTimeoutMS, pingPongLoggingEnabled: this.pingPongLoggingEnabled, }); + this.retire(previous); // Return a promise that resolves with the connection information return new Promise((resolve, reject) => { @@ -207,7 +233,7 @@ export class SocketModeClient extends EventEmitter { }; disconnectedCallback = (err) => { this.removeListener(State.Connected, connectedCallback); - reject(err); + reject(signal.aborted ? new SMConnectAbortedError() : err); }; this.once(State.Connected, connectedCallback); this.once(State.Disconnected, disconnectedCallback); @@ -221,9 +247,8 @@ export class SocketModeClient extends EventEmitter { * unless you call start() again later. */ public disconnect(): Promise { - this.shuttingDown = true; - clearTimeout(this.reconnectionTimer); - this.reconnectionTimer = undefined; + this.lifecycle.abort(); + this.closeRetiringWebsocket(); this.logger.debug('Manually disconnecting this Socket Mode client'); this.emit(State.Disconnecting); return new Promise((resolve, _reject) => { @@ -240,32 +265,73 @@ export class SocketModeClient extends EventEmitter { // PRIVATE/PROTECTED METHODS + private retire(previous: SlackWebSocket | undefined): void { + if (previous?.isActive()) { + this.closeRetiringWebsocket(); + this.retiringWebsocket = previous; + } else if (previous?.readyState !== undefined) { + previous.disconnect(); + } + } + + private closeRetiringWebsocket(): void { + this.retiringWebsocket?.disconnect(); + this.retiringWebsocket = undefined; + } + + private async openReplacement(reason: string): Promise { + if (this.connectOperations.size > 0) return; + this.logger.debug(`Received "disconnect" (${reason}) message - opening a replacement connection.`); + this.emit(State.Reconnecting); + try { + await this.start(); + } catch (error) { + if (!(error instanceof SMConnectAbortedError)) { + throw error; + } + } + } + + private async withConnectOperation(work: Promise): Promise { + this.connectOperations.add(work); + try { + return await work; + } finally { + this.connectOperations.delete(work); + } + } + /** * Initiates a reconnect, taking into account configurable delays and number of reconnect attempts and failures. * Accepts a callback to invoke after any calculated delays. */ - private delayReconnectAttempt(cb: () => Promise): Promise { + private async delayReconnectAttempt(cb: () => Promise, signal: AbortSignal): Promise { + if (signal.aborted) { + throw new SMConnectAbortedError(); + } this.numOfConsecutiveReconnectionFailures += 1; const msBeforeRetry = this.clientPingTimeoutMS * this.numOfConsecutiveReconnectionFailures; this.logger.debug(`Before trying to reconnect, this client will wait for ${msBeforeRetry} milliseconds`); - return new Promise((res, _rej) => { - this.reconnectionTimer = setTimeout(() => { - this.reconnectionTimer = undefined; - if (this.shuttingDown) { - this.logger.debug('Client shutting down, will not attempt reconnect.'); - } else { - this.logger.debug('Continuing with reconnect...'); - this.emit(State.Reconnecting); - cb.apply(this).then(res); - } - }, msBeforeRetry); - }); + try { + await this.withConnectOperation(setTimeout(msBeforeRetry, undefined, { signal })); + } catch (error) { + if (!signal.aborted) { + throw error; + } + } + if (signal.aborted) { + this.logger.debug('Client shutting down, will not attempt reconnect.'); + throw new SMConnectAbortedError(); + } + this.logger.debug('Continuing with reconnect...'); + this.emit(State.Reconnecting); + return cb(); } /** * Retrieves a new WebSocket URL to connect to. */ - private async retrieveWSSURL(): Promise { + private async retrieveWSSURL(signal: AbortSignal): Promise { // python equiv: BaseSocketModeClient.issue_new_wss_url try { this.logger.debug('Going to retrieve a new WSS URL ...'); @@ -294,7 +360,7 @@ export class SocketModeClient extends EventEmitter { isRecoverable = false; } if (this.autoReconnectEnabled && isRecoverable) { - return await this.delayReconnectAttempt(this.retrieveWSSURL); + return await this.delayReconnectAttempt(() => this.retrieveWSSURL(signal), signal); } throw error; } @@ -307,7 +373,11 @@ export class SocketModeClient extends EventEmitter { * - raising the State.Connected event (when Slack sends a type:hello message) * - disconnecting the underlying socket (when Slack sends a type:disconnect message) */ - protected async onWebSocketMessage(data: string | ArrayBuffer, isBinary: boolean): Promise { + protected async onWebSocketMessage( + data: string | ArrayBuffer, + isBinary: boolean, + socket?: SlackWebSocket, + ): Promise { if (isBinary) { this.logger.debug('Unexpected binary message received, ignoring.'); return; @@ -344,10 +414,18 @@ export class SocketModeClient extends EventEmitter { // Slack is recycling the pod handling the connection (or otherwise requires the client to reconnect) if (event.type === 'disconnect') { - this.logger.debug( - `Received "${event.type}" (${event.reason}) message - disconnecting.${this.autoReconnectEnabled ? ' Will reconnect.' : ''}`, - ); - this.websocket?.disconnect(); + if (socket && socket === this.retiringWebsocket) { + this.closeRetiringWebsocket(); + return; + } + if (socket && socket !== this.websocket) return; + const isRefresh = event.reason === 'warning' || event.reason === 'refresh_requested'; + if (this.autoReconnectEnabled && !this.shuttingDown && isRefresh) { + await this.openReplacement(event.reason); + } else { + this.logger.debug(`Received "${event.type}" (${event.reason}) message - disconnecting.`); + this.websocket?.disconnect(); + } return; } @@ -358,7 +436,7 @@ export class SocketModeClient extends EventEmitter { `Calling ack() - type: ${event.type}, envelope_id: ${event.envelope_id}, data: ${JSON.stringify(response)}`, ); } - await this.send(event.envelope_id, response); + await this.send(event.envelope_id, response, socket); }; // For events_api messages, expose the type of the event @@ -408,19 +486,20 @@ export class SocketModeClient extends EventEmitter { * @param id the envelope id * @param body the message body or string text */ - private send(id: string, body = {}): Promise { + private send(id: string, body = {}, socket?: SlackWebSocket): Promise { const _body = typeof body === 'string' ? { text: body } : body; const message = { envelope_id: id, payload: { ..._body } }; return new Promise((resolve, reject) => { - const wsState = this.websocket?.readyState; + const websocket = socket?.isActive() ? socket : this.websocket; + const wsState = websocket?.readyState; this.logger.debug( `send() method was called (WebSocket state: ${wsState ? WS_READY_STATES[wsState] : 'uninitialized'})`, ); - if (this.websocket === undefined) { + if (websocket === undefined) { this.logger.error('Failed to send a message as the client is not connected'); reject(new SMSendWhileDisconnectedError()); - } else if (!this.websocket.isActive()) { + } else if (!websocket.isActive()) { this.logger.error('Failed to send a message as the client has no active connection'); reject(new SMSendWhileNotReadyError()); } else { @@ -428,7 +507,7 @@ export class SocketModeClient extends EventEmitter { const flatMessage = JSON.stringify(message); this.logger.debug(`Sending a WebSocket message: ${flatMessage}`); - this.websocket.send(flatMessage, (error) => { + websocket.send(flatMessage, (error) => { if (error) { this.logger.error(`Failed to send a WebSocket message (error: ${error})`); return reject(new SMWebsocketError(error)); diff --git a/packages/socket-mode/src/errors.test.ts b/packages/socket-mode/src/errors.test.ts index a181fc8a3..752062657 100644 --- a/packages/socket-mode/src/errors.test.ts +++ b/packages/socket-mode/src/errors.test.ts @@ -4,6 +4,7 @@ import { describe, it } from 'node:test'; import { ErrorCode, SlackSocketModeError, + SMConnectAbortedError, SMNoReplyReceivedError, SMPlatformError, SMSendWhileDisconnectedError, @@ -76,4 +77,16 @@ describe('error classes', () => { assert.ok(err.message.includes('not ready')); }); }); + + describe('SMConnectAbortedError', () => { + it('should be an instance of Error, SlackSocketModeError, and SMConnectAbortedError', () => { + const err = new SMConnectAbortedError(); + assert.ok(err instanceof Error); + assert.ok(err instanceof SlackSocketModeError); + assert.ok(err instanceof SMConnectAbortedError); + assert.equal(err.code, ErrorCode.ConnectAbortedError); + assert.equal(err.name, 'SMConnectAbortedError'); + assert.ok(err.message.includes('disconnect()')); + }); + }); }); diff --git a/packages/socket-mode/src/errors.ts b/packages/socket-mode/src/errors.ts index 20e220db0..279ff859e 100644 --- a/packages/socket-mode/src/errors.ts +++ b/packages/socket-mode/src/errors.ts @@ -15,6 +15,7 @@ export enum ErrorCode { WebsocketError = 'slack_socket_mode_websocket_error', NoReplyReceivedError = 'slack_socket_mode_no_reply_received_error', InitializationError = 'slack_socket_mode_initialization_error', + ConnectAbortedError = 'slack_socket_mode_connect_aborted_error', } export type SMCallError = @@ -81,6 +82,14 @@ export class SMSendWhileDisconnectedError extends SlackSocketModeError { } } +export class SMConnectAbortedError extends SlackSocketModeError { + readonly code = ErrorCode.ConnectAbortedError; + + constructor() { + super('Stopped connecting because disconnect() was called before the connection was established'); + } +} + export class SMSendWhileNotReadyError extends SlackSocketModeError { readonly code = ErrorCode.SendWhileNotReadyError; diff --git a/packages/socket-mode/src/index.ts b/packages/socket-mode/src/index.ts index 05ed0ec2a..ab85b6bc2 100644 --- a/packages/socket-mode/src/index.ts +++ b/packages/socket-mode/src/index.ts @@ -4,6 +4,7 @@ export { ErrorCode, SlackSocketModeError, SMCallError, + SMConnectAbortedError, SMNoReplyReceivedError, SMPlatformError, SMPlatformErrorEvent, diff --git a/packages/socket-mode/test/integrations/integration.test.js b/packages/socket-mode/test/integrations/integration.test.js index 6724da519..2f490977d 100644 --- a/packages/socket-mode/test/integrations/integration.test.js +++ b/packages/socket-mode/test/integrations/integration.test.js @@ -97,6 +97,111 @@ describe('Integration tests with a WebSocket server', { timeout: 30000 }, () => await client.disconnect(); }); }); + describe('restarting during a reconnect (issue #2656)', () => { + it('opens one connection when disconnect() and start() run during the reconnect wait', async () => { + client = new SocketModeClient({ + appToken: 'whatever', + logLevel: LogLevel.ERROR, + clientOptions: { + slackApiUrl: `http://localhost:${HTTP_PORT}/`, + }, + clientPingTimeout: 100, + }); + await client.start(); + + exposed_ws_connection.terminate(); + await sleep(20); + await client.disconnect(); + await client.start(); + await sleep(300); + + assert.strictEqual(wss.clients.size, 1); + await client.disconnect(); + }); + }); + describe('overlapping refreshes (issue #2767)', () => { + beforeEach(() => { + client = new SocketModeClient({ + appToken: 'whatever', + logLevel: LogLevel.ERROR, + clientOptions: { + slackApiUrl: `http://localhost:${HTTP_PORT}/`, + }, + clientPingTimeout: 100, + }); + }); + it('opens only the replacement when the old connection closes while its URL is fetched', async () => { + await client.start(); + const old = exposed_ws_connection; + server.close(); + server = createServer((_req, res) => { + setTimeout(() => { + res.writeHead(200, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ ok: true, url: `ws://localhost:${WSS_PORT}/` })); + }, 300); + }); + server.listen(HTTP_PORT); + let connections = 0; + const replaced = new Promise((resolve) => { + wss.on('connection', () => { + connections += 1; + resolve(); + }); + }); + + old.send(JSON.stringify({ type: 'disconnect', reason: 'refresh_requested' })); + await sleep(50); + old.terminate(); + await replaced; + await sleep(300); + + assert.strictEqual(connections, 1); + assert.strictEqual(wss.clients.size, 1); + await client.disconnect(); + }); + it('opens one replacement when refresh_requested arrives on the old connection before the new one says hello', async () => { + await client.start(); + const old = exposed_ws_connection; + let connections = 0; + wss.removeAllListeners('connection'); + wss.on('connection', (ws) => { + connections += 1; + exposed_ws_connection = ws; + setTimeout(() => ws.send(JSON.stringify({ type: 'hello' })), 300); + }); + + old.send(JSON.stringify({ type: 'disconnect', reason: 'warning' })); + await sleep(100); + old.send(JSON.stringify({ type: 'disconnect', reason: 'refresh_requested' })); + await sleep(500); + + assert.strictEqual(connections, 1); + assert.strictEqual(wss.clients.size, 1); + await client.disconnect(); + }); + it('receives and acknowledges an envelope sent on the old connection after the new one says hello', async () => { + await client.start(); + const old = exposed_ws_connection; + const acks = []; + old.on('message', (data) => acks.push(JSON.parse(data).envelope_id)); + const connected = new Promise((resolve) => client.once('connected', resolve)); + const received = new Promise((resolve) => + client.once('interactive', async ({ ack, envelope_id }) => { + await ack(); + resolve(envelope_id); + }), + ); + + old.send(JSON.stringify({ type: 'disconnect', reason: 'warning' })); + await connected; + old.send(JSON.stringify({ type: 'interactive', envelope_id: 'e-old', payload: {} })); + + assert.strictEqual(await Promise.race([received, sleep(500)]), 'e-old'); + await sleep(50); + assert.deepStrictEqual(acks, ['e-old']); + await client.disconnect(); + }); + }); describe('`apps.connections.open` API failure modes', () => { beforeEach(() => { client = new SocketModeClient({ @@ -177,15 +282,8 @@ describe('Integration tests with a WebSocket server', { timeout: 30000 }, () => assert.strictEqual(debugLoggerSpy.calledWith(sinon.match('Unable to parse an incoming WebSocket message')), true); await client.disconnect(); }); - it('should maintain one serial reconnection attempt if WSS server sends unexpected HTTP response during handshake, like a 409', async (t) => { + it('should maintain one serial reconnection attempt if WSS server sends unexpected HTTP response during handshake, like a 409', async () => { // test for https://github.com/slackapi/node-slack-sdk/issues/2094 - // SocketModeClient's internal reconnection loop creates promises that reject - // with `undefined` after disconnect(). node:test detects these as unhandled - // rejections (mocha did not). Mock process.emit to swallow them during cleanup. - const originalEmit = process.emit.bind(process); - t.after(() => { - process.emit = originalEmit; - }); // override socket mode client instance with lower client ping timeout, which controls reconnection rate client = new SocketModeClient({ appToken: 'whatever', @@ -238,11 +336,6 @@ describe('Integration tests with a WebSocket server', { timeout: 30000 }, () => else res(); }); }); - // Swallow unhandled rejections from internal reconnection cleanup - process.emit = (event, ...args) => { - if (event === 'unhandledRejection') return true; - return originalEmit(event, ...args); - }; // Allow event loop to drain any pending async activity from reconnection attempts await sleep(100); assert.strictEqual(retries, 2, 'unexpected number of times `close` event was raised during reconnection!');