Skip to content

Commit 7b75d46

Browse files
committed
WIP(#27237): seat review r2 — a 2xx call stamps a set row's time and never clears rejected
Claude-Session: https://claude.ai/code/session_01Utp6CaqVRa7ow6CjE1JwaG Co-authored-by: Claude <noreply@anthropic.com>
1 parent d9df7ad commit 7b75d46

10 files changed

Lines changed: 129 additions & 71 deletions

File tree

‎.changeset/27237-connector-credential-rejected-marker.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Metadata-protocol: none
1010
Three additions for the connector credential store (#27237):
1111

1212
- `@objectstack/spec/integration` exports `CONNECTOR_CREDENTIAL_REJECTED`, `ConnectorCredentialRejectedError` (with `status`: 401 or 403) and the structural check `isConnectorCredentialRejected`. They sit beside the upstream-unavailable marker. A connector provider factory throws it when the upstream refuses the credential it sent, and the materializer then degrades the instance instead of failing the boot. The code is registered in the error-code ledger under `@objectstack/spec`.
13-
- `ConnectorMaterialization` gains the optional `credentialVerified`. A provider factory sets it when the upstream accepted the instance's credential while materializing (the MCP handshake over http); the automation service then records the credential as verified, and a stored one marked `rejected` returns to `set`. A provider that first sends the credential on a call leaves it unset.
13+
- `ConnectorMaterialization` gains the optional `credentialVerified`. A provider factory sets it when the upstream accepted the instance's credential while materializing (the MCP handshake over http); the automation service then records the credential as verified, and a stored one marked `rejected` returns to `set`. A provider that first sends the credential on a call leaves it unset; a call answered 2xx is not taken as proof.
1414
- `IAutomationService` gains three optional members, `listConnectorCredentialRefs`, `setConnectorCredential` and `clearConnectorCredential`, with their types `ConnectorCredentialRef`, `ConnectorCredentialSetRequest` and `ConnectorCredentialClearRequest`. A listing entry never carries a value.
1515

1616
`PLATFORM_OBJECT_NAMES['service-automation']` lists the new `sys_connector_credential` object.

‎.changeset/27237-connector-credential-store.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ The reference now resolves through three sources, in order: an environment varia
1313

1414
- A reference no source answers for degrades the instance naming it, with a reason that names the reference and how to set it. Before, it failed the boot. A `credentialResolver` that throws is still fatal at boot.
1515
- Setting or clearing a credential reconciles at once: every instance naming the reference re-materializes, or is torn down and degraded, with no restart.
16-
- A credential the upstream refuses (HTTP 401 / 403) degrades the instance with the upstream's message, never the value. An `mcp` instance is refused when it connects; a `rest` or `openapi` instance on the first call that answers 401 / 403, which still returns its answer. A stored credential is marked `rejected`, and its instance is not retried until the credential is set again, the declaration changes, or the server restarts; the mark itself stays until an upstream accepts the value (an `mcp` handshake, or a call answered 2xx). A credential from an environment variable or the host's resolver is retried on the usual backoff instead, since it can change where the service cannot see it, and the listing reports it `rejected` until it is accepted.
16+
- A credential the upstream refuses (HTTP 401 / 403) degrades the instance with the upstream's message, never the value. An `mcp` instance is refused when it connects; a `rest` or `openapi` instance on the first call that answers 401 / 403, which still returns its answer. A stored credential is marked `rejected`, and its instance is not retried until the credential is set again, the declaration changes, or the server restarts; the mark itself stays until the value is set again or an `mcp` handshake proves it. A call answered 2xx does not clear it, since the endpoint may not check the credential; it only stamps the last-verified time of a credential still `set`. A credential from an environment variable or the host's resolver is retried on the usual backoff instead, since it can change where the service cannot see it, and the listing reports it `rejected` until a provider proves it or the server restarts.
1717
- Under the `group` and `isolated` tenancy postures the store is not read, and writes to it are refused with a 403 naming the posture.
1818
- A value starting with `secret:` is refused with a 400: the data engine would store it as a handle to another stored secret. Writes run one at a time with their reconcile, so the instances end on the value the store ends on; a first write another replica makes at the same moment answers 409.
1919
- The value never appears in a log record, the listing, `GET /connectors`, an error body or a read of the object. A provider message that echoes it carries `[credential withheld]` instead.

‎content/docs/automation/connectors.mdx‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -279,8 +279,8 @@ DELETE /api/v1/automation/connectors/credential-refs/BILLING_API_TOKEN
279279
`missing` or `rejected`), a `source` (`env`, `object`, `vault` or `null`),
280280
whether this door can write it, and the instances naming it with their state.
281281
A stored one also says when and by whom it was last set (`updatedAt`,
282-
`updatedBy`) and when an upstream last accepted it (`lastVerifiedAt`). It
283-
never includes a value.
282+
`updatedBy`) and when it was last seen working (`lastVerifiedAt`). It never
283+
includes a value.
284284
- **`PUT`** stores the value, then reconciles. Every instance naming the
285285
reference re-materializes with it **without a restart**, and the answer is
286286
the reference's entry after that. The route refuses, before writing anything:
@@ -308,13 +308,15 @@ depends on where the value came from:
308308

309309
- **A value stored here** is marked `rejected`, and its instance is not
310310
retried until the credential is set again, its declaration changes, or the
311-
server restarts. The mark outlives a restart: it clears only when an upstream
312-
accepts the value, either an `mcp` handshake that carries it or a `rest` /
313-
`openapi` call answered `2xx`.
311+
server restarts. The mark outlives a restart: it clears only when the value
312+
is set again or an `mcp` handshake that carries it succeeds. A `rest` /
313+
`openapi` call answered `2xx` does not clear it, because the endpoint may not
314+
check the credential at all.
314315
- **A value from an environment variable or the host's resolver** can change
315316
where this door cannot see it (a rotation, or a `403` rate limit that lifts),
316317
so its instance is retried on the same 5s→5min backoff as an unreachable
317-
upstream. The listing reports it `rejected` until an upstream accepts it.
318+
upstream. The listing reports it `rejected` until an `mcp` handshake proves
319+
it or the server restarts, even while the instance is serving again.
318320

319321
Credentials are **environment-level**: one value per reference for the whole
320322
environment, read by whichever instance names it. On a multi-organization

‎docs/adr/0097-declarative-connector-instances.md‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -231,12 +231,15 @@ exactly the plausible-but-dead shape this ADR exists to kill.
231231
> until the credential is set again, the declaration changes, or the process restarts (a restart
232232
> lifts the hold). An instance with no `credentialRef`, whose credential is written in the
233233
> declaration, is held the same way. The `rejected` mark survives a re-materialization, since a
234-
> `rest` or `openapi` instance materializes without sending the value. Only an upstream accepting
235-
> the value clears it: a provider answering `credentialVerified` at materialization (the `mcp`
236-
> handshake over http), or a call answered `2xx`. A value from an environment variable or the
237-
> host's resolver can change where the process cannot see it (a rotation, or a `403` rate limit
238-
> that lifts), so its instance is retried on the backoff above instead, and the credential
239-
> listing reports the refusal until an upstream accepts it.
234+
> `rest` or `openapi` instance materializes without sending the value. Only proof clears it: a
235+
> provider answering `credentialVerified` at materialization (the `mcp` handshake over http), or
236+
> setting the value again. A call answered `2xx` is not proof, since the endpoint may not check
237+
> the credential (a public path, an operation with `security: []`); it only stamps the
238+
> last-verified time of a row still at `set`. A value from an environment variable or the host's
239+
> resolver can change where the process cannot see it (a rotation, or a `403` rate limit that
240+
> lifts), so its instance is retried on the backoff above instead, and the credential listing
241+
> reports the refusal until a provider proves the value, it stops resolving from that source, or
242+
> the process restarts.
240243
241244
The `openapi` provider applies the same split to its **remote-URL spec fetch**:
242245
an unreachable endpoint (network error) or a transient HTTP status

‎packages/services/service-automation/src/connector-credential-door.test.ts‎

Lines changed: 25 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -219,19 +219,12 @@ async function boot(
219219
return { ql, engine, provider, call, reload, kernel };
220220
}
221221

222-
/** Wait until the listing entry for {@link REF} matches — an acceptance is recorded off the call's path. */
223-
async function listingSettles(engine: AutomationEngine, expected: Record<string, unknown>) {
224-
await vi.waitFor(async () => {
225-
expect((await engine.listConnectorCredentialRefs()).find((e) => e.ref === REF)).toMatchObject(expected);
226-
});
227-
return (await engine.listConnectorCredentialRefs()).find((e) => e.ref === REF)!;
228-
}
229222

230223
const ADMIN = { userId: 'usr_admin_27237' } as never;
231224

232225
describe('[#27237] a missing credential degrades its instance; setting it materializes it, no restart', () => {
233226
it('missing → degraded naming the reference; set → ready; clear → degraded again', async () => {
234-
const { engine, provider, ql, call } = await boot([billing()]);
227+
const { engine, provider, ql, call, reload } = await boot([billing()]);
235228

236229
// Missing: no arm of the chain answers — degraded, named, factory never asked.
237230
expect(provider.state.materialized).toHaveLength(0);
@@ -261,9 +254,10 @@ describe('[#27237] a missing credential degrades its instance; setting it materi
261254
expect(entry.connectors[0]).toMatchObject({ name: 'billing', state: 'ready' });
262255
// Materializing sent nothing upstream (the rest shape), so nothing is verified yet…
263256
expect(entry.lastVerifiedAt).toBeUndefined();
264-
// …until a call the upstream accepts.
257+
// …until a call answered 2xx stamps the time on the row, which is `set`.
265258
expect(await call()).toMatchObject({ status: 200 });
266-
await vi.waitFor(async () => expect((await engine.listConnectorCredentialRefs())[0].lastVerifiedAt).toBeDefined());
259+
await reload(); // the reconcile queue has run the call's bookkeeping by now
260+
expect((await engine.listConnectorCredentialRefs())[0]).toMatchObject({ status: 'set', lastVerifiedAt: expect.any(String) });
267261

268262
// Set again: a new revision re-materializes with the new value.
269263
await engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE_2, context: ADMIN });
@@ -419,7 +413,7 @@ describe('[#27237] a wrong value degrades the instance and marks the row rejecte
419413
});
420414

421415
describe('[#27237] seat review r1: only a stored value holds its instance', () => {
422-
it('an environment value refused on a call retries on the backoff, says so, and recovers once accepted', async () => {
416+
it('an environment value refused on a call retries on the backoff, says so, and recovers', async () => {
423417
process.env[REF] = DUMMY_ENV_VALUE;
424418
envKeys.push(REF);
425419
const out = captureOutput();
@@ -437,18 +431,18 @@ describe('[#27237] seat review r1: only a stored value holds its instance', () =
437431
// On the backoff: the retry timer is armed for it (a held instance arms none).
438432
expect(out.text()).toContain('1 degraded connector instance(s); next retry in 5000ms');
439433

440-
// The retry — the timer's own soft reconcile — re-materializes it.
434+
// The retry — the timer's own soft reconcile — re-materializes it, and it serves again.
441435
provider.state.callStatus = 200;
442436
const before = provider.state.materialized.length;
443437
await reload();
444438
expect(provider.state.materialized).toHaveLength(before + 1);
445439
expect(engine.getConnectorDegradedReason('billing')).toBeUndefined();
446-
// Still reported refused until an upstream accepts it…
447-
expect((await engine.listConnectorCredentialRefs())[0]).toMatchObject({ status: 'rejected' });
448-
// …which the first accepted call does.
449440
expect(await call()).toMatchObject({ status: 200 });
450-
const recovered = await listingSettles(engine, { source: 'env', status: 'set' });
451-
expect(recovered.statusReason).toBeUndefined();
441+
await reload();
442+
// Still reported refused: a 2xx is no proof (the endpoint may not check the credential).
443+
const after = (await engine.listConnectorCredentialRefs())[0];
444+
expect(after).toMatchObject({ source: 'env', status: 'rejected' });
445+
expect(after.connectors[0]).toMatchObject({ state: 'ready' });
452446
});
453447

454448
it("a vault value refused at materialization retries, and a rotation the provider proves recovers it", async () => {
@@ -471,7 +465,7 @@ describe('[#27237] seat review r1: only a stored value holds its instance', () =
471465
expect((await engine.listConnectorCredentialRefs())[0]).toMatchObject({ source: 'vault', status: 'set' });
472466
});
473467

474-
it('a stored value refused on a call holds its instance; a restart lifts the hold, the row stays `rejected` until a call is accepted', async () => {
468+
it('a stored value refused on a call holds its instance; a restart lifts the hold, and the row stays `rejected` — a 2xx call does not clear it', async () => {
475469
const first = await boot([billing()]);
476470
await first.engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE, context: ADMIN });
477471
const set = (await first.engine.listConnectorCredentialRefs())[0];
@@ -501,11 +495,20 @@ describe('[#27237] seat review r1: only a stored value holds its instance', () =
501495
await second.reload();
502496
expect((await second.engine.listConnectorCredentialRefs())[0]).toMatchObject({ status: 'rejected' });
503497

504-
// The upstream accepting the same value is what clears it.
498+
// A call answered 2xx is no proof — a public path or an operation with
499+
// `security: []` never checks the credential — so the row stays
500+
// `rejected`, its reason and its unverified time unmoved.
505501
expect(await second.call()).toMatchObject({ status: 200 });
506-
const accepted = await listingSettles(second.engine, { status: 'set', source: 'object' });
507-
expect(accepted.statusReason).toBeUndefined();
508-
expect(accepted.lastVerifiedAt).toBeDefined();
502+
await second.reload(); // the reconcile queue has run the call's bookkeeping by now
503+
const afterCall = (await second.engine.listConnectorCredentialRefs())[0];
504+
expect(afterCall).toMatchObject({ status: 'rejected', source: 'object', statusReason: reason });
505+
expect(afterCall.lastVerifiedAt).toBeUndefined();
506+
507+
// Setting it again is what clears it.
508+
await second.engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE_2, context: ADMIN });
509+
const reset = (await second.engine.listConnectorCredentialRefs())[0];
510+
expect(reset).toMatchObject({ status: 'set', source: 'object' });
511+
expect(reset.statusReason).toBeUndefined();
509512
});
510513

511514
it('a provider that proves the credential while materializing (mcp) returns a `rejected` row to `set` on restart', async () => {

‎packages/services/service-automation/src/connector-credentials.ts‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -326,12 +326,13 @@ export class ConnectorCredentialStore {
326326
}
327327

328328
/**
329-
* Record that an upstream ACCEPTED revision `revision`: a provider that
330-
* proved it at materialization (`credentialVerified`, the mcp provider), or
331-
* a call its upstream answered `2xx`. The timestamp moves, and a `rejected`
332-
* row whose same value now works returns to `set`. A materialization alone
333-
* proves nothing (rest and openapi first send the value on a call), so it
334-
* never calls this. A row rewritten since (another revision) is left alone.
329+
* Record that a provider PROVED revision `revision` while materializing
330+
* (`credentialVerified` — the mcp handshake carried it and the upstream
331+
* answered). The timestamp moves, and a `rejected` row whose same value now
332+
* works returns to `set`: this and setting the value again are the only
333+
* ways back. A materialization alone proves nothing (rest and openapi
334+
* first send the value on a call), so it never calls this. A row rewritten
335+
* since (another revision) is left alone.
335336
*/
336337
async markVerified(record: ConnectorCredentialRecord, at: Date = new Date()): Promise<void> {
337338
const engine = this.resolveEngine();
@@ -345,6 +346,26 @@ export class ConnectorCredentialStore {
345346
);
346347
}
347348

349+
/**
350+
* Record that a call sent with revision `revision` was answered `2xx`:
351+
* the timestamp moves on a row still at `set`, and nothing else. A `2xx`
352+
* proves nothing about a credential the endpoint may not have checked (a
353+
* public path, an openapi operation with `security: []`), so it never
354+
* returns a `rejected` row to `set` — {@link markVerified} or a new value
355+
* does that.
356+
*/
357+
async markCallAccepted(record: ConnectorCredentialRecord, at: Date = new Date()): Promise<void> {
358+
const engine = this.resolveEngine();
359+
if (!engine) return;
360+
const current = await this.read(record.ref);
361+
if (!current || current.revision !== record.revision || current.status !== 'set') return;
362+
await engine.update(
363+
CONNECTOR_CREDENTIAL_OBJECT,
364+
{ id: current.id, last_verified_at: at.toISOString() },
365+
{ context: SYSTEM_CONTEXT },
366+
);
367+
}
368+
348369
/**
349370
* Record that an upstream refused revision `revision`. `reason` is the
350371
* connector's message, already scrubbed of the value by the caller
@@ -582,8 +603,9 @@ export function credentialRejectedOnCallReason(
582603
}
583604

584605
/**
585-
* A `2xx` result of a connector call — the upstream accepted the credential.
586-
* The rest and openapi providers answer `{ status, ok, body }`.
606+
* A `2xx` result of a connector call. The rest and openapi providers answer
607+
* `{ status, ok, body }`. Not a proof the credential is valid: the endpoint
608+
* may not have checked it.
587609
*/
588610
export function isAcceptedCallResult(result: unknown): boolean {
589611
if (typeof result !== 'object' || result === null) return false;

0 commit comments

Comments
 (0)