Skip to content

overrides: add makeRequired DSL + 6 input-shape quirks + 5 service-method JSDocs - #13

Merged
javrrr merged 3 commits into
mainfrom
override-makeRequired-and-quirks
May 7, 2026
Merged

overrides: add makeRequired DSL + 6 input-shape quirks + 5 service-method JSDocs#13
javrrr merged 3 commits into
mainfrom
override-makeRequired-and-quirks

Conversation

@javrrr

@javrrr javrrr commented May 7, 2026

Copy link
Copy Markdown
Owner

Summary

Three independent changes, one PR for review economy. Each is its own commit
so they can be split if needed.

  1. generate-types: extend the override DSL with makeRequired and
    add field-level validation. Symmetric to the existing makeOptional;
    covers the case "spec marks optional, API requires it" — distinct from
    addRequiredFields (used when the spec omits the field entirely).
  2. SCHEMA_OVERRIDES: 4 new entries + 2 extended notes for input-shape
    quirks observed during integration testing.
  3. src/resources/: behavioral JSDoc on five service methods —
    timing, async-teardown, asymmetric path-params, name-rewrite quirks
    that don't fit the type-shape DSL.

215 tests pass. tsc --noEmit clean. No public-API breakage; the only
type-narrowing changes are on input shapes that the API was already
rejecting at runtime when the new requirements were violated.

Why this matters

The SDK already has precedent for runtime overrides on schemas where the
OpenAPI spec is structurally accurate but doesn't capture empirical platform
behavior. This PR continues that work: each change is grounded in a concrete
server response that the prior typings would have let consumers compile and
ship.

A representative example — without this PR, the following compiles cleanly
but fails at deploy:

```ts
await client.searchIndex.create({
developerName: "MyIdx",
label: "MyIdx",
searchType: "HYBRID",
processingType: "NEAR_REALTIME",
sourceDmoDeveloperName: "X__dlm",
chunkDmoDeveloperName: "MyIdx_chunk",
chunkDmoName: "MyIdx_chunk",
vectorDmoDeveloperName: "MyIdx_index",
vectorDmoName: "MyIdx_index",
chunkingConfiguration: { fieldLevelConfigurations: [/.../] },
vectorEmbedding: {}, // ← server NPEs here
vectorEmbeddingConfiguration: { /.../ },
});
// → 500 UNKNOWN_EXCEPTION (empty diagnostic body)
```

After this PR, `vectorEmbeddingRelatedFields` is required on
`VectorEmbeddingInputRepresentation` and the omission is caught at
compile time with a helpful `@override` JSDoc explaining why.

Commit 1: `generate-types` — `makeRequired` + field validation

The override loader had two gaps:

  • No `makeRequired`. Workaround was `addRequiredFields`, which only
    triggers when the field is absent from the schema — wrong tool when the
    field exists and just needs its required-ness flipped.
  • No field-level validation. Schema-name removal failed loudly (good)
    but a renamed/removed field referenced by `makeOptional` or `fieldTypes`
    silently no-op'd, leaving stale overrides invisible until they were
    actively misleading.

Adds:

  • `makeRequired: string[]` symmetric to `makeOptional`.
  • Per-field validation on `makeOptional` / `makeRequired` / `fieldTypes`
    — a stale field name now fails generation with the same loud message
    the schema-name validation already produces.
  • Rejection of contradictions: a field appearing in both `makeOptional`
    and `makeRequired` errors at config-load time rather than last-wins.

Negative-tested: introducing a stale field name to an existing override
produces

Generation failed: SCHEMA_OVERRIDES.DataStreamInputRepresentation.makeOptional: field "thisFieldDoesNotExistOnTheSchema" not found on schema — remove stale override

Commit 2: 4 new overrides + 2 extended notes

`ConnectionSchemaFieldInputRepresentation` — `makeRequired: ["label"]`

`PUT /ssot/connections/{id}/schema` returns
`500 INTERNAL_SERVER_ERROR` with the Java NPE
`Cannot invoke java.lang.CharSequence.length() because "this.text" is null`
when any field in the schema is missing `label`.

Reproducer:

```http
PUT /services/data/v66.0/ssot/connections/{id}/schema
{
"schemas": [{
"schemaType": "IngestApi",
"name": "KB",
"label": "KB",
"fields": [
{ "name": "Id", "dataType": "Text" } // ← no label → NPE
]
}]
}
```

Adding `label: "Id"` succeeds. Spec marks `label` optional on the base
`ConnectionSchemaFieldInputRepresentation`; the upsert handler dereferences
it unconditionally.

`DataStreamFieldMappingInputRepresentation` — `makeRequired: ["targetFieldReturntype"]`

Two related issues, one structural and one documentary.

Structural. `targetFieldReturntype` is required by the data-stream
create handler, but the spec marks it optional. When omitted, the create
returns 201 but the mapping is silently dropped from the saved data
stream — no error, no warning, just missing data flow at runtime.

Documentary (note only). POST input uses `sourceFieldLabel`, but GET
responses echo `sourceFieldName` for the same field. Round-tripping a GET
shape into a POST without renaming fails with
`JSON_PARSER_ERROR: Unrecognized field "sourceFieldName"`.

`DataStreamSourceFieldInputRepresentation` — note only

Same input/output asymmetry pattern as above: POST uses `dataType`
(camelCase), GET echoes `datatype` (lowercase). Round-tripping fails with
`JSON_PARSER_ERROR: Unrecognized field "datatype"`.

`VectorEmbeddingInputRepresentation` — `makeRequired: ["vectorEmbeddingRelatedFields"]`

Server NPEs when the array is omitted, empty, or null. Spec marks it
optional. Typical minimum is a single entry pointing at the source DMO's
primary key.

`DataStreamInputRepresentation` — extended note

The spec types `dataAccessMode?: "Direct_Access" | "Ingest"` correctly,
but doesn't document that `Direct_Access` is required for federated /
BYOL connectors (Snowflake, Databricks, BigQuery, Iceberg). Without it,
the create handler routes to the ingest path and returns the misleading
error `400 INTERNAL_ERROR: Unable to post Data Stream: DATA_CONNECTORS
is not supported` even when the connector is GA. The connector's
`features` array exposes `BYOL` as a hint, but that's the only public
signal.

Direct_Access streams must also OMIT the top-level `datasource` field —
the server returns
`400 BAD_REQUEST: DataSource name should be empty for External data streams`
otherwise. The connection binding is established through
`connectorInfo.connectorDetails.name` instead.

`SemanticSearchInputRepresentation` — extended note

The input handler rejects output-only display-name fields that GET
responses include alongside the developer-name fields. Specifically:
`sourceDmoName`, `sourceDmoFieldName`, `relatedDmoName`,
`relatedDmoFieldName`. Round-tripping a GET shape into a POST returns
`500 UNKNOWN_EXCEPTION` with no diagnostic body. Pass developer-name
fields only.

Commit 3: Service-class JSDoc

These don't fit the type-shape DSL because they describe behavior rather
than shape: timing windows, async teardowns, retry guidance, path-param
asymmetries. Three resource files converted from thin re-exports into
subclasses solely to host the JSDoc; signatures unchanged, all methods
delegate via `super.*`.

  • `ConnectionsService.create` — name-lock race after DELETE returns
    `DUPLICATES_DETECTED` for several seconds; recommend ~30s retry budget
    on that error code. IngestApi name-rewrite (`_`) caveat.
  • `ConnectionsService.delete` — transient `500 UNKNOWN_EXCEPTION`
    from the platform's teardown race, even when `deletable: true` and no
    live dependents. Recommend ~30s 5xx retry budget.
  • `CalculatedInsightsService.create` — server-side validation +
    schedule registration commonly takes 30–60s; SDK's default 30s timeout
    aborts. Recommend `{ timeout: 120_000 }`.
  • `CalculatedInsightsService.delete` — async teardown; record sits at
    `status = "DELETING"` for minutes; same-name re-create fails until
    teardown completes.
  • `SearchIndexService.delete` — path-parameter accepts the
    platform-assigned ID but NOT the developer name (asymmetric vs. GET
    which accepts either).

A note on the diff

`src/generated/openapi.{yaml,d.ts}` and parts of `src/schemas.ts` include
unrelated spec drift picked up by `npm run generate` (new `activationType`
and `sourceDmoName` fields on Activation* schemas). Reviewing on a
branch where the maintainer regenerates fresh would suppress these.

Test plan

  • `npm run typecheck` clean
  • `npm test` — all 215 tests pass, 3 skipped (unchanged)
  • `npm run generate` regenerates without errors against the current
    upstream spec
  • Negative test on the field-level validation (a stale field name in
    an override fails generation with a clear message)
  • Reviewer confirms behavioral evidence matches their internal records
    on each platform quirk

javrrr added 3 commits May 7, 2026 16:43
The SCHEMA_OVERRIDES DSL handled "spec marks required, API doesn't"
(makeOptional) and "field missing from spec entirely" (addRequiredFields)
but had no way to express "spec marks optional, API requires it" — a
common case where the spec is structurally correct but doesn't capture
empirical platform behavior. Add `makeRequired: string[]` symmetric to
`makeOptional`.

Also tighten validation. Existing override loader fails generation when
a SCHEMA name is removed from the spec; extend the same loud-failure
rule to FIELD names referenced by makeOptional / makeRequired /
fieldTypes. A field renamed upstream would have silently no-op'd
before, leaving the override stale. The validation runs alongside the
existing application loop so per-field checks share the
collectFlatProperties view.

Reject contradictions: a single field appearing in both makeOptional
and makeRequired is a config error, not last-wins.

Negative-tested: introducing a stale field name to an existing
override now fails generation with a clear message; reverted to the
working set, all 5 existing overrides still emit.

Existing 215 tests pass.
Four schemas affected by API behaviors not captured in the OpenAPI spec:

ConnectionSchemaFieldInputRepresentation
  Server NPEs ("Cannot invoke java.lang.CharSequence.length() because
  this.text is null") when `label` is omitted from a schema field. The
  upsert handler dereferences label unconditionally despite the spec
  marking it optional. New `makeRequired: ["label"]`.

DataStreamFieldMappingInputRepresentation
  Two related issues:
    1. POST input uses `sourceFieldLabel` but GET responses echo
       `sourceFieldName` for the same field. Round-tripping GET→POST
       without renaming fails with JSON_PARSER_ERROR.
    2. `targetFieldReturntype` is required by the create handler —
       when omitted, the mapping is silently dropped from the saved
       data stream with no error.
  Documented in the note; `makeRequired: ["targetFieldReturntype"]`.

DataStreamSourceFieldInputRepresentation
  Same pattern as #1 above: POST uses `dataType` (camelCase), GET
  echoes `datatype` (lowercase). Documented; no shape change.

VectorEmbeddingInputRepresentation
  Server NPEs when vectorEmbeddingRelatedFields is omitted, empty, or
  null. Spec marks it optional. New `makeRequired:
  ["vectorEmbeddingRelatedFields"]`.

Also extend two existing override notes:

DataStreamInputRepresentation
  Add routing note: `dataAccessMode = "Direct_Access"` is required for
  federated/BYOL connectors (Snowflake, Databricks, BigQuery, Iceberg)
  — without it the create returns the misleading
  `400 INTERNAL_ERROR: Unable to post Data Stream: DATA_CONNECTORS is
  not supported`. Direct_Access streams must also OMIT `datasource`.

SemanticSearchInputRepresentation
  Add note: input rejects output-only display-name fields
  (sourceDmoName, sourceDmoFieldName, relatedDmoName,
  relatedDmoFieldName) that GET responses include — round-tripping
  GET→POST returns 500 UNKNOWN_EXCEPTION with no diagnostic body.

Spec drift (unrelated): regenerated output also picks up new fields
on Activation* schemas (activationType, sourceDmoName) added to the
upstream spec.

215 tests pass; typecheck clean.
Schema-shape quirks live in SCHEMA_OVERRIDES; behavioral quirks
(timing, race conditions, asymmetric path-param semantics) belong on
the service-class methods themselves. Convert three thin re-exports
into subclasses solely to host the JSDoc.

ConnectionsService
  - create: name-lock race after DELETE → DUPLICATES_DETECTED for
    several seconds; recommend ~30s retry budget on that error code.
    IngestApi name-rewrite caveat (`<label>_<uuid>`).
  - delete (new override): transient 500 UNKNOWN_EXCEPTION from the
    platform's teardown race even when deletable=true; recommend ~30s
    5xx retry budget.

CalculatedInsightsService
  - create: server-side validation+schedule registration commonly
    takes 30–60s; SDK's default 30s timeout aborts. Recommend
    `{ timeout: 120_000 }`.
  - delete: asynchronous teardown — 204 immediately but record sits
    at status=DELETING for minutes. GET still returns the record
    during that window; same-name re-create returns
    DUPLICATES_DETECTED until teardown completes.

SearchIndexService
  - delete: path parameter accepts the platform-assigned ID but NOT
    the developer name. Asymmetric vs. GET (which accepts either).

Each method retains its original signature and delegates via super.*;
no behavior change, only JSDoc. Existing 215 tests still pass; type
check clean.
@javrrr
javrrr merged commit fc489f5 into main May 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant