Skip to content

Commit 8dc1b3a

Browse files
committed
overrides: document BigQuery wire-shape quirks
Two override notes captured from a BigQuery zero-copy probe against awt 2026-06-11: DataConnectionInputRepresentation (new override) Connector-type casing on the wire is UPPERCASE — the spec lists TitleCase (e.g. "Snowflake", "BigQuery") but the live /ssot/connections endpoint returns ILLEGAL_QUERY_PARAMETER_VALUE for anything except the UPPERCASE form. Also documents the BigQuery credentials/parameters shape: authenticationOption=KeyPair, serviceAccountEmail, privateKey (which must be the full SA JSON key file content — the connector parses the JSON internally to extract project_id + private_key + client_email; sending bare PEM fails with "[BIGQUERY] [native] Failed to connect"). DataStreamInputRepresentation (extended note) advancedAttributes for Direct_Access streams uses GENERIC Snowflake-style keys regardless of connector flavor — `database`/`schema`/`object`. For BigQuery this maps to project/dataset/table; sending native BigQuery key names returns "INVALID_ARGUMENT: database cannot be empty in advanced attr". Also: INCREMENTAL refresh on BigQuery requires acceleration enabled on the connection — without it, returns "INVALID_ARGUMENT: acceleration should be enabled for incremental column"; use TOTAL_REPLACE if acceleration isn't configured. The schemas.ts JSDoc was hand-applied (not via npm run generate) because the upstream OpenAPI spec has churned since the last regen and pulling in all the spec changes would conflate this PR with unrelated upstream work. A future regen run should produce identical JSDoc from the override config.
1 parent 1e041f7 commit 8dc1b3a

2 files changed

Lines changed: 6 additions & 2 deletions

File tree

scripts/generate-types.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ interface SchemaOverride {
5858
*/
5959
const SCHEMA_OVERRIDES: Record<string, SchemaOverride> = {
6060
DataStreamInputRepresentation: {
61-
note: "Spec bugs: dataLakeObjectInfo should accept single or array; mappings and sourceFields are not required for all connector types. Routing note: dataAccessMode='Direct_Access' is required for federated/BYOL connectors (Snowflake, Databricks, BigQuery, Iceberg) — without it the server returns `400 INTERNAL_ERROR: Unable to post Data Stream: DATA_CONNECTORS is not supported` even when the connector is GA. Direct_Access streams must also OMIT the top-level `datasource` field (otherwise: `DataSource name should be empty for External data streams`); the connection binding is established via connectorInfo.connectorDetails.name instead.",
61+
note: "Spec bugs: dataLakeObjectInfo should accept single or array; mappings and sourceFields are not required for all connector types. Routing note: dataAccessMode='Direct_Access' is required for federated/BYOL connectors (Snowflake, Databricks, BigQuery, Iceberg) — without it the server returns `400 INTERNAL_ERROR: Unable to post Data Stream: DATA_CONNECTORS is not supported` even when the connector is GA. Direct_Access streams must also OMIT the top-level `datasource` field (otherwise: `DataSource name should be empty for External data streams`); the connection binding is established via connectorInfo.connectorDetails.name instead. advancedAttributes for Direct_Access streams uses GENERIC keys regardless of connector flavor: `database`/`schema`/`object` (Snowflake-shaped). For BigQuery this maps to project/dataset/table — sending the BigQuery-native key names returns `INVALID_ARGUMENT: database cannot be empty in advanced attr`. INCREMENTAL refresh on BigQuery requires acceleration enabled on the connection — without it, create returns `INVALID_ARGUMENT: acceleration should be enabled for incremental column`; use TOTAL_REPLACE if acceleration isn't configured.",
6262
makeOptional: ["mappings", "sourceFields"],
6363
fieldTypes: {
6464
dataLakeObjectInfo: "DataLakeObjectInputRepresentation | DataLakeObjectInputRepresentation[]",
@@ -74,6 +74,9 @@ const SCHEMA_OVERRIDES: Record<string, SchemaOverride> = {
7474
note: "Spec bugs: recordModifiedFieldName and orgUnitIdentifierFieldName are not required for all DLO types",
7575
makeOptional: ["recordModifiedFieldName", "orgUnitIdentifierFieldName"],
7676
},
77+
DataConnectionInputRepresentation: {
78+
note: "Connector-type casing on the wire is NOT what the spec lists. The spec types `connectorType` as the TitleCase form (e.g. \"Snowflake\", \"BigQuery\") but the live `/ssot/connections` endpoint rejects everything except UPPERCASE and returns `ILLEGAL_QUERY_PARAMETER_VALUE: ConnectorType [BigQuery] is not supported`. Use \"SNOWFLAKE\", \"BIGQUERY\", \"AMAZONS3\", etc. on every wire call (POST body, ?connectorType= query string). Probed against awt 2026-06-11. Per-connector credential/parameter shapes are NOT documented in the spec; the connector descriptor at GET `/ssot/connector-descriptors/{TYPE}` is the authoritative source. BigQuery specifically needs: credentials = [{paramName: \"authenticationOption\", value: \"KeyPair\"}, {paramName: \"serviceAccountEmail\", value: <SA email>}, {paramName: \"privateKey\", value: <full SA JSON key file content>}]; parameters = [{paramName: \"projectId\", value: <GCP project>}]. Sending just the PEM body (or just the bare PEM) for `privateKey` results in `Connection wasn't successful: [BIGQUERY] [native] Failed to connect`; the connector parses the JSON internally to extract project_id + private_key + client_email.",
79+
},
7780
DataObjectFieldInputRepresentation: {
7881
note: "Spec bugs: API expects `dataType` instead of `type` for field data type; isDynamicLookup missing but required for PATCH to work",
7982
makeOptional: ["type"],

src/schemas.ts

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1820,6 +1820,7 @@ export type DataCleanRoomUseCaseTypeRepresentation = {
18201820
url?: string;
18211821
useCaseTypes?: string[];
18221822
}
1823+
/** @override Connector-type casing on the wire is NOT what the spec lists. The spec types `connectorType` as the TitleCase form (e.g. "Snowflake", "BigQuery") but the live `/ssot/connections` endpoint rejects everything except UPPERCASE and returns `ILLEGAL_QUERY_PARAMETER_VALUE: ConnectorType [BigQuery] is not supported`. Use "SNOWFLAKE", "BIGQUERY", "AMAZONS3", etc. on every wire call (POST body, ?connectorType= query string). Probed against awt 2026-06-11. Per-connector credential/parameter shapes are NOT documented in the spec; the connector descriptor at GET `/ssot/connector-descriptors/{TYPE}` is the authoritative source. BigQuery specifically needs: credentials = [{paramName: "authenticationOption", value: "KeyPair"}, {paramName: "serviceAccountEmail", value: <SA email>}, {paramName: "privateKey", value: <full SA JSON key file content>}]; parameters = [{paramName: "projectId", value: <GCP project>}]. Sending just the PEM body (or just the bare PEM) for `privateKey` results in `Connection wasn't successful: [BIGQUERY] [native] Failed to connect`; the connector parses the JSON internally to extract project_id + private_key + client_email. */
18231824
export type DataConnectionInputRepresentation = {
18241825
credentials: Schemas["DataConnectionParameterInputRepresentation"][];
18251826
method: "Egress" | "Ingress";
@@ -2248,7 +2249,7 @@ export type DataStreamFieldMappingInputRepresentation = {
22482249
export type DataStreamFieldMappingRepresentation = Schemas["DataStreamFieldMappingRepresentation"];
22492250
export type DataStreamFrequencyInputRepresentation = Schemas["DataStreamFrequencyInputRepresentation"];
22502251
export type DataStreamFrequencyRepresentation = Schemas["DataStreamFrequencyRepresentation"];
2251-
/** @override Spec bugs: dataLakeObjectInfo should accept single or array; mappings and sourceFields are not required for all connector types. Routing note: dataAccessMode='Direct_Access' is required for federated/BYOL connectors (Snowflake, Databricks, BigQuery, Iceberg) — without it the server returns `400 INTERNAL_ERROR: Unable to post Data Stream: DATA_CONNECTORS is not supported` even when the connector is GA. Direct_Access streams must also OMIT the top-level `datasource` field (otherwise: `DataSource name should be empty for External data streams`); the connection binding is established via connectorInfo.connectorDetails.name instead. */
2252+
/** @override Spec bugs: dataLakeObjectInfo should accept single or array; mappings and sourceFields are not required for all connector types. Routing note: dataAccessMode='Direct_Access' is required for federated/BYOL connectors (Snowflake, Databricks, BigQuery, Iceberg) — without it the server returns `400 INTERNAL_ERROR: Unable to post Data Stream: DATA_CONNECTORS is not supported` even when the connector is GA. Direct_Access streams must also OMIT the top-level `datasource` field (otherwise: `DataSource name should be empty for External data streams`); the connection binding is established via connectorInfo.connectorDetails.name instead. advancedAttributes for Direct_Access streams uses GENERIC keys regardless of connector flavor: `database`/`schema`/`object` (Snowflake-shaped). For BigQuery this maps to project/dataset/table — sending the BigQuery-native key names returns `INVALID_ARGUMENT: database cannot be empty in advanced attr`. INCREMENTAL refresh on BigQuery requires acceleration enabled on the connection — without it, create returns `INVALID_ARGUMENT: acceleration should be enabled for incremental column`; use TOTAL_REPLACE if acceleration isn't configured. */
22522253
export type DataStreamInputRepresentation = {
22532254
advancedAttributes?: { [key: string]: string };
22542255
connectorInfo: Schemas["ConnectorInputRepresentation"];

0 commit comments

Comments
 (0)