[Python Sources] Automatic schema inference: detect field removals/type changes and infer types without a type pointer

[Chris van den Hurk (sun-cv)]

Automatic schema inference: detect field removals/type changes and infer types without a type pointer

Problem / Current Behavior

For sources with no well-defined schema, no current schema loader manages schema drift fully automatically. Inline schemas miss new fields until manually edited. DynamicSchemaLoader can detect new field names, but it can only infer field types when the API response includes a usable type field to reference via type_pointer + types_mapping - without that, new fields land as string type. Field removals and type changes, including inside nested objects, currently produce no alert at all; both fail silently.

Requested Enhancement

1. Land value-based schema inference (an InferredSchemaLoader) that can infer field types directly from record values, rather than requiring the source API to expose a dedicated type field.
2. Detect and surface field removals and type changes - including within nested objects - as an alert, rather than allowing silent schema drift.

Use Case

Many APIs don't publish a formal schema and don't include type metadata per field. Connector maintainers currently have two options: hardcode/manually maintain a schema (which misses new fields), or use dynamic schema detection that only gets real types if the API happens to expose a type field. Meanwhile, if a field is removed or changes type upstream, nothing alerts the user the sync just starts emitting different data silently. Automatic, value-based type inference plus drift alerts would let syncs stay correct without manual schema maintenance.

Additional Context

Related to CDK schema loader behavior (DynamicSchemaLoader, and a proposed InferredSchemaLoader). Tracked upstream in airbytehq/airbyte-python-cdk#831.

[Airbyte Support Bot (airbyte-support-bot)]

Hi Chris van den Hurk (@sun-cv), thank you for taking the time to write this up!

We're looking into this. Value-based type inference and drift alerting for sources without a formal published schema are framework-level capabilities in the Python CDK, so we've escalated your request to our team for investigation and prioritization: airbytehq/oncall#13296

A couple of things that would help the team scope this:

• Do you have a concrete source/API in mind where the current DynamicSchemaLoader approach falls short (no usable type field in the response)? A short, redacted example of a record shape would be useful.
• For the drift case, what behavior would you expect when a field is removed or changes type upstream — a warning surfaced in the sync logs, a failed sync, or a diff surfaced at discover/refresh time? Preferences here have real trade-offs for existing connections.

If you share any logs or sample payloads, please mask or remove any sensitive information (API keys, passwords, tokens, connection strings, customer data) first.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.