Configure per-connection CPU and memory resource requirements from the UI

[Ardalan Jalal (ardalan-jalal)]

Feature Request: Configure Per-Connection Resource Requirements from the UI

Airbyte version

Airbyte Self-Managed 2.0.1, running on Kubernetes.

Problem

Airbyte supports setting CPU and memory resource requirements at the connection level, but there is currently no way to configure these values from the Airbyte UI in my deployment.

Connection-specific resource requirements can currently be configured by directly updating the Airbyte database, for example:

UPDATE connection
SET resource_requirements = '{
  "cpu_request": "0.1",
  "cpu_limit": "0.5",
  "memory_request": "256Mi",
  "memory_limit": "1024Mi"
}'
WHERE id = '<connection-id>';

Directly modifying Airbyte's internal database should not be necessary for a normal operational configuration.

Use case

Different connections can have drastically different workloads.

For example, I have MongoDB → ClickHouse CDC connections where the initial snapshot has already completed. One connection processes approximately 500,000 CDC records per day, distributed throughout the day.

This connection requires significantly fewer resources during normal CDC operation than:

• a connection performing an initial snapshot;
• a high-volume CDC connection;
• a large batch connection.

Using one global resource configuration therefore causes either:

• small CDC connections to reserve unnecessarily large amounts of CPU and memory; or
• large connections to be under-provisioned.

Connection-level resource configuration already provides the correct solution, but currently requires direct database access.

Requested solution

Please expose connection-specific resource requirements in the Airbyte UI, ideally under:

Connection → Settings → Advanced settings → Resources

For example:

CPU request:       100m
CPU limit:         500m

Memory request:    256Mi
Memory limit:      1Gi

It would also be useful for the UI to show:

• the effective resource configuration for the connection;
• whether the value is inherited from a connector/global default;
• an option to override those values for this connection;
• an option to reset the connection back to inherited/default values.

Why this is important

Airbyte already supports connection-specific resource requirements internally, so this is primarily an operational/UI gap.

Requiring administrators to directly modify the connection.resource_requirements field in Airbyte's PostgreSQL database is difficult to automate safely and bypasses the normal Airbyte configuration interface.

For Kubernetes installations with many connections, exposing this in the UI would allow much better resource utilization and make it practical to run lightweight CDC connections alongside heavier workloads without over-reserving cluster resources.

Expected behavior

An Airbyte administrator should be able to configure CPU and memory requests/limits for each individual connection without directly modifying the Airbyte database.