[destination-s3-data-lake] Support AWS Lake Formation LF-tags with Glue catalog #84212
Replies: 1 comment
|
Hi Rich Ramsell (@rramsell), thank you for the detailed feature request! We're looking into this. Adding Lake Formation LF-tag configuration to the S3 Data Lake destination's Glue catalog path is a reasonable ask, and the context you provided about your LF-TBAC governance model and the existing AWS Datalake destination configuration is very helpful. We've escalated this to our team for consideration: airbytehq/oncall#13278 In the meantime, if you have any additional details that would help scope this — for example whether tagging the Glue database alone would be sufficient for your access model, or whether per-table tags are required — feel free to add them here. Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL. |
Uh oh!
There was an error while loading. Please reload this page.
Feature request
We'd like the S3 Data Lake destination to support applying AWS Lake Formation LF-tags when using AWS Glue as the Iceberg catalog.
Use case
Our data lake uses Lake Formation tag-based access control (LF-TBAC). Tables created in the Glue Catalog need specific LF-tags applied in order to participate in our existing data-access model.
The S3 Data Lake destination otherwise looks like a strong fit for us because it provides a supported Iceberg + Glue catalog path and supports Incremental Append + Deduped.
However, the current destination configuration does not appear to expose any options for configuring Lake Formation LF-tags on the Glue databases or tables it creates.
Desired behavior
Ideally, the destination configuration would allow users to specify:
and apply those LF-tags to the Glue database and/or tables created by the destination.
For example:
Existing implementation/example
Airbyte's AWS Datalake destination already exposes Lake Formation-related configuration, including:
A similar approach in the S3 Data Lake destination would let users adopt the newer Iceberg/Glue destination while continuing to use LF-tag-based governance.
Why this matters
Without LF-tag support, tables created automatically by Airbyte require an additional out-of-band tagging process before they are usable under an LF-TBAC governance model.
Native support would make the S3 Data Lake destination usable in environments where Glue + Lake Formation governance is required.
All reactions