target-iceberg is a Singer target for Iceberg.
Build with the Meltano Target SDK.
- Python 3.10, 3.11, or 3.12 (3.13 is not supported yet).
- Apache Iceberg access via PyIceberg with the PyArrow extra (installs
pyiceberg-corefor Arrow-based I/O). This project pins PyArrow 21.x alongside PyIceberg.
Install from GitHub:
pipx install git+https://github.com/ORG_NAME/target-iceberg.git@main| Setting | Required | Default | Description |
|---|---|---|---|
| credential | True | None | Rest catalog user credential |
| catalog_uri | True | None | Catalog URI, e.g. https://api.catalog.io/ws/ |
| warehouse | True | None | Warehouse name |
| catalog_type | True | None | rest or jdbc |
| namespace | True | None | The namespace where data will be written |
| add_record_metadata | False | None | Add metadata to records. |
| validate_records | False | 1 | Whether to validate the schema of the incoming streams. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an addtional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
A full list of supported settings and capabilities for this target is available by running:
target-iceberg --aboutThis Singer target will automatically import any environment variables within the working directory's
.env if the --config=ENV is provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the .env file.
You can easily run target-iceberg by itself or in a pipeline using Meltano.
target-iceberg --version
target-iceberg --help
# Test using the "Carbon Intensity" sample:
tap-carbon-intensity | target-iceberg --config /path/to/target-iceberg-config.jsonFollow these instructions to contribute to this project.
Use Poetry (2.x recommended). Runtime and tooling dependencies are declared in pyproject.toml; development-only packages (for example pytest and singer-sdk[testing]) live in the dev dependency group ([tool.poetry.group.dev.dependencies]).
pipx install poetry
poetry installEnsure your environment uses Python 3.10–3.12 (see Requirements).
Start by setting up the local catalog environment:
docker compose upCreate tests within the tests subfolder and
then run:
poetry run pytestYou can also test the target-iceberg CLI interface directly using poetry run:
poetry run target-iceberg --helpTesting with Meltano
Note: This target will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Next, install Meltano (if you haven't already). This repo is tested with Meltano 4.x (for example 4.1.2); poetry install also resolves Meltano into the same environment if you prefer not to use pipx.
# Install Meltano (e.g. via pipx)
pipx install meltano
# From the root of your clone of this repository
meltano installNow you can test and orchestrate using Meltano:
# Test invocation:
meltano invoke target-iceberg --version
# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
meltano run tap-carbon-intensity target-icebergSee the dev guide for more instructions on how to use the Meltano Singer SDK to develop your own Singer taps and targets.