Data Source Types

[cosmic-jester-uk]

Hi, I can see file reads in the code but I can't see anything relating to API calls or SharePoint or Shortcuts (Dataverse etc). Am I missing something or are they not included in the accelerator?
Thanks,
Dave

[bennyaustin]

Hi @cosmic-jester-uk, the ELT framework supports ingestion from file-based data sources, for files dropped into Event Grid–supported sources (e.g Azure Blob and ADLS). You'd use SourceFileDrop* columns of ELT.IngestDefinition instead to configure for this scenario. You ingestion pipeline will use [ELT].[GetIngestDefinition_FileDrop] instead of [ELT].[GetIngestDefinition], which is intended for database/API based data sources.

At this stage, there isn’t a sample pipeline included for file-drop ingestion yet. This is something I’m planning to add in a future release. Code contributions are very welcome if you’re interested.

For shortcuts and mirrored data sources (e.g. Dataverse), I’m working on a new ELT pattern as these don’t require traditional ingestion. A feature branch is now open where you can follow progress on that work.

Hope that helps, and thanks for checking it out 👍