File access and transfer

[amotl]

Pitch

omniload comes from ingesting and processing tabular data, mostly from databases and services. However, the file is still a major resource unit in personal and corporate computing, not killed by service-based computing yet, and files can also include tabular data.

In this spirit, it would be sweet if omniload can add connectors to integrate with more data sources / standards that host, provide, or are otherwise concerned with file transfer in one way or another, also beyond the existing integration lanes.

[amotl]

Connector requests

We proposed adding a few more connectors using the existing fsspec backend of omniload/dlt.

• Cloudflare R2 #163
• Object Storage Service (OSS) oss:// #180
• Dropbox #211
• WebDAV #212
• SharePoint and OneDrive #213

Other types of integrations still need discussions.

• Rclone #139
• rsync #210

[amotl]

There are also other types of ad hoc file transfer tools that save you from needing to use Dropbox, WeTransfer, or others. croc is one of them, see also croc-app. Thank you, @WalBeh.

[amotl]

SeaweedFS

It looks like SeaweedFS could be a good companion to omniload.