fix(storage): don't clobber the global fsspec registry on import

[cosmicBboy]

Symptom

In a hybrid v1/v2 setup — a flytekit (v1) task that imports flyte (v2) to remotely launch v2 tasks — merely importing flyte breaks all of flytekit's S3/GCS/Azure I/O in that process. Every output upload fails with:

Configuration key: 'cache_regions' is not valid for store 'S3'

surfacing as a FlyteUploadDataException / SYSTEM error that also masks the task's real error (the failed task can't even upload its error document).

Root cause

src/flyte/storage/_storage.py ran this at import time:

register(_OBSTORE_SUPPORTED_PROTOCOLS, asynchronous=True)  # obstore.fsspec.register

obstore.fsspec.register registers with clobber=True, replacing the global fsspec registry entries for s3, gs, abfs, and abfss. flytekit's data_persistence.py resolves filesystems through that registry (fsspec.filesystem("s3", cache_regions=True, ...) with s3fs-specific kwargs), and obstore's store rejects those kwargs.

Fix (two parts)

1. flyte's own I/O no longer depends on the global registry. For obstore-supported protocols (s3, gs, abfs, abfss), get_underlying_filesystem() now instantiates a flyte-owned per-protocol subclass of obstore.fsspec.FsspecStore directly (_obstore_filesystem_class, mirroring exactly what obstore.fsspec.register(..., asynchronous=True) builds). fsspec's _Cached metaclass still applies to direct instantiation, so instance caching is preserved, and the obstore bypasses (_split_path / _construct_store) and anonymous-access fallback keep getting obstore-backed instances. Other protocols still go through fsspec.filesystem(...).

2. Global registration is now non-destructive gap-filling. _register_obstore_for_missing_protocols() registers obstore for a protocol only if (a) nothing is already registered for it and (b) no other implementation is importable (fsspec.get_filesystem_class raises ImportError). The importability probe matters because fsspec resolves known_implementations lazily — in a hybrid image the registry can be empty at import flyte time even though s3fs is installed, so a registry-only check would still clobber. This keeps today's behavior in pure-v2 images (no s3fs installed → obstore registered so pandas/pyarrow s3:// paths keep working) while never stomping on flytekit's s3fs/gcsfs/adlfs, regardless of import order.

The flyte protocol registration (FlyteFS) is unchanged. fsspec.filesystem(protocol) uses elsewhere in the SDK (models.py, io/_file.py, _remote_fs.py) only read fs.sep for path joining and work with whichever implementation resolves, so they are intentionally untouched.

Testing

New tests/internal/storage/test_fsspec_registry.py:

• a pre-registered s3fs-style implementation survives the registration helper, and fsspec.filesystem("s3") still resolves to it
• get_underlying_filesystem("s3") returns an obstore-backed FS (_split_path/_construct_store) even when s3fs-style owns the registry
• the helper registers obstore when nothing is registered or importable, and leaves an importable-but-not-yet-registered implementation alone (lazy-resolution case)
• the flytekit collision itself: fsspec.filesystem("s3", cache_regions=True) reaches the registered class instead of obstore
• the flyte-owned class mirrors obstore.fsspec.register (protocol/asynchronous attrs, class + instance caching)

Also verified end-to-end in fresh subprocesses for both environments (pure-v2: obstore gets registered for all four protocols; hybrid: pre-registered s3fs-style survives and accepts cache_regions=True while flyte's own I/O stays obstore-backed). Existing storage/io/init suites pass (pytest -k "not integration and not sandbox"), plus ruff check, ruff format --check, and mypy on the touched module.

🤖 Generated with Claude Code