fix: auto-size multipart parts for File uploads (from_local_sync + hashed from_local)

[kumare3]

Follow-up to #1462, which fixed Dir.from_local_sync. File had the same class of bug in two places.

Summary

On obstore-backed filesystems (s3/gs/abfs), fsspec's BufferedFile("wb") and our _open_obstore_bypass both write with a fixed 10 MiB buffer, and obstore uploads one multipart part per buffer. That caps a single object at ~97.6 GiB (10,000 parts × 10 MiB) — 2× the Dir threshold from #1462, but the same failure (InvalidRange / part-limit rejection at CompleteMultipartUpload). The 160 GiB file in #1462 would also fail through File.from_local_sync.

Two paths were affected:

1. File.from_local_sync (non-hash and precomputed-hash branches) uploaded via raw fs.open(remote_path, "wb") + shutil.copyfileobj — never touched the obstore-aware storage.put that auto-sizes parts.
2. File.from_local / from_local_sync with a HashMethod streamed through storage.put_stream, which had no way to know the total size and so kept the fixed 10 MiB part. This one affected the async path too.

Changes

• File.from_local_sync: route through syncify(storage.put), mirroring Dir.from_local_sync (fix: auto-size multipart parts for Dir.from_local_sync uploads #1462) and Dir.download_sync.
• storage.put_stream(..., size_hint=): new optional kwarg. If the hinted size would blow past _MAX_SAFE_PARTS at the default buffer, chunk_size is scaled up via the existing _compute_upload_chunk_size. Streams below ~88 GiB are byte-for-byte unchanged (no override is passed). Explicit chunk_size=/buffer_size= kwargs still win.
• New module-level _upload_hashed() helper in _file.py, shared by from_local and from_local_sync (via syncify), passing os.path.getsize(local_path) as size_hint. This also collapses the two duplicated hashed-upload implementations into one.
• Factored the 10 * 2**20 literal into _STREAM_WRITE_BUFFER_SIZE.

Not changed (on purpose)

• DataFrame: encoders write via df.to_parquet(storage_options=…) / pq.write_table(filesystem=…), i.e. pandas/pyarrow drive an fsspec file-like with no size known up front, so there's no clean place to plumb a hint. A single >97 GiB parquet file written from one in-memory frame isn't a realistic path today, so leaving it.
• File.download_sync: still raw fs.get. Not a correctness issue (downloads have no part cap); it just skips the parallel reader. Can follow up separately if wanted.
• File.open_sync("wb"): stream API with no size knowledge; callers can pass buffer_size=.

Tests

• Rewrote the three from_local_sync remote tests (they mocked fs.open) to assert storage.put / storage.put_stream(size_hint=…) are used and the hash is still computed from the streamed bytes.
• Added an async from_local hashed test asserting size_hint is passed.
• Added a parametrized put_stream test: no hint / small hint → no chunk_size override; 200 GiB hint → chunk_size > default and ceil(size / chunk_size) <= _MAX_SAFE_PARTS.

🤖 Generated with Claude Code