Skip to content

Commit 02aa9f9

Browse files
committed
fix: handle missing content-length header in _get_tokenizer_config_size
When a server uses chunked transfer encoding, neither content-length nor x-goog-stored-content-length headers are present, leaving size as None. The subsequent int() cast then raises a TypeError. Add an explicit None check that raises a ValueError instead, which is caught gracefully by the existing try/except in both callers (get_hf_tokenizer and async_get_hf_tokenizer) — tokenizer download proceeds normally. Fixes: #762
1 parent 756b1d8 commit 02aa9f9

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

src/cohere/manually_maintained/tokenizers.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,4 +99,6 @@ def _get_tokenizer_config_size(tokenizer_url: str) -> float:
9999
if size:
100100
break
101101

102-
return round(int(typing.cast(int, size)) / 1024 / 1024, 2)
102+
if size is None:
103+
raise ValueError("No content-length header found (server may use chunked transfer encoding)")
104+
return round(int(size) / 1024 / 1024, 2)

0 commit comments

Comments
 (0)