Valid UTF-8 files are wrongly reported as [Binary file] when a multi-byte character straddles the 1024-byte read boundary

[kotophalk]

Summary

A fully valid UTF-8 source file gets its content replaced with [Binary file] whenever a multi-byte UTF-8 character happens to start within the last few bytes of the first 1024 bytes (_CHUNK_SIZE). The binary/text detection only reads and decodes the first chunk, so a multi-byte character split across the chunk boundary raises UnicodeDecodeError and the whole file is misclassified as binary.

The file itself is perfectly valid UTF-8 — git, editors, and python all read it without any issue.

Environment

• gitingest version: 0.3.1 (also reproducible on gitingest.com)
• Python: 3.12
• OS: Linux

Steps to reproduce

Create a UTF-8 file where a multi-byte character (e.g. a Cyrillic letter, an em dash, an emoji) begins at byte offset 1023, so its continuation byte lands at offset 1024 — just outside the chunk:

# make a repro file: 1023 ASCII bytes of padding, then a 2-byte Cyrillic 'ч' (D1 87)
data = b"# " + b"x" * 1021 + "ч".encode("utf-8") + b"\nprint('hello')\n"
with open("repro.py", "wb") as f:
    f.write(data)

# sanity check: the file is valid UTF-8
open("repro.py", encoding="utf-8").read()   # succeeds, no error

Then run gitingest on a repo containing this file. The output for repro.py is:

================================================
FILE: repro.py
================================================
[Binary file]

Root cause

Detection reads only the first _CHUNK_SIZE bytes and decodes that raw slice:

src/gitingest/utils/file_utils.py

_CHUNK_SIZE = 1024  # bytes

def _read_chunk(path: Path) -> bytes | None:
    ...
    return fp.read(_CHUNK_SIZE)

src/gitingest/schemas/filesystem.py

if not _decodes(chunk, "utf-8"):
    return "[Binary file]"

_decodes calls chunk.decode("utf-8"), which raises UnicodeDecodeError when the chunk ends in the middle of a multi-byte sequence:

'utf-8' codec can't decode byte 0xd1 in position 1023: unexpected end of data

Because the slice is cut at an arbitrary byte offset, a valid multi-byte character can be truncated even though the file is fully valid UTF-8. The classification is therefore position-dependent and fires on legitimate text files that contain any non-ASCII content near the boundary (comments/strings in Cyrillic, CJK, accented Latin, em dashes, emoji, etc.).

Expected behavior

Valid UTF-8 files should always be treated as text, regardless of where non-ASCII characters fall relative to the 1024-byte read window.

Suggested fix

Don't let an incomplete trailing multi-byte sequence in the chunk cause a false negative. A few options:

1. Decode the chunk incrementally so a truncated tail is tolerated:

   import codecs
   
   def _decodes(chunk: bytes, encoding: str) -> bool:
       decoder = codecs.getincrementaldecoder(encoding)()
       try:
           decoder.decode(chunk, final=False)  # don't force-finish the last char
       except UnicodeDecodeError:
           return False
       return True

2. Or read the chunk on a character boundary (e.g. io.TextIOWrapper/fp.read(n) in text mode with the candidate encoding), so no character is ever split.

3. Or, for the specific "unexpected end of data" case, retry after trimming the incomplete trailing bytes before deciding the file is binary.

Option 1 is the smallest change and keeps the existing chunk-based approach.

Additional notes

• The presence of a NUL byte remains a legitimate binary signal; this fix only addresses the truncated-multibyte false positive.

[kanavgoyal898]

Hi! I was able to reproduce this locally and confirmed the root cause.

In layman’s terms: the code checks only the first 1024 bytes to decide whether a file is text. But UTF-8 characters can span multiple bytes. If the 1024-byte cutoff slices a character in half, the sample looks invalid even though the whole file is perfectly valid text, so Gitingest wrongly labels it as binary.

For a valid UTF-8 file where the first 1024-byte chunk ends on the first byte of ч, the current behavior is:

file decodes fully as UTF-8: True
first chunk length: 1024 bytes
last byte in first chunk: 0xd1
before fix output: [Binary file]

A small fix would be, as already mentioned to decode the sampled chunk with an incremental decoder using final=False, so an incomplete trailing character at the end of the sample does not cause a false binary classification. Malformed UTF-8 inside the sampled chunk would still fail normally.

With that change, the same file would be treated as text and its contents would be included instead of [Binary file].

I’d be happy to work on this and open a PR with a focused regression test for the chunk-boundary case, plus a guard test to ensure malformed UTF-8 is still reported as binary.

[github-actions]

Hi there! We haven’t seen activity here for 45 days, so I’m marking this issue as stale.
If you’d like to keep it open, please leave a comment within 10 days. Thanks!