Skip to content

Support (almost) any bytes in a filename. - #166

Merged
galenhuntington merged 8 commits into
masterfrom
char-safe
Aug 27, 2026
Merged

Support (almost) any bytes in a filename.#166
galenhuntington merged 8 commits into
masterfrom
char-safe

Conversation

@galenhuntington

@galenhuntington galenhuntington commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Summary: Filenames with non-UTF-8 bytes can still be played, and will be displayed reasonably (with replacement characters). There is also a UI change: the .mp3 extension is not shown.

This PR is my attempt to deal comprehensively with the problem of funny bytes. This may be slightly heavyweight for a problem I've never encountered in practice, but the robustness and theoretical correctness are appealing. I decided I preferred this approach to rejecting files with problematic names. That is, I chose the first, more involved, option in #97.

Practical concerns include that there might be a single non-ASCII character in a filename that's (say) ISO-8859-1 or which got mangled by re-encoding (this happens), and this shouldn't make the file unplayable. And rejecting files itself adds complexity, as I'd need filtering and reporting steps. Indeed, semantics can become odd: if directory A has bad bytes, but its subdirectory B does not, does that mean A/*.mp3 are not playable (because A is shown in the UI), but A/B/*.mp3 are?

The key addition is that every file path is stored both as a raw POSIX path for sending to the decoder, and as UTF-8 text that can be shown on-screen.

Changes:

  • File and directories now have a .text field which represents how they are rendered (and searched across). The constraint is a bit like duck typing, as kind of an experiment with the record dot. This replaces class Lookup.
    • Bytes invalid under UTF-8 are replaced with the Unicode replacement character .
    • Unprintable characters are similarly replaced. This prevents them from messing up the display, and also their wcwidth reports -1 for all of these (except NUL). This includes ASCII 0 to 31 and 127. (Other Unicode codepoints have wrong widths in at least one terminal, Kitty, but this seems to be its idiosyncrasy which I either can't do anything about or wouldn't be worth dealing with.)
    • Filenames with CR and LF cannot be sent to mpg123's line protocol and so are quietly skipped, both individual files and whole directories. (Also NUL though that's not possible under POSIX, but I threw it in anyway.) Workarounds are possible: I could for example create a temporary symlink to the file and then play that; this would be very low power-to-weight but I'm still considering it out of a stubborn sense of completeness. A lesser mitigation is to throw up a warning, or an error if on the command line. However, for now I'm doing nothing but filtering them out.
    • Closes Decide how to deal with file paths with weird bytes. #97.
  • Filesystem encoding allows filenames with unusual bytes to be passed on the command line. Thus, this closes Non-UTF-8 filenames on the command line are mangled at the argv boundary #111.
  • I no longer display the .mp3 extensions in the app. I've been considering this change for a long time since it's a bit unsightly and not really useful, and .text opened a natural way to revisit it. One justification is that what is displayed should match what is searched, and for search it makes less sense to include extensions. E.g., someone searching st.mp would expect to find stomp and stamp, but not a file ending in st.mp3.
  • Some tests are added.

Other notes:

  • I have tested this by creating a directory full of files with bizarre filenames. They show up and can be played with no problems.
  • A dedup function in Text.hs keeps filenames stored only once in memory in the most common case where there is no encoding issue, by pointing .text into the same byte array.
  • The above brought to my attention an existing related issue: that the takeFileNames in Playlist.hs created a slice of the full path of every file, meaning the full paths are held in memory forever. I addressed this by copying out the slice we actually want. I still plan to totally overhaul Playlist, so this will also have to be handled then.
  • I renamed File and Dir fields since field name collision is no longer an issue. I had to add DuplicateRecordFields.
  • The size and hi fields are unneeded and dropped from Dir.
  • The once compelling case for ByteString everywhere is reduced by this change. UI text, such as .text, might be Text (strict or lazy, perhaps Builder) or some other type (maybe even a newtype) reflecting its distinct semantics from raw bytes. However, one consideration remains: the ultimate consumer of displayed text waddnstr takes bytes and can currently be directly passed the underlying ByteString array. This question needs further reflection.

@galenhuntington

Copy link
Copy Markdown
Owner Author

Description is updated to reflect the latest round of changes, many of which came out of Claude's review.

@galenhuntington
galenhuntington merged commit 51d1655 into master Aug 27, 2026
6 checks passed
@galenhuntington
galenhuntington deleted the char-safe branch August 27, 2026 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Non-UTF-8 filenames on the command line are mangled at the argv boundary Decide how to deal with file paths with weird bytes.

1 participant