Support (almost) any bytes in a filename. - #166
Merged
Conversation
Owner
Author
|
Description is updated to reflect the latest round of changes, many of which came out of Claude's review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary: Filenames with non-UTF-8 bytes can still be played, and will be displayed reasonably (with replacement characters). There is also a UI change: the
.mp3extension is not shown.This PR is my attempt to deal comprehensively with the problem of funny bytes. This may be slightly heavyweight for a problem I've never encountered in practice, but the robustness and theoretical correctness are appealing. I decided I preferred this approach to rejecting files with problematic names. That is, I chose the first, more involved, option in #97.
Practical concerns include that there might be a single non-ASCII character in a filename that's (say) ISO-8859-1 or which got mangled by re-encoding (this happens), and this shouldn't make the file unplayable. And rejecting files itself adds complexity, as I'd need filtering and reporting steps. Indeed, semantics can become odd: if directory A has bad bytes, but its subdirectory B does not, does that mean
A/*.mp3are not playable (because A is shown in the UI), butA/B/*.mp3are?The key addition is that every file path is stored both as a raw POSIX path for sending to the decoder, and as UTF-8 text that can be shown on-screen.
Changes:
.textfield which represents how they are rendered (and searched across). The constraint is a bit like duck typing, as kind of an experiment with the record dot. This replaces classLookup.�.wcwidthreports -1 for all of these (except NUL). This includes ASCII 0 to 31 and 127. (Other Unicode codepoints have wrong widths in at least one terminal, Kitty, but this seems to be its idiosyncrasy which I either can't do anything about or wouldn't be worth dealing with.).mp3extensions in the app. I've been considering this change for a long time since it's a bit unsightly and not really useful, and.textopened a natural way to revisit it. One justification is that what is displayed should match what is searched, and for search it makes less sense to include extensions. E.g., someone searchingst.mpwould expect to findstompandstamp, but not a file ending inst.mp3.Other notes:
dedupfunction inText.hskeeps filenames stored only once in memory in the most common case where there is no encoding issue, by pointing.textinto the same byte array.takeFileNames inPlaylist.hscreated a slice of the full path of every file, meaning the full paths are held in memory forever. I addressed this by copying out the slice we actually want. I still plan to totally overhaul Playlist, so this will also have to be handled then.DuplicateRecordFields..text, might be Text (strict or lazy, perhaps Builder) or some other type (maybe even a newtype) reflecting its distinct semantics from raw bytes. However, one consideration remains: the ultimate consumer of displayed textwaddnstrtakes bytes and can currently be directly passed the underlying ByteString array. This question needs further reflection.