[bug] epub-fixer accepts ISO 639-2/B language codes that calibre silently discards, making imported books invisible

[mysterfr]

Describe the bug

kindle_epub_fixer.py::fix_book_language() validates an epub's dc:language tag against an allowed_languages list (line 576) that was taken from Amazon's KDP documentation. That list uses ISO 639-2/B (bibliographic) codes fre, ger, dut, gre, cze, ice, per, etc.

Calibre only understands ISO 639-2/T (terminologic) and ISO 639-1. canonicalize_lang('fre') returns None.

The result is that the fixer inspects <dc:language>fre</dc:language>, declares it valid, and leaves it untouched. And then calibredb add silently discards it.
The book is created with no row in books_languages_link at all.

That matters because cps/db.py:1577 builds the visibility filter as:

lang_filter = Books.languages.any(Languages.lang_code == current_user.filter_language())

.any() is false for a book with no languages, so the book is excluded from every query (including search in UI) for any user whose "Books language" is set to anything other than "Show all".
The import reports success, the file lands in processed_books/imported/, a cwa_import row is written, and the book simply cannot be found in the UI. Nothing is logged.

23 codes in allowed_languages are accepted by the fixer and then dropped by calibre.
Verified against the calibre bundled in the image: every code on the left returns None, every code on the right canonicalizes cleanly:

alb -> sqi
arm -> hye
baq -> eus
bur -> mya
chi -> zho
cze -> ces
dut -> nld
fre -> fra
geo -> kat
ger -> deu
gre -> ell
ice -> isl
mac -> mkd
mao -> mri
may -> msa
per -> fas
rum -> ron
slo -> slk
tib -> bod
wel -> cym
in -> id
iw -> he
ji -> yi

(The last three are deprecated ISO 639-1 codes, leading into the same failure mode.)

To Reproduce

Steps to reproduce the behavior:

1. Go to Settings → Edit Users, open a user, and set Books language to French (anything other than "Show all").
2. Take an epub whose OEBPS/content.opf contains dc:languagefre</dc:language>. Many French epubs from mainstream publishers use this code.
3. Drop it into the ingest folder and wait for the ingest to finish. The log reports success:
   [ingest-processor] Added to Calibre database
4. Log in as that user and search for a word from the title. No result.
5. Log in as an admin whose Books language is "Show all" and run the same search. The book is there.
6. Confirm the cause directly:

$ sqlite3 metadata.db "select id from books where id not in (select book from books_languages_link);"
<the book's id is listed but it has no language>

$ calibre-debug -c "from calibre.utils.localization import canonicalize_lang; print(canonicalize_lang('fre'))"
None

Expected behavior

The epub-fixer should normalize the language tag to a form calibre can actually consume, so fre is stored as fra and the book is visible to users with a language filter.

Either map the 23 codes above to their /T equivalents before writing the tag, or remove them from allowed_languages so the existing "invalid tag" path handles them.

Failing that, an import that ends up with no language at all should at minimum be logged as a warning, since the book silently disappears from the UI for filtered users.

Screenshots

Not applicable — the bug reproduces on the command line as shown above. (Visually it is just an empty search result page.)

Configuration(please complete the following information):

• OS: Debian GNU/Linux 13 (trixie), kernel 6.12.95, rootless Docker 29.2.1
• Hardware: x86_64 KVM/QEMU VPS
• Version: CWA v4.1.38, image ghcr.io/new-usemame/calibre-web-nextgen@sha256:2474a0d1fdbb1eb47e016cd624d4aec6ec108dee92719eb46bab7d, bundled calibre 9.11

Additional context

Found on a 4500-book, mostly French library where 12 books had been silently invisible for months.
They all had <dc:language>fre</dc:language>.
Fixed after the fact with:

calibredb set_metadata --field languages:fra <id> --library-path=...

Two notes that may be useful:

• This is easy to mistake for a failed import, which sends you round a second loop: you re-drop the file, it is rejected as a duplicate, and the ingest reports success again.
• The library here uses split-library mode (metadata.db and the book files on separate mounts) with NETWORK_SHARE_MODE=true, but neither appears to be relevant as the language is lost at calibredb add time regardless.