Unhandled ArgumentException (invalid Unicode code points) in ReleaseTitleMatchScorer during RSS sync

[jbob06]

Version: v0.9.936.0 (running chaptarr/chaptarr:latest, develop branch)

Description

An unhandled System.ArgumentException in ReleaseTitleMatchScorer during RSS sync causes the entire release to be dropped from consideration instead of just being skipped/logged, which can silently prevent otherwise-good releases from being matched/grabbed.

Stack trace

System.ArgumentException: String contains invalid Unicode code points. (Parameter 'strInput')
   at System.Globalization.Normalization.IcuNormalize(String strInput, NormalizationForm normalizationForm)
   at NzbDrone.Core.Utilities.UnicodeComparisonNormalizer.NormalizeWordsWithSourceSpans(String text, IReadOnlyList`1 sourceSpans, Boolean stripDiacritics) in ./Chaptarr.Core/Utilities/UnicodeComparisonNormalizer.cs:line 58
   at NzbDrone.Core.Parser.ReleaseTitleMatchScorer.TokenizeWithSpans(String text) in ./Chaptarr.Core/Parser/ReleaseTitleMatchScorer.cs:line 1278
   at NzbDrone.Core.Parser.ReleaseTitleMatchScorer.BuildContradictoryVariants(Book targetBook, IEnumerable`1 authorCatalogBooks) in ./Chaptarr.Core/Parser/ReleaseTitleMatchScorer.cs:line 656
   at NzbDrone.Core.Parser.ReleaseTitleMatchScorer.FindBestMatch(String releaseTitle, String authorName, IEnumerable`1 candidateBooks, String releaseAuthorHint, IEnumerable`1 authorCatalogBooks) in ./Chaptarr.Core/Parser/ReleaseTitleMatchScorer.cs:line 185
   at NzbDrone.Core.DecisionEngine.DownloadDecisionMaker.<>c__DisplayClass13_0.<TryMapRssRelease>b__7(IGrouping`2 group) in ./Chaptarr.Core/DecisionEngine/DownloadDecisionMaker.cs:line 490
   at System.Linq.Enumerable.IteratorSelectIterator`2.MoveNext()
   at System.Linq.Enumerable.IEnumerableWhereIterator`1.ToList()
   at NzbDrone.Core.DecisionEngine.DownloadDecisionMaker.TryMapRssRelease(ReleaseInfo report, ParsedBookInfo parsedBookInfo)
   at NzbDrone.Core.DecisionEngine.DownloadDecisionMaker.GetBookDecisions(List`1 reports, Boolean pushedRelease, SearchCriteriaBase searchCriteria)+MoveNext() in ./Chaptarr.Core/DecisionEngine/DownloadDecisionMaker.cs:line 185

Repro

1. Trigger POST /api/v1/command {"name":"RssSync"} against an indexer with a reasonably large feed (500 releases in my case).
2. Watch logs at trace/debug level — DownloadDecisionMaker logs Processing 500 releases, then throws the above exception repeatedly (8 times in one ~40s sync) with Error|DownloadDecisionMaker|Couldn't process release. and no further detail about which release.

What I ruled out

I suspected our own book catalog had a corrupted title (since BuildContradictoryVariants tokenizes the target book's full author-catalog), so I dumped all 25,206 Books.Title rows from chaptarr.db and strict-decoded every one as UTF-8 — all valid, none malformed. So the invalid code point(s) causing IcuNormalize to throw must be coming from the incoming ReleaseInfo/release title on the indexer's RSS feed itself (a torrent name with corrupted/malformed encoding), not from anything already stored in Chaptarr's database. I wasn't able to capture the literal offending title since the request crashes before anything is persisted/logged with the release name attached.

Suggested fix

TokenizeWithSpans/NormalizeWordsWithSourceSpans should catch/sanitize invalid input (e.g. strip or replace ill-formed UTF-16 sequences before calling String.Normalize, or catch ArgumentException per-release) rather than letting one malformed release title propagate an unhandled exception up through the Select/ToList() chain in DownloadDecisionMaker.GetBookDecisions/TryMapRssRelease — as written, a single bad title in an RSS batch aborts processing for that batch/group instead of just being skipped, which risks other legitimate releases in the same batch never being evaluated.

[jbob06]

Correction on scope, found while preparing a fix: I originally described this as risking "other legitimate releases in the same batch never being evaluated." That's wrong — I misread the call chain. DownloadDecisionMaker.GetBookDecisions (DownloadDecisionMaker.cs:110-333) already wraps each report in its own try/catch inside the foreach loop; the Select().ToList() chain that actually crashes (TryMapRssRelease → FindBestMatch → BuildContradictoryVariants) is entirely local to a single report's processing, not a batch-wide LINQ pipeline. So this doesn't abort the RSS sync or drop other releases — only the one report with the offending title is affected.

The real (still worth fixing) impact: that one report gets caught by the generic handler and turned into a permanent, non-bypassable rejection ("Unexpected error processing release", RejectionType.Permanent) — so any release whose title (or, via BuildContradictoryVariants, any title in the matched author's catalog) contains a supplementary-plane character (an emoji, or a character like a CJK Extension B ideograph outside the Basic Multilingual Plane) can never be grabbed, silently, until this is fixed.

Root cause, precisely: UnicodeComparisonNormalizer.NormalizeWordsWithSourceSpans iterates the input one UTF-16 code unit (char) at a time and calls .Normalize(NormalizationForm.FormD) on each single unit. Any character outside the BMP is represented as a surrogate pair (two code units) in .NET strings — normalizing one half of that pair alone is what throws ArgumentException: String contains invalid Unicode code points, since a lone surrogate isn't well-formed UTF-16 on its own. This isn't data corruption on the indexer's or Chaptarr's side; it's a real, valid character being torn in half by the per-code-unit loop.

I have a fix + regression tests ready and will open a PR shortly.