Repository navigation
Conversation
getPageIds(String) bound the title as given and returned every entry the collation of PageMapLine.name treated as equal to it. It now normalizes the title with Title.getWikiStyleTitle(), like getPage(String) and existsPage(String), fetches the names along with the page ids in the collation of the column, so the name index is still used, and keeps only the entries whose name equals the normalized title by code point. A title the column charset cannot hold (MySQL/MariaDB errors 1267, 1270, 1271) and a null or unparsable title now mean not found, i.e. WikiPageNotFoundException, as the method already threw for unknown titles.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #670; will be rebased onto master after it is merged.
What
Wikipedia.getPageIds(String)nowTitle.getWikiStyleTitle(), likegetPage(String)andexistsPage(String)do, andPageMapLine.name(so the name index is used), but fetches(name, pageID)and filters in Java, asexistsPage(String)does since #546: Keep name lookups index-friendly on MySQL and MariaDB #670.Why
The title was bound as given, so
getPageIds("wikipedia API")threwWikiPageNotFoundExceptionwhilegetPage("wikipedia API")found page 1014. On case-insensitive collations (MySQL/MariaDB default,utf8mb4_general_ci,utf8mb3_general_ci,latin1_swedish_ci, HSQLDBVARCHAR_IGNORECASE, PostgreSQL nondeterministic ICU) it returned the ids of all case variants, and even answered a case variant that is not stored. Case variants are whatgetPageIdsCaseInsensitive(String)is for.Behaviour changes
getPageIds("wikipedia API")→[1014]).getPageIds("Mixed_case")→[800002]instead of[800001, 800002, 800003];getPageIds("MiXed_Case")(not stored) →WikiPageNotFoundException. Likewise accent,ß/ssand PAD SPACE variants are no longer returned.WikiPageNotFoundException(the existing contract andtestGetPageIdsByTitleInvalid): this now also covers anullor unparsable title and a title the column charset cannot hold (MySQL/MariaDB errors 1267/1270/1271, viaWikipedia.isCollationMismatch), which used to surface as an unchecked exception.getPageIds(String)among the lookups that keep only the exact match.Tests
name-lookups-divergences.tsv(case-ids,case-ids-unstored,normalized-ids; 12 rows).GET_PAGE_IDS:case-normalized-ids,accent-stripped-ids,sharp-s-ss-ids,emoji-stored-ids,emoji-unstored-ids,cjk-unstored-ids,trailing-space-ids.NameLookupContractTestC6 now binds the normalized probe forgetPageIdstoo, since the lookup normalizes it;NameLookupSpecmarksGET_PAGE_IDSas exact.WikipediaTest: normalization, case variants not found,null/empty not found.mvn -o -pl dkpro-jwpl-api -am verifyon HSQLDB (DEFAULT and IGNORECASE profiles) passes.Fixes #745.