Skip to content

#745: Normalize the title and match it exactly in getPageIds(String) - #752

Open
rzo1 wants to merge 1 commit into
fix/issue-546from
fix/issue-745
Open

rzo1 wants to merge 1 commit into
fix/issue-546from
fix/issue-745

Conversation

@rzo1

@rzo1 rzo1 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Stacked on #670; will be rebased onto master after it is merged.

What

Wikipedia.getPageIds(String) now

  • normalizes the title with Title.getWikiStyleTitle(), like getPage(String) and existsPage(String) do, and
  • returns only the entries whose name equals the normalized title by code point. The query still compares in the collation of PageMapLine.name (so the name index is used), but fetches (name, pageID) and filters in Java, as existsPage(String) does since #546: Keep name lookups index-friendly on MySQL and MariaDB #670.

Why

The title was bound as given, so getPageIds("wikipedia API") threw WikiPageNotFoundException while getPage("wikipedia API") found page 1014. On case-insensitive collations (MySQL/MariaDB default, utf8mb4_general_ci, utf8mb3_general_ci, latin1_swedish_ci, HSQLDB VARCHAR_IGNORECASE, PostgreSQL nondeterministic ICU) it returned the ids of all case variants, and even answered a case variant that is not stored. Case variants are what getPageIdsCaseInsensitive(String) is for.

Behaviour changes

  • Titles are normalized: blanks become underscores, the first letter is upper-cased (getPageIds("wikipedia API") → [1014]).
  • Exact matching on every collation: getPageIds("Mixed_case") → [800002] instead of [800001, 800002, 800003]; getPageIds("MiXed_Case") (not stored) → WikiPageNotFoundException. Likewise accent, ß/ss and PAD SPACE variants are no longer returned.
  • Not found keeps being signalled by WikiPageNotFoundException (the existing contract and testGetPageIdsByTitleInvalid): this now also covers a null or unparsable title and a title the column charset cannot hold (MySQL/MariaDB errors 1267/1270/1271, via Wikipedia.isCollationMismatch), which used to surface as an unchecked exception.
  • The Javadoc describes the normalization, the exact matching and the not-found case; the README "Case sensitivity" section lists getPageIds(String) among the lookups that keep only the exact match.

Tests

  • Removed the Wikipedia.getPageIds(String) does not normalize the title and returns all case variants on case-insensitive collations #745 ledger rows from name-lookups-divergences.tsv (case-ids, case-ids-unstored, normalized-ids; 12 rows).
  • New spec cases for GET_PAGE_IDS: case-normalized-ids, accent-stripped-ids, sharp-s-ss-ids, emoji-stored-ids, emoji-unstored-ids, cjk-unstored-ids, trailing-space-ids.
  • NameLookupContractTest C6 now binds the normalized probe for getPageIds too, since the lookup normalizes it; NameLookupSpec marks GET_PAGE_IDS as exact.
  • WikipediaTest: normalization, case variants not found, null/empty not found.
  • Local: mvn -o -pl dkpro-jwpl-api -am verify on HSQLDB (DEFAULT and IGNORECASE profiles) passes.

Fixes #745.

getPageIds(String) bound the title as given and returned every entry the
collation of PageMapLine.name treated as equal to it. It now normalizes
the title with Title.getWikiStyleTitle(), like getPage(String) and
existsPage(String), fetches the names along with the page ids in the
collation of the column, so the name index is still used, and keeps only
the entries whose name equals the normalized title by code point.

A title the column charset cannot hold (MySQL/MariaDB errors 1267, 1270,
1271) and a null or unparsable title now mean not found, i.e.
WikiPageNotFoundException, as the method already threw for unknown titles.
@rzo1 rzo1 added this to the 2.4.0 milestone Oct 1, 2026
@rzo1
rzo1 requested a review from mawiesne October 1, 2026 07:13
@rzo1 rzo1 self-assigned this Oct 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant