[rzo1]

Wikipedia.existsPage(String) and new Category(wiki, title) appended COLLATE utf8mb4_bin to their name comparison on MySQL and MariaDB. With utf8mb3, latin1 or varbinary name columns, which include the legacy DEFAULT CHARSET=utf8 layout, this kept MariaDB from using name_index / nameIndex, and every lookup scanned the whole index. On utf8mb4 columns the index was already used: ref on MariaDB, a range scan on MySQL 8.x. So the gain is mainly a MariaDB one on legacy charsets.

The issue proposed a dual predicate (name = :name AND name = :name COLLATE utf8mb4_bin). It did not hold up: MySQL 8.4 dropped the collated comparison as redundant with the index lookup, so case variants matched. Putting the collation on the column side fails on utf8mb3 databases ("COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8mb3'"). So this takes the issue's alternative.

Changes

• Both queries compare in the column's own collation, and the exact, case-sensitive match is picked in Java from the rows that come back. Depending on the collation, these rows are case, accent, ß/ss or width variants of the title. SQL_COLLATION is removed.
• CATEGORY_BY_NAME_QUERY orders by id, so duplicate exact category names resolve to the lowest id.
• A title the column charset cannot hold, e.g. an emoji on utf8mb3 or CJK on latin1, makes MySQL/MariaDB raise "Illegal mix of collations" (errors 1267/1270/1271). This is treated as not found, as on master, where COLLATE utf8mb4_bin converted the column first.
• The README "Case sensitivity" paragraph and the query Javadoc describe the exact matching.

Behaviour changes

• Duplicate exact category names: the lowest id is used. Before, uniqueResult() threw NonUniqueResultException.
• Stored names with a trailing space no longer match a title without it. utf8mb4_bin pads spaces, String.equals does not. Title never produces such names, so only hand-imported data is affected.
• Everything else is unchanged. Case, accent, ß/ss, NFC/NFD, full-width, ignorable and supplementary-character variants give the same answers as master on every collation tested.

Regression suite

This replaces NameLookupIndexTest. That test was enabled with != HSQLDB and would have failed on PostgreSQL (#727).

• Spec and oracle: contract/name-lookups.tsv is a readable table of 82 cases. The expected answers come from a plain-Java oracle using the production Title, not from the database under test. NameLookupContractSpecTest checks the spec without a database.
• Collations: NameLookupContractTest runs the cases on every engine against several column collations. These are created as extra databases inside the existing CI containers, so there are no new matrix jobs:
  • MySQL/MariaDB: default, utf8mb4_bin, utf8mb4_general_ci, utf8mb3_general_ci and latin1_swedish_ci;
  • PostgreSQL: default and a nondeterministic ICU collation;
  • HSQLDB: default and IGNORECASE.
• Plan test: NameLookupPlanTest checks the SQL Hibernate actually sends. On MySQL/MariaDB it counts rows read (Handler_read*); on PostgreSQL the JSON EXPLAIN must contain no Seq Scan. A control row runs the old COLLATE utf8mb4_bin query and must fail on MariaDB with utf8mb3/latin1 columns.
• Schema check: SchemaAssumptionTest pins the default collations of the images and checks that each variant has the collation it claims.
• Engine opt-in: @EnabledOnEngines(...) makes tests list their engines explicitly, so a new engine cannot silently fall into an engine-specific test.
• Known differences: a ledger records the known pre-existing behaviours of the lookups that follow the collation. An entry that stops occurring fails as stale.

Verification

• Full dkpro-jwpl-api suite:
  • HSQLDB: 1033 tests
  • MariaDB 11.4 and MySQL 8.4: 1316 tests each
  • PostgreSQL 17: 1044 tests
• Collation behaviour was also checked by hand on MySQL 8.0 and MariaDB 10.6/10.11.
• Mutation check. Each of these made the suite fail:
  • reintroducing COLLATE utf8mb4_bin;
  • dropping the Java exact filter;
  • dropping order by id;
  • dropping the collation-mismatch handling.

Not in this PR

getPage(String), getPageIds(String), getCategories(String) and the LIKE-based lookups still follow the column collation, as on master. On a case-insensitive MySQL/MariaDB column, existsPage(x) can be false while getPage(x) returns a case variant. Moving Page.fetchByTitle to the same fetch-and-filter approach is a follow-up.

Fixes #546.

[rzo1]

Rebased onto master to resolve the conflict with #669: Category(wiki, title) keeps #669's single native entity query (select * from Category where name = :name, now without COLLATE so the name index is used) and picks the entity whose name equals the title exactly in Java, throwing WikiPageNotFoundException if none does; NameLookupIndexTest now EXPLAINs that query (Category.CATEGORY_BY_NAME_QUERY) and passes on MariaDB and MySQL.

[rzo1]

Rebased onto master to resolve the conflict with #675: the title lookup now keeps #675's immutable Category.Row and cache fill, but runs the collation-free CATEGORY_BY_NAME_QUERY and builds the Row from the candidate whose name matches exactly in Java.

[mawiesne]

Needs strict / careful review and/or discussion.

[rzo1]

Agreed, this one deserves a careful look. A summary for the review:

What changes

• Wikipedia.existsPage(String) and new Category(wiki, title) no longer append COLLATE utf8mb4_bin. Both queries compare in the column's own collation, which lets MySQL use name_index / nameIndex again, and the exact, case-sensitive match is then picked in Java from the few case variants that come back.
• SQL_COLLATION is removed.

Why not the issue's dual predicate (name = :name AND name = :name COLLATE utf8mb4_bin)

• On MySQL 8.4 it returned wrong results: the optimizer treats the collated comparison as redundant with the index lookup and drops it, so case variants matched. Reproduced in a MySQL 8.4 container.
• Putting the collation on the column side instead fails on legacy utf8mb3 databases ("COLLATION 'utf8mb4_bin' is not valid for CHARACTER SET 'utf8mb3'").

Behaviour changes to check

1. Lookups stay case-sensitive and exact on every engine (WikipediaTest and CategoryTest have case-variant checks).
2. If several categories share exactly the same name, the first one is used; before, uniqueResult() threw.
3. On MariaDB 11.4 the index was used even with COLLATE, so the speed-up is a MySQL one; MariaDB behaves the same as before.

How it is verified

• NameLookupIndexTest runs EXPLAIN on both queries on MariaDB and MySQL and asserts index access; with COLLATE put back it fails on MySQL 8.4.
• The full dkpro-jwpl-api suite passes on HSQLDB, MariaDB 11.4 and MySQL 8.4.

Worth discussing

• Is "first exact match wins" acceptable for duplicate category names, or should it keep throwing?
• Moving the exact match into Java also suits the planned PostgreSQL support (Support PostgreSQL as a database backend #674), where = is already case-sensitive and there is no COLLATE utf8mb4_bin.

[rzo1]

@mawiesne Thanks for insisting on a careful look at the collation side. We checked the change against real servers: MySQL 8.0/8.4, MariaDB 10.6/10.11/11.4, PostgreSQL 17 and HSQLDB 2.7. On MySQL/MariaDB we tested each server's default collation plus utf8mb4_bin, utf8mb4_general_ci, utf8mb4_unicode_ci, utf8mb3_general_ci and latin1_swedish_ci. On PostgreSQL we tested the default and a nondeterministic ICU collation. We compared master with this branch rebased onto current master.

What holds: the exact comparison now happens in Java, so existsPage(String) and getCategory(String) give the same answers as master for case, accent, ß/ss, NFC/NFD, full-width, ignorable and supplementary-character variants on every collation we tried. The exceptions are listed below.

What was wrong, and is fixed in the new commits on this branch, now rebased onto master:

1. On utf8mb3 or latin1 name columns (the legacy DEFAULT CHARSET=utf8 layout), a title the column cannot store, e.g. an emoji or CJK on latin1, made both methods throw an unchecked "Illegal mix of collations" exception (MySQL errors 1267/1270/1271). Master returned false / WikiPageNotFoundException, because COLLATE utf8mb4_bin converted the column first. This is now treated as not found.
2. With duplicate exact category names, master threw NonUniqueResultException. The branch returned whichever row came first, and on HSQLDB and PostgreSQL that was not the lowest id. CATEGORY_BY_NAME_QUERY now has order by id, so the lowest id wins deterministically.
3. NameLookupIndexTest was enabled with selectEngine() != HSQLDB, which also matches PostgreSQL (added in #674: Support PostgreSQL in the API read path, tests and CI (stage 1) #727 after this branch was created). On current master it failed with "column name type was not found", so both postgresql CI jobs would have broken. It is replaced by the suite below.
4. The README "Case sensitivity" paragraph and the Javadoc ("case variants at most") are corrected. On _ai_ci collations the candidate rows can also be accent, ß/ss or width variants.

Stored names with a trailing space no longer match: utf8mb4_bin pads spaces, String.equals does not. We think the new behaviour is correct, but it deserves a changelog line.

Regression suite (the new NameLookupContract*, NameLookupPlanTest and SchemaAssumptionTest classes):

• The spec is contract/name-lookups.tsv, a readable table of 82 tricky cases. The expected answers come from a plain-Java oracle using the production Title, not from the database under test.
• Collations: every engine runs the cases against several column collations, created as extra databases inside the existing CI containers. No new matrix jobs are needed.
• Plan test: checks the SQL Hibernate actually sends. On MySQL/MariaDB it counts rows read (Handler_read*); on PostgreSQL the JSON EXPLAIN must contain no Seq Scan.
• Engine opt-in: tests use @EnabledOnEngines(...), so a new engine cannot silently fall into a test the way PostgreSQL did.
• Schema check: SchemaAssumptionTest pins the default collations of the images, so an image bump fails loudly.
• Proof it catches breakage: it fails when any of these is reintroduced: COLLATE utf8mb4_bin, dropping the Java exact filter, dropping order by id, or dropping the collation-mismatch handling.
• Results: green locally on HSQLDB (1033 tests), MariaDB 11.4 and MySQL 8.4 (1316 each), and PostgreSQL 17 (1044).

On performance: on utf8mb4 columns the old COLLATE utf8mb4_bin query already used the index on all servers we tried: ref on MariaDB, a range scan on MySQL. The measurable gain is on MariaDB with utf8mb3 or latin1 columns, where the old query scanned the whole index. So the earlier statement that "the speed-up is a MySQL one" was backwards. That is also why the plan test counts rows read instead of asserting type=ref.

Out of scope, already on master: getPage(String), getPageIds(String), getCategories(String) and the LIKE-based lookups still follow the column collation. On a case-insensitive MySQL/MariaDB column, existsPage(x) can be false while getPage(x) returns a case variant. We suggest a follow-up issue to move Page.fetchByTitle to the same fetch-and-filter approach. A few other older findings came up along the way (e.g. unescaped _ in LIKE patterns), and we will file them separately.

Open questions for you:

• Is "collation mismatch = not found" acceptable, or would you rather reject titles the column cannot store up front?
• Duplicate category names: lowest id, or keep throwing as before?
• Should the fetchByTitle follow-up be a merge condition for this PR?

[rzo1]

@mawiesne To check the change against real data rather than only our fixtures, could you run the following read-only script on one or two of your real MariaDB JWPL databases (ideally EN and DE) and attach the output? It only reads metadata and the name columns. Only the TSV at the end exports data, and it contains names only.

-- 1. Server and schema defaults
SELECT VERSION(), @@character_set_server, @@collation_server,
       @@character_set_database, @@collation_database;

-- 2. Charset/collation of the name columns and table sizes
SELECT TABLE_NAME, COLUMN_NAME, COLUMN_TYPE, CHARACTER_SET_NAME, COLLATION_NAME
  FROM information_schema.COLUMNS
 WHERE TABLE_SCHEMA = DATABASE() AND COLUMN_NAME = 'name';
SELECT TABLE_NAME, ENGINE, TABLE_COLLATION, TABLE_ROWS
  FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE();
SHOW CREATE TABLE PageMapLine;
SHOW CREATE TABLE Category;

-- 3. Plans and timings: old query (master) vs new query (#670), for a few titles
--    (e.g. 'Berlin', 'Köln', a multi-word title, a title with an emoji or CJK Ext-B if the wiki has one)
ANALYZE FORMAT=JSON SELECT name FROM PageMapLine WHERE name = 'Berlin' COLLATE utf8mb4_bin;
ANALYZE FORMAT=JSON SELECT name FROM PageMapLine WHERE name = 'Berlin';
ANALYZE FORMAT=JSON SELECT * FROM Category WHERE name = 'Berlin' COLLATE utf8mb4_bin;
ANALYZE FORMAT=JSON SELECT * FROM Category WHERE name = 'Berlin' ORDER BY id;

-- 4. Fan-out: how many rows the column collation treats as "equal"
SELECT cnt AS rows_per_name, COUNT(*) AS names
  FROM (SELECT COUNT(*) AS cnt FROM PageMapLine GROUP BY name) t
 GROUP BY cnt ORDER BY cnt;
SELECT name, COUNT(*) AS c FROM PageMapLine GROUP BY name ORDER BY c DESC LIMIT 20;

-- 5. Edge cases the PR changes
SELECT COUNT(*) FROM (SELECT BINARY name AS b FROM Category GROUP BY b HAVING COUNT(*) > 1) d;  -- duplicate exact category names
SELECT COUNT(*) FROM PageMapLine WHERE name LIKE '% ';                                       -- trailing spaces
SELECT COUNT(*) FROM PageMapLine WHERE name REGEXP '[^\\x{0000}-\\x{FFFF}]';                 -- supplementary characters

And one small export: the names of the largest collision groups from step 4, which we would add as real-world cases to name-lookups.tsv:

SELECT p.name, HEX(p.name) FROM PageMapLine p
  JOIN (SELECT name FROM PageMapLine GROUP BY name HAVING COUNT(*) > 1 ORDER BY COUNT(*) DESC LIMIT 200) g
    ON p.name = g.name
 ORDER BY p.name;

Why each part matters:

• Step 2 shows whether your databases use the utf8mb3/latin1 layout. That is where both the full-scan problem and the "Illegal mix of collations" case live. If the columns are utf8mb3, run step 3 anyway: the old query is exactly the one that misbehaves there.
• Step 3 replaces our synthetic plan measurements with real ones. On MariaDB, ANALYZE actually runs the query and reports rows read and time.
• Step 4 measures what the Java-side filtering costs on real data.
• Step 5 shows whether the behaviour changes (lowest id wins, trailing spaces no longer match) affect real data at all.
• The export: HEX(name) makes NFC/NFD differences visible, which plain text hides.

The GROUP BY queries scan the whole table once, so on EN they may take a few minutes.

[mawiesne]

@rzo1

Open questions for you:

• (A) Is "collation mismatch = not found" acceptable, or would you rather reject titles the column cannot store up front?
• (B) Duplicate category names: lowest id, or keep throwing as before?
• (C) Should the fetchByTitle follow-up be a merge condition for this PR?

(A) => acceptable
(B) => lowest id
(C) => no

[rzo1]

@mawiesne Thanks for the answers on (A), (B) and (C).

(A) and (B) are already implemented the way you decided: a collation mismatch (MySQL/MariaDB 1267/1270/1271, e.g. a title a utf8mb3 or latin1 column cannot hold) is treated as not found in existsPage(String) and getCategory(String), and among categories with the same name the lookup orders by id and returns the one with the lowest id.

Per (C), the follow-ups are filed as separate issues (milestone "Bug backlog"):

• Wikipedia.getPage(String) follows the column collation and returns case, accent or ß/ss variants of the title #744: getPage(String) / fetchByTitle follows the column collation and returns case, accent or ß/ss variants (and throws 1267 on titles the column cannot hold).
• Wikipedia.getPageIds(String) does not normalize the title and returns all case variants on case-insensitive collations #745: getPageIds(String) does not normalize the title and returns all case variants on case-insensitive collations.
• Wikipedia.getCategories(String) does not normalize the title and returns categories of case variants on case-insensitive collations #746: getCategories(String) does not normalize the title and returns the categories of case variants on case-insensitive collations.
• Wikipedia.getPageByExactTitle(String) binds the title with blanks and never finds a multi-word title #747: getPageByExactTitle(String) binds the title with blanks instead of underscores and never finds a multi-word title.
• PageQuery title pattern matches case-insensitively on case-insensitive collations #748: The PageQuery title pattern matches case-insensitively on case-insensitive collations.
• Unescaped '_' in LIKE patterns of getDiscussionArchives and the template fragment lookups matches any character #750: An unescaped _ in the LIKE patterns of getDiscussionArchives and the template fragment lookups matches any character.
• PageQuery title patterns and getDiscussionArchives fail on PostgreSQL columns with a nondeterministic collation #751: PageQuery title patterns and getDiscussionArchives fail on PostgreSQL columns with a nondeterministic collation (SQLSTATE 0A000).
• WikipediaTemplateInfo.checkTemplateId(String) does not lower-case the name and returns -1 on case-sensitive columns #749: WikipediaTemplateInfo.checkTemplateId(String) does not lower-case the name and returns -1 on case-sensitive columns. This one is not in the ledger, since the suite covers only the API module.

The divergence ledger (name-lookups-divergences.tsv) now references these issue numbers instead of the placeholders, and the spec test accepts only #<number> references.

One placeholder is gone without an issue: getPageIdsCaseInsensitive returning nothing on the HSQLDB IGNORECASE profile was a fixture problem, not an API bug. HSQLDB keeps the order of an existing index when ALTER COLUMN changes the type to VARCHAR_IGNORECASE, so lookups through name_index stayed case-sensitive. The profile now drops and recreates the name indexes around the type change. With that, IGNORECASE behaves like the case-insensitive MySQL/MariaDB collations, and the ledger records the same divergences for it (#744, #745, #746, #748).