Skip to content

Add async PAPI leaderboard cache and placeholders - #5301

Merged
nossr50 merged 3 commits into
masterfrom
papi_leaderboard_rfe
Jul 9, 2026
Merged

Add async PAPI leaderboard cache and placeholders#5301
nossr50 merged 3 commits into
masterfrom
papi_leaderboard_rfe

Conversation

@nossr50

@nossr50 nossr50 commented Jun 14, 2026

Copy link
Copy Markdown
Member

This PR adds an async cache updated at regular configurable intervals, which allows placeholders to be used for n leaderboard positions (default max tracked rank is 100).

Some examples:

%mcmmo_mctop_mining:1% -> value of top mining entry
%mcmmo_mctop_name_mining:1% -> name of top mining player
%mcmmo_mctop_name_mining:23% -> name of 23rd best mining player
%mcmmo_mctop_all:1% -> value of top player's power level
%mcmmo_mctop_name_all:1% -> name of player with highest power level (sum of skill values)

Powerlevel / Overall alias behavior:

  • overall, all, and powerlevel all map to the same powerlevel leaderboard data.

New config keys:

  • General.PlaceholderAPI.Leaderboards.Max_Tracked_Rank (default: 100, clamped between 10 and 1000)
  • General.Leaderboards.Refresh_Interval_Seconds.SQL (default: 60)
  • General.Leaderboards.Refresh_Interval_Seconds.FlatFile (default: 600)

Behavior notes:

  • Placeholder requests are cache-backed (no per-request DB reads).
  • Cache refresh and cache swap are async.
  • Each refresh reads every scope (17 skills + overall) in one bulk backend read: a single users-file scan on FlatFile, one query batch on SQL.
  • If no leaderboard placeholder was resolved since the previous refresh, the next scheduled refresh is skipped; the first lookup afterwards is served from cache and triggers one immediate async refresh. Servers that never use these placeholders do no recurring cache work.
  • Invalid positions (blank/non-numeric/0/negative) return empty.
  • Positions above Max_Tracked_Rank return empty.
  • Positions no player currently holds return empty.
  • If a refresh fails mid-flight (e.g. the database drops during the read), the cache keeps serving the last good snapshot instead of blanking out.
  • The cache starts only after PlaceholderAPI accepts the expansion registration, and refreshes stop cleanly on plugin disable.

This PR addresses: #5171, #4800, and #4291


Refresh intervals

The leaderboard cache is refreshed on a timer, with a separate interval per database backend under General.Leaderboards.Refresh_Interval_Seconds (Max_Tracked_Rank stays under PlaceholderAPI, since it caps in-memory placeholder storage).

  • SQL defaults to 60s. SQL leaderboard queries are indexed and cheap, so they can refresh frequently. On SQL this interval only governs the placeholder cache; /mctop and /mcrank always query live data.
  • FlatFile defaults to 600s. A FlatFile rebuild scans the entire user file, which is expensive on large servers, so it refreshes far less often. Cache refreshes force a rebuild, so placeholder data is never more than one interval old; the same value throttles command-triggered rebuilds (/mctop, /mcrank), and concurrent callers race a CAS on the throttle so only one full-file scan runs at a time. Failed rebuilds don't consume the throttle window, so the next caller retries immediately instead of being stuck with stale or empty boards.
  • Both intervals have a 60s floor; values below 60 are treated as 60.
  • PapiExpansion selects the interval based on the active backend.

SQL leaderboard query performance

readLeaderboard sorts each leaderboard on a single column with a stable tiebreak on skills.user_id DESC, which matches the implicit PK ordering at the end of each InnoDB secondary index — the engine resolves the whole sort with a backward scan of the per-skill index and touches only the LIMIT window, with no filesort. Every leaderboard column carries a per-skill index (plus total).

The tiebreak choice, the indexes, and the join order are all required. The query pins skills as the driving table with STRAIGHT_JOIN: the renamed-player filter is a range predicate on the indexed user column, and on large tables MySQL 8 otherwise drives from users via that index and filesorts every qualifying row, leaving the per-skill indexes unused. At 300k rows on MySQL 8 and MariaDB 10.11, indexed page-1 queries run ~1-2ms on both engines (median of 15 runs), versus roughly 40-320ms without the indexes depending on engine and cache warmth. A Docker-tagged test runs EXPLAIN on the exact production query and asserts no plan step filesorts on either engine.

Tie ordering: players with identical values order newest-registered-first on SQL (previously alphabetical), and /mcrank uses the same tiebreak so rank numbers agree with /mctop pages and the placeholders. FlatFile keeps its alphabetical tie order.

Ghost rows: the database keeps a row under _INVALID_OLD_USERNAME_ when a player renames. These were always meant to be hidden from /mctop, but the old filter used LIKE-style backslash escapes inside an equality comparison, which MySQL treats as literal backslashes — the filter never matched anything. The fixed filter actually excludes them, and /mcrank's rank counts now exclude them too, so ghost rows can neither appear on a page nor shift anyone's rank number.

Automatic per-skill leaderboard indexes

  • Fresh SQL installs get these indexes directly from the CREATE TABLE schema.
  • Existing databases get them via an idempotent ADD_SKILL_LEADERBOARD_INDEXES upgrade that checks INFORMATION_SCHEMA with a single grouped query and only adds the missing ones. This is portable across MySQL/MariaDB, whereas CREATE INDEX IF NOT EXISTS is not.
  • The probe only counts an index whose leading column matches (seq_in_index = 1) — a composite index that merely contains the skill column elsewhere can't serve the leaderboard ORDER BY, so it doesn't suppress creation of the dedicated index.
  • The migration is best-effort: each index is attempted independently, failures are logged and skipped rather than propagated, and the upgrade is only marked complete once every column is handled (so a partial failure retries on the next startup). mcMMO keeps running even if an index cannot be created.
  • If an index named idx_<skill> already exists but leads with a different column, mcMMO assumes it was created deliberately, logs one warning, and treats that column as handled instead of retrying the DDL on every startup.
  • The column list is derived from SkillTools.NON_CHILD_SKILLS plus total, and the legacy updateStructure ADD COLUMN path indexes newly added skill columns as it creates them, so skills added by future updates are covered on both fresh and upgraded databases.

FlatFile thread-safety

All FlatFile leaderboard scopes live in one immutable generation object behind a single volatile reference, published inside the file lock right after the scan: rebuilds are serialized scan-to-publish, a slower older scan can never overwrite a newer one, readers never observe lists from two different scans, and /mcrank resolves every skill against the same generation. The rebuild throttle is an AtomicLong CAS so concurrent callers can't perform duplicate full-file scans; only successful rebuilds arm it, and leaderboards that were never successfully built bypass it entirely.

Backend read contract

DatabaseManager gains readLeaderboardSnapshot(perScopeLimit), used by the cache: one call returns the top rows of every leaderboard scope, propagates backend read failures (instead of swallowing them into an empty result like the command-facing readLeaderboard), and bypasses backend-level staleness. On FlatFile it forces a single rebuild and slices every scope from that one generation — which is why a cache refresh costs one file scan rather than one per skill. This is what lets the cache distinguish "database outage" from "genuinely empty leaderboard" and keep its last good snapshot.

Tests

  • Unit tests for the parser, cache behavior, alias/name-value consistency, last-good-snapshot retention on refresh failure, idle-skip and reactivation on the next lookup, truncation of data sources that return more rows than requested, and no-refresh-after-shutdown.
  • FlatFile integration tests, including the 60s refresh floor, a snapshot-read-bypasses-throttle check, failed-rebuild retry behavior (both the throttled and snapshot paths), every-scope-in-one-rebuild coverage, and a concurrency regression test running forced rebuilds against concurrent page/rank reads.
  • SQL Testcontainers integration tests (MySQL + MariaDB), run under -Psql-tests (@Tag("docker")) and sharing one container pair per JVM: leaderboard indexes exist after a fresh install, the migration adds them on a pre-existing unindexed table and marks the upgrade complete, re-running the migration is idempotent, a composite index does not suppress the dedicated index, an index name collision is honored without wedging the migration, the legacy column-upgrade path indexes the columns it adds, EXPLAIN shows no filesort on the leaderboard query, /mcrank agrees with /mctop when a renamed player's ghost row ties or outranks a real player, and readLeaderboardSnapshot propagates read failures that readLeaderboard swallows.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds new PlaceholderAPI tokens to query mcMMO leaderboards by rank position (skill + overall/powerlevel), backed by an async, periodically refreshed in-memory snapshot cache to keep placeholder resolution fast and non-blocking.

Changes:

  • Added an async LeaderboardPlaceholderCache that periodically refreshes top-N leaderboard snapshots and serves placeholder lookups from immutable, atomically swapped snapshots.
  • Registered new %mcmmo_mctop_*:<position>% and %mcmmo_mctop_name_*:<position>% placeholders for all non-child skills plus overall aliases (overall, all, powerlevel).
  • Added config options for cache depth and refresh interval, plus unit/integration tests for parsing, cache behavior, and backend correctness.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/main/java/com/gmail/nossr50/placeholders/LeaderboardPlaceholderCache.java Implements async, single-flight snapshot refresh and position lookup APIs (contains a critical scheduler callback issue).
src/main/java/com/gmail/nossr50/placeholders/LeaderboardPlaceholderInputParser.java Adds shared parsing for :<position> placeholder params.
src/main/java/com/gmail/nossr50/placeholders/McTopNamePlaceholder.java New placeholder to return player names for a leaderboard position.
src/main/java/com/gmail/nossr50/placeholders/McTopPositionPlaceholder.java New placeholder to return values/levels for a leaderboard position.
src/main/java/com/gmail/nossr50/placeholders/PapiExpansion.java Wires up the cache + registers new placeholders; adds shutdown hook.
src/main/java/com/gmail/nossr50/config/GeneralConfig.java Adds config getters + validation for max tracked rank and refresh interval.
src/main/resources/config.yml Documents and provides defaults for the new PlaceholderAPI leaderboard cache settings.
src/main/java/com/gmail/nossr50/mcMMO.java Stores PapiExpansion reference and shuts it down/unregisters it during disable.
src/test/java/com/gmail/nossr50/placeholders/LeaderboardPlaceholderCacheTest.java Unit tests for snapshot building, bounds behavior, and refresh failure semantics.
src/test/java/com/gmail/nossr50/placeholders/LeaderboardPlaceholderInputParserTest.java Unit tests for position parsing behavior.
src/test/java/com/gmail/nossr50/placeholders/McTopPlaceholdersTest.java Ensures name/value placeholders map to the same cached row and alias naming is correct.
src/test/java/com/gmail/nossr50/placeholders/LeaderboardPlaceholderFlatFileIntegrationTest.java Integration tests against flatfile backend and refresh concurrency guard.
src/test/java/com/gmail/nossr50/placeholders/LeaderboardPlaceholderSqlIntegrationTest.java Docker-tagged integration tests against MySQL/MariaDB backends.
Changelog.txt Documents the new placeholders and config keys.
pom.xml Bumps project version to 2.2.054-SNAPSHOT.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/main/java/com/gmail/nossr50/placeholders/PapiExpansion.java
@nossr50
nossr50 force-pushed the papi_leaderboard_rfe branch 4 times, most recently from 43d911e to dd512b2 Compare July 8, 2026 00:37
@nossr50
nossr50 force-pushed the papi_leaderboard_rfe branch from dd512b2 to 340f2e1 Compare July 9, 2026 07:57
nossr50 added 3 commits July 9, 2026 10:47
PapiExpansion splits incoming params on the first ':' and resolves the
prefix against its token map, and every mctop placeholder depends on
that seam delivering the position suffix. These tests pin the routed
(empty string from an unrefreshed cache) versus unrouted (null) return
contract for the value, name, and overall-alias variants, missing
position segments, and case-insensitive token lookup.
Placeholder names were built with bare toLowerCase(), which follows
the JVM default locale; on Turkish-locale JVMs enum names containing
'I' produce a dotless ı, so tokens like mctop_mining would register
under a different spelling than documented. The case-insensitive token
map happens to absorb the mismatch, but the registered names should
not depend on that. All seven placeholder classes now lower-case with
Locale.ENGLISH.
@nossr50
nossr50 force-pushed the papi_leaderboard_rfe branch from 340f2e1 to 3bec5a4 Compare July 9, 2026 18:11
@nossr50
nossr50 merged commit 0dcf35b into master Jul 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants