Skip to content

perf(import-lists): optimize local library lookup and defer Hardcover cursor commit - #101

Open
Josh-Archer wants to merge 2 commits into
Chaptarr:developfrom
Josh-Archer:perf/import-list-sync-optimization
Open

perf(import-lists): optimize local library lookup and defer Hardcover cursor commit#101
Josh-Archer wants to merge 2 commits into
Chaptarr:developfrom
Josh-Archer:perf/import-list-sync-optimization

Conversation

@Josh-Archer

Copy link
Copy Markdown
Contributor

Description

This PR addresses two related performance and data-integrity issues in the import list subsystem:

  1. Local Lookup Optimization: ImportListSyncService.BuildImportListLocalLookup() previously performed full-table scans across the entire library (GetAllAuthors() and GetAllBooks()) on every list sync pass. On large libraries (thousands of books), this caused multi-minute CPU spikes and database contention. This change scopes the local lookup strictly to the batch of item provider IDs fetched during the current sync pass.
  2. Deferred State/Cursor Commit (CommitState): HardcoverLibraryImportList.Fetch() previously updated the sync cursor in the database immediately upon receiving the API response. If downstream processing failed (disk error, validation failure, or unhandled exception during ProcessListItems), the cursor had already moved forward, permanently dropping un-imported books. This change introduces CommitState() to the IImportList / ImportListBase lifecycle, ensuring state and cursors are persisted only after list items are successfully processed.

Database Migration

NO

How was this tested?

  • Unit Tests:
    • Added HardcoverLibraryImportListCursorFixture testing cursor non-persistence during Fetch() and proper persistence on CommitState().
    • Verified ImportListSyncServiceSearchOnAddFixture passes with the updated scoped local lookup.
  • Environment: Tested on Linux (K3s container runtime with PostgreSQL Flexible Server backend) across a ~9,000+ item library with Hardcover Want-to-Read and Owned sync lists.

Highlights & Assumptions

  • Highlights:
    • Drastically reduces memory and database overhead during scheduled and manual import list syncs.
    • Ensures transactional integrity of import list state cursors (no silent item drops on processing errors).
    • Default no-op CommitState() implementation on ImportListBase ensures zero breaking changes for existing or third-party import list implementations.
  • Assumptions:
    • Provider IDs can be split and matched via _authorService.FindByProviderId and _bookService.FindByProviderId.
    • Future stateful import lists (e.g. Goodreads, StoryGraph) can leverage the same CommitState() pattern.

AI Disclosure & Workflow Sharing

Note on AI: This implementation and test suite were developed and refined using agentic AI coding workflows (Codex / Claude 3.7). We understand and fully welcome the project's code review scrutiny. If the maintainers or community are interested, we are very happy to share our agentic harness, testing harness, and validation workflow setup!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant