Skip to content

feat(pgvector): add table replacement helpers - #4162

Open
31groot wants to merge 2 commits into
deepset-ai:mainfrom
31groot:feat/pgvector-zero-downtime-replacement
Open

31groot wants to merge 2 commits into
deepset-ai:mainfrom
31groot:feat/pgvector-zero-downtime-replacement

Conversation

@31groot

@31groot 31groot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes

  • Add synchronous and asynchronous helpers for replacing an existing Pgvector document table with a staging table.
  • Validate table identifiers, database and schema compatibility, and table structure before performing the replacement.
  • Rename indexes during the cutover to support the staging table becoming the active table.
  • Retain the previous active table as a backup rather than dropping it immediately.
  • Add tests covering successful replacement, missing staging tables, existing backup tables, and incompatible table configurations.
  • Update the relevant docstrings to document the new functionality and its usage.

The goal is to reduce downtime when replacing Pgvector tables while preserving the previous table as a backup.

Limitation: Writers should be paused during the cutover. This implementation reduces replacement downtime but does not guarantee uninterrupted concurrent writes.

How did you test it?

Ran the following commands from integrations/pgvector with a local Pgvector PostgreSQL instance:

  • hatch run fmt src/haystack_integrations/document_stores/pgvector/table_replacement.py tests/test_table_replacement.py
  • hatch run fmt-check src/haystack_integrations/document_stores/pgvector/document_store.py src/haystack_integrations/document_stores/pgvector/table_replacement.py tests/test_table_replacement.py
  • hatch run test:integration tests/test_table_replacement.py — 5 passed
  • hatch run test:integration tests/test_document_store.py — 57 passed, 2 skipped
  • hatch run test:integration tests/test_document_store_async.py — 53 passed, 2 skipped
  • hatch run test:types — passed
  • git diff --check — passed

Notes for the reviewer

  • The previous table is retained as a backup after replacement.
  • The replacement helpers support both synchronous and asynchronous usage.
  • Please review the transactional cutover, index renaming, validation, and backup-table behavior.
  • Writers must be paused during the cutover; uninterrupted concurrent writes are not guaranteed.

Checklist

@31groot
31groot requested a review from a team as a code owner October 26, 2026 12:54
@31groot
31groot requested review from davidsbatista and removed request for a team October 26, 2026 12:54
@github-actions github-actions Bot added topic:CI integration:pgvector type:documentation Improvements or additions to documentation labels Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report (pgvector)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/pgvector/src/haystack_integrations/document_stores/pgvector
  document_store.py
  table_replacement.py 144-145, 265-372, 394-508
Project Total  

This report was generated by python-coverage-comment-action

@github-actions

Copy link
Copy Markdown
Contributor

Coverage report (supabase)

This PR does not seem to contain any modification to coverable code.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage report (alloydb)

This PR does not seem to contain any modification to coverable code.

@31groot
31groot force-pushed the feat/pgvector-zero-downtime-replacement branch from e0f3fba to b7d8f31 Compare October 26, 2026 13:12
@31groot 31groot changed the title Feat/pgvector zero downtime replacement feat(pgvector): add table replacement helpers Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration:pgvector topic:CI type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support zero-downtime vectorization on PgvectorDocumentStore

1 participant