Skip to content

✨(backend) Export import instance data for migration - #2789

Draft
magopian wants to merge 21 commits into
mainfrom
feat/export-import-instance-data-for-migration
Draft

magopian wants to merge 21 commits into
mainfrom
feat/export-import-instance-data-for-migration

Conversation

@magopian

@magopian magopian commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

A partner is using both a pre-prod instance (version 3.4.2) and a prod instance (version 5.2.1). They want to migrate all the documents, accesses, requests... from the pre-prod to the prod instance.

Proposal

This PR adds two management commands. The assumption is that the import command is run on a version that is up to date with the version that ran the export, or more recent.

  • export management command: export in a JSONL file the models that are present in v3.4.2, and also export models that exist in main, if they're present on the instance (so this command exports as much as possible). Everything is exported without any filters.
  • import management command: import all the models found in the JSONL file. Some logic is applied: rewrite the UUID of users that are already existing, rewrite the TreeBeard files so as not to overwrite the existing documents...

@magopian
magopian force-pushed the feat/export-import-instance-data-for-migration branch 4 times, most recently from 35c8042 to d0620b4 Compare October 6, 2026 14:58
This is the first model to be exported. Dynamically export any existing
field of the model, this way this management command should be able to
run on any version.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
This only exports the documents metadata. The media lives in S3, the
content lives in either S3 or yhub depending on if the migration to the
yhub collaboration server has already occured.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export the document accesses

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export invitations to access a document

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export documents opened by authenticated users using a link share

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export the favorites

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export the access requests

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export user threads, comments and reactions

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export user reconciliations when a user had two accounts

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Export mentions of users in documents, and whether they've already been
notified by email

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
@magopian
magopian force-pushed the feat/export-import-instance-data-for-migration branch from d0620b4 to 39b8996 Compare October 6, 2026 15:08
Users that already exist aren't duplicated, instead the imported UUID is
stored in a mapping dict so all the future imported models will have
their users' FK rewritten to the existing UUID instead of the imported
one.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Document are only imported if not already existing.
The source path (computed using TreeBeard) is reconstructed so as not to
overwrite the ones in the target instance.

The `duplicated_from_id` field is not imported for now, it would require
a second pass over all documents.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Import DocumentAccess, remapping user FK if needed

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Import the table as is, bypassing the `clean()` that doesn't create an
Invitation for an existing user.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Straightforward LinkTrace imports

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Straightforward DocumentAskForAccess import

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Straightforward DocumentFavorite import

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Straightforward import of threads, comments and reactions.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
@magopian
magopian requested a review from lunika October 7, 2026 09:15
Straightforward import of user reconciliations.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Straightfoward import of mentions

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
Use a dict lookup for import helpers instead of a huge match statement.

Signed-off-by: Mathieu Agopian <mathieu@agopian.info>
@magopian
magopian force-pushed the feat/export-import-instance-data-for-migration branch from 73bff30 to 693bcb4 Compare October 7, 2026 09:31
@magopian
magopian removed the request for review from lunika October 7, 2026 10:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant