Skip to content

Add yarn twenty pull to write an installed application back to source - #25410

Open
Weiko wants to merge 7 commits into
mainfrom
c--pull-3-cli
Open

Add yarn twenty pull to write an installed application back to source#25410
Weiko wants to merge 7 commits into
mainfrom
c--pull-3-cli

Conversation

@Weiko

@Weiko Weiko commented Sep 4, 2026

Copy link
Copy Markdown
Member

What

yarn twenty pull reads an installed application out of a workspace and writes it back as local define files. It is the inverse of apply, and it consumes the exportApplication query added in #25311.

The command is experimental and says so: a warning on every run, plus a callout in the docs and a marker in both command tables.

yarn twenty pull -u <universalIdentifier>

How it works

  1. Export. exportApplication is added to ApplicationApi and ApiService, following the existing query on that class. Typed refusals (not exportable, standard application, not found) are relayed as the server worded them.
  2. Scan. The project is scanned through the manifest extractor to map each universalIdentifier to the file that currently defines it. The application is matched by kind rather than identifier, so pulling into a scaffold rewrites the config file it already ships instead of leaving two defineApplication files behind.
  3. Plan. Each entity is compared against the base recorded by the previous pull: unchanged entities leave their file untouched, changed ones rewrite the file that defines them, entities gone from the workspace have their file deleted, and entities the workspace never knew are reported and left alone. New entities land beside existing files of their kind, falling back to the dev:add layout.
  4. Write. Files are staged under .twenty/ and moved into place at the end, so a failure halfway leaves the tree as it was. No folder is ever deleted.
  5. Report. Files grouped as written, regenerated and deleted, then the coverage the server returned, grouped by reason with counts per kind. -v lists identifiers.

The base lives at .twenty/pull-base.json, gitignored and per checkout. A checkout without one writes everything and deletes nothing.

Scope

The export fills the application header, objects with their fields, standalone fields on objects the app does not own, and authored indexes. Everything else comes back classified and is reported, not silently dropped. Package files and stored source are not restored here, because the export returns no files yet.

Consequence worth stating plainly: a pruning apply of a pulled tree destroys the kinds that are not covered yet. That is why the docs and the report both point at --no-delete, and why plan lists exactly what would go.

Three things the implementation had to correct

  • Enum values are written as symbols, not raw strings. Writability, open-record-in, field type, number data type and date display format are real TypeScript enums; a tree written with raw strings builds but fails plan's typecheck. The writer emits symbols with a generated import block. Running tsc over a pulled tree is what caught the number data type case.
  • Objects are written verbatim. Stripping the build-injected name field loses writability, isUIEditable, isUnique and isLabelSyncedWithName, which the export carries and the injection does not restore.
  • The base is written by pull only, never by push. A push would record the built manifest, which carries injected fields, minted permission identifiers and checksums that the exported manifest does not. Comparing those two shapes needs a normaliser that does not exist, and getting it subtly wrong means silently skipping a file that should have been rewritten. The cost is that a pull after a push reformats your own files.

Objects whose label identifier is engine-derived

Junction objects created in the UI point their label identifier at the object's id field, which the engine derives and the export therefore drops. That pointer is portable: it is the deterministic identifier of that field, seeded on the application and object identifiers, so any workspace derives the same value.

Two things in the SDK blocked it, and both are fixed here:

  • defineObject rejected a label identifier naming no field in the object's own fields. It is now a warning saying the pointer must name a field the engine derives.
  • The build injected a name field into any object lacking one, which would have added a real column to a junction table. That injection is now skipped for exactly this configuration.

This cannot affect any existing app, because that configuration was previously a hard error, so no app can be in it.

Verification

Against the seeded Custom application on a running server:

  • pull writes 19 files covering all five objects, including both junction objects;
  • the written tree typechecks against the real SDK types;
  • plan --no-delete reports No changes. Twenty metadata matches your manifest.;
  • a pruning plan destroys exactly the kinds the coverage report named, and nothing else.

Offline, building the written tree reproduces the exported objects, fields, indexes and application header exactly once key and collection order are normalised, and a second pull produces zero writes with every entity unchanged.

Tests

  • write-define-file — enum symbols and the import block, members whose name differs from their value, wrapped imports, quoting and escaping.
  • build-pull-entities — application property stripping, file naming for fields on standard objects and for indexes, the engine-derived label identifier, skipping an index whose object is absent.
  • plan-pull-writes — first pull, base-driven second pull, regeneration into an existing file, deletion, local-only entities, placement beside existing files, collision qualification.
  • format-pull-report — file grouping, per-kind counts, verbose listing and its cap, unreadable files.
  • get-default-fields-in-object-fields — the injection rule, including the new condition.
  • pull-round-trip integration — writes a fixture export into a temp project, builds it, and asserts the built manifest matches the export, junction object included.
  • Updated define-object for the warning that replaced the error.

Docs

A pull row in the CLI command table, a row in the "which command, when" table, and a section on the sync and recovery page covering what is written, what is not, the report, the base file and the second pull.

Review in cubic

@mintlify

mintlify Bot commented Sep 4, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
twenty 🟢 Ready View Preview Sep 4, 2026, 4:32 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds the experimental yarn twenty pull workflow, including application export, source scanning, define-file generation, pull-base tracking, reporting, SDK validation changes, tests, and documentation.

  • Exports an installed application and maps supported metadata back to local define files.
  • Tracks the previous export in .twenty/pull-base.json to plan rewrites and deletions.
  • Generates TypeScript enum references and supports engine-derived object label fields.
  • Adds detailed coverage reporting and user guidance for unsupported metadata.
  • The filesystem application path still has two data-loss risks: partial commits on failure and overwriting unrecognized path collisions.

Confidence Score: 3/5

The PR is not safe to merge until pull prevents partial commits and preserves pre-existing files at unrecognized generated-path collisions.

A commit-phase filesystem error can leave only part of the planned pull applied, and generated writes can silently remove local files that scanning did not associate with the exported entity.

Files Needing Attention: packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts, packages/twenty-sdk/src/cli/utilities/pull/plan-pull-writes.ts

Important Files Changed

Filename Overview
packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts Stages generated files but commits replacements and deletions sequentially without rollback, allowing partial source-tree updates.
packages/twenty-sdk/src/cli/utilities/pull/plan-pull-writes.ts Plans base-aware writes and deletions, but generated destinations can collide with files not associated with the same identifier.
packages/twenty-sdk/src/cli/operations/pull.ts Orchestrates authentication, export, scanning, planning, filesystem writes, and pull-base persistence.
packages/twenty-sdk/src/cli/utilities/pull/build-pull-entities.ts Converts exported application, object, field, and index manifests into writable define entities.
packages/twenty-sdk/src/cli/utilities/pull/write-define-file.ts Generates define files with symbolic enum references and deterministic imports.
packages/twenty-sdk/src/cli/utilities/build/manifest/utils/get-default-fields-in-object-fields.ts Avoids injecting a name field when an object label points to an engine-derived field.
packages/twenty-sdk/src/sdk/define/objects/define-object.ts Changes absent label-field references from validation errors to explicit synchronization warnings.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI[yarn twenty pull] --> Export[Export application]
  Export --> Scan[Scan local define files]
  Scan --> Base[Read pull base]
  Base --> Plan[Plan writes and deletions]
  Plan --> Stage[Stage generated files]
  Stage --> Commit[Replace and delete destinations]
  Commit --> Save[Record new pull base]
  Save --> Report[Print writes and coverage]
Loading

Reviews (1): Last reviewed commit: "Mark pull as experimental and report def..." | Re-trigger Greptile

Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/plan-pull-writes.ts
@twenty-ci-bot-public

twenty-ci-bot-public Bot commented Sep 4, 2026

Copy link
Copy Markdown

🤖 PR Review

Check Result
🔍 Build safety ⏭️ skipped — external-only
🛡️ Security ✅ passed
🧭 Triage ⏭️ skipped — external-only
📐 Quality ✅ passed — 1 nit(s)
🚦 Auto-approve 👀 needs review — Too large to review reliably (+3216 lines across 30 file(s))

🛡️ Security Review

No high-severity vulnerabilities detected.

Summary

  • 🟡 1 lower-severity issue(s)

🚦 Auto-approve

🙋 Manual review recommended for the following reason(s):

  • Too large to review reliably (+3216 lines across 30 file(s))

  • Complexity is medium

  • 🧠 Complexity: medium

  • 📏 Size: +3216 / -18 lines across 30 file(s)


View details

Automated pre-review — human approval still required.

@twenty-ci-bot-public

twenty-ci-bot-public Bot commented Sep 4, 2026

Copy link
Copy Markdown

🟡 Quality review · 1 finding

Safe to merge — one non-blocking naming nit; all prior data-loss blockers and marker nits are fixed

High-level — SDK-only pull command with sound architecture (no FE/BE shape divergence, calls an existing server query); the +3110-line size was flagged by the bot and acknowledged by the author, so not re-raised.
Low-level — All prior marker/cubic/greptile findings are resolved at head; one minor naming imprecision on a serialized-config map is the only remaining line-level issue.

💬 1 inline comment on the diff.


Reviewed against the pr-review standard — high-level then low-level. Advisory; human review still required. Run details.

Comment thread packages/twenty-sdk/src/cli/utilities/pull/scan-project-define-files.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/build-pull-entities.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/build-pull-entities.ts Outdated
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

App docs drift check

Change Docs page Status Suggested fix
New pull command with -u/--universal-identifier packages/twenty-docs/developers/extend/apps/operations/cli.mdx already updated in this PR
New pull command section, experimental warning, base file description packages/twenty-docs/developers/extend/apps/operations/sync-and-recovery.mdx already updated in this PR
-v, --verbose flag on pull expands coverage report to list individual entity identifiers (up to 20 per metadata type) packages/twenty-docs/developers/extend/apps/operations/sync-and-recovery.mdx needs update Add a note to the pull section: "Add --verbose to list individual identifiers for each coverage group."

| `dev` | Watch source files and live-sync changes | [Quick Start](/developers/extend/apps/getting-started/quick-start) |
| `plan` | Preview metadata changes without applying them | [Syncing & recovery](/developers/extend/apps/operations/sync-and-recovery#previewing-changes-plan) |
| `apply` | Apply metadata changes after showing the plan | [Syncing & recovery](/developers/extend/apps/operations/sync-and-recovery) |
| `pull` | **Experimental.** Write the installed application back to local source files | [Syncing & recovery](/developers/extend/apps/operations/sync-and-recovery#pulling-an-installed-app-back-to-source) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would deserve a pull-plan mode (ok for doing it in another PR but i would start by this personnally)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and I would like it too. Two things make me want it as a follow-up rather than here.

It needs the writer to produce content without touching the tree, which is already how it works: the planner returns the full content of every write, and applying it is a separate step. So a plan mode is mostly a flag that stops before applyPullWrites and prints the same report, plus a diff of each file against what is on disk. That diff is the part worth designing properly rather than bolting on, since the useful output is "this file would change in these ways", not just "this file would be rewritten".

The other reason is size: this PR is already flagged as too large to review reliably, so I would rather not add a surface to it.

Happy to take it as the next one if you want it before the remaining entity families.

Comment thread packages/twenty-docs/developers/extend/apps/operations/sync-and-recovery.mdx Outdated
Comment thread packages/twenty-docs/developers/extend/apps/operations/sync-and-recovery.mdx Outdated
@martmull

martmull commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

how does conflicts are handled? Like if you pull an object from the remove with a universalId you already use in your local app

Comment thread packages/twenty-sdk/src/cli/utilities/pull/build-pull-entities.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/write-define-file.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/twenty-docs/developers/extend/apps/operations/sync-and-recovery.mdx Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

Comment thread packages/twenty-sdk/src/cli/utilities/pull/plan-pull-writes.ts
Comment thread packages/twenty-sdk/src/cli/utilities/pull/print-typescript-value.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts
Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/pull-base-file.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/scan-project-define-files.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/__tests__/plan-pull-writes.spec.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/scan-project-define-files.ts Outdated
@Weiko

Weiko commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

how does conflicts are handled? Like if you pull an object from the remote with a universalId you already use in your local app

Short answer: the identifier is the identity, so pull treats that as the same entity and rewrites the file that defines it. The server wins, and the file appears under regenerated in the report.

That is deliberate, but it is worth being precise about the three cases, because only one of them is a real conflict:

  1. Same identifier, and the base agrees. Normal case. If the entity did not change on the server since your last pull, the file is left completely untouched, even if you reformatted or refactored it.
  2. Same identifier, changed on the server. The file is rewritten from the server's version and listed as regenerated.
  3. Same identifier, and you also changed it locally without pushing. This is the real conflict. Today the server's version wins and you lose the local edit, with git as the safety net. The base file makes this detectable (the entity is in the base, the server's copy differs, and so does yours), but the report does not yet call it out separately from case 2. Refusing, --force and per-file resolution were all deferred past v1.

There is no case where two genuinely different entities collide on one identifier, because an identifier is unique within an application and the export only returns the identifiers that application owns. What can collide is the file name, and the review caught a bug there which is fixed in 24c38f8: a generated name that would land on an unrelated file is now qualified instead of overwriting it.

If you want case 3 surfaced in the report before this ships, that is a small addition and I am happy to add it here.

@Weiko

Weiko commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Review addressed in 24c38f8 and 9090708. Two data-loss bugs were real and are fixed, with tests that fail without the fix:

  • Partial commits. The apply step now backs up every destination it will replace or delete and restores them all if any later step fails. It also refuses a plan up front when a destination is a directory, and deletes files non-recursively, so no folder is ever removed. Staging and backup directories are per invocation, so two pulls in one checkout cannot corrupt each other.
  • Generated paths overwriting local files. The planner reserves every scanned path before choosing a name. Only the file that already defines the same identifier is written in place; anything else gets the identifier prefix. This also removes the duplicate-path case.

Also fixed: carriage returns and the two Unicode line separators are escaped (a carriage return really did produce an unterminated literal, verified by evaluating the output); a __proto__ key is written as a computed key; the base file is validated before use; an unreadable source file no longer aborts the scan; and all seven quality nits.

One suggested fix I did not take as written. Quoting __proto__ does not help: { '__proto__': v } still sets the prototype, and only { ['__proto__']: v } creates an own property. That is what the code does now, with a test asserting the prototype is untouched.

On splitting the SDK foundation into an earlier PR: fair in principle, but the defineObject and default-fields change is two conditions and is meaningless on its own, since nothing but pull produces the configuration it unblocks. Splitting it would ship a behaviour change with no caller and no way to test it end to end. I would rather keep it here where the round-trip test proves it.

Comment thread packages/twenty-sdk/src/cli/utilities/pull/plan-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/build-pull-entities.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/format-pull-report.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 14 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts
Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/pull-base-file.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/plan-pull-writes.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/apply-pull-writes.ts
Comment thread packages/twenty-sdk/src/cli/utilities/pull/__tests__/apply-pull-writes.spec.ts Outdated
Comment thread packages/twenty-sdk/src/cli/utilities/pull/build-pull-entities.ts Outdated
: undefined;
};

const buildConfigByUniversalIdentifier = (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit · Low-level · naming — name for what it returns

buildConfigByUniversalIdentifier returns a map of serialized JSON strings, not configs

The returned Map<string, string> holds JSON.stringify(entity.config) values for equality comparison, so a caller reading .get(id) gets a string rather than a config. Rename to reflect the serialized value, e.g. buildSerializedConfigByUniversalIdentifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants