Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 8 additions & 49 deletions docs/planning/open-code-intel-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -444,54 +444,13 @@ that would read as "this code has no duplicates".

## F7. Index export / import

### Delivery

New `src/lemoncrow/infra/code_intel/portable.py`, CLI subcommands under the
existing `lc code` group (`gateway/cli/commands/code.py`, live source):

```
lc code export [--out .lemoncrow/index.tar.zst] [--tier best|fast]
lc code import [--from .lemoncrow/index.tar.zst]
```

- `VACUUM INTO` each of the five DBs into a temp dir (compacts, drops WAL)
- tar + zstd; two tiers — `best` (zstd 9, drop derived indexes) on explicit
export, `fast` (zstd 3) for incremental refresh
- manifest: engine `index_version`, `indexer_semantics_version`, LemonCrow
version, repo HEAD sha, row counts, sidecar `schema_version`
- **import refuses on version mismatch** rather than producing a subtly wrong
graph. The engine owns those numbers; we cannot migrate its data.
- import bootstraps into an empty workspace, then the engine's normal
incremental pass fills the local diff

`zstandard` is a new dependency — put it behind an extra, not the base install.

### Tests

`tests/infra/code_intel/test_portable.py` — round-trip fidelity (row counts and
a sampled query match), version-mismatch refusal, corrupt-archive handling.

**Effort:** 5-8 days. **Risk:** low-medium. Deliberately does **not** commit the
artifact to git by default.

**Shipped** as `8235fdaf` — `infra/code_intel/portable.py`, plus `lc code
export` / `lc code import`. Two deviations:

> **`zstandard` is an accelerator, not a requirement.** It sits behind a new
> `portable` extra as planned, but export falls back to stdlib lzma when it is
> absent rather than failing. The manifest names the codec and import reads it,
> so the feature works on a base install and gets smaller archives with the
> extra.
>
> **The `best` tier does not drop derived indexes.** The engine's DDL is closed;
> an index dropped on export is one open code cannot recreate, so the import
> would hand back a database the engine expects to be complete. The two tiers
> differ by compression level only.

One addition the plan did not call for: the archive is treated as untrusted
input. Members must be regular files whose names are on a fixed allow-list, so a
traversal path or a symlink is refused outright. A tar file is a format someone
else can write, and "a teammate sent it" is not provenance.
**Removed.** Shipped as `8235fdaf` — `infra/code_intel/portable.py` plus two
`lc code` subcommands that packed a workspace's index into an archive and
restored it — and removed again under PRD-739 FR15 (PLN-2027 PR 7): the fork
added the surface, upstream never shipped it, and nothing consumed it. The
code, including the archive's untrusted-input handling, is at `8235fdaf`; the
delivery notes and the two deviations recorded against them are at
`git show 8d77d599:docs/planning/open-code-intel-plan.md`.

---

Expand Down Expand Up @@ -811,7 +770,7 @@ superseded generation just multiplies the blast radius of the defect.
| F11 | `8ea1eb15` | `infra/code_intel/freshness.py` |
| F2 | `f866112d` | `code_changes` tool |
| F5 | `e832fc70` | `code_query` tool |
| F7 | `8235fdaf` | `lc code export` / `import` |
| F7 | `8235fdaf` | index export / import (removed, PRD-739 FR15) |
| F12 | `67d56649` | routing + completeness contract (below) |

Deviations are recorded against each item above. The whole `tests/gateway/`
Expand Down
3 changes: 0 additions & 3 deletions docs/planning/phase-b-review-handoff.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,6 @@ exclusive) and re-query — expect an `IndexRebuilding` error, not `[]`.
any workspace without the optional tree-sitter `parsers` extra, and treating it
as a rebuild made every code tool fail forever on a perfectly valid workspace.

5. **`zstandard` is not installed**, so `lc code export` uses an lzma fallback.
Functional, larger archives. Not a bug.

---

## Also worth knowing
Expand Down
4 changes: 0 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,6 @@ semantic = [
parsers = [
"tree-sitter-languages>=1.10; python_version < '3.13'",
]
# Portable code-index archives (`lc code export` / `import`). zstd beats the
# stdlib codecs on both ratio and speed; without it export/import still work and
# fall back to lzma, so this is an accelerator, not a requirement.
portable = ["zstandard>=0.22"]
rename = ["rope>=0.23"]
ortools = ["ortools>=9.10"]
litellm = ["litellm>=1.83.14"]
Expand Down
72 changes: 4 additions & 68 deletions src/lemoncrow/gateway/cli/commands/code.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@

from lemoncrow.gateway.cli.commands._shared import _emit, require_pro
from lemoncrow.gateway.integrations.openmemory_lifecycle import project_root as _project_root
from lemoncrow.infra.code_intel.portable import TIERS as _PORTABLE_TIERS


@click.group("zoekt")
Expand Down Expand Up @@ -823,76 +822,14 @@ def _entry_age_days(entry: Path, now: float) -> float:
return max(0.0, (now - newest) / 86_400.0)


def _portable_repo_root(repo_root: str | None) -> Path:
def _resolve_repo_root(repo_root: str | None) -> Path:
if repo_root is not None:
return Path(repo_root).expanduser().resolve()
from lemoncrow.core.foundation.paths import resolve_workspace_root

return Path(resolve_workspace_root()).resolve()


@code_group.command("export")
@click.option(
"--out", default=None, type=click.Path(path_type=Path), help="Archive path (default: .lemoncrow/index.tar.*)."
)
@click.option(
"--tier",
type=click.Choice(sorted(_PORTABLE_TIERS)),
default="best",
show_default=True,
help="best: smaller archive, slower. fast: quicker, for incremental refresh.",
)
@click.option("--repo-root", default=None, help="Repository root (default: the resolved workspace root).")
@click.option("--json", "as_json", is_flag=True)
def code_export_cmd(out: Path | None, tier: str, repo_root: str | None, as_json: bool) -> None:
"""Pack this workspace's code index into a portable archive.

Compacts each database, records the engine's index and semantics versions
in a manifest, and compresses the bundle. The archive is deliberately not
committed anywhere by default -- it is a build artifact, not source.
"""
from lemoncrow.infra.code_intel.portable import PortableIndexError, export_index

try:
result = export_index(repo_root=_portable_repo_root(repo_root), out=out, tier=tier)
except PortableIndexError as exc:
raise click.ClickException(str(exc)) from exc
if as_json:
_emit(result.to_dict(), as_json=True)
return
click.echo(f"Wrote {result.path} ({result.size_bytes:,} bytes, {result.codec}, tier={result.tier})")
click.echo(f" databases: {', '.join(result.databases)}")
click.echo(f" engine index_version: {result.manifest['engine_index_version']}")


@code_group.command("import")
@click.option("--from", "source", required=True, type=click.Path(path_type=Path), help="Archive to import.")
@click.option("--repo-root", default=None, help="Repository root (default: the resolved workspace root).")
@click.option("--force", is_flag=True, help="Replace an index this workspace already holds.")
@click.option("--json", "as_json", is_flag=True)
def code_import_cmd(source: Path, repo_root: str | None, force: bool, as_json: bool) -> None:
"""Restore a code index from an archive built by `lc code export`.

Refuses on an indexer-semantics mismatch. The engine owns that number and
open code cannot migrate its data, so importing across it would produce a
graph whose edges mean something else -- confident and wrong. Re-index
instead. --force overrides only the already-populated check.
"""
from lemoncrow.infra.code_intel.portable import PortableIndexError, import_index

try:
result = import_index(archive=source, repo_root=_portable_repo_root(repo_root), force=force)
except PortableIndexError as exc:
raise click.ClickException(str(exc)) from exc
if as_json:
_emit(result.to_dict(), as_json=True)
return
click.echo(f"Restored {len(result.restored)} database(s) into {result.workspace}")
click.echo(f" from: {result.archive}")
click.echo(f" verified against: {result.verified_against}")
click.echo("Run `lc code index` to fill in the local diff.")


@code_group.command("clones")
@click.option(
"--threshold",
Expand Down Expand Up @@ -938,14 +875,13 @@ def code_clones_cmd(

try:
report = build_clones(
repo_root=_portable_repo_root(repo_root),
repo_root=_resolve_repo_root(repo_root),
threshold=DEFAULT_THRESHOLD if threshold is None else threshold,
min_tokens=MIN_TOKENS if min_tokens is None else min_tokens,
)
except (CodeIntelUnavailable, IndexRebuilding) as exc:
# Matches `lc code export` / `import` above: an unindexed workspace, or
# one mid-reindex, is a thing the user can act on, so it gets a one-line
# message rather than a traceback.
# An unindexed workspace, or one mid-reindex, is a thing the user can act
# on, so it gets a one-line message rather than a traceback.
raise click.ClickException(str(exc)) from exc
if as_json:
_emit(report.as_dict(), as_json=True)
Expand Down
Loading
Loading