Update Version Lab for integrated PyMuPDF builds

Author: hashiromer

.github/docker/pymupdf-source-stack/ENVIRONMENTS.md

@@ -17,7 +17,8 @@ rebuilt; they must not be used by benchmark jobs.
 
 Container publication is restricted to the repository's default branch and
 serialized so two publishers cannot race to update a version tag. The publisher
-checks the system environment and RapidOCR import, but does not promote its
+checks the system environment and RapidOCR/ONNX imports using the dependencies
+in `tools/version_lab/uv.lock`, but does not promote its
 digest into the benchmark workflow automatically. Before changing the pinned
 benchmark digest, run the all-latest 15-case benchmark against the candidate,
 verify exact MuPDF provenance and Layout/OCR compatibility, compare aggregate
@@ -61,13 +62,19 @@ The latest image uses:
 
 The image contains system and build dependencies, not the selected PyMuPDF
 source stack or the ParseBench Python environment. The benchmark workflow still
-needs to install its locked dependencies and build the selected MuPDF,
-PyMuPDF, PyMuPDF Layout, and PyMuPDF4LLM revisions inside the container. The
+needs to install its locked dependencies and build one PyMuPDF wheel containing
+the selected Layout and PyMuPDF4LLM sources. MuPDF is built from the native
+version declared by the selected PyMuPDF revision. The
 image includes `unzip` because current MuPDF source uses it while generating
 the Extract DOCX template. `rapidocr-onnxruntime==1.2.3` and
 `opencv-python==4.13.0.92` were installed only inside the publisher's
 validation environment; they were not baked into the image.
 
+The current publisher mounts the repository read-only and installs the Version
+Lab lockfile into a temporary environment, skipping the two local projects.
+This validates the same RapidOCR and ONNX dependencies used by benchmark runs
+without duplicating their version pins in the workflow.
+
 The image writes its principal versions to
 `/usr/local/share/parsebench/environment.txt` for runtime verification.
 
@@ -101,9 +108,11 @@ The production workflow uses the following integration:
 3. Run workflow shell steps with Bash inside the container.
 4. Mark the workspace and the four source checkouts as Git safe directories,
    because Actions mounts them with host ownership.
-5. Use `uv sync --locked --extra runners`, excluding the three source-built
-   PyMuPDF packages as the current workflow does.
+5. Use `uv sync --project tools/version_lab --locked`, excluding `pymupdf`,
+   `pymupdf-layout`, and `pymupdf4llm` as the current workflow does.
 6. Verify `/usr/local/share/parsebench/environment.txt`, Python, Tesseract,
    SWIG, and the RapidOCR import before compiling the selected source stack.
-7. Keep the existing behavioral Layout/OCR compatibility test as the final
-   environment and source-stack gate.
+7. Build PyMuPDF with `PYMUPDF_SETUP_PATH_LAYOUT` and `PYMUPDF_SETUP_PATH_4LLM`
+   pointing to the selected source checkouts, and the resolved MuPDF source.
+8. Verify all four source revisions and run the behavioral Layout/OCR
+   compatibility test before benchmarking.

.github/scripts/pymupdf_source_stack/README.md

@@ -10,9 +10,9 @@ the adapters in this directory translate GitHub environment variables and
 output files to that shared core. GitHub-only publishing and failure-summary
 behavior remains here.
 
-MuPDF is an independently selected source component. The workflow resolves its
-latest PyPI release tag or current branch head to a full SHA before checkout, then gives
-PyMuPDF's build system a reproducible
+MuPDF follows the version declared in the selected PyMuPDF commit's `setup.py`;
+developers do not select it separately. The workflow resolves that native
+release tag to a full SHA before checkout, then gives PyMuPDF's build system a reproducible
 `git:--sha <sha> https://github.com/ArtifexSoftware/mupdf.git` selector through
 `PYMUPDF_SETUP_MUPDF_BUILD`. The compatibility gate independently compares
 that selector with the source embedded in the installed
@@ -36,12 +36,16 @@ reports, scores, or upstream modules.
 Each adapter is a small command with one responsibility. `resolve_dataset.py`
 resolves the `current` Hugging Face branch or validates a user-supplied full
 commit SHA. `resolve_refs.py` uses the shared source resolver for all four
-components. Source inputs default to `latest`, which resolves the current
-PyPI release to its Git tag. Existing branch names, all default branches, and
-the newest branch head per repository are also supported. Historical release
+components. PyMuPDF, Layout, and PyMuPDF4LLM inputs default to `latest`, which
+resolves the current PyMuPDF PyPI release to its Git tag in each repository.
+Existing branch names, all default branches, and the newest branch head per
+repository are also supported; MuPDF always follows the selected PyMuPDF commit. Historical release
 tags and arbitrary source commits are rejected. Layout always comes from
 `ArtifexSoftware/pymupdf_layout`. `resolve_sources.py` verifies every source
-checkout against its resolved SHA before the source builds begin. Finally,
+checkout against its resolved SHA and exports absolute source roots. PyMuPDF
+builds one wheel using `PYMUPDF_SETUP_PATH_LAYOUT` and
+`PYMUPDF_SETUP_PATH_4LLM`; the embedded commit IDs are checked before benchmarking.
+There are no standalone Layout/4LLM installs or older packaging fallbacks. Finally,
 `benchmark.py download` reuses only a complete cached snapshot whose internal
 revision marker matches that SHA. A missing, stale, or incomplete snapshot is
 removed and downloaded again. Inputs supplied by the workflow are passed

.github/scripts/pymupdf_source_stack/common.py

@@ -16,6 +16,7 @@
     COMPONENT_SPECS,
     DATASET_BRANCHES,
     DATASET_REPOSITORY,
+    SELECTABLE_COMPONENTS,
 )
 from parsebench_version_lab.provenance import mupdf_build_spec  # noqa: E402, F401 - compatibility re-export
 from parsebench_version_lab.util import required_env as env  # noqa: E402, F401

.github/scripts/pymupdf_source_stack/configure.py

@@ -6,7 +6,7 @@
 import re
 from pathlib import Path
 
-from common import COMPONENT_SPECS, append_summary, env, write_github_outputs, write_json
+from common import COMPONENT_SPECS, SELECTABLE_COMPONENTS, append_summary, env, write_github_outputs, write_json
 
 RUN_SCOPES = {
     "Quick test (15 cases)": ("test", "data/test"),
@@ -20,7 +20,7 @@
     "Text content": "text_content",
     "Text formatting": "text_formatting",
 }
-LATEST_REFS = {name: component.default_branch for name, component in COMPONENT_SPECS.items()}
+LATEST_REFS = {name: component.default_branch for name, component in SELECTABLE_COMPONENTS.items()}
 LATEST_ANY_BRANCH_REFS = dict.fromkeys(LATEST_REFS, "latest-any-branch")
 
 
@@ -53,7 +53,7 @@ def main() -> int:
         raise SystemExit("Select either all latest default-branch commits or latest commits from any branch, not both")
     requested_refs = {
         "dataset": env("DATASET_REF"),
-        "mupdf": env("MUPDF_REF"),
+        "mupdf": "auto",
         "pymupdf": env("PYMUPDF_REF"),
         "pymupdf_layout": env("PYMUPDF_LAYOUT_REF"),
         "pymupdf4llm": env("PYMUPDF4LLM_REF"),
@@ -70,7 +70,6 @@ def main() -> int:
 
     stack_label = "_".join(
         (
-            f"mupdf-{safe_ref(refs['mupdf'])}",
             f"pymupdf-{safe_ref(refs['pymupdf'])}",
             f"layout-{safe_ref(refs['pymupdf_layout'])}",
             f"4llm-{safe_ref(refs['pymupdf4llm'])}",

.github/scripts/pymupdf_source_stack/resolve_refs.py

@@ -17,10 +17,16 @@ def main() -> int:
     request = json.loads(request_path.read_text(encoding="utf-8"))
     manager = SourceManager(CommandRunner())
     outputs = {}
+    resolved = {}
     for name, component in COMPONENT_SPECS.items():
         selection = request[name]["ref"]
         try:
-            source = manager.resolve_ref(component, selection)
+            source = (
+                manager.resolve_mupdf(resolved["pymupdf"])
+                if name == "mupdf"
+                else manager.resolve_ref(component, selection)
+            )
+            resolved[name] = source
         except (OSError, RuntimeError, ValueError) as error:
             message = f"Cannot resolve {component.label} selection {selection!r}: {error}"
             write_json(

.github/scripts/pymupdf_source_stack/resolve_sources.py

@@ -1,14 +1,12 @@
 #!/usr/bin/env python3
-"""Verify pinned source checkouts and locate their installable packages."""
+"""Verify pinned source checkouts for the integrated PyMuPDF build."""
 
 from __future__ import annotations
 
 import json
 from pathlib import Path
 
 from common import COMPONENT_SPECS, env, git_sha, mupdf_build_spec, write_github_outputs, write_json
-from parsebench_version_lab.local import package_dir
-from parsebench_version_lab.sources import ResolvedSource
 
 
 def main() -> int:
@@ -18,11 +16,9 @@ def main() -> int:
     for name, component in COMPONENT_SPECS.items():
         root = Path(".source") / component.checkout_dir
         sha = git_sha(root)
-        source = ResolvedSource(name, component.label, component.repository, request[name]["ref"], sha, root)
         try:
             if sha != request[name]["resolved_sha"]:
                 raise RuntimeError(f"{component.label} checkout {sha} differs from the resolved commit.")
-            package = root if name == "mupdf" else package_dir(source)
         except RuntimeError as error:
             write_json(
                 output_dir / "_failure.json",
@@ -35,7 +31,7 @@ def main() -> int:
                 },
             )
             raise SystemExit(str(error)) from error
-        outputs[f"{name}_dir"] = str(package)
+        outputs[f"{name}_dir"] = str(root.resolve())
         outputs[f"{name}_sha"] = sha
     outputs["mupdf_build"] = mupdf_build_spec(outputs["mupdf_sha"])
     write_github_outputs(outputs)

.github/workflows/pymupdf-source-stack-environment.yml

@@ -60,7 +60,7 @@ jobs:
         env:
           IMAGE_TAG: ${{ steps.candidate.outputs.tag }}
         run: |
-          docker run --rm "$IMAGE_TAG" bash -euo pipefail -c '
+          docker run --rm --volume "$GITHUB_WORKSPACE:/workspace:ro" "$IMAGE_TAG" bash -euo pipefail -c '
             set -x
             test "$(python --version)" = "Python 3.12.13"
             test "$(uv --version | cut -d" " -f2)" = "0.11.29"
@@ -76,11 +76,11 @@ jobs:
             cc --version
             swig -version
             cat /usr/local/share/parsebench/environment.txt
-            uv venv /tmp/rapidocr-smoke
-            uv pip install --python /tmp/rapidocr-smoke/bin/python \
-              "rapidocr-onnxruntime==1.2.3" "opencv-python==4.13.0.92"
+            UV_PROJECT_ENVIRONMENT=/tmp/rapidocr-smoke \
+              uv sync --project /workspace/tools/version_lab --locked \
+                --no-install-project --no-install-package parse-bench
             /tmp/rapidocr-smoke/bin/python -c \
-              "from rapidocr_onnxruntime import RapidOCR; print(RapidOCR)"
+              "from importlib.metadata import version; from rapidocr import RapidOCR; import onnxruntime; print(version(\"rapidocr\"), onnxruntime.__version__, RapidOCR)"
           '
 
       - name: Log in to GitHub Container Registry

.github/workflows/pymupdf-source-stack-parsebench.yml

@@ -7,13 +7,11 @@ run-name: >-
       inputs.pipeline,
       inputs.latest_any_branch && 'Latest branch' ||
         inputs.all_latest && 'Latest default' ||
-        inputs.mupdf_ref == inputs.pymupdf_ref &&
-        inputs.mupdf_ref == inputs.pymupdf_layout_ref &&
-        inputs.mupdf_ref == inputs.pymupdf4llm_ref &&
-        inputs.mupdf_ref ||
+        inputs.pymupdf_ref == inputs.pymupdf_layout_ref &&
+        inputs.pymupdf_ref == inputs.pymupdf4llm_ref &&
+        inputs.pymupdf_ref ||
         format(
-          'M:{0},P:{1},L:{2},4:{3}',
-          inputs.mupdf_ref,
+          'P:{0},L:{1},4:{2}',
           inputs.pymupdf_ref,
           inputs.pymupdf_layout_ref,
           inputs.pymupdf4llm_ref
@@ -30,12 +28,12 @@ on:
   workflow_dispatch:
     inputs:
       all_latest:
-        description: "Use latest default-branch commits for MuPDF, PyMuPDF, Layout and PyMuPDF4LLM. Their ref inputs are ignored."
+        description: "Use default branches for PyMuPDF, Layout and PyMuPDF4LLM. MuPDF follows PyMuPDF. Ref inputs are ignored."
         required: true
         default: false
         type: boolean
       latest_any_branch:
-        description: "Use the newest branch-head commit in each of the four source repositories. Cannot be combined with latest default-branch commits."
+        description: "Use newest branch heads for PyMuPDF, Layout and PyMuPDF4LLM. MuPDF follows PyMuPDF. Cannot combine with latest default branches."
         required: true
         default: false
         type: boolean
@@ -56,23 +54,18 @@ on:
         required: true
         default: current
         type: string
-      mupdf_ref:
-        description: "MuPDF: latest PyPI release or a current branch name. Ignored by either automatic branch mode."
-        required: true
-        default: "latest"
-        type: string
       pymupdf_ref:
         description: "PyMuPDF: latest PyPI release or a current branch name. Ignored by either automatic branch mode."
         required: true
         default: "latest"
         type: string
       pymupdf_layout_ref:
-        description: "PyMuPDF Layout: latest PyPI release or a current branch name. Ignored by either automatic branch mode."
+        description: "PyMuPDF Layout: current PyMuPDF release tag or a current branch name. Ignored by either automatic branch mode."
         required: true
         default: "latest"
         type: string
       pymupdf4llm_ref:
-        description: "PyMuPDF4LLM: latest PyPI release or a current branch name. Ignored by either automatic branch mode."
+        description: "PyMuPDF4LLM: current PyMuPDF release tag or a current branch name. Ignored by either automatic branch mode."
         required: true
         default: "latest"
         type: string
@@ -137,7 +130,6 @@ jobs:
           DATASET_REF: ${{ inputs.dataset_version }}
           GROUP_SELECTION: ${{ inputs.group }}
           LATEST_ANY_BRANCH: ${{ inputs.latest_any_branch }}
-          MUPDF_REF: ${{ inputs.mupdf_ref }}
           PYMUPDF4LLM_REF: ${{ inputs.pymupdf4llm_ref }}
           PYMUPDF_LAYOUT_REF: ${{ inputs.pymupdf_layout_ref }}
           PYMUPDF_REF: ${{ inputs.pymupdf_ref }}
@@ -249,64 +241,30 @@ jobs:
         run: python3 "$WORKFLOW_SCRIPTS/write_run_summary.py"
 
       - name: Install ParseBench dependencies without selected PyMuPDF packages
-        env:
-          MODERN_RAPIDOCR_PIPELINE: pymupdf4llm_html_tables_rapidocr_v3
         run: |
           export UV_PROJECT_ENVIRONMENT="$GITHUB_WORKSPACE/.venv"
-          sync_args=(--project tools/version_lab --locked)
-          if [[ "$PIPELINE" == "$MODERN_RAPIDOCR_PIPELINE" ]]; then
-            sync_args+=(--extra rapidocr-modern)
-          fi
-          uv sync "${sync_args[@]}" \
+          uv sync --project tools/version_lab --locked \
             --no-install-package pymupdf \
             --no-install-package pymupdf-layout \
             --no-install-package pymupdf4llm
-          uv pip install --python .venv "pipcl==12" "swig==4.3.1"
+          uv pip install --python .venv "swig==4.3.1"
           .venv/bin/swig -version
 
       - name: Verify RapidOCR native runtime
-        if: ${{ inputs.pipeline != 'pymupdf4llm_html_tables_rapidocr_v3' }}
-        run: .venv/bin/python -c "from rapidocr_onnxruntime import RapidOCR; print(RapidOCR)"
-
-      - name: Verify modern RapidOCR native runtime
-        if: ${{ inputs.pipeline == 'pymupdf4llm_html_tables_rapidocr_v3' }}
         run: >-
           .venv/bin/python -c
           "from importlib.metadata import version; from rapidocr import RapidOCR; import onnxruntime;
           print(f'rapidocr={version(\"rapidocr\")} onnxruntime={onnxruntime.__version__} engine={RapidOCR}')"
 
-      - name: Build and install selected PyMuPDF
+      - name: Build and install integrated PyMuPDF
         env:
           PACKAGE_DIR: ${{ steps.sources.outputs.pymupdf_dir }}
           PYMUPDF_SETUP_MUPDF_BUILD: ${{ steps.sources.outputs.mupdf_build }}
           PYMUPDF_SETUP_SWIG: ${{ github.workspace }}/.venv/bin/swig
+          PYMUPDF_SETUP_PATH_LAYOUT: ${{ steps.sources.outputs.pymupdf_layout_dir }}
+          PYMUPDF_SETUP_PATH_4LLM: ${{ steps.sources.outputs.pymupdf4llm_dir }}
         run: uv pip install --python .venv --reinstall --no-deps "$PACKAGE_DIR"
 
-      - name: Build and install selected PyMuPDF Layout
-        env:
-          PACKAGE_DIR: ${{ steps.sources.outputs.pymupdf_layout_dir }}
-          PYMUPDF_LAYOUT_SETUP_BUILD_PYMUPDF: "1"
-          PYMUPDF_LAYOUT_SETUP_SWIG: ${{ github.workspace }}/.venv/bin/swig
-        run: uv pip install --python .venv --reinstall --no-build-isolation --no-deps "$PACKAGE_DIR"
-
-      - name: Build and install selected PyMuPDF4LLM
-        env:
-          PACKAGE_DIR: ${{ steps.sources.outputs.pymupdf4llm_dir }}
-        run: uv pip install --python .venv --reinstall --no-deps "$PACKAGE_DIR"
-
-      - name: Verify modern RapidOCR and native HTML capabilities
-        if: ${{ inputs.pipeline == 'pymupdf4llm_html_tables_rapidocr_v3' }}
-        run: >-
-          .venv/bin/python -c
-          "import inspect, pymupdf4llm; from pymupdf4llm.helpers.pymupdf_rag import to_markdown as rag_to_markdown;
-          from pymupdf4llm.ocr.detect_rapidocr import detect_rapidocr_backend;
-          backend = detect_rapidocr_backend(); assert backend == 'rapidocr',
-          f'Expected modern rapidocr backend, received {backend!r}';
-          assert 'table_output' in inspect.signature(rag_to_markdown).parameters and
-          'table_output' in inspect.getsource(pymupdf4llm.to_markdown),
-          'Selected PyMuPDF4LLM revision lacks native table_output support';
-          print('modern RapidOCR and native HTML table output are available')"
-
       - name: Check source stack compatibility
         env:
           MUPDF_REF: ${{ steps.refs.outputs.mupdf_ref }}