The long-form version of the README: why each stage works the way it does, what was measured, and how the QA thresholds were calibrated. Kept because the reasoning is the expensive part.
Flat image in, separated layers out: background.png plus one RGBA cutout per
element, where every layer is complete — hide any single layer and recomposite
and there is no hole, no ghost rim, and no style discontinuity where the fill
sits.
Spec: docs/layer-decomposition-pipeline-spec-v2.md.
If a ball overlaps a plant, cutting out the ball leaves the plant incomplete — its hidden part was never in the source image. So removal runs back-to-front: peel the nearest element, reconstruct what it covered, and only then extract the next element from the filled image. Every layer comes out complete with respect to everything behind it, and what remains at the end is the background plate.
Model-driven stages are hard to unit test, so the engine is graded against
synthetic scenes with known ground truth (layerengine.synth). A synthetic
scene knows each element's complete appearance — including the parts a nearer
element covers — while the flattened image the pipeline receives shows only what
is visible. That gap is exactly what the peel loop must reconstruct, which turns
the most stochastic stage in the pipeline into something with a numeric answer.
Two properties keep the suite honest:
- The generator composites via a plain lerp over an opaque plate, which is a different code path from the general Porter-Duff compositor. The two verify each other rather than one asserting itself.
- Design claims are tested against their alternative. Stage C asserts that per-contact depth comparison beats a global median; the test builds the case where they disagree and pins both answers, so removing the contact-band logic fails the test instead of passing it vacuously.
Each stage reaches its model through a protocol, so backends are swappable without touching pipeline logic.
| Stage | Backend | Status on M4 Pro |
|---|---|---|
| A — planning | OpenAI / Anthropic / Gemini | Verified against gpt-5.5. ~12 s, one call. Provider-switchable; the pipeline never learns which one wrote the plan. |
| B — segmentation | SAM 3, or Grounding DINO + SAM 2 | Verified on MPS. ~1.9 s/prompt at 1920×1080 fp32, ~5.6 GB peak, 840M params. No CUDA kernels, no PYTORCH_ENABLE_MPS_FALLBACK needed. |
| C — depth | Depth Anything 3 (DA3MONO-LARGE) |
Verified on MPS. 334M params, ~1.1 s at 504 px. Needs scripts/setup_da3.sh — the PyPI package's declared deps are partly wrong for this platform. |
| D — matting | ViTMatte (transformers) |
Verified on MPS. Beats the binary mask it replaces by 5.2× inside the unknown band, measured against synthetic ground truth. |
| E — fill | LaMa + SDXL inpainting, routed by hole size | Verified on MPS. 0.4 GB + 6.9 GB. See below — neither wins alone. |
SAM 3's weights are gated (facebook/sam3, manual approval by Meta). Set
HF_TOKEN and point LAYERENGINE_SAM3_REPO at whichever checkpoint you have
access to — or run --segmenter grounded-sam2, which needs no approval at all.
SAM 2 cannot substitute for SAM 3 on its own, and that is checkable rather than a matter of opinion:
Sam2Processor.__call__: images, input_points, input_labels, input_boxes
Sam3Processor.__call__: images, text, input_boxes, ...
Sam2Config has vision_config, prompt_encoder_config and
mask_decoder_config — no text config. SAM 2 is a promptable segmenter, not
an open-vocabulary one, and Stage B is built on handing it a noun phrase. The
plan's hint boxes cannot stand in either: one box deliberately covers all
fifteen flowers together, and SAM 2 given that box returns one mask of the
cluster rather than fifteen flowers.
So the open vocabulary comes from a detector instead. Grounding DINO turns the phrase into per-instance boxes, SAM 2 turns each box into a mask; both are ungated and Apache-2.0, 3.7 GB together against SAM 3's 6.9 GB.
Same wallpaper, same plan, same filler:
| SAM 3 | Grounded SAM 2 | |
|---|---|---|
| armored knight | 14.9% coverage | 15.2% |
| steel sword | 1.0% | 1.0% |
| flowering bushes | 15.4%, one instance | 9.6% + 47.7%, over-matched |
| crimson cloak (not in the image) | correctly reported missing | invented at 0.30 |
| review reasons | 2 | 4 |
On things that are actually there the two agree closely, and one phrase still returns many instances — "pink flowers" came back as six separate boxes, which is the property Stage B depends on.
The difference is in reporting absence. Grounding DINO localises rather than classifies: it answers with whatever is most phrase-like instead of deciding whether the phrase is present at all. "yellow taxi" and "office desk" correctly returned nothing, but "a grand piano" grounded onto the knight at 0.54 — above any usable threshold — and on a synthetic image of two red discs it grounded "knight in armor" at 0.44, higher than "wooden chair" at 0.40. That last case is a strict xfail in the test suite rather than an omission, so it will say something if it ever changes.
Stage F caught every one of those failures on the real run, including the
invented cloak, which scored 0.88 on the occlusion-seam gate. That is the
system working — but sam3 stays the default, because a segmenter that reports
absence honestly is worth more than one that never says no.
The DA3 checkpoint is pinned by exact name. Sibling checkpoints in the same
family (DA3-LARGE, DA3-GIANT, the NESTED variants) are CC BY-NC, so choosing
by size rather than by name silently lands a commercial pipeline on a
non-commercial licence. DA3MONO-LARGE and DA3-SMALL are Apache-2.0.
Backend tests are marked models and excluded from the default run:
.venv/bin/python -m pytest # deterministic core, offline, ~0.2s
.venv/bin/python -m pytest -m models # adapter contract tests, downloads weightsOne real 1920×1080 wallpaper, no hand-written plan, all five models:
plan 5 elements (generated by openai:gpt-5.5)
armored knight, steel sword, crimson cloak, pink flowers, flowering bushes
ran in 486.6s
layers 10 + background
QA recomposite_psnr 35.0 dB
fill_texture_ratio armored knight 1.08 (was 0.40 with classical)
occlusion_seam flowering bushes 0.16 (no fragmented layers)
shift_ghost steel sword 0.51
REVIEW - segmentation found no instance for 'crimson cloak'
- suspect segmentation: 'flowering bushes' box is 3.33x the hint box
Re-running with the same saved plan reproduces every fill score to the last decimal, which is the point of seeding the diffusion filler: a pipeline whose output moves run to run cannot be regression-tested, and every threshold above was calibrated against fixed output.
Both remaining diagnostics are correct. The planner named a cloak that SAM could not find, and the bush mask genuinely over-matched. Neither is a fill problem any more, which is the thing that changed.
Classical inpainting was the pipeline's quality ceiling and both earlier runs routed to review because of it. Two models replaced it, and the interesting part is that the obvious choice is wrong half the time.
Square holes in a real 1920×1080 wallpaper, scored with fill_texture_ratio —
the same gate the pipeline routes on, where 1.0 is what untouched content
scores:
| hole span | classical | LaMa | SDXL |
|---|---|---|---|
| 32 px | 0.57 | 1.31 | — |
| 64 px | 0.28 | 1.01 | — |
| 128 px | 0.30 | 0.57 | — |
| 512 px | 0.25 | 0.34 | — |
| 594 px (a real knight) | 0.28 | 0.37 | 0.96 |
LaMa is superb on small holes and collapses on large ones; SDXL is the reverse. The cause is structural rather than a difference in model quality. SDXL crops a square around the hole and resizes it to its native 1024, so every hole is about the same size as far as the model is concerned. LaMa runs at native resolution, so a large hole is simply larger than it can synthesise across — its receptive field is wide, but finite, and measured in pixels.
That last point is why RoutedFiller switches on an absolute pixel count
rather than a fraction of the frame: the same 200 px hole is equally hard at
1080p and at 4K. It measures the hole's equivalent side (√area) rather than its
bounding box, because a thin diagonal wisp spans the frame, has almost no area,
and boundary propagation handles it perfectly — routing on the box would spend
70 seconds of diffusion on it.
Only SDXL reads the Stage A style paragraph, which is the reason the spec asks for that paragraph at all. It is also the only filler that can hallucinate an object into a gap, which is what the upper texture gate exists for.
Cost: --fill auto runs about 8 minutes on a 10-layer 1080p image, against 48
seconds for LaMa alone. Almost all of it is two SDXL passes.
Every Stage F threshold is now derived from a measured null distribution rather than from feel: 240 random square regions of two real wallpapers at each size, none of them filled at all, so every score is what untouched content produces.
| side | median | <0.6 | >1.6 | >3.0 |
|---|---|---|---|---|
| 64 | 1.01 | 12% | 6% | 1% |
| 128 | 1.03 | 5% | 3% | 0% |
| 384 | 1.09 | 1% | 1% | 0% |
The median is 1.0 at every size, so the metric is centred correctly. What changes with size is the spread — and that caught a real bug. The old upper gate of 1.6 was fitted to smeared fills without ever checking what normal content scores, and 5–6% of untouched regions exceed it. At ten holes per image that flags most images, which is the opposite of the spec's 10–15% manual queue. It fired as a false positive on a real run.
So the gates are 0.5 and 3.0 — the latter unreached by any of 1440 untouched samples — and neither applies below a 128 px equivalent hole side, where the spread swallows both thresholds. Scores are still reported at every size, because suppressing them would hide a regression across a batch.
Spec §7 asks for a shift test and two of its three checks now exist, both aimed at damage that is invisible while the layers sit in their original positions.
occlusion_seam_fraction measures how much of a layer's outline was drawn by a
nearer layer rather than by the object itself. On the 17-layer classical run,
five small flowers scored exactly 1.000 — SAM had returned only the island
visible through a gap in the knight, so every boundary pixel they had was the
knight's, and what shipped was a fragment rather than a flower. Ordinary contact
between large elements sat at 0.06–0.08, so the gate goes at 0.5.
It counts plain abutment too, and that is deliberate: two layers resting against each other share an edge that is pixel-for-pixel identical to an occlusion seam, and from masks alone the cases cannot be told apart. The honest claim is narrower than "this layer is occluded" — it is "this much of the outline is defined by something nearer, so moving this layer will expose whether it was".
shift_ghost_score slides a layer aside and asks whether the pixels it uncovers
look like the background beside them or like the layer that was on top. A matte
that under-covers leaves the object's outer pixels in the plate and the fill is
then cut to a hole that is too tight; stacked up nothing shows, because the
layer covers its own residue exactly.
Spec §2 says peeling gives amodal completion for free. That is true of appearance and not of coverage. When a nearer layer is peeled and its hole filled, the layer beneath has its hidden colour correctly reconstructed — but its alpha still comes from SAM's modal mask, so the finished layer keeps a hole shaped like whatever was in front of it.
fill_interior_holes closes the tractable half: occluders sitting fully inside
a silhouette. On the dog image that recovered 76 px against 274,934 px
missing versus the convex hull, because the cloak that crosses the dog runs to
the frame edge and no morphology can tell that from real silhouette shape.
There is a much better answer, and it is measured. The peel loop already reconstructs what each layer covered, so by the time a layer is reached the occluder in front of it is gone from the working image. Re-running the segmenter on that filled image with the same phrase returns a fuller silhouette. On the dog:
segmented on the original image 271,198 px seam 0.062
segmented on the filled image 289,486 px seam 0.130
recovered +18,288 px
of which sits under nearer layers 28,636 px
240× what morphology recovered, on a run whose fills were still classical — the technique is limited by how convincingly the occluder's region was reconstructed, and that has since improved a lot.
It is not implemented, and the reason is that doing it properly is a change to the pipeline's shape rather than a patch. Stage D currently mattes every element up front and Stage E then peels; re-segmentation requires segmentation and matting to move inside the peel loop, which means SAM stays resident through Stage E (~13 GB alongside SDXL — it fits, but the staging exists for a reason). It also cascades: growing a layer's alpha grows the hole that should have been filled beneath it, so the layers behind it need re-peeling. That is a design decision about memory and runtime, not a bug fix, and it is the largest correctness gap remaining.
src/layerengine/
compositing.py Porter-Duff over / flatten, straight alpha throughout
masks.py Euclidean morphology, IoU, bbox
plan.py Stage A: the validated scene plan
segmentation.py Stage B: hint cross-check, cross-prompt de-dup, area sanity
ordering.py Stage C: contact-band occlusion graph -> peel order
depth.py depth polarity + the estimator protocol
matting.py Stage D: trimaps, band policy, decontamination
peel.py Stage E: back-to-front peel and reconstruction
qa.py Stage F: recomposite, fill texture, edge rim
packaging.py Stage G: layer PNGs + manifest
pipeline.py the driver: staging and failure routing
cli.py python -m layerengine.cli
synth.py Ground-truth scene generator
images.py uint8/float encoding coercion
backends/
sam3.py SAM 3 concept segmentation, local PyTorch/MPS
da3.py Depth Anything 3 monocular depth
vitmatte.py ViTMatte trimap matting
classical_fill.py OpenCV inpainting
tests/
python -m venv .venv
.venv/bin/pip install -e ".[dev]" # core + test tooling, no torch
.venv/bin/pip install -e ".[models]" # adds torch / transformers / pymatting
./scripts/setup_da3.sh # Depth Anything 3 (see the script's header)
.venv/bin/python -m pytestRun one image. Without --plan, Stage A writes one with a VLM and saves it to
<out>/plan.json, so the run stays reproducible:
cp .env.example .env # add a key for whichever provider you want
.venv/bin/python -m layerengine.cli IMAGE --out out/name
.venv/bin/python -m layerengine.cli IMAGE --out out/name --plan-provider anthropic
.venv/bin/python -m layerengine.cli IMAGE --out out/name --plan examples/the-last-rest.plan.json--fill selects the fill backend: auto (default, routes by hole size),
lama, sdxl, or classical.
Conventions: images are float32 (H, W, 4) in [0, 1] with straight
(non-premultiplied) alpha; depth is disparity-like, so larger means nearer.