add perceptual near-duplicate clustering

Author: jiapengLi11

.github/workflows/ci.yml

@@ -40,6 +40,11 @@ jobs:
           python examples/create_consensus_fixture.py --output-dir /tmp/ylr-consensus-fixture
           yolo-label-recovery consensus /tmp/ylr-consensus-fixture/primary_candidates.csv /tmp/ylr-consensus-fixture/verifier_candidates.csv --output-dir /tmp/ylr-consensus --agreement-iou 0.50 --verifier-min-confidence 0.50 --redact-paths
           python -c "import json; d=json.load(open('/tmp/ylr-consensus/consensus.json')); assert d['totals']['primary_auto']==72 and d['totals']['agreed_auto']==48 and d['totals']['downgraded_to_review']==24"
+      - name: Run perceptual near-duplicate demo
+        run: |
+          python examples/create_near_duplicate_fixture.py --output /tmp/ylr-near-duplicates
+          yolo-label-recovery cluster /tmp/ylr-near-duplicates --output-dir /tmp/ylr-near-duplicate-output --redact-paths
+          python -c "import json; d=json.load(open('/tmp/ylr-near-duplicate-output/near_duplicate_summary.json')); assert d['totals']['images_discovered']==11 and d['totals']['fingerprint_failures']==1 and d['totals']['clusters']==3 and d['totals']['clustered_images']==7 and d['totals']['cross_split_clusters']==2"
       - name: Build wheel
         run: python -m pip wheel --no-deps --wheel-dir dist .
       - name: Reject private artifacts

.gitignore

@@ -46,3 +46,4 @@ Desktop.ini
 !examples/calibration/output/*.csv
 !examples/consensus/*.csv
 !examples/consensus/output/*.csv
+!examples/near_duplicates/output/*.csv

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.7.0
+
+- Added the model-free, read-only `cluster` command for perceptual near-duplicate review grouping.
+- Added 64-bit dHash/aHash fingerprints, BK-tree radius search and deterministic connected components.
+- Added aspect-ratio and low-texture luminance safeguards to reduce false-positive groups.
+- Added cross-split near-duplicate leakage evidence, corrupt-image isolation and representative review exports.
+- Added a reproducible public fixture, self-contained HTML report, static screenshot, bilingual documentation and CI coverage.
+
 ## 0.6.0
 
 - Added the model-free `consensus` command for independent cross-Teacher AUTO verification.

README.md

@@ -47,6 +47,12 @@ The public fixture contains `2,400` reviewed candidates across all six classes.
 
 The public fixture contains `96` primary candidates across six classes. Of `72` primary AUTO candidates, `48` receive one-to-one spatial support from an independent verifier and remain AUTO; `24` are safely downgraded to REVIEW. The stage is model-free and adds no GPU memory pressure.
 
+### Perceptual near-duplicate review groups
+
+![Pre-generated perceptual near-duplicate report](docs/assets/near-duplicates-preview.png)
+
+The public fixture groups resize, JPEG recompression and brightness variants without merging black and white low-texture frames. It finds `3` groups containing `7` images, reduces first-pass review to `3` representatives and flags `2` groups crossing dataset splits.
+
 ## Why this project exists
 
 Multi-class datasets often contain combined scenes such as `person + helmet + smoking` or `person + slipper`. If the original annotation process focused on one target at a time, valid objects from other classes can be missing. Training a new multi-class model on incomplete labels can make the model learn the wrong supervision signal.
@@ -86,6 +92,7 @@ flowchart TD
 - A model-free audit catches malformed labels, corrupt images and exact train/val/test leakage before GPU work starts.
 - Audited candidate decisions can calibrate class-specific AUTO policies using a Wilson precision lower bound and REVIEW policies using positive recall.
 - Independent Teacher candidate streams can gate AUTO decisions with one-to-one spatial agreement without loading two models together.
+- Perceptual hashes, a BK-tree and conservative visual guards group repeated review work and expose near-duplicate split leakage.
 - Every scan records a local manifest with parameters, image inventory, package versions, CUDA and GPU metadata.
 
 ## One-minute public demo
@@ -142,6 +149,16 @@ yolo-label-recovery consensus primary_candidates.csv verifier_candidates.csv `
   --redact-paths
 ```
 
+Group perceptual near-duplicates without loading a model or changing source data:
+
+```powershell
+yolo-label-recovery cluster D:\data\mining-safety `
+  --output-dir D:\data\near-duplicate-audit `
+  --workers 4 `
+  --max-distance 6 `
+  --redact-paths
+```
+
 For GPU-assisted label recovery, first install the CUDA-compatible PyTorch build required by the target GPU, then install the inference extra:
 
 ```powershell
@@ -258,6 +275,8 @@ See:
 - [Threshold calibration (Simplified Chinese)](docs/CALIBRATION.zh-CN.md)
 - [Cross-Teacher consensus](docs/CONSENSUS.md)
 - [Cross-Teacher consensus (Simplified Chinese)](docs/CONSENSUS.zh-CN.md)
+- [Perceptual near-duplicate grouping](docs/NEAR_DUPLICATES.md)
+- [Perceptual near-duplicate grouping (Simplified Chinese)](docs/NEAR_DUPLICATES.zh-CN.md)
 - [Interview presentation](docs/INTERVIEW_STORY.md)
 - [Portfolio and interview guide](docs/PORTFOLIO_GUIDE.md)
 - [Portfolio and interview guide (Simplified Chinese)](docs/PORTFOLIO_GUIDE.zh-CN.md)

README.zh-CN.md

@@ -47,6 +47,12 @@
 
 公开样本包含六类共 `96` 个主 Teacher 候选。在 `72` 个主 AUTO 中，`48` 个得到独立验证 Teacher 的一对一空间支持并保留为 AUTO，另外 `24` 个安全降级到 REVIEW。该阶段不依赖模型推理，不会增加显存压力。
 
+### 感知近重复审核分组
+
+![预生成感知近重复报告](docs/assets/near-duplicates-preview.png)
+
+公开样例能够聚合缩放、JPEG 重压缩和亮度变化图片，同时不会错误合并纯黑与纯白低纹理帧。结果包含 `3` 组、共 `7` 张图片，只需优先审核 `3` 张代表图，并发现 `2` 组跨数据划分近重复。
+
 ## 为什么需要这个项目
 
 多类别数据集经常包含 `person + helmet + smoking`、`person + slipper` 等联合场景。如果原始标注工作每次只关注一个目标，图中其他类别的有效目标就可能漏标。使用不完整标签训练多类别模型时，这些目标会被当作背景，从而向模型传递错误监督信号。
@@ -86,6 +92,7 @@ flowchart TD
 - GPU 推理前先进行无模型审计，检查错误标签、损坏图片和 train/val/test 精确重复。
 - 使用人工审核候选校准分类别策略：AUTO 采用 Wilson 精度置信下限，REVIEW 采用正样本召回约束。
 - 使用独立 Teacher 候选流进行一对一空间一致性门控，无需同时加载两个模型。
+- 使用感知哈希、BK-tree 和保守视觉约束压缩重复审核工作，并发现跨划分近重复泄漏。
 - 每次扫描生成 manifest，记录参数、图片清单、依赖版本、CUDA 和 GPU 信息。
 
 ## 一分钟公开演示
@@ -142,6 +149,16 @@ yolo-label-recovery consensus primary_candidates.csv verifier_candidates.csv `
   --redact-paths
 ```
 
+无需模型且不修改源数据，即可聚类感知近重复图片：
+
+```powershell
+yolo-label-recovery cluster D:\data\mining-safety `
+  --output-dir D:\data\near-duplicate-audit `
+  --workers 4 `
+  --max-distance 6 `
+  --redact-paths
+```
+
 如需使用 GPU 自动补标，请先安装与目标 GPU/CUDA 兼容的 PyTorch，再安装推理依赖：
 
 ```powershell
@@ -258,6 +275,8 @@ out-root/
 - [阈值校准（英文）](docs/CALIBRATION.md)
 - [跨 Teacher 一致性门控（中文）](docs/CONSENSUS.zh-CN.md)
 - [跨 Teacher 一致性门控（英文）](docs/CONSENSUS.md)
+- [感知近重复聚类（中文）](docs/NEAR_DUPLICATES.zh-CN.md)
+- [感知近重复聚类（英文）](docs/NEAR_DUPLICATES.md)
 - [面试项目讲解](docs/INTERVIEW_STORY.md)
 - [作品集与面试指南（中文）](docs/PORTFOLIO_GUIDE.zh-CN.md)
 - [作品集与面试指南（英文）](docs/PORTFOLIO_GUIDE.md)

docs/INTERVIEW_STORY.md

@@ -19,6 +19,7 @@ Long scans are checkpointed after every committed batch. Resume validates a run
 - Hardlinks reduce disk duplication when building a trainable dataset.
 - Atomic checkpoint/resume avoids restarting completed teacher/split work.
 - The HTML report turns audit artifacts into a reviewable delivery and GitHub demo.
+- Perceptual near-duplicate grouping uses compact hashes and BK-tree radius search to reduce repeated review without loading all pixels or comparing every pair.
 - `doctor` and `manifest.json` make environment differences visible instead of leaving CUDA and dependency drift implicit.
 
 ## Honest limitation

docs/NEAR_DUPLICATES.md

@@ -0,0 +1,63 @@
+# Perceptual near-duplicate grouping
+
+[English](NEAR_DUPLICATES.md) | [简体中文](NEAR_DUPLICATES.zh-CN.md)
+
+The `cluster` command reduces repeated human review and detects visually duplicated images crossing YOLO dataset splits. It is model-free, CPU-only and read-only: it never deletes, moves or rewrites an image or label.
+
+## Usage
+
+```powershell
+yolo-label-recovery cluster D:\data\mining-safety `
+  --output-dir D:\data\near-duplicate-audit `
+  --splits train val test `
+  --workers 4 `
+  --max-distance 6 `
+  --redact-paths
+```
+
+Use `--fail-on-cross-split` in CI when perceptual leakage must fail a data release.
+
+## Algorithm
+
+1. Decode one image per worker and apply EXIF orientation.
+2. Produce a 64-bit difference hash (dHash), a 64-bit average hash (aHash), dimensions and luminance statistics.
+3. Query dHash neighbors with a BK-tree instead of comparing every image pair.
+4. Reject candidates that violate the aHash radius or aspect-ratio guard.
+5. For two low-texture images, require similar mean luminance so black and white frames do not collide.
+6. Convert accepted neighbor edges into deterministic connected components.
+7. Select the highest-resolution image in each component as the first review representative.
+
+For `N` images, fingerprint storage is `O(N)`. BK-tree search is usually far below brute-force `O(N²)`, although adversarial hash distributions can degrade. Pixel buffers are bounded by the worker count; the entire dataset is never decoded into RAM at once.
+
+## Outputs
+
+| File | Purpose |
+|---|---|
+| `near_duplicate_members.csv` | One row per clustered image with hashes and representative distance |
+| `review_representatives.csv` | One highest-resolution representative per review group |
+| `fingerprint_failures.csv` | Corrupt or unreadable image evidence |
+| `near_duplicate_summary.json` | Policy, totals, group membership and leakage counts |
+| `near_duplicate_report.html` | Self-contained visual report for review and delivery |
+
+## Guardrails and limitations
+
+- A cluster means visual similarity, not permission to delete data automatically.
+- Connected components allow transitive chains: A may match B and B may match C even when A is farther from C. The report exposes maximum dHash distance to the selected representative so loose groups can be inspected.
+- dHash is useful for resize, JPEG recompression and modest brightness changes. It is not designed for major crops, rotations or semantic similarity.
+- Cross-split clusters are leakage candidates. Confirm provenance before moving images because visually similar frames may still represent intentional temporal evaluation.
+- Thresholds are policy. Validate them on a reviewed sample from the target domain before bulk cleanup.
+
+## Reproducible public fixture
+
+```powershell
+python examples\create_near_duplicate_fixture.py --output .near-duplicate-fixture
+yolo-label-recovery cluster .near-duplicate-fixture `
+  --output-dir .near-duplicate-output `
+  --redact-paths
+```
+
+Expected result: `11` discovered files, `1` intentional fingerprint failure, `3` groups, `7` grouped images and `2` cross-split groups. Black and white low-texture frames remain separate.
+
+## Interview framing
+
+This feature is evidence of data-centric ML engineering rather than another training wrapper. The key design choices are bounded decoding, sub-quadratic candidate search, conservative false-positive guards, deterministic review groups, explicit leakage evidence and immutable source data.

docs/NEAR_DUPLICATES.zh-CN.md

@@ -0,0 +1,63 @@
+# 感知近重复图片聚类
+
+[English](NEAR_DUPLICATES.md) | [简体中文](NEAR_DUPLICATES.zh-CN.md)
+
+`cluster` 命令用于减少重复人工审核，并发现跨 YOLO 数据划分出现的视觉重复图片。它不依赖模型和 GPU，且严格只读：不会删除、移动或改写任何图片与标签。
+
+## 使用方法
+
+```powershell
+yolo-label-recovery cluster D:\data\mining-safety `
+  --output-dir D:\data\near-duplicate-audit `
+  --splits train val test `
+  --workers 4 `
+  --max-distance 6 `
+  --redact-paths
+```
+
+如果数据发布流程要求近重复图片绝不能跨划分，可在 CI 中增加 `--fail-on-cross-split`。
+
+## 算法流程
+
+1. 每个工作线程逐张解码图片，并应用 EXIF 方向修正。
+2. 生成 64 位差值哈希 dHash、64 位均值哈希 aHash、尺寸及亮度统计。
+3. 使用 BK-tree 查询 dHash 半径近邻，避免所有图片两两暴力比较。
+4. 使用 aHash 距离和宽高比变化过滤错误候选。
+5. 两张图片都属于低纹理图时，额外要求平均亮度接近，避免纯黑帧和纯白帧被错误合并。
+6. 将通过检查的近邻关系转换为确定性的连通分量。
+7. 每组选择分辨率最高的图片，作为人工首轮审核代表图。
+
+对 `N` 张图片，指纹存储为 `O(N)`。BK-tree 在常见哈希分布下远少于暴力 `O(N²)` 比较，但极端哈希分布仍可能退化。像素内存受工作线程数限制，不会将全量数据集图片同时解码到内存。
+
+## 输出文件
+
+| 文件 | 用途 |
+|---|---|
+| `near_duplicate_members.csv` | 每张聚类成员图片、哈希及其到代表图的距离 |
+| `review_representatives.csv` | 每组一张最高分辨率首轮审核代表图 |
+| `fingerprint_failures.csv` | 损坏或无法读取的图片证据 |
+| `near_duplicate_summary.json` | 策略、统计、分组成员和跨划分数量 |
+| `near_duplicate_report.html` | 可直接交付的自包含可视化报告 |
+
+## 安全边界与局限
+
+- 分到同一组只代表视觉相似，不代表可以自动删除。
+- 连通分量允许传递链：A 接近 B、B 接近 C 时，即使 A 与 C 距离较远也可能同组。报告会显示成员到代表图的最大 dHash 距离，便于检查松散组。
+- dHash 适合发现缩放、JPEG 重压缩和轻微亮度变化，不擅长大幅裁剪、旋转或纯语义相似。
+- 跨划分组是数据泄漏候选，移动图片前仍需确认来源，因为连续视频帧也可能被有意用于时序评测。
+- 阈值属于数据策略。批量清理前应使用目标领域的人工样本验证。
+
+## 可复现公开样例
+
+```powershell
+python examples\create_near_duplicate_fixture.py --output .near-duplicate-fixture
+yolo-label-recovery cluster .near-duplicate-fixture `
+  --output-dir .near-duplicate-output `
+  --redact-paths
+```
+
+预期结果为：发现 `11` 个文件，其中 `1` 个故意损坏；形成 `3` 组、共 `7` 张聚类图片，其中 `2` 组跨数据划分。低纹理纯黑帧与纯白帧保持分离。
+
+## 面试表达
+
+该功能展示的是数据中心型 ML 工程能力，而不是简单封装训练命令。重点包括：有限内存解码、非暴力近邻搜索、保守防误合并、确定性审核分组、显式泄漏证据以及源数据不可变。

docs/PORTFOLIO_GUIDE.md

@@ -22,6 +22,7 @@ The difficult part was operational reliability on a long `K x N` scan. I added t
 | Human-in-the-loop ML | AUTO/REVIEW/IGNORE routing | Precision-first thresholds and audit evidence |
 | Evaluation and policy | `calibrate` command | Turning reviewed outcomes into precision/recall-constrained class policies |
 | Ensemble policy | `consensus` command | Independent evidence, one-to-one matching and coverage/risk tradeoffs |
+| Scalable similarity search | `cluster` command | Perceptual hashes, BK-tree radius search and conservative collision guards |
 | Software quality | package, CLI, tests and CI | Public fixture, privacy checks and release build |
 | Communication | HTML reports and architecture docs | Turning model work into reviewable project evidence |
 
@@ -33,7 +34,8 @@ The difficult part was operational reliability on a long `K x N` scan. I added t
 4. Open `examples/demo_output/report.html` to show the recovery quality report without exposing project data.
 5. Open `examples/calibration/output/calibration.html` to explain why each class receives a different threshold.
 6. Open `examples/consensus/output/consensus.html` to show unsupported AUTO candidates being downgraded.
-7. Run `yolo-label-recovery doctor` to show environment diagnostics.
+7. Open `examples/near_duplicates/output/near_duplicate_report.html` to show review compression and split leakage.
+8. Run `yolo-label-recovery doctor` to show environment diagnostics.
 
 This demonstration works without a GPU or private model weights. A full teacher scan remains an optional second demonstration when suitable public weights and data are available.
 
@@ -67,6 +69,10 @@ Separate scans preserve the one-model GPU memory bound and allow different detec
 
 No. It makes the prediction stronger evidence. The project retains REVIEW routing, visual samples and an audit trail because confidence alone does not prove correctness under domain shift.
 
+**Why not compare every image pair for near duplicates?**
+
+Brute force grows as `O(N²)`. The `cluster` command stores compact fingerprints and uses a BK-tree for Hamming-radius candidate search, then applies aHash, aspect-ratio and low-texture luminance safeguards. It still treats every group as review evidence rather than an automatic deletion decision.
+
 ## Claims to avoid
 
 - Do not claim the teachers eliminate all missing labels.

docs/PORTFOLIO_GUIDE.zh-CN.md

@@ -22,6 +22,7 @@
 | 人机协同 ML | AUTO/REVIEW/IGNORE | 分类别阈值与精度优先的审核策略 |
 | 评估与策略 | `calibrate` 命令 | 将人工审核结果转换为精度/召回约束的分类别策略 |
 | 集成决策 | `consensus` 命令 | 独立证据、一对一匹配与覆盖率/风险取舍 |
+| 可扩展相似度搜索 | `cluster` 命令 | 感知哈希、BK-tree 半径查询与保守碰撞防护 |
 | 软件质量 | Python 包、CLI、测试和 CI | 公开测试夹具、隐私扫描与 Release 构建 |
 | 技术沟通 | HTML 报告与架构文档 | 将模型工作转化为可审核、可展示的证据 |
 
@@ -33,7 +34,8 @@
 4. 打开 `examples/demo_output/report.html`，在不泄露项目数据的情况下展示补标质量报告。
 5. 打开 `examples/calibration/output/calibration.html`，解释为什么每个类别需要不同阈值。
 6. 打开 `examples/consensus/output/consensus.html`，展示不受支持的 AUTO 如何降级。
-7. 运行 `yolo-label-recovery doctor`，展示环境诊断能力。
+7. 打开 `examples/near_duplicates/output/near_duplicate_report.html`，展示审核压缩与跨划分泄漏。
+8. 运行 `yolo-label-recovery doctor`，展示环境诊断能力。
 
 该演示不需要 GPU 或私有权重。具备合适的公开模型与数据后，可将完整 Teacher 扫描作为第二阶段演示。
 
@@ -67,6 +69,10 @@ Teacher 的置信度校准和目标难度不同。吸烟、拖鞋等小目标不
 
 不是。高置信度只代表更强的证据。域偏移条件下，置信度不能直接证明标签正确，因此项目保留 REVIEW 分流、可视化抽样和完整审计链路。
 
+**为什么近重复检测不直接两两比较全部图片？**
+
+暴力比较会按 `O(N²)` 增长。`cluster` 命令仅保存紧凑指纹，使用 BK-tree 做汉明距离半径搜索，再用 aHash、宽高比和低纹理亮度约束过滤。即便如此，每个分组仍只是人工审核证据，而不是自动删除决定。
+
 ## 不应夸大的内容
 
 - 不要声称 Teacher 能消除全部漏标。