Fix external-review issues: dedup OFF_MARKERS, paginate Bilibili, validate creators, version cap, retire bidao pilot, add tests + CI

Robustness / bugs:
- shared_markers.py: single source for OFF_MARKERS (FULL for features, COMPARE for
  compare_form -- the deliberate livestream / 联名 divergence preserved, behavior
  unchanged), ending the silent drift between the two copies.
- fetch_bilibili.py: paginate up to num (was first-page-only -> missed data past 50);
  add BilibiliError mirroring YouTubeError; factor out _to_record / _get_page.
- creators.py: validate_creators() lists ALL config problems (typo'd keys, missing
  alias/zone, duplicate alias, youtube missing channel) instead of a KeyError at
  fetch time; called at the start of fetch_multi.
- Cap bilibili-api-python <18.0.0 (requirements.txt + pyproject.toml).
- schema.py: VideoRecord TypedDict documenting the standard video record.

Cleanup -- retire the single-creator pilot cluster (bidao-hardcoded, mutually coupled
via classified.json, superseded by the generalized features/scan_signals pipeline,
and absent from every pipeline / README command list):
- delete classify_and_stats.py, fetch_videos.py, and the orphaned consumers comments.py
  + subtitle.py (they read the now-gone classified.json); drop all four from the
  PyInstaller spec hiddenimports; add shared_markers + schema.
- anonymize remaining bidao / real-BV references in analyze_video.py + diagnose.py
  docstrings; reword the README L2-text line to the surviving compare_meme.py.

Tests + CI (the project had zero tests):
- conftest.py + tests/ (23 tests): features.extract, off_tag, OFF_MARKERS divergence,
  validate_creators, spearman, build_report formatters, runtime paths, and
  fetch_bilibili parse/record/error + pagination (incl. the >50-video multi-page path).
- pyproject dev extra + pytest config; CI now runs pytest.

Deliberately skipped (with reasons): scripts/ -> subpackage restructure (would break
the released PyInstaller packaging), web CSRF/rate-limit (localhost-only app), scipy
for spearman (intentional zero-dep). Two report claims were verified already-fine and
left alone: scan_signals already degrades on missing data; requirements/pyproject are
already aligned.

Verified: 23 tests pass, compileall clean, viralens --help ok, frozen build succeeds
and --vl-exec compare_form / fetch_multi / build_report all work (new modules bundled,
BilibiliError surfaces correctly).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: HarryXin0919

.github/workflows/ci.yml

@@ -15,9 +15,11 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: "3.11"
-      - name: Install (editable, deps from pyproject)
-        run: pip install -e .
+      - name: Install (editable + dev deps from pyproject)
+        run: pip install -e ".[dev]"
       - name: Syntax check (compile all scripts)
         run: python -m compileall scripts
       - name: Smoke test (console entry point)
         run: viralens --help
+      - name: Unit tests
+        run: pytest

README.md

@@ -239,7 +239,7 @@ creators.py ──▶ fetch_multi.py ──▶ data/<alias>_videos.json        (
 - **Signal scanner** (`scan_signals`): turns each video into a universal feature vector, then auto-tests
   every dimension (title patterns, length buckets, daypart, cover metrics…) for high/low-play separation,
   ranks by effect size, and reports which levers are *universal* vs *creator-specific*.
-- **L2 — text** (`subtitle`, `comments`): subtitles + hot comments → `jieba` keyword analysis.
+- **L2 — text**: subtitles + hot comments → `jieba` keyword analysis (cross-creator comment-engagement test: `compare_meme.py`).
 - **Cross-creator / cross-zone gate**: a pattern earns a ✅ only if it survives the *same test* on
   multiple independent creators **and** more than one zone.
 

conftest.py

@@ -0,0 +1,5 @@
+"""Let pytest import the flat scripts/ modules without installing the package."""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent / "scripts"))

packaging/viralens.spec

@@ -50,9 +50,9 @@ hiddenimports += [
     "fetch_multi", "fetch_bilibili", "fetch_youtube",
     "compare_form", "creator_profile", "scan_signals", "charts", "export_data", "build_report",
     "diagnose", "analyze_video", "import_private",
-    "creators", "features", "benchmarks",
-    "classify_and_stats", "comments", "compare_meme", "fetch_covers",
-    "fetch_videos", "resolve_creators", "subtitle",
+    "creators", "features", "benchmarks", "shared_markers", "schema",
+    "compare_meme", "fetch_covers",
+    "resolve_creators",
 ]
 
 # —— 只读资源:网页界面 + 配置模板 —— 放进打包根目录,app.py 用 runtime.ASSET_DIR 找它们 ——

pyproject.toml

@@ -19,16 +19,17 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Information Analysis",
 ]
 dependencies = [
-    "bilibili-api-python>=17.0.0",
+    "bilibili-api-python>=17.0.0,<18.0.0",
     "jieba>=0.42.1",
     "matplotlib>=3.7.0",
     "Pillow>=9.0.0",
     "numpy>=1.23.0",
 ]
 
-# 仅在「打包成桌面 app」时需要:原生窗口后端。源码 CLI / 网页界面都不依赖它。
+# gui = 打包成桌面 app 时的原生窗口后端;dev = 跑测试。源码 CLI / 网页界面都不依赖它们。
 [project.optional-dependencies]
 gui = ["pywebview>=5.0"]
+dev = ["pytest>=7.0"]
 
 [project.urls]
 Homepage = "https://github.com/HarryXin0919/viralens"
@@ -45,3 +46,6 @@ py-modules = ["viralens"]
 
 [tool.setuptools.package-dir]
 "" = "scripts"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

requirements.txt

@@ -1,6 +1,6 @@
 # viralens — runtime dependencies
 # Install:  pip install -r requirements.txt
-bilibili-api-python>=17.0.0   # async Bilibili public-data access
+bilibili-api-python>=17.0.0,<18.0.0   # async Bilibili public-data access (cap: 18.x may break API)
 jieba>=0.42.1                 # Chinese word segmentation (comment keyword analysis)
 matplotlib>=3.7.0             # static README charts (PNG)
 Pillow>=9.0.0                 # cover-image analysis (optional step: fetch_covers.py)

scripts/analyze_video.py

@@ -16,9 +16,9 @@
 按需调用:用户在诊断页点「分析开头+配乐」才下这一条,不批量下 700 条。
 
 命令行单测:
-    python analyze_video.py bidao BV1mT421Y7mE
-    python analyze_video.py mrbeast 0BjlBnfHcHM
-    python analyze_video.py bidao BV1mT421Y7mE --force   # 忽略缓存重算
+    python analyze_video.py <alias> <vid>            # 如 demo_b1 BVxxxxxxxxxx
+    python analyze_video.py <alias> <youtube_vid>    # YouTube 视频 id
+    python analyze_video.py <alias> <vid> --force    # 忽略缓存重算
 """
 import base64
 import json