Skip to content

Fix external-review issues: dedup OFF_MARKERS, paginate Bilibili, val… - #3

Merged
HarryXin0919 merged 1 commit into
mainfrom
report-driven-fixes
Jun 3, 2026
Merged

Fix external-review issues: dedup OFF_MARKERS, paginate Bilibili, val…#3
HarryXin0919 merged 1 commit into
mainfrom
report-driven-fixes

Conversation

@HarryXin0919

Copy link
Copy Markdown
Owner

…idate creators, version cap, retire bidao pilot, add tests + CI

Robustness / bugs:

  • shared_markers.py: single source for OFF_MARKERS (FULL for features, COMPARE for compare_form -- the deliberate livestream / 联名 divergence preserved, behavior unchanged), ending the silent drift between the two copies.
  • fetch_bilibili.py: paginate up to num (was first-page-only -> missed data past 50); add BilibiliError mirroring YouTubeError; factor out _to_record / _get_page.
  • creators.py: validate_creators() lists ALL config problems (typo'd keys, missing alias/zone, duplicate alias, youtube missing channel) instead of a KeyError at fetch time; called at the start of fetch_multi.
  • Cap bilibili-api-python <18.0.0 (requirements.txt + pyproject.toml).
  • schema.py: VideoRecord TypedDict documenting the standard video record.

Cleanup -- retire the single-creator pilot cluster (bidao-hardcoded, mutually coupled via classified.json, superseded by the generalized features/scan_signals pipeline, and absent from every pipeline / README command list):

  • delete classify_and_stats.py, fetch_videos.py, and the orphaned consumers comments.py
    • subtitle.py (they read the now-gone classified.json); drop all four from the PyInstaller spec hiddenimports; add shared_markers + schema.
  • anonymize remaining bidao / real-BV references in analyze_video.py + diagnose.py docstrings; reword the README L2-text line to the surviving compare_meme.py.

Tests + CI (the project had zero tests):

  • conftest.py + tests/ (23 tests): features.extract, off_tag, OFF_MARKERS divergence, validate_creators, spearman, build_report formatters, runtime paths, and fetch_bilibili parse/record/error + pagination (incl. the >50-video multi-page path).
  • pyproject dev extra + pytest config; CI now runs pytest.

Deliberately skipped (with reasons): scripts/ -> subpackage restructure (would break the released PyInstaller packaging), web CSRF/rate-limit (localhost-only app), scipy for spearman (intentional zero-dep). Two report claims were verified already-fine and left alone: scan_signals already degrades on missing data; requirements/pyproject are already aligned.

Verified: 23 tests pass, compileall clean, viralens --help ok, frozen build succeeds and --vl-exec compare_form / fetch_multi / build_report all work (new modules bundled, BilibiliError surfaces correctly).

…idate creators, version cap, retire bidao pilot, add tests + CI

Robustness / bugs:
- shared_markers.py: single source for OFF_MARKERS (FULL for features, COMPARE for
  compare_form -- the deliberate livestream / 联名 divergence preserved, behavior
  unchanged), ending the silent drift between the two copies.
- fetch_bilibili.py: paginate up to num (was first-page-only -> missed data past 50);
  add BilibiliError mirroring YouTubeError; factor out _to_record / _get_page.
- creators.py: validate_creators() lists ALL config problems (typo'd keys, missing
  alias/zone, duplicate alias, youtube missing channel) instead of a KeyError at
  fetch time; called at the start of fetch_multi.
- Cap bilibili-api-python <18.0.0 (requirements.txt + pyproject.toml).
- schema.py: VideoRecord TypedDict documenting the standard video record.

Cleanup -- retire the single-creator pilot cluster (bidao-hardcoded, mutually coupled
via classified.json, superseded by the generalized features/scan_signals pipeline,
and absent from every pipeline / README command list):
- delete classify_and_stats.py, fetch_videos.py, and the orphaned consumers comments.py
  + subtitle.py (they read the now-gone classified.json); drop all four from the
  PyInstaller spec hiddenimports; add shared_markers + schema.
- anonymize remaining bidao / real-BV references in analyze_video.py + diagnose.py
  docstrings; reword the README L2-text line to the surviving compare_meme.py.

Tests + CI (the project had zero tests):
- conftest.py + tests/ (23 tests): features.extract, off_tag, OFF_MARKERS divergence,
  validate_creators, spearman, build_report formatters, runtime paths, and
  fetch_bilibili parse/record/error + pagination (incl. the >50-video multi-page path).
- pyproject dev extra + pytest config; CI now runs pytest.

Deliberately skipped (with reasons): scripts/ -> subpackage restructure (would break
the released PyInstaller packaging), web CSRF/rate-limit (localhost-only app), scipy
for spearman (intentional zero-dep). Two report claims were verified already-fine and
left alone: scan_signals already degrades on missing data; requirements/pyproject are
already aligned.

Verified: 23 tests pass, compileall clean, viralens --help ok, frozen build succeeds
and --vl-exec compare_form / fetch_multi / build_report all work (new modules bundled,
BilibiliError surfaces correctly).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@HarryXin0919
HarryXin0919 merged commit 51a63b8 into main Jun 3, 2026
@HarryXin0919
HarryXin0919 deleted the report-driven-fixes branch June 3, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant