Fix external-review issues: dedup OFF_MARKERS, paginate Bilibili, val… - #3
Merged
Conversation
…idate creators, version cap, retire bidao pilot, add tests + CI Robustness / bugs: - shared_markers.py: single source for OFF_MARKERS (FULL for features, COMPARE for compare_form -- the deliberate livestream / 联名 divergence preserved, behavior unchanged), ending the silent drift between the two copies. - fetch_bilibili.py: paginate up to num (was first-page-only -> missed data past 50); add BilibiliError mirroring YouTubeError; factor out _to_record / _get_page. - creators.py: validate_creators() lists ALL config problems (typo'd keys, missing alias/zone, duplicate alias, youtube missing channel) instead of a KeyError at fetch time; called at the start of fetch_multi. - Cap bilibili-api-python <18.0.0 (requirements.txt + pyproject.toml). - schema.py: VideoRecord TypedDict documenting the standard video record. Cleanup -- retire the single-creator pilot cluster (bidao-hardcoded, mutually coupled via classified.json, superseded by the generalized features/scan_signals pipeline, and absent from every pipeline / README command list): - delete classify_and_stats.py, fetch_videos.py, and the orphaned consumers comments.py + subtitle.py (they read the now-gone classified.json); drop all four from the PyInstaller spec hiddenimports; add shared_markers + schema. - anonymize remaining bidao / real-BV references in analyze_video.py + diagnose.py docstrings; reword the README L2-text line to the surviving compare_meme.py. Tests + CI (the project had zero tests): - conftest.py + tests/ (23 tests): features.extract, off_tag, OFF_MARKERS divergence, validate_creators, spearman, build_report formatters, runtime paths, and fetch_bilibili parse/record/error + pagination (incl. the >50-video multi-page path). - pyproject dev extra + pytest config; CI now runs pytest. Deliberately skipped (with reasons): scripts/ -> subpackage restructure (would break the released PyInstaller packaging), web CSRF/rate-limit (localhost-only app), scipy for spearman (intentional zero-dep). Two report claims were verified already-fine and left alone: scan_signals already degrades on missing data; requirements/pyproject are already aligned. Verified: 23 tests pass, compileall clean, viralens --help ok, frozen build succeeds and --vl-exec compare_form / fetch_multi / build_report all work (new modules bundled, BilibiliError surfaces correctly). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…idate creators, version cap, retire bidao pilot, add tests + CI
Robustness / bugs:
Cleanup -- retire the single-creator pilot cluster (bidao-hardcoded, mutually coupled via classified.json, superseded by the generalized features/scan_signals pipeline, and absent from every pipeline / README command list):
Tests + CI (the project had zero tests):
Deliberately skipped (with reasons): scripts/ -> subpackage restructure (would break the released PyInstaller packaging), web CSRF/rate-limit (localhost-only app), scipy for spearman (intentional zero-dep). Two report claims were verified already-fine and left alone: scan_signals already degrades on missing data; requirements/pyproject are already aligned.
Verified: 23 tests pass, compileall clean, viralens --help ok, frozen build succeeds and --vl-exec compare_form / fetch_multi / build_report all work (new modules bundled, BilibiliError surfaces correctly).