Skip to content

Filter Whisper hallucinations (closes #5) - #11

Open
kenodressel wants to merge 12 commits into
mainfrom
feat/hallucination-filter
Open

Filter Whisper hallucinations (closes #5)#11
kenodressel wants to merge 12 commits into
mainfrom
feat/hallucination-filter

Conversation

@kenodressel

Copy link
Copy Markdown
Owner

Summary

Closes #5.

Whisper produces canned phrases ("Thank you.", "Untertitel im Auftrag des ZDF") and consecutive-token/n-gram repetition on noise, silence, or short audio. This change drops those after transcription and shows a brief ⚠️ "Likely noise" badge instead of pasting garbage.

  • Post-Whisper, two detectors in sabbel/hallucinations.py: token/n-gram repetition heuristic + normalized exact-match against a 22-phrase default phantom set (English + German + bracketed music/applause + dots/notes).
  • Users can extend the phantom set via a new [hallucinations] section in ~/.config/sabbel/dictionary.toml — hot-reloaded between recordings.
  • Filter runs between the empty-text check and apply_replacements so phantom matches aren't broken by dictionary substitutions.

Spec: docs/superpowers/specs/2026-05-13-hallucination-filtering-design.md

Test plan

  • Unit suite: uv run pytest — 113 passing (39 new tests across test_hallucinations.py + test_dictionary.py)
  • Manual on hardware (after make install-app):
    • Hold hotkey in a silent room → ⚠️ "Likely noise" badge, no paste
    • Speak a normal sentence → normal dictation, no filter trigger
    • Add a custom phrase to ~/.config/sabbel/dictionary.toml under [hallucinations] → picked up on next recording without restart
    • tail /tmp/sabbel-runtime.log after a noise rejection → shows Filtered hallucination: ...

Known limitations

  • Emphatic "nein nein nein nein" or "ja ja ja ja" (4+ consecutive identical tokens) is filtered. Accepted false positive — emphatic 4-repeat speech is rare in dictation.
  • Single-word "you" and "Bye." are dropped (Whisper noise-bias phantoms). Acceptable for dictation; users dictating one-word messages should add a custom replacement if it bites.
  • 5+-gram repetition (e.g., "a b c d e a b c d e a b c d e") is not flagged. Spec caps n at 4 — extremely unlikely in practice.

🤖 Generated with Claude Code

kenodressel and others added 12 commits May 13, 2026 16:40
Post-Whisper filter with two paths: consecutive-token/n-gram
repetition detector, and exact-match phantom phrase set (with
defaults + user-extensible list in dictionary.toml under
[hallucinations]). Drops the transcription and shows a brief
"Likely noise" menu-bar badge on match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rases

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dy docstring

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Whisper hallucination handling

1 participant