test: expand golden-utterance e2e coverage, fix template gaps - #84
test: expand golden-utterance e2e coverage, fix template gaps#84JarbasAl wants to merge 1 commit into
Conversation
Add many natural en-US phrasings across all four trivia intents
(number/math/date/year), including varied {number}/{year}/{date} slot
values, sibling-confusion negatives (each utterance must route to its
own intent and not a sibling), and dedicated slot-extraction tests that
prove the captured value actually reaches the fact-fetcher rather than
just matching the intent.
Fixed real template gaps surfaced by these natural phrasings:
- number_trivia/math_trivia/date_trivia/year_trivia only accepted a
'give me' opener; 'tell me' variants of the same phrasing failed to
match (e.g. 'tell me a fact about number seven').
- math_trivia had no {number} slot pattern at all, so 'give me a math
fact about the number 7' never routed anywhere.
- date_trivia and year_trivia had no bare 'date/year fact' pattern
without an explicit day/year, so plain requests like 'tell me a year
fact' or 'give me a random date fact' never matched.
Fail-before: reverting only the .intent changes made 22 of the new
tests fail (18 golden-utterance routing failures, 2 sibling-confusion
failures, 4 slot-extraction failures) against the unfixed templates;
restoring the fix makes the full suite (65 e2e + 12 unit tests) pass.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Beep! Your PR results are served. 🍽️I've aggregated the results of the automated checks for this PR below. 🏷️ Release PreviewEnsuring the release is as shiny as a new penny. ✨ Current:
✅ PR title follows conventional commit format. 🚀 Release Channel Compatibility Predicted next version:
🧪 Spec LintThe results are fresh out of the pipeline. 🏗️ 12 error(s), 10 warning(s) 🔍 LintA detailed summary of the latest automation run. 📝 ❌ ruff: issues found — see job log 🌍 Locale BuildChecking if there's anything else we need to do. 📋 ✅ Locale properly configured (114 files, 10 languages) Locale directories found:
Localization coverage:
pyproject.toml: ✅
Build manifest: ✅ 228 locale files included in package 🎙️ SkillIs the skill's voice clear and confident? Let's see. 🗣️ ℹ️ Not an OVOS skill repo — check skipped. 🔒 Security (pip-audit)Ensuring our password hashing is up to date. 🔨 ✅ No known vulnerabilities found (78 packages scanned). 📊 CoverageThe coverage audit is ready for your inspection. 📋 ❌ 40.2% total coverage Per-file coverage (2 files)
Full report: download the 📋 Repo HealthEnsuring the repo's joints are well-oiled (aka CI/CD). ⚙️ ✅ All required files present. Latest Version: ✅ 🔨 Build TestsThe compiler has spoken! Here is the verdict. 📜 ✅ All versions pass
🔌 Skill Tests (ovoscope)Testing the skill's 'vocabulary' across different languages. 🌍 ❌ 0/65 passed, 65 errors ❌ **TestDateTrivia** — 0/1
❌ **TestMathTrivia** — 0/1
❌ **TestNumberTrivia** — 0/3
❌ **TestTriviaRoutingWithoutAdapt** — 0/4
❌ **TestYearTrivia** — 0/1
❌ **test_date_slot_is_actually_extracted** — 0/1
❌ **test_golden_utterance[fact about december 3]** — 0/1
❌ **test_golden_utterance[fact about the date 5]** — 0/1
❌ **test_golden_utterance[fact about the year 1992]** — 0/1
❌ **test_golden_utterance[fact about today]** — 0/1
❌ **test_golden_utterance[give me a fact about march 14]** — 0/1
❌ **test_golden_utterance[give me a fact about numbers]** — 0/1
❌ **test_golden_utterance[give me a fact about the number 42]** — 0/1
❌ **test_golden_utterance[give me a fact about tomorrow]** — 0/1
❌ **test_golden_utterance[give me a math fact about the number 7]** — 0/1
❌ **test_golden_utterance[give me a math fact]** — 0/1
❌ **test_golden_utterance[give me a mathematical trivia]** — 0/1
❌ **test_golden_utterance[give me a random date fact]** — 0/1
❌ **test_golden_utterance[give me a random math fact]** — 0/1
❌ **test_golden_utterance[give me a random number curiosity]** — 0/1
❌ **test_golden_utterance[give me a random year fact]** — 0/1
❌ **test_golden_utterance[give me a trivia about the number 256]** — 0/1
❌ **test_golden_utterance[give me a trivia about the year 1999]** — 0/1
❌ **test_golden_utterance[give me a year curiosity]** — 0/1
❌ **test_golden_utterance[give me another number fact]** — 0/1
❌ **test_golden_utterance[number fact 7]** — 0/1
❌ **test_golden_utterance[tell me a curiosity about number 15]** — 0/1
❌ **test_golden_utterance[tell me a date fact]** — 0/1
❌ **test_golden_utterance[tell me a fact about mathematics]** — 0/1
❌ **test_golden_utterance[tell me a fact about number seven]** — 0/1
❌ **test_golden_utterance[tell me a fact about the year 2020]** — 0/1
❌ **test_golden_utterance[tell me a math curiosity]** — 0/1
❌ **test_golden_utterance[tell me a math trivia about 12]** — 0/1
❌ **test_golden_utterance[tell me a number trivia]** — 0/1
❌ **test_golden_utterance[tell me a random number fact]** — 0/1
❌ **test_golden_utterance[tell me a trivia about yesterday]** — 0/1
❌ **test_golden_utterance[tell me a year fact]** — 0/1
❌ **test_golden_utterance[year fact 2001]** — 0/1
❌ **test_negative_confusable_not_claimed[play a random song]** — 0/1
❌ **test_negative_confusable_not_claimed[play some music]** — 0/1
❌ **test_negative_confusable_not_claimed[set a timer for 5 minutes]** — 0/1
❌ **test_negative_confusable_not_claimed[tell me a random joke]** — 0/1
❌ **test_negative_confusable_not_claimed[what year is it]** — 0/1
❌ **test_negative_confusable_not_claimed[what's the weather]** — 0/1
❌ **test_negative_confusable_not_claimed[what's today's date]** — 0/1
❌ **test_number_slot_is_actually_extracted[fact about the year 1992-year_trivia-1992]** — 0/1
❌ **test_number_slot_is_actually_extracted[give me a fact about the number 42-number_trivia-42]** — 0/1
❌ **test_number_slot_is_actually_extracted[give me a math fact about the number 7-number_math-7]** — 0/1
❌ **test_number_slot_is_actually_extracted[give me a trivia about the number 256-number_trivia-256]** — 0/1
❌ **test_number_slot_is_actually_extracted[number fact 7-number_trivia-7]** — 0/1
❌ **test_number_slot_is_actually_extracted[tell me a curiosity about number 15-number_trivia-15]** — 0/1
❌ **test_number_slot_is_actually_extracted[tell me a fact about the year 2020-year_trivia-2020]** — 0/1
❌ **test_number_slot_is_actually_extracted[tell me a math trivia about 12-number_math-12]** — 0/1
❌ **test_number_slot_is_actually_extracted[year fact 2001-year_trivia-2001]** — 0/1
❌ **test_sibling_intent_not_confused[fact about december 3]** — 0/1
❌ **test_sibling_intent_not_confused[fact about the year 1992]** — 0/1
❌ **test_sibling_intent_not_confused[give me a math fact]** — 0/1
❌ **test_sibling_intent_not_confused[number fact 7]** — 0/1
❌ **test_sibling_intent_not_confused[tell me a date fact]** — 0/1
❌ **test_sibling_intent_not_confused[tell me a year fact]** — 0/1
🚌 Bus CoverageA bird's eye view of the message bus landscape. 🦅 Coded with logic, delivered with care. 🧠 |
Expands the golden-utterance end-to-end test suite for this skill and fixes gaps found across the
date_trivia,math_trivia,number_trivia, andyear_triviaintent templates while writing those tests.