Skip to content

test: expand golden-utterance e2e coverage, fix template gaps - #84

Draft
JarbasAl wants to merge 1 commit into
devfrom
qa/number-facts-golden-utterances
Draft

test: expand golden-utterance e2e coverage, fix template gaps#84
JarbasAl wants to merge 1 commit into
devfrom
qa/number-facts-golden-utterances

Conversation

@JarbasAl

@JarbasAl JarbasAl commented Sep 4, 2026

Copy link
Copy Markdown
Member

🤖 Auto-generated by Claude Opus 4.8 (claude-opus-4-8) via Claude Code — NOT human-reviewed. Verify before acting.

Expands the golden-utterance end-to-end test suite for this skill and fixes gaps found across the date_trivia, math_trivia, number_trivia, and year_trivia intent templates while writing those tests.

Add many natural en-US phrasings across all four trivia intents
(number/math/date/year), including varied {number}/{year}/{date} slot
values, sibling-confusion negatives (each utterance must route to its
own intent and not a sibling), and dedicated slot-extraction tests that
prove the captured value actually reaches the fact-fetcher rather than
just matching the intent.

Fixed real template gaps surfaced by these natural phrasings:
- number_trivia/math_trivia/date_trivia/year_trivia only accepted a
  'give me' opener; 'tell me' variants of the same phrasing failed to
  match (e.g. 'tell me a fact about number seven').
- math_trivia had no {number} slot pattern at all, so 'give me a math
  fact about the number 7' never routed anywhere.
- date_trivia and year_trivia had no bare 'date/year fact' pattern
  without an explicit day/year, so plain requests like 'tell me a year
  fact' or 'give me a random date fact' never matched.

Fail-before: reverting only the .intent changes made 22 of the new
tests fail (18 golden-utterance routing failures, 2 sibling-confusion
failures, 4 slot-extraction failures) against the unfixed templates;
restoring the fix makes the full suite (65 e2e + 12 unit tests) pass.
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Beep! Your PR results are served. 🍽️

I've aggregated the results of the automated checks for this PR below.

🏷️ Release Preview

Ensuring the release is as shiny as a new penny. ✨

Current: 0.1.15a1Next: 0.1.15a2

Signal Value
Label (none)
PR title test: expand golden-utterance e2e coverage, fix template gaps
Bump alpha

✅ PR title follows conventional commit format.


🚀 Release Channel Compatibility

Predicted next version: 0.1.15a2

Channel Status Note Current Constraint
Stable Compatible ovos-skill-number-facts>=0.1.12,<0.2.0
Testing Compatible ovos-skill-number-facts>=0.1.12,<1.0.0
Alpha Compatible ovos-skill-number-facts>=0.1.15a1

🧪 Spec Lint

The results are fresh out of the pipeline. 🏗️

12 error(s), 10 warning(s)

locale/ca-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/ca-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/da-DK/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/da-DK/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/de-DE/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/de-DE/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/de-DE/api.voc: error: single-branch group (Pi): a group must offer a choice between at least two branches  [in: '(Zahl) (Pi)']
locale/en-US/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/en-US/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/en-US/no.number.found.dialog: error: base name 'no.number.found' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/es-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/es-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/eu-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/eu-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/fr-FR/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/fr-FR/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/gl-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/gl-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/kab/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/kab/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/pt-PT/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)

🔍 Lint

A detailed summary of the latest automation run. 📝

ruff: issues found — see job log

🌍 Locale Build

Checking if there's anything else we need to do. 📋

✅ Locale properly configured (114 files, 10 languages)

Locale directories found:

  • locale

Localization coverage:

  • locale: 114 files in 10 languages (ca-ES, pt-PT, de-DE, fr-FR, da-DK...)

pyproject.toml:[tool.setuptools.package-data.ovos_skill_number_facts] includes locale

  • locale/**/*

Build manifest: ✅ 228 locale files included in package

🎙️ Skill

Is the skill's voice clear and confident? Let's see. 🗣️

ℹ️ Not an OVOS skill repo — check skipped.

🔒 Security (pip-audit)

Ensuring our password hashing is up to date. 🔨

✅ No known vulnerabilities found (78 packages scanned).

📊 Coverage

The coverage audit is ready for your inspection. 📋

40.2% total coverage

Per-file coverage (2 files)
File Coverage Missing lines
version.py 0.0% 5
__init__.py 42.9% 44

Full report: download the coverage-report artifact.

📋 Repo Health

Ensuring the repo's joints are well-oiled (aka CI/CD). ⚙️

✅ All required files present.

Latest Version: 0.1.15a1

version.py — Version file
README.md — README
LICENSE — License file
pyproject.toml — pyproject.toml
⚠️ setup.py — setup.py
🗑️ MANIFEST.in — legacy: pyproject.toml is present; this file is no longer needed
CHANGELOG.md — Changelog
version.py has valid version block markers

🔨 Build Tests

The compiler has spoken! Here is the verdict. 📜

✅ All versions pass

Python Build Install Tests
3.10
3.11
3.12
3.13
3.14

🔌 Skill Tests (ovoscope)

Testing the skill's 'vocabulary' across different languages. 🌍

0/65 passed, 65 errors

❌ **TestDateTrivia** — 0/1
Test Result
test_date_fact ❌ error
❌ **TestMathTrivia** — 0/1
Test Result
test_math_fact ❌ error
❌ **TestNumberTrivia** — 0/3
Test Result
test_number_fact ❌ error
test_give_me_a_fact_about_numbers ❌ error
test_random_number_fact ❌ error
❌ **TestTriviaRoutingWithoutAdapt** — 0/4
Test Result
test_math_fact ❌ error
test_date_fact ❌ error
test_year_fact ❌ error
test_number_fact ❌ error
❌ **TestYearTrivia** — 0/1
Test Result
test_year_fact ❌ error
❌ **test_date_slot_is_actually_extracted** — 0/1
Test Result
test_date_slot_is_actually_extracted ❌ error
❌ **test_golden_utterance[fact about december 3]** — 0/1
Test Result
test_golden_utterance[fact about december 3] ❌ error
❌ **test_golden_utterance[fact about the date 5]** — 0/1
Test Result
test_golden_utterance[fact about the date 5] ❌ error
❌ **test_golden_utterance[fact about the year 1992]** — 0/1
Test Result
test_golden_utterance[fact about the year 1992] ❌ error
❌ **test_golden_utterance[fact about today]** — 0/1
Test Result
test_golden_utterance[fact about today] ❌ error
❌ **test_golden_utterance[give me a fact about march 14]** — 0/1
Test Result
test_golden_utterance[give me a fact about march 14] ❌ error
❌ **test_golden_utterance[give me a fact about numbers]** — 0/1
Test Result
test_golden_utterance[give me a fact about numbers] ❌ error
❌ **test_golden_utterance[give me a fact about the number 42]** — 0/1
Test Result
test_golden_utterance[give me a fact about the number 42] ❌ error
❌ **test_golden_utterance[give me a fact about tomorrow]** — 0/1
Test Result
test_golden_utterance[give me a fact about tomorrow] ❌ error
❌ **test_golden_utterance[give me a math fact about the number 7]** — 0/1
Test Result
test_golden_utterance[give me a math fact about the number 7] ❌ error
❌ **test_golden_utterance[give me a math fact]** — 0/1
Test Result
test_golden_utterance[give me a math fact] ❌ error
❌ **test_golden_utterance[give me a mathematical trivia]** — 0/1
Test Result
test_golden_utterance[give me a mathematical trivia] ❌ error
❌ **test_golden_utterance[give me a random date fact]** — 0/1
Test Result
test_golden_utterance[give me a random date fact] ❌ error
❌ **test_golden_utterance[give me a random math fact]** — 0/1
Test Result
test_golden_utterance[give me a random math fact] ❌ error
❌ **test_golden_utterance[give me a random number curiosity]** — 0/1
Test Result
test_golden_utterance[give me a random number curiosity] ❌ error
❌ **test_golden_utterance[give me a random year fact]** — 0/1
Test Result
test_golden_utterance[give me a random year fact] ❌ error
❌ **test_golden_utterance[give me a trivia about the number 256]** — 0/1
Test Result
test_golden_utterance[give me a trivia about the number 256] ❌ error
❌ **test_golden_utterance[give me a trivia about the year 1999]** — 0/1
Test Result
test_golden_utterance[give me a trivia about the year 1999] ❌ error
❌ **test_golden_utterance[give me a year curiosity]** — 0/1
Test Result
test_golden_utterance[give me a year curiosity] ❌ error
❌ **test_golden_utterance[give me another number fact]** — 0/1
Test Result
test_golden_utterance[give me another number fact] ❌ error
❌ **test_golden_utterance[number fact 7]** — 0/1
Test Result
test_golden_utterance[number fact 7] ❌ error
❌ **test_golden_utterance[tell me a curiosity about number 15]** — 0/1
Test Result
test_golden_utterance[tell me a curiosity about number 15] ❌ error
❌ **test_golden_utterance[tell me a date fact]** — 0/1
Test Result
test_golden_utterance[tell me a date fact] ❌ error
❌ **test_golden_utterance[tell me a fact about mathematics]** — 0/1
Test Result
test_golden_utterance[tell me a fact about mathematics] ❌ error
❌ **test_golden_utterance[tell me a fact about number seven]** — 0/1
Test Result
test_golden_utterance[tell me a fact about number seven] ❌ error
❌ **test_golden_utterance[tell me a fact about the year 2020]** — 0/1
Test Result
test_golden_utterance[tell me a fact about the year 2020] ❌ error
❌ **test_golden_utterance[tell me a math curiosity]** — 0/1
Test Result
test_golden_utterance[tell me a math curiosity] ❌ error
❌ **test_golden_utterance[tell me a math trivia about 12]** — 0/1
Test Result
test_golden_utterance[tell me a math trivia about 12] ❌ error
❌ **test_golden_utterance[tell me a number trivia]** — 0/1
Test Result
test_golden_utterance[tell me a number trivia] ❌ error
❌ **test_golden_utterance[tell me a random number fact]** — 0/1
Test Result
test_golden_utterance[tell me a random number fact] ❌ error
❌ **test_golden_utterance[tell me a trivia about yesterday]** — 0/1
Test Result
test_golden_utterance[tell me a trivia about yesterday] ❌ error
❌ **test_golden_utterance[tell me a year fact]** — 0/1
Test Result
test_golden_utterance[tell me a year fact] ❌ error
❌ **test_golden_utterance[year fact 2001]** — 0/1
Test Result
test_golden_utterance[year fact 2001] ❌ error
❌ **test_negative_confusable_not_claimed[play a random song]** — 0/1
Test Result
test_negative_confusable_not_claimed[play a random song] ❌ error
❌ **test_negative_confusable_not_claimed[play some music]** — 0/1
Test Result
test_negative_confusable_not_claimed[play some music] ❌ error
❌ **test_negative_confusable_not_claimed[set a timer for 5 minutes]** — 0/1
Test Result
test_negative_confusable_not_claimed[set a timer for 5 minutes] ❌ error
❌ **test_negative_confusable_not_claimed[tell me a random joke]** — 0/1
Test Result
test_negative_confusable_not_claimed[tell me a random joke] ❌ error
❌ **test_negative_confusable_not_claimed[what year is it]** — 0/1
Test Result
test_negative_confusable_not_claimed[what year is it] ❌ error
❌ **test_negative_confusable_not_claimed[what's the weather]** — 0/1
Test Result
test_negative_confusable_not_claimed[what's the weather] ❌ error
❌ **test_negative_confusable_not_claimed[what's today's date]** — 0/1
Test Result
test_negative_confusable_not_claimed[what's today's date] ❌ error
❌ **test_number_slot_is_actually_extracted[fact about the year 1992-year_trivia-1992]** — 0/1
Test Result
test_number_slot_is_actually_extracted[fact about the year 1992-year_trivia-1992] ❌ error
❌ **test_number_slot_is_actually_extracted[give me a fact about the number 42-number_trivia-42]** — 0/1
Test Result
test_number_slot_is_actually_extracted[give me a fact about the number 42-number_trivia-42] ❌ error
❌ **test_number_slot_is_actually_extracted[give me a math fact about the number 7-number_math-7]** — 0/1
Test Result
test_number_slot_is_actually_extracted[give me a math fact about the number 7-number_math-7] ❌ error
❌ **test_number_slot_is_actually_extracted[give me a trivia about the number 256-number_trivia-256]** — 0/1
Test Result
test_number_slot_is_actually_extracted[give me a trivia about the number 256-number_trivia-256] ❌ error
❌ **test_number_slot_is_actually_extracted[number fact 7-number_trivia-7]** — 0/1
Test Result
test_number_slot_is_actually_extracted[number fact 7-number_trivia-7] ❌ error
❌ **test_number_slot_is_actually_extracted[tell me a curiosity about number 15-number_trivia-15]** — 0/1
Test Result
test_number_slot_is_actually_extracted[tell me a curiosity about number 15-number_trivia-15] ❌ error
❌ **test_number_slot_is_actually_extracted[tell me a fact about the year 2020-year_trivia-2020]** — 0/1
Test Result
test_number_slot_is_actually_extracted[tell me a fact about the year 2020-year_trivia-2020] ❌ error
❌ **test_number_slot_is_actually_extracted[tell me a math trivia about 12-number_math-12]** — 0/1
Test Result
test_number_slot_is_actually_extracted[tell me a math trivia about 12-number_math-12] ❌ error
❌ **test_number_slot_is_actually_extracted[year fact 2001-year_trivia-2001]** — 0/1
Test Result
test_number_slot_is_actually_extracted[year fact 2001-year_trivia-2001] ❌ error
❌ **test_sibling_intent_not_confused[fact about december 3]** — 0/1
Test Result
test_sibling_intent_not_confused[fact about december 3] ❌ error
❌ **test_sibling_intent_not_confused[fact about the year 1992]** — 0/1
Test Result
test_sibling_intent_not_confused[fact about the year 1992] ❌ error
❌ **test_sibling_intent_not_confused[give me a math fact]** — 0/1
Test Result
test_sibling_intent_not_confused[give me a math fact] ❌ error
❌ **test_sibling_intent_not_confused[number fact 7]** — 0/1
Test Result
test_sibling_intent_not_confused[number fact 7] ❌ error
❌ **test_sibling_intent_not_confused[tell me a date fact]** — 0/1
Test Result
test_sibling_intent_not_confused[tell me a date fact] ❌ error
❌ **test_sibling_intent_not_confused[tell me a year fact]** — 0/1
Test Result
test_sibling_intent_not_confused[tell me a year fact] ❌ error

🚌 Bus Coverage

A bird's eye view of the message bus landscape. 🦅

⚠️ Bus coverage report unavailable — check the job log.


Coded with logic, delivered with care. 🧠

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant