Skip to content

Add m2v-multilingual test gate to end2end suite - #86

Draft
JarbasAl wants to merge 1 commit into
devfrom
feat/m2v-test-gate
Draft

Add m2v-multilingual test gate to end2end suite#86
JarbasAl wants to merge 1 commit into
devfrom
feat/m2v-test-gate

Conversation

@JarbasAl

@JarbasAl JarbasAl commented Sep 6, 2026

Copy link
Copy Markdown
Member

🤖 Auto-generated by Claude (via Claude Code) — NOT human-reviewed. Verify before acting.

This adds an m2v-multilingual test gate to the end2end suite, part of the ecosystem-wide campaign to validate every default skill against the candidate default intent engine before it ships.

The new test (test/end2end/test_m2v_gate.py) boots the skill through ovoscope's get_m2v_minicroft helper, which loads the model2vec multilingual classifier (OpenVoiceOS/ovos-m2v-intents-multi-128M-v5), and sends five utterances lifted from the existing golden corpus, one per intent plus an extra number-fact phrasing. It stubs the numbersapi.com fetchers the same way test_golden_utterances.py already does, so the suite stays network-free, and checks that the spoken text is exactly the stubbed fact sentinel for the intent that should have handled it.

All five cases route correctly and speak the right stubbed fact under the candidate model — no mispredicts found. The existing padacioso-based golden suite is untouched — this is an addition, not a replacement.

Test plan

  • pytest test/end2end/test_m2v_gate.py -v — 6 passed (booted live against the real model in a throwaway venv)

Boots the skill through the candidate default intent engine (the
model2vec multilingual classifier) via ovoscope's get_m2v_minicroft and
exercises a representative slice of the golden utterances, asserting
both routing and the spoken fact content (stubbing numbersapi.com the
same way the existing padacioso golden suite does).

All five golden cases route to the correct intent and speak the
stubbed fact content under the candidate model.
@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

Beep boop! The automated check sequence is complete. 🦾

I've aggregated the results of the automated checks for this PR below.

🏷️ Release Preview

What's in the box? A preview of the next release! 📦

Current: 0.1.15a1Next: 0.1.15a2

Signal Value
Label (none)
PR title Add m2v-multilingual test gate to end2end suite
Bump alpha

⚠️ No conventional commit prefix — alpha-only bump.
Suggested: fix: update the thing or feat: update the thing


🚀 Release Channel Compatibility

Predicted next version: 0.1.15a2

Channel Status Note Current Constraint
Stable Compatible ovos-skill-number-facts>=0.1.12,<0.2.0
Testing Compatible ovos-skill-number-facts>=0.1.12,<1.0.0
Alpha Compatible ovos-skill-number-facts>=0.1.15a1

🔍 Lint

Ensuring the codebase remains stable and healthy. 🛡️

ruff: issues found — see job log

🎙️ Skill

I've peered into the heart of this skill. 💖

ℹ️ Not an OVOS skill repo — check skipped.

🔒 Security (pip-audit)

Checking if we're following security best practices. 📏

✅ No known vulnerabilities found (78 packages scanned).

📋 Repo Health

I've checked the repo's flexibility (aka refactorability). 🧘‍♂️

✅ All required files present.

Latest Version: 0.1.15a1

version.py — Version file
README.md — README
LICENSE — License file
pyproject.toml — pyproject.toml
⚠️ setup.py — setup.py
🗑️ MANIFEST.in — legacy: pyproject.toml is present; this file is no longer needed
CHANGELOG.md — Changelog
version.py has valid version block markers

🌍 Locale Build

The automated gnomes have finished their shift. 🍄

✅ Locale properly configured (114 files, 10 languages)

Locale directories found:

  • locale

Localization coverage:

  • locale: 114 files in 10 languages (ca-ES, pt-PT, de-DE, fr-FR, da-DK...)

pyproject.toml:[tool.setuptools.package-data.ovos_skill_number_facts] includes locale

  • locale/**/*

Build manifest: ✅ 228 locale files included in package

🧪 Spec Lint

Evaluating the overall quality of your PR. ✨

11 error(s), 10 warning(s)

locale/ca-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/ca-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/da-DK/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/da-DK/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/de-DE/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/de-DE/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/en-US/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/en-US/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/en-US/no.number.found.dialog: error: base name 'no.number.found' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/es-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/es-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/eu-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/eu-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/fr-FR/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/fr-FR/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/gl-ES/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/gl-ES/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/kab/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)
locale/kab/Numbers.voc: warning: file name should be lowercase (OVOS-INTENT-2 §2)
locale/pt-PT/Numbers.voc: error: base name 'Numbers' must be lowercase ASCII letters, digits and underscores only (OVOS-INTENT-2 §2)

📊 Coverage

A bird's eye view of your test coverage landscape. 🦅

40.2% total coverage

Per-file coverage (2 files)
File Coverage Missing lines
version.py 0.0% 5
__init__.py 42.9% 44

Full report: download the coverage-report artifact.

⚖️ License Check

Are we all good on the legal front? Let's find out! 📑

✅ No license violations found.

Policy: Apache 2.0 (universal donor). StrongCopyleft / NetworkCopyleft / WeakCopyleft / Other / Error categories fail. MPL allowed.

🔨 Build Tests

Checking the plumbing of your data flows. 🚰

✅ All versions pass

Python Build Install Tests
3.10
3.11
3.12
3.13
3.14

🔌 Skill Tests (ovoscope)

I've checked the skill's 'privacy' (aka data handling). 🔒

29/29 passed

TestDateTrivia — 1/1
TestMathTrivia — 1/1
TestNumberTrivia — 3/3
TestTriviaRoutingWithoutAdapt — 4/4
TestYearTrivia — 1/1
test_golden_utterance[fact about december 3] — 1/1
test_golden_utterance[fact about the date 5] — 1/1
test_golden_utterance[fact about the year 1992] — 1/1
test_golden_utterance[fact about today] — 1/1
test_golden_utterance[give me a fact about the number 42] — 1/1
test_golden_utterance[give me a math fact] — 1/1
test_golden_utterance[give me a mathematical trivia] — 1/1
test_golden_utterance[give me a trivia about the year 1999] — 1/1
test_golden_utterance[number fact 7] — 1/1
test_golden_utterance[tell me a fact about mathematics] — 1/1
test_golden_utterance[tell me a random number fact] — 1/1
test_golden_utterance[year fact 2001] — 1/1
test_negative_confusable_not_claimed[play a random song] — 1/1
test_negative_confusable_not_claimed[play some music] — 1/1
test_negative_confusable_not_claimed[set a timer for 5 minutes] — 1/1
test_negative_confusable_not_claimed[tell me a random joke] — 1/1
test_negative_confusable_not_claimed[what year is it] — 1/1
test_negative_confusable_not_claimed[what's the weather] — 1/1
test_negative_confusable_not_claimed[what's today's date] — 1/1

🚌 Bus Coverage

A forensic look at what's being emitted. 🔎

⚠️ Bus coverage report unavailable — check the job log.


Stay curious and keep coding! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant