Skip to content

ci: run the unit tests outside flow/test #4808

Open

Description

@oharboe

Description

Not urgent, just documenting a chore. I tried to clean this up in a PR but I think this is more of a policy decision than coding work these days.

The Test Util Scripts workflow (.github/workflows/github-actions-cron-util-test.yml) runs flow/test/*.py. Some unit tests elsewhere in the tree are not run by CI, so they are run on demand and can break unnoticed:

Tests Cover Runner
flow/scripts/memories/{detect,gen_memories,idiomatic,schema}_test.py AUTO_MEMORIES none
flow/util/genMetrics_test.py genMetrics.py none
tools/FakeRAM2.0/test/*_test.py FakeRAM2.0 tools/FakeRAM2.0/test/test_all.sh

All of them pass on master (9ca2a9e7).

#4807 tried to fix this by documenting these tests as run on demand, and was closed. Moving them is little coding now that an AI can do the edit; what is open is policy:

  • Does every unit test in the tree run in CI?
  • Do unit tests live in flow/test/, where CI and contributors find them, or next to the code they cover?
  • Does a vendored tool such as FakeRAM2.0 keep its own suite and runner, called from the workflow?

Suggested Solution

If the answers are yes, in flow/test/, and yes:

  • Move the memories tests and genMetrics_test.py into flow/test/ as test_<name>.py. They use only the Python standard library (plus git for genMetrics_test.py), so the workflow needs nothing installed. Each reaches the code under test the way flow/test/test_convertDrc.py already does, a sys.path entry for ../scripts/memories or ../util.
  • Add flow/scripts/memories/*.py to the workflow's paths triggers. Today a pull request that changes only that directory does not run the workflow.
  • Run tools/FakeRAM2.0/test/test_all.sh as a workflow step instead of moving its tests: its tests import FakeRAM2.0's modules relative to tools/FakeRAM2.0, and the suite has its own runner. Add tools/FakeRAM2.0/** to the triggers.

Additional Context

The workflow's loop, for file in *.py; do python3 "$file"; done, runs under the GitHub Actions default bash -e, so it stops at the first failing test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions