Skip to content

TESTING.md

Latest commit

 

History

History
180 lines (147 loc) · 7.02 KB

TESTING.md

File metadata and controls

180 lines (147 loc) · 7.02 KB
Raw

Manual test scenarios — install & usage

The automated suite (uv run pytest) covers the engine. This document is the human acceptance pass: prove a real user can install ASPIS and run the loop on both Windows and Linux. Run every scenario on each OS and compare.

Legend: ✅ pass · ⚠️ warn-but-ok · ❌ fix before publish.

S0 — Automated smoke test (run this first)

One command proves a fresh install drives the whole pipeline — install → version → doctor → init → status → models (--sync/--available) → per-agent pin → bootstrap — in a throwaway venv + temp project, with zero effect on your global tools or this repo.

Linux / macOS / WSL

./scripts/smoke-test.sh           # add --keep to inspect the sandbox

Windows (PowerShell)

.\scripts\smoke-test.ps1          # add -Keep to inspect the sandbox

Expect: [smoke] ALL 14 CHECKS PASSED — fresh install works end-to-end. (exit 0). Any FAIL line names the step and what it expected. Run it on both Windows and Linux/WSL — that is the cross-platform proof (S6) for the core flow. The scenarios below (S1–S7) are the deeper manual pass for things the smoke test cannot assert automatically (real shells, gh, the agent loop).

S1 — One-command install

Linux / macOS

git clone https://github.com/mahmoud-emad-dev/aspis.git && cd aspis
./install.sh

Windows (PowerShell)

git clone https://github.com/mahmoud-emad-dev/aspis.git; cd aspis
.\install.ps1

Expect: prerequisite checks pass, uv installed if missing, aspis --version prints the installed version, aspis doctor runs. ✅ when aspis resolves from a new shell.

Check: command -v aspis (Linux) / (Get-Command aspis).Source (Windows) → points at ~/.local/bin/aspis (Linux) or %USERPROFILE%\.local\bin\aspis.exe (Windows).

S2 — aspis doctor outside a project

aspis doctor

Expect: [ok] python, [ok] aspis, [ok] git, [warn] project not an ASPIS project, and All checks passed. (exit 0). The project warning is expected (⚠️, not ❌).

Edge cases: run with an old Python (should refuse at install), with git absent (doctor warns, still works).

S3 — Initialize a fresh project

mkdir demo && cd demo
aspis init --write

Expect: .aspis/ (brain: context, config, rules, templates), plus .opencode/ and/or .claude/ (agents, skills, commands), AGENTS.md, and CLAUDE.md if Claude is a target. Then:

aspis doctor          # now [ok] project
aspis status

✅ when the project is recognized and the runtime dirs contain agents + skills.

S3b — Bootstrap (make the project live)

aspis bootstrap --write -y       # non-interactive happy path
aspis doctor                     # 0-FAIL, tree clean (auto-committed)
aspis bootstrap --write -y       # run again

Expect: the brain is filled (goal, stack, default mode), the canonical structure has every folder filled, none empty or stray, the onboarding package self-cleans once the project is live, and history is clean. ✅ when the second run is idempotent (no changes, no error). ❌ if a bootstrap agent/skill lingers after the project is live.

S4 — Model detection & assignment

aspis models                 # per-runtime tiers, resolved to your connected models
aspis models --available     # the full menu from your connected plans only
aspis models --sync          # writes .aspis/config/agent-models.yaml

Open .aspis/config/agent-models.yaml: the menu (ranked per capability) is at the top, every agent pre-assigned a best-fit available model, with a by_capability block.

  • ✅ only models your plan provides appear.
  • Edit a by_capability value, run aspis models → the change is reflected; a pin to a non-available model is flagged [!] not available.
  • Connect a new provider, run aspis doctor → [warn] connected plans changed — run aspis models --sync.

S5 — Commit flow (single writer + hooks)

# inside an initialized project that is a git repo
echo "test" > notes.md
aspis commit notes.md --type docs --title "add notes"
git log -1

Expect: the hooks run (pre-commit checks, commit-msg validates the convention, post-commit refreshes context), the message follows the convention, and no AI/tool attribution is present. ❌ if -A staging or attribution appears.

S6 — Cross-platform parity

Run S3–S4 on Windows and on Linux/WSL against the same runtime/plan. The rendered agent files and aspis models output should match (modulo line endings). ❌ on any encoding, path, or line-ending divergence.

S7 — Reinstall / uninstall

uv tool install --reinstall .     # after pulling changes
uv tool uninstall aspis           # clean removal

Expect: reinstall picks up the new version; uninstall removes the shim and leaves project files untouched.

S8 — Git hygiene (auto-clean + gitignore)

# inside an initialized project (git repo)
touch junk.tmp .aspis/templates/.gitkeep   # a populated dir shouldn't keep .gitkeep
aspis commit notes.md --type docs --title "trigger hooks"
aspis gitignore                            # offline-first, stack-aware

Expect: post-commit cleans junk + reaps stale .gitkeep from populated dirs; the commit-msg hook strips any AI/tool attribution; aspis gitignore maintains a nature-based ignore (generated/state ignored, source/durable tracked) without network. ❌ if attribution survives or a populated dir keeps a .gitkeep.

S9 — Catalog ↔ export parity (in the ASPIS repo)

uv run pytest tests/test_catalog.py tests/test_consistency.py -v

Expect: the runtime-neutral catalog reproduces the live .claude/ and .opencode/ byte-for-byte — the moat that proves there is no second source of truth. ❌ on any drift.

test_catalog.py exercises the catalog → per-runtime rendering (agents, skills, commands, plugins ship into .claude/ and .opencode/ for every bundled lead); test_consistency.py is the machine-checked "nothing is missing" guard (every asset a profile selects exists, every skill and delegate an agent references ships in that profile, every command binds to a shipped agent). Together they are the narrowest suite that enforces the S9 invariant.

Where things live (verify the layout)

Item Linux / macOS Windows
aspis shim ~/.local/bin/aspis %USERPROFILE%\.local\bin\aspis.exe
Tool environment uv-managed (uv tool dir) uv-managed (uv tool dir)
Global model config (optional) ~/.aspis/config/project.yaml %USERPROFILE%\.aspis\config\project.yaml
Per-project brain (tracked) <project>/.aspis/ <project>\.aspis\
Per-project generated state (gitignored) <project>/.aspis/index/, config/reference/.runtime-inventory.json same, under <project>\.aspis\
Detected provider config (read, not written) ~/.local/share/opencode/auth.json, ~/.claude/settings.json %USERPROFILE%\.local\share\opencode\auth.json, %USERPROFILE%\.claude\settings.json