Add Agent QA eval harness

[pranshuchittora]

Summary

Add Agent QA as a product-level regression harness under Build AI systems > Evals and reliability.

Scope and problem

This serves the Building AI systems pillar. Output-level evals do not cover complete user journeys through a deployed web or mobile product. Agent QA exercises those interfaces from natural-language regression suites and retains testing context between runs.

The proposed README sentence is deliberately limited to inspectable capabilities. It makes no adoption, quality, or production-use claim.

Evidence and distinction

Primary evidence:

• The project README documents natural-language web/mobile testing, memory built from product, suite, and test observations, and self-healing behavior.
• The official quickstart documents installation and a runnable first test.
• The memory implementation exposes curation, retrieval, trust, and runtime-injection behavior as reviewable source.

The strongest overlapping entry is Promptfoo, which focuses on test cases, assertions, model comparisons, and red-team checks. Agent QA is materially different at the interface boundary: it drives browser and mobile workflows around the assembled product and retains scoped execution memory. It complements Promptfoo rather than replacing it.

Current non-qualitative signals are 781 GitHub stars and 3,251 npm downloads from 2026-07-11 through 2026-08-09. These are usage signals, not proof of production adoption, and I make no production-use claim.

Contributor-affiliated rubric estimate: technical/production quality 21/25, applicability 18/20, currentness 18/20, real-world evidence 11/20, documentation 9/10, and distinctiveness 5/5, for 82/100. The real-world-evidence score is the least certain and should receive independent review.

Maintenance and limitations

• The repository was created on 2026-05-07 and last pushed on 2026-08-03.
• The current npm release is agent-qa@0.1.21 and requires Node.js 24 or newer.
• The current FSL-1.1-ALv2 license is source-available, not OSI open source, and converts to Apache-2.0 after two years.
• Model execution requires a configured compatible model endpoint and may incur provider costs.
• I found no independent production case study, so none is claimed.

Affiliation

I am affiliated with Vostride and Agent QA.

Test plan

• python3 -m unittest discover -s tests -v: all 12 tests pass.
• python3 scripts/validate_readme.py README.md --base origin/master: 78 resources, zero errors, zero warnings.
• python3 scripts/validate_readme.py README.md --check-links --base origin/master: 78 resources, zero errors, one unrelated warning for a pre-existing Packt URL returning 403.
• git diff --check: passes.
• README, issues, and pull requests contain no existing Agent QA proposal.

Risks

The project is young and lacks independent production-adoption evidence. Reviewers may reasonably score real-world evidence below my estimate or decide that product-level UI regression is too narrow for this intentionally small guide.

Related issue

None.

Summary by CodeRabbit

• Documentation
  • Added the Agent QA resource to the “Evals and reliability” section of the documentation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7288edcf-fe28-41f7-b7b2-4e938c5c7ea7

📥 Commits

Reviewing files that changed from the base of the PR and between ab1c3cc and 3f06b77.

📒 Files selected for processing (1)

• README.md

Included review availability: Your plan includes up to 10 reviews per rolling hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

README.md adds the Agent QA resource and its description to the Evals and reliability section.

Changes

Agent QA Resource

Layer / File(s)	Summary
Add Agent QA resource README.md	The Evals and reliability list now includes the Agent QA resource and its description.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 3f06b

This PR adds a single documented project entry without changing product behavior or runtime configuration; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: owainlewis

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the addition of Agent QA as an evaluation harness and matches the pull request's main objective.
Description check	✅ Passed	The description covers every required template section with detailed scope, evidence, limitations, affiliation, tests, risks, and issue status.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

Warning

⚠️ This pull request has been flagged as potential spam (promotional) by CodeRabbit slop detection and should be reviewed carefully.

[greptile-apps]

Greptile Summary

Adds Agent QA to the “Evals and reliability” resources in the README.

• Describes it as a source-available QA harness for natural-language web and mobile regression suites.
• Highlights persistent test memory and self-healing execution.

Confidence Score: 5/5

The documentation-only change appears safe to merge.

The new resource entry follows the repository’s README format and placement conventions, with no blocking or non-blocking defect identified.

Important Files Changed

Filename	Overview
README.md	Adds one correctly formatted resource entry under the relevant evaluation and reliability section; no actionable issue identified.

_{Reviews (1): Last reviewed commit: "docs: add Agent QA eval harness" | Re-trigger Greptile}