Skip to content

EVAL-USAGE.md

Latest commit

 

History

History
262 lines (185 loc) · 9.48 KB

EVAL-USAGE.md

File metadata and controls

262 lines (185 loc) · 9.48 KB
Raw

Evaluate a Skill with comet eval

This document explains, from the user's point of view, how to evaluate a Skill in the current release. In normal usage, you do not need to understand pytest, task registry details, profiles, treatments, or Docker internals. The user-facing entry point is comet eval.

First understand where comet eval fits in Comet

/comet-any creates or optimizes a Skill, while comet eval evaluates whether that Skill can be discovered by the eval harness, run, and reported correctly. The current /comet-any flow also feeds eval results and project-level preference evidence into publish readiness: preferenceHash, the composition proposal, resolved Skill evidence, and eval evidence must all match the current draft.

Think about the flow like this:

/comet-any generates a Skill
  -> produces comet/eval.yaml
  -> comet eval --collect performs discovery precheck
  -> comet eval --html performs the real evaluation
  -> /comet-any or comet creator status/next reads the result and continues into readiness / review / publish / distribute

comet eval does not publish. Publishing is still handled by the Bundle backend behind /comet-any, surfaced to ordinary users as comet publish. Eval's job is to provide publish evidence.

For ordinary users, the recommended path remains:

/comet-any -> comet eval -> comet creator status/next -> comet publish review/approve/run -> comet publish distribute --preview -> comet publish distribute

The stable composed Skill Bundle required capability set is skills/scripts/rules/hooks/references, with scripts/rules/hooks as the required control plane. hooks/*.yaml only become active after comet publish distribute compiles them for the target platform.

Recommended path: evaluate a Skill generated by /comet-any

After /comet-any generates a Skill, first look for:

generated-skill/
  comet/
    eval.yaml

Then run these two commands:

comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html

The first command, --collect, only confirms whether tasks can be discovered. It is a low-cost precheck right after generation.

The second command, --html, performs the real evaluation and produces a browsable report. After it passes, /comet-any can use the result as publish evidence.

How eval results enter publish readiness

After /comet-any or the backend records eval results, they are merged into publish readiness. What the user needs to know is just:

  1. Results from comet eval become evidence for Publish readiness:.
  2. If current-hash eval evidence is missing, User next steps: must point to running comet eval before publish continues.

The usual sequence is:

comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json
comet publish review <name> --platform <reference-platform> --json

comet creator next prints only the single recommended user command; comet publish review must directly show Publish readiness:, User next steps:, Readiness:, Blockers:, Warnings:, and Evidence:.

Why run collect first

collect is the cheapest user-facing debugging entry point. It answers:

  • is the comet/eval.yaml path correct
  • can the eval harness read this manifest
  • can the tasks referenced by the manifest be discovered
  • are the current repo's eval dependency paths usable

It should not start a full model evaluation first, and it should not spend long-running cost before manifest and discovery work.

What --html prints

During execution, the CLI prints a set of execution facts:

  • Eval root: which eval/ root was actually used
  • Mode: collect or run
  • Target: whether the target is a manifest or local Skill directory
  • Experiment: the experiment id for this run
  • Profile: which profile the run used
  • Task: which task was run
  • Report path: where the report was written
  • Report config: the temporary report config used when --html is enabled

--html requests both markdown and HTML reports. Reports are typically written under:

eval/local/logs/experiments/<experiment-id>/summary.html

If CLI output shows the placeholder <experiment-id>, use the Experiment value printed in the same output block to resolve the final path.

How to read the report

Users do not need to read low-level logs line by line. Start with:

  • whether evaluation passed
  • whether failures come from harness, workflow, task, or model
  • whether failures are relevant to the Skill's goal
  • whether expected artifacts are missing
  • whether the problem is path, manifest, or environment related
  • whether token / cost / duration look abnormal

comet eval surfaces failure attribution so the report groups failures under harness, workflow, task, model, and related categories. That attribution helps decide whether the next step is fixing the Skill, the eval config, or the environment.

Reports also distinguish the raw set from the analysis set. The raw set keeps every run for auditability; the analysis set is the default source for headline metrics, pass@k/pass^k, cost, charts, and the verdict. excluded usually means API timeout, rate limiting, auth/network failure, Docker/container failure, or an outer runner timeout; these runs stay visible in the report but do not affect headline metrics. flagged means a harness or task assumption looks suspicious; the run remains in the analysis set but is called out as a risk. Real Skill, workflow, model, or validator failures remain included and are not filtered away just because they lower the score.

If the report says Insufficient clean data or Inconclusive due to data quality, rerun the affected task/treatment pair or inspect the environment before treating the verdict as final.

How /comet-any uses eval results

From the user's point of view, after comet eval finishes you can hand control back to /comet-any or run comet creator next <name> to see the single recommended next step. /comet-any merges eval evidence into readiness:

  • no eval evidence: cannot publish
  • eval failed: cannot publish
  • eval evidence points to an old hash: cannot publish
  • .comet/skill-preferences.yaml changed while in strict mode: cannot publish; confirm or regenerate first
  • eval passed and hash matches: continue into review / publish decisions

Users should not edit Bundle state manually, and they should not write report paths into internal JSON by hand. /comet-any records structured evidence through the Bundle backend.

How to evaluate when you only have a local Skill directory

If you do not yet have comet/eval.yaml and only have a local Skill directory, start with quick smoke:

comet eval ./my-skill --quick

This is appropriate for early validation:

  • whether the Skill directory can be read
  • whether the eval harness can inject it as a dynamic Skill
  • whether the generic smoke task runs

Current quick smoke uses:

generic-skill-smoke

This is only an early smoke path, not final publish evidence. Before publishing, it is still recommended to let /comet-any generate comet/eval.yaml and then use the manifest path.

When to choose manifest vs skill-path

The rule is simple:

  • if comet/eval.yaml exists: pass that file as the target
  • if you only have a local directory and are still in early debugging: pass the Skill directory with --quick
  • if the Skill was generated by /comet-any: pass its comet/eval.yaml
  • if you want publish readiness: pass its comet/eval.yaml

Do not treat a local Skill --quick smoke as the final publish evaluation.

What to do when it fails

If collect fails

Check first:

  • whether the manifest path is correct
  • whether comet/eval.yaml exists
  • whether the manifest's recommended task exists
  • whether you are in the Comet repo root or passed the right --project

If run fails

Look at failure attribution in the report first:

  • harness: usually eval harness, dependency, Docker, path, or environment issues
  • workflow: usually the Skill execution flow did not behave as intended
  • task: usually task definition, verification rule, or fixture issues
  • model: usually model behavior, tool use, or unstable output

If the HTML report cannot be found

First read the CLI's Experiment and Report path. If the path still contains <experiment-id>, use the real experiment id under:

eval/local/logs/experiments/

comet eval and comet skill check are different

The commands serve different roles.

comet eval is the shared eval-harness entry point for evaluating a Skill package or comet/eval.yaml.

comet skill check is the local Engine Run completion check used to decide whether a run or change meets the runtime checks in comet/checks.yaml.

If your question is "Can this Skill pass product evaluation?", use:

comet eval ./generated-skill/comet/eval.yaml --html

If your question is "Is this deterministic Skill Run missing artifacts or status?", then use:

comet skill check --change ./changes/demo --scope completion

What the user needs to remember

In practice, only remember these three points:

  1. If /comet-any generated the Skill, prefer comet/eval.yaml
  2. Run --collect first, then --html
  3. Eval results are publish readiness evidence, not the publish action itself

Recommended commands:

comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json