Skip to content

Semantic rules that depend on other rules' results #241

Open

Description

@mindsocket

Problem

Some semantic rules only make sense once the nodes they look at pass a more basic check. For example, a rule that judges whether sibling goals overlap gives noise when the goals are too vague to judge. A "detailed enough to judge" rule can flag the vague goals, but nothing lets one rule's result affect whether another rule runs or how its result counts.

Cascade gating does not cover this. Gating flows down the tree. A rule on a parent that judges its children runs even when those children fail a basic rule.

Options

No decision yet. The options sort onto two axes.

When a rule runs (decided before any judge call). This saves tokens, but adds configuration and ordering logic.

  • Gating that works in other directions, for example from children up to the parent rule that judges them.
  • Extend when so it can refer to other rules' results.
  • Run rules in layers or priority classes, where each layer adapts to the results of the one before.

How results combine (decided after the calls). This spends tokens up front, then reconciles.

  • Speculative fan-out: run every rule that might apply, then let other results override or discard some of them.
  • Expression rules that apply logic to the results of other rules.
  • Decision trees or declared dependencies between rules.

Baseline: define more rules, run them all, and report every result.

A node's rules that need the same context already share one judge input (src/judge/judge-node.ts). So when the extra rules share that context, fan-out costs extra output tokens, not extra requests.

Criteria

  • Token cost per space.
  • A user can explain why a rule did or did not report.
  • Configuration a schema author has to write.
  • It works across schemas, not only one.
  • A rule that does not run, or is overridden, is never shown as a pass.

Next step

Measure before choosing. Run the detail rules alongside the comparison rules in the evals. Count how often a comparison verdict on a node that fails a detail rule disagrees with its label, and what each option would have saved or changed. Related: #116.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions