Skip to content

Log rejected attack plans in reports (hallucination transparency) #12

Description

@xhulz

Context

When the AI proposes an attack plan with hallucinated functions (e.g., functions that don't exist in the contract), Raze rejects the plan at validateAttackPlan(). But today those rejected plans disappear silently — they're not logged anywhere.

Community request: https://www.reddit.com/r/... — a dev asked whether Raze reports on hallucinated and impossible attack vectors.

What to do

  1. Capture rejected plans in validateAttackPlan() (reason + original plan)
  2. Add a rejectedPlans field to AttackPipelineResult and AttackSuiteResult
  3. Add a Rejected Plans section to the fuzz report:
    ## Rejected Plans
    
    | # | Contract | Functions | Reason |
    |---|---|---|---|
    | 1 | Counter | transferAll | Function not found in contract |
    
  4. Surface rejectedPlans in MCP tool responses so the AI can self-correct

Why it matters

  • Transparency — users see that Raze is actively filtering AI hallucinations
  • Data — helps the community understand what kinds of hallucinations AIs generate
  • Self-correction — the AI can retry with corrected function names

Files likely affected

  • src/core/orchestrator.ts — capture rejection reason
  • src/core/types.ts — add RejectedPlan type
  • src/core/reporter.ts — add rejected plans section
  • src/core/pipeline.ts / attackSuite.ts — propagate rejected plans

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions