Skip to content

report.md

Latest commit

 

History

History
249 lines (186 loc) · 13.5 KB

report.md

File metadata and controls

249 lines (186 loc) · 13.5 KB
Raw

Agents Shipgate Report

Project: simple-openai-api-agent Agent: api-refund-assistant Target: production_like

Release Decision

Decision: blocked Reason: 2 active findings block release.

Blockers (2):

  • HIGH SHIP-ACTION-EXTERNAL-COMMUNICATION-AUDIT-MISSING — send_customer_email has external communication capability without required controls
  • CRITICAL SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING — create_refund has financial write capability without required controls

Review items (13):

  • HIGH SHIP-AUTH-MISSING-SCOPE — send_customer_email lacks declared auth scopes
  • HIGH SHIP-AUTH-MISSING-SCOPE — create_refund lacks declared auth scopes
  • HIGH SHIP-SIDEFX-IDEMPOTENCY-MISSING — create_refund lacks idempotency evidence
  • HIGH SHIP-API-FUNCTION-SCHEMA-STRICTNESS — send_customer_email function schema is not strict enough
  • HIGH SHIP-API-FUNCTION-SCHEMA-STRICTNESS — create_refund function schema is not strict enough
  • MEDIUM SHIP-API-STRUCTURED-OUTPUT-READINESS — Response format schemas/refund_decision.schema.json is under-specified
  • MEDIUM SHIP-API-TIMEOUT-MISSING — OpenAI API flow lacks timeout metadata
  • HIGH SHIP-API-RETRY-WITHOUT-IDEMPOTENCY — send_customer_email may be retried without idempotency evidence
  • MEDIUM SHIP-API-TOOL-OUTPUT-SCHEMA-MISSING — create_refund lacks success/failure output modeling
  • HIGH SHIP-API-RETRY-WITHOUT-IDEMPOTENCY — create_refund may be retried without idempotency evidence
  • MEDIUM SHIP-API-TRACE-APPROVAL-MISSING — Trace sample shows create_refund without approval
  • HIGH SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING — send_customer_email is high-risk but has no owner
  • HIGH SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING — create_refund is high-risk but has no owner

Evidence coverage: static (2/2 catalog tools reachable; 2 semantic review concern(s); 0/2 actions pass-eligible; human review recommended)

Baseline delta: not enabled

Fail policy: ci_mode=advisory, fail_on=[none], new_findings_only=false, would_fail_ci=false (exit 0)

Summary

  • Critical: 1
  • High: 10
  • Medium: 4
  • Low: 0
  • Suppressed: 0
  • Status: Release blockers detected (legacy; see Release Decision above)

Top Findings

15 findings across 3 subjects, most urgent first.

  • create_refund [openai_api] (at tools/openai-tools.json#/tools/0) — BLOCKS RELEASE (1 critical, 5 high, 1 medium)
    • critical SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING (blocks release) — missing: approval.required, safeguards.audit_log, safeguards.idempotency
    • high SHIP-API-FUNCTION-SCHEMA-STRICTNESS — create_refund function schema is not strict enough
      • Make create_refund a strict function schema: object parameters, additionalProperties=false, complete required list, and bounded risky fields.
    • high SHIP-API-RETRY-WITHOUT-IDEMPOTENCY — create_refund may be retried without idempotency evidence
      • Add idempotency evidence for create_refund or avoid retrying this side effect.
    • high SHIP-AUTH-MISSING-SCOPE — create_refund lacks declared auth scopes
      • Declare operation-specific auth scopes for create_refund, or explicitly declare anonymous authority when the operation requires no credentials.
    • high SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING — create_refund is high-risk but has no owner
      • Declare an owner for each high-risk production tool in risk_overrides.tools.
    • … and 2 more findings for this subject
  • send_customer_email [openai_api] (at tools/openai-tools.json#/tools/1) — BLOCKS RELEASE (5 high)
    • high SHIP-ACTION-EXTERNAL-COMMUNICATION-AUDIT-MISSING (blocks release) — missing: safeguards.audit_log, confirmation.required
    • high SHIP-API-FUNCTION-SCHEMA-STRICTNESS — send_customer_email function schema is not strict enough
      • Make send_customer_email a strict function schema: object parameters, additionalProperties=false, complete required list, and bounded risky fields.
    • high SHIP-API-RETRY-WITHOUT-IDEMPOTENCY — send_customer_email may be retried without idempotency evidence
      • Add idempotency evidence for send_customer_email or avoid retrying this side effect.
    • high SHIP-AUTH-MISSING-SCOPE — send_customer_email lacks declared auth scopes
      • Declare operation-specific auth scopes for send_customer_email, or explicitly declare anonymous authority when the operation requires no credentials.
    • high SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING — send_customer_email is high-risk but has no owner
      • Declare an owner for each high-risk production tool in risk_overrides.tools.
  • api-refund-assistant (agent-wide) (at shipgate.yaml) — review (3 medium)
    • medium SHIP-API-STRUCTURED-OUTPUT-READINESS — Response format schemas/refund_decision.schema.json is under-specified
      • Tighten the structured output schema with enums, needs_review/refusal/error modeling, and declared critical fields.
    • medium SHIP-API-TIMEOUT-MISSING — OpenAI API flow lacks timeout metadata
      • Declare tool-call timeout metadata for high-risk OpenAI API flows.
    • medium SHIP-API-TRACE-APPROVAL-MISSING — Trace sample shows create_refund without approval
      • Require approval before calling create_refund.

Finding Provenance

Reviewer triage signal only. Provenance kind does not change severity, release decision, fingerprints, baselines, or CI exit codes.

Provenance kind Active findings
static_declaration 14
ast_extraction 0
keyword_heuristic 0
regex_heuristic 0
policy_pack 0
runtime_trace 1

Suppressed findings excluded: 0

Capability <-> Intent Diff

Agent intent:

  • prohibited_action: issue refund without approval (tags: financial_action)
  • prohibited_action: send customer email without confirmation (tags: external_write, customer_communication)
  • instruction_preview: You are a support refund assistant. You should only advise the support representative and prepare a draft response. Do not take action on the customer's account. (tags: financial_action)

Actual capabilities:

  • create_refund: capability=financial_action, risk=financial_action, write, control=missing
  • send_customer_email: capability=external_write, risk=customer_communication, external_write, write, control=missing

Policy/control gaps:

  • CRITICAL undetected_gap [create_refund]: create_refund has financial write capability without required controls. (at tools/openai-tools.json) Requires: Static review requires deterministic evidence for release gaps. Release implication: Human review is required to interpret this finding.
  • HIGH control_missing [create_refund]: create_refund function schema is not strict enough. (at tools/openai-tools.json) Requires: API function schemas must be strict enough for reliable tool calls. Release implication: The model may send ambiguous or overbroad tool arguments.
  • HIGH control_missing [create_refund]: create_refund is high-risk but has no owner. (at tools/openai-tools.json) Requires: Manifest metadata must match the active release surface. Release implication: Release review metadata is incomplete or stale.
  • HIGH control_missing [create_refund]: create_refund lacks idempotency evidence. (at tools/openai-tools.json) Requires: Risky write tools need idempotency evidence before retryable release. Release implication: Retries could duplicate financial, destructive, or external effects.
  • HIGH control_missing [send_customer_email]: send_customer_email function schema is not strict enough. (at tools/openai-tools.json) Requires: API function schemas must be strict enough for reliable tool calls. Release implication: The model may send ambiguous or overbroad tool arguments.
  • 11 more in report.json

Release implication:

  • Decision: blocked
  • 2 release-relevant finding(s) map to active release blockers; resolve required controls or remove the capability.

Next validation:

  • Tool schema boundary check: The tool accepts bounded structured inputs and returns structured outputs where needed.
  • High-risk tool validation case: A declared test or review scenario covers the high-risk tool path.
  • Retry behavior for risky write: Retries use idempotency evidence or the side effect is not retried.
  • Least-privilege scope review: Manifest and tool scopes match the narrow permissions needed for the release.
  • Approval gate for high-risk action: The run records human approval before the tool call and denies calls without approval.

Recommended Next Actions

  • Declare approval.required, safeguards.audit_log, and safeguards.idempotency for this financial write action.
  • Declare confirmation policy and safeguards.audit_log for this external communication action.
  • Make send_customer_email a strict function schema: object parameters, additionalProperties=false, complete required list, and bounded risky fields.
  • Make create_refund a strict function schema: object parameters, additionalProperties=false, complete required list, and bounded risky fields.
  • Add idempotency evidence for send_customer_email or avoid retrying this side effect.
  • Add idempotency evidence for create_refund or avoid retrying this side effect.
  • Declare operation-specific auth scopes for send_customer_email, or explicitly declare anonymous authority when the operation requires no credentials.
  • Declare operation-specific auth scopes for create_refund, or explicitly declare anonymous authority when the operation requires no credentials.

Control Pack

default — Shipgate default controls v1. Shipgate's built-in requirements: money, destruction, production operations, code execution, and outbound communication carry controls.

  • external communication requires confirmation policy and safeguards.audit_log — 1 action short
  • financial write requires approval.required, safeguards.audit_log, and safeguards.idempotency — 1 action short

Tool Surface Summary

  • Total tools: 2
  • High-risk tools: 2
  • Wildcard tools: 0
  • Missing descriptions: 0
  • Sources: openai_api=2

Action Surface Diff

  • Status: disabled - No action-surface comparison source was provided.
  • Base: none

Capability Runtime Evidence

  • Sources: 1
  • Trace rows: 1
  • Matched rows: 1
  • Unmatched rows: 0
  • Warnings: 0

Matched trace rows:

  • ctrace\_8669c1b40747c28a create_refund (openai_api_trace, tool_name)

  • Declared local trace artifacts are audit evidence only; no live trace collection or tool execution occurred.

  • Trace normalization retains only allowlisted scalar fields and discards prompts, messages, arguments, outputs, and payload bodies.

Tool Surface Diff

  • Status: disabled - No --diff-from report or v0.3 baseline snapshot was provided.
  • Base: none

OpenAI API Surface Summary

  • Prompt files: 1
  • Tool files: 1
  • Response formats: 1
  • Model config present: True
  • Test cases: 1
  • Trace samples: 1
  • Policy rule files: 1

Findings By Category

Action Surface

  • CRITICAL: SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING [create_refund] - create_refund has financial write capability without required controls
  • HIGH: SHIP-ACTION-EXTERNAL-COMMUNICATION-AUDIT-MISSING [send_customer_email] - send_customer_email has external communication capability without required controls

Api

  • HIGH: SHIP-API-FUNCTION-SCHEMA-STRICTNESS [create_refund] - create_refund function schema is not strict enough
  • HIGH: SHIP-API-FUNCTION-SCHEMA-STRICTNESS [send_customer_email] - send_customer_email function schema is not strict enough
  • HIGH: SHIP-API-RETRY-WITHOUT-IDEMPOTENCY [create_refund] - create_refund may be retried without idempotency evidence
  • HIGH: SHIP-API-RETRY-WITHOUT-IDEMPOTENCY [send_customer_email] - send_customer_email may be retried without idempotency evidence
  • MEDIUM: SHIP-API-STRUCTURED-OUTPUT-READINESS - Response format schemas/refund_decision.schema.json is under-specified
  • MEDIUM: SHIP-API-TIMEOUT-MISSING - OpenAI API flow lacks timeout metadata
  • MEDIUM: SHIP-API-TOOL-OUTPUT-SCHEMA-MISSING [create_refund] - create_refund lacks success/failure output modeling
  • MEDIUM: SHIP-API-TRACE-APPROVAL-MISSING - Trace sample shows create_refund without approval

Auth

  • HIGH: SHIP-AUTH-MISSING-SCOPE [create_refund] - create_refund lacks declared auth scopes
  • HIGH: SHIP-AUTH-MISSING-SCOPE [send_customer_email] - send_customer_email lacks declared auth scopes

Manifest

  • HIGH: SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING [create_refund] - create_refund is high-risk but has no owner
  • HIGH: SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING [send_customer_email] - send_customer_email is high-risk but has no owner

Side Effects

  • HIGH: SHIP-SIDEFX-IDEMPOTENCY-MISSING [create_refund] - create_refund lacks idempotency evidence

Agent Binding Surface

Status: declared Root agent: root Entry points: root Pass eligible: true Catalog partition: 2 reachable, 0 possible, 0 unbound

Appendix: Root-Reachable Tool Inventory

Tool Source Risk Tags Risk Confidence Auth Scopes Owner
send_customer_email openai_api customer_communication, external_write, write customer_communication=high, external_write=high, write=medium - -
create_refund openai_api financial_action, write financial_action=high, write=medium - -

Disclaimer

Agents Shipgate is an advisory tool: the deterministic merge gate for AI-generated agent capability changes, run as a local-first, static Tool-Use Readiness review. It does not certify agent safety or compliance. Findings are based on static configuration, declared policies, tool schemas, and optional SDK metadata. Runtime behavior, actual tool routing, and output interpretation are not verified.