Skip to content

Add wildcard matching options for message conversation items #21

Description

@rowan-stein

User Request

Add wildcard options to testllm for message conversation items. For the role field and content field, add an any checkbox. When enabled, the matching logic should skip equality comparison for that field.

Specification

1. Domain Types (src/lib/responses/types.ts)

Add optional wildcard boolean fields to MessageContent:

interface MessageContent {
  role: MessageRole;
  content: string;
  any_role?: boolean;    // NEW: when true, skip role comparison during matching
  any_content?: boolean; // NEW: when true, skip content comparison during matching
}

2. Matching Logic (src/lib/responses/matching.ts)

Update StoredMessageContentSchema to accept optional wildcard fields:

const StoredMessageContentSchema = z.object({
  role: z.enum(["user", "system", "developer", "assistant"]),
  content: z.string(),
  any_role: z.boolean().optional(),
  any_content: z.boolean().optional(),
});

Update compareItems() for the message branch — skip comparison when wildcard is enabled:

if (expected.type === "message" && actual.type === "message") {
  const content = expected.content;
  const roleMismatch = !content.any_role && content.role !== actual.role;
  const contentMismatch = !content.any_content && content.content !== actual.content;
  if (roleMismatch || contentMismatch) {
    return { ... "input_mismatch" };
  }
  return null;
}

3. Matching Tests (src/lib/responses/__tests__/matching.test.ts)

Add test cases:

  • Message with any_role: true matches any input role
  • Message with any_content: true matches any input content
  • Message with both any_role and any_content true matches any message
  • Message without wildcard flags still requires exact match (backwards compatible)

4. API Validation Schema (src/lib/schemas/test-items.ts)

Add wildcard fields to InputMessageContentSchema only (not OutputMessageContentSchema — wildcards don't apply to assistant output):

const InputMessageContentSchema = z.object({
  role: z.enum(["user", "system", "developer"]),
  content: z.string(),
  any_role: z.boolean().optional(),
  any_content: z.boolean().optional(),
});

5. UI Draft Types (src/components/test-item-editor/types.ts)

Add any_role?: boolean and any_content?: boolean to the message MessageContent type.

6. UI Form Fields (src/components/test-item-editor/item-content-fields.tsx)

For message-type items:

  • Add a checkbox labeled "Any" next to the Role select. When checked, disable the role dropdown.
  • Add a checkbox labeled "Any" next to the Content textarea. When checked, disable the textarea.
  • When checkbox is toggled on, set the corresponding any_* field to true on the content object.
  • When toggled off, set to false / remove the field.

7. UI Checkbox Component

Add shadcn/ui Checkbox component (src/components/ui/checkbox.tsx) if not already present. The project uses Radix UI (already installed) and shadcn/ui.

8. Read-only View (src/components/test-item-list.tsx)

Show a visual indicator (badge or label) when any_role or any_content is true on a message item.

9. E2E Tests

Add e2e test fixtures and test cases in e2e/ that verify wildcard matching works through the full responses API endpoint.

Notes

  • No database migration neededcontent is a JSONB column, new fields are absorbed automatically
  • Wildcards only apply to input message items (user/system/developer), not output (assistant)
  • Existing test data without wildcard fields continues to work (backwards compatible — undefined treated as false)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions