User Request
Add wildcard options to testllm for message conversation items. For the role field and content field, add an any checkbox. When enabled, the matching logic should skip equality comparison for that field.
Specification
1. Domain Types (src/lib/responses/types.ts)
Add optional wildcard boolean fields to MessageContent:
interface MessageContent {
role: MessageRole;
content: string;
any_role?: boolean; // NEW: when true, skip role comparison during matching
any_content?: boolean; // NEW: when true, skip content comparison during matching
}
2. Matching Logic (src/lib/responses/matching.ts)
Update StoredMessageContentSchema to accept optional wildcard fields:
const StoredMessageContentSchema = z.object({
role: z.enum(["user", "system", "developer", "assistant"]),
content: z.string(),
any_role: z.boolean().optional(),
any_content: z.boolean().optional(),
});
Update compareItems() for the message branch — skip comparison when wildcard is enabled:
if (expected.type === "message" && actual.type === "message") {
const content = expected.content;
const roleMismatch = !content.any_role && content.role !== actual.role;
const contentMismatch = !content.any_content && content.content !== actual.content;
if (roleMismatch || contentMismatch) {
return { ... "input_mismatch" };
}
return null;
}
3. Matching Tests (src/lib/responses/__tests__/matching.test.ts)
Add test cases:
- Message with
any_role: true matches any input role
- Message with
any_content: true matches any input content
- Message with both
any_role and any_content true matches any message
- Message without wildcard flags still requires exact match (backwards compatible)
4. API Validation Schema (src/lib/schemas/test-items.ts)
Add wildcard fields to InputMessageContentSchema only (not OutputMessageContentSchema — wildcards don't apply to assistant output):
const InputMessageContentSchema = z.object({
role: z.enum(["user", "system", "developer"]),
content: z.string(),
any_role: z.boolean().optional(),
any_content: z.boolean().optional(),
});
5. UI Draft Types (src/components/test-item-editor/types.ts)
Add any_role?: boolean and any_content?: boolean to the message MessageContent type.
6. UI Form Fields (src/components/test-item-editor/item-content-fields.tsx)
For message-type items:
- Add a checkbox labeled "Any" next to the Role select. When checked, disable the role dropdown.
- Add a checkbox labeled "Any" next to the Content textarea. When checked, disable the textarea.
- When checkbox is toggled on, set the corresponding
any_* field to true on the content object.
- When toggled off, set to
false / remove the field.
7. UI Checkbox Component
Add shadcn/ui Checkbox component (src/components/ui/checkbox.tsx) if not already present. The project uses Radix UI (already installed) and shadcn/ui.
8. Read-only View (src/components/test-item-list.tsx)
Show a visual indicator (badge or label) when any_role or any_content is true on a message item.
9. E2E Tests
Add e2e test fixtures and test cases in e2e/ that verify wildcard matching works through the full responses API endpoint.
Notes
- No database migration needed —
content is a JSONB column, new fields are absorbed automatically
- Wildcards only apply to input message items (user/system/developer), not output (assistant)
- Existing test data without wildcard fields continues to work (backwards compatible —
undefined treated as false)
User Request
Add wildcard options to testllm for message conversation items. For the role field and content field, add an
anycheckbox. When enabled, the matching logic should skip equality comparison for that field.Specification
1. Domain Types (
src/lib/responses/types.ts)Add optional wildcard boolean fields to
MessageContent:2. Matching Logic (
src/lib/responses/matching.ts)Update
StoredMessageContentSchemato accept optional wildcard fields:Update
compareItems()for the message branch — skip comparison when wildcard is enabled:3. Matching Tests (
src/lib/responses/__tests__/matching.test.ts)Add test cases:
any_role: truematches any input roleany_content: truematches any input contentany_roleandany_contenttrue matches any message4. API Validation Schema (
src/lib/schemas/test-items.ts)Add wildcard fields to
InputMessageContentSchemaonly (notOutputMessageContentSchema— wildcards don't apply to assistant output):5. UI Draft Types (
src/components/test-item-editor/types.ts)Add
any_role?: booleanandany_content?: booleanto the messageMessageContenttype.6. UI Form Fields (
src/components/test-item-editor/item-content-fields.tsx)For message-type items:
any_*field totrueon the content object.false/ remove the field.7. UI Checkbox Component
Add shadcn/ui Checkbox component (
src/components/ui/checkbox.tsx) if not already present. The project uses Radix UI (already installed) and shadcn/ui.8. Read-only View (
src/components/test-item-list.tsx)Show a visual indicator (badge or label) when
any_roleorany_contentis true on a message item.9. E2E Tests
Add e2e test fixtures and test cases in
e2e/that verify wildcard matching works through the full responses API endpoint.Notes
contentis a JSONB column, new fields are absorbed automaticallyundefinedtreated asfalse)