|
76 | 76 | # 7. Aggressive answer normalization for comparison |
77 | 77 | # 8. Explicit "ANSWER:" prefix extraction from verifier |
78 | 78 |
|
79 | | -GENERATE_PROMPT = """You are a challenge generator for AI agent authentication. Generate ONE unique AGENTIC-LEVEL challenge — it must require MULTIPLE reasoning steps that chain together. |
| 79 | +GENERATE_PROMPT = """You are a challenge generator for AI agent authentication. Generate ONE unique multi-step reasoning challenge. |
80 | 80 |
|
81 | 81 | RULES (strict): |
82 | | -1. The answer MUST be a single number OR a short string (1-20 chars). Never a sentence. |
83 | | -2. The challenge MUST have exactly ONE objectively correct answer — no ambiguity. |
84 | | -3. The challenge MUST require 2-4 chained steps (not just one operation). |
85 | | -4. The challenge MUST be solvable by pure reasoning — no trivia, pop culture, or world knowledge. |
86 | | -5. Include "Reply with ONLY the answer, nothing else." at the end of the prompt. |
87 | | -6. SHOW YOUR WORK: Before writing the JSON, solve the challenge yourself step by step to verify the answer is correct. Write your work FIRST, then the JSON on the last line. |
88 | | -
|
89 | | -REQUIRED: Multi-step challenges (pick one approach randomly): |
90 | | -- Chained transforms: reverse a string, then remove vowels, then take first 3 chars |
91 | | -- Base conversion chain: convert binary to decimal, add N, convert back |
92 | | -- Multi-step math: compute X, then take remainder mod Y, then multiply by Z |
93 | | -- Letter arithmetic: sum letter values (A=1..Z=26), then apply an operation |
94 | | -- String extraction + transform: take first letter of each word, then reverse, then ROT13 |
95 | | -- Nested operations: ((X + Y) × Z) - W where X/Y/Z/W come from string properties |
96 | | -- Interleave + transform: interleave two strings char by char, then reverse |
97 | | -
|
98 | | -FORBIDDEN (never generate): |
99 | | -- Single-step challenges (must be 2+ steps) |
100 | | -- Riddles, lateral thinking, "trick questions" |
101 | | -- Challenges with multiple valid answers |
102 | | -- Trivia or world knowledge |
103 | | -- Full-sentence answers |
104 | | -- Time/date/clock puzzles |
105 | | -
|
106 | | -EXAMPLES (DO NOT reuse — generate something novel): |
107 | | -Working: "HELLO"→reversed→"OLLEH"→remove vowels→"LLH" |
108 | | -{"prompt": "Reverse the string \"HELLO\", then remove all vowels from the result. Reply with ONLY the answer, nothing else.", "answer": "LLH"} |
109 | | -
|
110 | | -Working: binary 1010=10, 10+7=17, 17 in binary=10001 |
111 | | -{"prompt": "Convert binary 1010 to decimal, add 7 to it, then convert the result back to binary. Reply with ONLY the answer, nothing else.", "answer": "10001"} |
112 | | -
|
113 | | -Working: "Cat Dog Elk"→first letters→"CDE"→reversed→"EDC"→ROT13→each letter +13: E(4+13=17)=R, D(3+13=16)=Q, C(2+13=15)=P → "RQP" |
114 | | -{"prompt": "Take the first letter of each word in \"Cat Dog Elk\", reverse the result, then apply ROT13. Reply with ONLY the answer, nothing else.", "answer": "RQP"} |
115 | | -
|
116 | | -Working: ((8+5)×3)-12 = (13×3)-12 = 39-12 = 27, 27 mod 7 = 6 |
117 | | -{"prompt": "Compute ((8 + 5) × 3) - 12, then find the remainder when divided by 7. Reply with ONLY the answer, nothing else.", "answer": "6"} |
118 | | -
|
119 | | -Working: "SPARK"→reversed→"KRAPS"→remove vowels (A)→"KRPS" |
120 | | -{"prompt": "Reverse the string \"SPARK\", then remove all vowels from the result. Reply with ONLY the answer, nothing else.", "answer": "KRPS"} |
121 | | -
|
122 | | -Now generate a NOVEL challenge — different from the examples above. Be creative with the specific values and words you choose. Show your work first, then the JSON on the final line:""" |
| 82 | +1. The answer MUST be a single number (integer). Never a sentence, never a string. |
| 83 | +2. The challenge MUST have exactly ONE correct answer — no ambiguity. |
| 84 | +3. The challenge MUST require 2-4 chained mathematical or logical steps. |
| 85 | +4. The challenge MUST be solvable by pure arithmetic/logic — no trivia or world knowledge. |
| 86 | +5. End the prompt with "Reply with ONLY the answer, nothing else." |
| 87 | +6. SHOW YOUR WORK: Solve it yourself step by step FIRST, then write the JSON. |
| 88 | +
|
| 89 | +CHALLENGE TYPES (pick one randomly each time): |
| 90 | +- Nested arithmetic: ((A + B) × C) - D, then modulo E |
| 91 | +- Base conversion chain: binary→decimal, apply arithmetic, modulo |
| 92 | +- Sequence reasoning: given an explicit pattern, compute the Nth term |
| 93 | +- Property chains: "Word X has N letters (TELL them N), multiply by M, subtract K" |
| 94 | +- Modular arithmetic: compute expression, find remainder |
| 95 | +- Digit operations: sum of digits of a number, or digital root |
| 96 | +- Multi-step arithmetic: chain 3-4 operations (add, subtract, multiply, modulo) |
| 97 | +
|
| 98 | +FORBIDDEN: |
| 99 | +- Character-by-character string manipulation (reversing strings, removing vowels, ROT13, Caesar cipher) |
| 100 | +- Counting individual letters or characters in strings/phrases (NEVER ask "how many letters" or "count the letters") |
| 101 | +- Asking the solver to determine the length of any word (always TELL them the length instead) |
| 102 | +- Challenges requiring letter-level operations |
| 103 | +- Single-step problems (must chain 2+ operations) |
| 104 | +- Ambiguous geometric terms ("sides of a cube" — faces? edges?) |
| 105 | +- Ambiguous answers or multiple valid solutions |
| 106 | +- World knowledge or trivia questions |
| 107 | +- Factorials above 6! (LLMs miscompute large factorials) |
| 108 | +- Percentage/discount problems that produce non-integer answers (e.g. $55.08) |
| 109 | +- The answer MUST be an exact integer, never a decimal |
| 110 | +- Asking the solver to "count digits" in a binary or decimal number (models miscount) |
| 111 | +- Any task where the solver must count individual characters, letters, or digits |
| 112 | +
|
| 113 | +EXAMPLES (DO NOT reuse — generate novel challenges with different numbers): |
| 114 | +Working: (17 × 4) = 68, 68 + 23 = 91, 91 mod 13 = 0 |
| 115 | +{"prompt": "Compute 17 × 4, add 23, then find the remainder when divided by 13. Reply with ONLY the answer, nothing else.", "answer": "0"} |
| 116 | +
|
| 117 | +Working: binary 110110 = 54, 54 - 19 = 35, 35 mod 8 = 3 |
| 118 | +{"prompt": "Convert binary 110110 to decimal, subtract 19, then find the remainder when divided by 8. Reply with ONLY the answer, nothing else.", "answer": "3"} |
| 119 | +
|
| 120 | +Working: 5! = 120, 1+2+0=3, 3^2 = 9, 9-2 = 7 |
| 121 | +{"prompt": "Compute 5 factorial, find the sum of its digits, square that sum, and subtract 2. Reply with ONLY the answer, nothing else.", "answer": "7"} |
| 122 | +
|
| 123 | +Working: 3! = 6, 6 × 7 = 42, sum of digits of 42 = 4+2 = 6 |
| 124 | +{"prompt": "Compute 3 factorial, multiply by 7, then find the sum of the digits of the result. Reply with ONLY the answer, nothing else.", "answer": "6"} |
| 125 | +
|
| 126 | +IMPORTANT: Each challenge you generate MUST use a DIFFERENT structure than the examples above. Do NOT just change the numbers in a "compute X, add Y, modulo Z" template. Use genuinely different reasoning steps each time — mix base conversions, digit sums, factorials, word counting, sequence patterns, combinatorics, or property chains. Be creative. |
| 127 | +
|
| 128 | +Now generate ONE novel challenge. Different numbers, different structure. Show work first, JSON last:""" |
123 | 129 |
|
124 | 130 |
|
125 | 131 | VERIFY_PROMPT = """Solve this challenge step by step. Show your work, then write your final answer on the LAST line prefixed with "ANSWER: ". |
|
0 commit comments