Skip to content

Commit de41606

Browse files
author
root
committed
Dynamic mode: iterated prompt for 100% GPT-5.2 solve rate
Prompt refinements through 4 test rounds (10 challenges each): - Round 1: 100% (but all same structure - modulo chains) - Round 2: 70% (letter counting, ambiguous geometry, factorial errors) - Round 3: 90% (percentage decimals, binary digit counting) - Round 4: 100% ✅ Forbidden list now blocks: - Character/letter counting (NEVER ask 'how many letters') - Digit counting in binary/decimal - Factorials above 6! - Percentage problems with non-integer answers - Ambiguous geometric terms - Word length inference (always TELL the length) Challenge types focus on: - Nested arithmetic with modulo - Base conversion + arithmetic - Sequence reasoning - Digit sums / digital roots - Multi-step chains (3-4 operations) Frontend: Toggle centered (display:flex + width:fit-content), agent mode hides all sections, description updated.
1 parent 2972ae4 commit de41606

1 file changed

Lines changed: 48 additions & 42 deletions

File tree

src/agentchallenge/dynamic.py

Lines changed: 48 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -76,50 +76,56 @@
7676
# 7. Aggressive answer normalization for comparison
7777
# 8. Explicit "ANSWER:" prefix extraction from verifier
7878

79-
GENERATE_PROMPT = """You are a challenge generator for AI agent authentication. Generate ONE unique AGENTIC-LEVEL challenge — it must require MULTIPLE reasoning steps that chain together.
79+
GENERATE_PROMPT = """You are a challenge generator for AI agent authentication. Generate ONE unique multi-step reasoning challenge.
8080
8181
RULES (strict):
82-
1. The answer MUST be a single number OR a short string (1-20 chars). Never a sentence.
83-
2. The challenge MUST have exactly ONE objectively correct answer — no ambiguity.
84-
3. The challenge MUST require 2-4 chained steps (not just one operation).
85-
4. The challenge MUST be solvable by pure reasoning — no trivia, pop culture, or world knowledge.
86-
5. Include "Reply with ONLY the answer, nothing else." at the end of the prompt.
87-
6. SHOW YOUR WORK: Before writing the JSON, solve the challenge yourself step by step to verify the answer is correct. Write your work FIRST, then the JSON on the last line.
88-
89-
REQUIRED: Multi-step challenges (pick one approach randomly):
90-
- Chained transforms: reverse a string, then remove vowels, then take first 3 chars
91-
- Base conversion chain: convert binary to decimal, add N, convert back
92-
- Multi-step math: compute X, then take remainder mod Y, then multiply by Z
93-
- Letter arithmetic: sum letter values (A=1..Z=26), then apply an operation
94-
- String extraction + transform: take first letter of each word, then reverse, then ROT13
95-
- Nested operations: ((X + Y) × Z) - W where X/Y/Z/W come from string properties
96-
- Interleave + transform: interleave two strings char by char, then reverse
97-
98-
FORBIDDEN (never generate):
99-
- Single-step challenges (must be 2+ steps)
100-
- Riddles, lateral thinking, "trick questions"
101-
- Challenges with multiple valid answers
102-
- Trivia or world knowledge
103-
- Full-sentence answers
104-
- Time/date/clock puzzles
105-
106-
EXAMPLES (DO NOT reuse — generate something novel):
107-
Working: "HELLO"→reversed→"OLLEH"→remove vowels→"LLH"
108-
{"prompt": "Reverse the string \"HELLO\", then remove all vowels from the result. Reply with ONLY the answer, nothing else.", "answer": "LLH"}
109-
110-
Working: binary 1010=10, 10+7=17, 17 in binary=10001
111-
{"prompt": "Convert binary 1010 to decimal, add 7 to it, then convert the result back to binary. Reply with ONLY the answer, nothing else.", "answer": "10001"}
112-
113-
Working: "Cat Dog Elk"→first letters→"CDE"→reversed→"EDC"→ROT13→each letter +13: E(4+13=17)=R, D(3+13=16)=Q, C(2+13=15)=P → "RQP"
114-
{"prompt": "Take the first letter of each word in \"Cat Dog Elk\", reverse the result, then apply ROT13. Reply with ONLY the answer, nothing else.", "answer": "RQP"}
115-
116-
Working: ((8+5)×3)-12 = (13×3)-12 = 39-12 = 27, 27 mod 7 = 6
117-
{"prompt": "Compute ((8 + 5) × 3) - 12, then find the remainder when divided by 7. Reply with ONLY the answer, nothing else.", "answer": "6"}
118-
119-
Working: "SPARK"→reversed→"KRAPS"→remove vowels (A)→"KRPS"
120-
{"prompt": "Reverse the string \"SPARK\", then remove all vowels from the result. Reply with ONLY the answer, nothing else.", "answer": "KRPS"}
121-
122-
Now generate a NOVEL challenge — different from the examples above. Be creative with the specific values and words you choose. Show your work first, then the JSON on the final line:"""
82+
1. The answer MUST be a single number (integer). Never a sentence, never a string.
83+
2. The challenge MUST have exactly ONE correct answer — no ambiguity.
84+
3. The challenge MUST require 2-4 chained mathematical or logical steps.
85+
4. The challenge MUST be solvable by pure arithmetic/logic — no trivia or world knowledge.
86+
5. End the prompt with "Reply with ONLY the answer, nothing else."
87+
6. SHOW YOUR WORK: Solve it yourself step by step FIRST, then write the JSON.
88+
89+
CHALLENGE TYPES (pick one randomly each time):
90+
- Nested arithmetic: ((A + B) × C) - D, then modulo E
91+
- Base conversion chain: binary→decimal, apply arithmetic, modulo
92+
- Sequence reasoning: given an explicit pattern, compute the Nth term
93+
- Property chains: "Word X has N letters (TELL them N), multiply by M, subtract K"
94+
- Modular arithmetic: compute expression, find remainder
95+
- Digit operations: sum of digits of a number, or digital root
96+
- Multi-step arithmetic: chain 3-4 operations (add, subtract, multiply, modulo)
97+
98+
FORBIDDEN:
99+
- Character-by-character string manipulation (reversing strings, removing vowels, ROT13, Caesar cipher)
100+
- Counting individual letters or characters in strings/phrases (NEVER ask "how many letters" or "count the letters")
101+
- Asking the solver to determine the length of any word (always TELL them the length instead)
102+
- Challenges requiring letter-level operations
103+
- Single-step problems (must chain 2+ operations)
104+
- Ambiguous geometric terms ("sides of a cube" — faces? edges?)
105+
- Ambiguous answers or multiple valid solutions
106+
- World knowledge or trivia questions
107+
- Factorials above 6! (LLMs miscompute large factorials)
108+
- Percentage/discount problems that produce non-integer answers (e.g. $55.08)
109+
- The answer MUST be an exact integer, never a decimal
110+
- Asking the solver to "count digits" in a binary or decimal number (models miscount)
111+
- Any task where the solver must count individual characters, letters, or digits
112+
113+
EXAMPLES (DO NOT reuse — generate novel challenges with different numbers):
114+
Working: (17 × 4) = 68, 68 + 23 = 91, 91 mod 13 = 0
115+
{"prompt": "Compute 17 × 4, add 23, then find the remainder when divided by 13. Reply with ONLY the answer, nothing else.", "answer": "0"}
116+
117+
Working: binary 110110 = 54, 54 - 19 = 35, 35 mod 8 = 3
118+
{"prompt": "Convert binary 110110 to decimal, subtract 19, then find the remainder when divided by 8. Reply with ONLY the answer, nothing else.", "answer": "3"}
119+
120+
Working: 5! = 120, 1+2+0=3, 3^2 = 9, 9-2 = 7
121+
{"prompt": "Compute 5 factorial, find the sum of its digits, square that sum, and subtract 2. Reply with ONLY the answer, nothing else.", "answer": "7"}
122+
123+
Working: 3! = 6, 6 × 7 = 42, sum of digits of 42 = 4+2 = 6
124+
{"prompt": "Compute 3 factorial, multiply by 7, then find the sum of the digits of the result. Reply with ONLY the answer, nothing else.", "answer": "6"}
125+
126+
IMPORTANT: Each challenge you generate MUST use a DIFFERENT structure than the examples above. Do NOT just change the numbers in a "compute X, add Y, modulo Z" template. Use genuinely different reasoning steps each time — mix base conversions, digit sums, factorials, word counting, sequence patterns, combinatorics, or property chains. Be creative.
127+
128+
Now generate ONE novel challenge. Different numbers, different structure. Show work first, JSON last:"""
123129

124130

125131
VERIFY_PROMPT = """Solve this challenge step by step. Show your work, then write your final answer on the LAST line prefixed with "ANSWER: ".

0 commit comments

Comments
 (0)