Skip to content

explain.md

Latest commit

 

History

History
223 lines (151 loc) · 49.8 KB

explain.md

File metadata and controls

223 lines (151 loc) · 49.8 KB
Raw

explain.md — how this system works and why

Purpose. This is not the README (that tells someone how to run the app). This doc explains why every piece is the way it is, so you can defend any schema decision or trade-off cold in the live review. The AI-usage policy says understanding matters more than output — this is where that understanding lives. Fill in each phase's section as you build it, while the reasoning is fresh.

0. The mental model (read this first)

There are two systems that barely know about each other:

  • The Forum — posts, comments, solutions. It is a normal CRUD app. Its only obligation to the engine is: whenever a user does something meaningful, emit an event.
  • The Engine — knows nothing about posts or comments. It knows about events, challenges, progress, and rewards. It receives events, evaluates them against data-driven challenge configs, updates progress, and disburses rewards.

The seam between them is a single event stream. This decoupling is the whole point of the assignment, and it's why the design below keeps insisting that the forum never calls the evaluator directly.

The data flow, end to end:

user action (comment)
  → forum handler writes the comment
  → AND emits an event  (POST /api/events, or the internal equivalent)
      → event stored (idempotent on event_id)  ── same DB txn ──▶  job enqueued
  → API returns 202 immediately (no evaluation on the request path)
                                   … later, asynchronously …
  → worker picks up the job
  → generic evaluator runs the user's active challenges against the config
  → progress rows updated
  → if a challenge just completed → reward disbursed (idempotently) into the ledger
  → frontend, which has been polling, sees progress move and the reward appear

Everything else is detail hanging off this spine.

1. The generic, data-driven evaluator (the core)

The requirement: an admin creates a challenge by POSTing a config; the engine evaluates all challenges from that config with no hardcoded per-type logic. If the domain layer ever branches on type, the design has failed.

The shape. A challenge row carries:

  • event_type — which event this challenge cares about (e.g. comment_posted).
  • type — how to interpret progress (count or streak).
  • rule_config (JSONB) — the parameters for that interpretation.
  • reward (JSONB) — what to grant on completion.

rule_config examples:

// count: do event_type N times in the window
{ "target": 3 }
// streak: do event_type on N consecutive days
{ "length": 5 }

Where type is allowed to be switched on — exactly one place: a strategy registry.

STRATEGIES: dict[str, ChallengeStrategy] = {
    "count":  CountStrategy(),
    "streak": StreakStrategy(),
}

Each strategy implements the same interface: given a user, a challenge, and the user's relevant events, return (current_value, target_value, is_complete). The evaluator loops over a user's active challenges, looks up STRATEGIES[challenge.type], and asks it to compute progress. Adding a "unique-days-with-2-events" challenge later = add one strategy, touch nothing else. That extensibility is the exact thing the live review will test ("now add a challenge type that…").

Why JSONB for rule_config and reward: the config is polymorphic by design — a count challenge and a streak challenge don't share a fixed column set. Columns would force nullable sprawl or a table per type; JSONB keeps the schema honest ("this is opaque config the strategy interprets") and validates at the application edge (Pydantic per type) rather than the DB. Trade-off: no DB-level constraints on config contents — mitigated by strict validation on the admin write path.

explain.md note (fill when Phase 4 lands): paste your ChallengeStrategy interface and the two implementations, and note any defaulting you did for missing config fields.

2. Idempotency — it lives in two places

People conflate these; keeping them distinct is a fast way to look sharp in review.

Place #1 — event ingestion. The client generates event_id. Re-submitting the same event_id must return the original response without reprocessing. So:

  • events.event_id has a unique constraint.
  • On insert conflict, we don't just "skip" — we fetch and return the stored original response. (Store the response, or enough to reconstruct it, at first processing.)
  • Why it matters: networks retry. A double-submitted "comment_posted" must not count twice toward a challenge.

Place #2 — reward disbursal. A challenge rewards a user at most once per qualifying completion. So:

  • The ledger has a unique constraint on (user_id, challenge_id, completion_key).
  • Disbursal is an insert guarded by that constraint (insert-or-ignore / catch-unique). If the row exists, the reward was already given — do nothing.
  • completion_key encodes which completion: for a one-shot challenge it's constant; for a weekly challenge it's the ISO week (so it can pay out once per week and reset Monday — see §4).

Together: event idempotency stops double-counting input; reward idempotency stops double-paying output. The worker can safely run a job twice (see §3) precisely because both guards exist.

explain.md note (fill when Phases 3 & 5 land): paste the two unique constraints and the exact insert-on-conflict code.

3. The async job — why not the obvious thing

The requirement: evaluation runs asynchronously via a background job; ingestion returns 202 immediately; document the choice.

The trap: FastAPI BackgroundTasks. It runs in the same process, has no durability (a crash between 202 and evaluation loses the work), no retries, and breaks with >1 worker. It demos fine and fails the review question.

The choice here: a Postgres-backed outbox / job table, polled by a worker.

  • When an event is ingested, in the same transaction we insert the event row and a job row (status=pending). Same-txn means we never have an event with no job or a job with no event — this sidesteps the dual-write problem you'd get by publishing to an external queue after committing.
  • A worker loop selects pending jobs (FOR UPDATE SKIP LOCKED so multiple workers don't collide), evaluates, marks done. Failures → retry with backoff / failed after N attempts.
  • Delivery is at-least-once. Combined with the idempotent progress/reward writes from §2, the effective semantics are exactly-once. That sentence is the one to say out loud in review.

Why this over Celery/RQ + Redis: those are perfectly good and show queue fluency, but they add infra to run and reason about. For a 5-day, single-reviewer submission, "it's just Postgres, here's the table, here's the worker loop" is the most explainable option — and explainability is the graded axis. (If you'd rather show Redis/Celery, the trade-off note flips: more infra, but battle-tested retries/scheduling out of the box.)

explain.md note (fill when Phase 4 lands): paste the job table schema, the SKIP LOCKED select, and your retry policy.

4. Time — streaks and the Monday reset

Two requirements are secretly the same problem: "N consecutive days" and "resets every Monday" are both undefined until you pick a timezone. Decide once; apply everywhere.

Decision (default): a fixed forum timezone, set as a single config constant (default UTC). Rationale: streak-day bucketing and the weekly window must agree; a fixed TZ makes both trivially testable and removes per-user ambiguity. The alternative — per-user local time — is more "correct" for a global forum but multiplies edge cases (DST, travel) and is exactly the kind of extension they might ask you to add live. Keeping it a single constant means that extension is a localized change, and you can say so.

Streak-day bucketing: map each qualifying event's timestamp to a day in the forum TZ, take the distinct set of days (two comments same day = one day), sort, and find consecutive runs.

  • Current streak = the run ending today (define your grace rule: does a gap yesterday break it? Write down the choice).
  • Best streak = the longest run ever.

Weekly window: "this week" = from Monday 00:00 forum-TZ to now. Progress for the weekly challenge counts events in that window; the reward's completion_key is the ISO week so a new week is a fresh completion (§2). The widget "resets Monday" for free because the window and the key both roll over.

explain.md note (fill when Phases 4–5 land): state your chosen TZ, your grace rule for current streak, and paste the day-bucketing function.

5. The event boundary — one road in

Forum handlers emit events; they must not evaluate. Concretely: POST /posts writes the post and produces a post_created event through the same ingestion path that the public POST /api/events uses. One code path means one place for idempotency, one place jobs get enqueued, and no way for a forum action to secretly bypass the engine.

Why the public /events endpoint exists at all alongside server-side emission: it's the engine's contract with any producer, and it's what makes the engine genuinely decoupled/reusable. Document that forum-originated events and externally-posted events converge on the same handler.

explain.md note: describe how your forum handlers call into ingestion (shared function vs internal HTTP) and why.

6. Frontend — where the required behaviours live

Quick map of each graded behaviour to where it should live, so nothing gets dropped:

  • Optimistic UI (post + comment): TanStack Query onMutate writes the cache immediately, onError rolls back to the snapshot and toasts. Reviewers will force a failure — make rollback real, not cosmetic.
  • Polling (Challenges page + weekly widget): refetchInterval. Interval choice: ~30s (the wireframes' own note). Reasoning to document: evaluation is async and human-scale — sub-10s hammers the API for progress that changes on the order of user actions; 30s keeps the widget feeling live without wasteful load. State it explicitly.
  • Data-viz: streak heatmap or progress rings via Recharts/D3. A shadcn/Tailwind bar is explicitly disqualified.
  • Loading: skeletons, never spinners, on every fetch surface.
  • Error boundaries: every fetch section degrades to a visible "couldn't load · retry" fallback; the page stays usable.
  • URL state: feed sort/page in the querystring — shareable, and back-navigation restores it. Use the router + useSearchParams.
  • Custom hook: at least one named hook, single responsibility. Good candidates: useWeeklyChallenge() (encapsulates the polling + reset math) or useFeedParams() (URL ↔ filter state).

explain.md note (fill across Phases 6–8): name your custom hook and its single responsibility in one sentence; note your final polling interval.

7. Per-phase build log

Fill one entry per phase as it lands. Keep each to: what I built, the one decision I'd be asked about, anything I'd do differently.

Phase 0 — Foundations

  • Built:
  • Key decision (error/response envelope, TZ constant):
  • Notes:

Phase 1 — Auth & roles

  • Built: users table (id UUID pk, email unique+indexed, password_hash, role string with a CHECK (role in ('user','admin')) constraint, created_at) via Alembic migration c1cf4340c3f3; POST /api/auth/register, POST /api/auth/login, GET /api/auth/me; app/core/security.py (bcrypt hashing, PyJWT encode/decode); app/api/deps.py with get_current_user/require_admin exposed as reusable Annotated aliases (CurrentUser, AdminUser).
  • Key decision (JWT vs session, role gating): JWT, stateless — no session table, no server-side revocation list needed for a 5-day assignment scope. The token payload only carries sub (user id) + iat/exp, deliberately not a role claim: require_admin re-reads role from the DB on every request via get_current_user, so a role change (e.g. promoting a user to admin) takes effect on the user's very next request instead of waiting for a fresh login/token. The trade-off is one extra DB read per authenticated request — acceptable at this scale, and it closes the "stale role in an old token" class of bug for free.
  • Notes: Email is lowercased at the schema layer (RegisterRequest/LoginRequest validators) before it ever reaches a query or an insert, so uniqueness and login lookups are case-insensitive without needing a DB-side citext extension. There's no self-serve "become an admin" endpoint by design — admin promotion is an out-of-band DB write (a seed script arrives in Phase 9); Phase 1's tests promote a user directly via UPDATE users SET role = 'admin' to exercise require_admin. Since Phase 4's admin challenge CRUD is the first real consumer of require_admin, Phase 1 adds a test-only probe route (GET /api/auth/_admin_probe, mounted only inside tests/test_auth.py) purely to prove the dependency rejects a user token with 403 and accepts an admin token with 200 — this is the "admin-only test route" the plan called for, not a shipped product endpoint.

Phase 2 — Forum domain

  • Built: posts (title, body, denormalized comment_count/view_count, solution_comment_id) and comments tables, plus the events table one phase early (migration d3a1f9b2c7e4) — event_id UUID primary key doubles as invariant #4's uniqueness guarantee. app/services/events.py::ingest_event() is the one ingestion function (invariant #2); POST /posts, GET /posts/{id} (detail view), POST /posts/{id}/comments, POST /posts/{id}/solution (owner-only, 403 otherwise) each write their domain row and call ingest_event in the same session before a single commit() — so the domain write and the event write are one DB transaction, not two. GET /posts supports sort=latest|trending + page/limit.
  • Key decision (trending formula, emit-not-evaluate): trending_score = (comment_count*3 + view_count) / power(age_hours + 2, 1.5) — comments outweigh views 3:1 as the stronger engagement signal, and the +2 offset keeps a just-created post's score finite (no divide-by-~0 spike) while still decaying fast with age. Computed at query time from denormalized counters rather than stored, since "now" keeps moving. Forum handlers never call the evaluator or write progress/reward rows directly — they only ever call ingest_event, which for now (pre-Phase-3) just inserts an Event row; Phase 3 will extend that same function with event_id conflict-replay and the transactional job-row enqueue, not replace it, so the "one road in" invariant holds before and after.
  • Notes: events and its unique event_id PK were built now instead of deferred to Phase 3 so the forum handlers had something real to call — the plan's Phase 3 scope (idempotent replay of the stored response, and the outbox job row) is intentionally not implemented yet. Mark-solution validates that the comment actually belongs to the post (cross-post comment_id → 404, not silently accepted); re-marking a different comment as solution is allowed (last call wins), each mark still emits its own solution_marked event — de-duplicating reward eligibility for that is an engine-side (Phase 4/5) concern, not the forum's. posts.solution_comment_id and comments.post_id form a genuine FK cycle; the migration creates posts with a bare (FK-less) solution_comment_id column, creates comments referencing posts, then adds the posts → comments FK constraint afterward.

Phase 3 — Event ingestion & idempotency

  • Built: jobs table (migration d2d29db3cc1e) — id, event_id (FK → events.event_id, unique), status (pending|processing|done|failed, CHECK-constrained, mirroring the UserRole/enum.StrEnum pattern from Phase 1), attempts, last_error, created_at/updated_at. ingest_event() (app/services/events.py) is extended in place, not replaced: it now returns (Event, is_new: bool), wraps the event insert + job insert in a single SAVEPOINT (db.begin_nested()), and on an event_id conflict fetches and returns the existing row instead of reprocessing. POST /api/events (app/api/events.py) is a thin wrapper — auth via CurrentUser, calls ingest_event, commits, returns 202 with an EventOut built straight from the (possibly pre-existing) Event row.
  • Key decision (stored-response replay, transactional enqueue): I didn't add a separate "stored response" column. The POST /api/events response body is a pure function of the Event row's own columns (event_id, event_type, payload, created_at) — so replaying means fetching the original row and serializing it again, which is byte-identical by construction, no extra storage needed. The job enqueue lives inside ingest_event itself (not just the public endpoint) because forum handlers need jobs enqueued too — a comment_posted event from POST /posts/{id}/comments has to reach the evaluator exactly like one from the public endpoint. Putting both the event insert and the job insert in one SAVEPOINT means: a genuinely new event always gets exactly one job, and a replayed event never gets a second one, without ever risking a mid-request rollback of whatever the calling forum handler had already flushed (the Post/Comment row) in the same outer transaction — that's what the savepoint isolates.
  • Notes: On conflict, resubmission's event_type/payload are silently ignored in favor of the stored original — that's what an idempotency-key contract means, not "detect and reject mismatches" (this could be added if the review pushes on it, but isn't in the plan). The assert-vs-explicit-raise call: on IntegrityError from the savepoint, I only treat it as a replay if a row with that event_id genuinely exists; anything else (e.g. a broken user_id FK) re-raises so an unrelated bug doesn't get silently swallowed as if it were an idempotent replay. Forum handlers (Phase 2) didn't need any changes — they already call ingest_event and never use its return value, so widening its return type to a tuple was invisible to them; they now transitively get a Job row per event, which is what makes them Phase-4-ready. Could not exercise alembic upgrade head / pytest in this session (no reachable Postgres in the sandbox — docker unavailable, native Windows Postgres service present but credentials unknown); verified via ruff + mypy --strict only. Run alembic upgrade head and pytest locally before relying on this phase.

Phase 4 — Challenge engine + async job

  • Built: challenges table (migration f6a2c8d4e1b3) — name, description, type, event_type, rule_config/reward (JSONB), status (CHECK-constrained draft|active|expired|archived), start_at/end_at. progress table alongside it — (user_id, challenge_id) unique, current_value/target_value/is_complete/completed_at. app/services/strategies.py holds the registry (STRATEGIES: dict[str, ChallengeStrategy]) plus CountStrategy ({"target": N}) and StreakStrategy ({"length": N}), each with validate_config() (schema-time) and evaluate() (worker-time). app/services/evaluator.py::evaluate_event() is the generic loop: find active challenges for event.event_type, look up STRATEGIES[challenge.type], upsert the result into progress via Postgres INSERT ... ON CONFLICT DO UPDATE. app/worker.py is the durable poller (claim_and_process_one does the SELECT ... FOR UPDATE SKIP LOCKED claim + evaluate + mark done/failed; run_forever loops it with a 1s backoff when the queue is empty). Admin CRUD lives in app/api/admin_challenges.py (POST/GET/PATCH /api/admin/challenges, DELETE archives rather than deletes) — added as its own worker service in docker-compose.yml running python -m app.worker alongside backend.
  • Key decision (strategy registry, outbox worker): Neither evaluate_event nor the worker ever branches on challenge.type — even the admin-write-path validation doesn't hardcode Literal["count","streak"]; app/schemas/challenges.py::validate_challenge_config() looks the type up in STRATEGIES itself and delegates to that strategy's own validate_config(). So adding a third strategy is genuinely one file (implement ChallengeStrategy, add one registry line) — the schema, the evaluator, and the worker are all unaware anything changed. The bigger decision: progress is a materialized cache the worker recomputes wholesale on every run, not a counter it increments. CountStrategy.evaluate() re-counts matching events in the window from scratch; StreakStrategy.evaluate() re-derives the whole day-set and re-runs the streak scan. This is what makes "the worker can run a job twice safely" (invariant #5) true for progress writes specifically, not just reward disbursal (Phase 5) — there's no "+1" step to accidentally double, so replaying evaluate_event for the same event is a no-op by construction (test_evaluate_event_is_idempotent_when_run_twice). The trade-off is a full re-scan of the user's matching events on every evaluation instead of an O(1) increment — acceptable at this scale, and it's the difference between "idempotent because we recompute truth" and "idempotent because we hope a lock held."
  • Notes: Streak grace rule (documented as promised in §4): the current streak is the run ending on the most recent active day, as long as that day is today or yesterday — a day with no event yet today doesn't break a streak that ran through yesterday, but a full skipped day does. Implemented as a pure function, compute_streak(days: set[date], today: date) -> tuple[current, best], in app/services/strategies.py, tested in isolation in tests/test_strategies.py without touching the DB. Retry policy: job.attempts increments on any exception from evaluate_event; below MAX_ATTEMPTS = 5 the job goes back to pending (picked up on the worker's next poll — a poll-interval's worth of backoff, not exponential); at MAX_ATTEMPTS it's marked failed permanently. Gotcha worth flagging: zoneinfo.ZoneInfo("UTC") raises ZoneInfoNotFoundError on both Windows and slim Linux Docker images (python:3.12-slim ships no IANA tzdata) — added tzdata as an explicit dependency rather than relying on the OS. Discovered this by actually trying to instantiate ZoneInfo("UTC") in this sandbox before assuming stdlib zoneinfo was enough; would have silently 500'd every streak evaluation in the built Docker image otherwise. RewardConfig ({"type": str, "amount": int > 0}) is intentionally generic and unvalidated beyond shape — Phase 5 owns what "disbursing" a reward means. Could not exercise alembic upgrade head / pytest against a live DB in this session (same constraint as Phase 3 — no reachable Postgres in the sandbox); verified via ruff, mypy --strict, and by exercising compute_streak/validate_config directly as plain Python outside the test harness. Run alembic upgrade head and pytest locally before relying on this phase.

Phase 5 — Rewards + progress/streaks

  • Built: rewards table (migration a7f3e91c2d05) — user_id, reward_type, amount, source_challenge_id, completion_key, created_at, unique (user_id, source_challenge_id, completion_key). Same migration adds challenges.period (one_time|weekly, CHECK-constrained, default one_time) — a field orthogonal to type that Phase 4 didn't need yet. app/services/rewards.py::disburse_reward() is a plain INSERT ... ON CONFLICT DO NOTHING against that constraint, called from app/services/evaluator.py::evaluate_event() in the same transaction as the progress upsert, whenever a strategy reports is_complete. Read endpoints: GET /api/challenges and GET /api/challenges/weekly (app/api/challenges.py, outer-joins challenges to the caller's own progress row), GET /api/users/me/progress, GET /api/users/me/streaks, GET /api/users/me/rewards (paginated, same page/limit/meta shape as GET /posts) — all in app/api/users.py.
  • Key decision (completion_key, ledger constraint): completion_key is "once" for a one_time challenge (any later completion collides with the same key, so the unique constraint blocks a second payout forever) and the ISO week ("2026-W27") for a weekly one (a new week is a new key, so it's a fresh completion each Monday). Both app/services/rewards.py::completion_key() and app/services/strategies.py::effective_window() are pure functions taking now/tz as explicit arguments rather than reading the wall clock internally — that's what makes the weekly rollover unit-testable (test_completion_key_rolls_over_by_iso_week_for_weekly_challenges) without freezing time. period deliberately isn't folded into type: invariant #3 ("no branching on type outside the strategy registry") only restricts type, and a weekly count challenge and a one-time count challenge use the same CountStrategy — only the event window it counts within differs. So effective_window() clips the strategy's [start_at, end_at] to [max(start_at, this-week-Monday), end_at] when period == weekly, and both CountStrategy and StreakStrategy call it identically; adding period touched zero if type == branches.
  • Notes: Disbursal is called unconditionally whenever result.is_complete is True — not just "on the transition from incomplete to complete" — because the unique constraint (not an application-level check) is what actually prevents a double payout; this mirrors invariant #5's progress-recompute pattern (recompute-and-let-the-constraint-absorb-it, not detect-and-guard). GET /users/me/streaks needed best_streak, which the progress cache doesn't store (it only keeps the length-capped current_value needed for completion) — rather than add a column, compute_user_streak() recomputes (current, best) directly from source events, reusing the same _fetch_qualifying_days()/compute_streak() machinery StreakStrategy.evaluate() uses internally. The endpoint selects which challenges are "streak-shaped" via STREAK_CHALLENGE_TYPES (frozenset derived from isinstance(STRATEGIES[name], StreakStrategy)), not a hardcoded Challenge.type == "streak" literal, so a second streak-like strategy would show up there for free. Same DB-availability constraint as Phases 3–4: no reachable Postgres in this sandbox (docker absent; the native Windows Postgres service is running but its credentials aren't known here), so alembic upgrade head and pytest were not run — verified via ruff and mypy --strict only. Run both locally before relying on this phase.

Phase 6 — Frontend foundation

  • Built: Auth flow (app/(auth)/login, .../register) backed by lib/auth-context.tsx's AuthProvider/useAuth, storing {token, user} in localStorage and re-derived through useSyncExternalStore (not an effect+setState — see the key decision below). lib/api.ts is the one fetch wrapper: unwraps the {data, meta} envelope, throws a typed ApiError on non-2xx (mirroring error.code/message/details/request_id), and dispatches a cre:unauthorized window event on any 401 so AuthProvider can drop the session without every caller handling it. Shell A lives in components/shell/{sidebar,app-shell}.tsx, mounted once by app/(app)/layout.tsx — a client-side auth gate (skeleton while status === "loading", redirect to /login otherwise) wrapping all five route placeholders (feed, posts/new, posts/[id], challenges, profile). The weekly widget (components/weekly-challenge-widget.tsx) sits in the shell's right rail behind hooks/use-weekly-challenge.ts and a general-purpose ErrorBoundary (components/error-boundary.tsx). Primitives added via shadcn add: skeleton, card, avatar, dropdown-menu, input, label, badge, separator, sonner.
  • Key decision (polling interval, custom hook): Polling — 30s (WEEKLY_POLL_INTERVAL_MS in lib/config.ts), because evaluation is async and human-scale; anything sub-10s would hammer the API for progress that only moves on the order of user actions. Custom hook — useWeeklyChallenge(token), single responsibility: encapsulate the polling fetch of /challenges/weekly so WeeklyChallengeWidget stays purely presentational (loading skeleton / error+retry / data, nothing else). Auth storage — the JWT lives in localStorage, read via useSyncExternalStore rather than useEffect + setState. The naive effect version (useEffect(() => { setSession(read()) }, [])) has two problems: (1) it's exactly what the newer react-hooks/set-state-in-effect ESLint rule flags — reading an external store and pushing it into state via an effect is a textbook cascading-render case; (2) it causes a real one-tick flash: the server always renders "logged out" (no window), so an already logged-in user visiting /feed would flash-redirect to /login before the effect fires and corrects it. useSyncExternalStore's third argument (getServerSnapshot) makes "don't know yet" an explicit status === "loading" value instead of quietly aliasing it to unauthenticated — that's what the auth gate in app/(app)/layout.tsx checks before ever deciding to redirect.
  • Notes: Route protection is deliberately client-side only, not Next middleware — middleware runs on the edge before any client JS, and the backend is a pure bearer-token API (no cookies for it to read), so there is no server-visible signal to gate on. The trade-off: a hard refresh of a protected route always paints the loading skeleton for one tick before the real shell (or a redirect) appears — accepted for a 5-day scope; the anticipated fix if this became a real product would be to move the token into an httpOnly cookie and add matching Next middleware. apiRequest intentionally has no retry/backoff of its own — that's TanStack Query's job (providers.tsx sets retry: 1 globally); the wrapper's only responsibilities are the envelope shape and the 401 event. Verified end-to-end with a Playwright smoke pass against next dev (no backend/Postgres available in this sandbox, same constraint as Phases 3–5): root / redirects to /login when logged out, /feed redirects to /login when hit directly without a session, and — after seeding a fake session into localStorage — the shell renders (sidebar nav with active-route highlighting, avatar/email, logout), the weekly widget shows its skeleton then its error+retry fallback (since there's no live backend to answer /challenges/weekly), and it persists unremounted across a /feed → /profile navigation. Zero console errors beyond the expected ERR_CONNECTION_REFUSED from the absent backend. Run npm run typecheck / npm run lint / npm run build locally before relying on this phase against a real backend.

Phase 7 — Core pages

  • Built: lib/posts-api.ts (types + fetch functions for list/detail/create/comment/mark-solution, mirroring app/schemas/posts.py field-for-field). lib/api.ts gained apiRequestEnvelope (returns the full {data, meta} envelope) with apiRequest now a thin wrapper around it — needed because the feed is the first consumer that cares about meta.total_pages, not just data. Two new hooks: hooks/use-feed-params.ts (sort/page ↔ URL querystring, replacing useState) and hooks/use-create-post.ts (the optimistic-create mutation, kept separate from the page component since it's invoked from /posts/new but writes into the feed's cache — a cross-page concern that didn't belong inlined in either page). Feed (app/(app)/feed/page.tsx): sort tabs + pagination driven by useFeedParams, skeleton cards while loading, error+retry, wrapped in <Suspense> (required by useSearchParams in a static client page) and ErrorBoundary. Create post (.../posts/new/page.tsx): plain controlled form (matching the Phase 6 login/register pattern, no form library) calling useCreatePost(). Post detail (.../posts/[id]/page.tsx): converted from an async Server Component to a Client Component using React's use() to unwrap the still-Promise-typed params (needed client-side hooks for auth/query/mutations); nested comment list, an optimistic comment-post mutation, and an owner-only "Mark as solution" mutation, both following the same optimistic-cache-write/rollback/toast shape as the post-creation mutation. Added components/ui/textarea.tsx (plain <textarea> styled like Input — @base-ui/react has no textarea primitive) and formatRelativeTime() in lib/utils.ts.
  • Key decision (optimistic rollback, URL state): Rollback — useCreatePost's onMutate snapshots the feed's ["posts","list",{sort:"latest",page:1}] query, prepends an optimistic PostSummary with a optimistic-<uuid> id, then calls router.push("/feed") immediately — the user is looking at the feed with their new post already in it before the request has even resolved. On failure, onError restores the exact snapshot (the optimistic post disappears) and fires a sonner toast; onSettled invalidates the query either way so a real success reconciles with the server's actual row. This only fires the mutation's own onMutate/onError/onSuccess because they're declared on useMutation itself (not passed to .mutate()) — TanStack Query keeps those attached to the mutation, not the component, so they still run after the triggering page has unmounted from the router.push. The post-detail comment mutation and mark-solution mutation reuse the identical shape (snapshot → optimistic write → rollback-on-error → invalidate-on-settle) against ["posts","detail",id] instead. URL state — useFeedParams reads sort/page from useSearchParams() and writes via router.push with a rebuilt URLSearchParams, never local useState; this is what makes a /feed?sort=trending&page=2 link land on the right tab/page on a cold load, survive a reload, and have the browser back button restore the exact prior filter state (verified in the Playwright smoke pass below) — state that lived in useState would lose all three.
  • Notes: apiRequestEnvelope exists because apiRequest alone (Phase 6) discards meta, which every other Phase-6 caller didn't need but paginated feed does — extending the existing function's return shape rather than adding a parallel fetch path. Comments are a flat list under a post (the schema has no parent_comment_id), so "nested comments" here means comments rendered nested within the post detail view, not a reply tree — matches what app/schemas/posts.py::PostDetailOut actually returns. Mark-solution intentionally has no "unmark" affordance in the UI (mirrors the backend, which only supports re-marking a different comment, not clearing it). Verified end-to-end with a Playwright smoke pass against next dev (no backend/Postgres available in this sandbox, same constraint as Phases 3–6): seeded a fake session into localStorage, then confirmed — feed shows skeleton rows then a visible error+retry (no live backend to answer /posts); sort-tab clicks update the URL even mid-error; a direct load of /feed?sort=trending&page=2 renders with Trending active, survives a reload, and going back after clicking Latest restores sort=trending&page=2 exactly; publishing a post from /posts/new navigates to /feed with the optimistic post visible immediately, shows an error toast around ~2.5s later when the (absent) backend request fails, and the optimistic post is gone from the feed by the time the toast appears — the rollback is real, not cosmetic; post detail shows skeleton then error+retry the same way. Zero console errors beyond the expected ERR_CONNECTION_REFUSED from the absent backend. Run npm run typecheck / npm run lint / npm run build (all pass) and a manual pass against a real backend before relying on this phase.

Phase 8 — Challenges/Progress + Profile

  • Built: Backend — one new read endpoint, GET /users/me/rewards/summary (app/api/users.py), added because Phase 5 never exposed an aggregate: the paginated ledger alone can't answer "what's my points balance" correctly (a client-side sum only ever totals whatever page happened to be loaded). It's a single GROUP BY reward_type query returning RewardsSummaryOut { total_points: int, badges: list[RewardTypeSummary] } (app/schemas/rewards.py) — "points" rows collapse into the scalar balance, every other reward_type an admin configured becomes a badge with a count and latest_at. No migration, no invariant touched — it's a pure read alongside the four from Phase 5. Frontend — lib/progress-api.ts (fetchStreaks) and lib/rewards-api.ts (fetchRewards paginated, fetchRewardsSummary), mirroring the existing per-domain file convention (posts-api.ts, challenges-api.ts). app/(app)/challenges/page.tsx: three polling sections — a weekly breakdown (reuses the same useWeeklyChallenge query/cache the sidebar widget already polls, so it's not a second request for identical data), a Recharts RadialBarChart of percent-to-target across all active challenges (labels rendered as a plain adjacent list rather than Recharts' own Legend, which groups by data key, not by bar, for this chart shape), and a Recharts BarChart comparing current_streak vs best_streak per streak challenge. app/(app)/profile/page.tsx: a points-balance + badges card backed by the new summary endpoint, and a paginated reward ledger (local page state, not URL — unlike the feed, Phase 8's "done when" doesn't require shareable/back-restorable pagination here) with skeleton rows matching the feed's loading-row shape. Every fetch section on both pages is its own <ErrorBoundary> wrapping an inline isLoading/isError/empty/data branch (same shape as FeedContent and WeeklyChallengeWidget), so one section's failure (e.g. streaks) doesn't take down the others (e.g. progress rings) on the same page. Added recharts as the only new frontend dependency.
  • Key decision (which data-viz, why): Progress rings, not a streak heatmap. The backend only ever exposes current_streak/best_streak as scalars (GET /users/me/streaks) — there's no day-by-day activity endpoint, so a genuine calendar heatmap isn't supportable without adding one. Rather than build a fake/degenerate heatmap out of two numbers, I used a Recharts RadialBarChart (real per-challenge percent-to-target rings — reads progress.current_value/progress.target_value, falling back to rule_config.target when a challenge has no progress row yet) plus a Recharts BarChart for the streak comparison, both driven by the theme's existing --chart-1..5 CSS variables (already wired for light/dark in globals.css, evidently anticipating this exact phase). Both are genuinely chart-library-rendered, not a shadcn/Tailwind div-bar — the risk register's explicit disqualification.
  • Notes: The reward-ledger pagination intentionally does not live in the URL (unlike the feed's sort/page) — CLAUDE.md/plan.md's Phase 7 "done when" criterion about shareable/back-restorable state was scoped to the feed specifically; adding useRewardsParams here would be an unrequested abstraction for a page with no such requirement. All three Challenges-page sections poll at the existing 30s cadence (WEEKLY_POLL_INTERVAL_MS) rather than introducing a second interval constant — same reasoning as Phase 6 (evaluation is async/human-scale). Verified via npm run typecheck, npm run lint, and npm run build (all pass) and a next dev smoke check that /challenges and /profile return 200 with no server-side errors; could not exercise the authenticated data-fetching paths (charts, ledger, summary) end-to-end against a live backend in this sandbox — same constraint as every prior phase (no reachable Postgres; docker unavailable). Backend verified via ruff and mypy --strict only, same constraint. Run alembic isn't needed for this phase (no schema change); run a manual pass against a real backend + logged-in session before relying on the charts/ledger rendering correctly.

Phase 9 — Docs / deploy / bonus

  • Built: README.md finalized (quickstart, provisioning, full-flow verification steps, design-decision summary, bonus status). backend/app/scripts/seed.py — idempotent provisioning script (python -m app.scripts.seed): upserts one admin user by email, and inserts five sample challenges matched by name (skip if present) covering every axis the engine supports — count/streak type, one_time/weekly period, and a non-points reward type (badge) to demonstrate reward.type was already free-form since Phase 4. Bonus — rate limiting: app/core/rate_limit.py::TokenBucketRateLimiter, a plain in-memory token bucket keyed by user_id, checked at the top of POST /api/events before any DB work (app/api/events.py); raises AppError("RATE_LIMITED", ..., 429) with a computed retry_after_seconds. New settings EVENT_RATE_LIMIT_CAPACITY/EVENT_RATE_LIMIT_WINDOW_SECONDS (default 20/60s). Tested both in isolation (tests/test_rate_limit.py, pure — injects now instead of sleeping) and live against the endpoint (tests/test_events.py::test_exceeding_the_event_rate_limit_returns_429, which shrinks the module-global limiter's capacity for one test and restores it).
  • Key decision (rate limiting infra, deploy scope): In-memory token bucket, not Redis — same reasoning this repo already used once for the job queue (§3: the most explainable infra that satisfies the requirement beats the more "correct" one for a 5-day, single-reviewer submission). Trade-off stated in the module docstring rather than hidden: state is per-process, so the ceiling is per uvicorn worker, not global across replicas; swapping in Redis later is a localized change to one file. Deploy — descoped to "polish the local run" (user's explicit call): no public URL for this submission, but every piece (Dockerfiles, .env.example, the seed script) is deploy-ready, so pointing it at a host is a config exercise, not a code change. Remaining bonus items (leaderboard, a second reward-type UI) were explicitly descoped by the user to leave more time for hardening quality over bonus surface area — unit tests (already done across Phases 1–8) and rate limiting were the two the user asked for.
  • Notes: This phase is the first time the backend ran against a real, reachable Postgres instance (a native Windows Postgres 18 install, credentials supplied by the user mid-session) — every prior phase's explain.md notes say verification was ruff/mypy/static-only because no DB was reachable in the sandbox. Getting a live DB immediately surfaced two real bugs invisible to static checks: (1) a test-suite bug — app/core/db.py caches the async engine/pool as a process-global singleton, but pytest-asyncio's default is a new event loop per test function; every test after the first tried to reuse pooled connections bound to a dead loop ("Future attached to a different loop" / asyncpg.exceptions._base.InternalClientError: got result for unknown protocol state 3"). Fixed by pinning both asyncio_default_fixture_loop_scopeandasyncio_default_test_loop_scopeto"session"inpyproject.tomlso the whole test session shares one loop, matching the session-scoped_schemafixture that creates the engine. (2) **A real production bug** — Pydantic v2 puts the raised exception object itself underctx.errorin.errors()output for a failedmodel_validator(e.g.validate_challenge_config's ValueErrorfor an unknown challenge type or badrule_config); that's not JSON-serializable, so app/core/errors.py::handle_validation_errorcrashedjson.dumpsand turned every such 422 into an unhandled 500. Fixed with_sanitize_validation_errors(), which stringifies ctx.errorbefore it reaches the response body — the only non-serializable field pydantic ever puts in.errors(). Also fixed a trivial HTTP_422_UNPROCESSABLE_ENTITY→HTTP_422_UNPROCESSABLE_CONTENTStarlette deprecation warning noticed while in that file. After both fixes: all 63 tests (58 pre-existing + 5 new for the rate limiter) pass,ruffandmypy --strictare clean, and the seed script plus the full event→job→progress→reward chain were manually verified end-to-end against the live DB (acomment_posted` event moved four seeded challenges' progress in one worker pass and correctly disbursed the "First Comment" reward on the first try) — the strongest verification any phase in this project has had. The lesson worth stating out loud in review: static checks (ruff/mypy) and a green test suite are not the same claim as "runs against a live database" — this project spent eight phases only able to make the former claim, and both bugs found here were exactly the class of thing that only shows up under the latter.

Phase 10 — Admin console UI

  • Built: frontend/src/lib/admin-challenges-api.ts (types + CRUD calls mirroring ChallengeCreate/ChallengeUpdate/ChallengeOut field-for-field, plus the CHALLENGE_TYPES/EVENT_TYPES/CHALLENGE_STATUSES/CHALLENGE_PERIODS option lists the form's dropdowns render from), hooks/use-admin-challenges.ts (list + single-item queries) and hooks/use-admin-challenge-mutations.ts (create/update/archive — no optimistic writes, unlike the forum mutations, since this is an admin tool where the re-fetched row matters more than perceived latency). components/admin/challenge-form.tsx is one shared form for both create and edit, switching its one type-dependent field's label ("Target count" vs "Streak length") off the type dropdown. Three routes: app/(app)/admin/challenges/{page,new/page,[id]/edit/page}.tsx, gated by a second layout (app/(app)/admin/layout.tsx) stacked on top of the existing auth gate — it only adds the role === "admin" check and redirects to /feed otherwise, reusing the same skeleton-while-unknown shape as (app)/layout.tsx. Added a plain components/ui/native-select.tsx (styled <select>) rather than pulling in Base UI's Select composition, since the form's five dropdowns didn't need combobox behavior. Sidebar gained a conditional "Admin" nav item (user?.role === "admin").
  • Key decision (two-layout gating, no new backend): The Phase 4 admin API (/admin/challenges CRUD) was already complete and untouched — this phase is frontend-only. Gating is two stacked layouts rather than one combined check: (app)/layout.tsx proves "logged in at all" (existing, Phase 6), (app)/admin/layout.tsx additionally proves "and is an admin" — same pattern as the backend's CurrentUser → AdminUser dependency chain (app/api/deps.py), so the frontend's authorization shape mirrors the backend's instead of inventing a new one. CHALLENGE_TYPES = ["count", "streak"] is hardcoded in the frontend rather than fetched from the backend's STRATEGIES registry — this doesn't violate invariant #3 (no if type == in domain/business logic): it's a form's dropdown options, not evaluator branching, and the backend still rejects an invalid type on submit regardless of what the dropdown offered.
  • Notes: Incidental bug found while live-testing this phase (Playwright against a real next dev + uvicorn, JWTs minted directly via app.core.security.create_access_token so a valid session could be exercised even with the sandbox's Postgres unreachable — see Phase 3–9's recurring note): every unhandled 500 was reaching the browser as an opaque CORS failure, not a readable error, because Starlette's @app.exception_handler(Exception) — unlike the AppError/RequestValidationError/StarletteHTTPException handlers beside it — is invoked by ServerErrorMiddleware, which sits outside CORSMiddleware in the stack app/main.py builds. This is a real, general bug (affects every endpoint, not just the new admin ones) that just happened to surface here because this was the first time a page was driven against a genuinely broken backend call live in a browser rather than checked via isError assertions. Fixed in app/core/errors.py::_cors_headers_for_unhandled_exception, which replicates CORSMiddleware's origin check by hand for that one handler; regression-tested in tests/test_errors.py (a bare RuntimeError route, asserting the header is present for an allowed origin and absent for a disallowed one). Also fixed a real Base UI accessibility warning caught in the same pass: Button's render prop refuses to be pointed at an <a> (Base UI's own docs say link-styled-as-button should style the <a> directly, not compose through Button) — switched both "New challenge" and "Edit" from <Button render={<Link/>}> to <Link className={buttonVariants(...)}>. Same DB-availability constraint as every prior phase: the sandbox's Postgres (Supabase, over its session pooler) is unreachable here (TLS interception rewrites the chain with a self-signed cert, which asyncpg's handshake rejects), so tests/test_errors.py and the full suite couldn't be executed against a live DB in this session — verified instead via ruff/mypy --strict (both clean) and the Playwright pass described above, plus a direct curl comparison of a 200 vs. a genuinely-unhandled 500's response headers against the running (DB-broken) server, which is what actually caught and confirmed the CORS bug and its fix.

8. If you remember nothing else (review cheat-sheet)

  • The engine is decoupled; the forum only emits events; there is one road in.
  • The evaluator is data-driven; type is switched on in exactly one place (the strategy registry).
  • Idempotency is in two places: event replay (unique event_id, return stored response) and reward once-only (unique (user, challenge, completion_key)).
  • The job is durable (Postgres outbox, same-txn enqueue, SKIP LOCKED, retries) → at-least-once + idempotent writes = effectively-once. BackgroundTasks was rejected because it isn't durable.
  • One timezone, fixed as a constant, drives both streak-day bucketing and the Monday weekly reset. Per-user TZ is the anticipated extension.
  • Frontend: optimistic with real rollback, URL-as-state, skeletons not spinners, charting-lib data-viz, ~30s polling (justified).