Skip to content

System_Architecture_and_Algorithms_Guide.md

Latest commit

 

History

History
195 lines (166 loc) · 7.15 KB

System_Architecture_and_Algorithms_Guide.md

File metadata and controls

195 lines (166 loc) · 7.15 KB
Raw

Quick Guide: AI Hide and Seek Arena

1. Tech Stack Used

Core Language

  • Python 3.12
  • JavaScript (vanilla, browser-side)
  • HTML5 + CSS3

Backend

  • FastAPI for REST and WebSocket APIs
  • Uvicorn for ASGI server runtime
  • Pydantic for request validation

AI and Learning

  • Custom tabular Q-Learning (backend/learning.py)
  • Epsilon-greedy exploration with epsilon decay
  • Recent replay buffer for short-term experience replay

Physics and RL Expansion

  • PyBullet for 3D rigid-body simulation (backend/physics_world.py)
  • Gymnasium wrapper for RL environment (backend/physics_selfplay_env.py)
  • Stable-Baselines3 PPO for self-play training (backend/ppo_self_play.py)

Frontend and Visualization

  • Custom 3D-style renderer in Canvas 2D (backend/static/app.js)
  • No CDN dependency required for rendering
  • WebSocket stream for grid mode state updates
  • HTTP polling for physics spectator mode state

2. Algorithms Used

A) Grid Mode Agent Learning (Q-Learning)

  • State value is string-encoded with local neighborhood, relative target direction, distance bucket, and visibility/memory flags.
  • Action space: UP, DOWN, LEFT, RIGHT, STAY.
  • Policy: epsilon-greedy over Q-table values.
  • Update rule:
    • Q(s,a) <- Q(s,a) + alpha * (reward + gamma * max_a' Q(s',a') - Q(s,a))
  • Replay:
    • Every step, each agent stores transition in memory.
    • A random batch from recent memory is replayed to reinforce useful patterns.

B) Seeker Tactical Logic (Hybrid Rule + Learning)

  • Field-of-view cone with angle limit and range limit.
  • Line-of-sight check via Bresenham obstacle blocking.
  • Short-term target memory (seeker_memory_ticks) after losing visual contact.
  • Straight-line pursuit preference to reduce zig-zag movement.
  • Falls back to Q-learning when deterministic pursuit path is not available.

C) Environment and Game Mechanics

  • Procedural room generation with blocked doorway cells.
  • Hiders spawn in room interiors; seekers spawn outside.
  • Obstacles are pushable in grid mode if destination cell is valid.
  • Round-based objective:
    • Seekers win if all hiders are captured.
    • Hiders win by surviving until max round steps.

D) Reward Shaping (Grid Mode)

  • Per-step living cost.
  • Positive reward for seekers reducing distance to hiders.
  • Positive reward for hiders increasing distance from seekers.
  • Capture reward for seeker and penalty for captured hider.
  • Round-end rewards for winning team and penalties for losing team.

E) Physics Action Primitives (PyBullet Mode)

  • Action keys: move, turn, yaw, grab, release, push.
  • Objects:
    • Static arena walls and room walls.
    • Movable boxes, ramps, movable wall segments, doorway block.
  • Interaction mechanics:
    • Nearest interactable object selection with distance and front-direction checks.
    • Grab attaches object to agent hold position.
    • Release detaches object.
    • Push applies physical force impulse.

F) PPO Self-Play Baseline

  • Gymnasium environment outputs compact observation vector.
  • Continuous action head mapped to physics primitives.
  • Supports:
    • Role-wise training (seeker or hider)
    • Alternating league cycles
    • Checkpointing and evaluation matches

3. How the Game Works End-to-End

Startup Flow

  • main.py launches Uvicorn with FastAPI app.
  • backend/main.py creates runtime state and starts background engine loop.
  • Loop ticks at configurable interval and steps:
    • Grid engine (HideSeekEngine)
    • Physics world (if available)

Frontend Data Flow

  • WebSocket /ws continuously streams grid snapshots.
  • Frontend tries physics endpoint /api/physics/state.
  • If physics is available, UI auto-switches to physics spectator mode.
  • If physics is unavailable, UI stays in grid mode.

Grid Step Pipeline

  • Validate legal actions considering boundaries and obstacle-push rules.
  • Choose actions for each alive agent.
  • Execute moves in randomized order to avoid deterministic movement bias.
  • Resolve captures.
  • Compute shaped rewards.
  • Update Q-learning transitions and replay recent memory.
  • Emit events and snapshot for UI.

Physics Step Pipeline

  • Convert action maps into movement/orientation/interaction commands.
  • Apply velocity and heading updates.
  • Process grab/release/push interactions.
  • Sync held objects to carriers.
  • Run Bullet simulation step.
  • Return full world snapshot (agents, objects, room metadata).

4. Current Features Snapshot

  • Multi-agent hider vs seeker gameplay
  • 3D browser spectator UI
  • Live controls and score panels
  • Event timeline
  • Seeker FOV + memory pursuit
  • Room-based spawn strategy
  • Pushable obstacle mechanics (grid mode)
  • Real physics sandbox with movable tactical props
  • PPO self-play training entry points

5. Future Advancements (Detailed Roadmap)

A) Learning and Intelligence

  • Replace tabular Q-table in grid mode with Deep Q-Networks for better scaling to larger maps.
  • Add centralized critic multi-agent methods (MAPPO or QMIX-style variants) for coordinated team behavior.
  • Train separate policies for sub-roles:
    • Door blocker
    • Scout seeker
    • Decoy hider
  • Add curriculum learning:
    • Start with simple maps and no movable tools.
    • Increase room complexity, object count, and enemy intelligence gradually.

B) Physics and Emergent Strategy

  • Add true ramp climbing heuristics and reward terms for successful room breach via ramps.
  • Add object affordances:
    • Stackable boxes
    • Lockable door-block states
    • Hinged doors and breakable barricades
  • Add contact-based penalties to avoid unrealistic jitter exploits.
  • Add deterministic replay system for physics episodes to debug learned strategies.

C) Environment Diversity

  • Procedural map generator with themes:
    • Single-room fortress
    • Multi-room maze
    • Open arena with sparse cover
  • Dynamic hazards:
    • Shrinking safe zones
    • Timed gates
    • Moving barriers
  • Randomized spawn policies with fairness constraints and anti-camping rules.

D) UI and UX

  • True WebGL renderer (Three.js or Babylon.js) with real lighting, shadows, and post-processing.
  • Camera presets:
    • Tactical top-down
    • Follow seeker
    • Follow hider
    • Free spectator
  • Timeline scrubber to replay last N seconds.
  • Visual analytics overlay:
    • FOV cones
    • LOS rays
    • Predicted target trajectory
    • Heatmaps of movement density

E) Training Operations

  • Add structured experiment config files (yaml) for reproducible runs.
  • Add TensorBoard logging for rewards, win rates, entropy, capture time.
  • Add model registry for checkpoint tagging by metrics.
  • Add automated nightly evaluation tournament and leaderboard generation.

F) Production and Scale

  • Move simulation workers to distributed execution for faster self-play throughput.
  • Separate inference server from game server for lower latency.
  • Add persistent database for:
    • Match history
    • Agent stats
    • Model lineage
  • Add room-based multiplayer spectator sessions with shareable links.

6. Recommended Next Milestones

  1. Finalize PPO self-play loop quality with stable evaluation metrics.
  2. Add WebGL renderer with camera presets and replay timeline.
  3. Introduce multi-map curriculum and automated benchmark suite.
  4. Add distributed training runner and experiment tracking dashboard.