VLM-Safe Drone Agent Demo

This repository is a public-safe demo of an onboard drone-photography agent architecture.

It is not a flight controller and it does not execute movement. Instead, it shows the part that is often missing from flashy demos:

• how user intent is turned into a mission mode
• how a VLM is constrained to structured perception outputs
• how local validators and safety gates retain execution authority
• how target framing can be matched to a pose library without letting the model talk directly to the flight stack
• how a validated dry-run plan could sit in front of a companion-computer chain such as VINS, MAVROS, PX4, and an obstacle-aware local planner

At A Glance

Item	Summary
Role	AI agent architecture / safety-gated planning
Problem	Turn user photo intent into structured, auditable, non-executable planning outputs
Model boundary	VLM outputs visual facts and target framing, not flight control
Local authority	Validators, pose selection, and dry-run gate retain execution authority
Robotics context	Designed to sit upstream of stacks such as VINS, MAVROS, PX4, and local planners

Why This Repo Exists

This repository demonstrates:

• AI agent system design
• structured LLM / VLM contracts
• perception-to-action decoupling
• safety-gated planning for robotics workflows

The original production-like environment included many private details that should not be published, such as prompts, runtime config, mission logs, images, internal paths, and operational handoff documents. This repo is a clean-room, public-safe reconstruction of the core architecture.

Why This Is An Agent Project And Not Just "LLM Integration"

This repo is intentionally structured to demonstrate agent architecture rather than just a model call:

• explicit task state through mission_mode
• structured model outputs instead of free-form text
• local validation before planning continues
• tool-like modules for validation, pose selection, and dry-run generation
• a final action object that stays auditable and execution-disabled

Architecture

flowchart LR
    U["User Intent"] --> M["Mission Planner<br/>quick / precision / director"]
    M --> C["Perception Contract"]
    C --> V["VLM Output<br/>facts only"]
    V --> N["Validator / Normalizer"]
    N --> P["Pose Library Selector"]
    P --> G["Safety Gate<br/>dry run only"]
    G --> E["Execution Adapter<br/>disabled in this demo"]
    E --> R["Result / Audit Record"]

Loading

What The Demo Shows

• A mission request enters with mission_mode and a user instruction
• The "VLM output" is represented by structured JSON fixtures
• The validator rejects forbidden control fields
• The pose selector matches target framing to a small public-safe pose library
• The safety gate always produces a dry-run plan with execution disabled
• In a larger robotics stack, this dry-run layer can sit before a VINS / MAVROS / PX4 / local-planner execution adapter

Engineering Themes

• I understand how to constrain model authority in a robotics workflow
• I can break a system into intent, perception, validation, planning, and execution-boundary layers
• I think about agent outputs as contracts, not just strings
• I know how to describe real system boundaries without pretending a demo is a product

Example Output Shape

The public demo intentionally ends with a dry-run object shaped like:

{
  "mission_mode": "quick",
  "pipeline_safe": true,
  "execution_enabled": false,
  "actions": [
    {
      "pose_id": "P009_CENTER_HALF",
      "forward_m": 1.2,
      "right_m": 0.0,
      "up_m": 0.4,
      "turn_right_deg": 170.0
    }
  ]
}

That is the point: the model can influence framing intent, but the system still controls how planning results are represented and whether execution is ever enabled.

What It Does Not Claim

• It does not control PX4
• It does not implement a real obstacle-avoidance stack
• It does not ship a production prompt
• It does not represent a full commercial system

The point is to demonstrate the architecture and safety boundaries.

Repo Layout

• src/drone_agent_demo/contracts.py Data structures for mission requests, perception outputs, and dry-run plans
• src/drone_agent_demo/validator.py Contract validation and forbidden-field checks
• src/drone_agent_demo/pose_library.py A tiny pose library and selector
• src/drone_agent_demo/planner.py Mission planning pipeline and dry-run generation
• fixtures/ Example VLM outputs for quick, precision, and director-style flows
• tests/ Unit tests for the public-safe pipeline
• ARCHITECTURE.md Notes on how this maps to a more realistic robotics agent stack

Run

python3 demo.py fixtures/quick_half_body.json
python3 demo.py fixtures/precision_reference.json
python3 demo.py fixtures/director_three.json
python3 -m unittest discover -s tests

Suggested Project Summary

If the architecture needs a concise summary:

1. The user provides an intent and a mission mode.
2. The VLM is only allowed to return visual facts and target framing.
3. Local validators reject forbidden control fields and normalize the output.
4. A pose selector turns target framing into candidate camera poses.
5. The safety gate returns a dry-run plan with execution disabled.
6. In a real system, only a separate execution adapter would talk to MAVROS, VINS, PX4, or a local planner.

Short Project Summary

Suggested phrasing:

Designed a constrained VLM-agent architecture for onboard drone photography, using structured perception contracts, local validation, pose-library-based target selection, and safety-gated dry-run planning instead of direct model-to-flight control.