AirForesight: Current-to-Future Spatial Map Imagination with Cross-Space Planning Consistency for UAV-VLN
ACM MM 2026

Yutong Liu^1*, Xiaojie Li^1*✉, Mingzhu Xu², Jianlong Wu^1,3✉,

¹Harbin Institute of Technology, Shenzhen    ²Shandong University   
³Shenzhen Loop Area Institute   
^* Equal contribution   ^✉ Corresponding authors  

---

Table of Contents

• Introduction
• Project Structure
• Installation
• Checkpoints
• Data and Model Preparation
• Usage
• Acknowledgement

---

Introduction

This project is the official implementation of the ACM MM 2026 paper “AirForesight: Current-to-Future Spatial Map Imagination with Cross-Space Planning Consistency for UAV-VLN.”

• Problem Addressed: AirForesight is a framework for UAV Vision-and-Language Navigation (UAV-VLN). It addresses three key challenges: insufficient grounding in the current scene, limited future spatial reasoning, and inconsistency between map-space planning and action-space waypoint prediction.

• Core Idea: AirForesight grounds waypoint prediction in a structured current-to-future spatial reasoning process by constructing a current map, predicting a future trajectory in map space, and inferring the corresponding future map. It further introduces a cross-space planning consistency objective to align map-space trajectory reasoning with 3D waypoint prediction.

• Repository Contents: Training and inference code, data processing tools, evaluation scripts, and pretrained checkpoints.

---

Project Structure

.
├── Model/LLaMA-UAV/       # Model implementation and training code
├── airsim_plugin/         # AirSim client/server interfaces
├── scripts/               # Collection, inference and metric scripts
├── src/                   # Inference code
├── tools_generate_map/    # Map generation tools
├── utils/                 # Common utility functions
├── requirements.txt     
└── LICENSE    

The following directories are created locally or populated by downloaded assets:

data/                      # Annotations, metadata, and TravelUAV trajectories
model_zoo/                 # Vicuna and LAVIS model weights
simulators/envs/           # Simulator environments
test_weights/              # Trajectory-completion checkpoint
work_dirs/                 # Training, inference, and evaluation outputs

---

Installation

1. Clone the repository

git clone https://github.com/iLearn-Lab/MM26-AirForesight.git
cd MM26-AirForesight

2. Create environment

conda create -n airforesight python=3.10 -y
conda activate airforesight

3. Install dependencies

# Install PyTorch with CUDA 11.8 support 
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

# Install the main project dependencies 
pip install -r requirements.txt 

# Install LLaMA-UAV in editable mode 
pip install -e Model/LLaMA-UAV 

# Install additional dependencies 
pip install transformers==4.31.0 ninja 
pip install flash-attn==2.5.9.post1 --no-build-isolation

For compatibility with the AirSim Python API, install the patched msgpack-rpc-python described in Microsoft AirSim issue #3333:

git clone -b fix-msgpack-dep https://github.com/tbelhalfaoui/msgpack-rpc-python.git
cd msgpack-rpc-python;
python setup.py install
cd ..

Data and Model Preparation

1. Simulator environments

Download the simulator environments from TravelUAV_env, and arrange them as follows:

simulators/envs/
├── carla_town_envs/
├── closeloop_envs/
└── extra_envs/

The simulator server uses simulators/envs/ by default. A different location can be passed to airsim_plugin/AirVLNSimulatorServerTool.py with --root_path.

2. Download and preprocess the dataset

Download the split annotations from TravelUAV_data_json and the trajectories from TravelUAV. Prepare this layout:

data/
├── meta/
│   ├── map_spawnarea_info.json
│   └── object_description.json
├── train_dataset/          # TravelUAV trajectories
└── uav_dataset/
    ├── trainset.json
    └── seen_valset.json

Generate the merged trajectory annotation file:

cd Model/LLaMA-UAV/
python tools/generate_merged_json.py --root_dir <path to your dataset>

Preprocess the multi-view camera images and convert them into tensor files:

cd Model/LLaMA-UAV/
python tools/preprocess_image2tensor.py --root_dir <path to your dataset>

3. Download pretrained model

• GroundingDINO:

  cd tools_generate_map/tools/
  git clone https://github.com/IDEA-Research/GroundingDINO.git
  cd GroundingDINO
  pip install .
  
  # Download the pretrained GroundingDINO checkpoint
  wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
  

• Mobilesam:

  cd tools_generate_map/tools/
  git clone https://github.com/ChaoningZhang/MobileSAM.git
  cd MobileSAM
  pip install .
  

• Vision-Language Model Checkpoints: Download the pretrained weights from Vicuna-7b-v1.5, EVA-ViT-G, QFormer-7b, and put them in the directory ./model_zoo using the following structure:

  model_zoo 
  ├── vicuna-7b-v1.5 
  │	├── config.json 
  │ 	├── tokenizer.model 
  │ 	├── tokenizer_config.json 
  │ 	├── special_tokens_map.json 
  │ 	└── ... 
  └── LAVIS 
      ├── eva_vit_g.pth
      └── instruct_blip_vicuna7b_trimmed.pth
  

• Trajectory Completion Model: Download the trajectory completion model and save it in the directory ./test_weights/traveluav-traj-model.

---

Checkpoints

Download the AirForesight checkpoint and place it in its own experiment directory under work_dirs/, for example work_dirs/AirForesight/.

---

Usage

Before starting collection or evaluation, configure the simulator executable paths in airsim_plugin/AirVLNSimulatorServerTool.py and make sure the required dataset, simulator environments, and pretrained models weights described above are available.

1. Collection

Collect training rollouts from the annotation file and save the simulator's floating-point depth output.

bash scripts/run_collect_with_restart.sh <model_name> <base_gpu_id> <base_server_port> <num_clients>

The arguments are:

• <model_name>: name of the experiment directory under work_dirs/.
• <base_gpu_id>: GPU assigned to client 0. Client i uses GPU base_gpu_id + i.
• <base_server_port>: simulator port assigned to client 0. Client i uses port base_server_port + 1000 * i.
• <num_clients>: number of simulator/client pairs to launch in parallel.

The collection pipeline reads annotations from data/uav_dataset/trainset.json, uses the source trajectories in data/train_dataset/, and writes collected samples to data/train_dataset_collect_waypoint/. Each camera's metric depth array is stored as a .npy file alongside its recorded image. The restart wrapper monitors client processes and relaunches the collection job if a client terminates unexpectedly. Ensure that the selected GPU IDs and the complete port range are free before launching the job.

2. Generate Map and Trajectory Masks

The preprocessing pipeline first extracts task-relevant object categories from the navigation instructions, then generates the current semantic map, future-trajectory mask, and future semantic map:

python tools_generate_map/generate_instruction_info.py
python tools_generate_map/generate_pointcloud_4views.py
python tools_generate_map/generate_topdown_mask_4views.py

The three commands must be executed in order:

1. `generate_instruction_info.py` parses each navigation instruction, extracts object categories relevant to the task, and saves the structured instruction information for subsequent semantic processing.
2. `generate_pointcloud_4views.py` performs category-aware semantic segmentation and projects the RGB observations and floating-point depth maps into semantic pointclouds for each camera view.
3. `generate_topdown_mask_4views.py` combines the four camera views to produce aligned `224 × 224` top-down semantic masks covering a `200 m × 200 m` area. It also generates a future-trajectory mask from up to the next 10 waypoints.

Generated masks and visualizations are saved in each trajectory's topdown_map_4views_local_align_traj/ directory.

Important

The point-cloud and top-down-mask scripts contain fixed gpu_ids and processes_per_gpu settings. Adjust them according to the available hardware before running. The processing order must be preserved because each stage consumes outputs generated by the preceding stage.

3. Train

Training consists of two stages. The scripts resolve data and output paths relative to Model/LLaMA-UAV/, so launch them from that directory:

cd Model/LLaMA-UAV/

# Stage 1: train the base navigation model.
bash scripts/llm/train.sh

# Stage 2: fine-tune the model with current map and future prediction objectives.
bash scripts/llm/finetune.sh

Stage 1 writes the base checkpoint to the output_dir configured in scripts/llm/train.sh. Before Stage 2, verify that pretrain_checkpoint_path in scripts/llm/finetune.sh points to that checkpoint. Training logs and checkpoints are written to the configured directory under work_dirs/. Adjust GPUS, start_gpu, batch size, output paths, and other hyperparameters in the scripts when using a different setup.

Return to the repository root before running inference or evaluation:

cd ../..

4. Inference

Run closed-loop inference with multiple simulator clients:

bash scripts/run_eval_with_restart.sh <model_name> <base_gpu_id> <base_server_port> <num_clients>

The argument and resource-allocation rules are the same as for collection. <model_name> must identify a trained checkpoint directory under work_dirs/. Predictions are written to work_dirs/<model_name>/eval_test_results/. The wrapper monitors failed clients and restarts the complete evaluation job up to five times.

The current evaluation script reads data/uav_dataset/seen_valset.json. Change EVAL_JSON_PATH in scripts/eval.sh when evaluating another split.

5. Evaluation

Compute navigation metrics after inference has completed:

bash scripts/metric.sh <model_name>

The metric script reads work_dirs/<model_name>/eval_test_results/, evaluates the configured full, easy, and hard path groups, and writes the summary to work_dirs/<model_name>/result.txt.

---

Citation

If you find this work useful in your research, please consider citing our paper:

@inproceedings{liu2026airforesight,
  title     = {AirForesight: Current-to-Future Spatial Map Imagination with Cross-Space Planning Consistency for UAV-VLN},
  author    = {Liu, Yutong and Li, Xiaojie and Xu, Mingzhu and Wu, Jianlong},
  booktitle = {Proceedings of the ACM International Conference on Multimedia},
  year      = {2026}
}

---

Acknowledgement

This codebase builds on TravelUAV, AirVLN, and LLaMA-VID. We thank their authors for making their work publicly available. 🙏