Human Keypoint Autolabelling

This repo generates custom 2D human keypoint annotations from images by:

1. Defining mesh-surface markers on an SMPL body.
2. Fitting people in images with the vendored CameraHMR inference stack.
3. Projecting those markers into each image.
4. Exporting the result as COCO-style keypoint annotations.

The source tree is standalone. You do not need a separate CameraHMR checkout. Runtime assets and a few Python dependencies are still external.

Entry Points

• python gui_define.py Create or edit a YAML keypoint-definition file with an Open3D GUI.
• python process.py --image-folder ... --keypoint-config ... --output-dir ... Run batch fitting and export output/annotations/dataset.json.
• python gui_verify.py --coco-json ... --image-folder ... Review generated annotations with an Open3D image browser.

What "Standalone" Means Here

• CameraHMR source is vendored under vendor/camerahmr/.
• Asset paths are resolved inside the repo by src/runtime_paths.py.
• The smplx Python package is still installed from pip.
• Official SMPL / SMPL-H / SMPL-X body-model files and pretrained checkpoints are still supplied locally under data/ or redirected through config/paths.yaml / HKA_* environment variables.

Installation

Prerequisites

• Python 3.10 (recommended)
• NVIDIA GPU with CUDA support (recommended for usable inference speed)
• Git (for Detectron2 installation)

Step 1: Create Python Environment

conda create -n autolabel python=3.10
conda activate autolabel

Or using venv:

python -m venv venv
# Windows
venv\Scripts\activate
# Linux/Mac
source venv/bin/activate

Step 2: Install PyTorch

Install PyTorch with CUDA support (adjust CUDA version for your system):

pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118

For CPU-only installation:

pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1

Step 3: Install Dependencies

cd human_keypoint_autolabelling
pip install -r requirements.txt

This installs:

• Core dependencies: numpy, opencv-python, trimesh, open3d, pyyaml
• SMPL utilities: smplx, chumpy
• CameraHMR dependencies: pytorch-lightning, einops, timm, etc.
• Detectron2 (from GitHub)

Step 4: Set Up Runtime Assets

Download and place the required model files under data/:

data/
├── pretrained-models/
│   ├── cam_model_cleaned.ckpt
│   ├── camerahmr_checkpoint_cleaned.ckpt
│   └── model_final_f05665.pkl
├── smpl_mean_params.npz
└── models/
    └── SMPL/
        └── SMPL_NEUTRAL.pkl

Where to get these files:

• SMPL models: Register at SMPL website
• CameraHMR checkpoints: See CameraHMR repository
• Detectron2 checkpoint: Available from Detectron2 Model Zoo

Step 5: Verify Installation

python tests/run_tests.py

Expected output: ALL TESTS PASSED

Step 6: Set Up Pre-commit Hooks (Optional)

pip install pre-commit
pre-commit install

How to Run

Workflow Overview

1. Define custom keypoints → gui_define.py
2. Process images → process.py
3. Verify annotations → gui_verify.py

1. Define Custom Keypoints

Launch the keypoint definition GUI:

python gui_define.py

Optional arguments:

• --model-type {smpl,smplh,smplx} - Choose body model type (default: smpl)
• --output config/my_keypoints.yaml - Specify output file path

GUI workflow:

1. Select Joint A and Joint B from dropdowns
2. Adjust Length % (distance along bone) and Azimuth (rotation angle)
3. Preview the marker position on the 3D body mesh
4. Click "Add Keypoint" to save it to the configuration
5. Export the configuration to a YAML file

2. Process Images

Run batch processing to fit SMPL models and generate annotations:

python process.py --image-folder data/my_images --keypoint-config config/my_keypoints.yaml --output-dir output

Required arguments:

• --image-folder PATH - Directory containing images to process
• --keypoint-config PATH - YAML file with keypoint definitions
• --output-dir PATH - Output directory for annotations and logs

Optional arguments:

• --save-viz - Save visualization overlays to output/visualizations/
• --verbose - Enable INFO-level logging
• --device {cuda,cpu} - Force specific device (default: auto-detect)
• --detection-threshold FLOAT - Detectron2 person detection threshold (default: 0.9)

Example with all options:

python process.py \
    --image-folder data/my_images \
    --keypoint-config config/custom_markers.yaml \
    --output-dir output \
    --save-viz \
    --verbose \
    --detection-threshold 0.85

Outputs:

• output/annotations/dataset.json - COCO-format keypoint annotations
• output/visualizations/*_viz.jpg - Overlay images (if --save-viz used)
• output/logs/ - Processing logs

Supported image formats: .jpg, .jpeg, .png, .bmp, .tiff, .webp

3. Verify Annotations

Review generated annotations with the verification GUI:

python gui_verify.py --coco-json output/annotations/dataset.json --image-folder data/my_images

Required arguments:

• --coco-json PATH - Path to generated COCO JSON file
• --image-folder PATH - Directory containing source images

GUI features:

• Navigate between images using arrow keys or buttons
• Jump to specific image index
• Toggle keypoint labels on/off
• Save current visualization
• View dataset-wide visibility statistics

Keypoint visibility colors:

• 🟢 Green: Visible (COCO v=2)
• 🟡 Yellow: Occluded (COCO v=1)
• 🔴 Red: Unlabeled (COCO v=0)

Configuration

Environment Variables

Override default asset paths using HKA_* environment variables:

export HKA_CAMERAHMR_CHECKPOINT=/path/to/camerahmr_checkpoint_cleaned.ckpt
export HKA_CAM_MODEL_CHECKPOINT=/path/to/cam_model_cleaned.ckpt
export HKA_DETECTRON_CHECKPOINT=/path/to/model_final_f05665.pkl
export HKA_SMPL_MODEL_PATH=/path/to/SMPL_NEUTRAL.pkl

Config File

Alternatively, edit config/paths.yaml to set custom paths.

Asset resolution order:

1. Explicit function arguments
2. HKA_* environment variables
3. config/paths.yaml
4. Default paths under data/

Quick Reference

Common Commands

# Define keypoints
python gui_define.py --model-type smpl --output config/keypoints.yaml

# Process images with visualizations
python process.py --image-folder data/images --keypoint-config config/keypoints.yaml --output-dir output --save-viz

# Verify results
python gui_verify.py --coco-json output/annotations/dataset.json --image-folder data/images

# Run tests
python tests/run_tests.py

Example Workflow

# 1. Activate environment
conda activate autolabel

# 2. Define 5 custom keypoints
python gui_define.py

# 3. Process a folder of images
python process.py \
    --image-folder data/my_dataset \
    --keypoint-config config/custom_keypoints.yaml \
    --output-dir output \
    --save-viz \
    --verbose

# 4. Review the annotations
python gui_verify.py \
    --coco-json output/annotations/dataset.json \
    --image-folder data/my_dataset

Troubleshooting

Issue: ModuleNotFoundError: No module named 'detectron2'

• Solution: Reinstall Detectron2: pip install 'git+https://github.com/facebookresearch/detectron2.git'

Issue: CUDA out of memory

• Solution: Process fewer images at once or use --device cpu

Issue: Missing checkpoint files

• Solution: Ensure all required files are in data/ or set HKA_* environment variables

Issue: No persons detected

• Solution: Lower detection threshold: --detection-threshold 0.5

Additional Documentation

• Getting Started Guide - Detailed setup instructions
• Architecture Overview - System design and components
• API Reference - Python API documentation
• YAML Format - Keypoint configuration format
• COCO Output Format - Annotation schema details

Current Limitations

• The keypoint schema is currently built around the SMPL 24-joint skeleton
• gui_define.py exposes smpl, smplh, and smplx, but non-SMPL modes are only partially documented. The stored keypoint definitions still use the SMPL 24-joint names and IDs
• The lightweight test runner covers pure modules only. It does not exercise the full Detectron2 + CameraHMR + Open3D runtime path
• Image discovery is not recursive; only the specified folder is scanned

Contributing

This repository uses pre-commit hooks for code quality:

pip install pre-commit
pre-commit install
pre-commit run --all-files  # Run all hooks manually

Hooks include:

• YAML validation
• End-of-file fixing
• Trailing whitespace removal
• Secret detection (gitleaks)