Can candle-yolo do training and inference?

[Raj2032]

Hi guys I have some questions, so if I understand correctly candle is a replacement of pytorch and can be used with yolo to train images? I assume you use candle-yolo to do training for a custom model?

I also wanted to ask can candle run models after training on images, videos and live cameras, similar to this code for yolo/ultralytics with python:

from ultralytics import YOLO

model = YOLO("custom_ncnn_model/")

model.predict(source = "video.mp4",
show = True, conf = 0.6, line_thickness = 2, save = False)

[Charlie284]

Short answer: Candle can be used to write training code, but candle-yolo-v8 is not a Rust replacement for the full Ultralytics package.

There are three separate capabilities here:

1. Training: Candle itself supports model training and autograd, as noted in the framework features. However, the YOLOv8 example is inference-only. It loads an existing safetensors checkpoint, constructs the model, and calls forward on image tensors; it has no YOLO loss, labeled-dataset loader, augmentation pipeline, optimizer, backward pass, validation, or checkpoint-writing loop (the complete run path). So you cannot currently point candle-yolo-v8 at a custom dataset and train it.

2. Still-image inference: This is what the example supports. It accepts one or more image paths and a local --model file in safetensors format (arguments). The local file must match Candle's port exactly; an Ultralytics NCNN export directory is not a compatible input. The detection example also currently constructs the head with 80 classes, so a custom model requires changing that value and the COCO label list as well as supplying correctly mapped weights (hard-coded class count).

   If you still have the trained Ultralytics .pt checkpoint, the Candle YOLO model owner posted the weight-name conversion script used for the example. The owner calls it "very hacky"; it is a YOLOv8-specific mapping, not a general NCNN importer. Keep the original .pt file and validate the converted model's output against Ultralytics before relying on it.

3. Video and live-camera inference: The Candle example does not provide an equivalent of model.predict(source=...). Its input is Vec<String> image paths and it opens each with image::ImageReader. You can build video/camera support around it by decoding frames with something such as OpenCV, FFmpeg, GStreamer, or nokhwa, applying the same resize/RGB/CHW/0–1 preprocessing, calling model.forward for each frame, running the existing NMS/report code, and then displaying or encoding the result. That capture/display loop is application code you would need to add.

For the least work today, train the custom detector with Ultralytics Train mode and use Ultralytics for image/video/webcam prediction; its Predict mode already supports video files, streams, and webcam indices, and its NCNN integration can load the exported directory you show. Use Candle when you specifically want a Rust inference implementation and are prepared to convert the original weights and implement the media-input loop.