This repository contains the source code for "Chimera: A Block-Based Neural Architecture Search Framework for Event-Based Object Detection", published in the Frontiers in Artificial Intelligence journal.
You can read the full paper on:
Chimera paper
bash chimera_env.sh --create-yml pytorch
The general setup for Chimera-NAS is:
python ChimeraSearch.py --exp_name ${EXP_NAME} --population ${INDIVIDUALS} --iterations ${IT} --max_size ${MAX_PARAMS} --features ${LIST_OF_PROXIES} --w_macs ${W_MACS} --w_meco ${W_MECO} --w_zen ${W_ZEN} --w_azexpr ${W_AZEXPR} --w_aztrain ${W_AZTRAIN} --w_azprogr ${W_AZPROGR}
where:
EXP_NAME: Name of the folder where your run will be stored. Default: "chimera-nas"
INDIVIDUALS: Number of individuals in your population. Default is 100.
IT: Number of iterations on the evolutionary search. Default is 1000.
MAX_PARAMS: Maximum number of parameters that an individual from the population can have. It is in the scale of millions already. (If you want 10M, just set it as 10)
LIST_OF_PROXIES: An additive list of proxies that will be used by the search algorithm. The options are "macs", "meco", "zen score", "az-train", "az-prog", "az-expr".
W_MACS, W_MECO, W_ZEN, W_AZEXPR, W_AZTRAIN, W_AZPROGR: The weights used alongside each proxy. Make sure to align them with your list of proxies. They are float values that must add up to 1.0. They are set to zero as the default.
Here is a usage example, based on the final setup adopted in the paper:
python ChimeraSearch.py --exp_name chimera-nas --population 100 --iterations 1000 --max_size 10 --features "macs" "meco" --w_macs 0.35 --w_meco 0.65
First of all, you need to open the "yaml" file for the corresponding dataset you want to test and modify the paths to the location you are using. Then, you can run:
python val.py --data ${DATASET}.yaml --model ${WEIGHTS}.pt --channels ${CH} --split ${SPLIT} --show_sequences ${SEQ} --img ${IMGH} ${IMGW}
SPLIT: val, test
DATASET: You can find the "yaml" files inside the "data" folder. Example: "data/gen1/gen1_shist.yaml"
WEIGHTS: They are stored in the "weights" folder. Example: "weights/gen1/chimera-n0-gen1.pt"
SEQ: Number of sequences you want to see the predictions, default is -1
CH: Number of channels. In this work, the value 10 was adopted for all experiments
IMGH, IMGW: Image size. We adopted IMGH=256, IMGW=320
The preprocessed versions from the GEN1 and PeDRo datasets adopted in this work can be found here:
python train.py --batch ${BATCH} --nbs ${BATCH//2} --epochs ${NUM_EPOCH} --data ${DATASET}.yaml --model ${MODEL_NAME}.yaml --channels ${CH} --name ${WANDB_RUN_NAME} --project ${WANDB_PROJECT_NAME} --hyp ${HYP}.yaml --zoom_out ${Z} --flip ${F} --val_epoch ${VAL_EPOCH} --clip_length ${CLIP_LENGTH} --img ${IMGH} ${IMGW}
where:
BATCH: Batch size
NUM_EPOCH: Number of epochs
MODEL_NAME: The "yaml" files are stored inside the folder "models". Example: "models/chimera-n0.yaml"
CH: Number of channels. In all the experiments in this work, this number was set to 10.
HYP: The "yaml" files for the hyperparameters are stored inside the folder "hyps". It contains some hyperparameters such as learning rate, weight decay, momentum, and the loss function multipliers. Example: "hyps/hyp_gen1.yaml"
F: Horizontal flip probability
Z: Zoom-out probability
VAL_EPOCH: Number of epochs to perform validation
CLIP_LENGTH: Length of the clips used for training
IMGH, IMGW: Image size. We adopted IMGH=256, IMGW=320
To accelerate the training, we adopted some tricks:
1- We validated only at each 10 epochs
2- During training, instead of running the validation steps on full sequences, we divided the val set into batches that can be processed faster.
3- On the training pipeline, only the final validation step over the test set is calculated over full sequences.
4- Values reported in the paper that refer to the val set come from running val.py after training
The factor --nbs stands for Normalized Batch Size. It is also present in the original Ultralytics repo and is utilized to make the training more robust to different batch sizes. Accordingly, the Weight Decay was set taking into consideration the nbs and the clip length according to:
W_Decay = W0*Batch_size*Clip_Length/NBS
Where W0 is the weight decay defined in the hyperparameter files and W_Decay is the one adopted during training (and reported in the paper)
torchrun --nnodes 1 --nproc_per_node ${GPUS} train.py --device ${LIST_OF_GPUS} --batch ${BATCH} --nbs ${BATCH//2} --epochs ${NUM_EPOCH} --data ${DATASET}.yaml --model ${MODEL_NAME}.yaml --channels ${CH} --name ${WANDB_RUN_NAME} --project ${WANDB_PROJECT_NAME} --hyp ${HYP}.yaml --zoom_out ${Z} --flip ${F} --val_epoch ${VAL_EPOCH} --clip_length ${CLIP_LENGTH} --clip_stride ${CLIP_STRIDE}
where:
GPUS: The number of GPUs in your node
LIST_OF_GPUS: The list of the devices from your node. For example, for three GPUs, use [0,1,2]
The reported results from the paper related to the Chimera Testbed are stored in the "Testbed" sub-folder. The file "testbed_results.csv" has all the proxies and training results from the testbed models.
cd Testbed
The first step is to check the distribution of the mAPs across the different event representations:
python plot_map_format.py
Then, you can check the different weights derived through Regression Trees in the paper:
python testbed_regression.py
Finally, the Spearman and Kendall correlations, alongside the Mean-Squared Error over the 10% models from the testbed, are calculated through:
python Testbed_Correlations.py
The folder "nas_experiments" contains more details about some NAS experiments, including tests with different weights. The experiments related to the final reported architectures are inside the sub-folder "macs035_meco065". Inside each folder, there are, for each parameter constraint adopted, the final population, the log with the scores for each of them, and the weights related to the top-5 ranked architectures from the ZS-NAS search after being trained with the PeDRo dataset.
Silva, D. A., Elsheikh, A., Smagulova, K., Fouda, M., & Eltawil, A. M. (2025). Chimera: a block-based neural architecture search framework for event-based object detection. Frontiers in Artificial Intelligence, 8, 1644889.