CoCMT
This repo is the official implementation of "CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception" (IROS 2025).
Environment Setup
Step 1: Clone Repository
git clone https://github.com/taco-group/COCMT.git
cd COCMTStep 2: Install MMDetection3D Dependencies
First, install the required MMDetection3D ecosystem packages:
pip install mmcv-full==1.6.0
pip install mmdet==2.24.0
pip install mmsegmentation==0.29.1
pip install mmdet3d==1.0.0rc5For more details on MMDetection3D installation, please refer to the CMT project.
Step 3: Install CoCMT
cd mmdetection3d
pip install -v -e .
cd ..
pip install -v -e .
pip install -r requirements.txt
python setup.py develop
python3 opencood/utils/setup.py build_ext --inplace
pip listStep 4: Setup Directories
mkdir opencood/logs
cp -r opencood/modality_assign opencood/logs/heter_modality_assignDataset Preparation
OPV2V Dataset
Please refer to OpenCOOD for dataset download and preparation.
V2V4Real Dataset
Please refer to V2V4Real for dataset download and preparation.
Train
OPV2V 3 Branch
# Train CoCMT Hetero-LiDAR-Camera based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/hetero/CMT-Camera_Lidar_att_fuse.yaml --load_cmt_camera_nofusion_weights --cmt_camera_nofusion_weights_path [CMT_Camera_NoFusion_weights]
--load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights]
# Train CoCMT Hetero-LiDAR-Camera based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/hetero/CMT-Camera_Lidar_att_fuse.yaml --load_cmt_camera_nofusion_weights --cmt_camera_nofusion_weights_path [CMT_Camera_NoFusion_weights]
--load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights] --fix_camera_backbone --fix_lidar_backbone
# Train CoCMT Homo-LiDAR based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/lidar/CMT-L_attfuse.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights]
# Train CoCMT Homo-LiDAR based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/lidar/CMT-L_attfuse.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights] --fix_lidar_backbone
# Train CoCMT Homo-Camera based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/camera/CMT-C_attfuse.yaml --load_cmt_camera_nofusion_weights --load_cmt_camera_nofusion_weights [CMT_Camera_NoFusion_weights]
# Train CoCMT Homo-Camera based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/camera/CMT-C_attfuse.yaml --load_cmt_camera_nofusion_weights --load_cmt_camera_nofusion_weights [CMT_Camera_NoFusion_weights] --fix_camera_backbone
# Train CoCMT LiDAR nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/lidar/CMT-L_pointpillar.yaml
# Train CoCMT Camera nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/camera/CMT-C_resnet50.yaml
V2V4Real
# Train CoCMT Homo-LiDAR based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/v2v4real/CMT/lidar/CMT-L_attfuse_v2v4real.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights]
# Train CoCMT Homo-LiDAR based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/v2v4real/CMT/lidar/CMT-L_attfuse_v2v4real.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights] --fix_lidar_backbone
# Train CoCMT nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/v2v4real/CMT/lidar/CMT-L_pointpillar_v2v4real.yamlEval
OPV2V
bash test_epochs.sh <inference_script> <model_dir> <fusion_method> <epoch list> --lightning --qbm --range "102.4,102.4"
bash test_epochs.sh opencood/logs/cmt_camera_lidar_att_fuse_2024_03_25_10_57_59 intermediate 0,1 --lightning --qbm --range "102.4,102.4"
V2V4Real
bash test_epochs.sh <inference_script> <model_dir> <fusion_method> <epoch list> --lightning --qbm --homo --range "70.4,38.4"
bash test_epochs.sh opencood/logs/cmt_lidar_intermediatefusion_v2v4real intermediate 0,1 --lightning --qbm --homo --range "70.4,38.4"
Acknowledgement
We build our framework on top of CMT, HEAL, OpenCOOD, and V2V4Real. Please refer to the following repos: