GDIP: Gated Differentiable Image Processing for Object-Detection in Adverse Conditions [Website][Paper] | Accepted for ICRA 2023

UPDATE: Accepted for ICRA 2023

Abstract

Detecting objects under adverse weather and
lighting conditions is crucial for the safe and continuous
operation of an autonomous vehicle, and remains an unsolved
problem. We present a Gated Differentiable Image Processing
(GDIP) block, a domain-agnostic network architecture, which
can be plugged into existing object detection networks (e.g.,
Yolo) and trained end-to-end with adverse condition images
such as those captured under fog and low lighting. Our proposed GDIP block learns to enhance images directly through the
downstream object detection loss. This is achieved by learning
parameters of multiple image pre-processing (IP) techniques
that operate concurrently, with their outputs combined using
weights learned through a novel gating mechanism. We further
improve GDIP through a multi-stage guidance procedure for
progressive image enhancement. Finally, trading off accuracy
for speed, we propose a variant of GDIP that can be used as
a regularizer for training Yolo, which eliminates the need for
GDIP-based image enhancement during inference, resulting in
higher throughput and plausible real-world deployment. We
demonstrate significant improvement in detection performance
over several state-of-the-art methods through quantitative and
qualitative studies on synthetic datasets such as PascalVOC, and
real-world foggy (RTTS) and low-lighting (ExDark) datasets.

Datasets

Weights File

Setting up the environment:

# Install PyTorch 1.12.0+cu116
> pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

Steps for evaluation:

# Evaluate GDIP-Yolo on ExDark images
> python3 test_GDIP_ExDark.py --weights /path/to/best.pt

# Evaluate GDIP-Yolo on RTTS images
> python3 test_GDIP_RTTS.py --weights /path/to/best.pt

# Evaluate MGDIP-Yolo on ExDark images
> python3 test_MGDIP_ExDark.py --weights /path/to/best.pt

# Evaluate MGDIP-Yolo on RTTS images
> python3 test_MGDIP_RTTS.py --weights /path/to/best.pt

# Evaluate GDIP-REG on ExDark images
> python3 test_GDIP_REG_ExDark.py --weights /path/to/best.pt

# Evaluate GDIP-REG on RTTS images
> python3 test_GDIP_REG_RTTS.py --weights /path/to/best.pt

Running Inference:

# Infer GDIP-Yolo(trained on ExDark) on custom images
> python3 infer_GDIP_ExDark.py --weights /path/to/best.pt --visiual /path/to/images

# Infer GDIP-Yolo(trained on RTTS) on custom images
> python3 infer_GDIP_RTTS.py --weights /path/to/best.pt --visiual /path/to/images

# Infer MGDIP-Yolo(trained on ExDark) on custom images
> python3 infer_MGDIP_ExDark.py --weights /path/to/best.pt --visiual /path/to/images

# Infer MGDIP-Yolo(trained on RTTS) on custom images
> python3 infer_MGDIP_RTTS.py --weights /path/to/best.pt --visiual /path/to/images

# Infer GDIP-REG(trained on ExDark) on custom images
> python3 infer_GDIP_REG_ExDark.py --weights /path/to/best.pt --visiual /path/to/images

# Infer GDIP-REG(trained on RTTS) on custom images
> python3 infer_GDIP_REG_RTTS.py --weights /path/to/best.pt --visiual /path/to/images

Training Script:

This work is a funded project, so we have only released evaluation and inference scripts for now. We shall release the training script in the future. However, for now, you can follow the implementation details for training from the paper. As we have provided the class for GDIP-Yolo, importing it along with the hyperparameters mentioned in the paper should work.

Citation:

If you find our work useful, please consider citing us:

@misc{https://doi.org/10.48550/arxiv.2209.14922,
  doi = {10.48550/ARXIV.2209.14922},
  
  url = {https://arxiv.org/abs/2209.14922},
  
  author = {Kalwar, Sanket and Patel, Dhruv and Aanegola, Aakash and Konda, Krishna Reddy and Garg, Sourav and Krishna, K Madhava},
  
  keywords = {Computer Vision and Pattern Recognition (cs.CV), Robotics (cs.RO), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {GDIP: Gated Differentiable Image Processing for Object-Detection in Adverse Conditions},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}

Who to Contact:

• Sanket H Kalwar
• Dhruv Patel