Return to Computer Vision

Physics-Based Rendering for Improving Robustness to Rain

We present a physically-based weather rendering pipeline for realistically inserting rain/fog into clear weather images.
Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. Using our generated weather augmented Kitti and Cityscapes dataset, we conduct a thorough evaluation of deep object detection and semantic segmentation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection and 60% for semantic segmentation.
Furthermore, we show refining existing networks with our augmented images improves the robustness of both object detection and semantic segmentation algorithms. We experiment on nuScenes and measure an improvement of 15% for object detection and 35% for semantic segmentation compared to original rainy performance. Code and datasets are available.

Articles

Physics-Based Rendering for Improving Robustness to Rain
Shirsendu Sukanta Halder, Jean-François Lalonde, Raoul de Charette*
International Conference on Computer Vision (ICCV’19)

Weather augmented datasets

We provide our Weather Kitti and Weather Cityscapes, the weather augmentation of Kitti and Cityscapes generated with our ICCV’19 physics-based rendering. For each sequence, we provide 7+ rain levels (from dizzle to storm conditions) and 7 fog intensities (from light to dense fog). Additionally, we provide the post-processed depth maps we generated with Spaded for Kitti or MonoDepth for Cityscapes, and refined with guided-filter.
Details on our weather datasets may be found in our paper. All together, dataset contains 390k+ images.

You can download sequences individually below. Alternately, use our dev toolkit (cf. below) to download & generate automatically all data with:
weather_download-generate.py kitti --kitti_root %PATH%
or
weather_download-generate.py cityscapes --cityscapes_root %PATH% (cf. command help for more options)

  • Weather Kitti


    More previews? 1, 2, 3

  • Weather Cityscapes

  • More previews? 1, 2, 3

  • Dev toolkit [Download]
    Python 3.x.
    Contains: all-in-one download/generation scripts, scripts to generate fog images from transmittance (Kitti/Cityscapes), scripts to generate rain images from rain_diff images (Cityscapes only).

Citation

When using these datasets, please cite our paper:

@inproceedings{halder2019physics,
   title={Physics-Based Rendering for Improving Robustness to Rain},
   author={Halder, Shirsendu Sukanta and Lalonde, Jean-Fran{\c{c}}ois and de Charette, Raoul},
   booktitle={IEEE/CVF International Conference on Computer Vision},
   year={2019}
}

Notes

  • We do not provide clear weather images (i.e. rain=0mm, fog vmax=∞) and ground-truth. They should be downloaded separately from the original websites Kitti or Cityscapes.
  • For simplicity, we use the same file structure as the original Kitti/Cityscapes. As an example, 50mm rain augmentation for ‘Kitti object detection’ dataset will be located under: weather_kitti/data_object/training/image_2/rain/50mm. Similarly, fog augmentation of Cityscapes for max. visibility of 75m will be under: weather_cityscapes/leftImg8bit/train/fog/75m.
  • To meet depth map generation constraints Kitti images are slightly center-cropped to 1216×352, and Cityscapes images are downsampled to 1024×512 (i.e.: original_size/2).
  • Depth map are encoded as 16bits, and similar to Kitti format. depth_in_meter = depth/256.
  • Kitti/Cityscapes fog images must be generated using fog transmittance map and original datasets. A script is included in our Dev toolkit to generate foggy images automatically. Check weather_generate.py all --weather fog
  • To avoid conflicts with Cityscapes license we CANNOT provide the final Cityscapes rain images. Instead, we provide 16bits encoded rain_diff image: rain_diff = uint16(original – rainy + 2^8 – 1). A script is provided in the Dev toolkit to generate the Cityscape rain images from the rain_diff images and the original Cityscapes. Check weather_generate.py cityscapes --weather rain
  • Rain images come with a rain mask (e.g. rain/100mm/rainy_mask) which might be of high interest for rain removal / deraining researches.

Licensing

  • Datasets are released under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 license term. This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license.
  • Scripts are released under MIT license term, free for non-commercial purposes.

Any problems ? License/commercial questions ? Send an email to Raoul de Charette (first.last-name@inria.fr)

Code

Code for rain/fog augmentation of any image will be out soon.
Coming up (in November)!

Media



More media will be added!

Grants / Fundings

This work was partially supported by the Service de coopération et d’action culturelle du Consulat général de France à Québec with the grant Samuel de Champlain.

 

Permanent link to this article: https://team.inria.fr/rits/computer-vision/weather-augment/