Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. Using our generated weather augmented Kitti and Cityscapes dataset, we conduct a thorough evaluation of deep object detection and semantic segmentation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection and 60% for semantic segmentation.
Furthermore, we show refining existing networks with our augmented images improves the robustness of both object detection and semantic segmentation algorithms. We experiment on nuScenes and measure an improvement of 15% for object detection and 35% for semantic segmentation compared to original rainy performance. Code and datasets are available.
Update 01/08/2022: Fixed some download bugs in the weather dev toolkit.
Update 18/07/2022: We added the rainy Cityscapes validation set (see links below). Many thanks to @Theo2021 for sharing these.
Update 06/09/2020: Code released on github.
Update 01/07/2020: New IJCV paper out with exciting results.
Articles
Code (ICCV’19 & IJCV’20)
2020-09: The code is finally out on GitHub.
https://github.com/cv-rits/rain-rendering
Weather, rainy and foggy, augmented datasets (ICCV’19)
We provide our Weather Kitti and Weather Cityscapes, which contains the rainy and foggy augmentation of Kitti and Cityscapes generated with our ICCV’19 physics-based rendering. For each sequence, we provide 7+ rain levels (from dizzle to storm conditions) and 7 fog intensities (from light to dense fog). Additionally, we provide the post-processed depth maps we generated with Spaded for Kitti or MonoDepth for Cityscapes, and refined with guided-filter.
Details on our weather datasets may be found in our paper. All together, dataset contains 390k+ images.
While individual files are available, we recommend using our dev toolkit to download & generate automatically all data with:
weather_download-generate.py kitti --kitti_root %PATH%
or
weather_download-generate.py cityscapes --cityscapes_root %PATH% (cf. command help for more options)
- Dev toolkit [Download] *** RECOMMENDED ***
Python 3.x.
Contains: all-in-one download/generation scripts, scripts to generate fog images from transmittance (Kitti/Cityscapes), scripts to generate rain images from rain_diff images (Cityscapes only). - Weather Kitti
- data_object/training (7 481 frames): rain (55GB, 7 rains), fog transmittance (11GB, 7 fogs), depth (3GB)
- raw_data/2011_09_26_drive_0032_sync (390 frames): rain (5GB, 18 rains), fog transmittance (533MB, 7 fogs), depth (158MB)
- raw_data/2011_09_26_drive_0056_sync (294 frames): rain (3GB, 18 rains), fog transmittance (472MB, 7 fogs), depth (142MB)
- Weather Cityscapes
- leftImg8bit/train (2995 frames): rain_diff (22GB, 8 rains. ***CHECK NOTES***), fog transmittance (5GB, 7 fogs), depth (2GB)
- leftImg8bit/val (500 frames): rain_diff (4GB, 7 rains. ***CHECK NOTES***) — Thanks to @Theo2021 —
Citation
When using these datasets, please cite our paper:
title={Physics-Based Rendering for Improving Robustness to Rain},
author={Halder, Shirsendu Sukanta and Lalonde, Jean-Fran{\c{c}}ois and de Charette, Raoul},
booktitle={ICCV},
year={2019}
}
Notes
- We do not provide clear weather images (i.e. rain=0mm, fog vmax=∞) and ground-truth. They should be downloaded separately from the original websites Kitti or Cityscapes.
- For simplicity, we use the same file structure as the original Kitti/Cityscapes. As an example, 50mm rain augmentation for ‘Kitti object detection’ dataset will be located under:
weather_kitti/data_object/training/image_2/rain/50mm. Similarly, fog augmentation of Cityscapes for max. visibility of 75m will be under:weather_cityscapes/leftImg8bit/train/fog/75m. - To meet depth map generation constraints Kitti images are slightly center-cropped to 1216×352, and Cityscapes images are downsampled to 1024×512 (i.e.: original_size/2).
- Depth map are encoded as 16bits, and similar to Kitti format.
depth_in_meter = depth/256. - Kitti/Cityscapes fog images must be generated using fog transmittance map and original datasets. A script is included in our Dev toolkit to generate foggy images automatically. Check
weather_generate.py all --weather fog - To avoid conflicts with Cityscapes license we CANNOT provide the final Cityscapes rain images. Instead, we provide 16bits encoded rain_diff image: rain_diff = uint16(original – rainy + 2^8 – 1). A script is provided in the Dev toolkit to generate the Cityscape rain images from the rain_diff images and the original Cityscapes. Check
weather_generate.py cityscapes --weather rain - Rain images come with a rain mask (e.g. rain/100mm/rainy_mask) which might be of high interest for rain removal / deraining researches.
Licensing
- Datasets are released under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 license term. This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license.
- Scripts are released under MIT license term, free for non-commercial purposes.
Any problems ? License/commercial questions ? Send an email to Raoul de Charette (first.last-name@inria.fr)
Media
Grants / Fundings
This work was partially supported by the Service de coopération et d’action culturelle du Consulat général de France à Québec with the grant Samuel de Champlain.
