Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. Using our generated weather augmented Kitti and Cityscapes dataset, we conduct a thorough evaluation of deep object detection and semantic segmentation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection and 60% for semantic segmentation.
Furthermore, we show refining existing networks with our augmented images improves the robustness of both object detection and semantic segmentation algorithms. We experiment on nuScenes and measure an improvement of 15% for object detection and 35% for semantic segmentation compared to original rainy performance. Code and datasets are available.
Update 14/06: A new paper with new exciting results will be out soon.
Weather augmented datasets
We provide our Weather Kitti and Weather Cityscapes, the weather augmentation of Kitti and Cityscapes generated with our ICCV’19 physics-based rendering. For each sequence, we provide 7+ rain levels (from dizzle to storm conditions) and 7 fog intensities (from light to dense fog). Additionally, we provide the post-processed depth maps we generated with Spaded for Kitti or MonoDepth for Cityscapes, and refined with guided-filter.
Details on our weather datasets may be found in our paper. All together, dataset contains 390k+ images.
You can download sequences individually or use our dev toolkit to download & generate automatically all data with:
weather_download-generate.py kitti --kitti_root %PATH%
weather_download-generate.py cityscapes --cityscapes_root %PATH% (cf. command help for more options)
- Dev toolkit [Download]
Contains: all-in-one download/generation scripts, scripts to generate fog images from transmittance (Kitti/Cityscapes), scripts to generate rain images from rain_diff images (Cityscapes only).
- Weather Kitti
- data_object/training (7 481 frames): rain (55GB, 7 rains), fog transmittance (11GB, 7 fogs), depth (3GB)
- raw_data/2011_09_26_drive_0032_sync (390 frames): rain (5GB, 18 rains), fog transmittance (533MB, 7 fogs), depth (158MB)
- raw_data/2011_09_26_drive_0056_sync (294 frames): rain (3GB, 18 rains), fog transmittance (472MB, 7 fogs), depth (142MB)
- Weather Cityscapes
When using these datasets, please cite our paper:
- We do not provide clear weather images (i.e. rain=0mm, fog vmax=∞) and ground-truth. They should be downloaded separately from the original websites Kitti or Cityscapes.
- For simplicity, we use the same file structure as the original Kitti/Cityscapes. As an example, 50mm rain augmentation for ‘Kitti object detection’ dataset will be located under:
weather_kitti/data_object/training/image_2/rain/50mm. Similarly, fog augmentation of Cityscapes for max. visibility of 75m will be under:
- To meet depth map generation constraints Kitti images are slightly center-cropped to 1216×352, and Cityscapes images are downsampled to 1024×512 (i.e.: original_size/2).
- Depth map are encoded as 16bits, and similar to Kitti format.
depth_in_meter = depth/256.
- Kitti/Cityscapes fog images must be generated using fog transmittance map and original datasets. A script is included in our Dev toolkit to generate foggy images automatically. Check
weather_generate.py all --weather fog
- To avoid conflicts with Cityscapes license we CANNOT provide the final Cityscapes rain images. Instead, we provide 16bits encoded rain_diff image: rain_diff = uint16(original – rainy + 2^8 – 1). A script is provided in the Dev toolkit to generate the Cityscape rain images from the rain_diff images and the original Cityscapes. Check
weather_generate.py cityscapes --weather rain
- Rain images come with a rain mask (e.g. rain/100mm/rainy_mask) which might be of high interest for rain removal / deraining researches.
- Datasets are released under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 license term. This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license.
- Scripts are released under MIT license term, free for non-commercial purposes.
Any problems ? License/commercial questions ? Send an email to Raoul de Charette (email@example.com)
(note 14/06) Code for rain/fog augmentation of any image has been delayed a lot, but will be out soon.
Coming up (July 2020)!
More media will be added!
Grants / Fundings
This work was partially supported by the Service de coopération et d’action culturelle du Consulat général de France à Québec with the grant Samuel de Champlain.