Flexible SVBRDF Capture with a Multi-Image Deep Network

Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis and Adrien Bousseau

Abstract

Empowered by deep learning, recent methods for material capture can estimate a spatially-varying reflectance from a single photograph. Such lightweight capture is in stark contrast with the tens or hundreds of pictures required by traditional optimization-based approaches. However, a single image is often simply not enough to observe the rich appearance of real-world materials. We present a deep-learning method capable of estimating material appearance from a variable number of uncalibrated and unordered pictures captured with a handheld camera and flash. Thanks to an order-independent fusing layer, this architecture extracts the most useful information from each picture, while benefiting from strong priors learned from data. The method can handle both view and light direction variation without calibration. We show how our method improves its prediction with the number of input pictures, and reaches high quality reconstructions with as little as 1 to 10 images — a sweet spot between existing single-image and complex multi-image approaches.

Paper, code, slides and supplementals

Paper’s author version
Basilic webpage
HAL webpage

Supplemental materials for browsers

Supplemental materials
Slides

Source code (Github)
Source code (Zip)
Trained weights
Dataset

Acknowledgements and Funding

We thank Yulia Gryaditskaya, Simon Rodriguez and Stavros Diolatzis for their support during the deadline as well as Anthony Jouanin and Vincent Hourdin for regular feedback. We also thank Zhengqin Li and Kalyan Sunkavalli for their help with evaluation.This work was partially funded by an ANRT (http://www.anrt.asso.fr/en) CIFRE scholarship between Inria and Optis, the ERC Advanced Grant FUNGRAPH (No. 788065, http://fungraph.inria.fr), and by software and hardware donations from Adobe and Nvidia.

Bibtex

@Article{DADDB19,
  author       = "Deschaintre, Valentin and Aittala, Miika and Durand, Fr\'edo and Drettakis, George and Bousseau, Adrien",
  title        = "Flexible SVBRDF Capture with a Multi-Image Deep Network",
  journal      = "Computer Graphics Forum(Eurographics Symposium on Rendering Conference Proceedings)",
  number       = "4",
  volume       = "38",
  pages        = "13",
  month        = "jul",
  year         = "2019",
  keywords     = "Reflectance modeling, Image Processing, material capture, appearance capture, SVBRDF, deep learning",
  url          = "http://www-sop.inria.fr/reves/Basilic/2019/DADDB19"
}