Standard physical pen-and-paper creation and computer graphics tools tend to evolve in separate tracks. In this paper, we propose a new interface, PapARt, that bridges the gap between these two worlds. We developed a system that allow users to visualize, manipulate and edit a 3D scene projected onto a paper sheet. Using multitouch and tangible interfaces, users can directly interact with the 3D scene to prepare their drawings. Then, thanks to the projection of the 3D scene directly on the final surface medium, they can draw using standard tools while relying on the underlying 3D scene. Hence, users benefit from both the power of interactive 3D graphics and fast and easy interaction metaphors, while keeping a direct link with the physical material. PapARt has been tested during a large scale exhibition for general public. Such an interface, which combines computer-assisted drawing and free form user expressiveness on a standard sheet of paper, opens new perspectives for enhancing user creation.
System description 
Our system allows highly precise projection of images inside paper sheets. The projection is carried out by an overhead projector on which a camera detecting the paper sheets is glued. We chose overhead projection to allow any kind of drawing medium. The touch input is based on a Kinect device. Two markerboards are put on the table. The drawing medium or canvas can be put on top of the large markerboard. We call it the “drawing area”. The smaller markerboard is a remote controller for controlling the application, referred as the “menu”.
Projection et interaction
The markerboard detection is done by a camera, using ARToolkitPlus (though JavaCV). Although it is not the most aesthetic way to detect the paper sheet, it is very accurate and stable. The camera and the projector require to be calibrated together: the transformation, from the camera’s view to the projector’s view needs to be estimated. This estimation is done using Audet et al. technique and software.
The multi-touch detection is based on a depth-camera detection. It is made by the computation of the distance from any detected object and a pre-set plane. This kind of detection also enables 3D pointing. We use our touch and 3D pointing input for 3D scene manipulation such as rotate and scale operations. The pointing can be used to set a light position for example.
3D scene and drawing
The goal of the 3D interactive application, is to let the user modify a 3D scene and choose a point of view of it. He or she can switch between edition modes using the menu. As the scene is visualized directly on the drawing medium, the artist can have a preview of drawing or painting to create.
Once the scene is set, the projection is used as tracing paper. The projected image is not changing with the paper sheet’s position. Using this, the artist can draw on the paper sheet, relying on the projected image. Furthermore, the paper is still tracked, so, the artist can move the canvas or paper sheet within the display space. While drawing, the menu interface can be cast ouside the drawing space. Consequently, the artist have the whole table for his creation.
Publications
- [hal-page] Touch-Based Interfaces for Interacting with 3D Content in Public Exhibitions
Martin Hachet, Jean-Baptiste De La Rivière, Jérémy Laviole , Aurélie Cohé, Sebastien Cursan - – IEEE Computer Graphics and Applications 33, 2 (2013) 80-85
- Spatial augmented reality to enhance physical artistic creation.
Jérémy Laviole, Martin Hachet. - – Doctoral symposium, 25th ACM Symposium on User Interface Software and Technology (UIST) (2012)
- Demo: Spatial augmented reality for physical drawing
Jérémy Laviole, Martin Hachet. - – Demonstration at 25th ACM Symposium on User Interface Software and Technology (UIST) (2012)
- Digitally assisted stereo drawing
Jérémy Laviole, Fabien Lotte, Martin Hachet. - – The 3rd Dimension of CHI (3DCHI) – Workshop at CHI 2012 (2012)
- PapARt: interactive 3Dgraphics and multi-touch augmented paper for artistic creation.
Jérémy Laviole, Martin Hachet. - – IEEE Symposium on 3D User Interfaces 2012 in Orange County, CA, USA, March 4-5, 2012.
- (preleminary work) Tabletop games using real environment and physical simulation.
- Jérémy Laviole, Martin Hachet, Christophe Schlick,
- – Foundations of Digital Games (FDG) ’11 in Bordeaux, France, June 28 – July 1 2011.
General public exhibitions
- TechFest 2015, Mumbai, India. Jan 2 – 5. 2015.
- Living Lab at Cap Sciences in Bordeaux, for nearly 2 years (2012 – 2014).
- Digital Week 2013 in Bordeaux.
- Inria-Industry Meetings on Health, Strasbourg, November, 21st, 2012.
- Palais de la découverte, Paris. September to December 2011, for the exhibition «Un chercheur, une manip».
