Anima - Authoring and Directing Story Worlds
Welcome to ANIMA, a Computer Graphics team at Inria and Univ. Grenoble Alpes, created in July 2020.
The long term goal of the ANIMA team is to provide the necessary tools to author and direct computational story worlds that can serve as stage and screen for virtual reality storytelling. In previous work, we have already tackled the problem of generating responsive shapes such as virtual paper, i.e. surfaces that “act like paper” (they can be folded, crumpled and torn apart like paper, even creating the same sounds as paper). This work was not a faithful physical simulation of paper, but a computational mimicry of some important semantic and aesthetic properties of paper. In future work, we would like to further emphasize the metaphor of a computer theatre of virtual props, actors and cameras that can be authored and directed by artists and domain experts to tell stories with computers.
In an animated story world, virtual props need affordances, i.e. actions that can be performed with them. Similarly, virtual actors need skills, i.e. actions that they can perform within the given virtual world. This opens new scientific problems – how do we represent the story computationally? How do we assemble virtual sets for a story? How do we assign characters and objects in the story to actors and props in the virtual world? How do we design virtual actors with the necessary skills, and virtual props with the necessary affordances? How do we sketch and sculpt the choreography of their movements and actions? To address those scientific problems, the central themes of the ANIMA team are (i) to use the metaphor of movie storyboards as a multimodal depiction of a scene and to sketch entire stories from storyboards; and (ii) to use the metaphor of theatre rehearsals for iteratively improving the quality of virtual performances authored and directed with our systems.
In this context, the research program for ANIMA focuses on developing computer tools for authoring and directing animated movies, interactive games and mixed-reality applications, using virtual sets, actors, cameras and lights.
This raises several scientific challenges. Firstly, we need to build a representation of the story that the user/director has in mind, and this requires dedicated user interfaces for communicating the story. Secondly, we need to offer tools for authoring the necessary shapes and motions for communicating the story visually, and this requires a combination of high-level geometric, physical and semantic models that can be manipulated in real-time under the user’s artistic control. Thirdly, we need to offer tools for directing the story, and this requires new interaction models for controlling the virtual actors and cameras to communicate the desired story while maintaining the coherence of the story world.
Stories can come in many forms. An anatomy lesson is a story. A cooking recipe is a story. A geological sketch is a story. Many paintings and sculptures are stories. Stories can be told with words, but also with drawings and gestures. For the purpose of creating animated story worlds, we are particularly interested in communicating the story with words in the form of a screenplay or with pictures in the form of a storyboard. We also foresee the possibility of communicating the story in space using spatial gestures.
The first scientific challenge for the ANIMA team is to propose new computational models and representations for screenplays and storyboards, and practical methods for parsing and interpreting screenplays and storyboards from multimodal user input. To do this, we reverse engineer existing screenplays and storyboards, which are well suited for generating animation in traditional formats. We also explore new representations for communicating stories with a combination of speech commands, 3D sketches and 3D gestures, which promise to be more suited for communicating stories in new media including virtual reality, augmented reality and mixed reality.
Telling stories visually creates additional challenges not found in traditional, text-based storytelling. Even the simplest story requires a large vocabulary of shapes and animations to be told visually. This is a major bottleneck for all narrative animation synthesis systems.
The second scientific challenge for the ANIMA team is to propose methods for quickly authoring shapes and animations that can be used to tell stories visually. We therefore need to devise methods for generating shapes and shape families, understanding their functions and styles, their material properties and affordances; authoring animations for a large repertoire of actions, that can be easily edited and transfered between shapes; and printing and fabricating articulated and deformable shapes suitable for creating physical story worlds with tangible interaction.
Lastly, we aim to develop methods for controlling virtual actors and cameras in virtual worlds and editing them into movies in a variety of situations ranging from 2D and 3D professional animation, to virtual reality movies and real-time video games. Starting from the well-established tradition of the storyboard, we would like to offer new tools for directing movies in 3D animation, where the user is really the director, and the computer is in charge of its technical execution using a library of film idioms.
Research Goals
Narrative contents such as interactive games and animated movies are a major application domain for computer graphics, with implications in entertainment, education, cultural heritage, scientific communication and professional training. In those applications, the creation of 3-D content cannot be limited to the production of shapes and motions; it should also include the necessary steps to organize shapes and motions into compelling stories, using adequate staging, directing and editing. As a result, it becomes essential to conduct research in authoring and directing animated story worlds.The long term goal of the ANIMA team is to provide the necessary tools to author and direct computational story worlds that can serve as stage and screen for virtual reality storytelling. In previous work, we have already tackled the problem of generating responsive shapes such as virtual paper, i.e. surfaces that “act like paper” (they can be folded, crumpled and torn apart like paper, even creating the same sounds as paper). This work was not a faithful physical simulation of paper, but a computational mimicry of some important semantic and aesthetic properties of paper. In future work, we would like to further emphasize the metaphor of a computer theatre of virtual props, actors and cameras that can be authored and directed by artists and domain experts to tell stories with computers.
In an animated story world, virtual props need affordances, i.e. actions that can be performed with them. Similarly, virtual actors need skills, i.e. actions that they can perform within the given virtual world. This opens new scientific problems – how do we represent the story computationally? How do we assemble virtual sets for a story? How do we assign characters and objects in the story to actors and props in the virtual world? How do we design virtual actors with the necessary skills, and virtual props with the necessary affordances? How do we sketch and sculpt the choreography of their movements and actions? To address those scientific problems, the central themes of the ANIMA team are (i) to use the metaphor of movie storyboards as a multimodal depiction of a scene and to sketch entire stories from storyboards; and (ii) to use the metaphor of theatre rehearsals for iteratively improving the quality of virtual performances authored and directed with our systems.
In this context, the research program for ANIMA focuses on developing computer tools for authoring and directing animated movies, interactive games and mixed-reality applications, using virtual sets, actors, cameras and lights.
This raises several scientific challenges. Firstly, we need to build a representation of the story that the user/director has in mind, and this requires dedicated user interfaces for communicating the story. Secondly, we need to offer tools for authoring the necessary shapes and motions for communicating the story visually, and this requires a combination of high-level geometric, physical and semantic models that can be manipulated in real-time under the user’s artistic control. Thirdly, we need to offer tools for directing the story, and this requires new interaction models for controlling the virtual actors and cameras to communicate the desired story while maintaining the coherence of the story world.
Understanding stories
Stories can come in many forms. An anatomy lesson is a story. A cooking recipe is a story. A geological sketch is a story. Many paintings and sculptures are stories. Stories can be told with words, but also with drawings and gestures. For the purpose of creating animated story worlds, we are particularly interested in communicating the story with words in the form of a screenplay or with pictures in the form of a storyboard. We also foresee the possibility of communicating the story in space using spatial gestures.
The first scientific challenge for the ANIMA team is to propose new computational models and representations for screenplays and storyboards, and practical methods for parsing and interpreting screenplays and storyboards from multimodal user input. To do this, we reverse engineer existing screenplays and storyboards, which are well suited for generating animation in traditional formats. We also explore new representations for communicating stories with a combination of speech commands, 3D sketches and 3D gestures, which promise to be more suited for communicating stories in new media including virtual reality, augmented reality and mixed reality.
Authoring story worlds
Telling stories visually creates additional challenges not found in traditional, text-based storytelling. Even the simplest story requires a large vocabulary of shapes and animations to be told visually. This is a major bottleneck for all narrative animation synthesis systems.
The second scientific challenge for the ANIMA team is to propose methods for quickly authoring shapes and animations that can be used to tell stories visually. We therefore need to devise methods for generating shapes and shape families, understanding their functions and styles, their material properties and affordances; authoring animations for a large repertoire of actions, that can be easily edited and transfered between shapes; and printing and fabricating articulated and deformable shapes suitable for creating physical story worlds with tangible interaction.
Directing story worlds
Lastly, we aim to develop methods for controlling virtual actors and cameras in virtual worlds and editing them into movies in a variety of situations ranging from 2D and 3D professional animation, to virtual reality movies and real-time video games. Starting from the well-established tradition of the storyboard, we would like to offer new tools for directing movies in 3D animation, where the user is really the director, and the computer is in charge of its technical execution using a library of film idioms.
