107 | Portrait animation technology provides a ultra-low cost 108 | and highly effective way to creating expressive, realistic 109 | character animations and video footages: users only need 110 | to provide a static portrait image and a driving 111 | performance video, and the model can then use these inputs 112 | to generate videos by transferring the driving expression 113 | to the subject in the portrait. This can drastically 114 | reduce the complexity of existing motion capture, 115 | character animation and content creation pipeline. 116 |
117 |122 | We introduce X-Portrait 2, which builds upon our previous 123 | work X-Portrait and brings the expressiveness of portrait 124 | animation to a whole new level. To achieve this, we build 125 | a state-of-the-art expression encoder model that 126 | implicitly encodes every minuscule expressions from the 127 | input by training it on large-scale datasets. This encoder 128 | is then combined with powerful generative diffusion models 129 | to generate fluid and expressive videos. Our X-Portrait 2 130 | model can transfer subtle and minuscule facial expressions 131 | from the actors as well as challenging expressions 132 | including pouting, tougue-out, cheek-puffing and 133 | frowning. High fidelity of emotion preservation can also 134 | be achieved in the generated videos. 135 |
136 |195 | We ensure strong disentanglement of appearance and 196 | motion in training our expression encoder such that 197 | the encoder only pay attention to the 198 | expression-related information in the driving 199 | video. Our model can achieve strong cross-style and 200 | cross-domain expression transfer, covering both 201 | realistic portraits and cartoon images. This makes our 202 | model highly adaptive to a wide range of use cases 203 | including real-world storytelling, character 204 | animation, virtual agents and visual effects. 205 |
206 |235 | Comparing to state-of-the-art methods, such 236 | as X-Portrait 237 | and 238 | recently-released Runyway 239 | Act-One, our model can faithfully transfer fast 240 | head movements, minuscule expression changes and 241 | strong personal emotions. These aspects are crucial 242 | for high-quality animated content creation, such as 243 | those in the animation and movie productions. 244 |
245 |263 | The video clips used as driving videos on this webpage 264 | are sourced from Pexels, NeRSemble dataset, 265 | DFEW dataset, and movie Fences (2016), No Strings Attached (2011), It Ends With Us (2024), 266 | I Need You (2010), Face Off (1997) and The Shining (1980). The static images used as reference are sourced from 267 | Pexels, Midjourney and Deviantart. 268 | These materials are 269 | used solely for research and demonstration purposes 270 | under the principles of fair use, and no copyright 271 | infringement is intended. If you are a copyright holder 272 | and believe that our use of your video materials is 273 | inappropriate or violates copyright laws, please contact 274 | us, and we will promptly address the matter. This 275 | webpage is based 276 | on nerfies 277 | template. 278 |
279 |