├── docs ├── img │ ├── logo3.png │ ├── logo4.png │ └── magicstick_logo.png └── gif_results │ ├── teaser │ ├── cup.gif │ ├── parrot.gif │ ├── rabbti.gif │ └── ocean_bird.gif │ ├── object_size_editing │ ├── bear.gif │ ├── panda.gif │ ├── bird_enlarge.gif │ ├── mountain111.gif │ └── swan2small.gif │ ├── human_motion │ ├── human_motion.gif │ └── human_motion1.gif │ ├── object_position_editing │ ├── shark.gif │ ├── parrot.gif │ ├── cup_moving.gif │ └── astronaut_moving.gif │ └── object_appearance_editing │ ├── swan_fl.gif │ ├── swan_duck.gif │ ├── truck_bus.gif │ └── truck_train.gif └── README.md /docs/img/logo3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/img/logo3.png -------------------------------------------------------------------------------- /docs/img/logo4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/img/logo4.png -------------------------------------------------------------------------------- /docs/img/magicstick_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/img/magicstick_logo.png -------------------------------------------------------------------------------- /docs/gif_results/teaser/cup.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/teaser/cup.gif -------------------------------------------------------------------------------- /docs/gif_results/teaser/parrot.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/teaser/parrot.gif -------------------------------------------------------------------------------- /docs/gif_results/teaser/rabbti.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/teaser/rabbti.gif -------------------------------------------------------------------------------- /docs/gif_results/teaser/ocean_bird.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/teaser/ocean_bird.gif -------------------------------------------------------------------------------- /docs/gif_results/object_size_editing/bear.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_size_editing/bear.gif -------------------------------------------------------------------------------- /docs/gif_results/human_motion/human_motion.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/human_motion/human_motion.gif -------------------------------------------------------------------------------- /docs/gif_results/human_motion/human_motion1.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/human_motion/human_motion1.gif -------------------------------------------------------------------------------- /docs/gif_results/object_size_editing/panda.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_size_editing/panda.gif -------------------------------------------------------------------------------- /docs/gif_results/object_position_editing/shark.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_position_editing/shark.gif -------------------------------------------------------------------------------- /docs/gif_results/object_position_editing/parrot.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_position_editing/parrot.gif -------------------------------------------------------------------------------- /docs/gif_results/object_size_editing/bird_enlarge.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_size_editing/bird_enlarge.gif -------------------------------------------------------------------------------- /docs/gif_results/object_size_editing/mountain111.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_size_editing/mountain111.gif -------------------------------------------------------------------------------- /docs/gif_results/object_size_editing/swan2small.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_size_editing/swan2small.gif -------------------------------------------------------------------------------- /docs/gif_results/object_appearance_editing/swan_fl.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_appearance_editing/swan_fl.gif -------------------------------------------------------------------------------- /docs/gif_results/object_position_editing/cup_moving.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_position_editing/cup_moving.gif -------------------------------------------------------------------------------- /docs/gif_results/object_appearance_editing/swan_duck.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_appearance_editing/swan_duck.gif -------------------------------------------------------------------------------- /docs/gif_results/object_appearance_editing/truck_bus.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_appearance_editing/truck_bus.gif -------------------------------------------------------------------------------- /docs/gif_results/object_appearance_editing/truck_train.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_appearance_editing/truck_train.gif -------------------------------------------------------------------------------- /docs/gif_results/object_position_editing/astronaut_moving.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mayuelala/FollowYourHandle/HEAD/docs/gif_results/object_position_editing/astronaut_moving.gif -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |
2 | Your Image 3 |

MagicStick🪄: Controllable Video Editing via Control Handle Transformations

4 | 5 | 6 | [Yue Ma](https://mayuelala.github.io/), [Xiaodong Cun](http://vinthony.github.io/), [Yingqing He](https://github.com/YingqingHe), [Chenyang Qi](https://chenyangqiqi.github.io/), [Xintao Wang](https://xinntao.github.io/), [Ying Shan](https://scholar.google.com/citations?user=4oXBp9UAAAAJ&hl=zh-CN), [Xiu Li](https://scholar.google.com/citations?user=Xrh1OIUAAAAJ&hl=zh-CN) and [Qifeng Chen](https://cqf.io) 7 | 8 | 9 | [![GitHub](https://img.shields.io/github/stars/mayuelala/MagicStick?style=social)](https://github.com/mayuelala/MagicStick) 10 |
11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 |
"Zoom in the bird""Move the parrot""Zoom in the rabbit & rabbit ➜ tiger"
22 | 23 | 24 | ## 🎏 Abstract 25 | TL; DR: `MagicStick` is the first unified framework to modify video properties(e.g., shape, size, location, motion) leveraging the keyframe transformations on the extracted internal control signals. 26 | 27 |
CLICK for the full abstract 28 | Text-based video editing has recently attracted considerable interest in changing the style or replacing the objects with 29 | a similar structure. Beyond this, we demonstrate that properties such as shape, size, location, motion, etc., can also be 30 | edited in videos. Our key insight is that the keyframe’s transformations of the specific internal feature (e.g., edge maps 31 | of objects or human pose), can easily propagate to other frames to provide generation guidance. We thus propose MagicStick, 32 | a controllable video editing method that edits the video properties by utilizing the transformation on the extracted internal 33 | control signals. In detail, to keep the appearance, we inflate both the pretrained image diffusion model and ControlNet to 34 | the temporal dimension and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in editing, we perform 35 | an inversion and editing framework. Differently, finetuned ControlNet is introduced in both inversion and generation for 36 | attention guidance with the proposed attention remix between the spatial attention maps of inversion and editing. 37 | Yet succinct, our method is the first method to show the ability of video property editing from the pre-trained text-to-image model. 38 | We present experiments on numerous examples within our unified framework. We also compare with shape-aware text-based editing 39 | and handcrafted motion video generation, demonstrating our superior temporal consistency and editing capability than previous works. 40 |
41 | 42 | ## 📋 Changelog 43 | 44 | - 2023.12.01 Release Code and Paper! 45 | 46 | ## 🚧 Todo 47 | 48 | 55 | - [ ] Release the edit config and data for all results, Tune-a-video optimization 56 | - [ ] Memory and runtime profiling and Editing guidance documents 57 | - [ ] Colab and hugging-face 58 | - [ ] code refactoring 59 | - [ ] time & memory optimization 60 | - [ ] Release more application 61 | 62 | 230 | 231 | ## Object Size Editing 232 | We show the difference between the source prompt and the target prompt in the box below each video. 233 | 234 | Note mp4 and gif files in this GitHub page are compressed. 235 | Please check our [Project Page](https://magic-stick-edit.github.io/) for mp4 files of original video editing results. 236 | 237 | 238 | 239 | 240 | 241 | 243 | 244 | 245 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 261 | 262 |
263 | 264 | ## Object Position Editing 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 287 | 288 | 289 |
290 | 291 | ## Object Appearance Editing 292 | 293 | 294 | 295 | 296 | 297 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 |
"Truck ➜ Bus""Truck ➜ Train"
"A swan ➜ A flamingo""A swan ➜ A duck"
318 | 319 | 320 | 334 | 335 | ## 📀 Demo Video 336 | 337 | 338 | https://github.com/mayuelala/MagicStick/assets/38033523/20f689e1-7c38-4e77-b2c0-8e75b7601cd9 339 | 340 | 341 | 342 | 343 | ## 📍 Citation 344 | If you think this project is helpful, please feel free to leave a star⭐️⭐️⭐️ and cite our paper: 345 | ``` 346 | @article{ma2023magicstick, 347 | title={MagicStick: Controllable Video Editing via Control Handle Transformations}, 348 | author={Ma, Yue and Cun, Xiaodong and He, Yingqing and Qi, Chenyang and Wang, Xintao and Shan, Ying and Li, Xiu and Chen, Qifeng}, 349 | year={2023}, 350 | journal={arXiv:2312.03047}, 351 | } 352 | ``` 353 | 354 | 355 | ## 💗 Acknowledgements 356 | 357 | This repository borrows heavily from [FateZero](https://github.com/ChenyangQiQi/FateZero) and [FollowYourPose](https://github.com/mayuelala/FollowYourPose). Thanks to the authors for sharing their code and models. 358 | 359 | ## 🧿 Maintenance 360 | 361 | This is the codebase for our research work. We are still working hard to update this repo, and more details are coming in days. If you have any questions or ideas to discuss, feel free to contact [Yue Ma](mayuefighting@gmail.com) or [Xiaodong Cun](vinthony@gmail.com). 362 | 363 | --------------------------------------------------------------------------------