├── README.md └── assets ├── pipeline.png └── task.png /README.md: -------------------------------------------------------------------------------- 1 | ## FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction 2 | 3 | Pytorch implementation of [FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction](https://arxiv.org/abs/2409.18071) 4 | 5 | Runze He, 6 | Kai Ma, 7 | Linjiang Huang, 8 | Shaofei Huang, 9 | Jialin Gao, 10 | Xiaoming Wei, 11 | Jiao Dai, 12 | Jizhong Han, 13 | Si Liu 14 | 15 | [![arXiv](https://img.shields.io/badge/ArXiv-2409.18071-brightgreen)](https://arxiv.org/abs/2409.18071) 16 | [![Project page](https://img.shields.io/badge/Project-Page-brightgreen)](https://freeedit.github.io/) 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 28 | 29 | 30 | ## Introduction 31 | 32 | 33 | 34 | FreeEdit consists of three components: (a) Multi-modal instruction encoder. (b) Detail extractor. (c) Denosing U-Net. Text instruction and reference image are firstly fed into the multi-modal instruction encoder to generate multi-modal instruction embedding. The reference image is additionally fed into the detail extractor to obtain fine-grained features. The original image latent is concatenated with the noise latent to introduce the original image condition. Denosing U-Net accepts the 8-channel input and interacts with the multi-modal instruction embedding through cross-attention. The DRRA modules which connect the detail extractor and the denoising U-Net, are used to integrate fine-grained features from the detail extractor to promote ID consistency with the reference image. (d) The editing examples obtained using FreeEdit. 35 | 36 | 37 | 38 | 39 | ## Citation 40 | 41 | ```bibtex 42 | @misc{he2024freeedit, 43 | title={FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction}, 44 | author={Runze He and Kai Ma and Linjiang Huang and Shaofei Huang and Jialin Gao and Xiaoming Wei and Jiao Dai and Jizhong Han and Si Liu}, 45 | year={2024}, 46 | eprint={2409.18071}, 47 | archivePrefix={arXiv}, 48 | primaryClass={cs.CV}, 49 | url={https://arxiv.org/abs/2409.18071}, 50 | } 51 | ``` 52 | 53 | ## Contact 54 | 55 | If you have any comments or questions, feel free to contact [Runze He](https://github.com/hrz2000). -------------------------------------------------------------------------------- /assets/pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hrz2000/FreeEdit/115a63b128f23d11e0829c2f3e2f4a613c8f6746/assets/pipeline.png -------------------------------------------------------------------------------- /assets/task.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hrz2000/FreeEdit/115a63b128f23d11e0829c2f3e2f4a613c8f6746/assets/task.png --------------------------------------------------------------------------------