├── sources ├── img2img.png ├── origin.png ├── img2img_depth.png ├── inpaint_face.png ├── face_swap&img2img.png ├── img2img_zjz_lora.png ├── img2img_depth_softedge.png └── inpaint_face_and_neck.png └── README.md /sources/img2img.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/img2img.png -------------------------------------------------------------------------------- /sources/origin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/origin.png -------------------------------------------------------------------------------- /sources/img2img_depth.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/img2img_depth.png -------------------------------------------------------------------------------- /sources/inpaint_face.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/inpaint_face.png -------------------------------------------------------------------------------- /sources/face_swap&img2img.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/face_swap&img2img.png -------------------------------------------------------------------------------- /sources/img2img_zjz_lora.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/img2img_zjz_lora.png -------------------------------------------------------------------------------- /sources/img2img_depth_softedge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/img2img_depth_softedge.png -------------------------------------------------------------------------------- /sources/inpaint_face_and_neck.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ykk648/miaoya_pipeline/HEAD/sources/inpaint_face_and_neck.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## miaoya_pipeline 2 | 3 | 复现妙鸭相机证件照生成的pipeline 4 | 5 | 结论： V2的效果最好，V3的速度最快，但是面临妆容和ID不够匹配的问题，如果能训练一个 天真蓝/海马体/古装照 半身lora作为基底模型，效果会更好 6 | 7 | 此后开源的一些工作： 8 | 9 | - 2023.8 [facechain](https://github.com/modelscope/facechain) 工具链，解决一些工程问题，在训练lora前处理增加了landmark对齐旋转 10 | - 2023.9 [sd-webui-EasyPhoto](https://github.com/aigc-apps/sd-webui-EasyPhoto) 易用性改进，增加通过landmark对齐+openpose controlnet监督inpainting过程 11 | 12 | ### V1 inpainting 13 | 14 | Head/face inpainting based on sd-webui and lora models from civitai: 15 | 16 | | origin | face inpainting | face&neck inpainting | 17 | |:-------------------------:|:-------------------------------:|------------------------------------------| 18 | | ![](./sources/origin.png) | ![](./sources/inpaint_face.png) | ![](./sources/inpaint_face_and_neck.png) | 19 | 20 | - using chilloutmix as base model (realistic asian women) 21 | - using tagger(wd14) for prompts reverse 22 | - using textual inversion negative prompts (ng_deepnegative_v1_75t,badhandv4) 23 | - optim mask blur for cheek edge >15 24 | - DPM++SDE Karras sample step 30 for speed up 25 | - controlnet reference 0.5 26 | - controlnet softedge_pidinet 0.75 27 | - controlnet openpose face 0.7 28 | 29 | ### V2 inpainting + img2img 30 | 31 | Fix face color different from background, more realistic 32 | 33 | | face&neck inpainting | img2img depth | img2img depth&softedge | 34 | |:----:|:--------------------------------:|------------------------------------------| 35 | | ![](./sources/inpaint_face_and_neck.png) | ![](./sources/img2img_depth.png) | ![](./sources/img2img_depth_softedge.png)| 36 | 37 | - have tried more realistic/style/portrait lora(not shown here) 38 | - 由于没有风格lora约束，人物lora会带入风格（假设该lora训练时拟合了五官以外信息） 39 | 40 | ### V3 faceswap + img2img 41 | 42 | Only do one time img2img and can be faster. 43 | 44 | | origin | img2img | faceswap + img2img | 45 | |:----:|:--------------------------:|--------------------------------------| 46 | | ![](./sources/origin.png) | ![](./sources/img2img.png) | ![](./sources/face_swap&img2img.png) | 47 | 48 | 49 | - using [inswapper](https://github.com/deepinsight/insightface/blob/master/python-package/insightface/model_zoo/inswapper.py) for faceswap 50 | - private face swap model (not shown here) 51 | - reference/softedge_pidinet/depth to control details 52 | 53 | ## supply materials 54 | 55 | ### lora training experience 56 | 57 | using 20 upper body images to train a face lora model 58 | 59 | 20张半身像（参考妙鸭）稳定训练出一个比较好的lora还是有一点困难，目前一些tips： 60 | 61 | - 产品逻辑上，参考妙鸭，四组参数训练四个模型供用户选择 62 | - 有人在用证件照训练lora底模 [知乎](https://www.zhihu.com/question/613419782/answer/3133599555) 63 | - 不做任何优化的前提下，20张图片A100大约20分钟收敛（batch4 不考虑显存和多路复用），还可以更快（降低rank等） 64 | - 专注训练脸部lora（加入五官提示词） 65 | - 一些更新的社区工作（Lion SDXL LyCORIS等等）,lora在同类型图片（半身像）上预训练后可能会提升效果/减少训练时间 66 | 67 | 68 | 69 | ### Support 70 | 71 | - base model from [chilloutmix](https://civitai.com/models/6424) 72 | - lora from on [iu](https://civitai.com/models/11722/iu) 73 | - origin photo from 小红书 74 | 75 | ## TODO 76 | 77 | - [ ] pipeline extract from sd-webui based on sd-api 78 | - [ ] lora training deploy%speedup 79 | - [ ] 妙鸭数据爬取和对应的lora风格训练 --------------------------------------------------------------------------------