├── output0.png
├── 000_crop1.png
├── 000_crop_bad0.png
├── 000_crop_mask0.png
└── README.md


/output0.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/DonaldRR/Stable-diffusion-defect-generation/HEAD/output0.png


--------------------------------------------------------------------------------
/000_crop1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/DonaldRR/Stable-diffusion-defect-generation/HEAD/000_crop1.png


--------------------------------------------------------------------------------
/000_crop_bad0.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/DonaldRR/Stable-diffusion-defect-generation/HEAD/000_crop_bad0.png


--------------------------------------------------------------------------------
/000_crop_mask0.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/DonaldRR/Stable-diffusion-defect-generation/HEAD/000_crop_mask0.png


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Stable-diffusion-defect-generation
 2 | Previously [Stable Diffusion](https://github.com/runwayml/stable-diffusion) has great impact on image generation with supported unconditional and conditional generation. Especially, the conditional generation enpowers users to create specific types of "arts", manifesting a great bussiness potential to be discovered. Follow the work of Stable Diffusion, this repo shows a method to generate realistic texture defect with few defect samples, in our extreme case, only one sample is available.
 3 | 
 4 | (Code would be updated later)
 5 | 
 6 | # Method 
 7 | <img src="./000_crop_bad0.png" width="224"/>  <img src="./000_crop_mask0.png" width="224"/>  <img src="./000_crop1.png" width="224"/>  <img src="./output0.png" width="224"/>
 8 | 
 9 | From left to right: defect sample, defect mask, non-defect image, generated defect image
10 | 
11 | Inspired by the text2img and inpainting applicaions from Stable Diffusion, my propose method follows:
12 | 1. Mask out the defect area and concat the defect mask as input, following the inpainting training schedule
13 | 2. Use the CLIP image encoder to extract features from defect area
14 | 3. Project the defect features to proper dimension and inject to the cross-attention in Latent Diffusion Model
15 | 4. Freeze the Encoder&Decoder(`AutoencoderKL` I use)
16 | 5. Train the stable diffusion
17 | 


--------------------------------------------------------------------------------