├── LICENSE ├── README.md ├── discontinued ├── local │ ├── HISTORY.md │ ├── README.md │ ├── gimp-stable-diffusion-local.py │ └── version.json └── notebook │ ├── HISTORY.md │ ├── README.md │ ├── gimp-stable-diffusion.ipynb │ └── gimp-stable-diffusion.py └── stablehorde ├── HISTORY.md ├── README.md ├── gimp-stable-diffusion-horde.py └── version.json /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 blueturtleai 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # gimp-stable-diffusion 2 | This repository includes a GIMP plugin, which can be used for generating images with stable-diffusion: 3 | 4 | ### This plugin is no longer actively developed. 5 | 6 | ### Stablehorde 7 | This plugin can be used without running a stable-diffusion server yourself. It uses [Stablehorde](https://stablehorde.net) as the backend. Stablehorde is a cluster of stable-diffusion servers run by volunteers. [Check it out](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/stablehorde). 8 | -------------------------------------------------------------------------------- /discontinued/local/HISTORY.md: -------------------------------------------------------------------------------- 1 | # History 2 | ## GIMP Plugin 3 | ### 1.1.0 4 | Changes 5 | - Inpainting is supported now. 6 | - Image sizes between 384x384 and 1024x1024 are supported now. Before only 512x512. If one side is 1024, the other can be max. 768. 7 | 8 | ### 1.0.0 9 | Initial version 10 | -------------------------------------------------------------------------------- /discontinued/local/README.md: -------------------------------------------------------------------------------- 1 | # gimp-stable-diffusion-local 2 | 3 | This repository includes a GIMP plugin for communication with a locally installed stable-diffusion server. Linux, macOS and Windows 11 (WSL2) are supported. Please check the section "Limitations" to better understand where the limits are. 4 | 5 | Please check HISTORY.md for the latest changes. 6 | 7 | ## Installation 8 | ### Download files 9 | 10 | To download the files of this repository click on "Code" and select "Download ZIP". In the ZIP you will find the file "gimp-stable-diffusion-local.py" in the subfolder "local". This is the code for the GIMP plugin. You don't need the other files in the ZIP. 11 | 12 | ### GIMP 13 | 14 | The plugin has been tested in GIMP 2.10, but should run in all 2.* releases. Excluded is 2.99, as this is already based on Python 3. 15 | 16 | 1. Start GIMP and open the preferences dialog via "edit/preferences" and scroll down to "folders". Expand "folders" and click on "plug-ins". Select the folder which includes your username and copy the path. 17 | 18 | 2. Open the file explorer, navigate to this directory and copy the file "gimp-stable-diffusion.py" from the repository into this directory. If you are on MacOS or Linux, change the file permissions to 755. 19 | 20 | 3. Restart GIMP. You should now see the new menu "AI". If you don't see this, something went wrong. Please check in this case "Troubleshooting/GIMP" for possible solutions. The menu has one item "Stablehorde text2img". This item can't currently be selected. This only works, when you opened an image before. More about this below. 21 | 22 | ### Stable-Diffusion server 23 | #### Prerequisites 24 | You need an account on huggingface.co for downloading the model file. If you want to run on Windows 11, WSL2 needs to be prepared before. Details follow below. 25 | 26 | #### Model file 27 | The model file includes all the data which is needed for stable-diffusion to generate the images. 28 | 1. Create an account on https://huggingface.co. 29 | 30 | 2. Nagivate here https://huggingface.co/CompVis/stable-diffusion-v-1-4-original and agree to the shown agreement. 31 | 32 | 3. Go to "Settings/Access Tokens" on the left side. 33 | 34 | 4. Click on "New Token", enter a name, select "Read" as role, click on create and copy the token. 35 | 36 | #### Local server 37 | If you want to run the server on Windows 11, WSL2 needs to be prepared before. Please follow [these instructions](https://github.com/blueturtleai/cog/blob/main/docs/wsl2/wsl2.md) until after you reach executing "wsl.exe" in 7. Afterwards please start with 2. from the instructions below, as Docker is already installed. 38 | 39 | **1. Install and start docker** 40 | 41 | Please don't execute these commands, if you are running on Windows 11. Jump directly to 2. 42 | 43 | ``` 44 | sudo yum update -y 45 | sudo yum -y install docker 46 | sudo service docker start 47 | sudo usermod -aG docker ${USER} 48 | ``` 49 | Log off and log in again. 50 | 51 | **2. Install cog and cog-stable-diffusion** 52 | ``` 53 | sudo curl -o /usr/local/bin/cog -L https://github.com/replicate/cog/releases/download/v0.4.4/cog_`uname -s`_`uname -m` 54 | sudo chmod +x /usr/local/bin/cog 55 | sudo yum install git -y 56 | git clone https://github.com/blueturtleai/cog-stable-diffusion 57 | cd cog-stable-diffusion 58 | ``` 59 | 60 | **3. Install stable-diffusion server** 61 | ``` 62 | cog run script/download-weights 63 | sudo chown -R $USER . 64 | ``` 65 | 66 | **4. Create docker image and start stable-diffusion server** 67 | ``` 68 | cog build -t stable-diffusion 69 | docker run -d -p 5000:5000 --gpus all stable-diffusion 70 | ``` 71 | 72 | **5. Test stable-diffusion server** 73 | 74 | Wait a minute after server start. 75 | ``` 76 | curl http://localhost:5000/predictions -X POST \ 77 | -H 'Content-Type: application/json' \ 78 | -d '{"input": {"prompt": "A beautiful day, digital painting"}}' 79 | ``` 80 | If the test was successful, the generated image will be shown as a base64 encoded very long string. 81 | 82 | **6. Stop stable-diffusion server** 83 | 84 | If you are done generating images, you can stop the server that way. 85 | 86 | Get the container name. 87 | ``` 88 | docker ps -q 89 | ``` 90 | Stop the container. 91 | ``` 92 | docker stop 93 | ``` 94 | If you want to run the stable-diffusion server the next time, just execute 95 | ``` 96 | docker run -d -p 5000:5000 --gpus all stable-diffusion 97 | ``` 98 | 99 | ## Generate images 100 | Now we are ready for generating images. 101 | 102 | 1. Start GIMP and open any image or create a new one. If you want to use img2img, open the init image. 103 | 104 | 2. Select the new "AI/Stable Local" menu item. A dialog will open, where you can enter the details for the image generation. 105 | 106 | - **Generation Mode:** 107 | - **Text -> Image:** Generate an image based on your prompt. 108 | - **Image -> Image:** Generate an image based on an init image and on your prompt. 109 | - **Inpainting:** Erase a part of an image and generate a new image which has the erased part filled. The erased part is filled based on your prompt. Please read the section "Inpainting" below for an explanation how inpainting works. 110 | 111 | - **Init Strength:** How much the AI should take the init image into account. The higher the value, the more will the generated image look like the init image. 0.3 is a good value to use. 112 | 113 | - **Prompt Strength:** How much the AI should follow the prompt. The higher the value, the more the AI will generate an image which looks like your prompt. 7.5 is a good value to use. 114 | 115 | - **Steps:** How many steps the AI should use to generate the image. The higher the value, the more the AI will work on details. But it also means, the longer the generation takes and the more the GPU is used. 50 is a good value to use. 116 | 117 | - **Seed:** This parameter is optional. If it is empty, a random seed will be generated on the server. If you use a seed, the same image is generated again in the case the same parameters for init strength, steps, etc. are used. A slightly different image will be generated, if the parameters are modified. You find the seed in an additional layer at the top left. 118 | 119 | - **Prompt:** How the generated image should look like. 120 | 121 | - **Backend root URL:** Insert the URL of your local server. If you used the instructions from above, the URL should look like this ```http://localhost:5000```. 122 | 123 | 3. Click on the OK button. The values you inserted into the dialog will be transmitted to the server. When the image has been generated successfully, it will be shown as a new image in GIMP. The used seed is shown at the top left in an additional layer. 124 | 125 | ## Inpainting 126 | Inpainting means replacing a part of an existing image. For example if you don't like the face on an image, you can replace it. **Inpainting is currently still in experimental stage. So, please don't expect perfect results.** The experimental stage is caused by the server side and not by GIMP. 127 | 128 | For inpainting it's necessary to prepare the input image because the AI needs to know which part you want to replace. For this purpose you replace this image part by transparency. To do so, open the init image in GIMP and select "Layer/Transparency/Add alpha channel". Select now the part of the image which should be replaced and delete it. You can also use the eraser tool. 129 | 130 | For the prompt you use now a description of the new image. For example the image shows currently "a little girl running over a meadow with a balloon" and you want to replace the balloon by a parachute. You just write now "a little girl running over a meadow with a parachute". 131 | 132 | ## Limitations 133 | - **Testing:** The local server runs on Linux, macOS and Windows 11. Due to limited availability, it has only been tested on Linux. If you run in any problems on macOS or Windows 11, please open an issue. 134 | 135 | - **NSFW:** The server doesn't support NSFW (Not Safe For Work) images. If the server generates a NSFW image, an error is displayed in GIMP. 136 | 137 | ## Troubleshooting 138 | ### GIMP 139 | #### AI menu is not shown 140 | ##### Linux 141 | - If you get this error ```gimp: LibGimpBase-WARNING: gimp: gimp_wire_read(): error```, it's very likely, that you have a GIMP version installed, which doesn't include Python. Check, if you have got the menu "Filters > Python-Fu > Console". If it is missing, please install GIMP from here: https://flathub.org/apps/details/org.gimp.GIMP. 142 | 143 | - Please try https://flathub.org/apps/details/org.gimp.GIMP if you have got any other problems. 144 | 145 | ##### macOS 146 | - Please double check, if the permissions of the plugin py file are set to 755. It seems, that changing permissions doesn't work via the file manager. Please open a terminal, cd to the plugins directory and run "chmod ugo+x *py". 147 | 148 | ##### macOS/Linux 149 | - Open a terminal an try to run the plugin py file manually via ```python /gimp-stable-diffusion.py```. You should see the error message, that "gimpfu" is unknown. Make sure, that you are running Python 2, as this version is used by GIMP. If other errors occur, please reinstall GIMP. 150 | 151 | ## FAQ 152 | 153 | **Will GIMP 3 be supported?** Yes, the plugin will be ported to GIMP 3. 154 | 155 | **Will Out-Painting be supported?** This depends on which features the cog stable-diffusion server will support in the future. 156 | 157 | **How do I report an error or request a new feature?** Please open a new issue in this repository. 158 | 159 | 160 | -------------------------------------------------------------------------------- /discontinued/local/gimp-stable-diffusion-local.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # v1.1.0 4 | 5 | import urllib2 6 | import tempfile 7 | import os 8 | import base64 9 | import json 10 | import re 11 | import random 12 | import math 13 | import gimp 14 | from gimpfu import * 15 | 16 | VERSION = 110 17 | INIT_FILE = "init.png" 18 | GENERATED_FILE = "generated.png" 19 | API_ENDPOINT = "predictions" 20 | 21 | initFile = r"{}".format(os.path.join(tempfile.gettempdir(), INIT_FILE)) 22 | generatedFile = r"{}".format(os.path.join(tempfile.gettempdir(), GENERATED_FILE)) 23 | 24 | def checkUpdate(): 25 | try: 26 | gimp.get_data("update_checked") 27 | updateChecked = True 28 | except Exception as ex: 29 | updateChecked = False 30 | 31 | if updateChecked is False: 32 | try: 33 | url = "https://raw.githubusercontent.com/blueturtleai/gimp-stable-diffusion/main/local/version.json" 34 | response = urllib2.urlopen(url) 35 | data = response.read() 36 | data = json.loads(data) 37 | gimp.set_data("update_checked", "1") 38 | 39 | if VERSION < int(data["version"]): 40 | pdb.gimp_message(data["message"]) 41 | except Exception as ex: 42 | ex = ex 43 | 44 | def cleanup(): 45 | try: 46 | if os.path.exists(initFile): 47 | os.remove(initFile) 48 | 49 | if os.path.exists(generatedFile): 50 | os.remove(generatedFile) 51 | except Exception as ex: 52 | ex = ex 53 | 54 | def getImages(data, seed): 55 | images = [] 56 | 57 | for counter in range(len(data["output"])): 58 | image = re.match("data:image/png;base64,(.*)", data["output"][counter]).group(1) 59 | image = {"img": image, "seed": seed} 60 | images.append(image) 61 | 62 | return images 63 | 64 | def getImageData(image, drawable): 65 | pdb.file_png_save_defaults(image, drawable, initFile, initFile) 66 | initImage = open(initFile, "rb") 67 | encoded = base64.b64encode(initImage.read()) 68 | return encoded 69 | 70 | def displayGenerated(images): 71 | color = pdb.gimp_context_get_foreground() 72 | pdb.gimp_context_set_foreground((0, 0, 0)) 73 | 74 | for image in images: 75 | imageFile = open(generatedFile, "wb+") 76 | imageFile.write(base64.b64decode(image["img"])) 77 | imageFile.close() 78 | 79 | imageLoaded = pdb.file_png_load(generatedFile, generatedFile) 80 | pdb.gimp_display_new(imageLoaded) 81 | # image, drawable, x, y, text, border, antialias, size, size_type, fontname 82 | pdb.gimp_text_fontname(imageLoaded, None, 2, 2, str(image["seed"]), -1, TRUE, 12, 1, "Sans") 83 | pdb.gimp_image_set_active_layer(imageLoaded, imageLoaded.layers[1]) 84 | 85 | pdb.gimp_context_set_foreground(color) 86 | return 87 | 88 | def generate(image, drawable, mode, initStrength, promptStrength, steps, seed, prompt, url): 89 | if image.width < 384 or image.width > 1024 or image.height < 384 or image.height > 1024: 90 | raise Exception("Invalid image size. Image needs to be between 384x384 and 1024x1024.") 91 | 92 | if image.width * image.height > 786432: 93 | raise Exception("Invalid image size. Maximum size is 1024x768 or 768x1024.") 94 | 95 | if prompt == "": 96 | raise Exception("Please enter a prompt.") 97 | 98 | if mode == "MODE_INPAINTING" and drawable.has_alpha == 0: 99 | raise Exception("Invalid image. For inpainting an alpha channel is needed.") 100 | 101 | pdb.gimp_progress_init("", None) 102 | 103 | input = { 104 | "prompt": prompt, 105 | "num_inference_steps": int(steps), 106 | "guidance_scale": float(promptStrength) 107 | } 108 | 109 | if image.width % 64 != 0: 110 | width = math.floor(image.width/64) * 64 111 | else: 112 | width = image.width 113 | 114 | if image.height % 64 != 0: 115 | height = math.floor(image.height/64) * 64 116 | else: 117 | height = image.height 118 | 119 | input.update({"width": int(width)}) 120 | input.update({"height": int(height)}) 121 | 122 | if not seed: 123 | seed = random.randint(0, 2**31) 124 | else: 125 | seed = int(seed) 126 | 127 | input.update({"seed": seed}) 128 | 129 | if mode == "MODE_IMG2IMG" or mode == "MODE_INPAINTING": 130 | imageData = getImageData(image, drawable) 131 | input.update({"init_image": imageData}) 132 | input.update({"prompt_strength": (1 - float(initStrength))}) 133 | 134 | if mode == "MODE_INPAINTING": 135 | input.update({"mask": imageData}) 136 | 137 | data = {"input": input} 138 | data = json.dumps(data) 139 | 140 | headers = {"Content-Type": "application/json", "Accept": "application/json"} 141 | 142 | url = url + "/" if not re.match(".*/$", url) else url 143 | url = url + API_ENDPOINT 144 | 145 | request = urllib2.Request(url=url, data=data, headers=headers) 146 | pdb.gimp_progress_set_text("starting dreaming now...") 147 | 148 | try: 149 | response = urllib2.urlopen(request) 150 | data = response.read() 151 | 152 | try: 153 | data = json.loads(data) 154 | except Exception as ex: 155 | raise Exception(data) 156 | 157 | if data["status"] == "failed": 158 | raise Exception("The image couldn't be generated: " + data["error"]) 159 | 160 | images = getImages(data, seed) 161 | displayGenerated(images) 162 | except Exception as ex: 163 | raise ex 164 | finally: 165 | pdb.gimp_progress_end() 166 | cleanup() 167 | checkUpdate() 168 | 169 | return 170 | 171 | register( 172 | "stable-local", 173 | "stable-local", 174 | "stable-local", 175 | "BlueTurtleAI", 176 | "BlueTurtleAI", 177 | "2022", 178 | "/AI/Stable Local", 179 | "*", 180 | [ 181 | (PF_RADIO, "mode", "Generation Mode", "MODE_TEXT2IMG", ( 182 | ("Text -> Image", "MODE_TEXT2IMG"), 183 | ("Image -> Image", "MODE_IMG2IMG"), 184 | ("Inpainting", "MODE_INPAINTING") 185 | )), 186 | (PF_SLIDER, "initStrength", "Init Strength", 0.3, (0.0, 1.0, 0.1)), 187 | (PF_SLIDER, "promptStrength", "Prompt Strength", 7.5, (0, 20, 0.5)), 188 | (PF_SLIDER, "steps", "Steps", 50, (10, 150, 1)), 189 | (PF_STRING, "seed", "Seed (optional)", ""), 190 | (PF_STRING, "prompt", "Prompt", ""), 191 | (PF_STRING, "url", "Backend root URL", "") 192 | ], 193 | [], 194 | generate 195 | ) 196 | 197 | main() 198 | -------------------------------------------------------------------------------- /discontinued/local/version.json: -------------------------------------------------------------------------------- 1 | {"version": 110, "message": "New version of stable-diffusion local plugin is available. Please update! Please also rebuild your server."} 2 | -------------------------------------------------------------------------------- /discontinued/notebook/HISTORY.md: -------------------------------------------------------------------------------- 1 | # History 2 | ## GIMP Plugin 3 | ### Discontinued 4 | The GIMP plugin notebook was based on the deforum notebook. As this notebook is broking now, the GIMP plugin notebook doesn't work any longer either. To make it working again, a complete rewrite would be necessary. This would happen in the future again again. For this reason and combined with the fact that the usage of the GIMP notebook seems to be pretty low, the GIMP plugin for colab will be discontinued. Please check out the [GIMP plugin, which uses stablehorde](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/stablehorde) as an alternative. 5 | 6 | ### 1.2.0 7 | #### Changes 8 | - Now text2img is supported. 9 | - The dialog has been optimized. 10 | 11 | ### 1.1.2 12 | #### Changes 13 | - Exception handling further improved 14 | 15 | ### 1.1.1 16 | #### Changes 17 | - Data is now transferred to the server using https 18 | 19 | ### 1.1.0 20 | #### Changes 21 | - Inpainting is now supported 22 | 23 | ### 1.0.2 24 | #### Changes 25 | - **Image count**: It is now possible to create up to 4 images in one run 26 | 27 | - **Seed**: Can now be blank if not used (before -1) 28 | 29 | - **Backend URL**: Needs no longer to end with "/" 30 | 31 | - **Exception handling**: Has been improved 32 | 33 | ### 1.0.1 34 | #### CHANGES 35 | - **API**: Data is now transferred as JSON. It is checked, if the API version of the client and the server matches. If mismatch, user is requested to update plugin and/or server. 36 | 37 | - **Seed**: The seed of the generated image is now displayed in an additional layer. It can be passed to the server to generate the same image again. This only works, if the same parameters (init strength, number of steps, etc) are used. 38 | 39 | ### 1.0.0 40 | Initial version 41 | 42 | ## Stable-Diffusion server 43 | ### Discontinued 44 | The GIMP plugin notebook was based on the deforum notebook. As this notebook is broking now, the GIMP plugin notebook doesn't work any longer either. To make it working again, a complete rewrite would be necessary. This would happen in the future again again. For this reason and combined with the fact that the usage of the GIMP notebook seems to be pretty low, the GIMP plugin for colab will be discontinued. Please check out the [GIMP plugin, which uses stablehorde](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/stablehorde) as an alternative. 45 | 46 | ### 1.3.0 47 | Changes 48 | - Now text2img is supported. 49 | 50 | ### 1.2.1 51 | #### Changes 52 | - Stable-Diffusion Model v1.5 is now supported. 53 | 54 | ### 1.2.0 55 | #### Changes 56 | - It's no longer necessary to upload model files yourself to your Google drive. If a model file is missing, it is automatically downloaded from Huggingface. A Huggingface account and acceptance of the terms of service is still needed. 57 | 58 | ### 1.1.2 59 | #### Changes 60 | - Instead of ngrok now cloudflare is used as the tunnel provider. The switch is done because several users reported problems. Additional benefit is, that it's no longer necessary to register on ngrok and enter the authkey. Thanks for suggesting this @opencoca. 61 | 62 | ### 1.1.1 63 | #### Changes 64 | - Mounting Google drive and setting model path are now two separated steps 65 | 66 | ### 1.1.0 67 | #### Changes 68 | - The notebook is now based on the deforum notebook (before pharmapsychotic) 69 | - Inpainting is now supported 70 | 71 | ### 1.0.3 72 | #### Changes 73 | - **Image count**: It is now possible to create up to 4 images in one run 74 | 75 | ### 1.0.2 76 | #### Changes 77 | - **API**: Data is now transferred in JSON. It is checked, if API version of the client and server matches. If mismatch, code 405 is returned. 78 | 79 | - **Seed**: If a seed != -1 is transferred to the server, this seed is used for image generation. 80 | 81 | ### 1.0.1 82 | #### Changes 83 | - **Image dimensions**: The image dimensions are now no longer fixed to 512x512. The dimensions of the init image are now used instead. The dimensions are adjusted to a multiple of 64. 84 | 85 | ### 1.0.0 86 | Initial version 87 | -------------------------------------------------------------------------------- /discontinued/notebook/README.md: -------------------------------------------------------------------------------- 1 | # gimp-stable-diffusion 2 | 3 | This repository includes a GIMP plugin for communication with a stable-diffusion server and a Google colab notebook for running the server. 4 | 5 | ### Discontinued 6 | The GIMP plugin notebook was based on the deforum notebook. As this notebook is broking now, the GIMP plugin notebook doesn't work any longer either. To make it working again, a complete rewrite would be necessary. This would happen in the future again again. For this reason and combined with the fact that the usage of the GIMP notebook seems to be pretty low, the GIMP plugin for colab will be discontinued. Please check out the [GIMP plugin, which uses stablehorde](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/stablehorde) as an alternative. 7 | 8 | **More flavours of the plugin are available:** 9 | - Free image generation without running a Colab notebook or a local server: [GIMP plugin](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/stablehorde) 10 | - Image generation with a local server: [GIMP plugin](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/local) 11 | 12 | Please check HISTORY.md for the latest changes. 13 | 14 | Click here [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/blueturtleai/gimp-stable-diffusion/blob/main/gimp-stable-diffusion.ipynb) to open the notebook, if you already read the setup instructions below. 15 | 16 | ## Overview 17 | 18 | The server exposes a REST API, which is used by the GIMP plugin to communicate with the server. Currently the plugins offers text2img, img2img and inpainting. 19 | 20 | https://user-images.githubusercontent.com/113246030/190710535-9fb23f88-954f-4f73-afea-1475c8690754.MOV 21 | 22 | ## Manual 23 | It doesn't exist a separate manual. Please check the following sections for installation and image generation for a detailed explanation. 24 | 25 | ## Installation 26 | ### Download files 27 | 28 | To download the files of this repository click on "Code" and select "Download ZIP". In the ZIP you will find the file "gimp-stable-diffusion.py". This is the code for the GIMP plugin. You don't need the other files in the ZIP. 29 | 30 | ### GIMP 31 | 32 | The plugin has been tested in GIMP 2.10, but should run in all 2.* releases. Excluded is 2.99, as this is already based on Python 3. 33 | 34 | 1. Start GIMP and open the preferences dialog via edit/preferences and scroll down to "folders". Expand "folders" and click on "plug-ins". Select the folder which includes your username and copy the path. 35 | 36 | 2. Open the file explorer, navigate to this directory and copy the file "gimp-stable-diffusion.py" from the repository into this directory. If you are on MacOS or Linux, change the file permissions to 755. 37 | 38 | 3. Restart GIMP. You should now see the new menu "AI". If you don't see this, something went wrong. Please check in this case "Troubleshooting/GIMP" for possible solutions. The menu has one item "Stable Colab". This item can't currently be selected. This only works, when you opened an image before. More about this, when the server is running. 39 | 40 | ### Stable-Diffusion server 41 | #### Prerequisites 42 | You need a Google account and on huggingface.co. Google is needed for running a colab server and huggingface for downloading the model file. Details follow below. 43 | 44 | #### Model file 45 | The model file includes all the data which is needed for stable-diffusion to generate the images. 46 | 1. Create an account on https://huggingface.co. 47 | 48 | 2. Nagivate here https://huggingface.co/CompVis/stable-diffusion-v-1-4-original and agree to the shown agreement. 49 | 50 | 3. Go to "Settings/Access Tokens" on the left side. 51 | 52 | 4. Click on "New Token", enter a name, select "Read" as role, click on create and copy the token. 53 | 54 | #### Colab server 55 | 1. Open this link in a new tab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/blueturtleai/gimp-stable-diffusion/blob/main/gimp-stable-diffusion.ipynb) 56 | 57 | 2. Click on "connect" and wait until the status changes to "connected". 58 | 59 | 3. Click on the arrow left to "NVIDIA GPU" and wait until you see a checkmark on the left. 60 | 61 | 4. Click on the arrow left to "Mount Google Drive" and confirm the mount. Wait until you see a checkmark on the left. 62 | 63 | 5. Set the model path. It is recommended to use the default path. That way you don't have to adjust the path manually every time. If the model file doesn't exist at this location, it is automatically downloaded from Huggingface. Please make sure you have enough free space on your drive (about 4 GB). If you choose an individual path, please read the hints in the notebook. Execute the step. 64 | 65 | 6. Execute the step "Setup Environment". 66 | 67 | 7. Execute the step "Python Definitions". 68 | 69 | 8. Execute the step "Select and Load Model". In the selector for the model files there is currently only one entry. When v1.5 has been released, this model will be added to the selector. If the model file doesn't exist at the location you selected above, it will automatically be downloaded from Huggingface. Please make sure you have enough free space on your drive (about 4 GB). It is necessary, that you created an account on Huggingface and accepted the terms of service. Otherwise it's not possible to download the file. You will also need the Huggingface access token you created before. 70 | 71 | 9. Click on the arrow left to "Waiting for GIMP requests". The arrow on the left won't stop spinning in this case. If everything is okay, you should see something like this: 72 | ``` 73 | * Serving Flask app "__main__" (lazy loading) 74 | * Environment: production 75 | WARNING: This is a development server. Do not use it in a production deployment. 76 | Use a production WSGI server instead. 77 | * Debug mode: off 78 | 79 | INFO:werkzeug: * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 80 | 81 | * Running on https://*.trycloudflare.com <- copy this URL 82 | * Traffic stats available on http://127.0.0.1:4040 83 | ``` 84 | 85 | 10. Copy the URL from above, which reads like ```https://*.trycloudflare.com```. This is the URL, which is used for the communication between the GIMP plugin and the server. 86 | 87 | ## Generate images 88 | Now we are ready for generating images. 89 | 90 | 1. Start GIMP and open an image or create a new one. It is recommended, that the image size is not larger than 512x512 as the model has been trained on this size. If you want to have larger images, use an external upscaler instead. The generated image will have the dimensions of the init image. But it may be resized to make sure, that the dimensions are a multiple of 64. The larger the image, the longer it takes to generate it and the more GPU ressources and RAM is used. If it is too larger, you will run out of memory. 91 | 92 | 2. Select the new AI/Stable Colab menu item. A dialog will open, where you can enter the details for the image generation. 93 | 94 | - **Generation Mode:** 95 | - **Text -> Image:** Generate an image based on your prompt. 96 | - **Image -> Image:** Generate an image based on an init image and on your prompt. 97 | - **Inpainting:** Erase a part of an image and generate a new image which has the erased part filled. The erased part is filled based on your prompt. Please read the section "Inpainting" below for an explanation how inpainting works. 98 | 99 | - **Init Strength:** How much the AI should take the init image into account. The higher the value, the more will the generated image look like the init image. 0.3 is a good value to use. 100 | 101 | - **Prompt Strength:** How much the AI should follow the prompt. The higher the value, the more the AI will generate an image which looks like your prompt. 7.5 is a good value to use. 102 | 103 | - **Steps:** How many steps the AI should use to generate the image. The higher the value, the more the AI will work on details. But it also means, the longer the generation takes and the more the GPU is used. 50 is a good value to use. 104 | 105 | - **Seed:** This parameter is optional. If it is empty, a random seed will be generated on the server. If you use a seed, the same image is generated again in the case the same parameters for init strength, steps, etc. are used. A slightly different image will be generated, if the parameters are modified. You find the seed in an additional layer at the top left. 106 | 107 | - **Number of images:** Number of images, which are created in one run. The more images you create, the more server ressources will be used and the longer you have to wait until the generated images are displayed in GIMP. 108 | 109 | - **Prompt:** How the generated image should look like. 110 | 111 | - **Backend root URL:** Insert the trycloudflare.com URL you copied from the server. The URL should look like this ```https://*.trycloudflare.com```. 112 | 113 | 3. Click on the OK button. The values you inserted into the dialog and the init image will be transmitted to the server, which starts now generating the image. On the colab browser tab you can see what's going on. When the image has been generated successfully, it will be shown as a new image in GIMP. The used seed is shown at the top left in an additional layer. 114 | 115 | ### Inpainting 116 | Inpainting means replacing a part of an existing image. For example if you don't like the face on an image, you can replace it. **Inpainting is currently still in experimental stage. So, please don't expect perfect results.** The experimental stage is caused by the server side and not by GIMP. 117 | 118 | For inpainting it's necessary to prepare the input image because the AI needs to know which part you want to replace. For this purpose you replace this image part by transparency. To do so, open the init image in GIMP and select "Layer/Transparency/Add alpha channel". Select now the part of the image which should be replaced and delete it. You can also use the eraser tool. 119 | 120 | For the prompt you use now a description of the new image. For example the image shows currently "a little girl running over a meadow with a balloon" and you want to replace the balloon by a parachute. You just write now "a little girl running over a meadow with a parachute". 121 | 122 | ## Troubleshooting 123 | ### GIMP 124 | #### AI menu is not shown 125 | ##### Linux 126 | - If you get this error ```gimp: LibGimpBase-WARNING: gimp: gimp_wire_read(): error```, it's very likely, that you have a GIMP version installed, which doesn't include Python. Check, if you have got the menu "Filters > Python-Fu > Console". If it is missing, please install GIMP from here: https://flathub.org/apps/details/org.gimp.GIMP. 127 | 128 | ##### macOS 129 | - Please double check, if the permissions of the plugin py file are set to 755. It seems, that changing permissions doesn't work via the file manager. Please open a terminal, cd to the plugins directory and run "chmod ugo+x *py". 130 | 131 | ##### macOS/Linux 132 | - Open a terminal an try to run the plugin py file manually via ```python /gimp-stable-diffusion.py```. You should see the error message, that "gimpfu" is unknown. Make sure, that you are running Python 2, as this version is used by GIMP. If other errors occur, please reinstall GIMP. 133 | 134 | ### Colab server 135 | #### Ressource limits 136 | The ressources on the colab server are limited. So, it's a good idea to stop it when you don't use it. 137 | - If you only don't use it for a short time, just stop the last step (Waiting for GIMP requests). To do so, click on the spinning circle on the left. If you want to use it again, just execute the last step again. The URL for accessing the server will be different, so copy it again. 138 | 139 | - If you don't use if for a longer time, the best is to release all ressources. To do so, select "Runtime/Disconnect and delete runtime". If you want to use it again, you have to start again at step 1. 140 | 141 | If you generated several images, the ressources of the colab server will be exhausted at some point. This happens pretty quickly, if you use the free plan. It takes longer for the pro plans. If this happens, an error will occur and you have to wait for some time until you can generate images again. 142 | 143 | ## FAQ 144 | 145 | **Will GIMP 3 be supported?** 146 | Yes, the plugin will be ported to GIMP 3. 147 | 148 | **Does it run locally?** According to Google it should be possible to run the notebook locally. As I don't have a local GPU, I can't try it myself. If you give it a try, I would be happy to know, if it really works. 149 | 150 | **Will Out-Painting be supported?** I first need to check the details and see what's possible. So, unfortunately I can't promise it currently. 151 | 152 | **Can the official stable-diffusion API be used?** Unfortunately, this is currently not possible. The reason is, that this API currently can only be accessed via gRPC and it's not possible to use this protocol in a GIMP plugin. As soon as the API is available as a REST API, it will be possible to port the plugin. 153 | 154 | **How do I report an error or request a new feature?** Please open a new issue in this repository. 155 | 156 | 157 | -------------------------------------------------------------------------------- /discontinued/notebook/gimp-stable-diffusion.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "c442uQJ_gUgy" 7 | }, 8 | "source": [ 9 | "## GIMP Stable Diffusion v1.3.0\n", 10 | "###**IMPORTANT: The GIMP plugin notebook was based on the deforum notebook. As this notebook is broking now, the GIMP plugin notebook doesn't work any longer either. To make it working again, a complete rewrite would be necessary. This would happen in the future again again. For this reason and combined with the fact that the usage of the GIMP notebook seems to be pretty low, the GIMP plugin for colab will be discontinued. Please check out the [GIMP plugin, which uses stablehorde](https://github.com/blueturtleai/gimp-stable-diffusion/tree/main/stablehorde) as an alternative.** \n", 11 | "###**IMPORTANT: Please use v1.2.0 of the GIMP plugin (current version in the github repository)** \n", 12 | "###The plugin version can be found at the top of the plugin file \"gimp-stable-diffusion.py\" which is located in your GIMP plugins folder.\n", 13 | "\n", 14 | "Notebook by [deforum](https://discord.gg/upmXXsrwZc), modified by [BlueTurtleAI](https://twitter.com/BlueTurtleAI) for the usage with the GIMP plugin." 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "source": [ 20 | "By using this Notebook, you agree to the following Terms of Use, and license:\n", 21 | "\n", 22 | "**Stablity.AI Model Terms of Use**\n", 23 | "\n", 24 | "This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.\n", 25 | "\n", 26 | "The CreativeML OpenRAIL License specifies:\n", 27 | "\n", 28 | "You can't use the model to deliberately produce nor share illegal or harmful outputs or content.\n", 29 | "CompVis claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license.\n", 30 | "You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)\n", 31 | "\n", 32 | "\n", 33 | "Please read the full license here: https://huggingface.co/spaces/CompVis/stable-diffusion-license" 34 | ], 35 | "metadata": { 36 | "id": "3poIEhV1oA4z" 37 | } 38 | }, 39 | { 40 | "cell_type": "code", 41 | "metadata": { 42 | "id": "2g-f7cQmf2Nt", 43 | "cellView": "form" 44 | }, 45 | "source": [ 46 | "#@title NVIDIA GPU\n", 47 | "import subprocess\n", 48 | "sub_p_res = subprocess.run(['nvidia-smi', '--query-gpu=name,memory.total,memory.free', '--format=csv,noheader'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n", 49 | "print(sub_p_res)" 50 | ], 51 | "outputs": [], 52 | "execution_count": null 53 | }, 54 | { 55 | "cell_type": "code", 56 | "source": [ 57 | "#@title Mount Google Drive\n", 58 | "from google.colab import drive # type: ignore\n", 59 | "\n", 60 | "try:\n", 61 | " drive_path = \"/content/drive\"\n", 62 | " drive.mount(drive_path,force_remount=False)\n", 63 | "except:\n", 64 | " print(\"...error mounting drive or with drive path variables\")\n", 65 | "\n" 66 | ], 67 | "metadata": { 68 | "cellView": "form", 69 | "id": "wsOTioAILxjx" 70 | }, 71 | "execution_count": null, 72 | "outputs": [] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "metadata": { 77 | "id": "TxIOPT0G5Lx1", 78 | "cellView": "form" 79 | }, 80 | "source": [ 81 | "#@title Set Model Path\n", 82 | "#@markdown **Hints**\n", 83 | "#@markdown - It is recommended to use the default path. That way you don't have to adjust the path manually every time. \n", 84 | "#@markdown - If the model file doesn't exist at this location, it is automatically downloaded from Huggingface. Please make sure you have enough free space on your drive (about 4 GB).\n", 85 | "#@markdown - **For an individual path:**\n", 86 | "#@markdown - Click on the folder symbol on the left. Open the \"drive/MyDrive\" folder and navigate to the model file which you uploaded before. Select the model file, click on the three dots and select \"copy path\". Close the file explorer via the cross. \n", 87 | "#@markdown - Insert the copied path into the field. Remove the filename and the last \"/\" at the end. The path should now look for example like this ```/content/drive/MyDrive/AI/models```.\n", 88 | "\n", 89 | "import os\n", 90 | "\n", 91 | "# ask for the link\n", 92 | "print(\"Local Path Variables:\\n\")\n", 93 | "\n", 94 | "models_path = \"/content/models\"\n", 95 | "output_path = \"/content/output\"\n", 96 | "\n", 97 | "models_path_gdrive = \"/content/drive/MyDrive/AI/models\" #@param {type:\"string\"}\n", 98 | "output_path_gdrive = \"/content/drive/MyDrive/AI/StableDiffusion\"\n", 99 | "models_path = models_path_gdrive\n", 100 | "output_path = output_path_gdrive\n", 101 | "\n", 102 | "os.makedirs(models_path, exist_ok=True)\n", 103 | "os.makedirs(output_path, exist_ok=True)\n", 104 | "\n", 105 | "print(f\"models_path: {models_path}\")\n", 106 | "print(f\"output_path: {output_path}\")" 107 | ], 108 | "outputs": [], 109 | "execution_count": null 110 | }, 111 | { 112 | "cell_type": "code", 113 | "metadata": { 114 | "id": "VRNl2mfepEIe", 115 | "cellView": "form" 116 | }, 117 | "source": [ 118 | "#@title Setup Environment\n", 119 | "\n", 120 | "setup_environment = True\n", 121 | "print_subprocess = True\n", 122 | "\n", 123 | "if setup_environment:\n", 124 | " import subprocess, time\n", 125 | " print(\"Setting up environment...\")\n", 126 | " start_time = time.time()\n", 127 | " all_process = [\n", 128 | " ['pip', 'install', 'torch==1.12.1+cu113', 'torchvision==0.13.1+cu113', '--extra-index-url', 'https://download.pytorch.org/whl/cu113'],\n", 129 | " ['pip', 'install', 'omegaconf==2.2.3', 'einops==0.4.1', 'pytorch-lightning==1.7.4', 'torchmetrics==0.9.3', 'torchtext==0.13.1', 'transformers==4.21.2', 'kornia==0.6.7'],\n", 130 | " ['git', 'clone', 'https://github.com/deforum/stable-diffusion'],\n", 131 | " ['pip', 'install', '-e', 'git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers'],\n", 132 | " ['pip', 'install', '-e', 'git+https://github.com/openai/CLIP.git@main#egg=clip'],\n", 133 | " ['pip', 'install', 'accelerate', 'ftfy', 'jsonmerge', 'matplotlib', 'resize-right', 'timm', 'torchdiffeq'],\n", 134 | " ['git', 'clone', 'https://github.com/shariqfarooq123/AdaBins.git'],\n", 135 | " ['git', 'clone', 'https://github.com/isl-org/MiDaS.git'],\n", 136 | " ['git', 'clone', 'https://github.com/MSFTserver/pytorch3d-lite.git'],\n", 137 | " ['pip', 'install', 'pyngrok', 'flask-cloudflared']\n", 138 | " ]\n", 139 | " for process in all_process:\n", 140 | " running = subprocess.run(process,stdout=subprocess.PIPE).stdout.decode('utf-8')\n", 141 | " if print_subprocess:\n", 142 | " print(running)\n", 143 | " \n", 144 | " print(subprocess.run(['git', 'clone', 'https://github.com/deforum/k-diffusion/'], stdout=subprocess.PIPE).stdout.decode('utf-8'))\n", 145 | " with open('k-diffusion/k_diffusion/__init__.py', 'w') as f:\n", 146 | " f.write('')\n", 147 | "\n", 148 | " end_time = time.time()\n", 149 | " print(f\"Environment set up in {end_time-start_time:.0f} seconds\")" 150 | ], 151 | "outputs": [], 152 | "execution_count": null 153 | }, 154 | { 155 | "cell_type": "code", 156 | "metadata": { 157 | "id": "81qmVZbrm4uu", 158 | "cellView": "form" 159 | }, 160 | "source": [ 161 | "#@title Python Definitions\n", 162 | "import json\n", 163 | "from IPython import display\n", 164 | "\n", 165 | "import gc, math, os, pathlib, subprocess, sys, time\n", 166 | "import cv2\n", 167 | "import numpy as np\n", 168 | "import pandas as pd\n", 169 | "import random\n", 170 | "import requests\n", 171 | "import torch\n", 172 | "import torch.nn as nn\n", 173 | "import torchvision.transforms as T\n", 174 | "import torchvision.transforms.functional as TF\n", 175 | "from contextlib import contextmanager, nullcontext\n", 176 | "from einops import rearrange, repeat\n", 177 | "from omegaconf import OmegaConf\n", 178 | "from PIL import Image\n", 179 | "from pytorch_lightning import seed_everything\n", 180 | "from skimage.exposure import match_histograms\n", 181 | "from torchvision.utils import make_grid\n", 182 | "from tqdm import tqdm, trange\n", 183 | "from types import SimpleNamespace\n", 184 | "from torch import autocast\n", 185 | "\n", 186 | "sys.path.extend([\n", 187 | " 'src/taming-transformers',\n", 188 | " 'src/clip',\n", 189 | " 'stable-diffusion/',\n", 190 | " 'k-diffusion',\n", 191 | " 'pytorch3d-lite',\n", 192 | " 'AdaBins',\n", 193 | " 'MiDaS',\n", 194 | "])\n", 195 | "\n", 196 | "import py3d_tools as p3d\n", 197 | "\n", 198 | "from helpers import DepthModel, sampler_fn\n", 199 | "from k_diffusion.external import CompVisDenoiser\n", 200 | "from ldm.util import instantiate_from_config\n", 201 | "from ldm.models.diffusion.ddim import DDIMSampler\n", 202 | "from ldm.models.diffusion.plms import PLMSSampler\n", 203 | "\n", 204 | "def sanitize(prompt):\n", 205 | " whitelist = set('abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ')\n", 206 | " tmp = ''.join(filter(whitelist.__contains__, prompt))\n", 207 | " return tmp.replace(' ', '_')\n", 208 | "\n", 209 | "def anim_frame_warp_2d(prev_img_cv2, args, anim_args, keys, frame_idx):\n", 210 | " angle = keys.angle_series[frame_idx]\n", 211 | " zoom = keys.zoom_series[frame_idx]\n", 212 | " translation_x = keys.translation_x_series[frame_idx]\n", 213 | " translation_y = keys.translation_y_series[frame_idx]\n", 214 | "\n", 215 | " center = (args.W // 2, args.H // 2)\n", 216 | " trans_mat = np.float32([[1, 0, translation_x], [0, 1, translation_y]])\n", 217 | " rot_mat = cv2.getRotationMatrix2D(center, angle, zoom)\n", 218 | " trans_mat = np.vstack([trans_mat, [0,0,1]])\n", 219 | " rot_mat = np.vstack([rot_mat, [0,0,1]])\n", 220 | " xform = np.matmul(rot_mat, trans_mat)\n", 221 | "\n", 222 | " return cv2.warpPerspective(\n", 223 | " prev_img_cv2,\n", 224 | " xform,\n", 225 | " (prev_img_cv2.shape[1], prev_img_cv2.shape[0]),\n", 226 | " borderMode=cv2.BORDER_WRAP if anim_args.border == 'wrap' else cv2.BORDER_REPLICATE\n", 227 | " )\n", 228 | "\n", 229 | "def anim_frame_warp_3d(prev_img_cv2, depth, anim_args, keys, frame_idx):\n", 230 | " TRANSLATION_SCALE = 1.0/200.0 # matches Disco\n", 231 | " translate_xyz = [\n", 232 | " -keys.translation_x_series[frame_idx] * TRANSLATION_SCALE, \n", 233 | " keys.translation_y_series[frame_idx] * TRANSLATION_SCALE, \n", 234 | " -keys.translation_z_series[frame_idx] * TRANSLATION_SCALE\n", 235 | " ]\n", 236 | " rotate_xyz = [\n", 237 | " math.radians(keys.rotation_3d_x_series[frame_idx]), \n", 238 | " math.radians(keys.rotation_3d_y_series[frame_idx]), \n", 239 | " math.radians(keys.rotation_3d_z_series[frame_idx])\n", 240 | " ]\n", 241 | " rot_mat = p3d.euler_angles_to_matrix(torch.tensor(rotate_xyz, device=device), \"XYZ\").unsqueeze(0)\n", 242 | " result = transform_image_3d(prev_img_cv2, depth, rot_mat, translate_xyz, anim_args)\n", 243 | " torch.cuda.empty_cache()\n", 244 | " return result\n", 245 | "\n", 246 | "def add_noise(sample: torch.Tensor, noise_amt: float) -> torch.Tensor:\n", 247 | " return sample + torch.randn(sample.shape, device=sample.device) * noise_amt\n", 248 | "\n", 249 | "def get_output_folder(output_path, batch_folder):\n", 250 | " out_path = os.path.join(output_path,time.strftime('%Y-%m'))\n", 251 | " if batch_folder != \"\":\n", 252 | " out_path = os.path.join(out_path, batch_folder)\n", 253 | " os.makedirs(out_path, exist_ok=True)\n", 254 | " return out_path\n", 255 | "\n", 256 | "def load_img(path, shape, use_alpha_as_mask=False):\n", 257 | " # use_alpha_as_mask: Read the alpha channel of the image as the mask image\n", 258 | " if path.startswith('http://') or path.startswith('https://'):\n", 259 | " image = Image.open(requests.get(path, stream=True).raw)\n", 260 | " else:\n", 261 | " image = Image.open(path)\n", 262 | "\n", 263 | " if use_alpha_as_mask:\n", 264 | " image = image.convert('RGBA')\n", 265 | " else:\n", 266 | " image = image.convert('RGB')\n", 267 | "\n", 268 | " image = image.resize(shape, resample=Image.LANCZOS)\n", 269 | "\n", 270 | " mask_image = None\n", 271 | " if use_alpha_as_mask:\n", 272 | " # Split alpha channel into a mask_image\n", 273 | " red, green, blue, alpha = Image.Image.split(image)\n", 274 | " mask_image = alpha.convert('L')\n", 275 | " image = image.convert('RGB')\n", 276 | "\n", 277 | " image = np.array(image).astype(np.float16) / 255.0\n", 278 | " image = image[None].transpose(0, 3, 1, 2)\n", 279 | " image = torch.from_numpy(image)\n", 280 | " image = 2.*image - 1.\n", 281 | "\n", 282 | " return image, mask_image\n", 283 | "\n", 284 | "def load_mask_latent(mask_input, shape):\n", 285 | " # mask_input (str or PIL Image.Image): Path to the mask image or a PIL Image object\n", 286 | " # shape (list-like len(4)): shape of the image to match, usually latent_image.shape\n", 287 | " \n", 288 | " if isinstance(mask_input, str): # mask input is probably a file name\n", 289 | " if mask_input.startswith('http://') or mask_input.startswith('https://'):\n", 290 | " mask_image = Image.open(requests.get(mask_input, stream=True).raw).convert('RGBA')\n", 291 | " else:\n", 292 | " mask_image = Image.open(mask_input).convert('RGBA')\n", 293 | " elif isinstance(mask_input, Image.Image):\n", 294 | " mask_image = mask_input\n", 295 | " else:\n", 296 | " raise Exception(\"mask_input must be a PIL image or a file name\")\n", 297 | "\n", 298 | " mask_w_h = (shape[-1], shape[-2])\n", 299 | " mask = mask_image.resize(mask_w_h, resample=Image.LANCZOS)\n", 300 | " mask = mask.convert(\"L\")\n", 301 | " return mask\n", 302 | "\n", 303 | "def prepare_mask(mask_input, mask_shape, mask_brightness_adjust=1.0, mask_contrast_adjust=1.0):\n", 304 | " # mask_input (str or PIL Image.Image): Path to the mask image or a PIL Image object\n", 305 | " # shape (list-like len(4)): shape of the image to match, usually latent_image.shape\n", 306 | " # mask_brightness_adjust (non-negative float): amount to adjust brightness of the iamge, \n", 307 | " # 0 is black, 1 is no adjustment, >1 is brighter\n", 308 | " # mask_contrast_adjust (non-negative float): amount to adjust contrast of the image, \n", 309 | " # 0 is a flat grey image, 1 is no adjustment, >1 is more contrast\n", 310 | " \n", 311 | " mask = load_mask_latent(mask_input, mask_shape)\n", 312 | "\n", 313 | " # Mask brightness/contrast adjustments\n", 314 | " if mask_brightness_adjust != 1:\n", 315 | " mask = TF.adjust_brightness(mask, mask_brightness_adjust)\n", 316 | " if mask_contrast_adjust != 1:\n", 317 | " mask = TF.adjust_contrast(mask, mask_contrast_adjust)\n", 318 | "\n", 319 | " # Mask image to array\n", 320 | " mask = np.array(mask).astype(np.float32) / 255.0\n", 321 | " mask = np.tile(mask,(4,1,1))\n", 322 | " mask = np.expand_dims(mask,axis=0)\n", 323 | " mask = torch.from_numpy(mask)\n", 324 | "\n", 325 | " if args.invert_mask:\n", 326 | " mask = ( (mask - 0.5) * -1) + 0.5\n", 327 | " \n", 328 | " mask = np.clip(mask,0,1)\n", 329 | " return mask\n", 330 | "\n", 331 | "def maintain_colors(prev_img, color_match_sample, mode):\n", 332 | " if mode == 'Match Frame 0 RGB':\n", 333 | " return match_histograms(prev_img, color_match_sample, multichannel=True)\n", 334 | " elif mode == 'Match Frame 0 HSV':\n", 335 | " prev_img_hsv = cv2.cvtColor(prev_img, cv2.COLOR_RGB2HSV)\n", 336 | " color_match_hsv = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2HSV)\n", 337 | " matched_hsv = match_histograms(prev_img_hsv, color_match_hsv, multichannel=True)\n", 338 | " return cv2.cvtColor(matched_hsv, cv2.COLOR_HSV2RGB)\n", 339 | " else: # Match Frame 0 LAB\n", 340 | " prev_img_lab = cv2.cvtColor(prev_img, cv2.COLOR_RGB2LAB)\n", 341 | " color_match_lab = cv2.cvtColor(color_match_sample, cv2.COLOR_RGB2LAB)\n", 342 | " matched_lab = match_histograms(prev_img_lab, color_match_lab, multichannel=True)\n", 343 | " return cv2.cvtColor(matched_lab, cv2.COLOR_LAB2RGB)\n", 344 | "\n", 345 | "\n", 346 | "def make_callback(sampler_name, dynamic_threshold=None, static_threshold=None, mask=None, init_latent=None, sigmas=None, sampler=None, masked_noise_modifier=1.0): \n", 347 | " # Creates the callback function to be passed into the samplers\n", 348 | " # The callback function is applied to the image at each step\n", 349 | " def dynamic_thresholding_(img, threshold):\n", 350 | " # Dynamic thresholding from Imagen paper (May 2022)\n", 351 | " s = np.percentile(np.abs(img.cpu()), threshold, axis=tuple(range(1,img.ndim)))\n", 352 | " s = np.max(np.append(s,1.0))\n", 353 | " torch.clamp_(img, -1*s, s)\n", 354 | " torch.FloatTensor.div_(img, s)\n", 355 | "\n", 356 | " # Callback for samplers in the k-diffusion repo, called thus:\n", 357 | " # callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})\n", 358 | " def k_callback_(args_dict):\n", 359 | " if dynamic_threshold is not None:\n", 360 | " dynamic_thresholding_(args_dict['x'], dynamic_threshold)\n", 361 | " if static_threshold is not None:\n", 362 | " torch.clamp_(args_dict['x'], -1*static_threshold, static_threshold)\n", 363 | " if mask is not None:\n", 364 | " init_noise = init_latent + noise * args_dict['sigma']\n", 365 | " is_masked = torch.logical_and(mask >= mask_schedule[args_dict['i']], mask != 0 )\n", 366 | " new_img = init_noise * torch.where(is_masked,1,0) + args_dict['x'] * torch.where(is_masked,0,1)\n", 367 | " args_dict['x'].copy_(new_img)\n", 368 | "\n", 369 | " # Function that is called on the image (img) and step (i) at each step\n", 370 | " def img_callback_(img, i):\n", 371 | " # Thresholding functions\n", 372 | " if dynamic_threshold is not None:\n", 373 | " dynamic_thresholding_(img, dynamic_threshold)\n", 374 | " if static_threshold is not None:\n", 375 | " torch.clamp_(img, -1*static_threshold, static_threshold)\n", 376 | " if mask is not None:\n", 377 | " i_inv = len(sigmas) - i - 1\n", 378 | " init_noise = sampler.stochastic_encode(init_latent, torch.tensor([i_inv]*batch_size).to(device), noise=noise)\n", 379 | " is_masked = torch.logical_and(mask >= mask_schedule[i], mask != 0 )\n", 380 | " new_img = init_noise * torch.where(is_masked,1,0) + img * torch.where(is_masked,0,1)\n", 381 | " img.copy_(new_img)\n", 382 | " \n", 383 | " if init_latent is not None:\n", 384 | " noise = torch.randn_like(init_latent, device=device) * masked_noise_modifier\n", 385 | " if sigmas is not None and len(sigmas) > 0:\n", 386 | " mask_schedule, _ = torch.sort(sigmas/torch.max(sigmas))\n", 387 | " elif len(sigmas) == 0:\n", 388 | " mask = None # no mask needed if no steps (usually happens because strength==1.0)\n", 389 | " if sampler_name in [\"plms\",\"ddim\"]: \n", 390 | " # Callback function formated for compvis latent diffusion samplers\n", 391 | " if mask is not None:\n", 392 | " assert sampler is not None, \"Callback function for stable-diffusion samplers requires sampler variable\"\n", 393 | " batch_size = init_latent.shape[0]\n", 394 | "\n", 395 | " callback = img_callback_\n", 396 | " else: \n", 397 | " # Default callback function uses k-diffusion sampler variables\n", 398 | " callback = k_callback_\n", 399 | "\n", 400 | " return callback\n", 401 | "\n", 402 | "def sample_from_cv2(sample: np.ndarray) -> torch.Tensor:\n", 403 | " sample = ((sample.astype(float) / 255.0) * 2) - 1\n", 404 | " sample = sample[None].transpose(0, 3, 1, 2).astype(np.float16)\n", 405 | " sample = torch.from_numpy(sample)\n", 406 | " return sample\n", 407 | "\n", 408 | "def sample_to_cv2(sample: torch.Tensor, type=np.uint8) -> np.ndarray:\n", 409 | " sample_f32 = rearrange(sample.squeeze().cpu().numpy(), \"c h w -> h w c\").astype(np.float32)\n", 410 | " sample_f32 = ((sample_f32 * 0.5) + 0.5).clip(0, 1)\n", 411 | " sample_int8 = (sample_f32 * 255)\n", 412 | " return sample_int8.astype(type)\n", 413 | "\n", 414 | "def transform_image_3d(prev_img_cv2, depth_tensor, rot_mat, translate, anim_args):\n", 415 | " # adapted and optimized version of transform_image_3d from Disco Diffusion https://github.com/alembics/disco-diffusion \n", 416 | " w, h = prev_img_cv2.shape[1], prev_img_cv2.shape[0]\n", 417 | "\n", 418 | " aspect_ratio = float(w)/float(h)\n", 419 | " near, far, fov_deg = anim_args.near_plane, anim_args.far_plane, anim_args.fov\n", 420 | " persp_cam_old = p3d.FoVPerspectiveCameras(near, far, aspect_ratio, fov=fov_deg, degrees=True, device=device)\n", 421 | " persp_cam_new = p3d.FoVPerspectiveCameras(near, far, aspect_ratio, fov=fov_deg, degrees=True, R=rot_mat, T=torch.tensor([translate]), device=device)\n", 422 | "\n", 423 | " # range of [-1,1] is important to torch grid_sample's padding handling\n", 424 | " y,x = torch.meshgrid(torch.linspace(-1.,1.,h,dtype=torch.float32,device=device),torch.linspace(-1.,1.,w,dtype=torch.float32,device=device))\n", 425 | " z = torch.as_tensor(depth_tensor, dtype=torch.float32, device=device)\n", 426 | " xyz_old_world = torch.stack((x.flatten(), y.flatten(), z.flatten()), dim=1)\n", 427 | "\n", 428 | " xyz_old_cam_xy = persp_cam_old.get_full_projection_transform().transform_points(xyz_old_world)[:,0:2]\n", 429 | " xyz_new_cam_xy = persp_cam_new.get_full_projection_transform().transform_points(xyz_old_world)[:,0:2]\n", 430 | "\n", 431 | " offset_xy = xyz_new_cam_xy - xyz_old_cam_xy\n", 432 | " # affine_grid theta param expects a batch of 2D mats. Each is 2x3 to do rotation+translation.\n", 433 | " identity_2d_batch = torch.tensor([[1.,0.,0.],[0.,1.,0.]], device=device).unsqueeze(0)\n", 434 | " # coords_2d will have shape (N,H,W,2).. which is also what grid_sample needs.\n", 435 | " coords_2d = torch.nn.functional.affine_grid(identity_2d_batch, [1,1,h,w], align_corners=False)\n", 436 | " offset_coords_2d = coords_2d - torch.reshape(offset_xy, (h,w,2)).unsqueeze(0)\n", 437 | "\n", 438 | " image_tensor = rearrange(torch.from_numpy(prev_img_cv2.astype(np.float32)), 'h w c -> c h w').to(device)\n", 439 | " new_image = torch.nn.functional.grid_sample(\n", 440 | " image_tensor.add(1/512 - 0.0001).unsqueeze(0), \n", 441 | " offset_coords_2d, \n", 442 | " mode=anim_args.sampling_mode, \n", 443 | " padding_mode=anim_args.padding_mode, \n", 444 | " align_corners=False\n", 445 | " )\n", 446 | "\n", 447 | " # convert back to cv2 style numpy array\n", 448 | " result = rearrange(\n", 449 | " new_image.squeeze().clamp(0,255), \n", 450 | " 'c h w -> h w c'\n", 451 | " ).cpu().numpy().astype(prev_img_cv2.dtype)\n", 452 | " return result\n", 453 | "\n", 454 | "def generate(args, return_latent=False, return_sample=False, return_c=False):\n", 455 | " seed_everything(args.seed)\n", 456 | " os.makedirs(args.outdir, exist_ok=True)\n", 457 | "\n", 458 | " sampler = PLMSSampler(model) if args.sampler == 'plms' else DDIMSampler(model)\n", 459 | " model_wrap = CompVisDenoiser(model)\n", 460 | " batch_size = args.n_samples\n", 461 | " prompt = args.prompt\n", 462 | " assert prompt is not None\n", 463 | " data = [batch_size * [prompt]]\n", 464 | " precision_scope = autocast if args.precision == \"autocast\" else nullcontext\n", 465 | "\n", 466 | " init_latent = None\n", 467 | " mask_image = None\n", 468 | " init_image = None\n", 469 | " if args.init_latent is not None:\n", 470 | " init_latent = args.init_latent\n", 471 | " elif args.init_sample is not None:\n", 472 | " with precision_scope(\"cuda\"):\n", 473 | " init_latent = model.get_first_stage_encoding(model.encode_first_stage(args.init_sample))\n", 474 | " elif args.use_init and args.init_image != None and args.init_image != '':\n", 475 | " init_image, mask_image = load_img(args.init_image, \n", 476 | " shape=(args.W, args.H), \n", 477 | " use_alpha_as_mask=args.use_alpha_as_mask)\n", 478 | " init_image = init_image.to(device)\n", 479 | " init_image = repeat(init_image, '1 ... -> b ...', b=batch_size)\n", 480 | " with precision_scope(\"cuda\"):\n", 481 | " init_latent = model.get_first_stage_encoding(model.encode_first_stage(init_image)) # move to latent space \n", 482 | "\n", 483 | " if not args.use_init and args.strength > 0 and args.strength_0_no_init:\n", 484 | " print(\"\\nNo init image, but strength > 0. Strength has been auto set to 0, since use_init is False.\")\n", 485 | " print(\"If you want to force strength > 0 with no init, please set strength_0_no_init to False.\\n\")\n", 486 | " args.strength = 0\n", 487 | "\n", 488 | " # Mask functions\n", 489 | " if args.use_mask:\n", 490 | " assert args.mask_file is not None or mask_image is not None, \"use_mask==True: An mask image is required for a mask. Please enter a mask_file or use an init image with an alpha channel\"\n", 491 | " assert args.use_init, \"use_mask==True: use_init is required for a mask\"\n", 492 | " assert init_latent is not None, \"use_mask==True: An latent init image is required for a mask\"\n", 493 | "\n", 494 | " mask = prepare_mask(args.mask_file if mask_image is None else mask_image, \n", 495 | " init_latent.shape, \n", 496 | " args.mask_contrast_adjust, \n", 497 | " args.mask_brightness_adjust)\n", 498 | " \n", 499 | " if (torch.all(mask == 0) or torch.all(mask == 1)) and args.use_alpha_as_mask:\n", 500 | " raise Warning(\"use_alpha_as_mask==True: Using the alpha channel from the init image as a mask, but the alpha channel is blank.\")\n", 501 | " \n", 502 | " mask = mask.to(device)\n", 503 | " mask = repeat(mask, '1 ... -> b ...', b=batch_size)\n", 504 | " else:\n", 505 | " mask = None\n", 506 | " \n", 507 | " t_enc = int((1.0-args.strength) * args.steps)\n", 508 | "\n", 509 | " # Noise schedule for the k-diffusion samplers (used for masking)\n", 510 | " k_sigmas = model_wrap.get_sigmas(args.steps)\n", 511 | " k_sigmas = k_sigmas[len(k_sigmas)-t_enc-1:]\n", 512 | "\n", 513 | " if args.sampler in ['plms','ddim']:\n", 514 | " sampler.make_schedule(ddim_num_steps=args.steps, ddim_eta=args.ddim_eta, ddim_discretize='fill', verbose=False)\n", 515 | "\n", 516 | " callback = make_callback(sampler_name=args.sampler,\n", 517 | " dynamic_threshold=args.dynamic_threshold, \n", 518 | " static_threshold=args.static_threshold,\n", 519 | " mask=mask, \n", 520 | " init_latent=init_latent,\n", 521 | " sigmas=k_sigmas,\n", 522 | " sampler=sampler) \n", 523 | "\n", 524 | " results = []\n", 525 | " with torch.no_grad():\n", 526 | " with precision_scope(\"cuda\"):\n", 527 | " with model.ema_scope():\n", 528 | " for prompts in data:\n", 529 | " uc = None\n", 530 | " if args.scale != 1.0:\n", 531 | " uc = model.get_learned_conditioning(batch_size * [\"\"])\n", 532 | " if isinstance(prompts, tuple):\n", 533 | " prompts = list(prompts)\n", 534 | " c = model.get_learned_conditioning(prompts)\n", 535 | "\n", 536 | " if args.init_c != None:\n", 537 | " c = args.init_c\n", 538 | "\n", 539 | " if args.sampler in [\"klms\",\"dpm2\",\"dpm2_ancestral\",\"heun\",\"euler\",\"euler_ancestral\"]:\n", 540 | " samples = sampler_fn(\n", 541 | " c=c, \n", 542 | " uc=uc, \n", 543 | " args=args, \n", 544 | " model_wrap=model_wrap, \n", 545 | " init_latent=init_latent, \n", 546 | " t_enc=t_enc, \n", 547 | " device=device, \n", 548 | " cb=callback)\n", 549 | " else:\n", 550 | " # args.sampler == 'plms' or args.sampler == 'ddim':\n", 551 | " if init_latent is not None and args.strength > 0:\n", 552 | " z_enc = sampler.stochastic_encode(init_latent, torch.tensor([t_enc]*batch_size).to(device))\n", 553 | " else:\n", 554 | " z_enc = torch.randn([args.n_samples, args.C, args.H // args.f, args.W // args.f], device=device)\n", 555 | " if args.sampler == 'ddim':\n", 556 | " samples = sampler.decode(z_enc, \n", 557 | " c, \n", 558 | " t_enc, \n", 559 | " unconditional_guidance_scale=args.scale,\n", 560 | " unconditional_conditioning=uc,\n", 561 | " img_callback=callback)\n", 562 | " elif args.sampler == 'plms': # no \"decode\" function in plms, so use \"sample\"\n", 563 | " shape = [args.C, args.H // args.f, args.W // args.f]\n", 564 | " samples, _ = sampler.sample(S=args.steps,\n", 565 | " conditioning=c,\n", 566 | " batch_size=args.n_samples,\n", 567 | " shape=shape,\n", 568 | " verbose=False,\n", 569 | " unconditional_guidance_scale=args.scale,\n", 570 | " unconditional_conditioning=uc,\n", 571 | " eta=args.ddim_eta,\n", 572 | " x_T=z_enc,\n", 573 | " img_callback=callback)\n", 574 | " else:\n", 575 | " raise Exception(f\"Sampler {args.sampler} not recognised.\")\n", 576 | "\n", 577 | " if return_latent:\n", 578 | " results.append(samples.clone())\n", 579 | "\n", 580 | " x_samples = model.decode_first_stage(samples)\n", 581 | " if return_sample:\n", 582 | " results.append(x_samples.clone())\n", 583 | "\n", 584 | " x_samples = torch.clamp((x_samples + 1.0) / 2.0, min=0.0, max=1.0)\n", 585 | "\n", 586 | " if return_c:\n", 587 | " results.append(c.clone())\n", 588 | "\n", 589 | " for x_sample in x_samples:\n", 590 | " x_sample = 255. * rearrange(x_sample.cpu().numpy(), 'c h w -> h w c')\n", 591 | " image = Image.fromarray(x_sample.astype(np.uint8))\n", 592 | " results.append(image)\n", 593 | " return results" 594 | ], 595 | "outputs": [], 596 | "execution_count": null 597 | }, 598 | { 599 | "cell_type": "code", 600 | "metadata": { 601 | "cellView": "form", 602 | "id": "CIUJ7lWI4v53" 603 | }, 604 | "source": [ 605 | "#@title **Select and Load Model**\n", 606 | "#@markdown **Hints**\n", 607 | "\n", 608 | "#@markdown If the model file doesn't exist at the location you selected above, it will automatically be downloaded from Huggingface. \n", 609 | "#@markdown Please make sure you have enough free space on your drive (about 4 GB). It is necessary, that you created an account on Huggingface and accepted the terms of service. Otherwise it's not possible to download the file. For the download you need a Huggingface token:\n", 610 | "#@markdown - Login on Huggingface and select \"Settings/Access Tokens\" on the left.\n", 611 | "#@markdown - Click on \"New Token\", enter a name, select \"Read\" as role, click on create and copy the token.\n", 612 | "model_config = \"v1-inference.yaml\"\n", 613 | "model_checkpoint = \"sd-v1-5.ckpt\" #@param [\"sd-v1-5.ckpt\", \"sd-v1-4.ckpt\"]\n", 614 | "custom_config_path = \"\"\n", 615 | "custom_checkpoint_path = \"\"\n", 616 | "\n", 617 | "load_on_run_all = True\n", 618 | "half_precision = True # check\n", 619 | "check_sha256 = True\n", 620 | "\n", 621 | "model_map = {\n", 622 | " \"sd-v1-5.ckpt\": {\n", 623 | " 'sha256': 'cc6cb27103417325ff94f52b7a5d2dde45a7515b25c255d8e396c90014281516',\n", 624 | " 'url': 'https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt',\n", 625 | " 'requires_login': True,\n", 626 | " },\n", 627 | " \"sd-v1-4.ckpt\": {\n", 628 | " 'sha256': 'fe4efff1e174c627256e44ec2991ba279b3816e364b49f9be2abc0b3ff3f8556',\n", 629 | " 'url': 'https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt',\n", 630 | " 'requires_login': True,\n", 631 | " }\n", 632 | "}\n", 633 | "\n", 634 | "# config path\n", 635 | "ckpt_config_path = custom_config_path if model_config == \"custom\" else os.path.join(models_path, model_config)\n", 636 | "if os.path.exists(ckpt_config_path):\n", 637 | " print(f\"{ckpt_config_path} exists\")\n", 638 | "else:\n", 639 | " ckpt_config_path = \"./stable-diffusion/configs/stable-diffusion/v1-inference.yaml\"\n", 640 | "print(f\"Using config: {ckpt_config_path}\")\n", 641 | "\n", 642 | "# checkpoint path or download\n", 643 | "ckpt_path = custom_checkpoint_path if model_checkpoint == \"custom\" else os.path.join(models_path, model_checkpoint)\n", 644 | "ckpt_valid = True\n", 645 | "if os.path.exists(ckpt_path):\n", 646 | " print(f\"{ckpt_path} exists\")\n", 647 | "elif 'url' in model_map[model_checkpoint]:\n", 648 | " url = model_map[model_checkpoint]['url']\n", 649 | "\n", 650 | " # CLI dialogue to authenticate download\n", 651 | " if model_map[model_checkpoint]['requires_login']:\n", 652 | " print(\"This model requires an authentication token\")\n", 653 | " print(\"Please ensure you have accepted its terms of service before continuing.\\n\")\n", 654 | " print(\"Press enter after you inserted your username\")\n", 655 | "\n", 656 | " username = input(\"What is your huggingface username?:\")\n", 657 | " print(\"\\n\")\n", 658 | " print(\"Press enter after you inserted your token\")\n", 659 | " token = input(\"What is your huggingface token?:\")\n", 660 | " print(\"\\n\")\n", 661 | "\n", 662 | " _, path = url.split(\"https://\")\n", 663 | "\n", 664 | " url = f\"https://{username}:{token}@{path}\"\n", 665 | "\n", 666 | " # contact server for model\n", 667 | " print(f\"Attempting to download {model_checkpoint}...this may take a while\")\n", 668 | " ckpt_request = requests.get(url)\n", 669 | " request_status = ckpt_request.status_code\n", 670 | "\n", 671 | " # inform user of errors\n", 672 | " if request_status == 403:\n", 673 | " raise ConnectionRefusedError(\"You have not accepted the license for this model.\")\n", 674 | " elif request_status == 404:\n", 675 | " raise ConnectionError(\"Could not make contact with server\")\n", 676 | " elif request_status != 200:\n", 677 | " raise ConnectionError(f\"Some other error has ocurred - response code: {request_status}\")\n", 678 | "\n", 679 | " # write to model path\n", 680 | " with open(os.path.join(models_path, model_checkpoint), 'wb') as model_file:\n", 681 | " model_file.write(ckpt_request.content)\n", 682 | " model_file.close()\n", 683 | "else:\n", 684 | " print(f\"Please download model checkpoint and place in {os.path.join(models_path, model_checkpoint)}\")\n", 685 | " ckpt_valid = False\n", 686 | "\n", 687 | "if check_sha256 and model_checkpoint != \"custom\" and ckpt_valid:\n", 688 | " import hashlib\n", 689 | " print(\"\\n...checking sha256\")\n", 690 | " with open(ckpt_path, \"rb\") as f:\n", 691 | " bytes = f.read() \n", 692 | " hash = hashlib.sha256(bytes).hexdigest()\n", 693 | " del bytes\n", 694 | " if model_map[model_checkpoint][\"sha256\"] == hash:\n", 695 | " print(\"hash is correct\\n\")\n", 696 | " else:\n", 697 | " print(\"hash in not correct\\n\")\n", 698 | " ckpt_valid = False\n", 699 | "\n", 700 | "if ckpt_valid:\n", 701 | " print(f\"Using ckpt: {ckpt_path}\")\n", 702 | "\n", 703 | "def load_model_from_config(config, ckpt, verbose=False, device='cuda', half_precision=True):\n", 704 | " map_location = \"cuda\"\n", 705 | " print(f\"Loading model from {ckpt}\")\n", 706 | " pl_sd = torch.load(ckpt, map_location=map_location)\n", 707 | " if \"global_step\" in pl_sd:\n", 708 | " print(f\"Global Step: {pl_sd['global_step']}\")\n", 709 | " sd = pl_sd[\"state_dict\"]\n", 710 | " model = instantiate_from_config(config.model)\n", 711 | " m, u = model.load_state_dict(sd, strict=False)\n", 712 | " if len(m) > 0 and verbose:\n", 713 | " print(\"missing keys:\")\n", 714 | " print(m)\n", 715 | " if len(u) > 0 and verbose:\n", 716 | " print(\"unexpected keys:\")\n", 717 | " print(u)\n", 718 | "\n", 719 | " if half_precision:\n", 720 | " model = model.half().to(device)\n", 721 | " else:\n", 722 | " model = model.to(device)\n", 723 | " model.eval()\n", 724 | " return model\n", 725 | "\n", 726 | "if load_on_run_all and ckpt_valid:\n", 727 | " local_config = OmegaConf.load(f\"{ckpt_config_path}\")\n", 728 | " model = load_model_from_config(local_config, f\"{ckpt_path}\", half_precision=half_precision)\n", 729 | " device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n", 730 | " model = model.to(device)" 731 | ], 732 | "outputs": [], 733 | "execution_count": null 734 | }, 735 | { 736 | "cell_type": "code", 737 | "metadata": { 738 | "id": "qH74gBWDd2oq", 739 | "cellView": "form" 740 | }, 741 | "source": [ 742 | "#@title Waiting for GIMP requests\n", 743 | "\n", 744 | "from io import BytesIO\n", 745 | "import json\n", 746 | "import base64\n", 747 | "from flask import Flask, Response, request, abort, make_response\n", 748 | "from flask_cloudflared import run_with_cloudflared\n", 749 | "\n", 750 | "prompts = []\n", 751 | "\n", 752 | "def DeforumArgs():\n", 753 | " W = 0\n", 754 | " H = 0\n", 755 | "\n", 756 | " seed = -1\n", 757 | " sampler = 'klms'\n", 758 | " steps = 0\n", 759 | " scale = 0\n", 760 | " ddim_eta = 0.0\n", 761 | " dynamic_threshold = None\n", 762 | " static_threshold = None \n", 763 | "\n", 764 | " save_samples = False\n", 765 | " save_settings = False\n", 766 | " display_samples = False\n", 767 | "\n", 768 | " n_batch = 1\n", 769 | " batch_name = \"GIMP\"\n", 770 | " filename_format = \"{timestring}_{index}_{prompt}.png\"\n", 771 | " make_grid = False\n", 772 | " grid_rows = 2 \n", 773 | " outdir = get_output_folder(output_path, batch_name)\n", 774 | "\n", 775 | " use_init = False\n", 776 | " strength = 0\n", 777 | " strength_0_no_init = True # Set the strength to 0 automatically when no init image is used\n", 778 | " init_image = \"\"\n", 779 | " # Whiter areas of the mask are areas that change more\n", 780 | " use_mask = False\n", 781 | " use_alpha_as_mask = False # use the alpha channel of the init image as the mask\n", 782 | " mask_file = \"\"\n", 783 | " invert_mask = False\n", 784 | " # Adjust mask image, 1.0 is no adjustment. Should be positive numbers.\n", 785 | " mask_brightness_adjust = 1.0\n", 786 | " mask_contrast_adjust = 1.0\n", 787 | "\n", 788 | " n_samples = 1 # doesnt do anything\n", 789 | " precision = 'autocast' \n", 790 | " C = 4\n", 791 | " f = 8\n", 792 | "\n", 793 | " prompt = \"\"\n", 794 | " timestring = \"\"\n", 795 | " init_latent = None\n", 796 | " init_sample = None\n", 797 | " init_c = None\n", 798 | "\n", 799 | " return locals()\n", 800 | "\n", 801 | "def render_image_batch(args):\n", 802 | " args.prompts = {k: f\"{v:05d}\" for v, k in enumerate(prompts)}\n", 803 | " \n", 804 | " # create output folder for the batch\n", 805 | " #os.makedirs(args.outdir, exist_ok=True)\n", 806 | " #if args.save_settings or args.save_samples:\n", 807 | " #print(f\"Saving to {os.path.join(args.outdir, args.timestring)}_*\")\n", 808 | "\n", 809 | " # save settings for the batch\n", 810 | " #if args.save_settings:\n", 811 | " #filename = os.path.join(args.outdir, f\"{args.timestring}_settings.txt\")\n", 812 | " #with open(filename, \"w+\", encoding=\"utf-8\") as f:\n", 813 | " #json.dump(dict(args.__dict__), f, ensure_ascii=False, indent=4)\n", 814 | "\n", 815 | " index = 0\n", 816 | " \n", 817 | " # function for init image batching\n", 818 | " init_array = []\n", 819 | " if args.use_init:\n", 820 | " if args.init_image == \"\":\n", 821 | " raise FileNotFoundError(\"No path was given for init_image\")\n", 822 | " if args.init_image.startswith('http://') or args.init_image.startswith('https://'):\n", 823 | " init_array.append(args.init_image)\n", 824 | " elif not os.path.isfile(args.init_image):\n", 825 | " if args.init_image[-1] != \"/\": # avoids path error by adding / to end if not there\n", 826 | " args.init_image += \"/\" \n", 827 | " for image in sorted(os.listdir(args.init_image)): # iterates dir and appends images to init_array\n", 828 | " if image.split(\".\")[-1] in (\"png\", \"jpg\", \"jpeg\"):\n", 829 | " init_array.append(args.init_image + image)\n", 830 | " else:\n", 831 | " init_array.append(args.init_image)\n", 832 | " else:\n", 833 | " init_array = [\"\"]\n", 834 | "\n", 835 | " # when doing large batches don't flood browser with images\n", 836 | " clear_between_batches = args.n_batch >= 32\n", 837 | "\n", 838 | " for iprompt, prompt in enumerate(prompts): \n", 839 | " args.prompt = prompt\n", 840 | " print(f\"Prompt {iprompt+1} of {len(prompts)}\")\n", 841 | " print(f\"{args.prompt}\")\n", 842 | "\n", 843 | " all_images = []\n", 844 | "\n", 845 | " for batch_index in range(args.n_batch):\n", 846 | " if clear_between_batches and batch_index % 32 == 0: \n", 847 | " display.clear_output(wait=True) \n", 848 | " print(f\"Batch {batch_index+1} of {args.n_batch}\")\n", 849 | " \n", 850 | " for image in init_array: # iterates the init images\n", 851 | " args.init_image = image\n", 852 | " results = generate(args)\n", 853 | " for image in results:\n", 854 | " if args.make_grid:\n", 855 | " all_images.append(T.functional.pil_to_tensor(image))\n", 856 | " if args.save_samples:\n", 857 | " if args.filename_format == \"{timestring}_{index}_{prompt}.png\":\n", 858 | " filename = f\"{args.timestring}_{index:05}_{sanitize(prompt)[:160]}.png\"\n", 859 | " else:\n", 860 | " filename = f\"{args.timestring}_{index:05}_{args.seed}.png\"\n", 861 | " image.save(os.path.join(args.outdir, filename))\n", 862 | " if args.display_samples:\n", 863 | " display.display(image)\n", 864 | " index += 1\n", 865 | " #args.seed = next_seed(args)\n", 866 | "\n", 867 | " #print(len(all_images))\n", 868 | " if args.make_grid:\n", 869 | " grid = make_grid(all_images, nrow=int(len(all_images)/args.grid_rows))\n", 870 | " grid = rearrange(grid, 'c h w -> h w c').cpu().numpy()\n", 871 | " filename = f\"{args.timestring}_{iprompt:05d}_grid_{args.seed}.png\"\n", 872 | " grid_image = Image.fromarray(grid.astype(np.uint8))\n", 873 | " grid_image.save(os.path.join(args.outdir, filename))\n", 874 | " display.clear_output(wait=True) \n", 875 | " display.display(grid_image)\n", 876 | "\n", 877 | " return results\n", 878 | "\n", 879 | "\n", 880 | "args = SimpleNamespace(**DeforumArgs())\n", 881 | "args.timestring = time.strftime('%Y%m%d%H%M%S')\n", 882 | "\n", 883 | "API_VERSION = 5\n", 884 | "\n", 885 | "app = Flask(__name__)\n", 886 | "\n", 887 | "@app.route(\"/api/generate\", methods=[\"POST\"])\n", 888 | "def generateImages():\n", 889 | " r = request\n", 890 | " data = r.data.decode(\"utf-8\")\n", 891 | " data = json.loads(data)\n", 892 | "\n", 893 | " api_version = 0\n", 894 | "\n", 895 | " if \"api_version\" in data:\n", 896 | " api_version = int(data[\"api_version\"])\n", 897 | "\n", 898 | " if api_version != API_VERSION:\n", 899 | " abort(405)\n", 900 | "\n", 901 | " print(\"\\n\")\n", 902 | " print(\"Parameters sent from Gimp\")\n", 903 | " print(\"mode: \" + data[\"mode\"] + \", init_strength: \" + str(data[\"init_strength\"]) + \", prompt_strength: \" + str(data[\"prompt_strength\"]) + \", steps: \" + str(data[\"steps\"]) + \", width: \" + str(data[\"width\"]) + \", height: \" + str(data[\"height\"]) + \", prompt: \" + data[\"prompt\"] + \", seed: \" + str(data[\"seed\"]) + \", api_version: \" + str(data[\"api_version\"]))\n", 904 | " print(\"\\n\")\n", 905 | "\n", 906 | " args.W, args.H = map(lambda x: x - x % 64, (int(data[\"width\"]), int(data[\"height\"])))\n", 907 | " args.strength = max(0.0, min(1.0, float(data[\"init_strength\"])))\n", 908 | " args.scale = float(data[\"prompt_strength\"])\n", 909 | " args.steps = int(data[\"steps\"])\n", 910 | " args.use_init = False\n", 911 | " args.use_mask = False\n", 912 | " args.init_img = \"\"\n", 913 | " args.mask_file = \"\"\n", 914 | "\n", 915 | " if data[\"mode\"] == \"MODE_IMG2IMG\" or data[\"mode\"] == \"MODE_INPAINTING\":\n", 916 | " init_img = os.path.join(output_path, \"init.png\")\n", 917 | " img_data = base64.b64decode(data[\"init_img\"])\n", 918 | " img_file = open(init_img, \"wb+\")\n", 919 | " img_file.write(img_data)\n", 920 | " img_file.close()\n", 921 | " args.init_image = init_img\n", 922 | " args.use_init = True\n", 923 | "\n", 924 | " if data[\"mode\"] == \"MODE_INPAINTING\":\n", 925 | " args.mask_file = init_img\n", 926 | " args.strength = 0.0\n", 927 | " args.use_mask = True\n", 928 | "\n", 929 | " global prompts\n", 930 | " prompts = [data[\"prompt\"]]\n", 931 | "\n", 932 | " print(\"Parameters used for generating\")\n", 933 | " print(args)\n", 934 | "\n", 935 | " imgs_return = []\n", 936 | "\n", 937 | " for counter in range(data[\"image_count\"]):\n", 938 | " # clean up unused memory\n", 939 | " gc.collect()\n", 940 | " torch.cuda.empty_cache()\n", 941 | " \n", 942 | " args.seed = int(data[\"seed\"]) if int(data[\"seed\"]) != -1 else random.randint(0, 2**32)\n", 943 | "\n", 944 | " img = render_image_batch(args)[0]\n", 945 | "\n", 946 | " img_data = BytesIO()\n", 947 | " img.save(img_data, format=\"PNG\")\n", 948 | " img_data.seek(0)\n", 949 | " img_encoded = base64.b64encode(img_data.read())\n", 950 | " img_encoded = img_encoded.decode(\"utf-8\")\n", 951 | "\n", 952 | " img_return = {\"seed\": args.seed, \"image\": img_encoded}\n", 953 | " imgs_return.append(img_return)\n", 954 | "\n", 955 | " data_return = {\"images\": imgs_return}\n", 956 | " data_return = json.dumps(data_return)\n", 957 | "\n", 958 | " if data[\"mode\"] == \"MODE_IMG2IMG\" or data[\"mode\"] == \"MODE_INPAINTING\":\n", 959 | " if os.path.exists(init_img):\n", 960 | " os.remove(init_img)\n", 961 | "\n", 962 | " response = make_response()\n", 963 | " response.headers[\"mimetype\"] = \"application/json\"\n", 964 | " response.data = data_return\n", 965 | " return response\n", 966 | "\n", 967 | "run_with_cloudflared(app)\n", 968 | "app.run()" 969 | ], 970 | "outputs": [], 971 | "execution_count": null 972 | } 973 | ], 974 | "metadata": { 975 | "accelerator": "GPU", 976 | "colab": { 977 | "provenance": [], 978 | "private_outputs": true 979 | }, 980 | "gpuClass": "standard", 981 | "kernelspec": { 982 | "display_name": "Python 3", 983 | "name": "python3" 984 | }, 985 | "language_info": { 986 | "name": "python" 987 | } 988 | }, 989 | "nbformat": 4, 990 | "nbformat_minor": 0 991 | } 992 | -------------------------------------------------------------------------------- /discontinued/notebook/gimp-stable-diffusion.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # v1.2.0 4 | 5 | import urllib2 6 | import tempfile 7 | import os 8 | import base64 9 | import json 10 | import re 11 | import ssl 12 | import math 13 | 14 | from gimpfu import * 15 | 16 | INIT_FILE = "init.png" 17 | GENERATED_FILE = "generated.png" 18 | API_ENDPOINT = "api/generate" 19 | API_VERSION = 5 20 | 21 | HEADER_ACCEPT = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" 22 | HEADER_USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" 23 | 24 | headers = {"Accept": HEADER_ACCEPT, "User-Agent": HEADER_USER_AGENT, "Content-Type": "application/json"} 25 | ssl._create_default_https_context = ssl._create_unverified_context 26 | 27 | initFile = r"{}".format(os.path.join(tempfile.gettempdir(), INIT_FILE)) 28 | generatedFile = r"{}".format(os.path.join(tempfile.gettempdir(), GENERATED_FILE)) 29 | 30 | def getImageData(image, drawable): 31 | pdb.file_png_save_defaults(image, drawable, initFile, initFile) 32 | initImage = open(initFile, "rb") 33 | encoded = base64.b64encode(initImage.read()) 34 | return encoded 35 | 36 | def displayGenerated(images): 37 | color = pdb.gimp_context_get_foreground() 38 | pdb.gimp_context_set_foreground((0, 0, 0)) 39 | 40 | for image in images: 41 | imageFile = open(generatedFile, "wb+") 42 | imageFile.write(base64.b64decode(image["image"])) 43 | imageFile.close() 44 | 45 | imageLoaded = pdb.file_png_load(generatedFile, generatedFile) 46 | pdb.gimp_display_new(imageLoaded) 47 | # image, drawable, x, y, text, border, antialias, size, size_type, fontname 48 | pdb.gimp_text_fontname(imageLoaded, None, 2, 2, str(image["seed"]), -1, TRUE, 12, 1, "Sans") 49 | pdb.gimp_image_set_active_layer(imageLoaded, imageLoaded.layers[1]) 50 | 51 | pdb.gimp_context_set_foreground(color) 52 | return 53 | 54 | def generate(image, drawable, mode, initStrength, promptStrength, steps, seed, imageCount, prompt, url): 55 | if image.width < 384 or image.width > 1024 or image.height < 384 or image.height > 1024: 56 | raise Exception("Invalid image size. Image needs to be between 384x384 and 1024x1024.") 57 | 58 | if prompt == "": 59 | raise Exception("Please enter a prompt.") 60 | 61 | if mode == "MODE_INPAINTING" and drawable.has_alpha == 0: 62 | raise Exception("Invalid image. For inpainting an alpha channel is needed.") 63 | 64 | pdb.gimp_progress_init("", None) 65 | 66 | data = { 67 | "mode": mode, 68 | "init_strength": float(initStrength), 69 | "prompt_strength": float(promptStrength), 70 | "steps": int(steps), 71 | "prompt": prompt, 72 | "image_count": int(imageCount), 73 | "api_version": API_VERSION 74 | } 75 | 76 | if image.width % 64 != 0: 77 | width = math.floor(image.width/64) * 64 78 | else: 79 | width = image.width 80 | 81 | if image.height % 64 != 0: 82 | height = math.floor(image.height/64) * 64 83 | else: 84 | height = image.height 85 | 86 | data.update({"width": int(width)}) 87 | data.update({"height": int(height)}) 88 | 89 | if mode == "MODE_IMG2IMG" or mode == "MODE_INPAINTING": 90 | imageData = getImageData(image, drawable) 91 | data.update({"init_img": imageData}) 92 | 93 | seed = -1 if not seed else int(seed) 94 | data.update({"seed": seed}) 95 | 96 | data = json.dumps(data) 97 | 98 | url = url + "/" if not re.match(".*/$", url) else url 99 | url = re.sub("http://", "https://", url) 100 | url = url + API_ENDPOINT 101 | 102 | request = urllib2.Request(url=url, data=data, headers=headers) 103 | pdb.gimp_progress_set_text("starting dreaming now...") 104 | 105 | try: 106 | response = urllib2.urlopen(request) 107 | data = response.read() 108 | 109 | try: 110 | data = json.loads(data) 111 | displayGenerated(data["images"]) 112 | 113 | if os.path.exists(initFile): 114 | os.remove(initFile) 115 | 116 | if os.path.exists(generatedFile): 117 | os.remove(generatedFile) 118 | except Exception as ex: 119 | raise Exception(data) 120 | 121 | except Exception as ex: 122 | if isinstance(ex, urllib2.HTTPError) and ex.code == 405: 123 | raise Exception("GIMP plugin and stable-diffusion server don't match. Please update the GIMP plugin. If the error still occurs, please reopen the colab notebook.") 124 | else: 125 | raise ex 126 | 127 | return 128 | 129 | register( 130 | "stable-colab", 131 | "stable-colab", 132 | "stable-colab", 133 | "BlueTurtleAI", 134 | "BlueTurtleAI", 135 | "2022", 136 | "/AI/Stable Colab", 137 | "*", 138 | [ 139 | (PF_RADIO, "mode", "Generation Mode", "MODE_TEXT2IMG", ( 140 | ("Text -> Image", "MODE_TEXT2IMG"), 141 | ("Image -> Image", "MODE_IMG2IMG"), 142 | ("Inpainting", "MODE_INPAINTING") 143 | )), 144 | (PF_SLIDER, "initStrength", "Init Strength", 0.3, (0.0, 1.0, 0.1)), 145 | (PF_SLIDER, "promptStrength", "Prompt Strength", 7.5, (0, 20, 0.5)), 146 | (PF_SLIDER, "steps", "Steps", 50, (10, 150, 1)), 147 | (PF_STRING, "seed", "Seed (optional)", ""), 148 | (PF_SLIDER, "imageCount", "Number of images", 1, (1, 4,1)), 149 | (PF_STRING, "prompt", "Prompt", ""), 150 | (PF_STRING, "url", "Backend root URL", "") 151 | ], 152 | [], 153 | generate 154 | ) 155 | 156 | main() 157 | -------------------------------------------------------------------------------- /stablehorde/HISTORY.md: -------------------------------------------------------------------------------- 1 | # History 2 | ## GIMP Plugin 3 | ### 1.3.5 4 | Bugfixes 5 | - In some cases generation failed due to invalid prompt strength values 6 | 7 | ### 1.3.4 8 | Changes 9 | - The generated images are now transferred via Cloudflare r2. 10 | 11 | ### 1.3.3 12 | Changes 13 | - The new parameter r2 is transferred with value False. This disables for now the new Cloudflare r2 image download. 14 | 15 | ### 1.3.2 16 | Changes 17 | - Now detailed error messages are displayed. Before only the standard HTTP error messages. 18 | 19 | ### 1.3.1 20 | Changes 21 | - In the case no worker is available for image generation, now a message is displayed. 22 | - Minimum size has been reduced from 512x512 to 384x384. 23 | 24 | ### 1.3.0 25 | Changes 26 | - Inpainting is supported now 27 | 28 | ### 1.2.0 29 | Changes 30 | - Now images with sizes between 512x512 and 1024x1024 can be generated. Before only 512x512. 31 | 32 | ### 1.1.0 33 | Changes 34 | - img2img is now supported. 35 | 36 | ### 1.0.2 37 | Changes 38 | - If a new version of the plugin exists, a message is now displayed. 39 | 40 | ### 1.0.1 41 | Changes 42 | - NSWF toggle added. If you want to use an explicitly NSWF prompt, you can flag the request now accordingly. 43 | 44 | ### 1.0.0 45 | Initial version 46 | -------------------------------------------------------------------------------- /stablehorde/README.md: -------------------------------------------------------------------------------- 1 | # gimp-stable-diffusion-horde 2 | 3 | This repository includes a GIMP plugin for communication with [stablehorde](https://stablehorde.net). Stablehorde is a cluster of stable-diffusion servers run by volunteers. You can create stable-diffusion images for free without running a colab notebook or a local server. Please check the section "Limitations" to better understand where the limits are. 4 | 5 | Please check HISTORY.md for the latest changes. 6 | 7 | ## Installation 8 | ### Download files 9 | 10 | To download the files of this repository click on "Code" and select "Download ZIP". In the ZIP you will find the file "gimp-stable-diffusion-horde.py" in the subfolder "stablehorde". This is the code for the GIMP plugin. You don't need the other files in the ZIP. 11 | 12 | ### GIMP 13 | 14 | To run the plugin GIMP 2.10 is needed. 15 | 16 | 1. Start GIMP and open the preferences dialog via "edit/preferences" and scroll down to "folders". Expand "folders" and click on "plug-ins". Select the folder which includes your username and copy the path. 17 | 18 | 2. Open the file explorer, navigate to this directory and copy the file "gimp-stable-diffusion.py" from the repository into this directory. If you are on MacOS or Linux, change the file permissions to 755. 19 | 20 | 3. Restart GIMP. You should now see the new menu "AI". If you don't see this, something went wrong. Please check in this case "Troubleshooting/GIMP" for possible solutions. The menu has one item "Stablehorde". This item can't currently be selected. This only works, when you opened an image before. More about this below. 21 | 22 | ## Generate images 23 | Now we are ready for generating images. 24 | 25 | 1. Start GIMP and create/open an image with a size between 512x512 and 1024x1024. The generated image will have the size of the opened image or is a bit smaller. Check below for an explanation. 26 | - Stable diffusion only generates image sizes which are a multiple of 64. This means, if your opened image has a size of 650x512, the generated image will have a size of 640x512. 27 | - The larger the image, the longer you have to wait for generation. The reason is, that all servers in the cluster support 512x512, but not all larger sizes. 28 | 29 | 2. Select the new AI/Stablehorde menu item. A dialog will open, where you can enter the details for the image generation. 30 | 31 | - **Generation Mode:** 32 | - **Text -> Image:** Generate an image based on your prompt. 33 | - **Image -> Image:** Generate an image based on an init image and on your prompt. 34 | - **Inpainting:** Erase a part of an image and generate a new image which has the erased part filled. The erased part is filled based on your prompt. Please read the section "Inpainting" below for an explanation how inpainting works. 35 | 36 | - **Init Strength:** How much the AI should take the init image into account. The higher the value, the more will the generated image look like the init image. 0.3 is a good value to use. 37 | 38 | - **Prompt Strength:** How much the AI should follow the prompt. The higher the value, the more the AI will generate an image which looks like your prompt. 7.5 is a good value to use. 39 | 40 | - **Steps:** How many steps the AI should use to generate the image. The higher the value, the more the AI will work on details. But it also means, the longer the generation takes and the more the GPU is used. 50 is a good value to use. 41 | 42 | - **Seed:** This parameter is optional. If it is empty, a random seed will be generated on the server. If you use a seed, the same image is generated again in the case the same parameters for init strength, steps, etc. are used. A slightly different image will be generated, if the parameters are modified. You find the seed in an additional layer at the top left. 43 | 44 | - **NSFW:** If you want to send a prompt, which is excplicitly NSFW (Not Safe For Work). 45 | - If you flag your request as NSFW, only servers, which accept NSFW prompts, work on the request. It's very likely, that it takes then longer than usual to generate the image. If you don't flag the prompt, but it is NSFW, you will receive a black image. 46 | - If you didn't flag your request as NSFW and don't prompt NSFW, you will receive in some cases a black image, although it's not NSFW (false positive). Just rerun the generation in that case. 47 | 48 | - **Prompt:** How the generated image should look like. 49 | 50 | - **API key:** This parameter is optional. If you don't enter an API key, you run the image generation as anonymous. The downside is, that you will have then the lowest priority in the generation queue. For that reason it is recommended registering for free on [stablehorde](https://stablehorde.net) and getting an API key. 51 | 52 | - **Max Wait:** The maximum time in minutes you want to wait until image generation is finished. When the max time is reached, a timeout happens and the generation request is stopped. 53 | 54 | 3. Click on the OK button. The values you inserted into the dialog will be transmitted to the server, which dispatches the request now to one of the stable-diffusion servers in the cluster. Your generation request is added to queue. You can see the queue position and the remaining wait time in the status bar of the dialog. When the image has been generated successfully, it will be shown as a new image in GIMP. The used seed is shown at the top left in an additional layer. 55 | 56 | ## Inpainting 57 | Inpainting means replacing a part of an existing image. For example if you don't like the face on an image, you can replace it. **Inpainting is currently still in experimental stage. So, please don't expect perfect results.** The experimental stage is caused by the server side and not by GIMP. 58 | 59 | For inpainting it's necessary to prepare the input image because the AI needs to know which part you want to replace. For this purpose you replace this image part by transparency. To do so, open the init image in GIMP and select "Layer/Transparency/Add alpha channel". Select now the part of the image which should be replaced and delete it. You can also use the eraser tool. 60 | 61 | For the prompt you use now a description of the new image. For example the image shows currently "a little girl running over a meadow with a balloon" and you want to replace the balloon by a parachute. You just write now "a little girl running over a meadow with a parachute". 62 | 63 | ## Limitations 64 | - **Stability:** Stablehorde is still pretty new and under heavy development. So, it's not unlikely, that the servers are not available for some time or unexpected errors occur. 65 | 66 | - **Generation speed:** Stablehorde is a cluster of stable-diffusion servers run by volunteers. The generation speed depends on how many servers are in the cluster, which hardware they use and how many others want to generate with stablehorde. The upside is, that stablehorde is free to use, the downside that the generation speed is unpredictable. 67 | 68 | - **Privacy:** The privacy stablehorde offers is similar to generating in a public discord channel. So, please assume, that neither your prompts nor your generated images are private. 69 | 70 | - **Features:** Currently text2img, img2img and inpainting are supported. As soon as stablehorde supports outpainting, this will be available in the plugin too. 71 | 72 | ## Troubleshooting 73 | ### GIMP 74 | #### AI menu is not shown 75 | ##### Linux 76 | - If you get this error ```gimp: LibGimpBase-WARNING: gimp: gimp_wire_read(): error```, it's very likely, that you have a GIMP version installed, which doesn't include Python. Check, if you have got the menu "Filters > Python-Fu > Console". If it is missing, please install GIMP from here: https://flathub.org/apps/details/org.gimp.GIMP. 77 | 78 | - Please try https://flathub.org/apps/details/org.gimp.GIMP if you have got any other problems. 79 | 80 | ##### macOS 81 | - Please double check, if the permissions of the plugin py file are set to 755. It seems, that changing permissions doesn't work via the file manager. Please open a terminal, cd to the plugins directory and run "chmod ugo+x *py". 82 | 83 | ##### macOS/Linux 84 | - Open a terminal an try to run the plugin py file manually via ```python /gimp-stable-diffusion.py```. You should see the error message, that "gimpfu" is unknown. Make sure, that you are running Python 2, as this version is used by GIMP. If other errors occur, please reinstall GIMP. 85 | 86 | ## FAQ 87 | **Why is the generated image smaller than the opened image?** Stable-diffusion only generates image sizes which are a multiple of 64. This means, if your opened image has a size of 650x512, the generated image will have a size of 640x512. 88 | 89 | **Will GIMP 3 be supported?** Yes, the plugin will be ported to GIMP 3. 90 | 91 | **Will outpainting be supported?** Pretty likely outpainting will be supported. This depends on which features the stablehorde cluster supports. 92 | 93 | **How do I report an error or request a new feature?** Please open a new issue in this repository. 94 | 95 | 96 | -------------------------------------------------------------------------------- /stablehorde/gimp-stable-diffusion-horde.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # v1.3.5 4 | 5 | import urllib2 6 | import tempfile 7 | import os 8 | import base64 9 | import json 10 | import ssl 11 | import sched, time 12 | import math 13 | import gimp 14 | import re 15 | 16 | from gimpfu import * 17 | 18 | VERSION = 135 19 | INIT_FILE = "init.png" 20 | GENERATED_FILE = "generated.png" 21 | API_ROOT = "https://stablehorde.net/api/v2/" 22 | 23 | # check every 5 seconds 24 | CHECK_WAIT = 5 25 | checkMax = None 26 | 27 | ssl._create_default_https_context = ssl._create_unverified_context 28 | 29 | initFile = r"{}".format(os.path.join(tempfile.gettempdir(), INIT_FILE)) 30 | generatedFile = r"{}".format(os.path.join(tempfile.gettempdir(), GENERATED_FILE)) 31 | s = sched.scheduler(time.time, time.sleep) 32 | 33 | checkCounter = 0 34 | id = None 35 | 36 | def checkUpdate(): 37 | try: 38 | gimp.get_data("update_checked") 39 | updateChecked = True 40 | except Exception as ex: 41 | updateChecked = False 42 | 43 | if updateChecked is False: 44 | try: 45 | url = "https://raw.githubusercontent.com/blueturtleai/gimp-stable-diffusion/main/stablehorde/version.json" 46 | response = urllib2.urlopen(url) 47 | data = response.read() 48 | data = json.loads(data) 49 | gimp.set_data("update_checked", "1") 50 | 51 | if VERSION < int(data["version"]): 52 | pdb.gimp_message(data["message"]) 53 | except Exception as ex: 54 | ex = ex 55 | 56 | def getImageData(image, drawable): 57 | pdb.file_png_save_defaults(image, drawable, initFile, initFile) 58 | initImage = open(initFile, "rb") 59 | encoded = base64.b64encode(initImage.read()) 60 | return encoded 61 | 62 | def displayGenerated(images): 63 | color = pdb.gimp_context_get_foreground() 64 | pdb.gimp_context_set_foreground((0, 0, 0)) 65 | 66 | for image in images: 67 | if re.match("^https.*", image["img"]): 68 | response = urllib2.urlopen(image["img"]) 69 | bytes = response.read() 70 | else: 71 | bytes = base64.b64decode(image["img"]) 72 | 73 | imageFile = open(generatedFile, "wb+") 74 | imageFile.write(bytes) 75 | imageFile.close() 76 | 77 | imageLoaded = pdb.file_webp_load(generatedFile, generatedFile) 78 | pdb.gimp_display_new(imageLoaded) 79 | # image, drawable, x, y, text, border, antialias, size, size_type, fontname 80 | pdb.gimp_text_fontname(imageLoaded, None, 2, 2, str(image["seed"]), -1, TRUE, 12, 1, "Sans") 81 | pdb.gimp_image_set_active_layer(imageLoaded, imageLoaded.layers[1]) 82 | 83 | pdb.gimp_context_set_foreground(color) 84 | return 85 | 86 | def getImages(): 87 | url = API_ROOT + "generate/status/" + id 88 | response = urllib2.urlopen(url) 89 | data = response.read() 90 | data = json.loads(data) 91 | 92 | return data["generations"] 93 | 94 | def checkStatus(): 95 | url = API_ROOT + "generate/check/" + id 96 | response = urllib2.urlopen(url) 97 | data = response.read() 98 | data = json.loads(data) 99 | 100 | global checkCounter 101 | checkCounter = checkCounter + 1 102 | 103 | if data["processing"] == 0: 104 | text = "Queue position: " + str(data["queue_position"]) + ", Wait time: " + str(data["wait_time"]) + "s" 105 | elif data["processing"] > 0: 106 | text = "Generating..." 107 | 108 | pdb.gimp_progress_set_text(text) 109 | 110 | if checkCounter < checkMax and data["done"] is False: 111 | if data["is_possible"] is True: 112 | s.enter(CHECK_WAIT, 1, checkStatus, ()) 113 | s.run() 114 | else: 115 | raise Exception("Currently no worker available to generate your image. Please try again later.") 116 | elif checkCounter == checkMax: 117 | minutes = (checkMax * CHECK_WAIT)/60 118 | raise Exception("Image generation timed out after " + str(minutes) + " minutes. Please try it again later.") 119 | elif data["done"] == True: 120 | return 121 | 122 | def generate(image, drawable, mode, initStrength, promptStrength, steps, seed, nsfw, prompt, apikey, maxWaitMin): 123 | if image.width < 384 or image.width > 1024 or image.height < 384 or image.height > 1024: 124 | raise Exception("Invalid image size. Image needs to be between 384x384 and 1024x1024.") 125 | 126 | if prompt == "": 127 | raise Exception("Please enter a prompt.") 128 | 129 | if mode == "MODE_INPAINTING" and drawable.has_alpha == 0: 130 | raise Exception("Invalid image. For inpainting an alpha channel is needed.") 131 | 132 | pdb.gimp_progress_init("", None) 133 | 134 | global checkMax 135 | checkMax = (maxWaitMin * 60)/CHECK_WAIT 136 | 137 | try: 138 | params = { 139 | "cfg_scale": float(promptStrength), 140 | "steps": int(steps), 141 | "seed": seed 142 | } 143 | 144 | data = { 145 | "params": params, 146 | "prompt": prompt, 147 | "nsfw": nsfw, 148 | "censor_nsfw": False, 149 | "r2": True 150 | } 151 | 152 | if image.width % 64 != 0: 153 | width = math.floor(image.width/64) * 64 154 | else: 155 | width = image.width 156 | 157 | if image.height % 64 != 0: 158 | height = math.floor(image.height/64) * 64 159 | else: 160 | height = image.height 161 | 162 | params.update({"width": int(width)}) 163 | params.update({"height": int(height)}) 164 | 165 | if mode == "MODE_IMG2IMG": 166 | init = getImageData(image, drawable) 167 | data.update({"source_image": init}) 168 | data.update({"source_processing": "img2img"}) 169 | params.update({"denoising_strength": (1 - float(initStrength))}) 170 | elif mode == "MODE_INPAINTING": 171 | init = getImageData(image, drawable) 172 | models = ["stable_diffusion_inpainting"] 173 | data.update({"source_image": init}) 174 | data.update({"source_processing": "inpainting"}) 175 | data.update({"models": models}) 176 | 177 | data = json.dumps(data) 178 | 179 | apikey = "0000000000" if not apikey else apikey 180 | 181 | headers = {"Content-Type": "application/json", "Accept": "application/json", "apikey": apikey} 182 | url = API_ROOT + "generate/async" 183 | 184 | request = urllib2.Request(url=url, data=data, headers=headers) 185 | 186 | response = urllib2.urlopen(request) 187 | data = response.read() 188 | 189 | try: 190 | data = json.loads(data) 191 | global id 192 | id = data["id"] 193 | except Exception as ex: 194 | raise Exception(data) 195 | 196 | checkStatus() 197 | images = getImages() 198 | displayGenerated(images) 199 | 200 | except urllib2.HTTPError as ex: 201 | try: 202 | data = ex.read() 203 | data = json.loads(data) 204 | 205 | if "message" in data: 206 | message = data["message"] 207 | else: 208 | message = str(ex) 209 | except Exception: 210 | message = str(ex) 211 | 212 | raise Exception(message) 213 | except Exception as ex: 214 | raise ex 215 | finally: 216 | pdb.gimp_progress_end() 217 | checkUpdate() 218 | 219 | return 220 | 221 | register( 222 | "stable-horde", 223 | "stable-horde", 224 | "stable-horde", 225 | "BlueTurtleAI", 226 | "BlueTurtleAI", 227 | "2022", 228 | "/AI/Stablehorde", 229 | "*", 230 | [ 231 | (PF_RADIO, "mode", "Generation Mode", "MODE_TEXT2IMG", ( 232 | ("Text -> Image", "MODE_TEXT2IMG"), 233 | ("Image -> Image", "MODE_IMG2IMG"), 234 | ("Inpainting", "MODE_INPAINTING") 235 | )), 236 | (PF_SLIDER, "initStrength", "Init Strength", 0.3, (0, 1, 0.1)), 237 | (PF_SLIDER, "promptStrength", "Prompt Strength", 8, (0, 20, 1)), 238 | (PF_SLIDER, "steps", "Steps", 50, (10, 150, 1)), 239 | (PF_STRING, "seed", "Seed (optional)", ""), 240 | (PF_TOGGLE, "nsfw", "NSFW", False), 241 | (PF_STRING, "prompt", "Prompt", ""), 242 | (PF_STRING, "apiKey", "API key (optional)", ""), 243 | (PF_SLIDER, "maxWaitMin", "Max Wait (minutes)", 5, (1, 5, 1)) 244 | ], 245 | [], 246 | generate 247 | ) 248 | 249 | main() 250 | -------------------------------------------------------------------------------- /stablehorde/version.json: -------------------------------------------------------------------------------- 1 | {"version": 135, "message": "New version of the stablehorde plugin is available. Please update!"} 2 | --------------------------------------------------------------------------------