├── .github ├── FUNDING.yml └── workflows │ └── publish.yml ├── README.md ├── __init__.py ├── nodes.py ├── pyproject.toml └── workflows ├── simple higher scale_.png └── tinybottle.png /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | patreon: extraltodeus 4 | -------------------------------------------------------------------------------- /.github/workflows/publish.yml: -------------------------------------------------------------------------------- 1 | name: Publish to Comfy registry 2 | on: 3 | workflow_dispatch: 4 | push: 5 | branches: 6 | - main 7 | - master 8 | paths: 9 | - "pyproject.toml" 10 | 11 | jobs: 12 | publish-node: 13 | name: Publish Custom Node to registry 14 | runs-on: ubuntu-latest 15 | steps: 16 | - name: Check out code 17 | uses: actions/checkout@v4 18 | - name: Publish Custom Node 19 | uses: Comfy-Org/publish-node-action@main 20 | with: 21 | ## Add your own personal access token to your Github Repository secrets and reference it here. 22 | personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }} 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Stable-Diffusion-temperature-settings 2 | Provides the ability to set the temperature for both UNET and CLIP. For ComfyUI. 3 | 4 | I also added a togglable function compatible with SD 1.x models and SDXL/Turbo which helps to preserve quality weither it is for downscaling or upscaling. 5 | 6 | ## Usage 7 | 8 | Like any other model patch: 9 | 10 | ![image](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/baafc52a-452b-499f-8aec-2092b019e71f) 11 | 12 | 13 | ## CLIP temperature at 0.75, 1 and 1.25. Prompt "a bio-organic living plant spaceship" 14 | 15 | ![combined_image_new](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/12034834-43d0-44a5-a603-6c87d1bc6e5d) 16 | 17 | 18 | ## The temperature scaling option allows to avoid smudged details during an upscale: 19 | 20 | Without / with 21 | 22 | ![234523423](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/9fad70c5-bbdc-4243-88a6-2963e7b47491) 23 | 24 | This does not include tiled upscales since the tiling uses smaller latents as the workaround for this well known issue. 25 | 26 | ## Using SDXL, 512x512, 256x256, 128x128 without / with modification on the U-Net: 27 | 28 | ![335007111-bf6d7ef0-9c18-4436-8037-6b60a6a37ce2](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/e47775e0-1b36-46f9-9eac-5467ed8b6715)![335007043-0c4540ab-1840-4230-940a-07a9e38ef38a](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/3aecf1b1-85a8-4362-a7e1-e76048ca5f4b) 29 | 30 | ![335007132-3379081c-2c4e-4af0-ba92-b57031b3845b](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/ec1680c2-bb8f-4c50-855d-aeb5e0858a05)![335007062-a4dc0de9-68b7-4158-b4fa-6c607862d04a](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/1304717f-91d6-48a9-991f-b9a725ddff9d) 31 | 32 | 33 | ![335007140-10991fbe-4123-46d2-8069-cfaece9e77ec](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/b8e33680-cc4c-420c-b128-6b790a05ca12)![335007077-4d17e360-0e28-4fd2-98ae-8f6944114815](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/ff1c7ef5-1bbb-49bb-b53d-f14c4efe44bd) 34 | 35 | 36 | # Other examples on SDXL. 37 | 38 | 256x256 without / with 39 | 40 | 41 | ![00752UI_00001_](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/66659a07-0532-4fc9-9993-7229ba3160e8) 42 | ![00750UI_00001_](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/e8525a12-5d9c-41a3-ba5a-7c2fd65e2c2c) 43 | 44 | 45 | 46 | ## Using SD v1-5-pruned-emaonly: 47 | 48 | This is one of my first tests and I only display it as the proof of concept, exposing the possibility to fix the scaling issues with the U-Net. 49 | 50 | - Lower temperature applied to all layers except input 1 and 2, output 9, 10 and 11 which were ignored. 51 | - At 0.71. 52 | - Only self-attention 53 | - Resolution at 1024*512 54 | 55 | ![combined_pair_2](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/5e5403ea-2cb3-462c-a9f1-6cc7b1ddbaea) 56 | ![combined_pair_1](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/84fed1e4-a7ba-4f2a-8562-e3573f0aab8f) 57 | ![combined_pair_3](https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings/assets/15731540/c6703c21-0d63-404e-9bf8-3a7c580f59e7) 58 | 59 | 60 | 61 | 62 | # Patreon 63 | 64 | Provide an incentive to contributors: 65 | 66 | https://www.patreon.com/extraltodeus 67 | 68 | 69 | # Pro tip: 70 | 71 | Did you know that my first activity is to write creative model merging functions? 72 | 73 | While the code is too much of a mess to be shared, I do expose and share my models. You can find them in this [gallery](https://github.com/Extraltodeus/shared_models_galleries)! 😁 74 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | from .nodes import * 2 | 3 | NODE_CLASS_MAPPINGS = { 4 | "Unet Temperature":UnetTemperaturePatch, 5 | "CLIP Temperature":CLIPTemperaturePatch, 6 | } 7 | -------------------------------------------------------------------------------- /nodes.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import math 3 | import types 4 | import comfy.model_management 5 | from functools import partial 6 | from math import * 7 | from comfy.model_patcher import ModelPatcher 8 | from comfy.sd import CLIP 9 | 10 | EPSILON = 1e-16 11 | 12 | SD_layer_dims = { 13 | "SD1" : {"input_1": 4096,"input_2": 4096,"input_4": 1024,"input_5": 1024,"input_7": 256,"input_8": 256,"middle_0": 64,"output_3": 256,"output_4": 256,"output_5": 256,"output_6": 1024,"output_7": 1024,"output_8": 1024,"output_9": 4096,"output_10": 4096,"output_11": 4096}, 14 | "SDXL": {"input_4": 4096,"input_5": 4096,"input_7": 1024,"input_8": 1024,"middle_0": 1024,"output_0": 1024,"output_1": 1024,"output_2": 1024,"output_3": 4096,"output_4": 4096,"output_5": 4096}, 15 | "Disabled":{} 16 | } 17 | 18 | models_by_size = {"1719049928": "SD1", "5134967368":"SDXL"} 19 | 20 | def cv_temperature(input_tensor, auto_mode="normal"): 21 | if "creative" in auto_mode: 22 | tensor_std = input_tensor.std() 23 | temperature = (torch.std(torch.abs(input_tensor - tensor_std))/tensor_std) 24 | temperature = 1 / temperature 25 | del tensor_std 26 | else: 27 | temperature = torch.std(input_tensor) 28 | if "squared" in auto_mode: 29 | temperature = temperature ** 2 30 | elif "sqrt" in auto_mode: 31 | temperature = temperature ** .5 32 | if not "reversed" in auto_mode: 33 | temperature = 1 / temperature 34 | return temperature 35 | 36 | def should_scale(mname,lname,q2): 37 | if mname == "" or mname == "CLIP": return False 38 | if mname != "Disabled" and lname in SD_layer_dims[mname]: 39 | if lname not in SD_layer_dims[mname]: 40 | return False 41 | return q2 != SD_layer_dims[mname][lname] 42 | return False 43 | 44 | class temperature_patcher(): 45 | def __init__(self, temperature, layer_name = "", model_name="", eval_string="", auto_temp="disabled", 46 | Original_scale=512, Target_scale_X=512, Target_scale_Y=512, rescale_adjust=1, 47 | scale_before=False,scale_after=False): 48 | self.temperature = max(temperature,EPSILON) 49 | self.layer_name = layer_name 50 | self.model_name = model_name 51 | self.eval_string = eval_string 52 | self.auto_temp = auto_temp 53 | self.Original_scale = Original_scale 54 | self.Target_scale_X = Target_scale_X 55 | self.Target_scale_Y = Target_scale_Y 56 | self.rescale_adjust = rescale_adjust 57 | self.scale_before = scale_before 58 | self.scale_after = scale_after 59 | 60 | def pytorch_attention_with_temperature(self, q, k, v, extra_options, mask=None, attn_precision=None): 61 | heads = extra_options if isinstance(extra_options, int) else extra_options['n_heads'] 62 | b, _, dim_head = q.shape 63 | dim_head //= heads 64 | q, k, v = map( 65 | lambda t: t.view(b, -1, heads, dim_head).transpose(1, 2), 66 | (q, k, v), 67 | ) 68 | 69 | if self.auto_temp != "disabled": 70 | extra_temperature = cv_temperature({'q_': q, 'k_': k, 'v_': v}[self.auto_temp[:2]], self.auto_temp) 71 | else: 72 | extra_temperature = 1 73 | 74 | temperature_pre_scale = 1 75 | if self.scale_before: 76 | if should_scale(self.model_name, self.layer_name,q.size(-2)): 77 | ldim = SD_layer_dims[self.model_name][self.layer_name] 78 | if self.eval_string != "": 79 | temperature_pre_scale = eval(self.eval_string) 80 | else: 81 | temperature_pre_scale = log(q.size(-2), ldim) # that's the actual resccale, everything else is for testing purpose 82 | elif (self.Target_scale_X*self.Target_scale_Y) != self.Original_scale**2: 83 | temperature_pre_scale = log((self.Target_scale_X*self.Target_scale_Y)**.5, self.Original_scale) 84 | temperature_scale = self.temperature / temperature_pre_scale 85 | 86 | scale = 1 / (math.sqrt(q.size(-1)) * temperature_scale * extra_temperature) 87 | out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, is_causal=False, scale=scale) 88 | 89 | if self.scale_after: 90 | if should_scale(self.model_name, self.layer_name,q.size(-2)): 91 | ldim = SD_layer_dims[self.model_name][self.layer_name] 92 | if self.eval_string != "": 93 | out *= eval(self.eval_string) 94 | else: 95 | out *= log(q.size(-2), ldim) 96 | elif (self.Target_scale_X*self.Target_scale_Y) != self.Original_scale**2: 97 | out *= log((self.Target_scale_X*self.Target_scale_Y)**.5, self.Original_scale) 98 | 99 | if self.rescale_adjust != 1: 100 | out *= self.rescale_adjust 101 | 102 | out = ( 103 | out.transpose(1, 2).reshape(b, -1, heads * dim_head) 104 | ) 105 | return out 106 | 107 | class UnetTemperaturePatch: 108 | @classmethod 109 | def INPUT_TYPES(s): 110 | required_inputs = {} 111 | required_inputs["model"] = ("MODEL",) 112 | required_inputs["Temperature"] = ("FLOAT", {"default": 1.0, "min": 0.0, "max": 10.0, "step": 0.01, "round": 0.01}) 113 | required_inputs["Attention"] = (["both","self","cross"],) 114 | required_inputs["Dynamic_Scale_Temperature"] = ("BOOLEAN", {"default": False}) 115 | required_inputs["Dynamic_Scale_Output"] = ("BOOLEAN", {"default": True}) 116 | # required_inputs["eval_string"] = ("STRING", {"multiline": True}) 117 | return {"required": required_inputs} 118 | 119 | TOGGLES = {} 120 | RETURN_TYPES = ("MODEL","STRING",) 121 | RETURN_NAMES = ("Model","String",) 122 | FUNCTION = "patch" 123 | 124 | CATEGORY = "model_patches/Temperature" 125 | 126 | def patch(self, model, Temperature, Attention, Dynamic_Scale_Temperature, Dynamic_Scale_Output, Dynamic_Scale_Adjust=1, eval_string="", **kwargs): 127 | Dynamic_Scale_Attention = Dynamic_Scale_Temperature or Dynamic_Scale_Output 128 | if not Dynamic_Scale_Attention and Temperature == 1: 129 | print("Dynamic_Scale_Attention/temperature: no patch applied.") 130 | return (model, "Fully disabled",) 131 | if Dynamic_Scale_Attention and str(model.size) in models_by_size: 132 | model_name = models_by_size[str(model.size)] 133 | print(f"Model detected for scaling: {model_name}") 134 | else: 135 | if Dynamic_Scale_Attention: 136 | print("No compatible model detected for dynamic scale attention!") 137 | model_name = "Disabled" 138 | 139 | m = model.clone() 140 | levels = ["input","middle","output"] 141 | layer_names = {f"{l}_{n}": True for l in levels for n in range(12)} 142 | 143 | for key, toggle in layer_names.items(): 144 | current_level = key.split("_")[0] 145 | b_number = int(key.split("_")[1]) 146 | patcher = temperature_patcher(Temperature,layer_name=key,model_name=model_name, eval_string=eval_string, 147 | rescale_adjust=Dynamic_Scale_Adjust, scale_before=Dynamic_Scale_Temperature, 148 | scale_after=Dynamic_Scale_Output) 149 | if Attention in ["both","self"]: 150 | m.set_model_attn1_replace(patcher.pytorch_attention_with_temperature, current_level, b_number) 151 | if Attention in ["both","cross"]: 152 | m.set_model_attn2_replace(patcher.pytorch_attention_with_temperature, current_level, b_number) 153 | 154 | parameters_as_string = f"Temperature: {Temperature}\nAttention: {Attention}\nDynamic scale: {Dynamic_Scale_Attention}" 155 | return (m, parameters_as_string,) 156 | 157 | class CLIPTemperaturePatch: 158 | @classmethod 159 | def INPUT_TYPES(cls): 160 | return {"required": { "clip": ("CLIP",), 161 | "Temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 10.0, "step": 0.01}), 162 | # "Auto_temp": ("BOOLEAN", {"default": False}) # It's just experimental but uncomment it if you want to try. 163 | }} 164 | 165 | RETURN_TYPES = ("CLIP",) 166 | FUNCTION = "patch" 167 | CATEGORY = "model_patches/Temperature" 168 | 169 | def patch(self, clip, Temperature, Auto_temp=False): 170 | c = clip.clone() 171 | def custom_optimized_attention(device, mask=None, small_input=True): 172 | return temperature_patcher(temperature=Temperature,auto_temp="k_creative" if Auto_temp else "disabled").pytorch_attention_with_temperature 173 | def new_forward(self, x, mask=None, intermediate_output=None): 174 | optimized_attention = custom_optimized_attention(x.device, mask=mask is not None, small_input=True) 175 | 176 | if intermediate_output is not None: 177 | if intermediate_output < 0: 178 | intermediate_output = len(self.layers) + intermediate_output 179 | 180 | intermediate = None 181 | for i, l in enumerate(self.layers): 182 | x = l(x, mask, optimized_attention) 183 | if i == intermediate_output: 184 | intermediate = x.clone() 185 | return x, intermediate 186 | 187 | if getattr(c.patcher.model, "clip_g", None) is not None: 188 | c.patcher.add_object_patch("clip_g.transformer.text_model.encoder.forward", partial(new_forward, c.patcher.model.clip_g.transformer.text_model.encoder)) 189 | 190 | if getattr(c.patcher.model, "clip_l", None) is not None: 191 | c.patcher.add_object_patch("clip_l.transformer.text_model.encoder.forward", partial(new_forward, c.patcher.model.clip_l.transformer.text_model.encoder)) 192 | 193 | return (c,) 194 | 195 | class CLIPTemperatureWithScalePatch: 196 | @classmethod 197 | def INPUT_TYPES(cls): 198 | return {"required": { "clip": ("CLIP",), 199 | "Temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 10.0, "step": 0.01}), 200 | "Original_scale": ("INT", {"default": 512, "min": 64, "max": 100000.0, "step": 64}), 201 | "Target_scale_X": ("INT", {"default": 512, "min": 64, "max": 100000.0, "step": 64}), 202 | "Target_scale_Y": ("INT", {"default": 512, "min": 64, "max": 100000.0, "step": 64}), 203 | "Dynamic_Scale_Temperature": ("BOOLEAN", {"default": False}), 204 | "Dynamic_Scale_Output": ("BOOLEAN", {"default": False}) 205 | }} 206 | 207 | RETURN_TYPES = ("CLIP",) 208 | FUNCTION = "patch" 209 | CATEGORY = "model_patches/Temperature" 210 | 211 | def patch(self, clip, Temperature, Dynamic_Scale_Temperature, Dynamic_Scale_Output, Original_scale=512, Target_scale_X=512, Target_scale_Y=512, Scale_Adjust=1): 212 | c = clip.clone() 213 | def custom_optimized_attention(device, mask=None, small_input=True): 214 | return temperature_patcher(temperature=Temperature,Target_scale_X=Target_scale_X,Target_scale_Y=Target_scale_Y,Original_scale=Original_scale, rescale_adjust=Scale_Adjust, scale_before=Dynamic_Scale_Temperature, scale_after=Dynamic_Scale_Output).pytorch_attention_with_temperature 215 | 216 | def new_forward(self, x, mask=None, intermediate_output=None): 217 | optimized_attention = custom_optimized_attention(x.device, mask=mask is not None, small_input=True) 218 | 219 | if intermediate_output is not None: 220 | if intermediate_output < 0: 221 | intermediate_output = len(self.layers) + intermediate_output 222 | 223 | intermediate = None 224 | for i, l in enumerate(self.layers): 225 | x = l(x, mask, optimized_attention) 226 | if i == intermediate_output: 227 | intermediate = x.clone() 228 | return x, intermediate 229 | 230 | if getattr(c.patcher.model, "clip_g", None) is not None: 231 | c.patcher.add_object_patch("clip_g.transformer.text_model.encoder.forward", partial(new_forward, c.patcher.model.clip_g.transformer.text_model.encoder)) 232 | 233 | if getattr(c.patcher.model, "clip_l", None) is not None: 234 | c.patcher.add_object_patch("clip_l.transformer.text_model.encoder.forward", partial(new_forward, c.patcher.model.clip_l.transformer.text_model.encoder)) 235 | 236 | return (c,) 237 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "stable-diffusion-temperature-settings" 3 | description = "Provides the ability to set the temperature for both UNET and CLIP. For ComfyUI." 4 | version = "1.0.0" 5 | license = "LICENSE" 6 | 7 | [project.urls] 8 | Repository = "https://github.com/Extraltodeus/Stable-Diffusion-temperature-settings" 9 | # Used by Comfy Registry https://comfyregistry.org 10 | 11 | [tool.comfy] 12 | PublisherId = "extraltodeus" 13 | DisplayName = "Stable-Diffusion-temperature-settings" 14 | Icon = "" 15 | -------------------------------------------------------------------------------- /workflows/simple higher scale_.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Extraltodeus/Stable-Diffusion-temperature-settings/125203d2f048e42f4cf98379d92444b5c474d260/workflows/simple higher scale_.png -------------------------------------------------------------------------------- /workflows/tinybottle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Extraltodeus/Stable-Diffusion-temperature-settings/125203d2f048e42f4cf98379d92444b5c474d260/workflows/tinybottle.png --------------------------------------------------------------------------------