├── Flux.1-dev-Union_CN-jupyter.ipynb
├── Flux.1-dev-XLabs_CN-jupyter.ipynb
└── README.md
/Flux.1-dev-Union_CN-jupyter.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {
7 | "id": "VjYy0F2gZIPR",
8 | "cellView": "form"
9 | },
10 | "outputs": [],
11 | "source": [
12 | "#@markdown Install
\n",
13 | "\n",
14 | "%cd /content\n",
15 | "!git clone -b totoro5 https://github.com/camenduru/ComfyUI /content/TotoroUI\n",
16 | "!git clone -b totoro https://github.com/LucipherDev/ComfyUI-GGUF /content/TotoroUI/custom_nodes/TotoroUI-GGUF\n",
17 | "%cd /content/TotoroUI\n",
18 | "\n",
19 | "!pip install -q torchsde einops diffusers accelerate xformers==0.0.28.post2\n",
20 | "!pip install -q git+https://github.com/LucipherDev/controlnet_aux.git\n",
21 | "!pip install -q -r /content/TotoroUI/custom_nodes/TotoroUI-GGUF/requirements.txt\n",
22 | "!apt -y install -qq aria2\n",
23 | "\n",
24 | "import nodes\n",
25 | "\n",
26 | "if not nodes.load_custom_node(\"custom_nodes/TotoroUI-GGUF\"):\n",
27 | " raise Exception(\"Failed to load GGUF custom node\")"
28 | ]
29 | },
30 | {
31 | "cell_type": "code",
32 | "source": [
33 | "#@markdown Load Models
\n",
34 | "\n",
35 | "import torch\n",
36 | "from nodes import NODE_CLASS_MAPPINGS\n",
37 | "\n",
38 | "DualCLIPLoaderGGUF = NODE_CLASS_MAPPINGS[\"DualCLIPLoaderGGUF\"]()\n",
39 | "UnetLoaderGGUF = NODE_CLASS_MAPPINGS[\"UnetLoaderGGUF\"]()\n",
40 | "VAELoader = NODE_CLASS_MAPPINGS[\"VAELoader\"]()\n",
41 | "ControlNetLoader = NODE_CLASS_MAPPINGS[\"ControlNetLoader\"]()\n",
42 | "\n",
43 | "system_ram = \"low ram (Q2_K)\" # @param [\"low ram (Q2_K)\", \"high ram (Q8_0)\"]\n",
44 | "\n",
45 | "if system_ram == \"low ram (Q2_K)\":\n",
46 | " print(f\"Downloading Flux1-dev-Q2_K...\")\n",
47 | " !aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/city96/FLUX.1-dev-gguf/resolve/main/flux1-dev-Q2_K.gguf -d /content/TotoroUI/models/unet -o flux1-dev.gguf\n",
48 | "elif system_ram == \"high ram (Q8_0)\":\n",
49 | " print(f\"Downloading Flux1-dev-Q8_0...\")\n",
50 | " !aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/city96/FLUX.1-dev-gguf/resolve/main/flux1-dev-Q8_0.gguf -d /content/TotoroUI/models/unet -o flux1-dev.gguf\n",
51 | "\n",
52 | "print(\"Downloading VAE...\")\n",
53 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/FLUX.1-dev/resolve/main/ae.sft -d /content/TotoroUI/models/vae -o ae.sft\n",
54 | "\n",
55 | "print(\"Downloading Clips...\")\n",
56 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/FLUX.1-dev/resolve/main/clip_l.safetensors -d /content/TotoroUI/models/clip -o clip_l.safetensors\n",
57 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/resolve/main/t5-v1_1-xxl-encoder-Q3_K_L.gguf -d /content/TotoroUI/models/clip -o t5-v1_1-xxl-encoder-Q3_K_L.gguf\n",
58 | "\n",
59 | "print(\"Downloading Controlnet Preprocessors...\")\n",
60 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/Kijai/DepthAnythingV2-safetensors/resolve/main/depth_anything_v2_vitl_fp16.safetensors -d /content/TotoroUI/models/controlnet_preprocessors -o depth_anything_v2_vitl_fp16.safetensors\n",
61 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/ControlNetHED.pth -d /content/TotoroUI/models/controlnet_preprocessors -o ControlNetHED.pth\n",
62 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/body_pose_model.pth -d /content/TotoroUI/models/controlnet_preprocessors -o body_pose_model.pth\n",
63 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/hand_pose_model.pth -d /content/TotoroUI/models/controlnet_preprocessors -o hand_pose_model.pth\n",
64 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/facenet.pth -d /content/TotoroUI/models/controlnet_preprocessors -o facenet.pth\n",
65 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/sk_model.pth -d /content/TotoroUI/models/controlnet_preprocessors -o sk_model.pth\n",
66 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/sk_model2.pth -d /content/TotoroUI/models/controlnet_preprocessors -o sk_model2.pth\n",
67 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/fal/teed/resolve/main/5_model.pth -d /content/TotoroUI/models/controlnet_preprocessors -o 5_model.pth\n",
68 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/table5_pidinet.pth -d /content/TotoroUI/models/controlnet_preprocessors -o table5_pidinet.pth\n",
69 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/scannet.pt -d /content/TotoroUI/models/controlnet_preprocessors -o scannet.pt\n",
70 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/mlsd_large_512_fp32.pth -d /content/TotoroUI/models/controlnet_preprocessors -omlsd_large_512_fp32.pth\n",
71 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/netG.pth -d /content/TotoroUI/models/controlnet_preprocessors -o netG.pth\n",
72 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/ybelkada/segment-anything/resolve/main/checkpoints/sam_vit_b_01ec64.pth -d /content/TotoroUI/models/controlnet_preprocessors -o sam_vit_b.pth\n",
73 | "\n",
74 | "print(\"Downloading Controlnet Union...\")\n",
75 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/resolve/main/diffusion_pytorch_model.safetensors -d /content/TotoroUI/models/controlnet -o FLUX.1-dev-Controlnet-Union-Pro.safetensors\n",
76 | "\n",
77 | "with torch.inference_mode():\n",
78 | " print(\"Loading Clips...\")\n",
79 | " clip = DualCLIPLoaderGGUF.load_clip(\"t5-v1_1-xxl-encoder-Q3_K_L.gguf\", \"clip_l.safetensors\", \"flux\")[0]\n",
80 | " print(\"Loading VAE...\")\n",
81 | " vae = VAELoader.load_vae(\"ae.sft\")[0]\n",
82 | " print(f\"Loading Flux.1-dev...\")\n",
83 | " unet = UnetLoaderGGUF.load_unet(f\"flux1-dev.gguf\")[0]\n",
84 | " print(\"Loading Controlnet Union...\")\n",
85 | " controlnet = ControlNetLoader.load_controlnet(\"FLUX.1-dev-Controlnet-Union-Pro.safetensors\")[0]\n",
86 | "\n",
87 | " unet_f, clip_f = unet, clip\n",
88 | "\n",
89 | "print(\"All Models Loaded!\")\n",
90 | "\n",
91 | "import warnings\n",
92 | "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
93 | "warnings.simplefilter(action='ignore', category=UserWarning)\n",
94 | "\n",
95 | "import re\n",
96 | "import os\n",
97 | "import gc\n",
98 | "import random\n",
99 | "import numpy as np\n",
100 | "from PIL import Image\n",
101 | "from google.colab import files\n",
102 | "\n",
103 | "import nodes\n",
104 | "from totoro_extras import nodes_custom_sampler\n",
105 | "from totoro_extras import nodes_post_processing\n",
106 | "from totoro_extras import nodes_flux\n",
107 | "from totoro_extras import nodes_controlnet\n",
108 | "from totoro import model_management\n",
109 | "\n",
110 | "from controlnet_aux import CannyDetector, DepthAnythingDetector, HEDdetector, OpenposeDetector, PidiNetDetector, TEEDdetector, LineartDetector, LineartAnimeDetector, MLSDdetector, NormalBaeDetector, SamDetector\n",
111 | "\n",
112 | "CLIPTextEncodeFlux = nodes_flux.NODE_CLASS_MAPPINGS[\"CLIPTextEncodeFlux\"]()\n",
113 | "RandomNoise = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"RandomNoise\"]()\n",
114 | "BasicGuider = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"BasicGuider\"]()\n",
115 | "KSamplerSelect = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"KSamplerSelect\"]()\n",
116 | "BasicScheduler = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"BasicScheduler\"]()\n",
117 | "SamplerCustomAdvanced = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"SamplerCustomAdvanced\"]()\n",
118 | "LoraLoader = NODE_CLASS_MAPPINGS[\"LoraLoader\"]()\n",
119 | "VAEDecode = NODE_CLASS_MAPPINGS[\"VAEDecode\"]()\n",
120 | "VAEEncode = NODE_CLASS_MAPPINGS[\"VAEEncode\"]()\n",
121 | "LoadImage = NODE_CLASS_MAPPINGS[\"LoadImage\"]()\n",
122 | "EmptyLatentImage = NODE_CLASS_MAPPINGS[\"EmptyLatentImage\"]()\n",
123 | "ImageScaleToTotalPixels = nodes_post_processing.NODE_CLASS_MAPPINGS[\"ImageScaleToTotalPixels\"]()\n",
124 | "SetUnionControlNetType = nodes_controlnet.NODE_CLASS_MAPPINGS[\"SetUnionControlNetType\"]()\n",
125 | "ControlNetApplyAdvanced = NODE_CLASS_MAPPINGS[\"ControlNetApplyAdvanced\"]()"
126 | ],
127 | "metadata": {
128 | "id": "k3aTOrdb8HxC",
129 | "cellView": "form"
130 | },
131 | "execution_count": null,
132 | "outputs": []
133 | },
134 | {
135 | "cell_type": "code",
136 | "source": [
137 | "# @markdown Functions
\n",
138 | "\n",
139 | "loras = {\n",
140 | " \"xlabs_flux_anime\":\n",
141 | " {\n",
142 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/anime_lora_comfy_converted.safetensors\",\n",
143 | " \"filename\": \"xlabs_anime_lora.safetensors\",\n",
144 | " \"triggers\": \"anime\"\n",
145 | " },\n",
146 | " \"xlabs_flux_art\":\n",
147 | " {\n",
148 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/art_lora_comfy_converted.safetensors\",\n",
149 | " \"filename\": \"xlabs_art_lora.safetensors\",\n",
150 | " \"triggers\": \"art\"\n",
151 | " },\n",
152 | " \"xlabs_flux_disney\":\n",
153 | " {\n",
154 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/disney_lora_comfy_converted.safetensors\",\n",
155 | " \"filename\": \"xlabs_disney_lora.safetensors\",\n",
156 | " \"triggers\": \"disney style\"\n",
157 | " },\n",
158 | " \"xlabs_flux_mjv6\":\n",
159 | " {\n",
160 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/mjv6_lora_comfy_converted.safetensors\",\n",
161 | " \"filename\": \"xlabs_mjv6_lora.safetensors\"\n",
162 | " },\n",
163 | " \"xlabs_flux_realism\":\n",
164 | " {\n",
165 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/realism_lora_comfy_converted.safetensors\",\n",
166 | " \"filename\": \"xlabs_realism_lora.safetensors\"\n",
167 | " },\n",
168 | " \"xlabs_flux_scenery\":\n",
169 | " {\n",
170 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/scenery_lora_comfy_converted.safetensors\",\n",
171 | " \"filename\": \"xlabs_scenery_lora.safetensors\",\n",
172 | " \"triggers\": \"scenery style\"\n",
173 | " },\n",
174 | " \"xlabs_flux_furry\":\n",
175 | " {\n",
176 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/furry_lora.safetensors\",\n",
177 | " \"filename\": \"xlabs_flux_furry_lora.safetensors\"\n",
178 | " }\n",
179 | "}\n",
180 | "\n",
181 | "\n",
182 | "def load_loras(prompt):\n",
183 | " # @markdown Load Loras
- Add <lora_name:model_strength> to Prompt
\n",
184 | "\n",
185 | " global unet, clip, unet_f, clip_f\n",
186 | "\n",
187 | " unet_f, clip_f = unet, clip\n",
188 | "\n",
189 | " matches = re.findall(r\"<\\s*([^:]+?)\\s*:\\s*([0-9.]+)\\s*>\", prompt)\n",
190 | "\n",
191 | " loras_list = [(name.strip(), float(value)) for name, value in matches]\n",
192 | "\n",
193 | " if len(loras_list):\n",
194 | " print(\"Loading Loras...\")\n",
195 | "\n",
196 | " for lora_tuple in loras_list:\n",
197 | " lora = loras.get(lora_tuple[0], None)\n",
198 | "\n",
199 | " if lora:\n",
200 | " !aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M {lora[\"url\"]} -d /content/TotoroUI/models/loras -o {lora[\"filename\"]}\n",
201 | "\n",
202 | " with torch.inference_mode():\n",
203 | " unet_f, clip_f = LoraLoader.load_lora(unet_f, clip_f, lora[\"filename\"], lora_tuple[1], lora_tuple[1])\n",
204 | "\n",
205 | " print(f\"Loaded Lora: {lora_tuple[0]}\")\n",
206 | " else:\n",
207 | " print(f\"Lora not listed: {lora_tuple[0]}\")\n",
208 | "\n",
209 | "def clean_prompt(prompt):\n",
210 | " cleaned_prompt = re.sub(r\"<.*?>\", \"\", prompt)\n",
211 | "\n",
212 | " return cleaned_prompt\n",
213 | "\n",
214 | "def cuda_gc():\n",
215 | " try:\n",
216 | " model_management.soft_empty_cache()\n",
217 | " gc.collect()\n",
218 | " torch.cuda.empty_cache()\n",
219 | " torch.cuda.ipc_collect()\n",
220 | " except:\n",
221 | " pass\n",
222 | "\n",
223 | "def img_tensor_to_np(img_tensor):\n",
224 | " img_tensor = img_tensor.clone() * 255.0\n",
225 | " return img_tensor.squeeze().numpy().astype(np.uint8)\n",
226 | "\n",
227 | "def img_np_to_tensor(img_np_list):\n",
228 | " return torch.from_numpy(img_np_list.astype(np.float32) / 255.0).unsqueeze(0)\n",
229 | "\n",
230 | "def controlnet_preprocess(width, height):\n",
231 | " # @markdown \n",
232 | "\n",
233 | " process_image = True # @param {\"type\":\"boolean\"}\n",
234 | " preprocessor_type = \"openpose\" # @param [\"none\", \"openpose\", \"depth\", \"hed\", \"pidi\", \"scribble\", \"ted\", \"canny\", \"lineart\", \"anime_lineart\", \"mlsd\", \"normal\", \"segment\"]\n",
235 | " input_image = \"/content/test.png\" # @param {\"type\":\"string\"}\n",
236 | " resolution = 512 # @param {\"type\":\"slider\",\"min\":512,\"max\":2048,\"step\":1}\n",
237 | " resize_mode = \"Just Resize\" # @param [\"Just Resize\",\"Resize and Fill\",\"Crop and Resize\"]\n",
238 | "\n",
239 | " image_input = LoadImage.load_image(input_image)[0]\n",
240 | "\n",
241 | " if process_image:\n",
242 | " image_np = img_tensor_to_np(image_input)\n",
243 | " img = Image.fromarray(image_np)\n",
244 | "\n",
245 | " if resize_mode == \"Just Resize\":\n",
246 | " img = img.resize((width, height), Image.Resampling.LANCZOS)\n",
247 | "\n",
248 | " elif resize_mode == \"Resize and Fill\":\n",
249 | " img.thumbnail((width, height), Image.ANTIALIAS)\n",
250 | " result = Image.new(\"RGB\", (width, height), (0, 0, 0))\n",
251 | " offset = ((width - img.width) // 2, (height - img.height) // 2)\n",
252 | " result.paste(img, offset)\n",
253 | " img = result\n",
254 | "\n",
255 | " elif resize_mode == \"Crop and Resize\":\n",
256 | " src_width, src_height = img.size\n",
257 | " src_aspect = src_width / src_height\n",
258 | " target_aspect = width / height\n",
259 | "\n",
260 | " if src_aspect > target_aspect:\n",
261 | " new_width = int(src_height * target_aspect)\n",
262 | " left = (src_width - new_width) // 2\n",
263 | " img = img.crop((left, 0, left + new_width, src_height))\n",
264 | " else:\n",
265 | " new_height = int(src_width / target_aspect)\n",
266 | " top = (src_height - new_height) // 2\n",
267 | " img = img.crop((0, top, src_width, top + new_height))\n",
268 | "\n",
269 | " img = img.resize((width, height), Image.Resampling.LANCZOS)\n",
270 | "\n",
271 | " image_np = np.array(img).astype(np.uint8)\n",
272 | "\n",
273 | " if preprocessor_type == \"none\":\n",
274 | " processed_image = Image.fromarray(image_np)\n",
275 | "\n",
276 | " if preprocessor_type == \"canny\":\n",
277 | " canny = CannyDetector()\n",
278 | " processed_image = canny(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
279 | "\n",
280 | " elif preprocessor_type == \"depth\":\n",
281 | " depth = DepthAnythingDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"depth_anything_v2_vitl_fp16.safetensors\")\n",
282 | " processed_image = depth(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
283 | "\n",
284 | " elif preprocessor_type == \"hed\":\n",
285 | " hed = HEDdetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"ControlNetHED.pth\")\n",
286 | " processed_image = hed(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
287 | "\n",
288 | " elif preprocessor_type == \"openpose\":\n",
289 | " openpose = OpenposeDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"body_pose_model.pth\", hand_filename=\"hand_pose_model.pth\", face_filename=\"facenet.pth\")\n",
290 | " processed_image = openpose(image_np, detect_resolution=resolution, image_resolution=resolution, include_hand=True, include_face=True)\n",
291 | "\n",
292 | " elif preprocessor_type == \"pidi\":\n",
293 | " pidi = PidiNetDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"table5_pidinet.pth\")\n",
294 | " processed_image = pidi(image_np, detect_resolution=resolution, image_resolution=resolution, safe=True)\n",
295 | "\n",
296 | " elif preprocessor_type == \"scribble\":\n",
297 | " scribble = HEDdetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"ControlNetHED.pth\")\n",
298 | " processed_image = scribble(image_np, detect_resolution=resolution, image_resolution=resolution, scribble=True)\n",
299 | "\n",
300 | " elif preprocessor_type == \"ted\":\n",
301 | " ted = TEEDdetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"5_model.pth\")\n",
302 | " processed_image = ted(image_np, detect_resolution=resolution)\n",
303 | "\n",
304 | " elif preprocessor_type == \"lineart\":\n",
305 | " lineart = LineartDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"sk_model.pth\", coarse_filename=\"sk_model2.pth\")\n",
306 | " processed_image = lineart(image_np, detect_resolution=resolution, image_resolution=resolution, coarse=True)\n",
307 | "\n",
308 | " elif preprocessor_type == \"anime_lineart\":\n",
309 | " anime_lineart = LineartAnimeDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"netG.pth\")\n",
310 | " processed_image = anime_lineart(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
311 | "\n",
312 | " elif preprocessor_type == \"mlsd\":\n",
313 | " mlsd = MLSDdetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"mlsd_large_512_fp32.pth\")\n",
314 | " processed_image = mlsd(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
315 | "\n",
316 | " elif preprocessor_type == \"normal\":\n",
317 | " normal = NormalBaeDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"scannet.pt\")\n",
318 | " processed_image = normal(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
319 | "\n",
320 | " elif preprocessor_type == \"segment\":\n",
321 | " segment = SamDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", model_type=\"vit_b\", filename=\"sam_vit_b.pth\")\n",
322 | " processed_image = segment(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
323 | "\n",
324 | " processed_image_np = np.array(processed_image)\n",
325 | " processed_image_tensor = img_np_to_tensor(processed_image_np)\n",
326 | "\n",
327 | " display(Image.fromarray(processed_image_np))\n",
328 | "\n",
329 | " return processed_image_tensor\n",
330 | "\n",
331 | " else:\n",
332 | " return image_input\n",
333 | "\n",
334 | "def apply_controlnet(cond, width, height):\n",
335 | " # @markdown \n",
336 | "\n",
337 | " controlnet_type = \"openpose\" # @param [\"openpose\", \"depth\", \"hed/pidi/scribble/ted\", \"canny/lineart/anime_lineart/mlsd\", \"normal\", \"segment\", \"tile\", \"repaint\" ]\n",
338 | " strength = 0.5 # @param {\"type\":\"slider\",\"min\":0.0,\"max\":10.0,\"step\":0.01}\n",
339 | " start_percent = 0 # @param {\"type\":\"slider\",\"min\":0.0,\"max\":1.0,\"step\":0.001}\n",
340 | " end_percent = 0.225 # @param {\"type\":\"slider\",\"min\":0.0,\"max\":1.0,\"step\":0.001}\n",
341 | "\n",
342 | " cond_neg = CLIPTextEncodeFlux.encode(clip_f, \"\", \"\", 0)[0]\n",
343 | "\n",
344 | " print(\"Loading Controlnet...\")\n",
345 | "\n",
346 | " preprocessed_image = controlnet_preprocess(width, height)\n",
347 | "\n",
348 | " controlnet_f = SetUnionControlNetType.set_controlnet_type(controlnet, controlnet_type)[0]\n",
349 | "\n",
350 | " cond = ControlNetApplyAdvanced.apply_controlnet(cond, cond_neg, controlnet_f, preprocessed_image, strength, start_percent, end_percent, vae)[0]\n",
351 | "\n",
352 | " del controlnet_f\n",
353 | "\n",
354 | " return cond\n",
355 | "\n",
356 | "def closestNumber(n, m):\n",
357 | " q = int(n / m)\n",
358 | " n1 = m * q\n",
359 | " if (n * m) > 0:\n",
360 | " n2 = m * (q + 1)\n",
361 | " else:\n",
362 | " n2 = m * (q - 1)\n",
363 | " if abs(n - n1) < abs(n - n2):\n",
364 | " return n1\n",
365 | " return n2\n",
366 | "\n",
367 | "def save_image(decoded, path, name, download=False):\n",
368 | " full_path = os.path.abspath(os.path.join(path, name))\n",
369 | " Image.fromarray(np.array(decoded*255, dtype=np.uint8)[0]).save( full_path)\n",
370 | "\n",
371 | " img = Image.open(full_path)\n",
372 | " display(img)\n",
373 | "\n",
374 | " if download:\n",
375 | " files.download(full_path)\n",
376 | "\n",
377 | "@torch.inference_mode()\n",
378 | "def generate(prompt, width, height, fixed_seed, guidance, steps, sampler_name, scheduler, batch_size, auto_download, mode=\"t2i\", input_img=None, denoise=1.0):\n",
379 | " global unet, clip, unet_f, clip_f\n",
380 | "\n",
381 | " print(\"Prompt Received\")\n",
382 | "\n",
383 | " load_loras(prompt)\n",
384 | " prompt = clean_prompt(prompt)\n",
385 | "\n",
386 | " if mode == \"t2i\":\n",
387 | " latent_image = EmptyLatentImage.generate(closestNumber(width, 16), closestNumber(height, 16))[0]\n",
388 | "\n",
389 | " elif mode == \"i2i\":\n",
390 | " image = LoadImage.load_image(input_img)[0]\n",
391 | " latent_image = ImageScaleToTotalPixels.upscale(image, \"lanczos\", 1.0)[0]\n",
392 | " latent_image = VAEEncode.encode(vae, latent_image)[0]\n",
393 | "\n",
394 | " cond = CLIPTextEncodeFlux.encode(clip_f, prompt, prompt, guidance)[0]\n",
395 | "\n",
396 | " cond = apply_controlnet(cond, width, height)\n",
397 | "\n",
398 | " guider = BasicGuider.get_guider(unet_f, cond)[0]\n",
399 | " sampler = KSamplerSelect.get_sampler(sampler_name)[0]\n",
400 | " sigmas = BasicScheduler.get_sigmas(unet_f, scheduler, steps, denoise)[0]\n",
401 | "\n",
402 | " for i in range(0, batch_size):\n",
403 | " if fixed_seed == 0:\n",
404 | " seed = random.randint(0, 18446744073709551615)\n",
405 | " else:\n",
406 | " seed = fixed_seed\n",
407 | "\n",
408 | " print(\"Seed:\", seed)\n",
409 | "\n",
410 | " noise = RandomNoise.get_noise(seed)[0]\n",
411 | " sample, sample_denoised = SamplerCustomAdvanced.sample(noise, guider, sampler, sigmas, latent_image)\n",
412 | " model_management.soft_empty_cache()\n",
413 | " decoded = VAEDecode.decode(vae, sample)[0].detach()\n",
414 | "\n",
415 | " save_image(decoded, \"/content\", f\"flux_{mode}_{seed}_{i}.png\", auto_download)\n",
416 | "\n",
417 | " cuda_gc()\n",
418 | "\n",
419 | "print(f\"{'Lora Name':<40} {'Trigger Words':<40}\")\n",
420 | "print(\"-\" * 80)\n",
421 | "\n",
422 | "for key_name, details in loras.items():\n",
423 | " trigger_words = details.get(\"triggers\", \"N/A\")\n",
424 | " print(f\"{key_name:<40} {trigger_words:<40}\")"
425 | ],
426 | "metadata": {
427 | "id": "mLGPKWvopwnC",
428 | "cellView": "form"
429 | },
430 | "execution_count": null,
431 | "outputs": []
432 | },
433 | {
434 | "cell_type": "code",
435 | "execution_count": null,
436 | "metadata": {
437 | "cellView": "form",
438 | "id": "Ur9TmMNwC2kR"
439 | },
440 | "outputs": [],
441 | "source": [
442 | "#@markdown Txt2Img
\n",
443 | "\n",
444 | "positive_prompt = \"\" # @param {\"type\":\"string\"}\n",
445 | "width = 512 # @param {\"type\":\"slider\",\"min\":256,\"max\":2048,\"step\":1}\n",
446 | "height = 512 # @param {\"type\":\"slider\",\"min\":256,\"max\":2048,\"step\":1}\n",
447 | "fixed_seed = 0 # @param {\"type\":\"slider\",\"min\":0,\"max\":18446744073709552000,\"step\":1}\n",
448 | "guidance = 4.5 # @param {\"type\":\"slider\",\"min\":0,\"max\":20,\"step\":0.5}\n",
449 | "steps = 25 # @param {\"type\":\"slider\",\"min\":4,\"max\":50,\"step\":1}\n",
450 | "sampler_name = \"euler\" # @param [\"euler\",\"heun\",\"heunpp2\",\"heunpp2\",\"dpm_2\",\"lms\",\"dpmpp_2m\",\"ipndm\",\"deis\",\"ddim\",\"uni_pc\",\"uni_pc_bh2\"]\n",
451 | "scheduler = \"simple\" # @param [\"normal\",\"sgm_uniform\",\"simple\",\"ddim_uniform\"]\n",
452 | "batch_size = 1 # @param {\"type\":\"slider\",\"min\":1,\"max\":20,\"step\":1}\n",
453 | "auto_download = False # @param {\"type\":\"boolean\"}\n",
454 | "\n",
455 | "generate(positive_prompt, width, height, fixed_seed, guidance, steps, sampler_name, scheduler, batch_size, auto_download)"
456 | ]
457 | },
458 | {
459 | "cell_type": "code",
460 | "execution_count": null,
461 | "metadata": {
462 | "cellView": "form",
463 | "id": "Dpd2sfrePYoA"
464 | },
465 | "outputs": [],
466 | "source": [
467 | "#@markdown Img2Img
\n",
468 | "\n",
469 | "positive_prompt = \"anime style\" # @param {\"type\":\"string\"}\n",
470 | "fixed_seed = 0 # @param {\"type\":\"slider\",\"min\":0,\"max\":18446744073709552000,\"step\":1}\n",
471 | "guidance = 4.5 # @param {\"type\":\"slider\",\"min\":0,\"max\":20,\"step\":0.5}\n",
472 | "steps = 25 # @param {\"type\":\"slider\",\"min\":4,\"max\":50,\"step\":1}\n",
473 | "sampler_name = \"euler\" # @param [\"euler\",\"heun\",\"heunpp2\",\"heunpp2\",\"dpm_2\",\"lms\",\"dpmpp_2m\",\"ipndm\",\"deis\",\"ddim\",\"uni_pc\",\"uni_pc_bh2\"]\n",
474 | "scheduler = \"simple\" # @param [\"normal\",\"sgm_uniform\",\"simple\",\"ddim_uniform\"]\n",
475 | "input_img = \"/content/test.png\" # @param {\"type\":\"string\"}\n",
476 | "denoise = 0.85 # @param {\"type\":\"slider\",\"min\":0,\"max\":1,\"step\":0.01}\n",
477 | "batch_size = 1 # @param {\"type\":\"slider\",\"min\":1,\"max\":20,\"step\":1}\n",
478 | "auto_download = False # @param {\"type\":\"boolean\"}\n",
479 | "\n",
480 | "\n",
481 | "generate(positive_prompt, 0, 0, fixed_seed, guidance, steps, sampler_name, scheduler, batch_size, auto_download, \"i2i\", input_img, denoise)"
482 | ]
483 | }
484 | ],
485 | "metadata": {
486 | "accelerator": "GPU",
487 | "colab": {
488 | "gpuType": "T4",
489 | "provenance": []
490 | },
491 | "kernelspec": {
492 | "display_name": "Python 3",
493 | "name": "python3"
494 | },
495 | "language_info": {
496 | "name": "python"
497 | }
498 | },
499 | "nbformat": 4,
500 | "nbformat_minor": 0
501 | }
--------------------------------------------------------------------------------
/Flux.1-dev-XLabs_CN-jupyter.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {
7 | "id": "VjYy0F2gZIPR",
8 | "cellView": "form"
9 | },
10 | "outputs": [],
11 | "source": [
12 | "#@markdown Install
\n",
13 | "\n",
14 | "%cd /content\n",
15 | "!git clone -b totoro6 https://github.com/LucipherDev/ComfyUI /content/TotoroUI\n",
16 | "!git clone -b totoro https://github.com/LucipherDev/ComfyUI-GGUF /content/TotoroUI/custom_nodes/TotoroUI-GGUF\n",
17 | "%cd /content/TotoroUI\n",
18 | "\n",
19 | "!pip install -q torchsde einops diffusers accelerate xformers==0.0.28.post2\n",
20 | "!pip install -q git+https://github.com/LucipherDev/controlnet_aux.git\n",
21 | "!pip install -q -r /content/TotoroUI/custom_nodes/TotoroUI-GGUF/requirements.txt\n",
22 | "!apt -y install -qq aria2\n",
23 | "\n",
24 | "import nodes\n",
25 | "\n",
26 | "if not nodes.load_custom_node(\"custom_nodes/TotoroUI-GGUF\"):\n",
27 | " raise Exception(\"Failed to load GGUF custom node\")"
28 | ]
29 | },
30 | {
31 | "cell_type": "code",
32 | "execution_count": null,
33 | "metadata": {
34 | "id": "k3aTOrdb8HxC",
35 | "cellView": "form"
36 | },
37 | "outputs": [],
38 | "source": [
39 | "#@markdown Load Models
\n",
40 | "\n",
41 | "import torch\n",
42 | "from nodes import NODE_CLASS_MAPPINGS\n",
43 | "\n",
44 | "DualCLIPLoaderGGUF = NODE_CLASS_MAPPINGS[\"DualCLIPLoaderGGUF\"]()\n",
45 | "UnetLoaderGGUF = NODE_CLASS_MAPPINGS[\"UnetLoaderGGUF\"]()\n",
46 | "VAELoader = NODE_CLASS_MAPPINGS[\"VAELoader\"]()\n",
47 | "ControlNetLoader = NODE_CLASS_MAPPINGS[\"ControlNetLoader\"]()\n",
48 | "\n",
49 | "print(f\"Downloading Flux1-dev-Q4_K_S...\")\n",
50 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/city96/FLUX.1-dev-gguf/resolve/main/flux1-dev-Q4_K_S.gguf -d /content/TotoroUI/models/unet -o flux1-dev-Q4_K_S.gguf\n",
51 | "\n",
52 | "print(\"Downloading VAE...\")\n",
53 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/FLUX.1-dev/resolve/main/ae.sft -d /content/TotoroUI/models/vae -o ae.sft\n",
54 | "\n",
55 | "print(\"Downloading Clips...\")\n",
56 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/FLUX.1-dev/resolve/main/clip_l.safetensors -d /content/TotoroUI/models/clip -o clip_l.safetensors\n",
57 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/resolve/main/t5-v1_1-xxl-encoder-Q6_K.gguf -d /content/TotoroUI/models/clip -o t5-v1_1-xxl-encoder-Q6_K.gguf\n",
58 | "\n",
59 | "print(\"Downloading Controlnet Preprocessors...\")\n",
60 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/Kijai/DepthAnythingV2-safetensors/resolve/main/depth_anything_v2_vitl_fp16.safetensors -d /content/TotoroUI/models/controlnet_preprocessors -o depth_anything_v2_vitl_fp16.safetensors\n",
61 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/Annotators/resolve/main/ControlNetHED.pth -d /content/TotoroUI/models/controlnet_preprocessors -o ControlNetHED.pth\n",
62 | "\n",
63 | "print(\"Downloading X-Labs Controlnets...\")\n",
64 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-canny-controlnet-v3.safetensors -d /content/TotoroUI/models/controlnet -o flux-canny-controlnet-v3.safetensors\n",
65 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-depth-controlnet-v3.safetensors -d /content/TotoroUI/models/controlnet -o flux-depth-controlnet-v3.safetensors\n",
66 | "!aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-hed-controlnet-v3.safetensors -d /content/TotoroUI/models/controlnet -o flux-hed-controlnet-v3.safetensors\n",
67 | "\n",
68 | "with torch.inference_mode():\n",
69 | " print(\"Loading VAE...\")\n",
70 | " vae = VAELoader.load_vae(\"ae.sft\")[0]\n",
71 | " print(f\"Loading Flux1-dev-Q4_K_S...\")\n",
72 | " unet = UnetLoaderGGUF.load_unet(f\"flux1-dev-Q4_K_S.gguf\")[0]\n",
73 | " print(\"Loading Clips...\")\n",
74 | " clip = DualCLIPLoaderGGUF.load_clip(\"t5-v1_1-xxl-encoder-Q6_K.gguf\", \"clip_l.safetensors\", \"flux\")[0]\n",
75 | "\n",
76 | " unet_f, clip_f = unet, clip\n",
77 | "\n",
78 | "print(\"All Models Loaded!\")\n",
79 | "\n",
80 | "import warnings\n",
81 | "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
82 | "warnings.simplefilter(action='ignore', category=UserWarning)\n",
83 | "\n",
84 | "import re\n",
85 | "import os\n",
86 | "import random\n",
87 | "import numpy as np\n",
88 | "import gc\n",
89 | "from PIL import Image\n",
90 | "from google.colab import files\n",
91 | "\n",
92 | "import nodes\n",
93 | "from totoro_extras import nodes_custom_sampler\n",
94 | "from totoro_extras import nodes_post_processing\n",
95 | "from totoro_extras import nodes_flux\n",
96 | "from totoro_extras import nodes_controlnet\n",
97 | "from totoro import model_management\n",
98 | "\n",
99 | "from controlnet_aux import CannyDetector, DepthAnythingDetector, HEDdetector\n",
100 | "\n",
101 | "CLIPTextEncodeFlux = nodes_flux.NODE_CLASS_MAPPINGS[\"CLIPTextEncodeFlux\"]()\n",
102 | "RandomNoise = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"RandomNoise\"]()\n",
103 | "BasicGuider = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"BasicGuider\"]()\n",
104 | "KSamplerSelect = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"KSamplerSelect\"]()\n",
105 | "BasicScheduler = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"BasicScheduler\"]()\n",
106 | "SamplerCustomAdvanced = nodes_custom_sampler.NODE_CLASS_MAPPINGS[\"SamplerCustomAdvanced\"]()\n",
107 | "LoraLoader = NODE_CLASS_MAPPINGS[\"LoraLoader\"]()\n",
108 | "VAEDecode = NODE_CLASS_MAPPINGS[\"VAEDecode\"]()\n",
109 | "VAEEncode = NODE_CLASS_MAPPINGS[\"VAEEncode\"]()\n",
110 | "LoadImage = NODE_CLASS_MAPPINGS[\"LoadImage\"]()\n",
111 | "EmptyLatentImage = NODE_CLASS_MAPPINGS[\"EmptyLatentImage\"]()\n",
112 | "ImageScaleToTotalPixels = nodes_post_processing.NODE_CLASS_MAPPINGS[\"ImageScaleToTotalPixels\"]()\n",
113 | "ControlNetApplyAdvanced = NODE_CLASS_MAPPINGS[\"ControlNetApplyAdvanced\"]()"
114 | ]
115 | },
116 | {
117 | "cell_type": "code",
118 | "execution_count": null,
119 | "metadata": {
120 | "id": "mLGPKWvopwnC",
121 | "cellView": "form"
122 | },
123 | "outputs": [],
124 | "source": [
125 | "# @markdown Functions
\n",
126 | "\n",
127 | "loras = {\n",
128 | " \"xlabs_flux_anime\":\n",
129 | " {\n",
130 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/anime_lora_comfy_converted.safetensors\",\n",
131 | " \"filename\": \"xlabs_anime_lora.safetensors\",\n",
132 | " \"triggers\": \"anime\"\n",
133 | " },\n",
134 | " \"xlabs_flux_art\":\n",
135 | " {\n",
136 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/art_lora_comfy_converted.safetensors\",\n",
137 | " \"filename\": \"xlabs_art_lora.safetensors\",\n",
138 | " \"triggers\": \"art\"\n",
139 | " },\n",
140 | " \"xlabs_flux_disney\":\n",
141 | " {\n",
142 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/disney_lora_comfy_converted.safetensors\",\n",
143 | " \"filename\": \"xlabs_disney_lora.safetensors\",\n",
144 | " \"triggers\": \"disney style\"\n",
145 | " },\n",
146 | " \"xlabs_flux_mjv6\":\n",
147 | " {\n",
148 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/mjv6_lora_comfy_converted.safetensors\",\n",
149 | " \"filename\": \"xlabs_mjv6_lora.safetensors\"\n",
150 | " },\n",
151 | " \"xlabs_flux_realism\":\n",
152 | " {\n",
153 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/realism_lora_comfy_converted.safetensors\",\n",
154 | " \"filename\": \"xlabs_realism_lora.safetensors\"\n",
155 | " },\n",
156 | " \"xlabs_flux_scenery\":\n",
157 | " {\n",
158 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/scenery_lora_comfy_converted.safetensors\",\n",
159 | " \"filename\": \"xlabs_scenery_lora.safetensors\",\n",
160 | " \"triggers\": \"scenery style\"\n",
161 | " },\n",
162 | " \"xlabs_flux_furry\":\n",
163 | " {\n",
164 | " \"url\": \"https://huggingface.co/XLabs-AI/flux-lora-collection/resolve/main/furry_lora.safetensors\",\n",
165 | " \"filename\": \"xlabs_flux_furry_lora.safetensors\"\n",
166 | " }\n",
167 | "}\n",
168 | "\n",
169 | "\n",
170 | "def load_loras(prompt):\n",
171 | " # @markdown Load Loras
- Add <lora_name:model_strength> to Prompt
\n",
172 | "\n",
173 | " global unet, clip, unet_f, clip_f\n",
174 | "\n",
175 | " unet_f, clip_f = unet, clip\n",
176 | "\n",
177 | " matches = re.findall(r\"<\\s*([^:]+?)\\s*:\\s*([0-9.]+)\\s*>\", prompt)\n",
178 | "\n",
179 | " loras_list = [(name.strip(), float(value)) for name, value in matches]\n",
180 | "\n",
181 | " if len(loras_list):\n",
182 | " print(\"Loading Loras...\")\n",
183 | "\n",
184 | " for lora_tuple in loras_list:\n",
185 | " lora = loras.get(lora_tuple[0], None)\n",
186 | "\n",
187 | " if lora:\n",
188 | " !aria2c --quiet --console-log-level=error --auto-file-renaming=false --allow-overwrite=false -c -x 16 -s 16 -k 1M {lora[\"url\"]} -d /content/TotoroUI/models/loras -o {lora[\"filename\"]}\n",
189 | "\n",
190 | " with torch.inference_mode():\n",
191 | " unet_f, clip_f = LoraLoader.load_lora(unet_f, clip_f, lora[\"filename\"], lora_tuple[1], lora_tuple[1])\n",
192 | "\n",
193 | " print(f\"Loaded Lora: {lora_tuple[0]}\")\n",
194 | " else:\n",
195 | " print(f\"Lora not listed: {lora_tuple[0]}\")\n",
196 | "\n",
197 | "def clean_prompt(prompt):\n",
198 | " cleaned_prompt = re.sub(r\"<.*?>\", \"\", prompt)\n",
199 | "\n",
200 | " return cleaned_prompt\n",
201 | "\n",
202 | "def cuda_gc():\n",
203 | " try:\n",
204 | " model_management.soft_empty_cache()\n",
205 | " gc.collect()\n",
206 | " torch.cuda.empty_cache()\n",
207 | " torch.cuda.ipc_collect()\n",
208 | " except:\n",
209 | " pass\n",
210 | "\n",
211 | "def img_tensor_to_np(img_tensor):\n",
212 | " img_tensor = img_tensor.clone() * 255.0\n",
213 | " return img_tensor.squeeze().numpy().astype(np.uint8)\n",
214 | "\n",
215 | "def img_np_to_tensor(img_np_list):\n",
216 | " return torch.from_numpy(img_np_list.astype(np.float32) / 255.0).unsqueeze(0)\n",
217 | "\n",
218 | "def controlnet_preprocess(preprocessor_type, width, height):\n",
219 | " # @markdown \n",
220 | "\n",
221 | " process_image = True # @param {\"type\":\"boolean\"}\n",
222 | " input_image = \"/content/test.png\" # @param {\"type\":\"string\"}\n",
223 | " resolution = 512 # @param {\"type\":\"slider\",\"min\":512,\"max\":2048,\"step\":1}\n",
224 | " resize_mode = \"Just Resize\" # @param [\"Just Resize\",\"Resize and Fill\",\"Crop and Resize\"]\n",
225 | "\n",
226 | " image_input = LoadImage.load_image(input_image)[0]\n",
227 | "\n",
228 | " if process_image:\n",
229 | " image_np = img_tensor_to_np(image_input)\n",
230 | " img = Image.fromarray(image_np)\n",
231 | "\n",
232 | " if resize_mode == \"Just Resize\":\n",
233 | " img = img.resize((width, height), Image.Resampling.LANCZOS)\n",
234 | "\n",
235 | " elif resize_mode == \"Resize and Fill\":\n",
236 | " img.thumbnail((width, height), Image.ANTIALIAS)\n",
237 | " result = Image.new(\"RGB\", (width, height), (0, 0, 0))\n",
238 | " offset = ((width - img.width) // 2, (height - img.height) // 2)\n",
239 | " result.paste(img, offset)\n",
240 | " img = result\n",
241 | "\n",
242 | " elif resize_mode == \"Crop and Resize\":\n",
243 | " src_width, src_height = img.size\n",
244 | " src_aspect = src_width / src_height\n",
245 | " target_aspect = width / height\n",
246 | "\n",
247 | " if src_aspect > target_aspect:\n",
248 | " new_width = int(src_height * target_aspect)\n",
249 | " left = (src_width - new_width) // 2\n",
250 | " img = img.crop((left, 0, left + new_width, src_height))\n",
251 | " else:\n",
252 | " new_height = int(src_width / target_aspect)\n",
253 | " top = (src_height - new_height) // 2\n",
254 | " img = img.crop((0, top, src_width, top + new_height))\n",
255 | "\n",
256 | " img = img.resize((width, height), Image.Resampling.LANCZOS)\n",
257 | "\n",
258 | " image_np = np.array(img).astype(np.uint8)\n",
259 | "\n",
260 | " if preprocessor_type == \"canny\":\n",
261 | " canny = CannyDetector()\n",
262 | " processed_image = canny(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
263 | "\n",
264 | " elif preprocessor_type == \"depth\":\n",
265 | " depth = DepthAnythingDetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"depth_anything_v2_vitl_fp16.safetensors\")\n",
266 | " processed_image = depth(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
267 | "\n",
268 | " elif preprocessor_type == \"hed\":\n",
269 | " hed = HEDdetector.from_pretrained(\"/content/TotoroUI/models/controlnet_preprocessors\", filename=\"ControlNetHED.pth\")\n",
270 | " processed_image = hed(image_np, detect_resolution=resolution, image_resolution=resolution)\n",
271 | "\n",
272 | " processed_image_np = np.array(processed_image)\n",
273 | " processed_image_tensor = img_np_to_tensor(processed_image_np)\n",
274 | "\n",
275 | " display(Image.fromarray(processed_image_np))\n",
276 | "\n",
277 | " return processed_image_tensor\n",
278 | "\n",
279 | " else:\n",
280 | " return image_input\n",
281 | "\n",
282 | "\n",
283 | "def apply_controlnet(cond, width, height):\n",
284 | " # @markdown \n",
285 | "\n",
286 | " controlnet_type = \"canny\" # @param [\"canny\", \"depth\", \"hed\"]\n",
287 | " strength = 0.5 # @param {\"type\":\"slider\",\"min\":0.0,\"max\":10.0,\"step\":0.01}\n",
288 | " start_percent = 0 # @param {\"type\":\"slider\",\"min\":0.0,\"max\":1.0,\"step\":0.001}\n",
289 | " end_percent = 0.225 # @param {\"type\":\"slider\",\"min\":0.0,\"max\":1.0,\"step\":0.001}\n",
290 | "\n",
291 | " cond_neg = CLIPTextEncodeFlux.encode(clip_f, \"\", \"\", 0)[0]\n",
292 | "\n",
293 | " print(\"Loading Controlnet...\")\n",
294 | "\n",
295 | " preprocessed_image = controlnet_preprocess(controlnet_type, width, height)\n",
296 | "\n",
297 | " if controlnet_type == \"canny\":\n",
298 | " controlnet = ControlNetLoader.load_controlnet(\"flux-canny-controlnet-v3.safetensors\")[0]\n",
299 | " elif controlnet_type == \"depth\":\n",
300 | " controlnet = ControlNetLoader.load_controlnet(\"flux-depth-controlnet-v3.safetensors\")[0]\n",
301 | " elif controlnet_type == \"hed\":\n",
302 | " controlnet = ControlNetLoader.load_controlnet(\"flux-hed-controlnet-v3.safetensors\")[0]\n",
303 | "\n",
304 | " cond = ControlNetApplyAdvanced.apply_controlnet(cond, cond_neg, controlnet, preprocessed_image, strength, start_percent, end_percent)[0]\n",
305 | "\n",
306 | " del controlnet\n",
307 | "\n",
308 | " return cond\n",
309 | "\n",
310 | "def closestNumber(n, m):\n",
311 | " q = int(n / m)\n",
312 | " n1 = m * q\n",
313 | " if (n * m) > 0:\n",
314 | " n2 = m * (q + 1)\n",
315 | " else:\n",
316 | " n2 = m * (q - 1)\n",
317 | " if abs(n - n1) < abs(n - n2):\n",
318 | " return n1\n",
319 | " return n2\n",
320 | "\n",
321 | "def save_image(decoded, path, name, download=False):\n",
322 | " full_path = os.path.abspath(os.path.join(path, name))\n",
323 | " Image.fromarray(np.array(decoded*255, dtype=np.uint8)[0]).save(full_path)\n",
324 | "\n",
325 | " img = Image.open(full_path)\n",
326 | " display(img)\n",
327 | "\n",
328 | " if download:\n",
329 | " files.download(full_path)\n",
330 | "\n",
331 | "@torch.inference_mode()\n",
332 | "def generate(prompt, width, height, fixed_seed, guidance, steps, sampler_name, scheduler, batch_size, auto_download, mode=\"t2i\", input_img=None, denoise=1.0):\n",
333 | " global unet, clip, unet_f, clip_f\n",
334 | "\n",
335 | " print(\"Prompt Received\")\n",
336 | "\n",
337 | " load_loras(prompt)\n",
338 | " prompt = clean_prompt(prompt)\n",
339 | "\n",
340 | " if mode == \"t2i\":\n",
341 | " latent_image = EmptyLatentImage.generate(closestNumber(width, 16), closestNumber(height, 16))[0]\n",
342 | "\n",
343 | " elif mode == \"i2i\":\n",
344 | " image = LoadImage.load_image(input_img)[0]\n",
345 | " latent_image = ImageScaleToTotalPixels.upscale(image, \"lanczos\", 1.0)[0]\n",
346 | " latent_image = VAEEncode.encode(vae, latent_image)[0]\n",
347 | "\n",
348 | " image_np = img_tensor_to_np(image)\n",
349 | " img = Image.fromarray(image_np)\n",
350 | " width, height = img.width, img.height\n",
351 | "\n",
352 | " cond = CLIPTextEncodeFlux.encode(clip_f, prompt, prompt, guidance)[0]\n",
353 | "\n",
354 | " cond = apply_controlnet(cond, width, height)\n",
355 | "\n",
356 | " guider = BasicGuider.get_guider(unet_f, cond)[0]\n",
357 | " sampler = KSamplerSelect.get_sampler(sampler_name)[0]\n",
358 | " sigmas = BasicScheduler.get_sigmas(unet_f, scheduler, steps, denoise)[0]\n",
359 | "\n",
360 | " for i in range(0, batch_size):\n",
361 | " if fixed_seed == 0:\n",
362 | " seed = random.randint(0, 18446744073709551615)\n",
363 | " else:\n",
364 | " seed = fixed_seed\n",
365 | "\n",
366 | " print(\"Seed:\", seed)\n",
367 | "\n",
368 | " noise = RandomNoise.get_noise(seed)[0]\n",
369 | " sample, sample_denoised = SamplerCustomAdvanced.sample(noise, guider, sampler, sigmas, latent_image)\n",
370 | " model_management.soft_empty_cache()\n",
371 | " decoded = VAEDecode.decode(vae, sample)[0].detach()\n",
372 | "\n",
373 | " save_image(decoded, \"/content\", f\"flux_{mode}_{seed}_{i}.png\", auto_download)\n",
374 | "\n",
375 | " cuda_gc()\n",
376 | "\n",
377 | "print(f\"{'Lora Name':<40} {'Trigger Words':<40}\")\n",
378 | "print(\"-\" * 80)\n",
379 | "\n",
380 | "for key_name, details in loras.items():\n",
381 | " trigger_words = details.get(\"triggers\", \"N/A\")\n",
382 | " print(f\"{key_name:<40} {trigger_words:<40}\")"
383 | ]
384 | },
385 | {
386 | "cell_type": "code",
387 | "execution_count": null,
388 | "metadata": {
389 | "cellView": "form",
390 | "id": "Ur9TmMNwC2kR"
391 | },
392 | "outputs": [],
393 | "source": [
394 | "#@markdown Txt2Img
\n",
395 | "\n",
396 | "positive_prompt = \"\" # @param {\"type\":\"string\"}\n",
397 | "width = 1024 # @param {\"type\":\"slider\",\"min\":256,\"max\":2048,\"step\":1}\n",
398 | "height = 1024 # @param {\"type\":\"slider\",\"min\":256,\"max\":2048,\"step\":1}\n",
399 | "fixed_seed = 0 # @param {\"type\":\"slider\",\"min\":0,\"max\":18446744073709552000,\"step\":1}\n",
400 | "guidance = 4.5 # @param {\"type\":\"slider\",\"min\":0,\"max\":20,\"step\":0.5}\n",
401 | "steps = 25 # @param {\"type\":\"slider\",\"min\":4,\"max\":50,\"step\":1}\n",
402 | "sampler_name = \"euler\" # @param [\"euler\",\"heun\",\"heunpp2\",\"heunpp2\",\"dpm_2\",\"lms\",\"dpmpp_2m\",\"ipndm\",\"deis\",\"ddim\",\"uni_pc\",\"uni_pc_bh2\"]\n",
403 | "scheduler = \"simple\" # @param [\"normal\",\"sgm_uniform\",\"simple\",\"ddim_uniform\"]\n",
404 | "batch_size = 1 # @param {\"type\":\"slider\",\"min\":1,\"max\":20,\"step\":1}\n",
405 | "auto_download = False # @param {\"type\":\"boolean\"}\n",
406 | "\n",
407 | "generate(positive_prompt, width, height, fixed_seed, guidance, steps, sampler_name, scheduler, batch_size, auto_download)"
408 | ]
409 | },
410 | {
411 | "cell_type": "code",
412 | "execution_count": null,
413 | "metadata": {
414 | "cellView": "form",
415 | "id": "Dpd2sfrePYoA"
416 | },
417 | "outputs": [],
418 | "source": [
419 | "#@markdown Img2Img
\n",
420 | "\n",
421 | "positive_prompt = \"anime style\" # @param {\"type\":\"string\"}\n",
422 | "fixed_seed = 0 # @param {\"type\":\"slider\",\"min\":0,\"max\":18446744073709552000,\"step\":1}\n",
423 | "guidance = 4.5 # @param {\"type\":\"slider\",\"min\":0,\"max\":20,\"step\":0.5}\n",
424 | "steps = 25 # @param {\"type\":\"slider\",\"min\":4,\"max\":50,\"step\":1}\n",
425 | "sampler_name = \"euler\" # @param [\"euler\",\"heun\",\"heunpp2\",\"heunpp2\",\"dpm_2\",\"lms\",\"dpmpp_2m\",\"ipndm\",\"deis\",\"ddim\",\"uni_pc\",\"uni_pc_bh2\"]\n",
426 | "scheduler = \"simple\" # @param [\"normal\",\"sgm_uniform\",\"simple\",\"ddim_uniform\"]\n",
427 | "input_img = \"/content/test.png\" # @param {\"type\":\"string\"}\n",
428 | "denoise = 0.85 # @param {\"type\":\"slider\",\"min\":0,\"max\":1,\"step\":0.01}\n",
429 | "batch_size = 1 # @param {\"type\":\"slider\",\"min\":1,\"max\":20,\"step\":1}\n",
430 | "auto_download = False # @param {\"type\":\"boolean\"}\n",
431 | "\n",
432 | "\n",
433 | "generate(positive_prompt, 0, 0, fixed_seed, guidance, steps, sampler_name, scheduler, batch_size, auto_download, \"i2i\", input_img, denoise)"
434 | ]
435 | }
436 | ],
437 | "metadata": {
438 | "accelerator": "GPU",
439 | "colab": {
440 | "gpuType": "T4",
441 | "provenance": []
442 | },
443 | "kernelspec": {
444 | "display_name": "Python 3",
445 | "name": "python3"
446 | },
447 | "language_info": {
448 | "name": "python"
449 | }
450 | },
451 | "nbformat": 4,
452 | "nbformat_minor": 0
453 | }
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Flux.1-dev-ControlNet-jupyter
2 |
3 | - Flux ControlNets with Txt2Img | Img2Img | + Multiple LoRAs, All in one notebook
4 | - Works in Google Colab Free Teir T4 GPU 🥳🥳
5 | - Uses [Flux.1-dev GGUF](https://huggingface.co/city96/FLUX.1-dev-gguf) models
6 |
7 | ## Notebooks
8 |
9 | | Notebook | Info |
10 | | ------------- |:-------------:|
11 | | [](https://colab.research.google.com/github/LucipherDev/Flux.1-dev-ControlNet-jupyter/blob/main/Flux.1-dev-XLabs_CN-jupyter.ipynb) | Flux.1-dev-XLabs_CN-jupyter ([flux1-dev-Q4_K_S.gguf](https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q4_K_S.gguf)) |
12 | | [](https://colab.research.google.com/github/LucipherDev/Flux.1-dev-ControlNet-jupyter/blob/main/Flux.1-dev-Union_CN-jupyter.ipynb) | Flux.1-dev-Union_CN-jupyter (flux1-dev-[Q2_K.gguf](https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q2_K.gguf)/[Q8_0.gguf](https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q8_0.gguf)) |
13 |
14 | ###### * *This is still unstable in google colab free tier and prone to crashing because of low system RAM. Use at your own risk.*
15 |
16 | ## Features
17 |
18 | *Basic Functions [Txt2Img | Img2Img | + Multiple LoRAs] are as shown in [LucipherDev/Flux.1-fp8-jupyter](https://github.com/LucipherDev/Flux.1-fp8-jupyter)*
19 |
20 | ## X-Labs ControlNet
21 |
22 | ### Preprocessors
23 | - Canny
24 | - Depth (DepthAnythingV2)
25 | - HED
26 |
27 | 
28 |
29 | 
30 |
31 | 
32 |
33 | 
34 |
35 | ## Union-Pro ControlNet
36 |
37 | *Low RAM uses flux1-dev-Q2_K.gguf and gives low quality results, so using high RAM is advised as it uses flux1-dev-Q8_0.gguf*
38 |
39 | ### Preprocessors
40 | - Openpose
41 | - Depth (DepthAnythingV2)
42 | - HED
43 | - Pidi
44 | - Scribble
45 | - Ted
46 | - Canny
47 | - Lineart
48 | - Anime_lineart
49 | - MLSD
50 | - Normal
51 | - Segment
52 |
53 | 
54 |
55 | 
56 |
57 | 
58 |
--------------------------------------------------------------------------------