├── README.md ├── TouchDiffusion.tox ├── TouchDiffusionExt.py └── webui.bat /README.md: -------------------------------------------------------------------------------- 1 | # TouchDiffusion 2 | Discord Shield 3 | 4 | TouchDesigner implementation for real-time Stable Diffusion interactive generation with [StreamDiffusion](https://github.com/cumulo-autumn/StreamDiffusion). 5 | 6 | **Benchmarks with stabilityai/sd-turbo, 512x512 and 1 batch size.** 7 | 8 | | GPU | FPS | 9 | | --- | --- | 10 | | 4090 | 55-60 FPS | 11 | | 4080 | 47 FPS | 12 | | 3090ti | 37 FPS | 13 | | 3090 | 30-32 FPS | 14 | | 4070 Laptop | 24 FPS | 15 | | 3060 12GB | 16 FPS | 16 | 17 | ## Disclaimer 18 | **Notice:** This repository is in an early testing phase and may undergo significant changes. Use it at your own risk. 19 | 20 | ## Usage 21 | > [!TIP] 22 | > TouchDiffusion can be installed in multiple ways. **Portable version** have prebuild dendencies, so it prefered way to install or **Manuall install** is step by step instruction. 23 | 24 | #### Portable version: 25 | Includes preinstalled configurations, ensuring everything is readily available for immediate use. 26 | 1. Download and extract [archive](https://boosty.to/vjschool/posts/39931cd6-b9c5-4c27-93ff-d7a09b0918c5?share=post_link) 27 | 2. Run ```webui.bat```. It will provide url to web interface (ex. ```http://127.0.0.1:7860```) 28 | 3. Open ```install & update``` tab and run ```Update dependencies```. 29 | 30 | #### Manuall install: 31 | You can follow [YouTube tutorial](https://youtu.be/3WqUrWfCX1A) 32 | 33 | Required TouchDesigner 2023 & Python 3.11 34 | 1. Install [Python 3.11](https://www.python.org/downloads/release/python-3118/) 35 | 2. Install [Git](https://git-scm.com/downloads) 36 | 3. Install [CUDA Toolkit](https://developer.nvidia.com/cuda-11-8-0-download-archive) 11.8 (required PC restart) 37 | 4. Download [TouchDiffusion](https://github.com/olegchomp/TouchDiffusion/archive/refs/heads/main.zip). 38 | 5. Open ```webui.bat``` with text editor and set path to Python 3.11 in ```set PYTHON_PATH=```. (ex. ```set PYTHON_PATH="C:\Program Files\Python311\python.exe"```) 39 | 6. Run ```webui.bat```. After installation it will provide url to web interface (ex. ```http://127.0.0.1:7860```) 40 | 7. Open ```install & update``` tab and run ```Update dependencies```. (could take ~10 minutes, depending on your internet connection) 41 | 8. If you get pop up window with error related to .dll, run ```Fix pop up``` 42 | 9. Restart webui.bat 43 | 44 | #### Accelerate model: 45 | Models in ```.safetensors``` format must be in ```models\checkpoints``` folder. (as for sd_turbo, it will be auto-downloaded). 46 | 47 | **Internet connection required, while making engines.** 48 | 49 | 1) Run ```webui.bat``` 50 | 2) Select model type. 51 | 3) Select model. 52 | 4) Set width, height and amount of sampling steps (Batch size) 53 | 5) Select acceleration lora if available. 54 | 6) Run ```Make engine``` and wait for acceleration to finish. (could take ~10 minutes, depending on your hardware) 55 | 56 | #### TouchDesigner inference: 57 | 1. Add **TouchDiffusion.tox** to project 58 | 2. On ```Settings``` page change path to ```TouchDiffusion``` folder (same as where webui.bat). 59 | 3. Save and restart TouchDesigner project. 60 | 4. On ```Settings``` page select Engine and click **Load Engine**. 61 | 5. Connect animated TOP to input. Component cook only if input updates. 62 | 63 | #### Known issues / Roadmap: 64 | - [x] Fix Re-init. Sometimes required to restart TouchDesigner for initializing site-packages. 65 | - [ ] Code clean-up and rework. 66 | - [x] Custom resolution (for now fixed 512x512) 67 | - [ ] CFG not affecting image 68 | - [ ] Add Lora 69 | - [x] Add Hyper Lora support 70 | - [ ] Add ControlNet support 71 | - [ ] Add SDXL support 72 | 73 | ## Acknowledgement 74 | Based on the following projects: 75 | * [StreamDiffusion](https://github.com/cumulo-autumn/StreamDiffusion) - Pipeline-Level Solution for Real-Time Interactive Generation 76 | * [TopArray](https://github.com/IntentDev/TopArray) - Interaction between Python/PyTorch tensor operations and TouchDesigner TOPs. 77 | -------------------------------------------------------------------------------- /TouchDiffusion.tox: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/olegchomp/TouchDiffusion/33dd7e822d3d3b96b2e85883ab8a132ae9aea9c2/TouchDiffusion.tox -------------------------------------------------------------------------------- /TouchDiffusionExt.py: -------------------------------------------------------------------------------- 1 | from TDStoreTools import StorageManager 2 | import TDFunctions as TDF 3 | import numpy as np 4 | import torch 5 | import os 6 | import webbrowser 7 | import json 8 | from datetime import datetime 9 | import webbrowser 10 | 11 | try: 12 | from StreamDiffusion.utils.wrapper import StreamDiffusionWrapper 13 | except Exception as e: 14 | current_time = datetime.now() 15 | formated_time = current_time.strftime("%H:%M:%S") 16 | op('fifo1').appendRow([formated_time, 'Error', e]) 17 | 18 | 19 | class TouchDiffusionExt: 20 | """ 21 | DefaultExt description 22 | """ 23 | def __init__(self, ownerComp): 24 | self.ownerComp = ownerComp 25 | 26 | self.source = op('null1') 27 | self.device = "cuda" 28 | self.to_tensor = TopArrayInterface(self.source) 29 | self.stream_toparray = torch.cuda.current_stream(device=self.device) 30 | self.rgba_tensor = torch.zeros((512, 512, 4), dtype=torch.float32).to(self.device) #512,768 31 | self.rgba_tensor[..., 3] = 0 32 | self.output_interface = TopCUDAInterface(512,512,4,np.float32) #768,512 33 | self.stream = None 34 | 35 | def activate_stream(self): 36 | self.update_size() 37 | 38 | acceleration_lora = op('parameter1')['Accelerationlora',1].val 39 | if acceleration_lora == 'LCM': 40 | use_lcm_lora = True 41 | elif acceleration_lora == 'HyperSD': 42 | use_hyper_lora = True 43 | else: 44 | use_lcm_lora = False 45 | use_hyper_lora = False 46 | 47 | try: 48 | self.stream = StreamDiffusionWrapper( 49 | model_id_or_path=f"{op('parameter1')['Checkpoint',1].val}", 50 | lora_dict=op('parameter1')['Loralist',1].val, 51 | t_index_list=self.generate_t_index_list(), 52 | frame_buffer_size=1, 53 | width= int(op('parameter1')['Sizex',1]), 54 | height=int(op('parameter1')['Sizey',1]), 55 | warmup=0, 56 | acceleration="tensorrt", 57 | mode= op('parameter1')['Checkpointmode',1].val, 58 | use_denoising_batch=True, 59 | cfg_type="self", 60 | seed=int(op('parameter1')['Seed',1]), 61 | use_lcm_lora=use_lcm_lora, 62 | use_hyper_lora=use_hyper_lora, 63 | output_type='pt', 64 | model_type=op('parameter1')['Checkpointtype',1].val, 65 | touchdiffusion=True, 66 | #turbo=False 67 | ) 68 | 69 | self.stream.prepare( 70 | prompt = parent().par.Prompt.val, 71 | negative_prompt = parent().par.Negprompt.val, 72 | guidance_scale=parent().par.Cfgscale.val, 73 | delta=parent().par.Deltamult.val, 74 | t_index_list=self.update_denoising_strength() 75 | ) 76 | 77 | self.fifolog('Status', 'Engine activated') 78 | except Exception as e: 79 | self.fifolog('Error', e) 80 | 81 | def generate(self, scriptOp): 82 | stream = self.stream 83 | self.to_tensor.update(self.stream_toparray.cuda_stream) 84 | image = torch.as_tensor(self.to_tensor, device=self.device) 85 | image_tensor = self.preprocess_image(image) 86 | 87 | if hasattr(self.stream, 'batch_size'): 88 | last_element = 1 if stream.batch_size != 1 else 0 89 | for _ in range(stream.batch_size - last_element): 90 | output_image = stream(image=image_tensor) 91 | 92 | output_tensor = self.postprocess_image(output_image) 93 | scriptOp.copyCUDAMemory( 94 | output_tensor.data_ptr(), 95 | self.output_interface.size, 96 | self.output_interface.mem_shape) 97 | 98 | def update_size(self): 99 | width = int(op('parameter1')['Sizex',1]) 100 | height = int(op('parameter1')['Sizey',1]) 101 | print(width,height) 102 | self.rgba_tensor = torch.zeros((height, width, 4), dtype=torch.float32).to(self.device) 103 | self.rgba_tensor[..., 3] = 0 104 | self.output_interface = TopCUDAInterface(width,height,4,np.float32) 105 | 106 | 107 | def preprocess_image(self, image): 108 | image = torch.flip(image, [1]) 109 | image = torch.clamp(image, 0, 1) 110 | image = image[:3, :, :] 111 | _, h, w = image.shape 112 | # Resize to integer multiple of 32 113 | h, w = map(lambda x: x - x % 32, (h, w)) 114 | #image = self.blend_tensors(self.prev_frame, image, 0.5) 115 | image = image.unsqueeze(0) 116 | return image 117 | 118 | def postprocess_image(self, image): 119 | image = torch.flip(image, [1]) 120 | image = image.permute(1, 2, 0) 121 | self.rgba_tensor[..., :3] = image 122 | return self.rgba_tensor 123 | 124 | def acceleration_mode(self): 125 | turbo = False 126 | lcm = False 127 | acceleration_mode = parent().par.Acceleration.val 128 | if acceleration_mode == 'LCM': 129 | lcm = True 130 | if acceleration_mode == 'sd_turbo': 131 | turbo = True 132 | 133 | return lcm, turbo 134 | 135 | 136 | def update_engines(self): 137 | menuNames = [] 138 | menuLabels = [] 139 | for root, dirs, files in os.walk('engines'): 140 | if 'unet.engine' in files: 141 | folder_name = os.path.basename(root) 142 | split_folder_name = folder_name.split('--') 143 | if len(split_folder_name) >= 10: 144 | name = [split_folder_name[0], 145 | split_folder_name[2], 146 | split_folder_name[3], 147 | split_folder_name[5]] 148 | name = '-'.join(name) 149 | menuLabels.append(name) 150 | menuNames.append(folder_name) 151 | 152 | parent().par.Enginelist.menuNames = menuNames 153 | parent().par.Enginelist.menuLabels = menuLabels 154 | self.update_selected_engine() 155 | 156 | def update_selected_engine(self): 157 | try: 158 | vals = parent().par.Enginelist.val.split('--') 159 | parent().par.Checkpoint = vals[0] 160 | parent().par.Checkpointtype = vals[1] 161 | parent().par.Accelerationlora = vals[4] 162 | parent().par.Checkpointmode = vals[7] 163 | parent().par.Controlnet = vals[9] 164 | parent().par.Loralist = vals[8] 165 | parent().par.Sizex = vals[2] 166 | parent().par.Sizey = vals[3] 167 | parent().par.Batchsizex = vals[5] 168 | parent().par.Batchsizey = vals[6] 169 | except: 170 | parent().par.Checkpoint, parent().par.Checkpointtype, parent().par.Accelerationlora = '', '', '' 171 | parent().par.Checkpointmode, parent().par.Controlnet, parent().par.Loralist = '', '', '' 172 | parent().par.Sizex, parent().par.Sizey, parent().par.Batchsizex, parent().par.Batchsizey = 0,0,0,0 173 | 174 | 175 | 176 | def update_prompt(self): 177 | prompt = parent().par.Prompt.val 178 | self.stream.touchdiffusion_prompt(prompt) 179 | 180 | def prompt_to_str(self): 181 | prompt_list = [] 182 | seq = parent().seq.Promptblock 183 | enable_weights = parent().par.Enableweight 184 | for block in seq.blocks: 185 | if block.par.Weight.val > 0: 186 | if enable_weights: 187 | prompt_with_weight = f'({block.par.Prompt.val}){block.par.Weight.val}' 188 | else: 189 | prompt_with_weight = block.par.Prompt.val 190 | 191 | prompt_list.append(prompt_with_weight) 192 | 193 | prompt_str = ", ".join(prompt_list) 194 | return prompt_str 195 | 196 | def update_scheduler(self): 197 | t_index_list = [] 198 | seq = parent().seq.Schedulerblock 199 | for block in seq.blocks: 200 | t_index_list.append(block.par.Step) 201 | self.stream.touchdiffusion_scheduler(t_index_list) 202 | 203 | def update_denoising_strength(self): 204 | amount = parent().par.Denoise 205 | mode = parent().par.Denoisemode 206 | #self.stream.touchdiffusion_generate_t_index_list(amount, mode) 207 | t_index_list = self.stream.touchdiffusion_generate_t_index_list(amount, mode) 208 | return t_index_list 209 | 210 | def generate_t_index_list(self): 211 | batchsize = op('parameter1')['Batchsizex',1] 212 | t_index_list = [] 213 | for i in range(int(batchsize)): 214 | t_index_list.append(i) 215 | return t_index_list 216 | 217 | 218 | def update_cfg_setting(self): 219 | guidance_scale = parent().par.Cfgscale 220 | delta = parent().par.Deltamult.val 221 | self.stream.touchdiffusion_update_cfg_setting(guidance_scale=guidance_scale, delta=delta) 222 | 223 | def update_noise(self): 224 | seed = parent().par.Seed.val 225 | self.stream.touchdiffusion_update_noise(seed=seed) 226 | 227 | 228 | def parexec_onValueChange(self, par, prev): 229 | if hasattr(self.stream, 'batch_size'): 230 | if par.name == 'Prompt': 231 | self.update_prompt() 232 | elif par.name == 'Denoise': 233 | self.update_denoising_strength() 234 | elif par.name == 'Cfgscale': 235 | self.update_cfg_setting() 236 | elif par.name == 'Seed': 237 | self.update_noise() 238 | 239 | def parexec_onPulse(self, par): 240 | if par.name == 'Loadengine': 241 | self.activate_stream() 242 | elif par.name == 'Refreshenginelist': 243 | self.update_engines() 244 | if par.name[0:3] == 'Url': 245 | self.about(par.name) 246 | 247 | def fifolog(self, status, message): 248 | current_time = datetime.now() 249 | formated_time = current_time.strftime("%H:%M:%S") 250 | op('fifo1').appendRow([formated_time, status, message]) 251 | 252 | def about(self, endpoint): 253 | if endpoint == 'Urlg': 254 | webbrowser.open('https://github.com/olegchomp/TouchDiffusion', new=2) 255 | if endpoint == 'Urld': 256 | webbrowser.open('https://discord.gg/wNW8xkEjrf', new=2) 257 | if endpoint == 'Urlt': 258 | webbrowser.open('https://www.youtube.com/vjschool', new=2) 259 | if endpoint == 'Urla': 260 | webbrowser.open('https://olegcho.mp/', new=2) 261 | if endpoint == 'Urldonate': 262 | webbrowser.open('https://boosty.to/vjschool/', new=2) 263 | 264 | class TopCUDAInterface: 265 | def __init__(self, width, height, num_comps, dtype): 266 | self.mem_shape = CUDAMemoryShape() 267 | self.mem_shape.width = width 268 | self.mem_shape.height = height 269 | self.mem_shape.numComps = num_comps 270 | self.mem_shape.dataType = dtype 271 | self.bytes_per_comp = np.dtype(dtype).itemsize 272 | self.size = width * height * num_comps * self.bytes_per_comp 273 | 274 | class TopArrayInterface: 275 | def __init__(self, top, stream=0): 276 | self.top = top 277 | mem = top.cudaMemory(stream=stream) 278 | self.w, self.h = mem.shape.width, mem.shape.height 279 | self.num_comps = mem.shape.numComps 280 | self.dtype = mem.shape.dataType 281 | shape = (mem.shape.numComps, self.h, self.w) 282 | dtype_info = {'descr': [('', ' nul 2>&1 7 | if %errorlevel% neq 0 ( 8 | echo Git is not installed or not found in the system PATH. 9 | pause 10 | exit /b 1 11 | ) else ( 12 | echo Git is installed. 13 | ) 14 | 15 | if not exist .venv ( 16 | echo Creating .venv directory... 17 | %PYTHON_PATH% -m venv ".venv" || ( 18 | echo Failed to create virtual environment. 19 | pause 20 | exit /b 1 21 | ) 22 | 23 | echo Activating virtual environment... 24 | call .venv\Scripts\activate || ( 25 | echo Failed to activate virtual environment. 26 | pause 27 | exit /b 1 28 | ) 29 | 30 | echo Installing dependencies... 31 | python -m pip install --upgrade pip || ( 32 | echo Failed to update pip. 33 | pause 34 | exit /b 1 35 | ) 36 | 37 | echo Installing dependencies... 38 | pip install gradio || ( 39 | echo Failed to install gradio. 40 | pause 41 | exit /b 1 42 | ) 43 | 44 | if not exist StreamDiffusion ( 45 | echo Downloading StreamDiffusion... 46 | git clone https://github.com/olegchomp/StreamDiffusion || ( 47 | echo Failed to download StreamDiffusion 48 | pause 49 | exit /b 1 50 | ) 51 | ) 52 | 53 | echo Installation complete. 54 | 55 | echo Launching WebUI... 56 | python StreamDiffusion\webui.py || ( 57 | echo No launch file found 58 | pause 59 | exit /b 1 60 | ) 61 | 62 | ) else ( 63 | echo Activating virtual environment... 64 | call .venv\Scripts\activate.bat || ( 65 | echo Failed to activate virtual environment. 66 | pause 67 | exit /b 1 68 | ) 69 | 70 | if not exist StreamDiffusion ( 71 | echo Downloading StreamDiffusion... 72 | git clone https://github.com/olegchomp/StreamDiffusion || ( 73 | echo Failed to download StreamDiffusion 74 | pause 75 | exit /b 1 76 | ) 77 | ) 78 | 79 | echo Launching WebUI... 80 | python StreamDiffusion\webui.py || ( 81 | echo No launch file found 82 | pause 83 | exit /b 1 84 | ) 85 | ) 86 | 87 | pause 88 | --------------------------------------------------------------------------------