├── README.md
├── TouchDiffusion.tox
├── TouchDiffusionExt.py
└── webui.bat
/README.md:
--------------------------------------------------------------------------------
1 | # TouchDiffusion
2 |
3 |
4 | TouchDesigner implementation for real-time Stable Diffusion interactive generation with [StreamDiffusion](https://github.com/cumulo-autumn/StreamDiffusion).
5 |
6 | **Benchmarks with stabilityai/sd-turbo, 512x512 and 1 batch size.**
7 |
8 | | GPU | FPS |
9 | | --- | --- |
10 | | 4090 | 55-60 FPS |
11 | | 4080 | 47 FPS |
12 | | 3090ti | 37 FPS |
13 | | 3090 | 30-32 FPS |
14 | | 4070 Laptop | 24 FPS |
15 | | 3060 12GB | 16 FPS |
16 |
17 | ## Disclaimer
18 | **Notice:** This repository is in an early testing phase and may undergo significant changes. Use it at your own risk.
19 |
20 | ## Usage
21 | > [!TIP]
22 | > TouchDiffusion can be installed in multiple ways. **Portable version** have prebuild dendencies, so it prefered way to install or **Manuall install** is step by step instruction.
23 |
24 | #### Portable version:
25 | Includes preinstalled configurations, ensuring everything is readily available for immediate use.
26 | 1. Download and extract [archive](https://boosty.to/vjschool/posts/39931cd6-b9c5-4c27-93ff-d7a09b0918c5?share=post_link)
27 | 2. Run ```webui.bat```. It will provide url to web interface (ex. ```http://127.0.0.1:7860```)
28 | 3. Open ```install & update``` tab and run ```Update dependencies```.
29 |
30 | #### Manuall install:
31 | You can follow [YouTube tutorial](https://youtu.be/3WqUrWfCX1A)
32 |
33 | Required TouchDesigner 2023 & Python 3.11
34 | 1. Install [Python 3.11](https://www.python.org/downloads/release/python-3118/)
35 | 2. Install [Git](https://git-scm.com/downloads)
36 | 3. Install [CUDA Toolkit](https://developer.nvidia.com/cuda-11-8-0-download-archive) 11.8 (required PC restart)
37 | 4. Download [TouchDiffusion](https://github.com/olegchomp/TouchDiffusion/archive/refs/heads/main.zip).
38 | 5. Open ```webui.bat``` with text editor and set path to Python 3.11 in ```set PYTHON_PATH=```. (ex. ```set PYTHON_PATH="C:\Program Files\Python311\python.exe"```)
39 | 6. Run ```webui.bat```. After installation it will provide url to web interface (ex. ```http://127.0.0.1:7860```)
40 | 7. Open ```install & update``` tab and run ```Update dependencies```. (could take ~10 minutes, depending on your internet connection)
41 | 8. If you get pop up window with error related to .dll, run ```Fix pop up```
42 | 9. Restart webui.bat
43 |
44 | #### Accelerate model:
45 | Models in ```.safetensors``` format must be in ```models\checkpoints``` folder. (as for sd_turbo, it will be auto-downloaded).
46 |
47 | **Internet connection required, while making engines.**
48 |
49 | 1) Run ```webui.bat```
50 | 2) Select model type.
51 | 3) Select model.
52 | 4) Set width, height and amount of sampling steps (Batch size)
53 | 5) Select acceleration lora if available.
54 | 6) Run ```Make engine``` and wait for acceleration to finish. (could take ~10 minutes, depending on your hardware)
55 |
56 | #### TouchDesigner inference:
57 | 1. Add **TouchDiffusion.tox** to project
58 | 2. On ```Settings``` page change path to ```TouchDiffusion``` folder (same as where webui.bat).
59 | 3. Save and restart TouchDesigner project.
60 | 4. On ```Settings``` page select Engine and click **Load Engine**.
61 | 5. Connect animated TOP to input. Component cook only if input updates.
62 |
63 | #### Known issues / Roadmap:
64 | - [x] Fix Re-init. Sometimes required to restart TouchDesigner for initializing site-packages.
65 | - [ ] Code clean-up and rework.
66 | - [x] Custom resolution (for now fixed 512x512)
67 | - [ ] CFG not affecting image
68 | - [ ] Add Lora
69 | - [x] Add Hyper Lora support
70 | - [ ] Add ControlNet support
71 | - [ ] Add SDXL support
72 |
73 | ## Acknowledgement
74 | Based on the following projects:
75 | * [StreamDiffusion](https://github.com/cumulo-autumn/StreamDiffusion) - Pipeline-Level Solution for Real-Time Interactive Generation
76 | * [TopArray](https://github.com/IntentDev/TopArray) - Interaction between Python/PyTorch tensor operations and TouchDesigner TOPs.
77 |
--------------------------------------------------------------------------------
/TouchDiffusion.tox:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/olegchomp/TouchDiffusion/33dd7e822d3d3b96b2e85883ab8a132ae9aea9c2/TouchDiffusion.tox
--------------------------------------------------------------------------------
/TouchDiffusionExt.py:
--------------------------------------------------------------------------------
1 | from TDStoreTools import StorageManager
2 | import TDFunctions as TDF
3 | import numpy as np
4 | import torch
5 | import os
6 | import webbrowser
7 | import json
8 | from datetime import datetime
9 | import webbrowser
10 |
11 | try:
12 | from StreamDiffusion.utils.wrapper import StreamDiffusionWrapper
13 | except Exception as e:
14 | current_time = datetime.now()
15 | formated_time = current_time.strftime("%H:%M:%S")
16 | op('fifo1').appendRow([formated_time, 'Error', e])
17 |
18 |
19 | class TouchDiffusionExt:
20 | """
21 | DefaultExt description
22 | """
23 | def __init__(self, ownerComp):
24 | self.ownerComp = ownerComp
25 |
26 | self.source = op('null1')
27 | self.device = "cuda"
28 | self.to_tensor = TopArrayInterface(self.source)
29 | self.stream_toparray = torch.cuda.current_stream(device=self.device)
30 | self.rgba_tensor = torch.zeros((512, 512, 4), dtype=torch.float32).to(self.device) #512,768
31 | self.rgba_tensor[..., 3] = 0
32 | self.output_interface = TopCUDAInterface(512,512,4,np.float32) #768,512
33 | self.stream = None
34 |
35 | def activate_stream(self):
36 | self.update_size()
37 |
38 | acceleration_lora = op('parameter1')['Accelerationlora',1].val
39 | if acceleration_lora == 'LCM':
40 | use_lcm_lora = True
41 | elif acceleration_lora == 'HyperSD':
42 | use_hyper_lora = True
43 | else:
44 | use_lcm_lora = False
45 | use_hyper_lora = False
46 |
47 | try:
48 | self.stream = StreamDiffusionWrapper(
49 | model_id_or_path=f"{op('parameter1')['Checkpoint',1].val}",
50 | lora_dict=op('parameter1')['Loralist',1].val,
51 | t_index_list=self.generate_t_index_list(),
52 | frame_buffer_size=1,
53 | width= int(op('parameter1')['Sizex',1]),
54 | height=int(op('parameter1')['Sizey',1]),
55 | warmup=0,
56 | acceleration="tensorrt",
57 | mode= op('parameter1')['Checkpointmode',1].val,
58 | use_denoising_batch=True,
59 | cfg_type="self",
60 | seed=int(op('parameter1')['Seed',1]),
61 | use_lcm_lora=use_lcm_lora,
62 | use_hyper_lora=use_hyper_lora,
63 | output_type='pt',
64 | model_type=op('parameter1')['Checkpointtype',1].val,
65 | touchdiffusion=True,
66 | #turbo=False
67 | )
68 |
69 | self.stream.prepare(
70 | prompt = parent().par.Prompt.val,
71 | negative_prompt = parent().par.Negprompt.val,
72 | guidance_scale=parent().par.Cfgscale.val,
73 | delta=parent().par.Deltamult.val,
74 | t_index_list=self.update_denoising_strength()
75 | )
76 |
77 | self.fifolog('Status', 'Engine activated')
78 | except Exception as e:
79 | self.fifolog('Error', e)
80 |
81 | def generate(self, scriptOp):
82 | stream = self.stream
83 | self.to_tensor.update(self.stream_toparray.cuda_stream)
84 | image = torch.as_tensor(self.to_tensor, device=self.device)
85 | image_tensor = self.preprocess_image(image)
86 |
87 | if hasattr(self.stream, 'batch_size'):
88 | last_element = 1 if stream.batch_size != 1 else 0
89 | for _ in range(stream.batch_size - last_element):
90 | output_image = stream(image=image_tensor)
91 |
92 | output_tensor = self.postprocess_image(output_image)
93 | scriptOp.copyCUDAMemory(
94 | output_tensor.data_ptr(),
95 | self.output_interface.size,
96 | self.output_interface.mem_shape)
97 |
98 | def update_size(self):
99 | width = int(op('parameter1')['Sizex',1])
100 | height = int(op('parameter1')['Sizey',1])
101 | print(width,height)
102 | self.rgba_tensor = torch.zeros((height, width, 4), dtype=torch.float32).to(self.device)
103 | self.rgba_tensor[..., 3] = 0
104 | self.output_interface = TopCUDAInterface(width,height,4,np.float32)
105 |
106 |
107 | def preprocess_image(self, image):
108 | image = torch.flip(image, [1])
109 | image = torch.clamp(image, 0, 1)
110 | image = image[:3, :, :]
111 | _, h, w = image.shape
112 | # Resize to integer multiple of 32
113 | h, w = map(lambda x: x - x % 32, (h, w))
114 | #image = self.blend_tensors(self.prev_frame, image, 0.5)
115 | image = image.unsqueeze(0)
116 | return image
117 |
118 | def postprocess_image(self, image):
119 | image = torch.flip(image, [1])
120 | image = image.permute(1, 2, 0)
121 | self.rgba_tensor[..., :3] = image
122 | return self.rgba_tensor
123 |
124 | def acceleration_mode(self):
125 | turbo = False
126 | lcm = False
127 | acceleration_mode = parent().par.Acceleration.val
128 | if acceleration_mode == 'LCM':
129 | lcm = True
130 | if acceleration_mode == 'sd_turbo':
131 | turbo = True
132 |
133 | return lcm, turbo
134 |
135 |
136 | def update_engines(self):
137 | menuNames = []
138 | menuLabels = []
139 | for root, dirs, files in os.walk('engines'):
140 | if 'unet.engine' in files:
141 | folder_name = os.path.basename(root)
142 | split_folder_name = folder_name.split('--')
143 | if len(split_folder_name) >= 10:
144 | name = [split_folder_name[0],
145 | split_folder_name[2],
146 | split_folder_name[3],
147 | split_folder_name[5]]
148 | name = '-'.join(name)
149 | menuLabels.append(name)
150 | menuNames.append(folder_name)
151 |
152 | parent().par.Enginelist.menuNames = menuNames
153 | parent().par.Enginelist.menuLabels = menuLabels
154 | self.update_selected_engine()
155 |
156 | def update_selected_engine(self):
157 | try:
158 | vals = parent().par.Enginelist.val.split('--')
159 | parent().par.Checkpoint = vals[0]
160 | parent().par.Checkpointtype = vals[1]
161 | parent().par.Accelerationlora = vals[4]
162 | parent().par.Checkpointmode = vals[7]
163 | parent().par.Controlnet = vals[9]
164 | parent().par.Loralist = vals[8]
165 | parent().par.Sizex = vals[2]
166 | parent().par.Sizey = vals[3]
167 | parent().par.Batchsizex = vals[5]
168 | parent().par.Batchsizey = vals[6]
169 | except:
170 | parent().par.Checkpoint, parent().par.Checkpointtype, parent().par.Accelerationlora = '', '', ''
171 | parent().par.Checkpointmode, parent().par.Controlnet, parent().par.Loralist = '', '', ''
172 | parent().par.Sizex, parent().par.Sizey, parent().par.Batchsizex, parent().par.Batchsizey = 0,0,0,0
173 |
174 |
175 |
176 | def update_prompt(self):
177 | prompt = parent().par.Prompt.val
178 | self.stream.touchdiffusion_prompt(prompt)
179 |
180 | def prompt_to_str(self):
181 | prompt_list = []
182 | seq = parent().seq.Promptblock
183 | enable_weights = parent().par.Enableweight
184 | for block in seq.blocks:
185 | if block.par.Weight.val > 0:
186 | if enable_weights:
187 | prompt_with_weight = f'({block.par.Prompt.val}){block.par.Weight.val}'
188 | else:
189 | prompt_with_weight = block.par.Prompt.val
190 |
191 | prompt_list.append(prompt_with_weight)
192 |
193 | prompt_str = ", ".join(prompt_list)
194 | return prompt_str
195 |
196 | def update_scheduler(self):
197 | t_index_list = []
198 | seq = parent().seq.Schedulerblock
199 | for block in seq.blocks:
200 | t_index_list.append(block.par.Step)
201 | self.stream.touchdiffusion_scheduler(t_index_list)
202 |
203 | def update_denoising_strength(self):
204 | amount = parent().par.Denoise
205 | mode = parent().par.Denoisemode
206 | #self.stream.touchdiffusion_generate_t_index_list(amount, mode)
207 | t_index_list = self.stream.touchdiffusion_generate_t_index_list(amount, mode)
208 | return t_index_list
209 |
210 | def generate_t_index_list(self):
211 | batchsize = op('parameter1')['Batchsizex',1]
212 | t_index_list = []
213 | for i in range(int(batchsize)):
214 | t_index_list.append(i)
215 | return t_index_list
216 |
217 |
218 | def update_cfg_setting(self):
219 | guidance_scale = parent().par.Cfgscale
220 | delta = parent().par.Deltamult.val
221 | self.stream.touchdiffusion_update_cfg_setting(guidance_scale=guidance_scale, delta=delta)
222 |
223 | def update_noise(self):
224 | seed = parent().par.Seed.val
225 | self.stream.touchdiffusion_update_noise(seed=seed)
226 |
227 |
228 | def parexec_onValueChange(self, par, prev):
229 | if hasattr(self.stream, 'batch_size'):
230 | if par.name == 'Prompt':
231 | self.update_prompt()
232 | elif par.name == 'Denoise':
233 | self.update_denoising_strength()
234 | elif par.name == 'Cfgscale':
235 | self.update_cfg_setting()
236 | elif par.name == 'Seed':
237 | self.update_noise()
238 |
239 | def parexec_onPulse(self, par):
240 | if par.name == 'Loadengine':
241 | self.activate_stream()
242 | elif par.name == 'Refreshenginelist':
243 | self.update_engines()
244 | if par.name[0:3] == 'Url':
245 | self.about(par.name)
246 |
247 | def fifolog(self, status, message):
248 | current_time = datetime.now()
249 | formated_time = current_time.strftime("%H:%M:%S")
250 | op('fifo1').appendRow([formated_time, status, message])
251 |
252 | def about(self, endpoint):
253 | if endpoint == 'Urlg':
254 | webbrowser.open('https://github.com/olegchomp/TouchDiffusion', new=2)
255 | if endpoint == 'Urld':
256 | webbrowser.open('https://discord.gg/wNW8xkEjrf', new=2)
257 | if endpoint == 'Urlt':
258 | webbrowser.open('https://www.youtube.com/vjschool', new=2)
259 | if endpoint == 'Urla':
260 | webbrowser.open('https://olegcho.mp/', new=2)
261 | if endpoint == 'Urldonate':
262 | webbrowser.open('https://boosty.to/vjschool/', new=2)
263 |
264 | class TopCUDAInterface:
265 | def __init__(self, width, height, num_comps, dtype):
266 | self.mem_shape = CUDAMemoryShape()
267 | self.mem_shape.width = width
268 | self.mem_shape.height = height
269 | self.mem_shape.numComps = num_comps
270 | self.mem_shape.dataType = dtype
271 | self.bytes_per_comp = np.dtype(dtype).itemsize
272 | self.size = width * height * num_comps * self.bytes_per_comp
273 |
274 | class TopArrayInterface:
275 | def __init__(self, top, stream=0):
276 | self.top = top
277 | mem = top.cudaMemory(stream=stream)
278 | self.w, self.h = mem.shape.width, mem.shape.height
279 | self.num_comps = mem.shape.numComps
280 | self.dtype = mem.shape.dataType
281 | shape = (mem.shape.numComps, self.h, self.w)
282 | dtype_info = {'descr': [('', ' nul 2>&1
7 | if %errorlevel% neq 0 (
8 | echo Git is not installed or not found in the system PATH.
9 | pause
10 | exit /b 1
11 | ) else (
12 | echo Git is installed.
13 | )
14 |
15 | if not exist .venv (
16 | echo Creating .venv directory...
17 | %PYTHON_PATH% -m venv ".venv" || (
18 | echo Failed to create virtual environment.
19 | pause
20 | exit /b 1
21 | )
22 |
23 | echo Activating virtual environment...
24 | call .venv\Scripts\activate || (
25 | echo Failed to activate virtual environment.
26 | pause
27 | exit /b 1
28 | )
29 |
30 | echo Installing dependencies...
31 | python -m pip install --upgrade pip || (
32 | echo Failed to update pip.
33 | pause
34 | exit /b 1
35 | )
36 |
37 | echo Installing dependencies...
38 | pip install gradio || (
39 | echo Failed to install gradio.
40 | pause
41 | exit /b 1
42 | )
43 |
44 | if not exist StreamDiffusion (
45 | echo Downloading StreamDiffusion...
46 | git clone https://github.com/olegchomp/StreamDiffusion || (
47 | echo Failed to download StreamDiffusion
48 | pause
49 | exit /b 1
50 | )
51 | )
52 |
53 | echo Installation complete.
54 |
55 | echo Launching WebUI...
56 | python StreamDiffusion\webui.py || (
57 | echo No launch file found
58 | pause
59 | exit /b 1
60 | )
61 |
62 | ) else (
63 | echo Activating virtual environment...
64 | call .venv\Scripts\activate.bat || (
65 | echo Failed to activate virtual environment.
66 | pause
67 | exit /b 1
68 | )
69 |
70 | if not exist StreamDiffusion (
71 | echo Downloading StreamDiffusion...
72 | git clone https://github.com/olegchomp/StreamDiffusion || (
73 | echo Failed to download StreamDiffusion
74 | pause
75 | exit /b 1
76 | )
77 | )
78 |
79 | echo Launching WebUI...
80 | python StreamDiffusion\webui.py || (
81 | echo No launch file found
82 | pause
83 | exit /b 1
84 | )
85 | )
86 |
87 | pause
88 |
--------------------------------------------------------------------------------