├── LICENSE ├── README.md ├── assets ├── cmd-line-python-udp-msg-args.gif └── cmd-line-python-udp-msg.gif ├── base_python_and_subprocess.toe └── scripts ├── cmd_line_python.py ├── cmd_line_python_args.py ├── cmd_line_python_udp_msg.py └── cmd_line_python_udp_msg_args.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Matthew Ragan 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Working with a Python Subprocess Call and TouchDesigner 2 | 3 | At the TouchDesigner Summit in Montreal we'll be taking some time to talk about working with external Python Modules in TouchDesigner. While we'll have time to cover lots of information about how to incorporate external modules in Touch, we won't have a lot of time to talk through the wobbles that you might run into when working with operations that might be slow, or otherwise unwieldy to run in Touch. 4 | 5 | "What do you mean Matt?" 6 | 7 | The types of pieces that usually fall into this category are blocking operations. For example, let's say that you want to upload an image to the web somewhere. Many of the libraries that you might find will have an approach that's probably blocking - as in it will appear as if TouchDesigner has frozen while the whole operation completes. While this is fine outside of Touch, we don't typically like it when our applications appear to freeze - especially in installations or live performances. You might be able to move that process to another thread, though that might be a little more hassle that you really want it to be in the long run. 8 | 9 | Enter the Python Subprocess module. 10 | 11 | ## Subprocess 12 | The subprocess module allows you to run a python script as a parallel execution that doesn't touch your TouchDesigner application. You could use this for all sorts of interesting an powerful applications - from starting media syncing between machines, running a process that talks to the internet, or any number of solutions that execute outside of TouchDesigner. In the past I've used this for things like sending emails, or uploading images to instagram - there's lots you can do with this approach, it's just a matter of wrangling python and the subprocess module. 13 | 14 | What exactly is happening when we use the subprocess module?! Well, we can think of this as a situation where we write a python script in a text file, and then ask your operating system to run that file. There are great ways to pass in arguments into those situations, and if you really need data there are ways to get a response before the process quits. This can be a very flexible solution for a number of situations, and worth looking into if you want something that's non-blocking and can be run outside of TouchDesigner. 15 | 16 | Subprocess calls can be infuriating if you're not familiar with them, so let's look at some simple anatomy of making this work from Touch. 17 | 18 | ---- 19 | 20 | ## Scenario 1 - Execute this Script 21 | Let's start with the most basic of scenarios. Here we have some Python script that normally takes a long time to run, that we just want to kick off from TouchDesigner. For this example let's just look at something that will print to a shell - nothing fancy, just a place to get our bearings. Our python script might look something like this: 22 | 23 | ### Pure Python 24 | ```python 25 | import time 26 | import sys 27 | 28 | # a variable a divider that we're going to use 29 | divider = '- ' * 10 30 | 31 | # a for loop to print all of the paths in our sys.path 32 | for each in sys.path: 33 | print(each) 34 | 35 | # our divider 36 | print(divider) 37 | 38 | # a for loop that prints numbers 39 | for each in range(10): 40 | print(each) 41 | 42 | # a call to time.sleep to keep our terminal open so we can see what's happening 43 | time.sleep(120) 44 | ``` 45 | This works just the way that we might expect if we run it in our OS. But how can we run this script from TouchDesigner? 46 | 47 | ### In TouchDesigner 48 | In TouchDesigner we'd add a DAT that has the following contents. Here we assume that the script above has been saved in a folder called `scripts` that's in the same folder as our project file, and the name of the script is `cmd_line_python.py`. 49 | ```python 50 | import subprocess 51 | 52 | # point to our script that we're going to execute 53 | cmd_python_script = '{}/scripts/cmd_line_python.py'.format(project.folder) 54 | 55 | # quick debug print 56 | print(cmd_python_script) 57 | 58 | # call our script with subprocess 59 | subprocess.Popen(['python', cmd_python_script], shell=False) 60 | ``` 61 | This is great, but this will actually execute with the version of Python that's packaged with TouchDesigner. In some cases that's exactly what we want... in other's we might want to use a different version of python that's installed on our OS. How can we do that? 62 | 63 | ---- 64 | 65 | ## Scenario 2 - Execute this Script with a Specific Python 66 | This is very similar to our first situation, but here we want to run the python that's installed on our OS, not the python that's packaged with Touch. In this case we can use the exact same python script we saw above. 67 | 68 | ### Pure Python 69 | ```python 70 | import time 71 | import sys 72 | 73 | # a variable a divider that we're going to use 74 | divider = '- ' * 10 75 | 76 | # a for loop to print all of the paths in our sys.path 77 | for each in sys.path: 78 | print(each) 79 | 80 | # our divider 81 | print(divider) 82 | 83 | # a for loop that prints numbers 84 | for each in range(10): 85 | print(each) 86 | 87 | # a call to time.sleep to keep our terminal open so we can see what's happening 88 | time.sleep(120) 89 | ``` 90 | 91 | Our real changes come when we're issuing the subprocess call in TouchDesigner. 92 | 93 | ### In TouchDesigner 94 | Again, we add a DAT that has the following contents. Here we assume that the script above has been saved in a folder called `scripts` that's in the same folder as our project file, and the name of the script is `cmd_line_python.py`. The additional wrinkle this time, is that we also need to specify which `python.exe` we want to use for this process. 95 | 96 | ```python 97 | import subprocess 98 | 99 | # point to our script that we're going to execute 100 | cmd_python_script = '{}/scripts/cmd_line_python.py'.format(project.folder) 101 | 102 | # point to the specific version of python that we want to use 103 | python_exe = 'C:/Program Files/Python35/python.exe' 104 | 105 | # quick debug print 106 | print(cmd_python_script) 107 | 108 | # call our script with subprocess 109 | subprocess.Popen([python_exe, cmd_python_script], shell=False) 110 | ``` 111 | If we look closely at the example above, we can see that we've been very specific about where our `python_exe` lives. This approach runs the same script, only this time with the `python.exe` that we've specifically pointed to. 112 | 113 | ---- 114 | 115 | ## Scenario 3 - Passing Over Args 116 | There are several ways to approach this challenge. One that will line up with the format of many pure pythonic approaches here would be to use the `argparse` library. We find this in lots of stand-alone scripts, and it allows us the flexibility of setting default arguments, and creating some relatively clean inputs when calling a script form the command line. In this approach we set up a function that we'll call, and pass arguments into - in the example below that's our `My_python_method()`. By using `ArgumentParser` we can pass in command line arguments that we can in turn pass through to our function. Notice the syntax at the bottom to see how this works. `ArgumentParser` returns keyword arguments, which is why we use `kwargs` as the mechanism for sending args into our simple for loop. 117 | 118 | ### Pure Python 119 | ```python 120 | import time 121 | import sys 122 | import time 123 | from argparse import ArgumentParser 124 | 125 | def My_python_method(kwargs): 126 | 127 | disp_str = 'key: {} | value: {} | type: {}' 128 | for each_key, each_value in kwargs.items(): 129 | formatted_str = disp_str.format(each_key, each_value, type(each_value)) 130 | print(formatted_str) 131 | 132 | # keep the shell open so we can debug 133 | time.sleep(int(kwargs.get('delay'))) 134 | 135 | # execution order matters -this puppy has to be at the bottom as our functions are defined above 136 | if __name__ == '__main__': 137 | parser = ArgumentParser(description='A simple argument input example') 138 | parser.add_argument("-i", "--input", dest="in", help="an input string", required=True) 139 | parser.add_argument("-i2", "--input2", dest="in2", help="another input", required=True) 140 | parser.add_argument("-d", "--delay", dest="delay", help="how long our terminal stays up", required=False, default=10) 141 | 142 | args = parser.parse_args() 143 | My_python_method(vars(args)) 144 | pass 145 | 146 | # example 147 | # python .\cmd_line_python_args.py -i="a string" -i2="another string" -d=15 148 | ``` 149 | 150 | Where this becomes more interesting is when we look at what's happening on the TouchDesigner side of this equation. 151 | 152 | ### In TouchDesigner 153 | A DAT in TouchDesigner needs to follow the same rules we established so far - we need to know what executable we're using, which file we're running, and finally we now need to send along some arguments that will be passed to that file. In Touch, our script this time should look something like this: 154 | 155 | ```python 156 | import subprocess 157 | 158 | # point to our script that we're going to execute 159 | cmd_python_script = '{}/scripts/cmd_line_python_args.py'.format(project.folder) 160 | 161 | # construct a list of arguments for out external script 162 | script_args = ['-i', 'Hello', '-i2', 'TouchDesigner'] 163 | 164 | # join our python instructions with our scirpt args 165 | command_list = ['python', cmd_python_script] + script_args 166 | 167 | # call our script with subprocess 168 | subprocess.Popen(command_list, shell=False) 169 | ``` 170 | 171 | ---- 172 | ## Scenario 4 - I Want a Message Back 173 | 174 | Like all things Touch, and all things Python there are LOTS of ways to accomplish this task. One way we might want to consider, however, is using UDP messages. Touch happens to have a handy `UDPIn DAT` that's ready to accept messages, and the other benefit here is that we could potentially target another machine on our network as the target for these messages. For this first exploration let's imagine that we're only sending a message locally, and that all of our variables are defined in the python script we're running. We'll need to use the `socket` library to help with the communication elements, and you'll notice that we import that at the top of our script. This silly example just creates a UDP connection, and sends messages at a regular interval. 175 | 176 | ### Pure Python 177 | ```python 178 | import time 179 | import sys 180 | import socket 181 | 182 | upd_ip = "127.0.0.1" 183 | udp_port = 7000 184 | sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 185 | 186 | num_iters = 11 187 | sleep_interval = 2 188 | 189 | def msg_to_bytes(msg): 190 | return msg.encode('utf-8') 191 | 192 | starting_msg = "Sending messages at an interval of {} seconds".format(sleep_interval) 193 | sock.sendto(msg_to_bytes(starting_msg), (upd_ip, udp_port)) 194 | 195 | for each in range(num_iters): 196 | msg = "{} of {}".format(each, num_iters-1) 197 | sock.sendto(msg_to_bytes(msg), (upd_ip, udp_port)) 198 | time.sleep(sleep_interval) 199 | 200 | ending_msg = "All messages sent" 201 | sock.sendto(msg_to_bytes(ending_msg), (upd_ip, udp_port)) 202 | ``` 203 | 204 | ### In TouchDesigner 205 | For this simple execution in TouchDesigner we only need to worry about kicking off the script. That looks almost exactly like the other pieces we've set up so far. 206 | ```python 207 | import subprocess 208 | 209 | # point to our script that we're going to execute 210 | cmd_python_script = '{}/scripts/cmd_line_python_udp_msg.py'.format(project.folder) 211 | 212 | # print our script path - quick debug 213 | print(cmd_python_script) 214 | 215 | # clear the last entries from the UDPin DAT 216 | op('udpin1').par.clear.pulse() 217 | 218 | # call our script with subprocess 219 | subprocess.Popen(['python', cmd_python_script], shell=True) 220 | ``` 221 | Our catch this time is that we nee dto use a `UDPIn DAT` to receive those messages. Let's also make sure the `Row/Callback Format` parameter is set to `One Per Message`. With this all set up we should see something like this when we kick off the script. 222 | 223 | ![](assets/cmd-line-python-udp-msg.gif) 224 | 225 | ---- 226 | 227 | ## Scenario 5 - Messages, I can has args?! 228 | That all seems mighty fine... but, what happens when I want to combine what we've done with passing along arguments, and messages? I'm so glad you asked. With a little extra work we can make exactly that happen. We do need to do a little more heavy lifting on the python front, but that work gives us some extra flexibility. You'll notice below that we're now passing along which port we want to use, how many iterations of our for loop, and the interval between repetitions. 229 | 230 | ### Pure Python 231 | ```python 232 | import time 233 | import sys 234 | import socket 235 | from argparse import ArgumentParser 236 | 237 | def msg_to_bytes(msg): 238 | return msg.encode('utf-8') 239 | 240 | def msg_loop(port, interval, loop): 241 | # localhost 242 | upd_ip = "127.0.0.1" 243 | 244 | # set udp port with input val and initialize socket connection 245 | udp_port = int(port) 246 | sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 247 | 248 | # set additional variables with input vals 249 | num_iters = int(loop) 250 | sleep_interval = int(interval) 251 | 252 | # send message that we're starting 253 | starting_msg = "Sending messages at an interval of {} seconds".format(sleep_interval) 254 | sock.sendto(msg_to_bytes(starting_msg), (upd_ip, udp_port)) 255 | 256 | # run message loop 257 | for each in range(num_iters): 258 | msg = "{} of {}".format(each, num_iters-1) 259 | sock.sendto(msg_to_bytes(msg), (upd_ip, udp_port)) 260 | time.sleep(sleep_interval) 261 | 262 | # send message that we're ending 263 | ending_msg = "All messages sent" 264 | sock.sendto(msg_to_bytes(ending_msg), (upd_ip, udp_port)) 265 | 266 | # execution order matters - this puppy has to be at the bottom as our functions are defined above 267 | if __name__ == '__main__': 268 | parser = ArgumentParser(description='A simple UDP example') 269 | parser.add_argument("-p", "--port", dest="port", help="UDP port", required=True, default=1234) 270 | parser.add_argument("-i", "--interval", dest="interval", help="loop interval", required=True, default=5) 271 | parser.add_argument("-l", "--loop", dest="loop", help="number of repetitions", required=True, default=10) 272 | args = parser.parse_args() 273 | 274 | msg_loop(args.port, args.interval, args.loop) 275 | pass 276 | 277 | # example 278 | # python .\cmd_line_python_udp_msg_args.py -p=5000 -i=2 -l=10 279 | ``` 280 | 281 | ### In TouchDesigner 282 | Over in TouchDesigner, our subprocess script looks very similar to what we've done so far with just a few modifications. 283 | 284 | ```python 285 | import subprocess 286 | 287 | # set up our variables for our subprocess call 288 | port = str(op('udpin2').par.port.val) 289 | interval = '1' 290 | loop = '15' 291 | 292 | # point to our script that we're going to execute 293 | cmd_python_script = '{}/scripts/cmd_line_python_udp_msg_args.py'.format(project.folder) 294 | 295 | # construct a list of our python args - which python and which script 296 | python_args = ['python', cmd_python_script] 297 | 298 | # construct a list of arguments for out external script 299 | script_args = ['-p', port, '-i', interval, '-l', loop] 300 | 301 | # join our two lists - python args and scirpt args 302 | cmd_args = python_args + script_args 303 | 304 | # quick debug print 305 | print(cmd_args) 306 | 307 | # clear the last entries from the UDPin DAT 308 | op('udpin2').par.clear.pulse() 309 | 310 | # call our script with subprocess 311 | subprocess.Popen(cmd_args, shell=True) 312 | ``` 313 | Here the resulting messages look veyr simlar, only we've not gotten to specify all the qualtiies about the for loop from our script. 314 | 315 | ![](assets/cmd-line-python-udp-msg-args.gif) 316 | 317 | --- 318 | 319 | ## What does this Matter? 320 | Well, there are lots of things you might do with this, but espeically interesting might be considering how you can use otherwise very costly and slow operations in Python with this approach. For example, a recent set of Style Transfer experments I was working on used this style of approach to essentially create a TouchDesigner front end / UI for a `pytorch` style transfer backned. This let me pass along arguments from the UI over to a pure python execution that didn't block or freeze touch while it was running. There are lots of ways you might get into mischief with this kind of work, and it's worth pointing out that while all this is focused on python, there's no reason you couldn't instead think of other applications you want to run with a particular file and a set of command line arguments. 321 | 322 | Happy Programming! -------------------------------------------------------------------------------- /assets/cmd-line-python-udp-msg-args.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raganmd/blog-td-subprocess/edd97efa7d6337a75a0f2a9709fb7e78218e87cd/assets/cmd-line-python-udp-msg-args.gif -------------------------------------------------------------------------------- /assets/cmd-line-python-udp-msg.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raganmd/blog-td-subprocess/edd97efa7d6337a75a0f2a9709fb7e78218e87cd/assets/cmd-line-python-udp-msg.gif -------------------------------------------------------------------------------- /base_python_and_subprocess.toe: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/raganmd/blog-td-subprocess/edd97efa7d6337a75a0f2a9709fb7e78218e87cd/base_python_and_subprocess.toe -------------------------------------------------------------------------------- /scripts/cmd_line_python.py: -------------------------------------------------------------------------------- 1 | import time 2 | import sys 3 | 4 | divider = '- ' * 10 5 | 6 | for each in sys.path: 7 | print(each) 8 | 9 | print(divider) 10 | 11 | for each in range(10): 12 | print(each) 13 | 14 | time.sleep(120) -------------------------------------------------------------------------------- /scripts/cmd_line_python_args.py: -------------------------------------------------------------------------------- 1 | import time 2 | from argparse import ArgumentParser 3 | 4 | def My_python_method(kwargs): 5 | 6 | disp_str = 'key: {} | value: {} | type: {}' 7 | for each_key, each_value in kwargs.items(): 8 | formatted_str = disp_str.format(each_key, each_value, type(each_value)) 9 | print(formatted_str) 10 | 11 | # keep the shell open so we can debug 12 | time.sleep(int(kwargs.get('delay'))) 13 | 14 | # execution order matters -this puppy has to be at the bottom as our functions are defined above 15 | if __name__ == '__main__': 16 | parser = ArgumentParser(description='A simple argument input example') 17 | parser.add_argument("-i", "--input", dest="in", help="an input string", required=True) 18 | parser.add_argument("-i2", "--input2", dest="in2", help="another input", required=True) 19 | parser.add_argument("-d", "--delay", dest="delay", help="how long our terminal stays up", required=False, default=10) 20 | args = parser.parse_args() 21 | print(vars(args)) 22 | My_python_method(vars(args)) 23 | # My_python_method(args.input, args.intput2, args.delay) 24 | pass 25 | 26 | # example 27 | # python .\cmd_line_python_args.py -i="a string" -i2="another string" -d=15 -------------------------------------------------------------------------------- /scripts/cmd_line_python_udp_msg.py: -------------------------------------------------------------------------------- 1 | import time 2 | import sys 3 | import socket 4 | 5 | upd_ip = "127.0.0.1" 6 | udp_port = 7000 7 | sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 8 | 9 | divider = '- ' * 10 10 | num_iters = 11 11 | sleep_interval = 2 12 | 13 | def msg_to_bytes(msg): 14 | return msg.encode('utf-8') 15 | 16 | starting_msg = "Sending messages at an interval of {} seconds".format(sleep_interval) 17 | sock.sendto(msg_to_bytes(starting_msg), (upd_ip, udp_port)) 18 | 19 | for each in range(num_iters): 20 | msg = "{} of {}".format(each, num_iters-1) 21 | sock.sendto(msg_to_bytes(msg), (upd_ip, udp_port)) 22 | time.sleep(sleep_interval) 23 | 24 | ending_msg = "All messages sent" 25 | sock.sendto(msg_to_bytes(ending_msg), (upd_ip, udp_port)) -------------------------------------------------------------------------------- /scripts/cmd_line_python_udp_msg_args.py: -------------------------------------------------------------------------------- 1 | import time 2 | import sys 3 | import socket 4 | from argparse import ArgumentParser 5 | 6 | def msg_to_bytes(msg): 7 | return msg.encode('utf-8') 8 | 9 | def msg_loop(port, interval, loop): 10 | # localhost 11 | upd_ip = "127.0.0.1" 12 | 13 | # set udp port with input val and initialize socket connection 14 | udp_port = int(port) 15 | sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 16 | 17 | # set additional variables with input vals 18 | num_iters = int(loop) 19 | sleep_interval = int(interval) 20 | 21 | # send message that we're starting 22 | starting_msg = "Sending messages at an interval of {} seconds".format(sleep_interval) 23 | sock.sendto(msg_to_bytes(starting_msg), (upd_ip, udp_port)) 24 | 25 | # run message loop 26 | for each in range(num_iters): 27 | msg = "{} of {}".format(each, num_iters-1) 28 | sock.sendto(msg_to_bytes(msg), (upd_ip, udp_port)) 29 | time.sleep(sleep_interval) 30 | 31 | # send message that we're ending 32 | ending_msg = "All messages sent" 33 | sock.sendto(msg_to_bytes(ending_msg), (upd_ip, udp_port)) 34 | 35 | 36 | 37 | # execution order matters -this puppy has to be at the bottom as our functions are defined above 38 | if __name__ == '__main__': 39 | parser = ArgumentParser(description='A simple UDP example') 40 | parser.add_argument("-p", "--port", dest="port", help="UDP port", required=True, default=1234) 41 | parser.add_argument("-i", "--interval", dest="interval", help="loop interval", required=True, default=5) 42 | parser.add_argument("-l", "--loop", dest="loop", help="number of repetitions", required=True, default=10) 43 | args = parser.parse_args() 44 | 45 | msg_loop(args.port, args.interval, args.loop) 46 | pass 47 | 48 | # example 49 | # python .\cmd_line_python_udp_msg_args.py -p=5000 -i=2 -l=10 --------------------------------------------------------------------------------