├── README.md ├── runforrest.py ├── setup.py └── test_runforrest.py /README.md: -------------------------------------------------------------------------------- 1 | # Run, Forrest, Run! 2 | 3 | Because sometimes a single Python is just not fast enough. 4 | 5 | You have some code that looks like this: 6 | 7 | ```python 8 | for item in long_list_of_stuff: 9 | intermediate = prepare(item) 10 | result = dostuff(intermediate[1], intermediate.thing) 11 | ``` 12 | 13 | And the problem is, both `prepare` and `dostuff` take too long, and 14 | `long_list_of_stuff` is just too damn long. Running the above script 15 | takes *ages*. 16 | 17 | What can you do? You could use [IPyParallel][ip]! 18 | Or [Dask.distributed][dd]! Or [multiprocessing][mp]! 19 | 20 | [ip]: https://ipyparallel.readthedocs.io/en/latest/ 21 | [dd]: https://distributed.readthedocs.io/en/latest/ 22 | [mp]: https://docs.python.org/3.6/library/multiprocessing.html 23 | 24 | Well, but IPyParallel has kind of a weird API, and Dask.distributed is 25 | hard to set up, and multiprocessing is kind of limited... I've been 26 | there, I've done that, and was not satisfied. 27 | 28 | Furthermore, none of the above can handle seriously buggy programs 29 | that throw segfaults, corrupt your file system, leak memory, and 30 | exhaust your process limits. All of this has happened to me when I 31 | tried to run scientific code. 32 | 33 | So why is RunForrest better? 34 | 35 | 1. Understandable. Just short of 200 lines of source code is manageable. 36 | 2. No complicated dependencies. A recent version of Python with `dill` is all you need. 37 | 3. Simple. The above call graph will now look like this: 38 | 4. Robust. Runforrest survives errors, crashes, and even reboots, without losing data. 39 | 40 | ```python 41 | from runforrest import TaskList, defer 42 | tasklist = TaskList('tasklist') 43 | 44 | for item in long_list_of_stuff: 45 | task = defer(prepare, item) 46 | task = defer(dostuff, task[1], task.thing) 47 | tasklist.schedule(task) 48 | 49 | for task in tasklist.run(nprocesses=4): 50 | result = task.returnvalue # or task.errorvalue 51 | ``` 52 | 53 | Wrap your function calls in `defer`, `schedule` them to a `TaskList`, 54 | then `run`. That's all there is to it. Deferred functions return 55 | objects that can be indexed and getattr'd as much as you'd like and 56 | all of that will be resolved once they are `run` (a single return 57 | object will never execute more than one time). 58 | 59 | But the best thing is, each `schedule` will just create a file in a 60 | new directory `tasklist/todo`. Then `run` will execute those files, 61 | and put them in `tasklist/done` or `tasklist/fail` depending on 62 | whether there were errors or not. 63 | 64 | This solves so many problems. Maybe you want to try to re-run a failed 65 | item? Just copy them back over to `tasklist/todo`, and `run` again. Or 66 | `python runforrest.py --do_raise infile outfile` them manually, and 67 | observe the error first hand! 68 | 69 | Yes, it's simple. Stupid simple even, you might say. But it is 70 | debuggable. It doesn't start a web server. It doesn't set up fancy 71 | kernels and messaging systems. It doesn't fork three ways and choke 72 | on its own memory consumption. It's just one simple script. 73 | 74 | Then again, maybe this won't run so well on more than a couple of 75 | computers, and it probably still contains bugs and stuff. 76 | 77 | Finally, writing this is not my main job, and I won't be able to 78 | answer and fix every pull request at enterprise speed. Please be civil 79 | if I can't respond quickly. 80 | 81 | ## More Details: 82 | 83 | When creating a `TaskList`, it is assumed that you create a new, empty 84 | list, and will keep that list around after you are done. By default, 85 | `TaskList` therefore throws an error if the chosen directory already 86 | exists (`exist_ok=False`) and will keep all items after you are done 87 | (`post_clean=False`). 88 | 89 | ```python 90 | # new, empty, unique list: 91 | >>> tasklist = TaskList('new_directory') 92 | ``` 93 | 94 | If you are experimenting, and will create and re-create a `TaskList` 95 | over and over, set `exist_ok=True`. This will automatically delete any 96 | tasks in the directory when you instantiate your `TaskList` 97 | (`pre_clean=True`): 98 | 99 | ```python 100 | >>> # new, empty, pre-existing list: 101 | >>> tasklist = TaskList('existing_directory', exist_ok=True) 102 | ``` 103 | 104 | If you want to add to a pre-existing `TaskList`, set 105 | `pre_clean=False`: 106 | 107 | ```python 108 | >>> # existing list: 109 | >>> tasklist = TaskList('existing_directory', exist_ok=True, pre_clean=False) 110 | ``` 111 | 112 | If your computer crashed, and you want to continue where you last ran, 113 | set `noschedule_if_exist=True`. This way, all your `schedule`s will be 114 | skipped, and the existing `TaskList` will `run` all remaining TODO 115 | items: 116 | 117 | ```python 118 | tasklist = TaskList('tasklist', noschedule_if_exist=True) 119 | 120 | for item in long_list_of_stuff: 121 | task = defer(prepare, item) 122 | task = defer(dostuff, task[1], task.thing) 123 | tasklist.schedule(task) # will not schedule! 124 | 125 | for task in tasklist.run(nprocesses=4): # will run remaining TODOs 126 | result = task.returnvalue # or task.errorvalue 127 | ``` 128 | 129 | If your tasks are noisy, and litter your terminal with status messages, 130 | you can supply the `TaskList` with a `logfile`. If given, all output, 131 | and some diagnostic information, will be saved to the logfile instead of 132 | the terminal. 133 | 134 | ### Tasks 135 | 136 | Now you are ready to add tasks. A task is any deferred return value 137 | from a function or method call, or plain data: 138 | 139 | ```python 140 | >>> task = defer(numpy.zeros, 5) 141 | >>> task = defer(dict, alpha='a', beta='b') 142 | >>> task = defer(lambda x: x, 42) 143 | >>> task = defer(42) # behaves like the previous line 144 | ``` 145 | 146 | Tasks can also be used as arguments to deferred function calls: 147 | 148 | ```python 149 | >>> task1 = defer(dict, value=42) 150 | >>> task2 = dever(lambda x: x['value'], task1) 151 | ``` 152 | 153 | You can test-run your tasks using `evaluate`: 154 | 155 | ```python 156 | >>> evaluate(task1) 157 | {'value': 42} 158 | >>> evaluate(task2) 159 | 42 160 | ``` 161 | 162 | Tasks can also access attributes or indexes of arguments of tasks. 163 | Each task attribute or task index is itself a task, that can be 164 | indexed or attributed further: 165 | 166 | ```python 167 | >>> task1 = defer(dict, value=42) 168 | >>> task2 = defer(lambda x: x, task1['value']) 169 | >>> evaluate(task2) 170 | 42 171 | >>> task1 = defer({'value': 42}) 172 | >>> task2 = task1['value'] # also returns a Task! 173 | >>> evaluate(task2) 174 | 42 175 | >>> task1 = defer(list, [23, 42]) 176 | >>> task2 = defer(lambda x: x, task1[1]) # or task2 = task1[1] 177 | >>> evaluate(task2) 178 | 42 179 | >>> task1 = defer(Exception, 'an error message') 180 | >>> task2 = defer(lambda x: x, task1.args) # or task2 = task1.args 181 | >>> evaluate(task2) 182 | ('an error message',) 183 | >>> task2 = defer(lambda x: x, task1.args[0]) # or task2 = task1.args[0] 184 | 'an error message' 185 | ``` 186 | 187 | This way, you can build arbitrary, deep trees of functions and methods 188 | that process your data. 189 | 190 | ### Scheduling and Running 191 | 192 | When you finally have a task that evaluates to 193 | the result you want, you can schedule it on a `TaskList`: 194 | 195 | ```python 196 | >>> tasklist.schedule(task2) 197 | ``` 198 | 199 | If you want, you can attach some metadata that you can retrieve later. 200 | This can be very useful if you are running many many tasks, and want 201 | to sort and organize them later: 202 | 203 | ```python 204 | >>> tasklist.schedule(task2, metadata={'date': date.date()}) 205 | ``` 206 | 207 | Once you have scheduled all the tasks you want, you can run them: 208 | 209 | ```python 210 | >>> for task in tasklist.run(nprocesses=10): 211 | >>> do_something_with(task) 212 | ``` 213 | 214 | This will run each of your tasks in its own process, with ten 215 | processes active at any time. In its default, this will yield every 216 | finished task, with either the return value in `task.returnvalue`, or 217 | the error in `task.errorvalue` if there was an error, and the 218 | aforementioned metadata in `task.metadata`. Additionally, the task 219 | run time is saved in `task.runtime`. 220 | 221 | If you want to get more feedback for failing tasks, you can run them 222 | with `print_errors=True`, which will print the full stack trace of 223 | every error the moment it occurs. 224 | 225 | Sometimes, `dill` won't catch some local functions or globals, and 226 | your tasks will fail. In that case, set `save_session=True` and try 227 | again. 228 | 229 | Sometimes, tasks are unreliable, and need to be stopped before they 230 | bring your computer to a halt. You can tell RunForrest to kill tasks if 231 | they take longer than a set amount of seconds, by adding the `autokill` 232 | argument: 233 | 234 | ```python 235 | >>> for task in tasklist.run(autokill=300): # kill after five minutes 236 | >>> ... 237 | ``` 238 | 239 | Note that `autokill` will `SIGKILL` the whole process group, and will 240 | not give the processes a chance to react or clean up after themselves. 241 | This is the only way to reliably kill stuck processes, and all their 242 | threads and child-processes they might have spawned. 243 | 244 | ### Accessing Tasks 245 | 246 | At any time, you can inspect all currently-scheduled tasks with 247 | 248 | ```python 249 | >>> for task in tasklist.todo_tasks(): 250 | >>> do_something_with(task) 251 | ``` 252 | 253 | or, accordingly for done and failed tasks: 254 | 255 | ```python 256 | >>> for task in tasklist.done_tasks(): 257 | >>> do_something_with(task) 258 | >>> for task in tasklist.fail_tasks(): 259 | >>> do_something_with(task) 260 | ``` 261 | 262 | These three task lists correspond to `*.pkl` files in the 263 | `{tasklist}/todo`, `{tasklist}/done`, and `{tasklist}/fail` directory, 264 | respectively. For example, you can manually re-schedule failed tasks 265 | by moving them from the `{tasklist}/fail` to `{tasklist}/todo`. 266 | 267 | ### Cleaning 268 | 269 | If you want to get rid of tasks, you can use 270 | 271 | ```python 272 | >>> tasklist.clean() 273 | ``` 274 | 275 | You can decide manually if you want to remove only selected tasks with 276 | `clean_todo=True`, `clean_done=True`, and `clean_fail=True`. 277 | 278 | ### Running Tasks Manually 279 | 280 | If one of your tasks is failing, and you can't figure out why, it is 281 | useful to manually run just a single task. For this, you can use 282 | `tasklist` as an executable library: 283 | 284 | ```bash 285 | $ python -m runforrest path/to/task.pkl path/to/result.pkl -r 286 | ``` 287 | 288 | where `-r` raises any error; Alternatively, `-p` would print them just 289 | like `run` does with `print_errors=True`. If needed, you can load a 290 | session file with `-s path/to/session.pkl`. 291 | 292 | ## FAQ: 293 | 294 | - *Code that calls Numpy crashes in `pthread_create`*: By default, 295 | Numpy creates a large number of threads. If you run many Numpies in 296 | parallel, this can exhaust your thread limit. You can either raise 297 | your thread limit 298 | (`resources.setrlimit(resources.RLIMIT_NPROC, ...)`) or force Numpy 299 | to use only one thread per process 300 | (`os.putenv('OMP_NUM_THREADS', '1')`). 301 | 302 | - *Tasks fail because `"name '' is not defined"`*: Sometimes, 303 | serialization can fail if `run` is called from a 304 | `if __name__ == "__main__"` block. Set `save_session=True` in run 305 | to fix this problem (for a small performance overhead). 306 | -------------------------------------------------------------------------------- /runforrest.py: -------------------------------------------------------------------------------- 1 | from uuid import uuid4 as uuid 2 | from pathlib import Path 3 | from subprocess import Popen, STDOUT, PIPE 4 | import sys 5 | import os 6 | import signal 7 | from argparse import ArgumentParser 8 | import dill 9 | import time 10 | 11 | def _identity(thing): 12 | """Just a helper.""" 13 | return thing 14 | 15 | def defer(fun, *args, **kwargs): 16 | """Wrap a function or data for execution. 17 | 18 | Returns a `Task` without running the function. This `Task` can 19 | be used as an argument for other deferred functions to build a 20 | call graph. The call graph can then be executed by an `Executor`. 21 | 22 | `fun` can also be non-callable data, in which case the resulting 23 | `Task` will evaluate to that data and function arguments are 24 | ignored. This can be useful to defer later attribute accesses. 25 | 26 | Additionally, you can access attributes and indexes of the 27 | `Task`. 28 | 29 | """ 30 | if not callable(fun) and not isinstance(fun, Task): 31 | # return non-functions varbatim 32 | return Task(_identity, [fun], {}) 33 | else: 34 | return Task(fun, args, kwargs) 35 | 36 | 37 | class Task: 38 | """A proxy for a function result. 39 | 40 | Accessing attributes, indexing, and calling returns more 41 | `Tasks`. 42 | 43 | """ 44 | 45 | def __init__(self, fun, args, kwargs): 46 | self._fun = fun 47 | self._args = args 48 | self._kwargs = kwargs 49 | self._id = str(id(self)) 50 | 51 | def __eq__(self, other): 52 | return self._id == other._id and self._args == other._args and self._fun == other._fun 53 | 54 | def __getattr__(self, name): 55 | if name in ['__getstate__', '_id']: 56 | raise AttributeError() 57 | return TaskAttribute(self, name) 58 | 59 | def __getitem__(self, key): 60 | return TaskItem(self, key) 61 | 62 | 63 | class PartOfTask(): 64 | """A proxy for a part of a Task. 65 | 66 | Tasks from accessing attributes, indexing, or calling a 67 | `Task`. Each `PartOfTask` has an `_id` that is shared for all 68 | equivalent attribute/index/call accesses. 69 | 70 | """ 71 | 72 | def __init__(self, parent, index): 73 | self._parent = parent 74 | self._index = index 75 | self._id = parent._id + str(index) 76 | 77 | def __eq__(self, other): 78 | return self._id == other._id and self._parent == other._parent 79 | 80 | def __getattr__(self, name): 81 | if name in ['__getstate__', '_id']: 82 | raise AttributeError() 83 | return TaskAttribute(self, name) 84 | 85 | def __getitem__(self, key): 86 | return TaskItem(self, key) 87 | 88 | 89 | class TaskAttribute(PartOfTask): 90 | pass 91 | 92 | 93 | class TaskItem(PartOfTask): 94 | pass 95 | 96 | 97 | class TaskList: 98 | """Schedule tasks and run them in many processes. 99 | 100 | A `TaskList` schedules `Tasks`, and then executes them on several 101 | processes in parallel. For each `Task`, it walks the call chain, 102 | and executes all the code necessary to calculate the return 103 | values. The `TaskList` takes great pride in not evaluating `Tasks` 104 | more often than necessary, even if several `PartOfTasks` lead to 105 | the same original `Task`. 106 | 107 | By default, `schedule` dumps each `Task` in the directory 108 | `{directory}/todo`. Once the `Task` has been executed, it is 109 | transferred to either `{directory}/done` or `{directory}/failed`, 110 | depending on whether it raised errors or not. 111 | 112 | The `TaskList` delegates all the actual running of code to 113 | `evaluate`, which is called by invoking this very script as a 114 | command line program. The `TaskList` merely makes sure that a 115 | fixed number of processes are running at all times. 116 | 117 | """ 118 | 119 | def __init__(self, directory, exist_ok=False, pre_clean=True, 120 | post_clean=False, logfile=None, noschedule_if_exist=False): 121 | self._directory = Path(directory) 122 | self._post_clean = post_clean 123 | self._logfile = Path(logfile) if logfile else None 124 | self._noschedule = False 125 | 126 | if self._directory.exists(): 127 | if noschedule_if_exist: 128 | self._noschedule = True 129 | pre_clean = False 130 | elif not exist_ok: 131 | raise RuntimeError(f'TaskList directory {str(self._directory)} already exists') 132 | if pre_clean: 133 | self.clean() 134 | 135 | for dir in [self._directory, self._directory / 'todo', 136 | self._directory / 'done', self._directory / 'fail']: 137 | if not dir.exists(): 138 | dir.mkdir() 139 | 140 | self._processes = {} 141 | 142 | def __del__(self): 143 | if self._post_clean: 144 | self.clean() 145 | 146 | def schedule(self, task, metadata=None): 147 | """Schedule a task for later execution. 148 | 149 | The task is saved to the `{directory}/todo` directory. Use 150 | `run` to execute all the tasks in the `{directory}/todo} 151 | directory. 152 | 153 | If you want, you can attach metadata to the task, which you 154 | can retrieve as `task.metadata` after the task has been run. 155 | 156 | """ 157 | 158 | if self._noschedule: 159 | return 160 | 161 | task.errorvalue = None 162 | task.returnvalue = None 163 | task.metadata = metadata 164 | 165 | taskfilename = (str(uuid()) + '.pkl') 166 | with (self._directory / 'todo' / taskfilename).open('wb') as f: 167 | dill.dump(task, f) 168 | self._log('schedule', taskfilename) 169 | 170 | def run(self, nprocesses=4, print_errors=False, save_session=False, autokill=None): 171 | """Execute all tasks in the `{directory}/todo}` directory. 172 | 173 | All tasks are executed in their own processes, and `run` makes 174 | sure that no more than `nprocesses` are active at any time. 175 | 176 | If `print_errors=True`, processes will print full stack traces 177 | of failing tasks. Since these errors happen on another 178 | process, this will not be caught by the debugger, and will not 179 | stop the `run`. 180 | 181 | Use `save_session` to recreate all current globals in each 182 | process. 183 | 184 | """ 185 | 186 | if save_session: 187 | dill.dump_session(self._directory / 'session.pkl') 188 | 189 | class TaskIterator: 190 | def __init__(self, parent, todos, save_session): 191 | self.parent = parent 192 | self.todos = todos 193 | self.save_session = save_session 194 | 195 | def __iter__(self): 196 | for todo in self.todos: 197 | yield from self.parent._finish_tasks(nprocesses, autokill=autokill) 198 | self.parent._start_task(todo.name, print_errors, save_session) 199 | # wait for running jobs to finish: 200 | yield from self.parent._finish_tasks(1, autokill=autokill) 201 | 202 | def __len__(self): 203 | return len(self.todos) 204 | 205 | return TaskIterator(self, list((self._directory / 'todo').iterdir()), save_session) 206 | 207 | def _start_task(self, taskfilename, print_errors, save_session): 208 | """Start a new process, and append to self._processes.""" 209 | args = ['python', '-m', 'runforrest', 210 | self._directory / 'todo' / taskfilename, 211 | self._directory / 'done' / taskfilename] 212 | if print_errors: 213 | args += ['-p'] 214 | if save_session: 215 | args += ['-s', self._directory / 'session.pkl'] 216 | kwargs = dict(start_new_session=True, cwd=os.getcwd()) 217 | if self._logfile: 218 | kwargs['stdout'] = PIPE 219 | kwargs['stderr'] = STDOUT 220 | self._processes[taskfilename] = Popen(args, **kwargs) 221 | self._processes[taskfilename].start_time = time.perf_counter() 222 | self._log('start', taskfilename) 223 | 224 | def _finish_tasks(self, nprocesses, autokill): 225 | """Wait while `nprocesses` are running and return finished tasks.""" 226 | while len(self._processes) >= nprocesses: 227 | for file, proc in list(self._processes.items()): 228 | if proc.poll() is not None: 229 | task = self._retrieve_task(file) 230 | try: 231 | stdout, _ = proc.communicate(timeout=10) 232 | self._log('done' if task.errorvalue is None else 'fail', file) 233 | if stdout: 234 | self._log(stdout, file) 235 | yield task 236 | except subprocess.TimeoutExpired as err: 237 | # something is wrong. Kill the process and move on. 238 | process_group = os.getpgid(proc.pid) 239 | os.killpg(process_group, signal.SIGKILL) 240 | self._log('lost contact', file) 241 | finally: 242 | del self._processes[file] 243 | elif autokill and time.perf_counter() - proc.start_time > autokill: 244 | try: 245 | # kill the whole process group. 246 | # This is a mean thing to do, and might leave dangling 247 | # intermedite files. But at this point, the program was 248 | # provably not able to terminate on its own, and drastic 249 | # measures are our last resort. 250 | process_group = os.getpgid(proc.pid) 251 | os.killpg(process_group, signal.SIGKILL) 252 | self._log('autokilled', file) 253 | # sometimes, even the above does not work. In this case, 254 | # we will leak the process, but continue anyway: 255 | del self._processes[file] 256 | except Exception as err: 257 | self._log(err.message, file) 258 | else: 259 | time.sleep(0.1) 260 | 261 | def _retrieve_task(self, taskfilename): 262 | """Load task, and sort into `{directory}/done` or `{directory}/fail`.""" 263 | try: 264 | with (self._directory / 'done' / taskfilename).open('rb') as f: 265 | task = dill.load(f) 266 | except Exception as error: 267 | with (self._directory / 'todo' / taskfilename).open('rb') as f: 268 | task = dill.load(f) 269 | task.returnvalue = None 270 | task.errorvalue = error 271 | with (self._directory / 'done' / taskfilename).open('wb') as f: 272 | dill.dump(task, f) 273 | 274 | (self._directory / 'todo' / taskfilename).unlink() 275 | 276 | if task.errorvalue is not None: 277 | (self._directory / 'done' / taskfilename).rename(self._directory / 'fail' / taskfilename) 278 | 279 | return task 280 | 281 | def _log(self, message, taskfilename): 282 | if not self._logfile: 283 | return 284 | with self._logfile.open('a') as f: 285 | f.write(f"{time.strftime('%Y-%m-%dT%H:%M:%S')} {taskfilename} {message}\n") 286 | 287 | def todo_tasks(self): 288 | """Yield all tasks in `{directory}/todo`.""" 289 | for todo in (self._directory / 'todo').iterdir(): 290 | with todo.open('rb') as f: 291 | yield dill.load(f) 292 | 293 | def done_tasks(self): 294 | """Yield all tasks in `{directory}/done`.""" 295 | for done in (self._directory / 'done').iterdir(): 296 | with done.open('rb') as f: 297 | try: # safeguard against broken tasks: 298 | yield dill.load(f) 299 | except EOFError as err: 300 | print(f'skipping {done.name} ({err})') 301 | 302 | def fail_tasks(self): 303 | """Yield all tasks in `{directory}/fail`.""" 304 | for fail in (self._directory / 'fail').iterdir(): 305 | with fail.open('rb') as f: 306 | yield dill.load(f) 307 | 308 | def clean(self, clean_todo=True, clean_done=True, clean_fail=True): 309 | """Remove `{directory}` and all todo/done/fail tasks.""" 310 | def remove(dir): 311 | if dir.exists(): 312 | for f in dir.iterdir(): 313 | f.unlink() 314 | dir.rmdir() 315 | if clean_todo: 316 | remove(self._directory / 'todo') 317 | if clean_fail: 318 | remove(self._directory / 'fail') 319 | if clean_done: 320 | remove(self._directory / 'done') 321 | if clean_todo and clean_fail and clean_done: 322 | if (self._directory / 'session.pkl').exists(): 323 | (self._directory / 'session.pkl').unlink() 324 | remove(self._directory) 325 | 326 | 327 | def main(): 328 | parser = ArgumentParser(description="Run an enqueued function") 329 | parser.add_argument('infile', type=Path, help='contains the enqueued function') 330 | parser.add_argument('outfile', type=Path, help='contains the evaluation results') 331 | parser.add_argument('-s', '--sessionfile', type=Path, action='store', default=None) 332 | parser.add_argument('-p', '--do_print', action='store_true', default=False) 333 | parser.add_argument('-r', '--do_raise', action='store_true', default=False) 334 | 335 | args = parser.parse_args() 336 | run_task(args.infile, args.outfile, args.sessionfile, args.do_print, args.do_raise) 337 | 338 | 339 | def run_task(infile, outfile, sessionfile, do_print, do_raise): 340 | """Execute `infile` and produce `outfile`. 341 | 342 | If `sessionfile` is given, load session from that file. 343 | 344 | Set `do_print` or `do_raise` to `True` if errors should be printed or 345 | raised. 346 | 347 | """ 348 | 349 | if sessionfile: 350 | dill.load_session(Path(sessionfile)) 351 | 352 | with infile.open('rb') as f: 353 | task = dill.load(f) 354 | 355 | try: 356 | start_time = time.perf_counter() 357 | task.returnvalue = evaluate(task) 358 | task.errorvalue = None 359 | except Exception as err: 360 | task.errorvalue = err 361 | task.returnvalue = None 362 | finally: 363 | task.runtime = time.perf_counter() - start_time 364 | with outfile.open('wb') as f: 365 | dill.dump(task, f) 366 | 367 | if task.errorvalue is not None and do_raise: 368 | raise task.errorvalue 369 | 370 | if task.errorvalue is not None and do_print: 371 | print(f'Error in {infile.name}: {task.errorvalue.__repr__()}') 372 | 373 | sys.exit(0 if task.errorvalue is None else -1) 374 | 375 | 376 | def evaluate(task, known_results=None): 377 | """Execute a `task` and calculate its return value. 378 | 379 | `evaluate` walks the call chain to the `task`, and executes all 380 | the code necessary to calculate the return values. No `task` are 381 | executed more than once, even if several `PartOfTasks` lead to 382 | the same original `Task`. 383 | 384 | This is a recursive function that passes its state in 385 | `known_results`, where return values of all executed `Tasks` are 386 | stored. 387 | 388 | """ 389 | 390 | # because pickling breaks isinstance(task, Task) 391 | if not 'Task' in task.__class__.__name__: 392 | return task 393 | 394 | if known_results is None: 395 | known_results = {} 396 | 397 | if task._id not in known_results: 398 | if task.__class__.__name__ in ['TaskItem', 'TaskAttribute']: 399 | returnvalue = evaluate(task._parent, known_results) 400 | if task.__class__.__name__ == 'TaskItem': 401 | known_results[task._id] = returnvalue[task._index] 402 | elif task.__class__.__name__ == 'TaskAttribute': 403 | known_results[task._id] = getattr(returnvalue, task._index) 404 | else: 405 | raise TypeError(f'unknown Task {type(task)}') 406 | else: # is Task 407 | args = [evaluate(arg, known_results) for arg in task._args] 408 | kwargs = {k: evaluate(v, known_results) for k, v in task._kwargs.items()} 409 | returnvalue = task._fun(*args, **kwargs) 410 | known_results[task._id] = returnvalue 411 | 412 | return known_results[task._id] 413 | 414 | 415 | if __name__ == '__main__': 416 | main() 417 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name='runforrest', 5 | version='0.4.3', 6 | description='Batch and run your code in parallel. Simply.', 7 | author='Bastian Bechtold', 8 | author_email='basti@bastibe.de', 9 | url='https://github.com/bastibe/RunForrest', 10 | license='BSD 3-clause', 11 | py_modules=['runforrest'], 12 | install_requires=['dill'], 13 | python_requires='>=3.4' 14 | ) 15 | -------------------------------------------------------------------------------- /test_runforrest.py: -------------------------------------------------------------------------------- 1 | import runforrest 2 | import pathlib 3 | 4 | def identity(n): 5 | return n 6 | 7 | 8 | def test_run(): 9 | tasklist = runforrest.TaskList('tmp', post_clean=True) 10 | task = runforrest.defer(identity, 42) 11 | tasklist.schedule(task) 12 | tasks = list(tasklist.run(nprocesses=4)) 13 | assert tasks[0].returnvalue == 42 14 | 15 | 16 | def test_data_task(): 17 | tasklist = runforrest.TaskList('tmp', post_clean=True) 18 | task = runforrest.defer(42) 19 | tasklist.schedule(task) 20 | tasks = list(tasklist.run(nprocesses=4)) 21 | assert tasks[0].returnvalue == 42 22 | 23 | 24 | def test_nested_run(): 25 | tasklist = runforrest.TaskList('tmp', post_clean=True) 26 | task = runforrest.defer(identity, 42) 27 | task = runforrest.defer(identity, task) 28 | tasklist.schedule(task) 29 | tasks = list(tasklist.run(nprocesses=4)) 30 | assert tasks[0].returnvalue == 42 31 | 32 | 33 | def test_multiple_runs(howmany=20): 34 | tasklist = runforrest.TaskList('tmp', post_clean=True) 35 | for v in range(howmany): 36 | task = runforrest.defer(identity, v) 37 | tasklist.schedule(task) 38 | tasks = list(tasklist.run(nprocesses=10)) 39 | assert len(tasks) == howmany 40 | for r, v in zip(sorted(t.returnvalue for t in tasks), range(howmany)): 41 | assert r == v 42 | 43 | 44 | def test_multiple_nested_runs(howmany=20): 45 | tasklist = runforrest.TaskList('tmp', post_clean=True) 46 | for v in range(howmany): 47 | task = runforrest.defer(identity, v) 48 | task = runforrest.defer(identity, task) 49 | tasklist.schedule(task) 50 | tasks = list(tasklist.run(nprocesses=10)) 51 | assert len(tasks) == howmany 52 | for r, v in zip(sorted(t.returnvalue for t in tasks), range(howmany)): 53 | assert r == v 54 | 55 | 56 | def test_task_accessor(): 57 | tasklist = runforrest.TaskList('tmp', post_clean=True) 58 | # send something that has an attribute: 59 | task = runforrest.defer(identity, Exception(42)) 60 | # retrieve the attribute: 61 | task = runforrest.defer(identity, task.args) 62 | tasklist.schedule(task) 63 | tasks = list(tasklist.run(nprocesses=1)) 64 | assert tasks[0].returnvalue == (42,) 65 | 66 | 67 | def test_task_indexing(): 68 | tasklist = runforrest.TaskList('tmp', post_clean=True) 69 | # send something that can be indexed: 70 | task = runforrest.defer(identity, [42]) 71 | # retrieve at index: 72 | task = runforrest.defer(identity, task[0]) 73 | tasklist.schedule(task) 74 | tasks = list(tasklist.run(nprocesses=1)) 75 | assert tasks[0].returnvalue == 42 76 | 77 | 78 | def test_todo_and_done_task_access(): 79 | tasklist = runforrest.TaskList('tmp', post_clean=True) 80 | task = runforrest.defer(identity, 42) 81 | tasklist.schedule(task) 82 | todo = list(tasklist.todo_tasks()) 83 | list(tasklist.run(nprocesses=1)) 84 | done = list(tasklist.done_tasks()) 85 | assert task == done[0] == todo[0] 86 | 87 | 88 | def test_metadata(): 89 | tasklist = runforrest.TaskList('tmp', post_clean=True) 90 | task = runforrest.defer(identity, 42) 91 | tasklist.schedule(task, metadata='the truth') 92 | list(tasklist.run(nprocesses=1)) 93 | done = list(tasklist.done_tasks()) 94 | assert done[0].metadata == 'the truth' 95 | 96 | 97 | def crash(): 98 | raise Exception('TESTING') 99 | 100 | 101 | def test_failing_task(): 102 | tasklist = runforrest.TaskList('tmp', post_clean=True) 103 | task = runforrest.defer(crash) 104 | tasklist.schedule(task) 105 | tasks = list(tasklist.run(nprocesses=1)) 106 | assert len(tasks) == 1 107 | assert tasks[0] == task 108 | fail = list(tasklist.fail_tasks()) 109 | assert len(fail) == 1 110 | assert fail[0] == task 111 | assert fail[0].errorvalue.args == ('TESTING',) 112 | done = list(tasklist.done_tasks()) 113 | assert len(done) == 0 114 | 115 | 116 | def test_post_clean_true(): 117 | tasklist = runforrest.TaskList('tmp', post_clean=True) 118 | task = runforrest.defer(crash) 119 | tasklist.schedule(task) 120 | list(tasklist.run(nprocesses=1)) 121 | del tasklist 122 | assert not pathlib.Path('tmp').exists() 123 | assert not pathlib.Path('tmp/todo').exists() 124 | assert not pathlib.Path('tmp/done').exists() 125 | assert not pathlib.Path('tmp/fail').exists() 126 | 127 | 128 | def test_post_clean_false(): 129 | tasklist = runforrest.TaskList('tmp', post_clean=False) 130 | task = runforrest.defer(crash) 131 | tasklist.schedule(task) 132 | list(tasklist.run(nprocesses=1)) 133 | del tasklist 134 | for path in ['tmp/todo', 'tmp/done', 'tmp/fail', 'tmp']: 135 | path = pathlib.Path(path) 136 | assert path.exists() 137 | for file in path.iterdir(): 138 | file.unlink() 139 | path.rmdir() 140 | 141 | 142 | def test_noschedule(): 143 | tasklist = runforrest.TaskList('tmp') 144 | task = runforrest.defer(identity, 42) 145 | tasklist.schedule(task) 146 | # do not run, but reopen: 147 | tasklist = runforrest.TaskList('tmp', noschedule_if_exist=True, post_clean=True) 148 | task = runforrest.defer(identity, 42) 149 | tasklist.schedule(task) # should now be a no-op 150 | tasks = list(tasklist.run(nprocesses=4)) 151 | assert len(tasks) == 1 152 | assert tasks[0].returnvalue == 42 153 | 154 | 155 | def test_logging(): 156 | logfile = pathlib.Path('tmp.log') 157 | if logfile.exists(): 158 | logfile.unlink() 159 | tasklist = runforrest.TaskList('tmp', post_clean=True, logfile=logfile) 160 | task = runforrest.defer(identity, 42) 161 | tasklist.schedule(task) 162 | tasks = list(tasklist.run(nprocesses=4)) 163 | with logfile.open() as f: 164 | lines = list(f) 165 | assert len(lines) == 3 166 | assert 'schedule' in lines[0] 167 | assert 'start' in lines[1] 168 | assert 'done' in lines[2] 169 | logfile.unlink() 170 | --------------------------------------------------------------------------------