├── README.md ├── more_multi_progress_bar.png ├── multi_progress.py ├── multi_progress_bar.png └── single_progress_bar.png /README.md: -------------------------------------------------------------------------------- 1 | Multiprocessing and multiple progress bars 2 | ------------------------------------------ 3 | 4 | Well this was a success. 5 | 6 | I've got multiple processes running at once in [parallel][my-multi] 7 | 8 | [my-multi]: http://aaren.github.com/notes/2012/04/embarassingly_parallel_python 9 | 10 | What I want to do here is have a progress bar for each process and 11 | have these displayed nicely on the screen. 12 | 13 | There is already a [decent module][progressbar] for printing 14 | progress bars, so we're going to try and use that. 15 | 16 | Writing stuff to specific places on the screen is a job for 17 | [curses][], which is messy to use. Luckily I came across 18 | [blessings][], which is an excellent clean wrapper around curses. 19 | 20 | I've used blessings version 1.5 and progressbar version 2.3. 21 | 22 | [progressbar]: https://pypi.python.org/pypi/progressbar/2.3-dev 23 | [curses]: http://docs.python.org/2/howto/curses.html 24 | [blessings]: https://pypi.python.org/pypi/blessings/ 25 | 26 | The main class in progressbar, ProgressBar, has an instantiation 27 | argument `fd=sys.stderr` that is an object with a `write(string)` 28 | method. This is how progressbar actually writes out to the screen. 29 | By default, this happens by `sys.stderr.write('[----progress...]')`, 30 | but we can supply our own writer. 31 | 32 | Blessings works something like this (the [documentation][blessings] 33 | is excellent btw): 34 | 35 | ```python 36 | from blessings import Terminal 37 | 38 | term = Terminal() 39 | 40 | location = (0, 10) 41 | text = 'blessings!' 42 | print term.location(*location), text 43 | 44 | # alternately, 45 | with term.location(*self.location): 46 | print text 47 | ``` 48 | 49 | Progressbar works something like this: 50 | 51 | ```python 52 | import time 53 | 54 | from progressbar import ProgressBar 55 | 56 | pbar.start() 57 | for i in range(100): 58 | # mimic doing some stuff 59 | time.sleep(0.01) 60 | pbar.update(i) 61 | pbar.finish() 62 | ``` 63 | 64 | We need to make something to connect progressbar and blessings. 65 | Create an object that can write like this: 66 | 67 | ```python 68 | class Writer(object): 69 | """Create an object with a write method that writes to a 70 | specific place on the screen, defined at instantiation. 71 | 72 | This is the glue between blessings and progressbar. 73 | """ 74 | def __init__(self, location): 75 | """ 76 | Input: location - tuple of ints (x, y), the position 77 | of the bar in the terminal 78 | """ 79 | self.location = location 80 | 81 | def write(self, string): 82 | with term.location(*self.location): 83 | print(string) 84 | ``` 85 | 86 | Then we can put our progress bar wherever we want by feeding our 87 | writer object to progressbar: 88 | 89 | ```python 90 | def test_function(location): 91 | writer = Writer(location) 92 | pbar = ProgressBar(fd=writer) 93 | pbar.start() 94 | for i in range(100): 95 | # mimic doing some stuff 96 | time.sleep(0.01) 97 | pbar.update(i) 98 | pbar.finish() 99 | 100 | x_pos = 0 # zero characters from left 101 | y_pos = 10 # ten characters from top 102 | location = (x_pos, y_pos) 103 | test_function(location) 104 | ``` 105 | 106 | ![Arbitrarily positioned progress bar](https://raw.github.com/aaren/multi_progress/master/single_progress_bar.png) 107 | 108 | #### Multiprocessing #### 109 | 110 | Now that we can put a progressbar where we choose it is fairly 111 | trivial to extend this to multiprocessing. 112 | 113 | Basic [multiprocessing][] usage, mapping a function `our_function` 114 | onto a list of arguments `arg_list`: 115 | 116 | ```python 117 | from multiprocessing import Pool 118 | 119 | p = Pool() 120 | p.map(our_function, arg_list) 121 | p.close() 122 | ``` 123 | 124 | [multiprocessing]: http://docs.python.org/2/library/multiprocessing.html 125 | 126 | In our case the function is `test_function` and the list of 127 | arguments is a list of locations. For example, to have a progress 128 | bar at the start of the line on the 2nd, 7th and 8th lines: 129 | 130 | ```python 131 | locations = [(0, 1), (0, 6), (0, 7)] 132 | p = Pool() 133 | p.map(test_function, locations) 134 | p.close() 135 | ``` 136 | 137 | ![Parallel progress bars](https://raw.github.com/aaren/multi_progress/master/multi_progress_bar.png) 138 | 139 | I've only got two active progress bars here because I've only got a 140 | two core processor. `Pool()` defaults to making a number of worker 141 | processes equal to the number of processors. Here is the same code 142 | run on more cores with a load more locations: 143 | 144 | ![Lots of parallel progress bars](https://raw.github.com/aaren/multi_progress/master/more_multi_progress_bar.png) 145 | 146 | #### Fullscreen output #### 147 | 148 | You might notice that the examples above mess the screen up a bit. 149 | This is because blessings, unlike curses, does not force a 150 | fullscreen view that disappears on completion. We can tell blessings 151 | to have this behaviour like this: 152 | 153 | ```python 154 | with term.fullscreen(): 155 | do_the_stuff_above() 156 | ``` 157 | 158 | I've made an [script][demo-script] that demonstrates all of the above. 159 | 160 | [demo-script]: https://github.com/aaren/multi_progress/blob/master/multi_progress.py 161 | 162 | And there you go, multiple independent progress bars implemented in 163 | Python with not much hassle at all. This took me about 4 hours, blog 164 | post included, thanks in large part to how easy [blessings][] makes 165 | curses. 166 | -------------------------------------------------------------------------------- /more_multi_progress_bar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aaren/multi_progress/85b2ed4ee91c6f439859203ba6feb7dae243c586/more_multi_progress_bar.png -------------------------------------------------------------------------------- /multi_progress.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | 3 | import sys 4 | import time 5 | import random 6 | from multiprocessing import Pool 7 | 8 | from blessings import Terminal 9 | from progressbar import ProgressBar 10 | 11 | term = Terminal() 12 | 13 | 14 | class Writer(object): 15 | """Create an object with a write method that writes to a 16 | specific place on the screen, defined at instantiation. 17 | 18 | This is the glue between blessings and progressbar. 19 | """ 20 | def __init__(self, location): 21 | """ 22 | Input: location - tuple of ints (x, y), the position 23 | of the bar in the terminal 24 | """ 25 | self.location = location 26 | 27 | def write(self, string): 28 | with term.location(*self.location): 29 | print(string) 30 | 31 | 32 | def test(location): 33 | """Test with a single bar. 34 | 35 | Input: location - tuple (x, y) defining the position on the 36 | screen of the progress bar 37 | """ 38 | # fd is an object that has a .write() method 39 | writer = Writer(location) 40 | pbar = ProgressBar(fd=writer) 41 | # progressbar usage 42 | pbar.start() 43 | for i in range(100): 44 | # do stuff 45 | # time taken for process is function of line number 46 | # t_wait = location[1] / 100 47 | # time take is random 48 | t_wait = random.random() / 50 49 | time.sleep(t_wait) 50 | # update calls the write method 51 | pbar.update(i) 52 | 53 | pbar.finish() 54 | 55 | 56 | def test_bars(locations): 57 | """Test with multiple bars. 58 | 59 | Input: locations - a list of location (x, y) tuples 60 | """ 61 | writers = [Writer(loc) for loc in locations] 62 | pbars = [ProgressBar(fd=writer) for writer in writers] 63 | for pbar in pbars: 64 | pbar.start() 65 | 66 | for i in range(100): 67 | time.sleep(0.01) 68 | for pbar in pbars: 69 | pbar.update(i) 70 | 71 | for pbar in pbars: 72 | pbar.finish() 73 | 74 | 75 | def test_parallel(locations): 76 | """Test case with multiprocessing. 77 | 78 | Input: locations - a list of location (x, y) tuples 79 | 80 | Think of locations as a list of jobs. Each job is going 81 | to be processed by the single bar test. 82 | """ 83 | pool = Pool() 84 | pool.map(test, locations) 85 | pool.close() 86 | 87 | 88 | def main(): 89 | if len(sys.argv) == 4 and sys.argv[1] == 'single': 90 | x = int(sys.argv[2]) 91 | y = int(sys.argv[3]) 92 | print("Printing at ({x}, {y})".format(x=x, y=y)) 93 | location = (x, y) 94 | test(location) 95 | elif len(sys.argv) == 4 and sys.argv[1] == 'multi': 96 | first = int(sys.argv[2]) 97 | last = int(sys.argv[3]) 98 | locations = [(0, i) for i in range(first, last)] 99 | test_bars(locations) 100 | elif len(sys.argv) == 4 and sys.argv[1] == 'parallel_single': 101 | line1 = int(sys.argv[2]) 102 | line2 = int(sys.argv[3]) 103 | locations = [(0, line1), (0, line2)] 104 | test_parallel(locations) 105 | elif len(sys.argv) == 4 and sys.argv[1] == 'parallel_multi': 106 | first = int(sys.argv[2]) 107 | last = int(sys.argv[3]) 108 | locations = [(0, i) for i in range(first, last)] 109 | test_parallel(locations) 110 | else: 111 | raise UserWarning 112 | 113 | 114 | def usage(): 115 | """Print help to the screen""" 116 | usage_str = """Usage: 117 | python multi.py single x_pos y_pos 118 | python multi.py multi first_line last_line 119 | python multi.py parallel_single line_one line_two 120 | python multi.py parallel_multi first_line last_line 121 | """ 122 | print(usage_str) 123 | 124 | 125 | if __name__ == '__main__': 126 | try: 127 | with term.fullscreen(): 128 | main() 129 | except UserWarning: 130 | usage() 131 | -------------------------------------------------------------------------------- /multi_progress_bar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aaren/multi_progress/85b2ed4ee91c6f439859203ba6feb7dae243c586/multi_progress_bar.png -------------------------------------------------------------------------------- /single_progress_bar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aaren/multi_progress/85b2ed4ee91c6f439859203ba6feb7dae243c586/single_progress_bar.png --------------------------------------------------------------------------------