├── .gitignore ├── .travis.yml ├── LICENSE ├── MANIFEST ├── README.md ├── __init__.py ├── parallel ├── __init__.py ├── parallel_collections.py └── tests.py ├── requirements.txt └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .bin 3 | .lib 4 | .Python 5 | *venv 6 | dist -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "2.7" 4 | install: "pip install -r requirements.txt" 5 | script: coverage run --source=parallel/parallel_collections.py parallel/tests.py 6 | after_success: 'coveralls' -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Gregory Terzian 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MANIFEST: -------------------------------------------------------------------------------- 1 | # file GENERATED by distutils, do NOT edit 2 | .gitignore 3 | README.md 4 | requirements.txt 5 | setup.py 6 | parallel/__init__.py 7 | parallel/parallel_collections.py 8 | parallel/tests.py 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### Python Parallel Collections 2 | 3 | [![Build Status](https://travis-ci.org/gterzian/Python-Parallel-Collections.svg?branch=master)](https://travis-ci.org/gterzian/Python-Parallel-Collections) 4 | [![Coverage Status](https://coveralls.io/repos/gterzian/Python-Parallel-Collections/badge.svg?branch=master)](https://coveralls.io/r/gterzian/Python-Parallel-Collections?branch=master) 5 | 6 | #### Who said Python was not setup for multicore computing? :smiley_cat: 7 | In this package you'll find a convenient interface to parallel map/reduce/filter style operations, internally using the [Python 2.7 backport](http://pythonhosted.org/futures/#processpoolexecutor-example) of the [concurrent.futures](http://docs.python.org/dev/library/concurrent.futures.html) package. 8 | 9 | If you can define your problem in terms of map/reduce/filter operations, it will run on several parallel Python processes on your machine, taking advantage of multiple cores. 10 | 11 | _Please note that although the below examples are written in interactive style, due to the nature of multiple processes they might not 12 | actually work in the interactive interpreter._ 13 | 14 | #### API changes in 2 15 | 16 | Flatmap is now "properly" implemented, mapping first, then flattening the results. This matches how the function is implemented in most places, [for example in Rust](https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.flat_map). 17 | 18 | #### API changes in 1.2 19 | 20 | Special thanks to https://github.com/kogecoo 21 | 22 | In version 1.2, the behavior of parallel ``filter`` method has been changed to match that of CPython's built-in ``filter``. 23 | The *pretty much backward compatible* changes are as follows: 24 | 25 | * It is now possible to use ``None`` as a predicate for ``filter``. 26 | * Previously, it would raise a ``TypeError``. 27 | * It is now possible to include ``None`` elements in an iterable. 28 | * Previously, ``None`` elements were always filtered out regardless of the predicate. 29 | 30 | These modifications are based on [Python's Reference](https://docs.python.org/3.5/library/functions.html#filter), and [source of CPython filter method](https://github.com/python/cpython/blob/master/Python/bltinmodule.c). 31 | 32 | #### Changes in 1.0 33 | Version 1.0 introduces a massive simplification of the code base. No longer do we operate on concrete 'parallel' data structures, rather we work with just one `ParallelGen` object. This object is essentially a generator with parallel map/filter/reduce methods wrapping around whatever data structure (or generator) you instantiate it with. You instantiate one by passing along an iterable to the `parallel` factory function. Every method call returns a new ParallelGen object, allowing you to chain calls and only evaluate the results when you need them. 34 | 35 | _API changes to note: 36 | There is no distinction anymore between using `parallel` and `lazy_parallel`, just use `parallel` and everything will be lazy._ 37 | 38 | #### Getting Started 39 | ```python 40 | pip install futures 41 | pip install python-parallel-collections 42 | ``` 43 | ```python 44 | from parallel import parallel 45 | ``` 46 | 47 | #### Examples 48 | ```python 49 | >>> def double_iterables(l): 50 | ... for item in l: 51 | ... return map(double, l) 52 | ... 53 | >>> parallel_gen = parallel([[1,2,3],[4,5,6]]) 54 | >>> list(parallel_gen.flatmap(double_iterables)) 55 | [2, 4, 6, 8, 10, 12] 56 | ``` 57 | 58 | Every method call returns a new ParallelGen, instead of changing the current one, with the exception of `reduce` and `foreach`. Also note that you will need to evaluate the generator in order to get the resuts, as is done above through the call to `list`. A call to `reduce` will also return the result. 59 | 60 | The foreach method is equivalent to map but instead of returning a new ParallelGen it operates directly on the 61 | current one and returns `None`. 62 | ```python 63 | >>> list(flat_gen) 64 | [1, 2, 3, 4, 5, 6] 65 | >>> flat_gen.foreach(double) 66 | None 67 | >>> list(flat_gen) 68 | [2, 4, 6, 8, 10, 12] 69 | ``` 70 | 71 | Since every operation (except `foreach` and `reduce`) returns a collection, operations can be chained. 72 | ```python 73 | >>> parallel_gen = parallel([[1,2,3],[4,5,6]]) 74 | >>> list(parallel_gen.flatmap(double_iterables).map(str)) 75 | ['2', '4', '6', '8', '10', '12'] 76 | ``` 77 | 78 | #### On being lazy 79 | The parallel function returns a ParallelGen instance, as will any subsequent call to `map`, `filter` and `flatmatp`. This allows you to chain method calls without evaluating the results on every operation. 80 | Instead, each element in the initial datastructure or generator will be processed throught the entire pipeline of method calls one at the time, without creating intermittent datastructures. This is a great way to save memory when working with large or infinite streams of data. 81 | 82 | _For more on this technique, see the [Python Cookbook](http://shop.oreilly.com/product/0636920027072.do) 3rd 4.13. Creating Data Processing Pipelines._ 83 | 84 | 85 | #### Regarding lambdas and closures 86 | Sadly lambdas, closures and partial functions cannot be passed around multiple processes, so every function that you pass to the higher order methods needs to be defined using the def statement. If you want the operation to carry extra state, use a class with a `__call__` method defined. 87 | 88 | ### Quick examples of map, reduce and filter 89 | 90 | #### Map and FlatMap 91 | 92 | Functions passed to the map method of a list will be passed every element in the sequence and should return one element. Flatmap will first flatten the sequence then apply map to it. 93 | 94 | ```python 95 | >>> def double_iterables(l): 96 | ... for item in l: 97 | ... return map(double, l) 98 | ... 99 | >>> parallel_gen = parallel([[1,2,3],[4,5,6]]) 100 | >>> parallel_gen.flatmap(double_iterables).map(str) 101 | ['2', '4', '6', '8', '10', '12'] 102 | >>> def to_upper(item): 103 | ... return item.upper() 104 | ... 105 | >>> p = parallel('qwerty') 106 | >>> mapped = p.map(to_upper) 107 | >>> ''.join(mapped) 108 | 'QWERTY' 109 | ``` 110 | 111 | #### Reduce 112 | Reduce accepts an optional initializer, which will be passed as the first argument to every call to the function passed as reducer and returned by the method. 113 | ```python 114 | >>> def group_letters(all, letter): 115 | ... all[letter].append(letter) 116 | ... return all 117 | ... 118 | >>> p = parallel(['a', 'a', 'b']) 119 | >>> reduced = p.reduce(group_letters, defaultdict(list)) 120 | {'a': ['a', 'a'], 'b': ['b']} 121 | >>> p = parallel('aab') 122 | >>> p.reduce(group_letters, defaultdict(list)) 123 | {'a': ['a', 'a'], 'b': ['b']} 124 | ``` 125 | 126 | #### Filter 127 | The Filter method should be passed a predicate, which means a function that will return True or False and will be called once for every element in the sequence. 128 | ```python 129 | >>> def is_digit(item): 130 | ... return item.isdigit() 131 | ... 132 | >>> p = parallel(['a','2','3']) 133 | >>> pred = is_digit 134 | >>> filtered = p.filter(pred) 135 | >>> list(filtered) 136 | ['2', '3'] 137 | 138 | >>> def is_digit_dict(item): 139 | ... return item[1].isdigit() 140 | ... 141 | >>> p = parallel(dict(zip(range(3), ['a','2', '3',]))) 142 | {0: 'a', 1: '2', 2: '3'} 143 | >>> pred = is_digit_dict 144 | >>> filtered = p.filter(pred) 145 | >>> filtered 146 | {1: '2', 2: '3'} 147 | >>> p = parallel('a23') 148 | >>> filtered = p.filter(is_digit) 149 | >>>''.join(filtered) 150 | '23' 151 | ``` 152 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gterzian/Python-Parallel-Collections/b1fb228696e954161c89dc4bdc5837d3b3208d13/__init__.py -------------------------------------------------------------------------------- /parallel/__init__.py: -------------------------------------------------------------------------------- 1 | from parallel_collections import parallel, lazy_parallel, ParallelSeq, ParallelList, ParallelDict, ParallelString -------------------------------------------------------------------------------- /parallel/parallel_collections.py: -------------------------------------------------------------------------------- 1 | """The main module. Use as 'from parallel import parallel'.""" 2 | 3 | import multiprocessing 4 | from collections import namedtuple 5 | from concurrent import futures 6 | from itertools import chain, izip 7 | 8 | 9 | Pool = futures.ProcessPoolExecutor() 10 | 11 | '''Helper for the filter methods. 12 | This is a container of the result of some_evaluation(arg) and arg itself.''' 13 | EvalResult = namedtuple('EvalResult', ['bool', 'item']) 14 | 15 | 16 | def _map(fn, *iterables): 17 | """Using our own internal map function. 18 | 19 | This is to avoid evaluation of the generator as done 20 | in futures.ProcessPoolExecutor().map. 21 | """ 22 | fs = (Pool.submit(fn, *args) for args in izip(*iterables)) 23 | for future in fs: 24 | yield future.result() 25 | 26 | 27 | class _Filter(object): 28 | """Helper for the filter methods. 29 | 30 | We need to use a class as closures cannot be 31 | pickled and sent to other processes. 32 | """ 33 | 34 | def __init__(self, predicate): 35 | if predicate is None: 36 | self.predicate = bool 37 | else: 38 | self.predicate = predicate 39 | 40 | def __call__(self, item): 41 | if self.predicate(item): 42 | return EvalResult(bool=True, item=item) 43 | else: 44 | return EvalResult(bool=False, item=None) 45 | 46 | 47 | class _Reducer(object): 48 | 49 | """Helper for the reducer methods.""" 50 | 51 | def __init__(self, func, init=None): 52 | self.func = func 53 | self.list = multiprocessing.Manager().list([init]) 54 | 55 | def __call__(self, item): 56 | aggregate = self.func(self.list[0], item) 57 | self.list[0] = aggregate 58 | 59 | @property 60 | def result(self): 61 | return self.list[0] 62 | 63 | 64 | class ParallelGen(object): 65 | 66 | def __init__(self, data_source): 67 | self.data = data_source 68 | 69 | def __iter__(self): 70 | for item in self.data: 71 | yield item 72 | 73 | def foreach(self, func): 74 | self.data = [i for i in _map(func, self)] 75 | return None 76 | 77 | def filter(self, pred): 78 | _filter = _Filter(pred) 79 | return self.__class__((i.item for i in _map(_filter, self, ) if i.bool)) 80 | 81 | def map(self, func): 82 | return self.__class__(_map(func, self, )) 83 | 84 | def flatmap(self, func): 85 | return self.__class__(chain(*_map(func, self))) 86 | 87 | def reduce(self, function, init=None): 88 | _reducer = _Reducer(function, init) 89 | for i in _map(_reducer, self, ): 90 | # need to consume the generator returned by _map 91 | pass 92 | return _reducer.result 93 | 94 | 95 | def parallel(data_source): 96 | """factory function that returns ParallelGen objects, 97 | pass an iterable as data_source""" 98 | if data_source.__class__.__name__ == 'function': 99 | if data_source().__class__.__name__ == 'generator': 100 | return ParallelGen(data_source()) 101 | else: 102 | try: 103 | iter(data_source) 104 | except TypeError: 105 | raise TypeError("""supplied data source must be a generator, 106 | a generator function or an iterable, 107 | not %s""" % data_source.__class__.__name__) 108 | return ParallelGen(data_source) 109 | 110 | 111 | """Below is all deprecated stuff, with DeprecationWarning""" 112 | 113 | def lazy_parallel(data_source): 114 | raise DeprecationWarning("""lazy_parallel has been deprecated, 115 | please use "parallel" instead, it has become lazy too :)""") 116 | 117 | 118 | class ParallelSeq(object): 119 | def __init__(self, *args, **kwargs): 120 | raise DeprecationWarning("""{0} has been deprecated, please use 121 | the "parallel" factory function instead 122 | """.format(self.__class__.__name__)) 123 | 124 | class ParallelList(ParallelSeq): 125 | pass 126 | 127 | class ParallelDict(ParallelSeq): 128 | pass 129 | 130 | class ParallelString(ParallelSeq): 131 | pass 132 | -------------------------------------------------------------------------------- /parallel/tests.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from itertools import chain, imap 3 | from collections import defaultdict 4 | 5 | from parallel_collections import ( 6 | parallel, lazy_parallel, ParallelSeq, 7 | ParallelList, ParallelDict, 8 | ParallelString, _Reducer, _Filter 9 | ) 10 | 11 | 12 | class TestHelpers(unittest.TestCase): 13 | 14 | def test_reducer_two_items(self): 15 | reducer = _Reducer(group_letters, defaultdict(list)) 16 | reducer('a') 17 | reducer('a') 18 | self.assertEquals(reducer.result, {'a': ['a', 'a']}) 19 | 20 | def test_reducer_three_items(self): 21 | reducer = _Reducer(group_letters, defaultdict(list)) 22 | reducer('a') 23 | reducer('a') 24 | reducer('b') 25 | self.assertEquals(reducer.result, {'a': ['a', 'a'], 'b': ['b']}) 26 | 27 | def test_filter_none(self): 28 | _filter = _Filter(is_digit) 29 | self.assertEquals(_filter('a'), (False, None)) 30 | 31 | def test_filter_returns_passing_item(self): 32 | _filter = _Filter(is_digit) 33 | self.assertEquals(_filter('1'), (True, '1')) 34 | 35 | class TestGen(unittest.TestCase): 36 | 37 | def test_foreach(self): 38 | p = parallel((d for d in [range(10),range(10)])) 39 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 40 | p.foreach(double) 41 | self.assertEquals(list(p), map(double, [range(10),range(10)])) 42 | self.assertTrue(p.foreach(double) is None) 43 | 44 | def test_map(self): 45 | p = parallel((d for d in [range(10),range(10)])) 46 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 47 | mapped = p.map(double) 48 | self.assertEquals(list(mapped), map(double, [range(10),range(10)])) 49 | self.assertFalse(mapped is p) 50 | 51 | def test_filter(self): 52 | p = parallel((d for d in ['a','2','3'])) 53 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 54 | pred = is_digit 55 | filtered = p.filter(pred) 56 | self.assertEquals(list(filtered), list(['2','3'])) 57 | self.assertFalse(filtered is p) 58 | 59 | def test_filter_with_ret_none_func(self): 60 | p = parallel((d for d in [True, False])) 61 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 62 | pred = ret_none 63 | filtered = p.filter(pred) 64 | self.assertEquals(list(filtered), list([])) 65 | self.assertFalse(filtered is p) 66 | 67 | def test_filter_with_none_as_func(self): 68 | p = parallel((d for d in [False, True])) 69 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 70 | pred = None 71 | filtered = p.filter(pred) 72 | self.assertEquals(list(filtered), list([True])) 73 | self.assertFalse(filtered is p) 74 | 75 | def test_filter_for_none_false_elements(self): 76 | p = parallel((d for d in [False, True, None])) 77 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 78 | pred = is_none_or_false 79 | filtered = p.filter(pred) 80 | self.assertEquals(list(filtered), list([False, None])) 81 | self.assertFalse(filtered is p) 82 | 83 | def test_flatmap(self): 84 | p = parallel([range(10),range(10)]) 85 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 86 | self.assertEquals(list(p.flatmap(double_iterables)), map(double, chain(*[range(10),range(10)]))) 87 | self.assertFalse(p.flatmap(double) is p) 88 | 89 | def test_chaining(self): 90 | p = parallel((d for d in chain(range(10),range(10)))) 91 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 92 | self.assertEquals(list(p.filter(is_even).map(double)), list(parallel([range(10),range(10)]).flatmap(double_evens_in_iterables))) 93 | 94 | def test_reduce(self): 95 | p = parallel((d for d in ['a', 'a', 'b'])) 96 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 97 | reduced = p.reduce(group_letters, defaultdict(list)) 98 | self.assertEquals(dict(a=['a','a'], b=['b',]), dict(reduced)) 99 | self.assertFalse(reduced is p) 100 | 101 | def test_gen_func_foreach(self): 102 | def inner_gen(): 103 | for d in [range(10),range(10)]: 104 | yield d 105 | p = parallel(inner_gen) 106 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 107 | p.foreach(double) 108 | self.assertEquals(list(p), map(double, [range(10),range(10)])) 109 | self.assertTrue(p.foreach(double) is None) 110 | 111 | def test_gen_func_map(self): 112 | def inner_gen(): 113 | for d in [range(10),range(10)]: 114 | yield d 115 | p = parallel(inner_gen) 116 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 117 | mapped = p.map(double) 118 | self.assertEquals(list(mapped), map(double, [range(10),range(10)])) 119 | self.assertFalse(mapped is p) 120 | 121 | def test_gen_func_filter(self): 122 | def inner_gen(): 123 | for d in ['a','2','3']: 124 | yield d 125 | p = parallel(inner_gen) 126 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 127 | pred = is_digit 128 | filtered = p.filter(pred) 129 | self.assertEquals(list(filtered), list(['2','3'])) 130 | self.assertFalse(filtered is p) 131 | 132 | def test_gen_func_flatmap(self): 133 | def inner_gen(): 134 | for d in [range(10),range(10)]: 135 | yield d 136 | p = parallel(inner_gen) 137 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 138 | self.assertEquals(list(p.flatmap(double_iterables)), map(double, chain(*[range(10),range(10)]))) 139 | self.assertFalse(p.flatmap(double) is p) 140 | 141 | def test_gen_func_chaining(self): 142 | def inner_gen(): 143 | for d in chain(*[range(10),range(10)]): 144 | yield d 145 | p = parallel(inner_gen) 146 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 147 | self.assertEquals(list(p.filter(is_even).map(double)), list(parallel([range(10),range(10)]).flatmap(double_evens_in_iterables))) 148 | 149 | def test_gen_func_reduce(self): 150 | def inner_gen(): 151 | for d in ['a', 'a', 'b']: 152 | yield d 153 | p = parallel(inner_gen) 154 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 155 | reduced = p.reduce(group_letters, defaultdict(list)) 156 | self.assertEquals(dict(a=['a','a'], b=['b',]), dict(reduced)) 157 | self.assertFalse(reduced is p) 158 | 159 | 160 | class TestFactories(unittest.TestCase): 161 | 162 | def test_returns_gen(self): 163 | p = parallel((d for d in [range(10),range(10)])) 164 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 165 | def inner_gen(): 166 | for i in range(10): 167 | yield i 168 | p = parallel(inner_gen()) 169 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 170 | p = parallel(list()) 171 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 172 | p = parallel(tuple()) 173 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 174 | p = parallel(set()) 175 | self.assertTrue(p.__class__.__name__ == 'ParallelGen') 176 | 177 | def test_lazy_parallel(self): 178 | with self.assertRaises(DeprecationWarning): 179 | p = lazy_parallel((d for d in [range(10),range(10)])) 180 | 181 | def test_class_depr_warning(self): 182 | with self.assertRaises(DeprecationWarning): 183 | p = ParallelSeq((d for d in [range(10),range(10)])) 184 | with self.assertRaises(DeprecationWarning): 185 | p = ParallelList((d for d in [range(10),range(10)])) 186 | with self.assertRaises(DeprecationWarning): 187 | p = ParallelDict((d for d in [range(10),range(10)])) 188 | with self.assertRaises(DeprecationWarning): 189 | p = ParallelString((d for d in [range(10),range(10)])) 190 | 191 | def test_raises_exception(self): 192 | def inner_func(): 193 | return 1 194 | class InnerClass(object): 195 | pass 196 | with self.assertRaises(TypeError): 197 | p = parallel(inner_func()) 198 | with self.assertRaises(TypeError): 199 | p = parallel(1) 200 | with self.assertRaises(TypeError): 201 | p = parallel(InnerClass()) 202 | 203 | 204 | def _print(item): 205 | print item 206 | return item 207 | 208 | def to_upper(item): 209 | return item.upper() 210 | 211 | def is_digit(item): 212 | return item.isdigit() 213 | 214 | def is_digit_dict(item): 215 | return item[1].isdigit() 216 | 217 | def add_up(x,y): 218 | return x+y 219 | 220 | def group_letters(all, letter): 221 | all[letter].append(letter) 222 | return all 223 | 224 | def group_letters_dict(all, letter): 225 | letter = letter[1] 226 | all[letter].append(letter) 227 | return all 228 | 229 | def chain_iterables(its): 230 | return chain(*its) 231 | 232 | def double(item): 233 | return item * 2 234 | 235 | def double_iterables(l): 236 | for item in l: 237 | return map(double, l) 238 | 239 | def is_even(i): 240 | if i & 1: 241 | return False 242 | else: 243 | return True 244 | 245 | def double_evens(i): 246 | if is_even(i): 247 | return i * 2 248 | 249 | def double_evens_in_iterables(l): 250 | for item in l: 251 | return map(double, filter(is_even, l)) 252 | 253 | def double_dict(item): 254 | k,v = item 255 | try: 256 | return [k, [i *2 for i in v]] 257 | except TypeError: 258 | return [k, v * 2] 259 | 260 | def ret_none(item): 261 | return None 262 | 263 | def is_none_or_false(item): 264 | return item is None or item is False 265 | 266 | if __name__ == '__main__': 267 | unittest.main() 268 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | futures==2.1.5 2 | wsgiref==0.1.2 3 | nose==1.3.0 4 | coverage==3.7.1 5 | coveralls==0.5 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | import parallel 3 | 4 | setup( 5 | name='python-parallel-collections', 6 | data_files=[('', ['requirements.txt', 'README.md', '.gitignore']),], 7 | version='2.0.0', 8 | packages=['parallel',], 9 | description='parallel support for map/reduce style operations', 10 | long_description=''' 11 | 12 | This package provides a convenient interface to perform map/filter/reduce style operation on standard Python data structures and generators in multiple processes. 13 | The parallelism is achieved using the Python 2.7 backport of the concurrent.futures package. 14 | If you can define your problem in terms of map/reduce/filter operations, it will run on several parallel Python processes on your machine, taking advantage of multiple cores. 15 | \n 16 | Examples at https://github.com/gterzian/Python-Parallel-Collections 17 | Feedback and contributions highly sought after!''', 18 | author='Gregory Terzian', 19 | license='BSD License', 20 | url='https://github.com/gterzian/Python-Parallel-Collections', 21 | platforms=["any"], 22 | requires=['futures',], 23 | classifiers=[ 24 | 'Natural Language :: English', 25 | 'Operating System :: OS Independent', 26 | 'Programming Language :: Python', 27 | 'Topic :: System :: Distributed Computing', 28 | ], 29 | ) 30 | --------------------------------------------------------------------------------