├── .gitignore ├── README.md ├── example.py ├── setup.py └── share_array ├── __init__.py └── share_array.py /.gitignore: -------------------------------------------------------------------------------- 1 | ### Python ### 2 | # Byte-compiled / optimized / DLL files 3 | __pycache__/ 4 | *.py[cod] 5 | *$py.class 6 | 7 | # C extensions 8 | *.so 9 | 10 | # Distribution / packaging 11 | .Python 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | pip-wheel-metadata/ 25 | share/python-wheels/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | MANIFEST 30 | 31 | # PyInstaller 32 | # Usually these files are written by a python script from a template 33 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 34 | *.manifest 35 | *.spec 36 | 37 | # Installer logs 38 | pip-log.txt 39 | pip-delete-this-directory.txt 40 | 41 | # Unit test / coverage reports 42 | htmlcov/ 43 | .tox/ 44 | .nox/ 45 | .coverage 46 | .coverage.* 47 | .cache 48 | nosetests.xml 49 | coverage.xml 50 | *.cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | 63 | # Flask stuff: 64 | instance/ 65 | .webassets-cache 66 | 67 | # Scrapy stuff: 68 | .scrapy 69 | 70 | # Sphinx documentation 71 | docs/_build/ 72 | 73 | # PyBuilder 74 | target/ 75 | 76 | # Jupyter Notebook 77 | .ipynb_checkpoints 78 | 79 | # IPython 80 | profile_default/ 81 | ipython_config.py 82 | 83 | # pyenv 84 | .python-version 85 | 86 | # pipenv 87 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 88 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 89 | # having no cross-platform support, pipenv may install dependencies that don’t work, or not 90 | # install all needed dependencies. 91 | #Pipfile.lock 92 | 93 | # celery beat schedule file 94 | celerybeat-schedule 95 | 96 | # SageMath parsed files 97 | *.sage.py 98 | 99 | # Environments 100 | .env 101 | .venv 102 | env/ 103 | venv/ 104 | ENV/ 105 | env.bak/ 106 | venv.bak/ 107 | 108 | # Spyder project settings 109 | .spyderproject 110 | .spyproject 111 | 112 | # Rope project settings 113 | .ropeproject 114 | 115 | # mkdocs documentation 116 | /site 117 | 118 | # mypy 119 | .mypy_cache/ 120 | .dmypy.json 121 | dmypy.json 122 | 123 | # Pyre type checker 124 | .pyre/ 125 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## multiprocess-shared-numpy-arrays 2 | 3 | ### Easily share numpy arrays between processes 4 | Convenience functions for sharing 5 | [numpy arrays](https://docs.scipy.org/doc/numpy/) 6 | between multiple processes using 7 | [multiprocessing.Array](https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.Array) 8 | as process safe shared memory arrays. 9 | 10 | #### Usage 11 | ```python 12 | # Create shared array from numpy array: 13 | make_shared_array(np_array, name='my_shared_array') 14 | 15 | # Access shared array as numpy array: 16 | array = get_shared_array('my_shared_array') 17 | ``` 18 | 19 | ### Installation 20 | 21 | ```bash 22 | pip install git+https://github.com/widmi/multiprocess-shared-numpy-arrays 23 | ``` 24 | 25 | 26 | ### Full example: 27 | 28 | ```python 29 | import numpy as np 30 | from multiprocessing import Pool 31 | from share_array.share_array import get_shared_array, make_shared_array 32 | 33 | 34 | def worker_function(i): 35 | """Function that uses the shared array""" 36 | array = get_shared_array('my_shared_array') # get shared memory array as numpy array 37 | array[:] += i # modify the shared memory array as numpy array 38 | 39 | 40 | if __name__ == '__main__': 41 | np_array = np.arange(3*5).reshape((3, 5)) # make a numpy array 42 | make_shared_array(np_array, name='my_shared_array') # create shared memory array from numpy array 43 | shared_array = get_shared_array('my_shared_array') # get shared memory array as numpy array 44 | 45 | print("Shared array before multiprocessing:") 46 | print(shared_array) # Print array 47 | 48 | with Pool(processes=2) as pool: 49 | _ = pool.map(worker_function, range(15)) # use a multiprocessing.Pool to distribute work to worker processes 50 | 51 | print("Shared array after multiprocessing:") 52 | print(shared_array) # Content of array was changed in worker processes 53 | ``` 54 | 55 | -------------------------------------------------------------------------------- /example.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """example.py 3 | 4 | 5 | Author -- Michael Widrich 6 | Contact -- widrich@ml.jku.at 7 | """ 8 | 9 | import numpy as np 10 | from multiprocessing import Pool 11 | from share_array.share_array import get_shared_array, make_shared_array 12 | 13 | 14 | def worker_function(i): 15 | """Function that uses the shared array""" 16 | array = get_shared_array('my_shared_array') # get shared memory array as numpy array 17 | array[:] += i # modify the shared memory array as numpy array 18 | 19 | 20 | if __name__ == '__main__': 21 | np_array = np.arange(3*5).reshape((3, 5)) # make a numpy array 22 | make_shared_array(np_array, name='my_shared_array') # create shared memory array from numpy array 23 | shared_array = get_shared_array('my_shared_array') # get shared memory array as numpy array 24 | 25 | print("Shared array before multiprocessing:") 26 | print(shared_array) # Print array 27 | 28 | with Pool(processes=2) as pool: 29 | _ = pool.map(worker_function, range(15)) # use a multiprocessing.Pool to distribute work to worker processes 30 | 31 | print("Shared array after multiprocessing:") 32 | print(shared_array) # Content of array was changed in worker processes 33 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.md", "r") as fh: 4 | long_description = fh.read() 5 | 6 | setuptools.setup( 7 | name="share-array-widmi", 8 | version="0.0.1", 9 | author="Michael Widrich", 10 | author_email="widrich@ml.jku.at", 11 | description="Easily share numpy arrays between processes", 12 | long_description=long_description, 13 | long_description_content_type="text/markdown", 14 | url="https://github.com/widmi/multiprocess-shared-numpy-arrays", 15 | packages=setuptools.find_packages(), 16 | classifiers=[ 17 | "Programming Language :: Python :: 3", 18 | "License :: OSI Approved :: MIT License", 19 | "Operating System :: OS Independent", 20 | ], 21 | ) -------------------------------------------------------------------------------- /share_array/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """share_array.py 3 | 4 | 5 | Author -- Michael Widrich 6 | Contact -- widrich@ml.jku.at 7 | """ 8 | -------------------------------------------------------------------------------- /share_array/share_array.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """share_array.py 3 | 4 | 5 | Author -- Michael Widrich 6 | Contact -- widrich@ml.jku.at 7 | """ 8 | 9 | import numpy as np 10 | from multiprocessing import Array 11 | 12 | 13 | def get_shared_array(name: str, shape=None): 14 | """ Get a shared memory array (multiprocessing.Array) with name and view it as a numpy array. 15 | 16 | The shared memory array is a flattened array but if created by share_array.share_array.make_shared_array(), it will 17 | be reshaped to the original shape of the numpy array. Optionally, a shape can be explicitly specified. 18 | 19 | Parameters 20 | ---------- 21 | name : str 22 | Name of the shared memory array as string. 23 | shape : tuple or None 24 | The shared memory array is a flattened array but if created by share_array.share_array.make_shared_array(), 25 | it will be reshaped to the original shape of the numpy array. 26 | Optionally, a shape can be explicitly specified. 27 | 28 | Returns 29 | ---------- 30 | np_array : np.ndarray 31 | Shared memory array viewed as numpy array. 32 | 33 | Example 34 | ------- 35 | >>> from share_array.share_array import make_shared_array, get_shared_array 36 | >>> import numpy as np 37 | >>> np_array = np.arange(5*3).reshape(5,3) # create a numpy array 38 | >>> if __name__ == '__main__': 39 | >>> make_shared_array(np_array, name='my_shared_array') # create shared memory array from numpy array 40 | >>> shared_np_array = get_shared_array('my_shared_array') # get process safe shared memory array as numpy array 41 | """ 42 | mp_array = globals()[name] 43 | np_array = np.frombuffer(mp_array.get_obj(), dtype=np.dtype(mp_array.get_obj()._type_)) 44 | if (shape is None) and (name + '_shape' in globals().keys()): 45 | shape = globals()[name + '_shape'] 46 | shape = np.frombuffer(shape.get_obj(), dtype=np.int) 47 | if shape is not None: 48 | np_array = np_array.reshape(shape) 49 | return np_array 50 | 51 | 52 | def make_shared_array(np_array: np.ndarray, name: str): 53 | """ Crate a shared memory array (multiprocessing.Array) with name from a numpy array. 54 | Shared memory array will be automatically initialized with the data from the numpy array. 55 | Shared memory array can accessed as numpy array using ShareArray.get_shared_array(). 56 | 57 | Flattened shared memory array will be created as global variable with name and it's shape information will 58 | be stored in another shared memory array _shape, which is used by get_shared_array() to get the numpy shape. 59 | 60 | Parameters 61 | ---------- 62 | np_array : np.ndarray 63 | Numpy array to create shared memory array from. 64 | Numpy datatype will be converted to ctypes datatype automatically. 65 | Data from numpy array will be used to initialize the shared array. 66 | name : str 67 | Name of the shared memory array as string. 68 | Shared memory array can be accessed as numpy array from multiple processes using get_shared_array(). 69 | 70 | Example 71 | ------- 72 | >>> from share_array.share_array import make_shared_array, get_shared_array 73 | >>> import numpy as np 74 | >>> np_array = np.arange(5*3).reshape(5,3) # create a numpy array 75 | >>> if __name__ == '__main__': 76 | >>> make_shared_array(np_array, name='my_shared_array') # create shared memory array from numpy array 77 | >>> shared_np_array = get_shared_array('my_shared_array') # get process safe shared memory array as numpy array 78 | """ 79 | mp_dtype = np.ctypeslib.as_ctypes(np_array.dtype.type())._type_ 80 | mp_array = Array(typecode_or_type=mp_dtype, size_or_initializer=int(np.prod(np_array.shape))) 81 | globals()[name] = mp_array 82 | shared_np_array = get_shared_array(name, shape=np_array.shape) 83 | shared_np_array[:] = np_array 84 | 85 | mp_array_shape = Array(typecode_or_type='l', size_or_initializer=len(np_array.shape)) 86 | globals()[name + '_shape'] = mp_array_shape 87 | shared_np_array = get_shared_array(name + '_shape') 88 | shared_np_array[:] = np_array.shape 89 | 90 | --------------------------------------------------------------------------------