├── .gitignore ├── A Beginner's Approach to Multi-processing Optimization ├── README.md ├── birdview.jpg ├── multi.py ├── single.ipynb ├── single.py └── single.py.lprof └── Python HTTP2 Experiments ├── README.md ├── app.py ├── h2web ├── __init__.py ├── app.py └── curio_server.py ├── localhost.crt.pem └── localhost.key /.gitignore: -------------------------------------------------------------------------------- 1 | # common 2 | .idea/ 3 | *.pyc 4 | 5 | # A Beginner's Approach to Multi-processing Optimization 6 | data/ 7 | .ipynb_checkpoints/ 8 | 9 | # Python HTTP2 Experiments 10 | hyper-h2-master/ 11 | public/ 12 | 13 | -------------------------------------------------------------------------------- /A Beginner's Approach to Multi-processing Optimization/README.md: -------------------------------------------------------------------------------- 1 | This note demonstrates how to use __multiple processes on data parallelism tasks__. 2 | 3 | The experiment is only suitable for GB level data. In my case, I used 12 GB of data. 4 | 5 | You need to create a `data/` dir to run this experiment. 6 | 7 | Under `data/`, you need to create `raw/` and `processed/` dirs. 8 | 9 | See the notebook for details. 10 | 11 | ### Please read the notebook first 12 | 13 | ### Multi-process code 14 | 15 | Now, we are pretty confident that our single version is fast enough. Let's summon the multi-process demon. 16 | 17 | We use the `multiprocessing.Pool` class. This class has a `map` function which allows you to do data parallelism like it's nothing. 18 | 19 | `map` function blocks until all results are computed. You don't need to worry about spawning, passing data, listening and joining processes at all. You just write your code as if it's single processed. 20 | 21 | This function is useful when your data are indenpendent from one anathor. In our case, each file can be handled individually, so that each process only has to be responsible for itself. 22 | 23 | [multiprocessing](https://docs.python.org/3.5/library/multiprocessing.html) module API here. 24 | 25 | [Pool](https://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.pool.Pool) API here. 26 | 27 | Our code goes like: 28 | ```python 29 | def f(path): 30 | view = bird_view_map(read_velodyne_data(path[0])) 31 | cv2.imwrite(''.join([path[1], '/', os.path.basename(path[0])[:-4], '.png']), view) 32 | 33 | def generate_birdviews(data_paths, to_dir, workers): 34 | """ 35 | This function process velodyne data to birdview in parallel 36 | :param data_paths: a list of paths to velodyne xxx.bin files 37 | :param to_dir: write birdview maps to this directory 38 | :param workers: number of processes 39 | """ 40 | with Pool(workers) as p: 41 | to_dirs = [to_dir] * len(data_paths) 42 | p.map(f, list(zip(data_paths, to_dirs))) 43 | 44 | 45 | if __name__ == '__main__': 46 | t = time.time() 47 | generate_birdviews(glob('data/raw/*.bin')[:100], to_dir='data/processed', workers=8) 48 | used = time.time() - t 49 | print('Simple multi-processed version:', used, 'seconds') 50 | ``` 51 | Notice 2 things: 52 | 1. In `Pool.map(func, iterable)`, `func` must be defined at the module level. In other words, it has to be a global function. 53 | 2. You have to call `.map` under `__main__` namespace. Or you will have a major problem. You can do some research to know why. 54 | 55 | The result on my i7 (4 cores, 8 hyperthreads) 56 | ``` 57 | Simple multi-processed version: 37.5 seconds 58 | ``` 59 | 60 | As you can see, we push down the time to less than 1/3. 61 | 62 | Why isn't it 1/8 ? First, it's hyper threadings, it's not real 8 cores. Second, inter process communication has overhead. 63 | 64 | But still, with mere 10 lines of code, we boost the performance 250%. This is incredible gain. 65 | 66 | However, we can still optimize further. Remember, interprocess communication is heavy. Even spawning a new process introduce a lot of overhead. 67 | 68 | Do you know how many processes our code spawns? 100 processes! 69 | 70 | Wait, isn't it 8? Yes, only 8 processes are alive at a time. But, since the list has 100 elements, there are 100 process ever alive in total. That's a huge waste since each process only process 1 file! 71 | 72 | It would be wise if we divide our data into 8 pieces instead of 100 pieces and let only 8 processes to deal with them. 73 | 74 | Let's modifly the code: 75 | ```Python 76 | def f2(paths): 77 | paths = pickle.loads(paths) 78 | _from, to = paths 79 | for i in range(len(to)): 80 | view = bird_view_map(read_velodyne_data(_from[i])) 81 | cv2.imwrite(''.join([to[i], '/', os.path.basename(_from[i])[:-4], '.png']), view) 82 | 83 | 84 | def generate_birdviews_2(data_paths, to_dir, workers): 85 | """ 86 | This function process velodyne data to birdview in parallel 87 | :param data_paths: a list of paths to velodyne xxx.bin files 88 | :param to_dir: write birdview maps to this directory 89 | :param workers: number of processes 90 | """ 91 | with Pool(workers) as p: 92 | to_dirs = [to_dir] * len(data_paths) 93 | n = np.int(np.ceil(len(data_paths)/workers)) 94 | _list = [pickle.dumps((data_paths[i:i+n], to_dirs[i:i+n])) for i in range(0, len(data_paths), n)] 95 | p.map(f2, _list) 96 | 97 | if __name__ == '__main__': 98 | t = time.time() 99 | generate_birdviews_2(glob('data/raw/*.bin')[:100], to_dir='data/processed', workers=8) 100 | used2 = time.time() - t 101 | print('Better multi-processed version:', used2, 'seconds') 102 | ``` 103 | As you can see, we divide the data into 8 pieces by 104 | ``` 105 | _list = [pickle.dumps((data_paths[i:i+n], to_dirs[i:i+n])) for i in range(0, len(data_paths), n)] 106 | ``` 107 | Notice that we use `pickle` to serialize our data, since Python uses the unix tradition that processes communicate through streams/files. Therefore, we can only pass `bytes`, `string` or `int` to another process. (I am not sure if you can pass float). Here we serialize a Python object to Python `bytes`. 108 | 109 | The result is: 110 | ``` 111 | Better multi-processed version: 36.4 seconds 112 | ``` 113 | A 3% gain. Yeah! 114 | 115 | ### Conclusion 116 | Through 2 steps of optimization, with less than 10 lines of code, we obtain an overall 293% performance gain. Remember, Python is slow, but, it's not hard to make it less slow. 117 | 118 | You might want to ask, why bother? If my Python code is slow, I can always write the slow part with C++. 119 | 120 | I have 2 arguments to justify it: 121 | 1. We, Python programmers say that we will rewrite slow code with C/C++. Nobody does it. It is not that easy to write a native extension for Python. Sometimes it's just not worth it to write a native code for some one-time use code. If you are writing a more general utility that will be used by many developers, then it's worth it. Numpy etc. We are not doing extreme optimization here. 122 | 2. No matter how fast your single thread code is, you still need to write milti-process parallel code someitmes. This experiment use Python as an example, but the principle applies to all languages. If you have data parallelism, consider multi-process. 123 | 124 | Ok, hope this is helpful to you. 125 | 126 | Best 127 | -------------------------------------------------------------------------------- /A Beginner's Approach to Multi-processing Optimization/birdview.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CreatCodeBuild/Python-Notes/dc07d56a30adf89e27c4d97109e9c07fac13d0b3/A Beginner's Approach to Multi-processing Optimization/birdview.jpg -------------------------------------------------------------------------------- /A Beginner's Approach to Multi-processing Optimization/multi.py: -------------------------------------------------------------------------------- 1 | from multiprocessing import Pool 2 | from glob import glob 3 | import cv2 4 | import os 5 | import numpy as np 6 | from io import BufferedReader 7 | import time 8 | import pickle 9 | 10 | 11 | """ 12 | First, let's define helper functions. 13 | We use the faster version of read_velodyne_data here. 14 | """ 15 | def read_velodyne_data(file_path): 16 | """ 17 | Read velodyne binary data and return a numpy array 18 | """ 19 | 20 | # First, check the size of this file to see if it's a valid velodyne binary file 21 | size = os.stat(file_path).st_size 22 | if size % 16 != 0: 23 | raise Exception('The size of '+file_path+' is not dividible by 16 bytes') 24 | 25 | with open(file_path, 'rb') as f: 26 | # Allocate memory for numpy array 27 | velodyne_data = np.empty(shape=(size//16, 4), dtype=np.float32) 28 | 29 | # Read the data, 16 bytes each time 30 | i = 0 31 | reader = BufferedReader(f) 32 | read_bytes = reader.read(16) 33 | while read_bytes: 34 | velodyne_data[i] = np.frombuffer(read_bytes, dtype=np.float32) 35 | read_bytes = reader.read(16) 36 | i += 1 37 | 38 | # Check whether correct amount of bytes were read 39 | if i != size/16: 40 | error = ' '.join(['The file size is', str(size), ', but', str(i), 'bytes were read']) 41 | raise Exception(error) 42 | 43 | return velodyne_data 44 | 45 | 46 | def bird_view_map(velodyne_data): 47 | """ 48 | Implements the method in https://arxiv.org/pdf/1611.07759.pdf 49 | :param velodyne_data: a list of velodyne cloud points 50 | :return: 2D image with 3 channels: height, intensity and density 51 | """ 52 | bird_view = np.zeros(shape=(1600, 1600, 3), dtype=np.float32) 53 | for point in velodyne_data: 54 | x = point[0] 55 | y = point[1] 56 | z = point[2] 57 | r = point[3] 58 | # if (-40 <= x <= 40) and (-40 <= y <= 40): 59 | xi = 800 - np.int(np.ceil(x/0.1)) 60 | yi = 800 - np.int(np.ceil(y/0.1)) 61 | if -z > bird_view[yi][xi][0]: 62 | bird_view[yi][xi][0] = -z 63 | bird_view[yi][xi][1] = r 64 | bird_view[yi][xi][2] += 1 65 | 66 | # todo: normalize birdview with the real method 67 | bird_view[:,:,0] = np.interp(bird_view[:,:,0], xp=(np.min(bird_view[:,:,0]), np.max(bird_view[:,:,0])), fp=(0, 255)) 68 | bird_view[:,:,1] = np.interp(bird_view[:,:,1], xp=(0, 1), fp=(0, 255)) 69 | bird_view[:,:,2] = np.interp(bird_view[:,:,2], xp=(np.min(bird_view[:,:,2]), np.max(bird_view[:,:,2])), fp=(0, 255)) 70 | return bird_view 71 | 72 | 73 | 74 | """ 75 | Now, let's define the multi-process code. 76 | We use the multiprocessing.Pool class. This class has a map function which allows you to do data parallelism like it's nothing. 77 | 78 | With map function, you don't need to worry about spawning, passing data, listening and joining processes at all. 79 | You just write your code as if it's single processed. 80 | 81 | This function is useful when your data are indenpendent from one anathor. 82 | In our case, each file can be handled individually, so that each process only has to be responsible for itself. 83 | """ 84 | def f(path): 85 | view = bird_view_map(read_velodyne_data(path[0])) 86 | cv2.imwrite(''.join([path[1], '/', os.path.basename(path[0])[:-4], '.png']), view) 87 | 88 | def generate_birdviews(data_paths, to_dir, workers): 89 | """ 90 | This function process velodyne data to birdview in parallel 91 | :param data_paths: a list of paths to velodyne xxx.bin files 92 | :param to_dir: write birdview maps to this directory 93 | :param workers: number of processes 94 | """ 95 | with Pool(workers) as p: 96 | to_dirs = [to_dir] * len(data_paths) 97 | p.map(f, list(zip(data_paths, to_dirs))) 98 | 99 | 100 | if __name__ == '__main__': 101 | t = time.time() 102 | generate_birdviews(glob('data/raw/*.bin')[:100], to_dir='data/processed', workers=8) 103 | used = time.time() - t 104 | print('Simple multi-processed version:', used, 'seconds') 105 | 106 | 107 | 108 | """ 109 | Can we optimize further? 110 | """ 111 | # def f2(paths): 112 | # paths = pickle.loads(paths) 113 | # _from, to = paths 114 | # for i in range(len(to)): 115 | # view = bird_view_map(read_velodyne_data(_from[i])) 116 | # cv2.imwrite(''.join([to[i], '/', os.path.basename(_from[i])[:-4], '.png']), view) 117 | 118 | 119 | # def generate_birdviews_2(data_paths, to_dir, workers): 120 | # """ 121 | # This function process velodyne data to birdview in parallel 122 | # :param data_paths: a list of paths to velodyne xxx.bin files 123 | # :param to_dir: write birdview maps to this directory 124 | # :param workers: number of processes 125 | # """ 126 | # with Pool(workers) as p: 127 | # to_dirs = [to_dir] * len(data_paths) 128 | # n = np.int(np.ceil(len(data_paths)/workers)) 129 | # _list = [pickle.dumps((data_paths[i:i+n], to_dirs[i:i+n])) for i in range(0, len(data_paths), n)] 130 | # p.map(f2, _list) 131 | 132 | # if __name__ == '__main__': 133 | # t = time.time() 134 | # generate_birdviews_2(glob('data/raw/*.bin')[:100], to_dir='data/processed', workers=8) 135 | # used2 = time.time() - t 136 | # print('Better multi-processed version:', used2, 'seconds') 137 | -------------------------------------------------------------------------------- /A Beginner's Approach to Multi-processing Optimization/single.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# First Time to Python Multi-processing\n", 8 | "I want to share something I learned about Python multi-processing on handling middle size data on a single machine.\n", 9 | "\n", 10 | "By middle size, I mean GB level data. In my case, that is 13.2GB of data, which is small enough to load into memory but once load into memeory, you may not have enough memory to run any algorithms.\n", 11 | "\n", 12 | "Let's dive into it.\n", 13 | "\n", 14 | "# Problem Description\n", 15 | "I have 13.2GB of data consisted of 7481 binary files. Each file is about 1.81M.\n", 16 | "\n", 17 | "My task is simple:\n", 18 | " 1. Read the file into a Python data structure\n", 19 | " 2. Process the data to an image\n", 20 | " 3. Write the image to disk\n", 21 | " \n", 22 | "## File Format\n", 23 | "The file stores a collection of vectors. Each vector has 4 elements: x, y, z, r\n", 24 | "\n", 25 | "Each element is a float32, so the size of a vector is 16 bytes.\n", 26 | "\n", 27 | "There is no delimiters between 2 vectors. Therefore, if the file size is 32 bytes, it has and only has 2 vectors. If the size of a file is not dividible by 16 bytes, it is not a valid file.\n", 28 | " \n" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 1, 34 | "metadata": { 35 | "collapsed": true 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "def read_velodyne_data(file_path):\n", 40 | " \"\"\"\n", 41 | " Read velodyne binary data and return a numpy array\n", 42 | " \"\"\"\n", 43 | "\n", 44 | " # First, check the size of this file to see if it's a valid velodyne binary file\n", 45 | " size = os.stat(file_path).st_size\n", 46 | " if size % 16 != 0:\n", 47 | " raise Exception('The size of '+file_path+' is not dividible by 16 bytes')\n", 48 | "\n", 49 | " with open(file_path, 'rb') as f:\n", 50 | " # Allocate memory for numpy array\n", 51 | " velodyne_data = np.empty(shape=(size//16, 4), dtype=np.float32)\n", 52 | "\n", 53 | " # Read the data, 16 bytes each time\n", 54 | " i = 0\n", 55 | " reader = BufferedReader(f)\n", 56 | " while reader.peek(16):\n", 57 | " read_bytes = reader.read(16)\n", 58 | " velodyne_data[i] = np.frombuffer(read_bytes, dtype=np.float32)\n", 59 | " i += 1\n", 60 | "\n", 61 | " # Check whether correct amount of bytes were read\n", 62 | " if i != size/16:\n", 63 | " error = ' '.join(['The file size is', str(size), ', but', str(i), 'bytes were read'])\n", 64 | " raise Exception(error)\n", 65 | "\n", 66 | " return velodyne_data" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "## Let's process the data\n", 74 | "Each file is a velodyne point cloud scan\n", 75 | "\n", 76 | "This is an sample processed image. This a a birdview of the lidar data\n", 77 | "\n", 78 | "\n", 79 | "The image is 1600 x 1600 pixel. Each pixel represent 10cm x 10cm of space. The range of the lidar in the x and y direction is 80m.\n", 80 | "\n", 81 | "Where x is left and right, y is front and back.\n", 82 | "\n", 83 | "We have 3 channels for this image." 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 2, 89 | "metadata": { 90 | "collapsed": true 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "def bird_view_map(velodyne_data):\n", 95 | " \"\"\"\n", 96 | " Implements the method in https://arxiv.org/pdf/1611.07759.pdf\n", 97 | " :param velodyne_data: a list of velodyne cloud points\n", 98 | " :return: 2D image with 3 channels: height, intensity and density\n", 99 | " \"\"\"\n", 100 | " bird_view = np.zeros(shape=(1600, 1600, 3), dtype=np.float32)\n", 101 | " for point in velodyne_data:\n", 102 | " x = point[0]\n", 103 | " y = point[1]\n", 104 | " z = point[2]\n", 105 | " r = point[3]\n", 106 | " # if (-40 <= x <= 40) and (-40 <= y <= 40):\n", 107 | " xi = 800 - np.int(np.ceil(x/0.1))\n", 108 | " yi = 800 - np.int(np.ceil(y/0.1))\n", 109 | " if -z > bird_view[yi][xi][0]:\n", 110 | " bird_view[yi][xi][0] = -z\n", 111 | " bird_view[yi][xi][1] = r\n", 112 | " bird_view[yi][xi][2] += 1\n", 113 | "\n", 114 | " # todo: normalize birdview with the real method\n", 115 | " bird_view[:,:,0] = np.interp(bird_view[:,:,0], xp=(np.min(bird_view[:,:,0]), np.max(bird_view[:,:,0])), fp=(0, 255))\n", 116 | " bird_view[:,:,1] = np.interp(bird_view[:,:,1], xp=(0, 1), fp=(0, 255))\n", 117 | " bird_view[:,:,2] = np.interp(bird_view[:,:,2], xp=(np.min(bird_view[:,:,2]), np.max(bird_view[:,:,2])), fp=(0, 255))\n", 118 | " return bird_view" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 3, 124 | "metadata": { 125 | "collapsed": false 126 | }, 127 | "outputs": [ 128 | { 129 | "name": "stdout", 130 | "output_type": "stream", 131 | "text": [ 132 | "Single processed version used 185.85564661026 seconds to process 100 files\n" 133 | ] 134 | } 135 | ], 136 | "source": [ 137 | "from glob import glob\n", 138 | "import cv2\n", 139 | "import os\n", 140 | "import numpy as np\n", 141 | "from io import BufferedReader\n", 142 | "import time\n", 143 | "\n", 144 | "\n", 145 | "paths = glob('data/raw/*.bin')[:100]\n", 146 | "t = time.time()\n", 147 | "for path in paths:\n", 148 | " data = read_velodyne_data(path)\n", 149 | " view = bird_view_map(data)\n", 150 | " cv2.imwrite(''.join(['data/processed/', os.path.basename(path)[:-4], '.png']), view)\n", 151 | "used = time.time() - t\n", 152 | "print('Single processed version used', used, 'seconds to process', len(paths), 'files')" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "### How can we optimize this program?\n", 160 | "Notice that we have 2 functions here:\n", 161 | "1. __read_velodyne_data__, which does the input.\n", 162 | "2. __bird_view_map__, which does the computation.\n", 163 | "\n", 164 | "Before we get into any of those parallelism things, the first thing is always try to optimize our individual functions. Because if your functions are slow, your parallel code is just slow code on more cores.\n", 165 | "\n", 166 | "We are not going to use our intuition or experiences to just staring at our code and hope something could happen. \n", 167 | "\n", 168 | "We are going to employ profiling tools to precisely measure our code performance.\n", 169 | "\n", 170 | "### Profile the code\n", 171 | "The traditional cProfile won't be useful here because it only profiles the code at a function level. It only tells you how much time each function call uses. Let's say you know that function __f1__ used 80% of your program time. You still don't know which line in __f1__ cost you so much time.\n", 172 | "\n", 173 | "That's why we need a line by line profiler [line_profiler](https://github.com/rkern/line_profiler) by [Robert Kern](https://github.com/rkern).\n", 174 | "\n", 175 | "Please look at my __README.md__ and line_profiler's README for profiling instructions.\n", 176 | "\n", 177 | "### Now we know\n", 178 | "Now we know that __while reader.peek(16)__ is the bottomneck of __read_velodyne_data__ function.\n", 179 | "\n", 180 | "__peek__ is unnecessary because we are going to read the data anyway. Therefore, we can change this function to" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 4, 186 | "metadata": { 187 | "collapsed": false 188 | }, 189 | "outputs": [], 190 | "source": [ 191 | "def read_velodyne_data_quick(file_path):\n", 192 | " \"\"\"\n", 193 | " Read velodyne binary data and return a numpy array\n", 194 | " \"\"\"\n", 195 | "\n", 196 | " # First, check the size of this file to see if it's a valid velodyne binary file\n", 197 | " size = os.stat(file_path).st_size\n", 198 | " if size % 16 != 0:\n", 199 | " raise Exception('The size of '+file_path+' is not dividible by 16 bytes')\n", 200 | "\n", 201 | " with open(file_path, 'rb') as f:\n", 202 | " # Allocate memory for numpy array\n", 203 | " velodyne_data = np.empty(shape=(size//16, 4), dtype=np.float32)\n", 204 | "\n", 205 | " # Read the data, 16 bytes each time\n", 206 | " i = 0\n", 207 | " reader = BufferedReader(f)\n", 208 | " read_bytes = reader.read(16) # As you can see here, we read directly and check if read_bytes has values.\n", 209 | " while read_bytes:\n", 210 | " velodyne_data[i] = np.frombuffer(read_bytes, dtype=np.float32)\n", 211 | " read_bytes = reader.read(16)\n", 212 | " i += 1\n", 213 | "\n", 214 | " # Check whether correct amount of bytes were read\n", 215 | " if i != size/16:\n", 216 | " error = ' '.join(['The file size is', str(size), ', but', str(i), 'bytes were read'])\n", 217 | " raise Exception(error)\n", 218 | "\n", 219 | " return velodyne_data" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "### Now let's run it for 100 files" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 10, 232 | "metadata": { 233 | "collapsed": false 234 | }, 235 | "outputs": [ 236 | { 237 | "name": "stdout", 238 | "output_type": "stream", 239 | "text": [ 240 | "Better version used 131.4069118499756 seconds\n", 241 | "It is 11.766444206237793 seconds faster for 100 files\n" 242 | ] 243 | } 244 | ], 245 | "source": [ 246 | "t = time.time()\n", 247 | "paths = glob('data/raw/*.bin')[:100]\n", 248 | "for path in paths:\n", 249 | " data = read_velodyne_data_quick(path)\n", 250 | " view = bird_view_map(data)\n", 251 | " cv2.imwrite(''.join(['data/processed/', os.path.basename(path)[:-4], '.png']), view)\n", 252 | "used2 = time.time() - t\n", 253 | "print('Better version used', used2, 'seconds')\n", 254 | "print('It is ', used - used2, 'seconds faster for', len(paths), 'files')" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": { 260 | "collapsed": false 261 | }, 262 | "source": [ 263 | "### 4% Performance Gain\n", 264 | "As you can see, conversativly speaking, it is at least 5 seconds faster for 100 files. This is about 4% faster.\n", 265 | "\n", 266 | "I don't know about you, but I am impressed. Just by changing a single line of IO, we can save 4% of time.\n", 267 | "\n", 268 | "For our data, we have 7821 files, we will save 391.05 seconds in total. \n", 269 | "\n", 270 | "Just image if you have much more data, 4% is incredible optimization.\n", 271 | "\n", 272 | "### Multi-process\n", 273 | "Because notebook doesn't work well with multi-process code, we will dicover the multi-process world in an ordinary Python script.\n", 274 | "\n", 275 | "Please see __multi.py__" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": null, 281 | "metadata": { 282 | "collapsed": true 283 | }, 284 | "outputs": [], 285 | "source": [] 286 | } 287 | ], 288 | "metadata": { 289 | "kernelspec": { 290 | "display_name": "Python 3", 291 | "language": "python", 292 | "name": "python3" 293 | }, 294 | "language_info": { 295 | "codemirror_mode": { 296 | "name": "ipython", 297 | "version": 3 298 | }, 299 | "file_extension": ".py", 300 | "mimetype": "text/x-python", 301 | "name": "python", 302 | "nbconvert_exporter": "python", 303 | "pygments_lexer": "ipython3", 304 | "version": "3.5.2" 305 | } 306 | }, 307 | "nbformat": 4, 308 | "nbformat_minor": 2 309 | } 310 | -------------------------------------------------------------------------------- /A Beginner's Approach to Multi-processing Optimization/single.py: -------------------------------------------------------------------------------- 1 | @profile 2 | def read_velodyne_data(file_path): 3 | """ 4 | Read velodyne binary data and return a numpy array 5 | """ 6 | 7 | # First, check the size of this file to see if it's a valid velodyne binary file 8 | size = os.stat(file_path).st_size 9 | if size % 16 != 0: 10 | raise Exception('The size of '+file_path+' is not dividible by 16 bytes') 11 | 12 | with open(file_path, 'rb') as f: 13 | # Allocate memory for numpy array 14 | velodyne_data = np.empty(shape=(size//16, 4), dtype=np.float32) 15 | 16 | # Read the data, 16 bytes each time 17 | i = 0 18 | reader = BufferedReader(f) 19 | while reader.peek(16): 20 | read_bytes = reader.read(16) 21 | velodyne_data[i] = np.frombuffer(read_bytes, dtype=np.float32) 22 | i += 1 23 | 24 | # Check whether correct amount of bytes were read 25 | if i != size/16: 26 | error = ' '.join(['The file size is', str(size), ', but', str(i), 'bytes were read']) 27 | raise Exception(error) 28 | 29 | return velodyne_data 30 | 31 | 32 | def bird_view_map(velodyne_data): 33 | """ 34 | Implements the method in https://arxiv.org/pdf/1611.07759.pdf 35 | :param velodyne_data: a list of velodyne cloud points 36 | :return: 2D image with 3 channels: height, intensity and density 37 | """ 38 | bird_view = np.zeros(shape=(1600, 1600, 3), dtype=np.float32) 39 | for point in velodyne_data: 40 | x = point[0] 41 | y = point[1] 42 | z = point[2] 43 | r = point[3] 44 | # if (-40 <= x <= 40) and (-40 <= y <= 40): 45 | xi = 800 - np.int(np.ceil(x/0.1)) 46 | yi = 800 - np.int(np.ceil(y/0.1)) 47 | if -z > bird_view[yi][xi][0]: 48 | bird_view[yi][xi][0] = -z 49 | bird_view[yi][xi][1] = r 50 | bird_view[yi][xi][2] += 1 51 | 52 | # todo: normalize birdview with the real method 53 | bird_view[:,:,0] = np.interp(bird_view[:,:,0], xp=(np.min(bird_view[:,:,0]), np.max(bird_view[:,:,0])), fp=(0, 255)) 54 | bird_view[:,:,1] = np.interp(bird_view[:,:,1], xp=(0, 1), fp=(0, 255)) 55 | bird_view[:,:,2] = np.interp(bird_view[:,:,2], xp=(np.min(bird_view[:,:,2]), np.max(bird_view[:,:,2])), fp=(0, 255)) 56 | return bird_view 57 | 58 | 59 | # You might wonder why I import stuff here. I know it's not a good practice. But it's just a demo 60 | # If you wonder why it's still valid Python code, take a look at how Python evaluate code 61 | from glob import glob 62 | import cv2 63 | import os 64 | import numpy as np 65 | from io import BufferedReader 66 | import time 67 | 68 | 69 | paths = glob('data/raw/*.bin')[:10] 70 | t = time.time() 71 | for path in paths: 72 | data = read_velodyne_data(path) 73 | view = bird_view_map(data) 74 | cv2.imwrite(''.join(['data/processed/', os.path.basename(path)[:-4], '.png']), view) 75 | used = time.time() - t 76 | print('Single processed version used', used, 'seconds to process', len(paths), 'files') 77 | -------------------------------------------------------------------------------- /A Beginner's Approach to Multi-processing Optimization/single.py.lprof: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CreatCodeBuild/Python-Notes/dc07d56a30adf89e27c4d97109e9c07fac13d0b3/A Beginner's Approach to Multi-processing Optimization/single.py.lprof -------------------------------------------------------------------------------- /Python HTTP2 Experiments/README.md: -------------------------------------------------------------------------------- 1 | # Python HTTP2 Web Server Framework 2 | 3 | I had heard of that HTTP2 was the future since the end of 2015 and I had watached some Python talks about HTTP2 in PyConf2016. 4 | 5 | But one year has passed, I have not found any full HTTP2 server framework yet. 6 | 7 | Also, async programming has been widely talked about but not many people seemed to actually do it. 8 | (Unless you consider JavaScript people, where they have no choice) 9 | 10 | Furthermore, generator style async io is much more fun than callback style. 11 | 12 | April 23rd, 2017, Sunday, I woke up and felt bored and decided to make one of my own. Since I had little prior web knowledge, this would be a super fun project for me. 13 | -------------------------------------------------------------------------------- /Python HTTP2 Experiments/app.py: -------------------------------------------------------------------------------- 1 | from h2web import app 2 | 3 | if __name__ == '__main__': 4 | 5 | # A basic callback style API is provided 6 | async def get_name(handler): 7 | print('GET name hit') 8 | await handler.send_and_end('GET name hit') 9 | 10 | app = app.App() 11 | app.get('name', get_name) 12 | app.up() 13 | -------------------------------------------------------------------------------- /Python HTTP2 Experiments/h2web/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CreatCodeBuild/Python-Notes/dc07d56a30adf89e27c4d97109e9c07fac13d0b3/Python HTTP2 Experiments/h2web/__init__.py -------------------------------------------------------------------------------- /Python HTTP2 Experiments/h2web/app.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from curio import Kernel 4 | 5 | from h2web.curio_server import h2_server 6 | 7 | 8 | class App: 9 | def __init__(self, port=5000, root='./public'): 10 | self.port = port 11 | self.root = os.path.abspath(root) 12 | print(self.root) 13 | self.server = None 14 | self.routes = {'GET': {}, 'POST': {}} 15 | 16 | def up(self): 17 | kernel = Kernel() 18 | print("Try GETting:") 19 | print(" (Accept all the warnings)") 20 | kernel.run(h2_server(address=("localhost", self.port), 21 | root=self.root, 22 | certfile="{}.crt.pem".format("localhost"), 23 | keyfile="{}.key".format("localhost"), 24 | app=self)) 25 | 26 | def register_route(self, method: str, route: str, handler): 27 | assert method in ['GET', 'POST'] 28 | self.routes[method][route] = handler 29 | 30 | def get(self, route: str, handler=None): 31 | self.register_route('GET', route, handler) 32 | -------------------------------------------------------------------------------- /Python HTTP2 Experiments/h2web/curio_server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3.5 2 | # -*- coding: utf-8 -*- 3 | """ 4 | curio_server.py 5 | ~~~~~~~~~~~~~~~ 6 | 7 | A fully-functional HTTP/2 server written for curio. 8 | 9 | Requires Python 3.5+. 10 | """ 11 | import mimetypes 12 | import os 13 | import sys 14 | 15 | from curio import Kernel, Event, spawn, socket, ssl 16 | 17 | import h2.config 18 | import h2.connection 19 | import h2.events 20 | 21 | 22 | # The maximum amount of a file we'll send in a single DATA frame. 23 | READ_CHUNK_SIZE = 8192 24 | 25 | 26 | class EndPointHandler: 27 | def __init__(self, sock, connection: h2.connection.H2Connection, header, stream_id): 28 | self.socket = sock 29 | self.connection = connection 30 | self.header = header 31 | self.stream_id = stream_id 32 | 33 | async def send_and_end(self, data): 34 | 35 | # Header 36 | content_type, content_encoding = mimetypes.guess_type(data) 37 | data = bytes(data, encoding='utf8') 38 | response_headers = [ 39 | (':status', '200'), 40 | ('content-length', str(len(data))), 41 | ('server', 'curio-h2'), 42 | ] 43 | if content_type: 44 | response_headers.append(('content-type', content_type)) 45 | if content_encoding: 46 | response_headers.append(('content-encoding', content_encoding)) 47 | 48 | self.connection.send_headers(self.stream_id, response_headers) 49 | await self.socket.sendall(self.connection.data_to_send()) 50 | 51 | # Body 52 | self.connection.send_data(self.stream_id, bytes(data), end_stream=True) 53 | await self.socket.sendall(self.connection.data_to_send()) 54 | 55 | 56 | def create_listening_ssl_socket(address, certfile, keyfile): 57 | """ 58 | Create and return a listening TLS socket on a given address. 59 | """ 60 | ssl_context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) 61 | ssl_context.options |= ( 62 | ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_COMPRESSION 63 | ) 64 | ssl_context.set_ciphers("ECDHE+AESGCM") 65 | ssl_context.load_cert_chain(certfile=certfile, keyfile=keyfile) 66 | ssl_context.set_alpn_protocols(["h2"]) 67 | 68 | sock = socket.socket() 69 | sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 70 | sock = ssl_context.wrap_socket(sock) 71 | sock.bind(address) 72 | sock.listen() 73 | 74 | return sock 75 | 76 | 77 | async def h2_server(address, root, certfile, keyfile, app): 78 | """ 79 | Create an HTTP/2 server at the given address. 80 | """ 81 | sock = create_listening_ssl_socket(address, certfile, keyfile) 82 | print("Now listening on %s:%d" % address) 83 | 84 | async with sock: 85 | while True: 86 | client, _ = await sock.accept() 87 | server = H2Server(client, root, app) 88 | app.server = server 89 | await spawn(server.run()) 90 | 91 | 92 | class H2Server: 93 | """ 94 | A basic HTTP/2 file server. This is essentially very similar to 95 | SimpleHTTPServer from the standard library, but uses HTTP/2 instead of 96 | HTTP/1.1. 97 | """ 98 | def __init__(self, sock, root, app): 99 | config = h2.config.H2Configuration(client_side=False, header_encoding='utf-8') 100 | self.sock = sock 101 | self.conn = h2.connection.H2Connection(config=config) 102 | self.root = root 103 | self.flow_control_events = {} 104 | 105 | # the Application that needs this server 106 | # this server runs this app 107 | self.app = app 108 | 109 | async def run(self): 110 | """ 111 | Loop over the connection, managing it appropriately. 112 | """ 113 | self.conn.initiate_connection() 114 | await self.sock.sendall(self.conn.data_to_send()) 115 | 116 | while True: 117 | # 65535 is basically arbitrary here: this amounts to "give me 118 | # whatever data you have". 119 | data = await self.sock.recv(65535) 120 | if not data: 121 | break 122 | 123 | events = self.conn.receive_data(data) 124 | for event in events: 125 | 126 | if isinstance(event, h2.events.RequestReceived): 127 | await spawn(self.request_received(event.headers, event.stream_id)) 128 | 129 | elif isinstance(event, h2.events.DataReceived): 130 | self.conn.reset_stream(event.stream_id) 131 | 132 | elif isinstance(event, h2.events.WindowUpdated): 133 | await self.window_updated(event) 134 | 135 | await self.sock.sendall(self.conn.data_to_send()) 136 | 137 | async def request_received(self, headers, stream_id): 138 | """ 139 | Handle a request by attempting to serve a suitable file. 140 | """ 141 | headers = dict(headers) 142 | for k, v in headers.items(): 143 | print(k, v) 144 | assert headers[':method'] == 'GET' 145 | 146 | if headers[':method'] == 'GET': 147 | route = headers[':path'].lstrip('/') 148 | 149 | if route in self.app.routes['GET']: 150 | await self.app.routes['GET'][route](EndPointHandler(self.sock, self.conn, headers, stream_id)) 151 | 152 | # if route is not registered, assume it is requesting files 153 | else: 154 | full_path = os.path.join(self.root, route) 155 | if not os.path.exists(full_path): 156 | response_headers = ( 157 | (':status', '404'), 158 | ('content-length', '0'), 159 | ('server', 'curio-h2'), 160 | ) 161 | self.conn.send_headers( 162 | stream_id, response_headers, end_stream=True 163 | ) 164 | await self.sock.sendall(self.conn.data_to_send()) 165 | else: 166 | await self.send_file(full_path, stream_id) 167 | 168 | else: 169 | raise NotImplementedError('Only GET is implemented') 170 | 171 | 172 | async def send_file(self, file_path, stream_id): 173 | """ 174 | Send a file, obeying the rules of HTTP/2 flow control. 175 | """ 176 | filesize = os.stat(file_path).st_size 177 | content_type, content_encoding = mimetypes.guess_type(file_path) 178 | response_headers = [ 179 | (':status', '200'), 180 | ('content-length', str(filesize)), 181 | ('server', 'curio-h2'), 182 | ] 183 | if content_type: 184 | response_headers.append(('content-type', content_type)) 185 | if content_encoding: 186 | response_headers.append(('content-encoding', content_encoding)) 187 | 188 | self.conn.send_headers(stream_id, response_headers) 189 | await self.sock.sendall(self.conn.data_to_send()) 190 | 191 | with open(file_path, 'rb', buffering=0) as f: 192 | await self._send_file_data(f, stream_id) 193 | 194 | async def _send_file_data(self, fileobj, stream_id): 195 | """ 196 | Send the data portion of a file. Handles flow control rules. 197 | """ 198 | while True: 199 | while not self.conn.local_flow_control_window(stream_id): 200 | await self.wait_for_flow_control(stream_id) 201 | 202 | chunk_size = min( 203 | self.conn.local_flow_control_window(stream_id), 204 | READ_CHUNK_SIZE, 205 | ) 206 | 207 | data = fileobj.read(chunk_size) 208 | keep_reading = (len(data) == chunk_size) 209 | 210 | self.conn.send_data(stream_id, data, not keep_reading) 211 | await self.sock.sendall(self.conn.data_to_send()) 212 | 213 | if not keep_reading: 214 | break 215 | 216 | async def wait_for_flow_control(self, stream_id): 217 | """ 218 | Blocks until the flow control window for a given stream is opened. 219 | """ 220 | evt = Event() 221 | self.flow_control_events[stream_id] = evt 222 | await evt.wait() 223 | 224 | async def window_updated(self, event): 225 | """ 226 | Unblock streams waiting on flow control, if needed. 227 | """ 228 | stream_id = event.stream_id 229 | 230 | if stream_id and stream_id in self.flow_control_events: 231 | evt = self.flow_control_events.pop(stream_id) 232 | await evt.set() 233 | elif not stream_id: 234 | # Need to keep a real list here to use only the events present at 235 | # this time. 236 | blocked_streams = list(self.flow_control_events.keys()) 237 | for stream_id in blocked_streams: 238 | event = self.flow_control_events.pop(stream_id) 239 | await event.set() 240 | return 241 | 242 | 243 | if __name__ == '__main__': 244 | host = sys.argv[2] if len(sys.argv) > 2 else "localhost" 245 | print(sys.argv) 246 | kernel = Kernel() 247 | print("Try GETting:") 248 | print(" On OSX after 'brew install curl --with-c-ares --with-libidn --with-nghttp2 --with-openssl':") 249 | print("/usr/local/opt/curl/bin/curl --tlsv1.2 --http2 -k https://localhost:5000/bundle.js") 250 | print("Or open a browser to: https://localhost:5000/") 251 | print(" (Accept all the warnings)") 252 | kernel.run(h2_server((host, 5000), 253 | sys.argv[1], 254 | "{}.crt.pem".format(host), 255 | "{}.key".format(host))) 256 | -------------------------------------------------------------------------------- /Python HTTP2 Experiments/localhost.crt.pem: -------------------------------------------------------------------------------- 1 | -----BEGIN CERTIFICATE----- 2 | MIIDhTCCAm2gAwIBAgIJAOrxh0dOYJLdMA0GCSqGSIb3DQEBCwUAMFkxCzAJBgNV 3 | BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX 4 | aWRnaXRzIFB0eSBMdGQxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xNTA5MTkxNDE2 5 | NDRaFw0xNTEwMTkxNDE2NDRaMFkxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21l 6 | LVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQxEjAQBgNV 7 | BAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMqt 8 | A1iu8EN00FU0eBcBGlLVmNEgV7Jkbukra+kwS8j/U2y50QPGJc/FiIVDfuBqk5dL 9 | ACTNc6A/FQcXvWmOc5ixmC3QKKasMpuofqKz0V9C6irZdYXZ9rcsW0gHQIr989yd 10 | R+N1VbIlEVW/T9FJL3B2UD9GVIkUELzm47CSOWZvAxQUlsx8CUNuUCWqyZJoqTFN 11 | j0LeJDOWGCsug1Pkj0Q1x+jMVL6l6Zf6vMkLNOMsOsWsxUk+0L3tl/OzcTgUOCsw 12 | UzY59RIi6Rudrp0oaU8NuHr91yiSqPbKFlX10M9KwEEdnIpcxhND3dacrDycj3ux 13 | eWlqKync2vOFUkhwiaMCAwEAAaNQME4wHQYDVR0OBBYEFA0PN+PGoofZ+QIys2Jy 14 | 1Zz94vBOMB8GA1UdIwQYMBaAFA0PN+PGoofZ+QIys2Jy1Zz94vBOMAwGA1UdEwQF 15 | MAMBAf8wDQYJKoZIhvcNAQELBQADggEBAEplethBoPpcP3EbR5Rz6snDDIcbtAJu 16 | Ngd0YZppGT+P0DYnPJva4vRG3bb84ZMSuppz5j67qD6DdWte8UXhK8BzWiHzwmQE 17 | QmbKyzzTMKQgTNFntpx5cgsSvTtrHpNYoMHzHOmyAOboNeM0DWiRXsYLkWTitLTN 18 | qbOpstwPubExbT9lPjLclntShT/lCupt+zsbnrR9YiqlYFY/fDzfAybZhrD5GMBY 19 | XdMPItwAc/sWvH31yztarjkLmld76AGCcO5r8cSR/cX98SicyfjOBbSco8GkjYNY 20 | 582gTPkKGYpStuN7GNT5tZmxvMq935HRa2XZvlAIe8ufp8EHVoYiF3c= 21 | -----END CERTIFICATE----- 22 | -------------------------------------------------------------------------------- /Python HTTP2 Experiments/localhost.key: -------------------------------------------------------------------------------- 1 | -----BEGIN RSA PRIVATE KEY----- 2 | MIIEpAIBAAKCAQEAyq0DWK7wQ3TQVTR4FwEaUtWY0SBXsmRu6Str6TBLyP9TbLnR 3 | A8Ylz8WIhUN+4GqTl0sAJM1zoD8VBxe9aY5zmLGYLdAopqwym6h+orPRX0LqKtl1 4 | hdn2tyxbSAdAiv3z3J1H43VVsiURVb9P0UkvcHZQP0ZUiRQQvObjsJI5Zm8DFBSW 5 | zHwJQ25QJarJkmipMU2PQt4kM5YYKy6DU+SPRDXH6MxUvqXpl/q8yQs04yw6xazF 6 | ST7Qve2X87NxOBQ4KzBTNjn1EiLpG52unShpTw24ev3XKJKo9soWVfXQz0rAQR2c 7 | ilzGE0Pd1pysPJyPe7F5aWorKdza84VSSHCJowIDAQABAoIBACp+nh4BB/VMz8Wd 8 | q7Q/EfLeQB1Q57JKpoqTBRwueSVai3ZXe4CMEi9/HkG6xiZtkiZ9njkZLq4hq9oB 9 | 2z//kzMnwV2RsIRJxI6ohGy+wR51HD4BvEdlTPpY/Yabpqe92VyfSYxidKZWaU0O 10 | QMED1EODOw4ZQ+4928iPrJu//PMB4e7TFao0b9Fk/XLWtu5/tQZz9jsrlTi1zthh 11 | 7n+oaGNhfTeIJJL4jrhTrKW1CLHXATtr9SJlfZ3wbMxQVeyj2wUlP1V0M6kBuhNj 12 | tbGbMpixD5iCNJ49Cm2PHg+wBOfS3ADGIpi3PcGw5mb8nB3N9eGBRPhLShAlq5Hi 13 | Lv4tyykCgYEA8u3b3xJ04pxWYN25ou/Sc8xzgDCK4XvDNdHVTuZDjLVA+VTVPzql 14 | lw7VvJArsx47MSPvsaX/+4hQXYtfnR7yJpx6QagvQ+z4ludnIZYrQwdUmb9pFL1s 15 | 8UNj+3j9QFRPenIiIQ8qxxNIQ9w2HsVQ8scvc9CjYop/YYAPaQyHaL8CgYEA1ZSz 16 | CR4NcpfgRSILdhb1dLcyw5Qus1VOSAx3DYkhDkMiB8XZwgMdJjwehJo9yaqRCLE8 17 | Sw5znMnkfoZpu7+skrjK0FqmMpXMH9gIszHvFG8wSw/6+2HIWS19/wOu8dh95LuC 18 | 0zurMk8rFqxgWMWF20afhgYrUz42cvUTo10FVB0CgYEAt7mW6W3PArfUSCxIwmb4 19 | VmXREKkl0ATHDYQl/Cb//YHzot467TgQll883QB4XF5HzBFurX9rSzO7/BN1e6I0 20 | 52i+ubtWC9xD4fUetXMaQvZfUGxIL8xXgVxDWKQXfLiG54c8Mp6C7s6xf8kjEUCP 21 | yR1F0SSA/Pzb+8RbY0p7eocCgYA+1rs+SXtHZev0KyoYGnUpW+Uxqd17ofOgOxqj 22 | /t6c5Z+TjeCdtnDTGQkZlo/rT6XQWuUUaDIXxUbW+xEMzj4mBPyXBLS1WWFvVQ5q 23 | OpzO9E/PJeqAH6rkof/aEelc+oc/zvOU1o9uA+D3kMvgEm1psIOq2RHSMhGvDPA0 24 | NmAk+QKBgQCwd1681GagdIYSZUCBecnLtevXmIsJyDW2yR1NNcIe/ukcVQREMDvy 25 | 5DDkhnGDgnV1D5gYcXb34g9vYvbfTnBMl/JXmMAAG1kIS+3pvHyN6f1poVe3yJV1 26 | yHVuvymnJxKnyaV0L3ntepVvV0vVNIkA3oauoUTLto6txBI+b/ImDA== 27 | -----END RSA PRIVATE KEY----- 28 | --------------------------------------------------------------------------------