![](\"images/euroscipy_logo.png\"){kind=logo}
\n",
10 | "\n",
11 | " "
12 | ]
13 | },
14 | {
15 | "cell_type": "markdown",
16 | "metadata": {},
17 | "source": [
18 | "## Goal of this Tutorial\n",
19 | "\n",
20 | "- Introduce the basics of scientific and numerical computation in Python using **Numpy**\n",
21 | "- Understand why `numpy` has a central role in the Python scientific ecosystem\n",
22 | " "
23 | ]
24 | },
25 | {
26 | "cell_type": "markdown",
27 | "metadata": {},
28 | "source": [
29 | "# Outline\n",
30 | "\n",
31 | "**11:00 - 11:45** (_45 mins_) Numpy Basics\n",
32 | "\n",
33 | "- Introduction to NumPy Arrays\n",
34 | " - numpy internals schematics\n",
35 | " - Reshaping and Resizing\n",
36 | "- Numerical Data Types\n",
37 | " - Record Array\n",
38 | " \n",
39 | "**11:50 - 12:30** (_40 mins_) Indexing and Slicing\n",
40 | " \n",
41 | "- Indexing numpy arrays\n",
42 | " - fancy indexing\n",
43 | " - array masking\n",
44 | "- Slicing & Stacking\n",
45 | "- Vectorization & Broadcasting\n",
46 | "\n",
47 | "**Follow up** \"Advanced NumPy: Bits of Data Science with NumPy\n",
48 | "\n",
49 | "- Serialisation & I/O\n",
50 | " - `.mat` files\n",
51 | "- Array and Matrix\n",
52 | " - Matlab compatibility\n",
53 | "- Sparse Matrices\n",
54 | "- Memmap \n",
55 | "- Ubiquitous NumPy: NumPy beyond `numpy`"
56 | ]
57 | },
58 | {
59 | "cell_type": "markdown",
60 | "metadata": {},
61 | "source": [
62 | "# Requirements"
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "metadata": {},
68 | "source": [
69 | "This tutorial has one main requirement: `numpy`.\n",
70 | "\n",
71 | "Materials are provided as Jupyter notebooks, so IPython notebook (`pip install notebook`) is also required.\n",
72 | "\n",
73 | "#### Advanced Part\n",
74 | "\n",
75 | "This part has more dependencies: `scipy`, `scikit-learn`, `matplotlib`, `torch`.\n",
76 | "All these dependencies have been collected in the `requirements.txt` file:\n",
77 | "\n",
78 | "```\n",
79 | "$ pip install -r requirements.txt\n",
80 | "```\n",
81 | "\n",
82 | "\n",
83 | "### Python version\n",
84 | "\n",
85 | "The minimum recommended version of Python to use for this tutorial is **Python 3.5**, although \n",
86 | "Python 2.7 should be fine, as well as previous versions of Python 3. \n",
87 | "\n",
88 | "Py3.5+ is recommended due to a reference to the `@` operator in the linear algebra notebook.\n",
89 | "\n",
90 | "\n"
91 | ]
92 | },
93 | {
94 | "cell_type": "markdown",
95 | "metadata": {},
96 | "source": [
97 | "# MyBinder\n",
98 | "\n",
99 | "If you don't want to bother setting up the environment on your own computer, you can use MyBinder\n",
100 | "\n",
101 | "(**Note**: recommended only with a proper Wi-Fi connection)"
102 | ]
103 | },
104 | {
105 | "cell_type": "markdown",
106 | "metadata": {},
107 | "source": [
108 | "[](https://mybinder.org/v2/gh/leriomaggio/numpy-euroscipy/master)"
109 | ]
110 | }
111 | ],
112 | "metadata": {
113 | "kernelspec": {
114 | "display_name": "Python 3.7 (NumPy EuroSciPy)",
115 | "language": "python",
116 | "name": "numpy-euroscipy"
117 | },
118 | "language_info": {
119 | "codemirror_mode": {
120 | "name": "ipython",
121 | "version": 3
122 | },
123 | "file_extension": ".py",
124 | "mimetype": "text/x-python",
125 | "name": "python",
126 | "nbconvert_exporter": "python",
127 | "pygments_lexer": "ipython3",
128 | "version": "3.7.3"
129 | }
130 | },
131 | "nbformat": 4,
132 | "nbformat_minor": 4
133 | }
134 |
--------------------------------------------------------------------------------
/06_numpy_internals.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Understanding NumPy Internals\n",
8 | "\n",
9 | "We can achieve significant performance speed enhancement with NumPy over native Python code, particularly when our computations follow the **Single Instruction, Multiple Data (SIMD)** paradigm. "
10 | ]
11 | },
12 | {
13 | "cell_type": "code",
14 | "execution_count": 1,
15 | "metadata": {},
16 | "outputs": [],
17 | "source": [
18 | "import numpy as np"
19 | ]
20 | },
21 | {
22 | "cell_type": "markdown",
23 | "metadata": {
24 | "slideshow": {
25 | "slide_type": "slide"
26 | }
27 | },
28 | "source": [
29 | "## Copy and \"deep copy\""
30 | ]
31 | },
32 | {
33 | "cell_type": "markdown",
34 | "metadata": {
35 | "slideshow": {
36 | "slide_type": "subslide"
37 | }
38 | },
39 | "source": [
40 | "To achieve high performance, assignments in Python usually do not copy the underlaying objects. \n",
41 | "\n",
42 | "This is important for example when objects are passed between functions, to avoid an excessive amount of memory copying when it is not necessary (techincal term: **pass by reference**).\n",
43 | "\n",
44 | "![](\"images/reference.png\")
"
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "metadata": {},
50 | "source": [
51 | "First, we need a way to check whether two arrays share the same underlying data buffer in memory. \n",
52 | "\n",
53 | "Let's define a function `aid()` that returns the memory location of the underlying data buffer:"
54 | ]
55 | },
56 | {
57 | "cell_type": "code",
58 | "execution_count": 3,
59 | "metadata": {},
60 | "outputs": [],
61 | "source": [
62 | "def aid(x):\n",
63 | " # This function returns the memory\n",
64 | " # block address of an array.\n",
65 | " return x.__array_interface__['data'][0]"
66 | ]
67 | },
68 | {
69 | "cell_type": "markdown",
70 | "metadata": {},
71 | "source": [
72 | "Two arrays with the same data location (as returned by `aid()`) share the same underlying data buffer. \n",
73 | "\n",
74 | "However, the opposite is true only if the arrays have the same offset (meaning that they have the same first element). "
75 | ]
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": 4,
80 | "metadata": {
81 | "collapsed": false,
82 | "jupyter": {
83 | "outputs_hidden": false
84 | },
85 | "slideshow": {
86 | "slide_type": "subslide"
87 | }
88 | },
89 | "outputs": [
90 | {
91 | "data": {
92 | "text/plain": [
93 | "array([[1, 2],\n",
94 | " [3, 4]])"
95 | ]
96 | },
97 | "execution_count": 4,
98 | "metadata": {},
99 | "output_type": "execute_result"
100 | }
101 | ],
102 | "source": [
103 | "A = np.array([[1, 2], [3, 4]])\n",
104 | "\n",
105 | "A"
106 | ]
107 | },
108 | {
109 | "cell_type": "code",
110 | "execution_count": 5,
111 | "metadata": {
112 | "collapsed": false,
113 | "jupyter": {
114 | "outputs_hidden": false
115 | },
116 | "slideshow": {
117 | "slide_type": "fragment"
118 | }
119 | },
120 | "outputs": [],
121 | "source": [
122 | "# now B is referring to the same array data as A \n",
123 | "B = A "
124 | ]
125 | },
126 | {
127 | "cell_type": "code",
128 | "execution_count": 6,
129 | "metadata": {},
130 | "outputs": [
131 | {
132 | "data": {
133 | "text/plain": [
134 | "True"
135 | ]
136 | },
137 | "execution_count": 6,
138 | "metadata": {},
139 | "output_type": "execute_result"
140 | }
141 | ],
142 | "source": [
143 | "aid(A) == aid(B)"
144 | ]
145 | },
146 | {
147 | "cell_type": "code",
148 | "execution_count": 7,
149 | "metadata": {
150 | "collapsed": false,
151 | "jupyter": {
152 | "outputs_hidden": false
153 | },
154 | "slideshow": {
155 | "slide_type": "fragment"
156 | }
157 | },
158 | "outputs": [
159 | {
160 | "data": {
161 | "text/plain": [
162 | "array([[10, 2],\n",
163 | " [ 3, 4]])"
164 | ]
165 | },
166 | "execution_count": 7,
167 | "metadata": {},
168 | "output_type": "execute_result"
169 | }
170 | ],
171 | "source": [
172 | "# changing B affects A\n",
173 | "B[0,0] = 10\n",
174 | "\n",
175 | "B"
176 | ]
177 | },
178 | {
179 | "cell_type": "code",
180 | "execution_count": 8,
181 | "metadata": {
182 | "collapsed": false,
183 | "jupyter": {
184 | "outputs_hidden": false
185 | },
186 | "slideshow": {
187 | "slide_type": "fragment"
188 | }
189 | },
190 | "outputs": [
191 | {
192 | "data": {
193 | "text/plain": [
194 | "array([[10, 2],\n",
195 | " [ 3, 4]])"
196 | ]
197 | },
198 | "execution_count": 8,
199 | "metadata": {},
200 | "output_type": "execute_result"
201 | }
202 | ],
203 | "source": [
204 | "A"
205 | ]
206 | },
207 | {
208 | "cell_type": "markdown",
209 | "metadata": {
210 | "slideshow": {
211 | "slide_type": "subslide"
212 | }
213 | },
214 | "source": [
215 | "* If we want to **avoid** this behavior, so that when we get a new completely independent object `B` copied from `A`, then we need to do a so-called **deep copy** using the function `np.copy`:"
216 | ]
217 | },
218 | {
219 | "cell_type": "code",
220 | "execution_count": 9,
221 | "metadata": {
222 | "collapsed": false,
223 | "jupyter": {
224 | "outputs_hidden": false
225 | },
226 | "slideshow": {
227 | "slide_type": "fragment"
228 | }
229 | },
230 | "outputs": [],
231 | "source": [
232 | "B = np.copy(A)"
233 | ]
234 | },
235 | {
236 | "cell_type": "code",
237 | "execution_count": 10,
238 | "metadata": {
239 | "collapsed": false,
240 | "jupyter": {
241 | "outputs_hidden": false
242 | },
243 | "slideshow": {
244 | "slide_type": "fragment"
245 | }
246 | },
247 | "outputs": [
248 | {
249 | "data": {
250 | "text/plain": [
251 | "array([[-5, 2],\n",
252 | " [ 3, 4]])"
253 | ]
254 | },
255 | "execution_count": 10,
256 | "metadata": {},
257 | "output_type": "execute_result"
258 | }
259 | ],
260 | "source": [
261 | "# now, if we modify B, A is not affected\n",
262 | "B[0,0] = -5\n",
263 | "\n",
264 | "B"
265 | ]
266 | },
267 | {
268 | "cell_type": "code",
269 | "execution_count": 11,
270 | "metadata": {
271 | "collapsed": false,
272 | "jupyter": {
273 | "outputs_hidden": false
274 | },
275 | "slideshow": {
276 | "slide_type": "fragment"
277 | }
278 | },
279 | "outputs": [
280 | {
281 | "data": {
282 | "text/plain": [
283 | "array([[10, 2],\n",
284 | " [ 3, 4]])"
285 | ]
286 | },
287 | "execution_count": 11,
288 | "metadata": {},
289 | "output_type": "execute_result"
290 | }
291 | ],
292 | "source": [
293 | "A"
294 | ]
295 | },
296 | {
297 | "cell_type": "code",
298 | "execution_count": 13,
299 | "metadata": {},
300 | "outputs": [
301 | {
302 | "data": {
303 | "text/plain": [
304 | "False"
305 | ]
306 | },
307 | "execution_count": 13,
308 | "metadata": {},
309 | "output_type": "execute_result"
310 | }
311 | ],
312 | "source": [
313 | "aid(A) == aid(B)"
314 | ]
315 | },
316 | {
317 | "cell_type": "markdown",
318 | "metadata": {},
319 | "source": [
320 | "---"
321 | ]
322 | },
323 | {
324 | "cell_type": "code",
325 | "execution_count": null,
326 | "metadata": {},
327 | "outputs": [],
328 | "source": []
329 | }
330 | ],
331 | "metadata": {
332 | "kernelspec": {
333 | "display_name": "Python 3.7 (NumPy EuroSciPy)",
334 | "language": "python",
335 | "name": "numpy-euroscipy"
336 | },
337 | "language_info": {
338 | "codemirror_mode": {
339 | "name": "ipython",
340 | "version": 3
341 | },
342 | "file_extension": ".py",
343 | "mimetype": "text/x-python",
344 | "name": "python",
345 | "nbconvert_exporter": "python",
346 | "pygments_lexer": "ipython3",
347 | "version": "3.7.3"
348 | }
349 | },
350 | "nbformat": 4,
351 | "nbformat_minor": 4
352 | }
353 |
--------------------------------------------------------------------------------
/04_sparse_matrices.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | ""
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {
13 | "slideshow": {
14 | "slide_type": "slide"
15 | }
16 | },
17 | "source": [
18 | "# Scipy Sparse Matrices"
19 | ]
20 | },
21 | {
22 | "cell_type": "markdown",
23 | "metadata": {
24 | "slideshow": {
25 | "slide_type": "subslide"
26 | }
27 | },
28 | "source": [
29 | "**Sparse Matrices** are very nice in some situations. \n",
30 | "\n",
31 | "For example, in some machine learning tasks, especially those associated\n",
32 | "with textual analysis, the data may be mostly zeros. \n",
33 | "\n",
34 | "Storing all these zeros is very inefficient. \n",
35 | "\n",
36 | "We can create and manipulate sparse matrices as follows:"
37 | ]
38 | },
39 | {
40 | "cell_type": "code",
41 | "execution_count": 1,
42 | "metadata": {
43 | "slideshow": {
44 | "slide_type": "skip"
45 | }
46 | },
47 | "outputs": [],
48 | "source": [
49 | "import numpy as np"
50 | ]
51 | },
52 | {
53 | "cell_type": "code",
54 | "execution_count": 2,
55 | "metadata": {
56 | "collapsed": false,
57 | "jupyter": {
58 | "outputs_hidden": false
59 | },
60 | "slideshow": {
61 | "slide_type": "subslide"
62 | }
63 | },
64 | "outputs": [
65 | {
66 | "name": "stdout",
67 | "output_type": "stream",
68 | "text": [
69 | "[[0.52508939 0.55969684 0.38059541 0.14994033 0.3561533 ]\n",
70 | " [0.94612104 0.20796991 0.18345058 0.03266521 0.71642811]\n",
71 | " [0.76801146 0.18143891 0.44346617 0.3509763 0.70771478]\n",
72 | " [0.96785438 0.64010409 0.20666769 0.99005094 0.42858088]\n",
73 | " [0.24971981 0.88585392 0.1683662 0.70119483 0.48374682]\n",
74 | " [0.01736319 0.87369042 0.19830546 0.56395574 0.20060824]\n",
75 | " [0.11881578 0.65524562 0.21570217 0.02114718 0.8527528 ]\n",
76 | " [0.7722977 0.44208694 0.01126588 0.80556187 0.07607147]\n",
77 | " [0.75409907 0.78761663 0.41863968 0.30373673 0.63332945]\n",
78 | " [0.99874432 0.37336682 0.14359151 0.76142434 0.1988419 ]]\n"
79 | ]
80 | }
81 | ],
82 | "source": [
83 | "# Create a random array with a lot of zeros\n",
84 | "X = np.random.random((10, 5))\n",
85 | "print(X)"
86 | ]
87 | },
88 | {
89 | "cell_type": "code",
90 | "execution_count": 4,
91 | "metadata": {
92 | "collapsed": false,
93 | "jupyter": {
94 | "outputs_hidden": false
95 | },
96 | "slideshow": {
97 | "slide_type": "subslide"
98 | }
99 | },
100 | "outputs": [
101 | {
102 | "name": "stdout",
103 | "output_type": "stream",
104 | "text": [
105 | "[[0. 0. 0. 0. 0. ]\n",
106 | " [0.94612104 0. 0. 0. 0.71642811]\n",
107 | " [0.76801146 0. 0. 0. 0.70771478]\n",
108 | " [0.96785438 0. 0. 0.99005094 0. ]\n",
109 | " [0. 0.88585392 0. 0.70119483 0. ]\n",
110 | " [0. 0.87369042 0. 0. 0. ]\n",
111 | " [0. 0. 0. 0. 0.8527528 ]\n",
112 | " [0.7722977 0. 0. 0.80556187 0. ]\n",
113 | " [0.75409907 0.78761663 0. 0. 0. ]\n",
114 | " [0.99874432 0. 0. 0.76142434 0. ]]\n"
115 | ]
116 | }
117 | ],
118 | "source": [
119 | "X[X < 0.7] = 0 # note: fancy indexing\n",
120 | "print(X)"
121 | ]
122 | },
123 | {
124 | "cell_type": "code",
125 | "execution_count": 5,
126 | "metadata": {
127 | "collapsed": false,
128 | "jupyter": {
129 | "outputs_hidden": false
130 | },
131 | "slideshow": {
132 | "slide_type": "subslide"
133 | }
134 | },
135 | "outputs": [
136 | {
137 | "name": "stdout",
138 | "output_type": "stream",
139 | "text": [
140 | " (1, 0)\t0.9461210440608149\n",
141 | " (1, 4)\t0.7164281142304602\n",
142 | " (2, 0)\t0.7680114556976801\n",
143 | " (2, 4)\t0.7077147754658187\n",
144 | " (3, 0)\t0.9678543752795629\n",
145 | " (3, 3)\t0.9900509407165115\n",
146 | " (4, 1)\t0.8858539179438214\n",
147 | " (4, 3)\t0.7011948276939008\n",
148 | " (5, 1)\t0.8736904234085155\n",
149 | " (6, 4)\t0.8527528049269587\n",
150 | " (7, 0)\t0.7722977020522017\n",
151 | " (7, 3)\t0.8055618728634483\n",
152 | " (8, 0)\t0.7540990714791828\n",
153 | " (8, 1)\t0.7876166309534933\n",
154 | " (9, 0)\t0.9987443167367364\n",
155 | " (9, 3)\t0.7614243372618548\n"
156 | ]
157 | }
158 | ],
159 | "source": [
160 | "from scipy import sparse\n",
161 | "\n",
162 | "# turn X into a csr (Compressed-Sparse-Row) matrix\n",
163 | "X_csr = sparse.csr_matrix(X)\n",
164 | "print(X_csr)"
165 | ]
166 | },
167 | {
168 | "cell_type": "code",
169 | "execution_count": 6,
170 | "metadata": {
171 | "collapsed": false,
172 | "jupyter": {
173 | "outputs_hidden": false
174 | },
175 | "slideshow": {
176 | "slide_type": "subslide"
177 | }
178 | },
179 | "outputs": [
180 | {
181 | "name": "stdout",
182 | "output_type": "stream",
183 | "text": [
184 | "[[0. 0. 0. 0. 0. ]\n",
185 | " [0.94612104 0. 0. 0. 0.71642811]\n",
186 | " [0.76801146 0. 0. 0. 0.70771478]\n",
187 | " [0.96785438 0. 0. 0.99005094 0. ]\n",
188 | " [0. 0.88585392 0. 0.70119483 0. ]\n",
189 | " [0. 0.87369042 0. 0. 0. ]\n",
190 | " [0. 0. 0. 0. 0.8527528 ]\n",
191 | " [0.7722977 0. 0. 0.80556187 0. ]\n",
192 | " [0.75409907 0.78761663 0. 0. 0. ]\n",
193 | " [0.99874432 0. 0. 0.76142434 0. ]]\n"
194 | ]
195 | }
196 | ],
197 | "source": [
198 | "# convert the sparse matrix to a dense array\n",
199 | "print(X_csr.toarray())"
200 | ]
201 | },
202 | {
203 | "cell_type": "code",
204 | "execution_count": 7,
205 | "metadata": {
206 | "collapsed": false,
207 | "jupyter": {
208 | "outputs_hidden": false
209 | },
210 | "slideshow": {
211 | "slide_type": "subslide"
212 | }
213 | },
214 | "outputs": [
215 | {
216 | "data": {
217 | "text/plain": [
218 | "True"
219 | ]
220 | },
221 | "execution_count": 7,
222 | "metadata": {},
223 | "output_type": "execute_result"
224 | }
225 | ],
226 | "source": [
227 | "# Sparse matrices support linear algebra:\n",
228 | "y = np.random.random(X_csr.shape[1])\n",
229 | "z1 = X_csr.dot(y)\n",
230 | "z2 = X.dot(y)\n",
231 | "np.allclose(z1, z2)"
232 | ]
233 | },
234 | {
235 | "cell_type": "markdown",
236 | "metadata": {
237 | "slideshow": {
238 | "slide_type": "subslide"
239 | }
240 | },
241 | "source": [
242 | "* The CSR representation can be very efficient for computations, but it is not as good for adding elements. \n",
243 | "\n",
244 | "* For that, the **LIL** (List-In-List) representation is better:"
245 | ]
246 | },
247 | {
248 | "cell_type": "code",
249 | "execution_count": 8,
250 | "metadata": {
251 | "collapsed": false,
252 | "jupyter": {
253 | "outputs_hidden": false
254 | },
255 | "slideshow": {
256 | "slide_type": "fragment"
257 | }
258 | },
259 | "outputs": [
260 | {
261 | "name": "stdout",
262 | "output_type": "stream",
263 | "text": [
264 | " (0, 1)\t1.0\n",
265 | " (0, 2)\t2.0\n",
266 | " (1, 1)\t2.0\n",
267 | " (1, 3)\t4.0\n",
268 | " (2, 0)\t2.0\n",
269 | " (2, 1)\t3.0\n",
270 | " (2, 2)\t4.0\n",
271 | " (2, 3)\t5.0\n",
272 | " (3, 0)\t3.0\n",
273 | " (4, 0)\t4.0\n",
274 | " (4, 1)\t5.0\n",
275 | " (4, 2)\t6.0\n",
276 | "[[0. 1. 2. 0. 0.]\n",
277 | " [0. 2. 0. 4. 0.]\n",
278 | " [2. 3. 4. 5. 0.]\n",
279 | " [3. 0. 0. 0. 0.]\n",
280 | " [4. 5. 6. 0. 0.]]\n"
281 | ]
282 | }
283 | ],
284 | "source": [
285 | "# Create an empty LIL matrix and add some items\n",
286 | "X_lil = sparse.lil_matrix((5, 5))\n",
287 | "\n",
288 | "for i, j in np.random.randint(0, 5, (15, 2)):\n",
289 | " X_lil[i, j] = i + j\n",
290 | "\n",
291 | "print(X_lil)\n",
292 | "print(X_lil.toarray())"
293 | ]
294 | },
295 | {
296 | "cell_type": "markdown",
297 | "metadata": {
298 | "slideshow": {
299 | "slide_type": "subslide"
300 | }
301 | },
302 | "source": [
303 | "* Often, once an LIL matrix is created, it is useful to convert it to a CSR format \n",
304 | " * **Note**: many scikit-learn algorithms require CSR or CSC format"
305 | ]
306 | },
307 | {
308 | "cell_type": "code",
309 | "execution_count": 9,
310 | "metadata": {
311 | "collapsed": false,
312 | "jupyter": {
313 | "outputs_hidden": false
314 | },
315 | "slideshow": {
316 | "slide_type": "fragment"
317 | }
318 | },
319 | "outputs": [
320 | {
321 | "name": "stdout",
322 | "output_type": "stream",
323 | "text": [
324 | " (0, 1)\t1.0\n",
325 | " (0, 2)\t2.0\n",
326 | " (1, 1)\t2.0\n",
327 | " (1, 3)\t4.0\n",
328 | " (2, 0)\t2.0\n",
329 | " (2, 1)\t3.0\n",
330 | " (2, 2)\t4.0\n",
331 | " (2, 3)\t5.0\n",
332 | " (3, 0)\t3.0\n",
333 | " (4, 0)\t4.0\n",
334 | " (4, 1)\t5.0\n",
335 | " (4, 2)\t6.0\n"
336 | ]
337 | }
338 | ],
339 | "source": [
340 | "X_csr = X_lil.tocsr()\n",
341 | "print(X_csr)"
342 | ]
343 | },
344 | {
345 | "cell_type": "markdown",
346 | "metadata": {
347 | "slideshow": {
348 | "slide_type": "subslide"
349 | }
350 | },
351 | "source": [
352 | "There are several other sparse formats that can be useful for various problems:\n",
353 | "\n",
354 | "- `CSC` (compressed sparse column)\n",
355 | "- `BSR` (block sparse row)\n",
356 | "- `COO` (coordinate)\n",
357 | "- `DIA` (diagonal)\n",
358 | "- `DOK` (dictionary of keys)"
359 | ]
360 | },
361 | {
362 | "cell_type": "markdown",
363 | "metadata": {
364 | "slideshow": {
365 | "slide_type": "slide"
366 | }
367 | },
368 | "source": [
369 | "## CSC - Compressed Sparse Column\n",
370 | "\n",
371 | "**Advantages of the CSC format**\n",
372 | "\n",
373 | " * efficient arithmetic operations CSC + CSC, CSC * CSC, etc.\n",
374 | " * efficient column slicing\n",
375 | " * fast matrix vector products (CSR, BSR may be faster)\n",
376 | "\n",
377 | "**Disadvantages of the CSC format**\n",
378 | "\n",
379 | " * slow row slicing operations (consider CSR)\n",
380 | " * changes to the sparsity structure are expensive (consider LIL or DOK)"
381 | ]
382 | },
383 | {
384 | "cell_type": "markdown",
385 | "metadata": {
386 | "slideshow": {
387 | "slide_type": "subslide"
388 | }
389 | },
390 | "source": [
391 | "### BSR - Block Sparse Row\n",
392 | "\n",
393 | "The Block Compressed Row (`BSR`) format is very similar to the Compressed Sparse Row (`CSR`) format. \n",
394 | "\n",
395 | "BSR is appropriate for sparse matrices with *dense sub matrices* like the example below. \n",
396 | "\n",
397 | "Block matrices often arise in *vector-valued* finite element discretizations. \n",
398 | "\n",
399 | "In such cases, BSR is **considerably more efficient** than CSR and CSC for many sparse arithmetic operations."
400 | ]
401 | },
402 | {
403 | "cell_type": "code",
404 | "execution_count": 10,
405 | "metadata": {
406 | "collapsed": false,
407 | "jupyter": {
408 | "outputs_hidden": false
409 | },
410 | "slideshow": {
411 | "slide_type": "subslide"
412 | }
413 | },
414 | "outputs": [
415 | {
416 | "data": {
417 | "text/plain": [
418 | "array([[1, 1, 0, 0, 2, 2],\n",
419 | " [1, 1, 0, 0, 2, 2],\n",
420 | " [0, 0, 0, 0, 3, 3],\n",
421 | " [0, 0, 0, 0, 3, 3],\n",
422 | " [4, 4, 5, 5, 6, 6],\n",
423 | " [4, 4, 5, 5, 6, 6]])"
424 | ]
425 | },
426 | "execution_count": 10,
427 | "metadata": {},
428 | "output_type": "execute_result"
429 | }
430 | ],
431 | "source": [
432 | "from scipy.sparse import bsr_matrix\n",
433 | "\n",
434 | "indptr = np.array([0, 2, 3, 6])\n",
435 | "indices = np.array([0, 2, 2, 0, 1, 2])\n",
436 | "data = np.array([1, 2, 3, 4, 5, 6]).repeat(4).reshape(6, 2, 2)\n",
437 | "bsr_matrix((data,indices,indptr), shape=(6, 6)).toarray()"
438 | ]
439 | },
440 | {
441 | "cell_type": "markdown",
442 | "metadata": {
443 | "slideshow": {
444 | "slide_type": "slide"
445 | }
446 | },
447 | "source": [
448 | "## COO - Coordinate Sparse Matrix\n",
449 | "\n",
450 | "**Advantages of the CSC format**\n",
451 | "\n",
452 | " * facilitates fast conversion among sparse formats\n",
453 | " * permits duplicate entries (see example)\n",
454 | " * very fast conversion to and from CSR/CSC formats\n",
455 | "\n",
456 | "**Disadvantages of the CSC format**\n",
457 | "\n",
458 | " * does not directly support arithmetic operations and slicing\n",
459 | " \n",
460 | "** Intended Usage**\n",
461 | "\n",
462 | " * COO is a fast format for constructing sparse matrices\n",
463 | " * Once a matrix has been constructed, convert to CSR or CSC format for fast arithmetic and matrix vector\n",
464 | " operations\n",
465 | " * By default when converting to CSR or CSC format, duplicate (i,j) entries will be summed together. \n",
466 | " This facilitates efficient construction of finite element matrices and the like.\n"
467 | ]
468 | },
469 | {
470 | "cell_type": "markdown",
471 | "metadata": {
472 | "slideshow": {
473 | "slide_type": "slide"
474 | }
475 | },
476 | "source": [
477 | "## DOK - Dictionary of Keys\n",
478 | "\n",
479 | "Sparse matrices can be used in arithmetic operations: they support addition, subtraction, multiplication, division, and matrix power.\n",
480 | "\n",
481 | "Allows for efficient O(1) access of individual elements. Duplicates are not allowed. Can be efficiently converted to a coo_matrix once constructed."
482 | ]
483 | },
484 | {
485 | "cell_type": "code",
486 | "execution_count": 11,
487 | "metadata": {
488 | "collapsed": false,
489 | "jupyter": {
490 | "outputs_hidden": false
491 | },
492 | "slideshow": {
493 | "slide_type": "subslide"
494 | }
495 | },
496 | "outputs": [
497 | {
498 | "data": {
499 | "text/plain": [
500 | "array([[0., 1., 2., 3., 4.],\n",
501 | " [0., 2., 3., 4., 5.],\n",
502 | " [0., 0., 4., 5., 6.],\n",
503 | " [0., 0., 0., 6., 7.],\n",
504 | " [0., 0., 0., 0., 8.]], dtype=float32)"
505 | ]
506 | },
507 | "execution_count": 11,
508 | "metadata": {},
509 | "output_type": "execute_result"
510 | }
511 | ],
512 | "source": [
513 | "from scipy.sparse import dok_matrix\n",
514 | "S = dok_matrix((5, 5), dtype=np.float32)\n",
515 | "for i in range(5):\n",
516 | " for j in range(i, 5):\n",
517 | " S[i,j] = i+j\n",
518 | " \n",
519 | "S.toarray()"
520 | ]
521 | },
522 | {
523 | "cell_type": "markdown",
524 | "metadata": {
525 | "slideshow": {
526 | "slide_type": "subslide"
527 | }
528 | },
529 | "source": [
530 | "The ``scipy.sparse`` submodule also has a lot of functions for sparse matrices\n",
531 | "including linear algebra, sparse solvers, graph algorithms, and much more."
532 | ]
533 | }
534 | ],
535 | "metadata": {
536 | "celltoolbar": "Slideshow",
537 | "kernelspec": {
538 | "display_name": "Python 3.7 (NumPy EuroSciPy)",
539 | "language": "python",
540 | "name": "numpy-euroscipy"
541 | },
542 | "language_info": {
543 | "codemirror_mode": {
544 | "name": "ipython",
545 | "version": 3
546 | },
547 | "file_extension": ".py",
548 | "mimetype": "text/x-python",
549 | "name": "python",
550 | "nbconvert_exporter": "python",
551 | "pygments_lexer": "ipython3",
552 | "version": "3.7.3"
553 | }
554 | },
555 | "nbformat": 4,
556 | "nbformat_minor": 4
557 | }
558 |
--------------------------------------------------------------------------------
/05_memmapping.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {
6 | "slideshow": {
7 | "slide_type": "slide"
8 | }
9 | },
10 | "source": [
11 | "# Processing large NumPy arrays with memory mapping\n",
12 | "\n",
13 | "\n",
14 | "**Reference**: _IPython Interactive Computing and Visualization Cookbook - Second Edition, by Cyrille Rossant_"
15 | ]
16 | },
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {},
20 | "source": [
21 | "---"
22 | ]
23 | },
24 | {
25 | "cell_type": "markdown",
26 | "metadata": {},
27 | "source": [
28 | "Sometimes, we need to deal with NumPy arrays that are too big to fit in the system memory. \n",
29 | "\n",
30 | "A common solution is to use memory mapping and implement **out-of-core** computations. \n",
31 | "\n",
32 | "The array is stored in a file on the hard drive, and we create a **memory-mapped** object to this file that can be used as a regular NumPy array. \n",
33 | "\n",
34 | "Accessing a portion of the array results in the corresponding data being automatically fetched from the hard drive. Therefore, we only consume what we use."
35 | ]
36 | },
37 | {
38 | "cell_type": "code",
39 | "execution_count": 1,
40 | "metadata": {
41 | "collapsed": false,
42 | "jupyter": {
43 | "outputs_hidden": false
44 | },
45 | "slideshow": {
46 | "slide_type": "subslide"
47 | }
48 | },
49 | "outputs": [],
50 | "source": [
51 | "import numpy as np"
52 | ]
53 | },
54 | {
55 | "cell_type": "code",
56 | "execution_count": 2,
57 | "metadata": {},
58 | "outputs": [],
59 | "source": [
60 | "# Let's create a Memory-Mapped Array in write mode\n",
61 | "\n",
62 | "nrows, ncols = 1000000, 100\n",
63 | "f = np.memmap('memmapped.dat', dtype=np.float32, mode='w+', shape=(nrows, ncols))"
64 | ]
65 | },
66 | {
67 | "cell_type": "markdown",
68 | "metadata": {},
69 | "source": [
70 | "Let's feed the array with random values, one column at a time because our system's memory is limited!"
71 | ]
72 | },
73 | {
74 | "cell_type": "code",
75 | "execution_count": 3,
76 | "metadata": {},
77 | "outputs": [],
78 | "source": [
79 | "for i in range(ncols):\n",
80 | " f[:, i] = np.random.rand(nrows)"
81 | ]
82 | },
83 | {
84 | "cell_type": "markdown",
85 | "metadata": {},
86 | "source": [
87 | "Save the last column of the Array"
88 | ]
89 | },
90 | {
91 | "cell_type": "code",
92 | "execution_count": 4,
93 | "metadata": {},
94 | "outputs": [],
95 | "source": [
96 | "x = f[:, -1]"
97 | ]
98 | },
99 | {
100 | "cell_type": "markdown",
101 | "metadata": {},
102 | "source": [
103 | "Now, we flush memory changes to disk by deleting the object:"
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": 5,
109 | "metadata": {},
110 | "outputs": [],
111 | "source": [
112 | "del f"
113 | ]
114 | },
115 | {
116 | "cell_type": "markdown",
117 | "metadata": {},
118 | "source": [
119 | "Reading a memory-mapped array from disk involves the same memmap() function. The data type and the shape need to be specified again, as this information is not stored in the file:"
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": 8,
125 | "metadata": {},
126 | "outputs": [],
127 | "source": [
128 | "f = np.memmap('memmapped.dat', dtype=np.float32,\n",
129 | " shape=(nrows, ncols))"
130 | ]
131 | },
132 | {
133 | "cell_type": "code",
134 | "execution_count": 9,
135 | "metadata": {},
136 | "outputs": [
137 | {
138 | "data": {
139 | "text/plain": [
140 | "True"
141 | ]
142 | },
143 | "execution_count": 9,
144 | "metadata": {},
145 | "output_type": "execute_result"
146 | }
147 | ],
148 | "source": [
149 | "np.array_equal(f[:, -1], x)"
150 | ]
151 | },
152 | {
153 | "cell_type": "code",
154 | "execution_count": 10,
155 | "metadata": {},
156 | "outputs": [],
157 | "source": [
158 | "del f"
159 | ]
160 | },
161 | {
162 | "cell_type": "markdown",
163 | "metadata": {},
164 | "source": [
165 | "**Note**:\n",
166 | "\n",
167 | ">This method is not adapted for long-term storage of data and data sharing. \n",
168 | ">A better file format for this specific case will be the **HDF5**."
169 | ]
170 | },
171 | {
172 | "cell_type": "markdown",
173 | "metadata": {},
174 | "source": [
175 | "## How `memmap` works"
176 | ]
177 | },
178 | {
179 | "cell_type": "markdown",
180 | "metadata": {},
181 | "source": [
182 | "Memory mapping lets you work with huge arrays almost as if they were regular arrays. Python code that accepts a NumPy array as input will also accept a `memmap` array. However, we need to ensure that the array is used efficiently. That is, the array is never loaded as a whole (otherwise, it would waste system memory and would obviate any advantage of the technique)."
183 | ]
184 | },
185 | {
186 | "cell_type": "markdown",
187 | "metadata": {},
188 | "source": [
189 | "Memory mapping is also useful when you have a huge file containing raw data in a homogeneous binary format with a known **data type and shape**. \n",
190 | "\n",
191 | "In this case, an alternative solution is to use NumPy's `fromfile()` function with a file handle created with Python's native `open()` function. \n",
192 | "\n",
193 | "Using `f.seek()` lets you position the cursor at any location and load a given number of bytes into a NumPy array."
194 | ]
195 | },
196 | {
197 | "cell_type": "markdown",
198 | "metadata": {
199 | "slideshow": {
200 | "slide_type": "subslide"
201 | }
202 | },
203 | "source": [
204 | "The numpy package makes it possible to memory map large contiguous chunks of binary files as shared memory for all the Python processes running on a given host:"
205 | ]
206 | },
207 | {
208 | "cell_type": "markdown",
209 | "metadata": {
210 | "slideshow": {
211 | "slide_type": "slide"
212 | }
213 | },
214 | "source": [
215 | "### Memmap Operations"
216 | ]
217 | },
218 | {
219 | "cell_type": "code",
220 | "execution_count": 11,
221 | "metadata": {
222 | "collapsed": false,
223 | "jupyter": {
224 | "outputs_hidden": false
225 | },
226 | "slideshow": {
227 | "slide_type": "subslide"
228 | }
229 | },
230 | "outputs": [
231 | {
232 | "name": "stdout",
233 | "output_type": "stream",
234 | "text": [
235 | "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
236 | ]
237 | }
238 | ],
239 | "source": [
240 | "mm_w = np.memmap('small_test.mmap', shape=10, dtype=np.float32, mode='w+')\n",
241 | "print(mm_w)"
242 | ]
243 | },
244 | {
245 | "cell_type": "markdown",
246 | "metadata": {
247 | "slideshow": {
248 | "slide_type": "subslide"
249 | }
250 | },
251 | "source": [
252 | "* This binary file can then be mapped as a new numpy array by all the engines having access to the same filesystem. \n",
253 | "* The `mode='r+'` opens this shared memory area in read write mode:"
254 | ]
255 | },
256 | {
257 | "cell_type": "code",
258 | "execution_count": 12,
259 | "metadata": {
260 | "collapsed": false,
261 | "jupyter": {
262 | "outputs_hidden": false
263 | },
264 | "slideshow": {
265 | "slide_type": "subslide"
266 | }
267 | },
268 | "outputs": [
269 | {
270 | "name": "stdout",
271 | "output_type": "stream",
272 | "text": [
273 | "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
274 | ]
275 | }
276 | ],
277 | "source": [
278 | "mm_r = np.memmap('small_test.mmap', dtype=np.float32, mode='r+')\n",
279 | "print(mm_r)"
280 | ]
281 | },
282 | {
283 | "cell_type": "code",
284 | "execution_count": 13,
285 | "metadata": {
286 | "collapsed": false,
287 | "jupyter": {
288 | "outputs_hidden": false
289 | },
290 | "slideshow": {
291 | "slide_type": "fragment"
292 | }
293 | },
294 | "outputs": [
295 | {
296 | "name": "stdout",
297 | "output_type": "stream",
298 | "text": [
299 | "[42. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
300 | ]
301 | }
302 | ],
303 | "source": [
304 | "mm_w[0] = 42\n",
305 | "print(mm_w)"
306 | ]
307 | },
308 | {
309 | "cell_type": "code",
310 | "execution_count": 14,
311 | "metadata": {
312 | "collapsed": false,
313 | "jupyter": {
314 | "outputs_hidden": false
315 | },
316 | "slideshow": {
317 | "slide_type": "fragment"
318 | }
319 | },
320 | "outputs": [
321 | {
322 | "name": "stdout",
323 | "output_type": "stream",
324 | "text": [
325 | "[42. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
326 | ]
327 | }
328 | ],
329 | "source": [
330 | "print(mm_r)"
331 | ]
332 | },
333 | {
334 | "cell_type": "markdown",
335 | "metadata": {
336 | "slideshow": {
337 | "slide_type": "subslide"
338 | }
339 | },
340 | "source": [
341 | "* Memory mapped arrays created with `mode='r+'` can be modified and the modifications are shared \n",
342 | " - in case of multiple process"
343 | ]
344 | },
345 | {
346 | "cell_type": "code",
347 | "execution_count": 15,
348 | "metadata": {
349 | "collapsed": false,
350 | "jupyter": {
351 | "outputs_hidden": false
352 | },
353 | "slideshow": {
354 | "slide_type": "fragment"
355 | }
356 | },
357 | "outputs": [],
358 | "source": [
359 | "mm_r[1] = 43"
360 | ]
361 | },
362 | {
363 | "cell_type": "code",
364 | "execution_count": 16,
365 | "metadata": {
366 | "collapsed": false,
367 | "jupyter": {
368 | "outputs_hidden": false
369 | },
370 | "slideshow": {
371 | "slide_type": "fragment"
372 | }
373 | },
374 | "outputs": [
375 | {
376 | "name": "stdout",
377 | "output_type": "stream",
378 | "text": [
379 | "[42. 43. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
380 | ]
381 | }
382 | ],
383 | "source": [
384 | "print(mm_r)"
385 | ]
386 | },
387 | {
388 | "cell_type": "markdown",
389 | "metadata": {
390 | "slideshow": {
391 | "slide_type": "subslide"
392 | }
393 | },
394 | "source": [
395 | "Memmap arrays generally behave very much like regular in-memory numpy arrays:"
396 | ]
397 | },
398 | {
399 | "cell_type": "code",
400 | "execution_count": 17,
401 | "metadata": {
402 | "collapsed": false,
403 | "jupyter": {
404 | "outputs_hidden": false
405 | },
406 | "slideshow": {
407 | "slide_type": "subslide"
408 | }
409 | },
410 | "outputs": [
411 | {
412 | "name": "stdout",
413 | "output_type": "stream",
414 | "text": [
415 | "85.0\n",
416 | "sum=85.0, mean=8.5, std=17.0014705657959\n"
417 | ]
418 | }
419 | ],
420 | "source": [
421 | "print(mm_r.sum())\n",
422 | "print(\"sum={0}, mean={1}, std={2}\".format(mm_r.sum(), \n",
423 | " np.mean(mm_r), np.std(mm_r)))"
424 | ]
425 | },
426 | {
427 | "cell_type": "markdown",
428 | "metadata": {
429 | "slideshow": {
430 | "slide_type": "subslide"
431 | }
432 | },
433 | "source": [
434 | "Before allocating more data let us define a couple of utility functions from the previous exercise (and more) to monitor what is used by which engine and what is still free on the cluster as a whole:"
435 | ]
436 | },
437 | {
438 | "cell_type": "markdown",
439 | "metadata": {
440 | "slideshow": {
441 | "slide_type": "subslide"
442 | }
443 | },
444 | "source": [
445 | "* Let's allocate a 80MB memmap array:"
446 | ]
447 | },
448 | {
449 | "cell_type": "code",
450 | "execution_count": 18,
451 | "metadata": {
452 | "collapsed": false,
453 | "jupyter": {
454 | "outputs_hidden": false
455 | },
456 | "slideshow": {
457 | "slide_type": "fragment"
458 | }
459 | },
460 | "outputs": [
461 | {
462 | "data": {
463 | "text/plain": [
464 | "memmap([0., 0., 0., ..., 0., 0., 0.])"
465 | ]
466 | },
467 | "execution_count": 18,
468 | "metadata": {},
469 | "output_type": "execute_result"
470 | }
471 | ],
472 | "source": [
473 | "np.memmap('bigger_test.mmap', shape=10 * int(1e6), dtype=np.float64, mode='w+')"
474 | ]
475 | },
476 | {
477 | "cell_type": "markdown",
478 | "metadata": {
479 | "slideshow": {
480 | "slide_type": "subslide"
481 | }
482 | },
483 | "source": [
484 | "No significant memory was used in this operation as we just asked the OS to allocate the buffer on the hard drive and just maitain a virtual memory area as a cheap reference to this buffer.\n",
485 | "\n",
486 | "Let's open new references to the same buffer from all the engines at once:"
487 | ]
488 | },
489 | {
490 | "cell_type": "code",
491 | "execution_count": 19,
492 | "metadata": {
493 | "collapsed": false,
494 | "jupyter": {
495 | "outputs_hidden": false
496 | },
497 | "slideshow": {
498 | "slide_type": "subslide"
499 | }
500 | },
501 | "outputs": [
502 | {
503 | "name": "stdout",
504 | "output_type": "stream",
505 | "text": [
506 | "CPU times: user 616 µs, sys: 778 µs, total: 1.39 ms\n",
507 | "Wall time: 17.3 ms\n"
508 | ]
509 | }
510 | ],
511 | "source": [
512 | "%time big_mmap = np.memmap('bigger_test.mmap', dtype=np.float64, mode='r+')"
513 | ]
514 | },
515 | {
516 | "cell_type": "code",
517 | "execution_count": 20,
518 | "metadata": {
519 | "collapsed": false,
520 | "jupyter": {
521 | "outputs_hidden": false
522 | },
523 | "slideshow": {
524 | "slide_type": "subslide"
525 | }
526 | },
527 | "outputs": [
528 | {
529 | "data": {
530 | "text/plain": [
531 | "memmap([0., 0., 0., ..., 0., 0., 0.])"
532 | ]
533 | },
534 | "execution_count": 20,
535 | "metadata": {},
536 | "output_type": "execute_result"
537 | }
538 | ],
539 | "source": [
540 | "big_mmap"
541 | ]
542 | },
543 | {
544 | "cell_type": "markdown",
545 | "metadata": {
546 | "slideshow": {
547 | "slide_type": "subslide"
548 | }
549 | },
550 | "source": [
551 | "* Let's trigger an actual load of the data from the drive into the in-memory disk cache of the OS, this can take some time depending on the speed of the hard drive (on the order of 100MB/s to 300MB/s hence 3s to 8s for this dataset):"
552 | ]
553 | },
554 | {
555 | "cell_type": "code",
556 | "execution_count": 21,
557 | "metadata": {
558 | "collapsed": false,
559 | "jupyter": {
560 | "outputs_hidden": false
561 | },
562 | "slideshow": {
563 | "slide_type": "subslide"
564 | }
565 | },
566 | "outputs": [
567 | {
568 | "name": "stdout",
569 | "output_type": "stream",
570 | "text": [
571 | "CPU times: user 20.5 ms, sys: 32.9 ms, total: 53.5 ms\n",
572 | "Wall time: 54.3 ms\n"
573 | ]
574 | },
575 | {
576 | "data": {
577 | "text/plain": [
578 | "0.0"
579 | ]
580 | },
581 | "execution_count": 21,
582 | "metadata": {},
583 | "output_type": "execute_result"
584 | }
585 | ],
586 | "source": [
587 | "%time np.sum(big_mmap)"
588 | ]
589 | },
590 | {
591 | "cell_type": "markdown",
592 | "metadata": {
593 | "slideshow": {
594 | "slide_type": "subslide"
595 | }
596 | },
597 | "source": [
598 | "* Now back into memory"
599 | ]
600 | },
601 | {
602 | "cell_type": "code",
603 | "execution_count": 22,
604 | "metadata": {
605 | "collapsed": false,
606 | "jupyter": {
607 | "outputs_hidden": false
608 | },
609 | "slideshow": {
610 | "slide_type": "fragment"
611 | }
612 | },
613 | "outputs": [
614 | {
615 | "name": "stdout",
616 | "output_type": "stream",
617 | "text": [
618 | "CPU times: user 15 ms, sys: 1.36 ms, total: 16.4 ms\n",
619 | "Wall time: 14.7 ms\n"
620 | ]
621 | },
622 | {
623 | "data": {
624 | "text/plain": [
625 | "0.0"
626 | ]
627 | },
628 | "execution_count": 22,
629 | "metadata": {},
630 | "output_type": "execute_result"
631 | }
632 | ],
633 | "source": [
634 | "%time np.sum(big_mmap)"
635 | ]
636 | }
637 | ],
638 | "metadata": {
639 | "celltoolbar": "Slideshow",
640 | "kernelspec": {
641 | "display_name": "Python 3.7 (NumPy EuroSciPy)",
642 | "language": "python",
643 | "name": "numpy-euroscipy"
644 | },
645 | "language_info": {
646 | "codemirror_mode": {
647 | "name": "ipython",
648 | "version": 3
649 | },
650 | "file_extension": ".py",
651 | "mimetype": "text/x-python",
652 | "name": "python",
653 | "nbconvert_exporter": "python",
654 | "pygments_lexer": "ipython3",
655 | "version": "3.7.3"
656 | }
657 | },
658 | "nbformat": 4,
659 | "nbformat_minor": 4
660 | }
661 |
--------------------------------------------------------------------------------
/extra_torch_tensor.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "## Original Notebook\n",
8 | "\n",
9 | "### Introduction to PyTorch Tensor\n",
10 | "\n",
11 | "**Reference**: [\"What is PyTorch?\"](https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py) by [Soumith Chintala](http://soumith.ch)"
12 | ]
13 | },
14 | {
15 | "cell_type": "markdown",
16 | "metadata": {},
17 | "source": [
18 | "\n",
19 | "What is PyTorch?\n",
20 | "================\n",
21 | "\n",
22 | "It’s a Python-based scientific computing package targeted at two sets of\n",
23 | "audiences:\n",
24 | "\n",
25 | "- A replacement for NumPy to use the power of GPUs\n",
26 | "- a deep learning research platform that provides maximum flexibility\n",
27 | " and speed\n",
28 | "\n",
29 | "Getting Started\n",
30 | "---------------\n",
31 | "\n",
32 | "Tensors\n",
33 | "^^^^^^^\n",
34 | "\n",
35 | "Tensors are similar to NumPy’s ndarrays, with the addition being that\n",
36 | "Tensors can also be used on a GPU to accelerate computing.\n",
37 | "\n"
38 | ]
39 | },
40 | {
41 | "cell_type": "code",
42 | "execution_count": 1,
43 | "metadata": {},
44 | "outputs": [],
45 | "source": [
46 | "import torch"
47 | ]
48 | },
49 | {
50 | "cell_type": "markdown",
51 | "metadata": {},
52 | "source": [
53 | "An uninitialized matrix is declared,\n", 54 | " but does not contain definite known\n", 55 | " values before it is used. When an\n", 56 | " uninitialized matrix is created,\n", 57 | " whatever values were in the allocated\n", 58 | " memory at the time will appear as the initial values.
``torch.Size`` is in fact a tuple, so it supports all tuple operations.
Any operation that mutates a tensor in-place is post-fixed with an ``_``.\n", 396 | " For example: ``x.copy_(y)``, ``x.t_()``, will change ``x``.
![](\"images/ndarray_with_details.png\")
"
1326 | ]
1327 | },
1328 | {
1329 | "cell_type": "markdown",
1330 | "metadata": {},
1331 | "source": [
1332 | "## Data vs Metadata (Attributes)"
1333 | ]
1334 | },
1335 | {
1336 | "cell_type": "markdown",
1337 | "metadata": {
1338 | "slideshow": {
1339 | "slide_type": "subslide"
1340 | }
1341 | },
1342 | "source": [
1343 | "This internal separation between actual data (i.e. the content of the array --> the `memory`) and metadata (i.e. properties and attributes of the data), allows for example for an efficient memory management.\n",
1344 | "\n",
1345 | "For example, the shape of an Numpy array **can be modified without copying and/or affecting** the actual data, which makes it a fast operation even for large arrays."
1346 | ]
1347 | },
1348 | {
1349 | "cell_type": "code",
1350 | "execution_count": 34,
1351 | "metadata": {
1352 | "collapsed": false,
1353 | "jupyter": {
1354 | "outputs_hidden": false
1355 | },
1356 | "slideshow": {
1357 | "slide_type": "fragment"
1358 | }
1359 | },
1360 | "outputs": [
1361 | {
1362 | "data": {
1363 | "text/plain": [
1364 | "array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,\n",
1365 | " 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
1366 | " 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44])"
1367 | ]
1368 | },
1369 | "execution_count": 34,
1370 | "metadata": {},
1371 | "output_type": "execute_result"
1372 | }
1373 | ],
1374 | "source": [
1375 | "a = np.arange(45)\n",
1376 | "\n",
1377 | "a"
1378 | ]
1379 | },
1380 | {
1381 | "cell_type": "code",
1382 | "execution_count": 35,
1383 | "metadata": {},
1384 | "outputs": [
1385 | {
1386 | "data": {
1387 | "text/plain": [
1388 | "(45,)"
1389 | ]
1390 | },
1391 | "execution_count": 35,
1392 | "metadata": {},
1393 | "output_type": "execute_result"
1394 | }
1395 | ],
1396 | "source": [
1397 | "a.shape"
1398 | ]
1399 | },
1400 | {
1401 | "cell_type": "code",
1402 | "execution_count": 36,
1403 | "metadata": {},
1404 | "outputs": [
1405 | {
1406 | "data": {
1407 | "text/plain": [
1408 | "array([[ 0, 1, 2, 3, 4],\n",
1409 | " [ 5, 6, 7, 8, 9],\n",
1410 | " [10, 11, 12, 13, 14],\n",
1411 | " [15, 16, 17, 18, 19],\n",
1412 | " [20, 21, 22, 23, 24],\n",
1413 | " [25, 26, 27, 28, 29],\n",
1414 | " [30, 31, 32, 33, 34],\n",
1415 | " [35, 36, 37, 38, 39],\n",
1416 | " [40, 41, 42, 43, 44]])"
1417 | ]
1418 | },
1419 | "execution_count": 36,
1420 | "metadata": {},
1421 | "output_type": "execute_result"
1422 | }
1423 | ],
1424 | "source": [
1425 | "A = a.reshape(9, 5)\n",
1426 | "\n",
1427 | "A"
1428 | ]
1429 | },
1430 | {
1431 | "cell_type": "code",
1432 | "execution_count": 37,
1433 | "metadata": {
1434 | "collapsed": false,
1435 | "jupyter": {
1436 | "outputs_hidden": false
1437 | },
1438 | "slideshow": {
1439 | "slide_type": "subslide"
1440 | }
1441 | },
1442 | "outputs": [],
1443 | "source": [
1444 | "n, m = A.shape"
1445 | ]
1446 | },
1447 | {
1448 | "cell_type": "code",
1449 | "execution_count": 38,
1450 | "metadata": {
1451 | "collapsed": false,
1452 | "jupyter": {
1453 | "outputs_hidden": false
1454 | },
1455 | "slideshow": {
1456 | "slide_type": "fragment"
1457 | }
1458 | },
1459 | "outputs": [
1460 | {
1461 | "data": {
1462 | "text/plain": [
1463 | "array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,\n",
1464 | " 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,\n",
1465 | " 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]])"
1466 | ]
1467 | },
1468 | "execution_count": 38,
1469 | "metadata": {},
1470 | "output_type": "execute_result"
1471 | }
1472 | ],
1473 | "source": [
1474 | "B = A.reshape((1,n*m))\n",
1475 | "B"
1476 | ]
1477 | },
1478 | {
1479 | "cell_type": "markdown",
1480 | "metadata": {},
1481 | "source": [
1482 | "**Q**: What is the difference (in terms of shape) between `B` and the original `a`?"
1483 | ]
1484 | },
1485 | {
1486 | "cell_type": "markdown",
1487 | "metadata": {
1488 | "slideshow": {
1489 | "slide_type": "slide"
1490 | }
1491 | },
1492 | "source": [
1493 | "### Flattening\n",
1494 | "\n",
1495 | "Another (quite common) reshaping operation you will end up performing on n-dimensional arrays is **flattening**.\n",
1496 | "\n",
1497 | "Flattening means _collapsing all the axis into a unique one_"
1498 | ]
1499 | },
1500 | {
1501 | "cell_type": "markdown",
1502 | "metadata": {
1503 | "slideshow": {
1504 | "slide_type": "subslide"
1505 | }
1506 | },
1507 | "source": [
1508 | "### `np.ravel`\n",
1509 | "\n",
1510 | "`numpy.ndarray` objects have a `ravel` method that generates a new version of the array as a `1D` vector. \n",
1511 | "\n",
1512 | "Also this time, the original memory is unaffected, and a pointer with different metadata is returned."
1513 | ]
1514 | },
1515 | {
1516 | "cell_type": "code",
1517 | "execution_count": 39,
1518 | "metadata": {
1519 | "collapsed": false,
1520 | "jupyter": {
1521 | "outputs_hidden": false
1522 | },
1523 | "slideshow": {
1524 | "slide_type": "fragment"
1525 | }
1526 | },
1527 | "outputs": [
1528 | {
1529 | "data": {
1530 | "text/plain": [
1531 | "array([1, 2, 3, 4, 5, 6])"
1532 | ]
1533 | },
1534 | "execution_count": 39,
1535 | "metadata": {},
1536 | "output_type": "execute_result"
1537 | }
1538 | ],
1539 | "source": [
1540 | "A = np.array([[1, 2, 3], [4, 5, 6]])\n",
1541 | "A.ravel()"
1542 | ]
1543 | },
1544 | {
1545 | "cell_type": "markdown",
1546 | "metadata": {},
1547 | "source": [
1548 | "By default, the `np.ravel` performs the operation _row-wise_ á-la-C. Numpy also support a Fortran-style order of indices (i.e. _column-major_ indexing)"
1549 | ]
1550 | },
1551 | {
1552 | "cell_type": "code",
1553 | "execution_count": 40,
1554 | "metadata": {},
1555 | "outputs": [
1556 | {
1557 | "data": {
1558 | "text/plain": [
1559 | "array([1, 4, 2, 5, 3, 6])"
1560 | ]
1561 | },
1562 | "execution_count": 40,
1563 | "metadata": {},
1564 | "output_type": "execute_result"
1565 | }
1566 | ],
1567 | "source": [
1568 | "A.ravel('F') # order F (Fortran) is column-major, C (default) row-major"
1569 | ]
1570 | },
1571 | {
1572 | "cell_type": "markdown",
1573 | "metadata": {
1574 | "slideshow": {
1575 | "slide_type": "subslide"
1576 | }
1577 | },
1578 | "source": [
1579 | "**Alternatively** We can also use the function `np.flatten` to make a higher-dimensional array into a vector. But this function create a copy of the data."
1580 | ]
1581 | },
1582 | {
1583 | "cell_type": "markdown",
1584 | "metadata": {},
1585 | "source": [
1586 | "### Transpose\n",
1587 | "\n",
1588 | "Similarly, we can transpose a matrix"
1589 | ]
1590 | },
1591 | {
1592 | "cell_type": "code",
1593 | "execution_count": 41,
1594 | "metadata": {
1595 | "collapsed": false,
1596 | "jupyter": {
1597 | "outputs_hidden": false
1598 | },
1599 | "slideshow": {
1600 | "slide_type": "subslide"
1601 | }
1602 | },
1603 | "outputs": [
1604 | {
1605 | "data": {
1606 | "text/plain": [
1607 | "array([[1, 4],\n",
1608 | " [2, 5],\n",
1609 | " [3, 6]])"
1610 | ]
1611 | },
1612 | "execution_count": 41,
1613 | "metadata": {},
1614 | "output_type": "execute_result"
1615 | }
1616 | ],
1617 | "source": [
1618 | "A.T"
1619 | ]
1620 | },
1621 | {
1622 | "cell_type": "code",
1623 | "execution_count": 42,
1624 | "metadata": {
1625 | "collapsed": false,
1626 | "jupyter": {
1627 | "outputs_hidden": false
1628 | },
1629 | "slideshow": {
1630 | "slide_type": "subslide"
1631 | }
1632 | },
1633 | "outputs": [
1634 | {
1635 | "data": {
1636 | "text/plain": [
1637 | "array([1, 4, 2, 5, 3, 6])"
1638 | ]
1639 | },
1640 | "execution_count": 42,
1641 | "metadata": {},
1642 | "output_type": "execute_result"
1643 | }
1644 | ],
1645 | "source": [
1646 | "A.T.ravel()"
1647 | ]
1648 | },
1649 | {
1650 | "cell_type": "markdown",
1651 | "metadata": {},
1652 | "source": [
1653 | "## Introducing `np.newaxis`\n",
1654 | "\n",
1655 | "In addition to shape, we can also manipulate the axis of an array."
1656 | ]
1657 | },
1658 | {
1659 | "cell_type": "markdown",
1660 | "metadata": {},
1661 | "source": [
1662 | "**(1)** We can always add as many axis as we want:"
1663 | ]
1664 | },
1665 | {
1666 | "cell_type": "code",
1667 | "execution_count": 43,
1668 | "metadata": {},
1669 | "outputs": [
1670 | {
1671 | "name": "stdout",
1672 | "output_type": "stream",
1673 | "text": [
1674 | "(1, 10, 2)\n"
1675 | ]
1676 | }
1677 | ],
1678 | "source": [
1679 | "A = np.arange(20).reshape(10, 2)\n",
1680 | "A = A[np.newaxis, ...] # this is called ellipsis\n",
1681 | "\n",
1682 | "print(A.shape)"
1683 | ]
1684 | },
1685 | {
1686 | "cell_type": "markdown",
1687 | "metadata": {},
1688 | "source": [
1689 | "**(2)** We can also _permute_ axis:"
1690 | ]
1691 | },
1692 | {
1693 | "cell_type": "code",
1694 | "execution_count": 44,
1695 | "metadata": {},
1696 | "outputs": [
1697 | {
1698 | "name": "stdout",
1699 | "output_type": "stream",
1700 | "text": [
1701 | "(2, 10, 1)\n"
1702 | ]
1703 | }
1704 | ],
1705 | "source": [
1706 | "A = A.swapaxes(0, 2) # swap axis 0 with axis 2 --> new shape: (2, 10, 1)\n",
1707 | "\n",
1708 | "print(A.shape)"
1709 | ]
1710 | },
1711 | {
1712 | "cell_type": "markdown",
1713 | "metadata": {},
1714 | "source": [
1715 | "Again, changin and manipulating the `axis` will not touch the memory, it will just change parameters (i.e. `strides` and `offset`) to navigate data."
1716 | ]
1717 | },
1718 | {
1719 | "cell_type": "markdown",
1720 | "metadata": {},
1721 | "source": [
1722 | "---"
1723 | ]
1724 | },
1725 | {
1726 | "cell_type": "markdown",
1727 | "metadata": {
1728 | "slideshow": {
1729 | "slide_type": "slide"
1730 | }
1731 | },
1732 | "source": [
1733 | "## Numerical Types and Precision\n",
1734 | "\n",
1735 | "In NumPy, talking about `int` or `float` does not make \"real sense\". This is mainly for two reasons:\n",
1736 | "\n",
1737 | "(a) `int` or `float` are assumed at the maximum precision available on your machine (presumably `int64` and \n",
1738 | "`float64`, respectively.\n",
1739 | "\n",
1740 | "(b) Different precision imply different numerical ranges, and so different memory size (i.e. _number of bytes_ required to represent all the numbers in the corresponding numerical range).\n",
1741 | "\n",
1742 | "Numpy support the following numerical types:"
1743 | ]
1744 | },
1745 | {
1746 | "cell_type": "markdown",
1747 | "metadata": {
1748 | "slideshow": {
1749 | "slide_type": "subslide"
1750 | }
1751 | },
1752 | "source": [
1753 | " bool | This stores boolean (True or False) as a bit\n",
1754 | "\n",
1755 | " int0 | This is a platform integer (normally either int32 or int64)\n",
1756 | " int8 | This is an integer ranging from -128 to 127\n",
1757 | " int16 | This is an integer ranging from -32768 to 32767\n",
1758 | " int32 | This is an integer ranging from -2 ** 31 to 2 ** 31 -1\n",
1759 | " int64 | This is an integer ranging from -2 ** 63 to 2 ** 63 -1\n",
1760 | " \n",
1761 | " uint8 | This is an unsigned integer ranging from 0 to 255\n",
1762 | " uint16 | This is an unsigned integer ranging from 0 to 65535\n",
1763 | " uint32 | This is an unsigned integer ranging from 0 to 2 ** 32 - 1\n",
1764 | " uint64 | This is an unsigned integer ranging from 0 to 2 ** 64 - 1\n",
1765 | "\n",
1766 | " float16 | This is a half precision float with sign bit, 5 bits exponent, and 10 bits mantissa\n",
1767 | " float32 | This is a single precision float with sign bit, 8 bits exponent, and 23 bits mantissa\n",
1768 | " float64 or float | This is a double precision float with sign bit, 11 bits exponent, and 52 bits mantissa\n",
1769 | " complex64 | This is a complex number represented by two 32-bit floats (real and imaginary components)\n",
1770 | " complex128 | This is a complex number represented by two 64-bit floats (real and imaginary components)\n",
1771 | " (or complex)\n"
1772 | ]
1773 | },
1774 | {
1775 | "cell_type": "markdown",
1776 | "metadata": {
1777 | "slideshow": {
1778 | "slide_type": "slide"
1779 | }
1780 | },
1781 | "source": [
1782 | "### Numerical Types and Representation"
1783 | ]
1784 | },
1785 | {
1786 | "cell_type": "markdown",
1787 | "metadata": {
1788 | "slideshow": {
1789 | "slide_type": "subslide"
1790 | }
1791 | },
1792 | "source": [
1793 | "The **numerical dtype** of an array should be selected very carefully, as it directly affects the numerical representation of elements, that is: \n",
1794 | "\n",
1795 | " * the number of **bytes** used; \n",
1796 | " * the *numerical range*"
1797 | ]
1798 | },
1799 | {
1800 | "cell_type": "markdown",
1801 | "metadata": {},
1802 | "source": [
1803 | "We can **always specify** the `dtype` of an array when we create one. If we do not, the `dtype` of the array will be inferred, namely `np.int_` or `np.float_` depending on the case."
1804 | ]
1805 | },
1806 | {
1807 | "cell_type": "code",
1808 | "execution_count": 45,
1809 | "metadata": {},
1810 | "outputs": [
1811 | {
1812 | "name": "stdout",
1813 | "output_type": "stream",
1814 | "text": [
1815 | "[0 1 2 3 4 5 6 7 8 9]\n",
1816 | "int64\n"
1817 | ]
1818 | }
1819 | ],
1820 | "source": [
1821 | "a = np.arange(10)\n",
1822 | "print(a)\n",
1823 | "\n",
1824 | "print(a.dtype)"
1825 | ]
1826 | },
1827 | {
1828 | "cell_type": "code",
1829 | "execution_count": 46,
1830 | "metadata": {},
1831 | "outputs": [
1832 | {
1833 | "name": "stdout",
1834 | "output_type": "stream",
1835 | "text": [
1836 | "[0 1 2 3 4 5 6 7 8 9]\n",
1837 | "uint8\n"
1838 | ]
1839 | }
1840 | ],
1841 | "source": [
1842 | "au = np.arange(10, dtype=np.uint8)\n",
1843 | "print(au)\n",
1844 | "\n",
1845 | "print(au.dtype)"
1846 | ]
1847 | },
1848 | {
1849 | "cell_type": "markdown",
1850 | "metadata": {
1851 | "slideshow": {
1852 | "slide_type": "subslide"
1853 | }
1854 | },
1855 | "source": [
1856 | "So, then: **What happens if I try to represent a number that is Out of range?**\n",
1857 | "\n",
1858 | "Let's have a go with **integers**, i.e., `int8` and `uint8`"
1859 | ]
1860 | },
1861 | {
1862 | "cell_type": "code",
1863 | "execution_count": 47,
1864 | "metadata": {
1865 | "slideshow": {
1866 | "slide_type": "fragment"
1867 | }
1868 | },
1869 | "outputs": [
1870 | {
1871 | "data": {
1872 | "text/plain": [
1873 | "array([0, 0, 0, 0], dtype=int8)"
1874 | ]
1875 | },
1876 | "execution_count": 47,
1877 | "metadata": {},
1878 | "output_type": "execute_result"
1879 | }
1880 | ],
1881 | "source": [
1882 | "x = np.zeros(4, 'int8') # Integer ranging from -128 to 127\n",
1883 | "x"
1884 | ]
1885 | },
1886 | {
1887 | "cell_type": "markdown",
1888 | "metadata": {},
1889 | "source": [
1890 | ">__Spoiler Alert__: _very simple example of indexing in NumPy_\n",
1891 | ">\n",
1892 | "> _Well...it works as expected, doesn't it?_"
1893 | ]
1894 | },
1895 | {
1896 | "cell_type": "code",
1897 | "execution_count": 48,
1898 | "metadata": {
1899 | "slideshow": {
1900 | "slide_type": "subslide"
1901 | }
1902 | },
1903 | "outputs": [
1904 | {
1905 | "data": {
1906 | "text/plain": [
1907 | "array([127, 0, 0, 0], dtype=int8)"
1908 | ]
1909 | },
1910 | "execution_count": 48,
1911 | "metadata": {},
1912 | "output_type": "execute_result"
1913 | }
1914 | ],
1915 | "source": [
1916 | "x[0] = 127\n",
1917 | "x"
1918 | ]
1919 | },
1920 | {
1921 | "cell_type": "code",
1922 | "execution_count": 49,
1923 | "metadata": {
1924 | "slideshow": {
1925 | "slide_type": "fragment"
1926 | }
1927 | },
1928 | "outputs": [
1929 | {
1930 | "data": {
1931 | "text/plain": [
1932 | "array([-128, 0, 0, 0], dtype=int8)"
1933 | ]
1934 | },
1935 | "execution_count": 49,
1936 | "metadata": {},
1937 | "output_type": "execute_result"
1938 | }
1939 | ],
1940 | "source": [
1941 | "x[0] = 128\n",
1942 | "x"
1943 | ]
1944 | },
1945 | {
1946 | "cell_type": "code",
1947 | "execution_count": 50,
1948 | "metadata": {
1949 | "slideshow": {
1950 | "slide_type": "fragment"
1951 | }
1952 | },
1953 | "outputs": [
1954 | {
1955 | "data": {
1956 | "text/plain": [
1957 | "array([-128, -127, 0, 0], dtype=int8)"
1958 | ]
1959 | },
1960 | "execution_count": 50,
1961 | "metadata": {},
1962 | "output_type": "execute_result"
1963 | }
1964 | ],
1965 | "source": [
1966 | "x[1] = 129\n",
1967 | "x"
1968 | ]
1969 | },
1970 | {
1971 | "cell_type": "code",
1972 | "execution_count": 51,
1973 | "metadata": {
1974 | "slideshow": {
1975 | "slide_type": "fragment"
1976 | }
1977 | },
1978 | "outputs": [
1979 | {
1980 | "data": {
1981 | "text/plain": [
1982 | "array([-128, -127, 1, 0], dtype=int8)"
1983 | ]
1984 | },
1985 | "execution_count": 51,
1986 | "metadata": {},
1987 | "output_type": "execute_result"
1988 | }
1989 | ],
1990 | "source": [
1991 | "x[2] = 257 # i.e. (128 x 2) + 1\n",
1992 | "x"
1993 | ]
1994 | },
1995 | {
1996 | "cell_type": "code",
1997 | "execution_count": 52,
1998 | "metadata": {
1999 | "slideshow": {
2000 | "slide_type": "subslide"
2001 | }
2002 | },
2003 | "outputs": [
2004 | {
2005 | "data": {
2006 | "text/plain": [
2007 | "array([0, 0, 0, 0], dtype=uint8)"
2008 | ]
2009 | },
2010 | "execution_count": 52,
2011 | "metadata": {},
2012 | "output_type": "execute_result"
2013 | }
2014 | ],
2015 | "source": [
2016 | "ux = np.zeros(4, 'uint8') # Integer ranging from 0 to 255, dtype also as string!\n",
2017 | "ux"
2018 | ]
2019 | },
2020 | {
2021 | "cell_type": "code",
2022 | "execution_count": 53,
2023 | "metadata": {
2024 | "slideshow": {
2025 | "slide_type": "subslide"
2026 | }
2027 | },
2028 | "outputs": [
2029 | {
2030 | "data": {
2031 | "text/plain": [
2032 | "array([255, 0, 1, 1], dtype=uint8)"
2033 | ]
2034 | },
2035 | "execution_count": 53,
2036 | "metadata": {},
2037 | "output_type": "execute_result"
2038 | }
2039 | ],
2040 | "source": [
2041 | "ux[0] = 255\n",
2042 | "ux[1] = 256\n",
2043 | "ux[2] = 257\n",
2044 | "ux[3] = 513 # (256 x 2) + 1\n",
2045 | "ux"
2046 | ]
2047 | },
2048 | {
2049 | "cell_type": "markdown",
2050 | "metadata": {},
2051 | "source": [
2052 | "### Machine Info and Supported Numerical Representation"
2053 | ]
2054 | },
2055 | {
2056 | "cell_type": "markdown",
2057 | "metadata": {},
2058 | "source": [
2059 | "Numpy provides two functions to inspect the information of supported integer and floating-point types, namely `np.iinfo` and `np.finfo`:"
2060 | ]
2061 | },
2062 | {
2063 | "cell_type": "code",
2064 | "execution_count": 54,
2065 | "metadata": {},
2066 | "outputs": [
2067 | {
2068 | "data": {
2069 | "text/plain": [
2070 | "iinfo(min=-2147483648, max=2147483647, dtype=int32)"
2071 | ]
2072 | },
2073 | "execution_count": 54,
2074 | "metadata": {},
2075 | "output_type": "execute_result"
2076 | }
2077 | ],
2078 | "source": [
2079 | "np.iinfo(np.int32)"
2080 | ]
2081 | },
2082 | {
2083 | "cell_type": "code",
2084 | "execution_count": 55,
2085 | "metadata": {},
2086 | "outputs": [
2087 | {
2088 | "data": {
2089 | "text/plain": [
2090 | "finfo(resolution=0.001, min=-6.55040e+04, max=6.55040e+04, dtype=float16)"
2091 | ]
2092 | },
2093 | "execution_count": 55,
2094 | "metadata": {},
2095 | "output_type": "execute_result"
2096 | }
2097 | ],
2098 | "source": [
2099 | "np.finfo(np.float16)"
2100 | ]
2101 | },
2102 | {
2103 | "cell_type": "markdown",
2104 | "metadata": {},
2105 | "source": [
2106 | "In addition, the `MachAr` class will provide information on the current machine : "
2107 | ]
2108 | },
2109 | {
2110 | "cell_type": "code",
2111 | "execution_count": 56,
2112 | "metadata": {},
2113 | "outputs": [],
2114 | "source": [
2115 | "machine_info = np.MachAr()"
2116 | ]
2117 | },
2118 | {
2119 | "cell_type": "code",
2120 | "execution_count": 57,
2121 | "metadata": {},
2122 | "outputs": [
2123 | {
2124 | "data": {
2125 | "text/plain": [
2126 | "2.220446049250313e-16"
2127 | ]
2128 | },
2129 | "execution_count": 57,
2130 | "metadata": {},
2131 | "output_type": "execute_result"
2132 | }
2133 | ],
2134 | "source": [
2135 | "machine_info.epsilon"
2136 | ]
2137 | },
2138 | {
2139 | "cell_type": "code",
2140 | "execution_count": 58,
2141 | "metadata": {},
2142 | "outputs": [
2143 | {
2144 | "data": {
2145 | "text/plain": [
2146 | "1.7976931348623157e+308"
2147 | ]
2148 | },
2149 | "execution_count": 58,
2150 | "metadata": {},
2151 | "output_type": "execute_result"
2152 | }
2153 | ],
2154 | "source": [
2155 | "machine_info.huge"
2156 | ]
2157 | },
2158 | {
2159 | "cell_type": "code",
2160 | "execution_count": 59,
2161 | "metadata": {},
2162 | "outputs": [
2163 | {
2164 | "data": {
2165 | "text/plain": [
2166 | "True"
2167 | ]
2168 | },
2169 | "execution_count": 59,
2170 | "metadata": {},
2171 | "output_type": "execute_result"
2172 | }
2173 | ],
2174 | "source": [
2175 | "np.finfo(np.float64).max == machine_info.huge"
2176 | ]
2177 | },
2178 | {
2179 | "cell_type": "code",
2180 | "execution_count": null,
2181 | "metadata": {},
2182 | "outputs": [],
2183 | "source": [
2184 | "# TRY THIS!\n",
2185 | "\n",
2186 | "help(machine_info)"
2187 | ]
2188 | },
2189 | {
2190 | "cell_type": "markdown",
2191 | "metadata": {
2192 | "slideshow": {
2193 | "slide_type": "slide"
2194 | }
2195 | },
2196 | "source": [
2197 | "# Data Type Object"
2198 | ]
2199 | },
2200 | {
2201 | "cell_type": "markdown",
2202 | "metadata": {
2203 | "slideshow": {
2204 | "slide_type": "subslide"
2205 | }
2206 | },
2207 | "source": [
2208 | "**Data type objects** are instances of the `numpy.dtype` class. \n",
2209 | "\n",
2210 | "Once again, arrays have a data type. \n",
2211 | "