├── assets └── images │ ├── pp.png │ ├── xb.png │ └── ntcd.png ├── week1 ├── NumPy-intro.ipynb ├── PyTorch-intro.ipynb ├── PytorchTut.ipynb ├── Google_Colab_Tutorial.ipynb └── Python_Tutorial_The_byte.ipynb └── README.md /assets/images/pp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ntu-nail/CE7455/HEAD/assets/images/pp.png -------------------------------------------------------------------------------- /assets/images/xb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ntu-nail/CE7455/HEAD/assets/images/xb.png -------------------------------------------------------------------------------- /assets/images/ntcd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ntu-nail/CE7455/HEAD/assets/images/ntcd.png -------------------------------------------------------------------------------- /week1/NumPy-intro.ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"},"colab":{"name":"NumPy-intro.ipynb","provenance":[]}},"cells":[{"cell_type":"markdown","metadata":{"id":"hFPxN_O-8MKZ","colab_type":"text"},"source":["# Introduction to NumPy\n","\n","\n","Before reading this introduction you should know a bit of:\n","1. Python - look at [official tutorial](https://docs.python.org/3/tutorial/)\n","2. Linear Algebra and Matrices - look at [Coursera tutorial](https://www.coursera.org/learn/linear-algebra-machine-learning) and/or book [Introduction to Applied Linear Algebra](http://vmls-book.stanford.edu/vmls.pdf)\n","\n","

\n","\n","From offical NumPy page we could read that \n","```\n","NumPy is the fundamental package for scientific computing with Python. It contains among other things:\n"," * a powerful N-dimensional array object\n"," * useful linear algebra, Fourier transform, and random number capabilities\n"," * sophisticated (broadcasting) functions\n"," * tools for integrating C/C++ and Fortran code \n","```\n","\n","All of that functionalities are very useful if you want to do Machine Learning using **CPU**. To install NumPy package, go to [link](https://scipy.org/install.html). There are also very good tutorials:\n","* [Official NumPy tutorial](https://numpy.org/devdocs/user/quickstart.html)\n","* [From Python to Numpy](https://www.labri.fr/perso/nrougier/from-python-to-numpy/) \n","\n","Here we want give you a quick crash course of using NumPy library."]},{"cell_type":"markdown","metadata":{"ExecuteTime":{"end_time":"2019-10-13T12:02:31.745617Z","start_time":"2019-10-13T12:02:31.737921Z"},"id":"TbFCBdTY8MKc","colab_type":"text"},"source":["## Basics: creating a numpy array \n","\n","Important notes:\n","* all items in NumPy array (a.k.a. `ndarray`) cantain only one data type e.g. `int8`, `float32`, ... ([all datatypes](https://numpy.org/devdocs/user/basics.types.html))\n","* you cannot modify NumPy array"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:31.979567Z","start_time":"2019-10-13T18:13:31.900397Z"},"id":"_4RAm4b98MKd","colab_type":"code","outputId":"e4d6bb8e-f754-469c-ef5f-d503a28d77d5","colab":{}},"source":["import numpy as np # there is a convention to always import numpy as `np`\n","\n","\n","print(\"1d NumPy array from Python list (with `int32` type)\")\n","list1d = [0, 1, 2, 3, 4, 5, 6, 7]\n","ndarray1d = np.array(list1d, dtype='int32') \n","print(ndarray1d) # print ndarray\n","print(ndarray1d.shape) # print ndarray shape\n","print()\n","\n","print(\"2d NumPy array from Python list of lists (with `float32` type)\")\n","list2d = [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13], [14, 15]]\n","ndarray2d = np.array(list2d, dtype='float32')\n","print(ndarray2d) # print ndarray\n","print(ndarray2d.shape) # print ndarray shape\n","print()\n","\n","print(\"1d random array (with `float32` type)\")\n","ndarray1d_random = np.random.rand(8, 2).astype('float32')\n","print(ndarray1d_random) # print ndarray\n","print(ndarray1d_random.shape) # print ndarray shape\n","print()\n","\n","print(\"1d array with uniform distribution\")\n","ndarray1d_uniform = np.random.uniform(-10, 10, 16)\n","print(ndarray1d_uniform) # print ndarray\n","print(ndarray1d_uniform.shape) # print ndarray shape\n","print()\n","\n","print(\"1d array based on linearly spaced vector\")\n","ndarray1d_linspace = np.linspace(0, 7, num=8, dtype='int8')\n","print(ndarray1d_linspace) # print ndarray\n","print(ndarray1d_linspace.shape) # print ndarray shape\n","print()\n","\n","print(\"1d array based on `arange` mechanism\")\n","ndarray1d_arange = np.arange(0, 10, 3)\n","print(ndarray1d_arange) # print ndarray\n","print(ndarray1d_arange.shape) # print ndarray shape\n","print()\n","\n","print(\"2d zeros array\")\n","ndarray2d_zeros = np.zeros([2, 4])\n","print(ndarray2d_zeros) # print ndarray\n","print(ndarray2d_zeros.shape) # print ndarray shape\n","print()\n","\n","print(\"2d ones array\")\n","ndarray2d_ones = np.ones([2, 4])\n","print(ndarray2d_ones) # print ndarray\n","print(ndarray2d_ones.shape) # print ndarray shape\n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["1d NumPy array from Python list (with `int32` type)\n","[0 1 2 3 4 5 6 7]\n","(8,)\n","\n","2d NumPy array from Python list of lists (with `float32` type)\n","[[ 0. 1.]\n"," [ 2. 3.]\n"," [ 4. 5.]\n"," [ 6. 7.]\n"," [ 8. 9.]\n"," [10. 11.]\n"," [12. 13.]\n"," [14. 15.]]\n","(8, 2)\n","\n","1d random array (with `float32` type)\n","[[0.07929848 0.04646103]\n"," [0.13551946 0.09415994]\n"," [0.31865713 0.7731082 ]\n"," [0.17045486 0.6933958 ]\n"," [0.09345574 0.99480355]\n"," [0.6637033 0.32966194]\n"," [0.72250724 0.711149 ]\n"," [0.9396972 0.8823931 ]]\n","(8, 2)\n","\n","1d array with uniform distribution\n","[-7.02870666 7.86882401 9.65007187 -7.83166989 -2.72224666 5.57936585\n"," -5.10631165 -9.66892718 3.56747523 -7.68795799 -1.05204412 6.419049\n"," -5.60641786 -2.30801985 -9.21079302 -4.18085661]\n","(16,)\n","\n","1d array based on linearly spaced vector\n","[0 1 2 3 4 5 6 7]\n","(8,)\n","\n","1d array based on `arange` mechanism\n","[0 3 6 9]\n","(4,)\n","\n","2d zeros array\n","[[0. 0. 0. 0.]\n"," [0. 0. 0. 0.]]\n","(2, 4)\n","\n","2d ones array\n","[[1. 1. 1. 1.]\n"," [1. 1. 1. 1.]]\n","(2, 4)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"ExecuteTime":{"end_time":"2019-10-13T13:12:01.187358Z","start_time":"2019-10-13T13:12:01.183488Z"},"id":"N0Vf9Z9J8MKh","colab_type":"text"},"source":["## Basics: extracting specific values from ndarrays\n","\n","Important notes:\n","* ndarrays can be indexed using the standard Python x[obj] syntax, wherea x is the array and obj the selection, see more at [NumPy indexing](https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html)"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:31.990206Z","start_time":"2019-10-13T18:13:31.982027Z"},"id":"kmjqaArx8MKi","colab_type":"code","outputId":"48ce7c6c-8189-47a0-dabe-1a6c63f72830","colab":{}},"source":["print(\"Get specific element\")\n","print(ndarray2d[1,1])\n","print()\n","\n","print(\"The basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step\")\n","print(ndarray1d[0:6:2])\n","print()\n","\n","print(\"Extract only one dimension from multidimensional ndarray\") \n","print(ndarray2d[:, 0])\n","print()\n","\n","print(\"Boolean array indexing\") \n","print(ndarray1d[([True, False, True, False, True, False, True, False])])\n","print()\n","\n","print(\"Using condition statement for indexing array\") \n","print(ndarray1d[(ndarray1d % 2 == 0)]) \n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Get specific element\n","3.0\n","\n","The basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step\n","[0 2 4]\n","\n","Extract only one dimension from multidimensional ndarray\n","[ 0. 2. 4. 6. 8. 10. 12. 14.]\n","\n","Boolean array indexing\n","[0 2 4 6]\n","\n","Using condition statement for indexing array\n","[0 2 4 6]\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"ExecuteTime":{"end_time":"2019-10-13T13:42:45.916975Z","start_time":"2019-10-13T13:42:45.911823Z"},"id":"YRlYpv_t8MKk","colab_type":"text"},"source":["## Basics: sum, min, max, mean, reshape \n"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:32.002505Z","start_time":"2019-10-13T18:13:31.992418Z"},"id":"w3ugauV88MKl","colab_type":"code","outputId":"45421697-3e69-4093-ffd3-260ab0d6c72d","colab":{}},"source":["print(\"represent `not a number value`\")\n","print(np.nan)\n","print()\n","\n","print(\"represent `infinite`\")\n","print(np.inf)\n","print()\n","\n","print(\"calculate mean, max and min in ndarray\")\n","print(\"sum \", ndarray2d.sum())\n","print(\"max \", ndarray2d.max())\n","print(\"min \", ndarray2d.min())\n","print(\"mean \", ndarray2d.mean())\n","print()\n","\n","print(\"calculate max on different axis\")\n","print(\"column max: \", np.max(ndarray2d, axis=0))\n","print(\"row max: \", np.max(ndarray2d, axis=1))\n","print()\n","\n","print(\"reshape 2d ndarray\")\n","print(ndarray2d.reshape(4, 4))\n","print(ndarray2d.shape, ndarray2d.reshape(4, 4).shape)\n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["represent `not a number value`\n","nan\n","\n","represent `infinite`\n","inf\n","\n","calculate mean, max and min in ndarray\n","sum 120.0\n","max 15.0\n","min 0.0\n","mean 7.5\n","\n","calculate max on different axis\n","column max: [14. 15.]\n","row max: [ 1. 3. 5. 7. 9. 11. 13. 15.]\n","\n","reshape 2d ndarray\n","[[ 0. 1. 2. 3.]\n"," [ 4. 5. 6. 7.]\n"," [ 8. 9. 10. 11.]\n"," [12. 13. 14. 15.]]\n","(8, 2) (4, 4)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"RzdLXvi_8MKn","colab_type":"text"},"source":["## Basics: array math\n"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:32.057925Z","start_time":"2019-10-13T18:13:32.005089Z"},"id":"1V26m-448MKo","colab_type":"code","outputId":"8cb35ac9-7a2b-4ee2-b11d-6f5bc4f81a8f","colab":{}},"source":["print(\"Initialize NumPy array, x1\")\n","x1 = np.ones([3, 3], dtype='float32')\n","x1[:, 0] = [2, 3, 4]\n","print(x1)\n","print()\n","\n","print(\"Transpose x1 array\")\n","print(x1.T)\n","print()\n","\n","print(\"Multiply by scalar, x2=x1*3\")\n","x2 = x1*3.\n","print(x2)\n","print()\n","\n","print(\"Element-wise sum, x1+x2\")\n","print(x1+x2)\n","print()\n","\n","print(\"Element-wise subtract, x1-x2\")\n","print(x1-x2)\n","print()\n","\n","print(\"Element-wise product, x1*x2\")\n","print(x1*x2)\n","print()\n","\n","print(\"Element-wise divide, x1/x2\")\n","print(x1/x2)\n","print()\n","\n","print(\"Element-wise power, x2^2\")\n","print(np.power(x2, 2))\n","print()\n","\n","print(\"Element-wise square root, sqrt(x2)\")\n","print(np.sqrt(x2))\n","print()\n","\n","print(\"Matrix multiplication, x1*x2\")\n","print(x1.dot(x2))\n","print()\n","\n","print(\"Vector multiplication, x1*x2[0]\")\n","print(x1.dot(x2[0]))\n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Initialize NumPy array, x1\n","[[2. 1. 1.]\n"," [3. 1. 1.]\n"," [4. 1. 1.]]\n","\n","Transpose x1 array\n","[[2. 3. 4.]\n"," [1. 1. 1.]\n"," [1. 1. 1.]]\n","\n","Multiply by scalar, x2=x1*3\n","[[ 6. 3. 3.]\n"," [ 9. 3. 3.]\n"," [12. 3. 3.]]\n","\n","Element-wise sum, x1+x2\n","[[ 8. 4. 4.]\n"," [12. 4. 4.]\n"," [16. 4. 4.]]\n","\n","Element-wise subtract, x1-x2\n","[[-4. -2. -2.]\n"," [-6. -2. -2.]\n"," [-8. -2. -2.]]\n","\n","Element-wise product, x1*x2\n","[[12. 3. 3.]\n"," [27. 3. 3.]\n"," [48. 3. 3.]]\n","\n","Element-wise divide, x1/x2\n","[[0.33333334 0.33333334 0.33333334]\n"," [0.33333334 0.33333334 0.33333334]\n"," [0.33333334 0.33333334 0.33333334]]\n","\n","Element-wise power, x2^2\n","[[ 36. 9. 9.]\n"," [ 81. 9. 9.]\n"," [144. 9. 9.]]\n","\n","Element-wise square root, sqrt(x2)\n","[[2.4494898 1.7320508 1.7320508]\n"," [3. 1.7320508 1.7320508]\n"," [3.4641016 1.7320508 1.7320508]]\n","\n","Matrix multiplication, x1*x2\n","[[33. 12. 12.]\n"," [39. 15. 15.]\n"," [45. 18. 18.]]\n","\n","Vector multiplication, x1*x2[0]\n","[18. 24. 30.]\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"ExecuteTime":{"end_time":"2019-10-13T15:46:53.797483Z","start_time":"2019-10-13T15:46:53.793264Z"},"id":"K-VvAo_y8MKr","colab_type":"text"},"source":["## Advanced: broadcasting\n","\n","Broadcasting allows universal functions to deal in a meaningful way with inputs that do not have exactly the same shape (see more at [this link](https://numpy.org/deavdocs/user/basics.broadcasting.html))"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:32.065016Z","start_time":"2019-10-13T18:13:32.059750Z"},"id":"hdhOb-_Z8MKr","colab_type":"code","outputId":"c114b5e2-eec8-4f4e-8f7f-6d67503c5d43","colab":{}},"source":["print(\"Add vector x2 to each row of matrix x1 using broadcasting mechanism\")\n","x1 = np.array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13], [14, 15]])\n","x2 = np.array([1, 3])\n","print(x1+x2)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Add vector x2 to each row of matrix x1 using broadcasting mechanism\n","[[ 1 4]\n"," [ 3 6]\n"," [ 5 8]\n"," [ 7 10]\n"," [ 9 12]\n"," [11 14]\n"," [13 16]\n"," [15 18]]\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /week1/PyTorch-intro.ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"},"colab":{"name":"PyTorch-intro.ipynb","provenance":[]}},"cells":[{"cell_type":"markdown","metadata":{"id":"1aG6DTtWokLY","colab_type":"text"},"source":["# Introduction to PyTorch\n","\n","Before reading this introduction you should know a bit of:\n","1. Python - look at [official tutorial](https://docs.python.org/3/tutorial/)\n","2. Linear Algebra and Matrices - look at [Coursera tutorial](https://www.coursera.org/learn/linear-algebra-machine-learning) and/or book [Introduction to Applied Linear Algebra](http://vmls-book.stanford.edu/vmls.pdf)\n","\n","\n","\n","

\n","\n","\n","From official NumPy page we could read that\n","\n","```\n","PyTorch is a Python-based scientific computing package targeted at two sets of audiences:\n","\n","* A replacement for NumPy to use the power of GPUs\n","* a deep learning research platform that provides maximum flexibility and speed\n","```\n","\n","Contrary to NumPy, PyTorch was designed mostly to work on **GPU**. PyTorch represents n-dimensional array object as `Tensor`. To install PyTorch library, go to [link](https://pytorch.org/get-started/locally/). There are also very good tutorials:\n","* [Official PyTorch tutorials](https://pytorch.org/tutorials/)\n","* [Deep Learning for Natural Language Processing with Pytorch](https://github.com/rguthrie3/DeepLearningForNLPInPytorch/blob/master/Deep%20Learning%20for%20Natural%20Language%20Processing%20with%20Pytorch.ipynb) \n","\n","Here we want give you a quick crash course of using PyTorch library, especially Tensor object. "]},{"cell_type":"markdown","metadata":{"id":"wk2YydsWokLb","colab_type":"text"},"source":["## Basics: creating a PyTorch tensor\n","\n","Important notes:\n","* all items in PyTorch array (a.k.a. `Tensor`) cantain only one data type e.g. `int8`, `float32`, ... ([all datatypes](https://pytorch.org/docs/stable/tensors.html))"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:36.933429Z","start_time":"2019-10-13T18:13:36.703426Z"},"id":"ZeKrtCJiokLc","colab_type":"code","outputId":"fb7ffe8d-ea99-4e85-a3d8-1d2bdada17d8","colab":{}},"source":["import torch \n","\n","print(\"1d Tensor from Python list (with `int32` type)\")\n","list1d = [0, 1, 2, 3, 4, 5, 6, 7]\n","tensor1d = torch.tensor(list1d, dtype=torch.int32) \n","print(tensor1d) # print tensor\n","print(tensor1d.size()) # print tensor shape\n","print()\n","\n","print(\"2d Tensor from Python list of lists (with `float32` type)\")\n","list2d = [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13], [14, 15]]\n","tensor2d = torch.tensor(list2d, dtype=torch.float32)\n","print(tensor2d) # print tensor\n","print(tensor2d.size()) # print tensor shape\n","print()\n","\n","print(\"1d random tensor (with `float32` type)\")\n","tensor1d_random = torch.rand(8, 2, dtype=torch.float32)\n","print(tensor1d_random) # print tensor\n","print(tensor1d_random.size()) # print tensor shape\n","print()\n","\n","print(\"1d tensor with uniform distribution\")\n","tensor1d_uniform = torch.FloatTensor(16, 1).uniform_(-10, 10)\n","print(tensor1d_uniform) # print tensor\n","print(tensor1d_uniform.size()) # print tensor shape\n","print()\n","\n","print(\"1d tensor based on linearly spaced vector\")\n","tensor1d_linspace = torch.linspace(0, 7, steps=8, dtype=torch.float32)\n","print(tensor1d_linspace) # print tensor\n","print(tensor1d_linspace.size()) # print tensor shape\n","print()\n","\n","print(\"1d tensor based on `arange` mechanism\")\n","tensor1d_arange = torch.arange(0, 10, 3)\n","print(tensor1d_arange) # print tensor\n","print(tensor1d_arange.size()) # print tensor shape\n","print()\n","\n","print(\"2d zeros tensor\")\n","torch2d_zeros = torch.zeros([2, 4])\n","print(torch2d_zeros) # print tensor\n","print(torch2d_zeros.size()) # print tensor shape\n","print()\n","\n","print(\"2d ones tensor\")\n","torch2d_ones = torch.ones([2, 4])\n","print(torch2d_ones) # print tensor\n","print(torch2d_ones.size()) # print tensor shape\n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["1d Tensor from Python list (with `int32` type)\n","tensor([0, 1, 2, 3, 4, 5, 6, 7], dtype=torch.int32)\n","torch.Size([8])\n","\n","2d Tensor from Python list of lists (with `float32` type)\n","tensor([[ 0., 1.],\n"," [ 2., 3.],\n"," [ 4., 5.],\n"," [ 6., 7.],\n"," [ 8., 9.],\n"," [10., 11.],\n"," [12., 13.],\n"," [14., 15.]])\n","torch.Size([8, 2])\n","\n","1d random tensor (with `float32` type)\n","tensor([[0.2339, 0.1720],\n"," [0.7893, 0.9454],\n"," [0.3148, 0.3166],\n"," [0.7262, 0.8609],\n"," [0.0912, 0.0767],\n"," [0.5113, 0.8339],\n"," [0.7812, 0.9421],\n"," [0.1299, 0.1272]])\n","torch.Size([8, 2])\n","\n","1d tensor with uniform distribution\n","tensor([[-6.8103],\n"," [ 0.8485],\n"," [ 6.5259],\n"," [ 2.5182],\n"," [ 0.6380],\n"," [-3.1469],\n"," [-3.4691],\n"," [ 4.6706],\n"," [ 2.0206],\n"," [-8.8526],\n"," [-3.8986],\n"," [ 0.9371],\n"," [-9.4456],\n"," [-8.4750],\n"," [-1.1130],\n"," [ 3.2142]])\n","torch.Size([16, 1])\n","\n","1d tensor based on linearly spaced vector\n","tensor([0., 1., 2., 3., 4., 5., 6., 7.])\n","torch.Size([8])\n","\n","1d tensor based on `arange` mechanism\n","tensor([0, 3, 6, 9])\n","torch.Size([4])\n","\n","2d zeros tensor\n","tensor([[0., 0., 0., 0.],\n"," [0., 0., 0., 0.]])\n","torch.Size([2, 4])\n","\n","2d ones tensor\n","tensor([[1., 1., 1., 1.],\n"," [1., 1., 1., 1.]])\n","torch.Size([2, 4])\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"r0xD9G7TokLh","colab_type":"text"},"source":["## Basics: extracting specific values from tensors\n","\n","Important notes:\n","* tensor can be indexed using the standard Python x[obj] syntax, wherea x is the array and obj the selection"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:36.942523Z","start_time":"2019-10-13T18:13:36.935148Z"},"id":"SLhk6f7uokLi","colab_type":"code","outputId":"d828dc61-1e0d-4568-dc1d-dbf7976325e8","colab":{}},"source":["print(\"Get specific element\")\n","print(tensor2d[1,1])\n","print()\n","\n","print(\"The basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step\")\n","print(tensor1d[0:6:2])\n","print()\n","\n","print(\"Extract only one dimension from multidimensional ndarray\") \n","print(tensor2d[:, 0])\n","print()\n","\n","print(\"Boolean array indexing\") \n","print(tensor1d[([True, False, True, False, True, False, True, False])])\n","print()\n","\n","print(\"Using condition statement for indexing array\") \n","print(tensor1d[(tensor1d % 2 == 0)]) \n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Get specific element\n","tensor(3.)\n","\n","The basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step\n","tensor([0, 2, 4], dtype=torch.int32)\n","\n","Extract only one dimension from multidimensional ndarray\n","tensor([ 0., 2., 4., 6., 8., 10., 12., 14.])\n","\n","Boolean array indexing\n","tensor([0, 2, 4, 6], dtype=torch.int32)\n","\n","Using condition statement for indexing array\n","tensor([0, 2, 4, 6], dtype=torch.int32)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"LxsaRlGkokLk","colab_type":"text"},"source":["## Basics: sum, min, max, mean, reshape "]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:36.959263Z","start_time":"2019-10-13T18:13:36.944221Z"},"id":"fKwA8BISokLl","colab_type":"code","outputId":"216dcc03-23a8-4371-e58b-e943d167cacf","colab":{}},"source":["print(\"represent `not a number value`\")\n","print(torch.tensor(float('nan')))\n","print()\n","\n","print(\"represent `infinite`\")\n","print(torch.tensor(float('Inf')))\n","print()\n","\n","print(\"calculate mean, max and min in tensor\")\n","print(\"sum \", tensor2d.sum())\n","print(\"max \", tensor2d.max())\n","print(\"min \", tensor2d.min())\n","print(\"mean \", tensor2d.mean())\n","print()\n","\n","print(\"calculate max on different axis\")\n","print(\"column max: \", tensor2d.max(dim=0)[0])\n","print(\"row max: \", tensor2d.max(dim=1)[0])\n","print()\n","\n","print(\"reshape 2d tensor\")\n","print(tensor2d.view(4, 4))\n","print(tensor2d.size(), tensor2d.view(4, 4).size())\n","print()\n","\n","print(\"reshape 2d tensor (second way)\")\n","print(tensor2d.view(4, -1)) # the second dimention is adjusted to size of the matrix\n","print(tensor2d.size(), tensor2d.view(4, -1).size())\n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["represent `not a number value`\n","tensor(nan)\n","\n","represent `infinite`\n","tensor(inf)\n","\n","calculate mean, max and min in tensor\n","sum tensor(120.)\n","max tensor(15.)\n","min tensor(0.)\n","mean tensor(7.5000)\n","\n","calculate max on different axis\n","column max: tensor([14., 15.])\n","row max: tensor([ 1., 3., 5., 7., 9., 11., 13., 15.])\n","\n","reshape 2d tensor\n","tensor([[ 0., 1., 2., 3.],\n"," [ 4., 5., 6., 7.],\n"," [ 8., 9., 10., 11.],\n"," [12., 13., 14., 15.]])\n","torch.Size([8, 2]) torch.Size([4, 4])\n","\n","reshape 2d tensor (second way)\n","tensor([[ 0., 1., 2., 3.],\n"," [ 4., 5., 6., 7.],\n"," [ 8., 9., 10., 11.],\n"," [12., 13., 14., 15.]])\n","torch.Size([8, 2]) torch.Size([4, 4])\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"MfFeOqbwokLn","colab_type":"text"},"source":["## Basics: tensor math"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:36.979309Z","start_time":"2019-10-13T18:13:36.961184Z"},"id":"u9qNagd4okLo","colab_type":"code","outputId":"af6f17cc-b86a-47f6-ca9e-ffadf4a005dc","colab":{}},"source":["print(\"Initialize tensor, x1\")\n","x1 = torch.ones([3, 3], dtype=torch.float32)\n","x1[:, 0] = torch.tensor([2., 3., 4.])\n","print(x1)\n","print()\n","\n","print(\"Transpose x1 tensor\")\n","print(x1.transpose(0, 1))\n","print()\n","\n","print(\"Multiply by scalar, x2=x1*3\")\n","x2 = x1*3.\n","print(x2)\n","print()\n","\n","print(\"Element-wise sum, x1+x2\")\n","print(x1+x2)\n","print()\n","\n","print(\"Element-wise subtract, x1-x2\")\n","print(x1-x2)\n","print()\n","\n","print(\"Element-wise product, x1*x2\")\n","print(x1*x2)\n","print()\n","\n","print(\"Element-wise divide, x1/x2\")\n","print(x1/x2)\n","print()\n","\n","print(\"Element-wise power, x2^2\")\n","print(torch.pow(x2, 2))\n","print()\n","\n","print(\"Element-wise square root, sqrt(x2)\")\n","print(torch.sqrt(x2))\n","print()\n","\n","print(\"Matrix multiplication, x1*x2\")\n","print(x1.mm(x2))\n","print()\n","\n","print(\"Vector multiplication, x1*x2[0]\")\n","print(x1.mm(x2[0].view([-1, 1])).view(-1))\n","print()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Initialize tensor, x1\n","tensor([[2., 1., 1.],\n"," [3., 1., 1.],\n"," [4., 1., 1.]])\n","\n","Transpose x1 tensor\n","tensor([[2., 3., 4.],\n"," [1., 1., 1.],\n"," [1., 1., 1.]])\n","\n","Multiply by scalar, x2=x1*3\n","tensor([[ 6., 3., 3.],\n"," [ 9., 3., 3.],\n"," [12., 3., 3.]])\n","\n","Element-wise sum, x1+x2\n","tensor([[ 8., 4., 4.],\n"," [12., 4., 4.],\n"," [16., 4., 4.]])\n","\n","Element-wise subtract, x1-x2\n","tensor([[-4., -2., -2.],\n"," [-6., -2., -2.],\n"," [-8., -2., -2.]])\n","\n","Element-wise product, x1*x2\n","tensor([[12., 3., 3.],\n"," [27., 3., 3.],\n"," [48., 3., 3.]])\n","\n","Element-wise divide, x1/x2\n","tensor([[0.3333, 0.3333, 0.3333],\n"," [0.3333, 0.3333, 0.3333],\n"," [0.3333, 0.3333, 0.3333]])\n","\n","Element-wise power, x2^2\n","tensor([[ 36., 9., 9.],\n"," [ 81., 9., 9.],\n"," [144., 9., 9.]])\n","\n","Element-wise square root, sqrt(x2)\n","tensor([[2.4495, 1.7321, 1.7321],\n"," [3.0000, 1.7321, 1.7321],\n"," [3.4641, 1.7321, 1.7321]])\n","\n","Matrix multiplication, x1*x2\n","tensor([[33., 12., 12.],\n"," [39., 15., 15.],\n"," [45., 18., 18.]])\n","\n","Vector multiplication, x1*x2[0]\n","tensor([18., 24., 30.])\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:08:48.844141Z","start_time":"2019-10-13T18:08:48.839846Z"},"id":"m-TIC6l7okLq","colab_type":"text"},"source":["## Advanced: broadcasting\n","\n","In short, if a PyTorch operation supports broadcast, then its Tensor arguments can be automatically expanded to be of equal sizes (see more at [this link](https://pytorch.org/docs/stable/notes/broadcasting.html))"]},{"cell_type":"code","metadata":{"ExecuteTime":{"end_time":"2019-10-13T18:13:36.990697Z","start_time":"2019-10-13T18:13:36.981482Z"},"id":"0Ox25KQSokLr","colab_type":"code","outputId":"838b2457-843e-45b7-efa5-a109eef4aadc","colab":{}},"source":["print(\"Add vector x2 to each row of matrix x1 using broadcasting mechanism\")\n","x1 = torch.tensor([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13], [14, 15]])\n","x2 = torch.tensor([1, 3])\n","print(x1+x2)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Add vector x2 to each row of matrix x1 using broadcasting mechanism\n","tensor([[ 1, 4],\n"," [ 3, 6],\n"," [ 5, 8],\n"," [ 7, 10],\n"," [ 9, 12],\n"," [11, 14],\n"," [13, 16],\n"," [15, 18]])\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CE7455: Deep Learning For Natural Language Processing 2 | 3 | # Course Objectives 4 | 5 | Natural language processing (NLP) is one of the most important fields in artificial intelligence (AI). It has become very crucial in the information age because most of the information is in the form of unstructured text. NLP technologies are applied everywhere as people communicate mostly in language: language translation, web search, customer support, emails, forums, advertisement, radiology reports, to name a few. 6 | 7 | There are several core NLP tasks and machine learning models behind NLP applications. Deep learning, a sub-field of machine learning, has recently brought a paradigm shift from traditional task-specific feature engineering to end-to-end systems and has obtained high performance across many different NLP tasks and downstream applications. Tech companies like Google, Baidu, Alibaba, Apple, Amazon, Facebook, Tencent, and Microsoft are now actively working on deep learning methods to improve their products. For example, Google recently replaced its traditional statistical machine translation and speech-recognition systems with systems based on deep learning methods. 8 | 9 | **Optional Textbooks** 10 | 11 | - Deep Learning by Goodfellow, Bengio, and Courville [free online](http://www.deeplearningbook.org/) 12 | - Machine Learning — A Probabilistic Perspective by Kevin Murphy [online](https://doc.lagout.org/science/Artificial%20Intelligence/Machine%20learning/Machine%20Learning_%20A%20Probabilistic%20Perspective%20%5BMurphy%202012-08-24%5D.pdf) 13 | - Natural Language Processing by Jacob Eisenstein [free online](https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf) 14 | - Speech and Language Processing by Dan Jurafsky and James H. Martin [(3rd ed. draft)](https://web.stanford.edu/~jurafsky/slp3/) 15 | 16 | # Intended Learning Outcomes 17 | 18 | In this course, students will learn state-of-the-art deep learning methods for NLP. Through lectures and practical assignments, students will learn the necessary tricks for making their models work on practical problems. They will learn to implement, and possibly invent their deep learning models using available deep learning libraries like [Pytorch](https://pytorch.org/). 19 | 20 | **Our Approach** 21 | 22 | - Thorough and Detailed: How to write from scratch, debug, and train deep neural models 23 | 24 | - State of the art: Most lecture materials are new from the research world in the past 1-5 years. 25 | 26 | - Practical: Focus on practical techniques for training the models on GPUs. 27 | 28 | - Fun: Cover exciting new advancements in NLP (e.g., Transformer, BERT). 29 | 30 | # Assessment Approach 31 | 32 | **Weekly Workload** 33 | 34 | - Lecture and practical problems implemented in PyTorch. 35 | - There will be NO office hours. 36 | 37 | **Assignments (individually graded)** 38 | 39 | - Two (2) assignments will contribute to ***2 * 25% = 50%*** of the total assessment. 40 | - Students will be graded individually on the assignments. They will be allowed to discuss with each other on the homework assignments, but they are required to submit individual write-ups and coding exercises. 41 | 42 | **Final Project (Group work but individually graded)** 43 | 44 | - There will be a final project contributing to the remaining 50% of the total coursework assessment. 45 | - ***3–5*** students per group 46 | - Project proposal: ***5%***, presentation: ***15%***, report: ***30%*** 47 | - The project will be a group work. Students will be graded individually, depending on their contribution to the group. The final project presentation will ensure the student’s understanding of the project. 48 | 49 | # Course Prerequisites 50 | 51 | - Proficiency in Python (using Numpy and PyTorch). There is a lecture for those who are not familiar with Python. 52 | - Linear Algebra, basic Probability and Statistics 53 | - Machine Learning basics 54 | 55 | # Teaching 56 | 57 | ## Instructors 58 | 59 |

60 | 61 |

Luu Anh Tuan (Part 2)

62 | 63 | ## Teaching Assistants 64 | 65 |

He Qiyuan (Part 1)

66 |

qiyuan001@e.ntu.edu.sg

67 | 68 |

Nguyen Tran Cong Duy (Part 2)

69 |

nguyentr003@e.ntu.edu.sg

70 | 71 | # Schedule & Course Content 72 | 73 | ## Week 1: Introduction 74 | 75 | [Lecture Slide](https://drive.google.com/file/d/1cVAxrntk1Q4R_LC1zFQufkfyakG21E-y/view?usp=sharing) 76 | 77 | ### Lecture Content 78 | 79 | - What is Natural Language Processing? 80 | - Why is language understanding difficult? 81 | - What is Deep Learning? 82 | - Deep learning vs. other machine learning methods? 83 | - Why deep learning for NLP? 84 | - Applications of deep learning to NLP 85 | - Knowing the target group (background, field of study, programming experience) 86 | - Expectation from the course 87 | 88 | ### Python & PyTorch Basics 89 | 90 | - Programming in Python 91 | 92 | - Jupiter Notebook and [Google Colab](https://colab.research.google.com/drive/16pBJQePbqkz3QFV54L4NIkOn1kwpuRrj) 93 | - [Introduction to Python](https://colab.research.google.com/drive/1bQG32CFoMZ-jBk02uaFon60tER3yFx4c) 94 | - Deep Learning Frameworks 95 | - Why Pytorch? 96 | - [Deep learning with PyTorch](https://drive.google.com/file/d/1c33y8bkdr7SJ_I8-wmqTAhld-y7KcspA/view?usp=sharing) 97 | - [Supplementary] 98 | - Numerical programming with Numpy/Scipy - [Numpy intro](https://drive.google.com/file/d/1cUzRzQGURrCKes8XynvTTA4Zvl_gUJdc/view?usp=sharing) 99 | - Numerical programming with Pytorch - [Pytorch intro](https://drive.google.com/file/d/18cgPOj2QKQN0WR9_vXoz6BoravvS9mTm/view?usp=sharing) 100 | 101 | 102 | ## Week 2: Machine Learning Basics 103 | 104 | [Lecture Slide](https://docs.google.com/presentation/d/1oRnv9yGMOm2vbznzS7ii4TB7PCxVD1bawtAGBAj0kwU/edit?usp=sharing) 105 | 106 | ### Lecture Content 107 | 108 | - What is Machine Learning? 109 | - Machine learning for text classification 110 | - Naive Bayes 111 | - Logistic Regression 112 | - Multi-class classification 113 | - Gradient-based optimization 114 | 115 | ### Practical exercise with Pytorch 116 | 117 | - [Deep learning with PyTorch](https://colab.research.google.com/drive/1aZVfsPUko-ugt1TVCmRwqGJXlxEJVaTq?usp=sharing) 118 | - [Linear Regression](https://colab.research.google.com/drive/12QpBf7x_Jt6-zypN4OrUFFHXz1u6CmYe?usp=sharing) 119 | - [Logistic Regression](https://colab.research.google.com/drive/1nTrYW5dUu6WO9cx7SGEvP9oX7qRbsGJk?usp=sharing) 120 | - [Supplementary] 121 | - Numerical programming with Pytorch - [Pytorch intro](https://drive.google.com/file/d/18cgPOj2QKQN0WR9_vXoz6BoravvS9mTm/view?usp=sharing) 122 | 123 | 124 | 125 | ## Week 3: Neural Networks & Optimization Basics 126 | 127 | [Lecture Slide](https://drive.google.com/file/d/1jc9T1-6WJ6GizPyRXKq7boLl0G9DgH9D/view?usp=sharing) 128 | 129 | ### Lecture Content 130 | 131 | - From Logistic Regression to Feed-forward NN 132 | - Activation functions 133 | - SGD with Backpropagation 134 | - Adaptive SGD (adagrad, adam, RMSProp) 135 | - Regularization (Dropout, Batch normalization, L1/L2 norm, Gradient clipping) 136 | 137 | ### Practical exercise with Pytorch 138 | 139 | - [Deep learning with PyTorch](https://colab.research.google.com/drive/1aZVfsPUko-ugt1TVCmRwqGJXlxEJVaTq?usp=sharing) 140 | - [Numpy notebook](https://colab.research.google.com/drive/1IAonxZnZjJb0_xUVWHt5atIxaI5GTJQ2) [Pytorch notebook](https://colab.research.google.com/drive/1YzZrMAmJ3hjvJfNIdGxae9kxGABG6yaT) 141 | - Backpropagation 142 | - Dropout 143 | - Batch normalization 144 | - Initialization 145 | - Gradient clipping 146 | 147 | 148 | ## Week 4: Word Vectors 149 | 150 | [Lecture Slide](https://drive.google.com/file/d/1uLkxAm1vDEPBvYn5lOkMHTtPxSiMOyU5/view?usp=sharing) 151 | 152 | 153 | [Project Proposal Instruction](https://docs.google.com/document/d/1i6QlzNX-HDjkW5HwRTLX8lYC6F4tCwhO40oqUItMP_s/edit?usp=sharing) 154 | 155 | [Group Allocation (form your own group)](https://docs.google.com/spreadsheets/d/13zTwv7U5AFvr7SfBUFFT2W7tWx8PCJm1kVUqpVPFqWg/edit?usp=sharing) 156 | 157 | 158 | ### Lecture Content 159 | 160 | - Word meaning 161 | - Distributed representation of words 162 | - Word2Vec models (Skip-gram, CBOW) 163 | - Negative sampling 164 | - Glove 165 | - FastText 166 | - Evaluating word vectors 167 | - Intrinsic evaluation 168 | - Extrinsic evaluation 169 | - Cross-lingual word embeddings 170 | 171 | ### Practical exercise with Pytorch 172 | 173 | [Skip-gram training](https://colab.research.google.com/drive/164dB-Vemzwavf1ffqDDVNtx7Y5VtcmQh) 174 | 175 | ### Suggested Readings 176 | 177 | - Word2Vec Tutorial - The Skip-Gram Model, [blog](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/) 178 | - [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781) - Original word2vec paper 179 | - [Distributed Representations of Words and Phrases and their Compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) - negative sampling paper 180 | - [GloVe: Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf) 181 | - [FastText: Enriching Word Vectors with Subword Information](https://www.mitpressjournals.org/doi/abs/10.1162/tacl_a_00051?mobileUi=0) 182 | - [Linguistic Regularities in Sparse and Explicit Word Representations.](https://levyomer.files.wordpress.com/2014/04/linguistic-regularities-in-sparse-and-explicit-word-representations-conll-2014.pdf) 183 | - [Neural Word Embeddings as Implicit Matrix Factorization.](https://arxiv.org/abs/1702.02098) 184 | - [Survey on Cross-lingual embedding methods](https://arxiv.org/abs/1706.04902) 185 | - [Slides on Cross-lingual embedding](https://www.dropbox.com/s/3eq5apr75yrz9ix/Cross-lingual%20word%20embeddings%20and%20beyond.pdf?dl=0) 186 | - [Adversarial autoencoder for unsupervised word translation](https://arxiv.org/abs/1904.04116) 187 | - [Evaluating Cross-Lingual Word Embeddings](https://www.aclweb.org/anthology/P19-1070) 188 | 189 | 190 | ## Week 5: Window-based Approach and Convolutional Nets 191 | 192 | [Lecture Slide](https://drive.google.com/file/d/1MpVGB3UBvPhUdcuj0r_ui_WkoMJ5IBUE/view?usp=sharing) 193 | 194 | ### Lecture Content 195 | 196 | - Classification tasks in NLP 197 | - Window-based Approach for language modeling 198 | - Window-based Approach for NER, POS tagging, and Chunking 199 | - Convolutional Neural Net for NLP 200 | - Loss Functions 201 | 202 | ### Suggested Readings 203 | 204 | - [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://transacl.org/ojs/index.php/tacl/article/viewFile/1346/320) 205 | - [Improving Distributional Similarity with Lessons Learned from Word Embeddings](https://www.aclweb.org/anthology/Q15-1016/) 206 | - [Natural Language Processing (Almost) from Scratch](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) 207 | - [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882) 208 | - [Fast and Accurate Entity Recognition with Iterated Dilated Convolutions](https://arxiv.org/abs/1702.02098) 209 | 210 | ### Practical exercise with Pytorch 211 | 212 | [Named Entity Recognition](https://colab.research.google.com/drive/1aDFQtSCYBpgRwkj9eq8Rao9eq5_68C31) 213 | 214 | [CNN for text classification](https://colab.research.google.com/drive/1r3qUTV0AQBV9cUAF0GML6Yu3aMfPdBGQ#scrollTo=mc6ZP_xosSil) 215 | 216 | 217 | ## Week 6: Recurrent Neural Nets 218 | 219 | [Lecture Slide](https://drive.google.com/file/d/1jGUvuEp1rhNXsXK-BBpnlmmucqGt4uLc/view?usp=sharing) 220 | 221 | 222 | Assignment 1 is out [here](https://docs.google.com/document/d/1oVHB2kby-G9ikIdS6dxnIMiDz1gPbQw4dd7_QghXt7M/edit?usp=sharing). **Deadline: 23 March 2025**. 223 | 224 | ### Lecture Content 225 | 226 | 227 | - Language modeling with RNNs 228 | - Backpropagation through time 229 | - Text generation with RNN LM 230 | - Sequence labeling with RNNs 231 | - Sequence classification with RNNs 232 | - Issues with Vanilla RNNs 233 | - Gated Recurrent Units (GRUs) and LSTMs 234 | - Bidirectional RNNs 235 | - Multi-layer RNNs 236 | - Recursive Neural Nets 237 | 238 | ### Practical exercise with Pytorch (RNN for POS Tagging) 239 | 240 | - [POS Tagging](https://colab.research.google.com/drive/1a_0JGhg-N9pItzUDwfULtuxdbnh1m1y8) 241 | 242 | ### Suggested Readings 243 | 244 | - [N-gram Language Models](https://web.stanford.edu/~jurafsky/slp3/3.pdf) 245 | - [Karpathy’s nice blog on Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) 246 | - [Building an Efficient Neural Language Model](https://research.fb.com/building-an-efficient-neural-language-model-over-a-billion-words/) 247 | - [On the difficulty of training recurrent neural networks](http://proceedings.mlr.press/v28/pascanu13.pdf) 248 | - [Colah’s blog on LSTMs/GRUs](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) 249 | - [Neural Architectures for Named Entity Recognition](https://www.aclweb.org/anthology/N16-1030/) 250 | - [Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings](https://www.aclweb.org/anthology/D15-1168/) 251 | 252 | 253 | 254 | ## Week 7: Machine translation and Seq2Seq Models 255 | 256 | [Lecture Slide](https://drive.google.com/file/d/1BxNxakiFSN9FYEbA2VKSuHQSmWA1sXyz/view?usp=sharing) 257 | 258 | ### Lecture Content 259 | 260 | - Machine translation 261 | - Early days (1950s) 262 | - Statistical machine translation or SMT (1990-2010) 263 | - Alignment in SMT 264 | - Neural machine translation or NMT (2014 - ) 265 | - Encoder-decoder model for NMT 266 | - Advantages and disadvantages of NMT 267 | - Decoding strategies such as beam search and sampling 268 | - MT evaluation 269 | - Other evaluation methods for language generation 270 | - Review 271 | 272 | ### Suggested Readings 273 | 274 | - [Statistical Machine Translation slides, CS224n 2015 (lectures 2/3/4)](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/syllabus.shtml) 275 | - [Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)](https://arxiv.org/pdf/1409.3215.pdf) 276 | - [Statistical Machine Translation (book by Philipp Koehn)](https://www.cambridge.org/core/books/statistical-machine-translation/94EADF9F680558E13BE759997553CDE5) 277 | - [A Neural Conversational Model](https://arxiv.org/abs/1506.05869) 278 | - [BLEU (original paper)](https://www.aclweb.org/anthology/P02-1040.pdf) 279 | 280 | ### Practical exercise with Pytorch 281 | 282 | - [Neural Machine Translation](https://colab.research.google.com/drive/1htCUnTIFY9NAhbANuuZEeHGxvfPGHBdK) 283 | 284 | 285 | ## Week 8: Seq2Seg Models, Attentions, Subwords 286 | 287 | [Lecture Slide](https://drive.google.com/file/d/14stMogPaHp6-s2bQqo-eR1FvzJ5Dt_T1/view?usp=share_link) 288 | 289 | ### Lecture Content 290 | 291 | - Information bottleneck issue with vanilla Seq2Seq 292 | - Attention to the rescue 293 | - Details of attention mechanism 294 | - Sub-word models 295 | - Byte-pair encoding 296 | - Hybrid models 297 | 298 | 299 | ### Practical exercise with Pytorch 300 | 301 | - [Neural machine translation tutorial in pytorch](https://colab.research.google.com/drive/1cYyBxmdjFjKls0CEsPc8WIHfxPDy4eSq) 302 | 303 | 304 | ### Suggested Readings 305 | 306 | - [Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)](https://arxiv.org/pdf/1409.0473.pdf) 307 | - [Effective Approaches to Attention-based Neural Machine Translation](https://nlp.stanford.edu/~lmthang/data/papers/emnlp15_attn.pdf) 308 | - [Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models](https://arxiv.org/abs/1604.00788) 309 | 310 | 311 | 312 | ## Week 9: Seq2Seq Variants and Transformer 313 | 314 | [Lecture Slide](https://drive.google.com/file/d/1wWxMg_tIVdShRccSyH-v38g-LKmOcO3L/view?usp=sharing) 315 | 316 | ### Lecture Content 317 | 318 | - Seq2Seq Variants (Pointer nets, Pointer Generator Nets) 319 | - Machine Translation 320 | - Summarization 321 | - Transformer architecture 322 | - Self-attention 323 | - Positional encoding 324 | - Multi-head attention 325 | 326 | [The Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) 327 | 328 | ### Suggested Readings 329 | 330 | - [Get To The Point: Summarization with Pointer-Generator Networks](https://arxiv.org/abs/1704.04368) 331 | - [Pointer Networks](https://papers.nips.cc/paper/5866-pointer-networks) 332 | - [Stack-Pointer Networks for Dependency Parsing](https://www.aclweb.org/anthology/P18-1130.pdf) 333 | - [A Unified Linear-Time Framework for Sentence-Level Discourse Parsing](https://arxiv.org/abs/1905.05682) 334 | - [Attention Is All You Need](https://arxiv.org/abs/1706.03762) 335 | - [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) 336 | - [Resurrecting Submodularity in Neural Abstractive Summarization](https://arxiv.org/abs/1911.03014) 337 | 338 | 339 | 340 | ## Week 10: Contextual embeddings and self-supervised learning 341 | 342 | [Lecture Slide](https://drive.google.com/file/d/18upNeGM8YdUMl9dn-FqfVrIeFVfb3mtb/view?usp=share_link) 343 | 344 | 345 | 346 | ### Lecture Content 347 | 348 | - Why semi-supervsied? 349 | - Semisupervised learning dimensions 350 | - Pre-training and fine-tuning methods 351 | 352 | - CoVe 353 | - TagLM 354 | - ELMo 355 | - GPT 356 | - ULMfit 357 | - BERT 358 | - BART 359 | - Evaluation benchmarks 360 | 361 | - GLUE 362 | - SQuAD 363 | - NER 364 | - SuperGLUE 365 | - XNLI 366 | 367 | [Pre-train Fine-tune with HF](https://colab.research.google.com/drive/1L_hwnQISoIBrH7W_r83I62hJ4FBlfNsz?usp=sharing) 368 | 369 | ### Suggested Readings 370 | 371 | - [Cove Paper](https://arxiv.org/pdf/1708.00107.pdf) 372 | - [ULMFit paper](https://arxiv.org/abs/1801.06146) 373 | - [BERT Paper](https://arxiv.org/abs/1810.04805) 374 | - [ELMo paper](https://arxiv.org/abs/1802.05365) 375 | - [BART paper](https://arxiv.org/abs/1910.13461) 376 | 377 | 378 | ## Week 11: Large Language Models & Multilingual NLP 379 | 380 | 381 | Assignment 2 is out [here](https://docs.google.com/document/d/1AonpEdaFc3Tgu0RdEux-ZMaJ0-RHOrPp/edit?usp=sharing&ouid=118265590333925180950&rtpof=true&sd=true). **Deadline: 23 Apr 2025, 11:59pm**. 382 | 383 | [Final project report instruction](https://drive.google.com/file/d/1zV7cV4hOMWaq2NcvoHumkx6U16HDATeL/view?usp=sharing) 384 | 385 | [Lecture Slide](https://drive.google.com/file/d/1L5H0_e7tVQw-lBELcejppVs2DTY6hb2b/view?usp=share_link) 386 | 387 | ### Lecture Content 388 | 389 | - Large Language Models 390 | - Examples of Large Language Models 391 | - Multilingual NLP 392 | - Why do we need Multilingual NLP? 393 | - Low-resource NLP 394 | - Cross-lingual models 395 | - Multilingual models 396 | 397 | ### Suggested Readings 398 | 399 | - [XLM paper](https://arxiv.org/abs/1901.07291) 400 | - [Transformer XL paper](https://arxiv.org/pdf/1901.02860.pdf) 401 | - [XLNet paper](https://arxiv.org/abs/1906.08237) 402 | - [mBART paper](https://arxiv.org/abs/2001.08210) 403 | 404 | ## Week 12: LLM Prompting 405 | 406 | [Lecture Slide](https://drive.google.com/file/d/1bH3rNigKxn3mhJ8KKUeaTi6mMLiRL0SH/view?usp=sharing) 407 | 408 | ### Lecture Content 409 | 410 | - Chain-of-Thought Prompting 411 | - Self-Consistency Improves Chain of Thought Reasoning in Language Models 412 | - Program of Thoughts Prompting 413 | - Least-to-Most Prompting Enables Complex Reasoning in Large Language Models 414 | - Measuring and Narrowing the Compositionality Gap in Language Models 415 | 416 | 437 | -------------------------------------------------------------------------------- /week1/PytorchTut.ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"PytorchTut.ipynb","provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"q-nLVTY_LwXh"},"source":["# Pytorch Basics\n","\n","\n","#### Table of contents\n","0. Pytorch tensors\n","\n","1. Basic autograd example 1\n","\n","2. Basic autograd example 2\n","\n","3. Loading data from numpy \n","\n","4. Input pipeline \n","\n","5. Input pipeline for custom dataset \n","\n","6. Pretrained model \n","\n","7. Save and load model\n","\n","8. Train a simple MNIST Neural nets\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"LeO-UXC-OZ_A"},"source":["import torch \n","import torchvision\n","import torch.nn as nn\n","import numpy as np\n","import torchvision.transforms as transforms\n","import matplotlib.pyplot as plt\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"bwEVQF5xagPM"},"source":["##0. Pytorch tensors"]},{"cell_type":"code","metadata":{"id":"e4YZOt2tajAB"},"source":["# Very similar to numpy\n","x = torch.tensor(1.)\n","print(x)\n","\n","x = torch.tensor([1,2,3,4,5]).float()\n","print(x)\n","print(x.size())\n","\n","ran = torch.Tensor(2,5,5).uniform_()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"rQjMkMYia60M"},"source":["# min, max, mean, reshaping\n","ran = torch.Tensor(2,5,5).uniform_()\n","print(ran)\n","print(ran.size())\n","print(ran.min())\n","print(ran.max())\n","print(ran.mean())\n","rsan = ran.view(5,2,5)\n","print(rsan)\n","print(rsan.size())"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"hZthx9bibTog"},"source":["# Tensor math\n","x = torch.Tensor(2, 4, 5).uniform_()\n","y = torch.Tensor(2, 4, 5).uniform_()\n","print(x.size())\n","print(y.size())\n","print(x)\n","print(y)\n","a = x + y\n","m = x * y\n","# transpose of y (the last 2 dimension)\n","yt = y.transpose(1, 2)\n","print(yt.size())\n","matmul = torch.matmul(x, yt)\n","print(matmul.size())\n","print(matmul)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NvmBBRDnPjIH"},"source":["##1. Basic autograd example 1"]},{"cell_type":"code","metadata":{"id":"5iHZVz5GOdbb"},"source":["# Create tensors.\n","x = torch.tensor(1., requires_grad=True)\n","w = torch.tensor(2., requires_grad=True)\n","b = torch.tensor(3., requires_grad=True)\n","\n","# Build a computational graph.\n","y = w * x + b # y = 2 * x + 3\n","\n","# Compute gradients.\n","y.backward()\n","\n","# Print out the gradients.\n","print(x.grad) # x.grad = 2 \n","print(w.grad) # w.grad = 1 \n","print(b.grad) # b.grad = 1 \n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9nMe0u_oPyD-"},"source":["##1. Basic autograd example 1"]},{"cell_type":"code","metadata":{"id":"oUvKhddOPcBM","colab":{"base_uri":"https://localhost:8080/","height":187},"executionInfo":{"status":"ok","timestamp":1573203612612,"user_tz":-480,"elapsed":2969,"user":{"displayName":"Xuan Phi Nguyen","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBMe29t94knDbWFUTdAsYnOA_RwiofW5MdraN0v1TU=s64","userId":"15291677822697214066"}},"outputId":"4e684706-ef6c-453d-e5af-cb895b7a4856"},"source":["# Create tensors of shape (10, 3) and (10, 2).\n","x = torch.randn(10, 3)\n","y = torch.randn(10, 2)\n","\n","# Build a fully connected layer.\n","linear = nn.Linear(3, 2)\n","print ('w: ', linear.weight)\n","print ('b: ', linear.bias)\n","\n","# Build loss function and optimizer.\n","criterion = nn.MSELoss()\n","optimizer = torch.optim.SGD(linear.parameters(), lr=0.01)\n","\n","# Forward pass.\n","pred = linear(x)\n","\n","# Compute loss.\n","loss = criterion(pred, y)\n","print('loss: ', loss.item())\n","\n","# Backward pass.\n","loss.backward()\n","\n","# Print out the gradients.\n","print ('dL/dw: ', linear.weight.grad) \n","print ('dL/db: ', linear.bias.grad)\n","\n","# 1-step gradient descent.\n","optimizer.step()\n","\n","# You can also perform gradient descent at the low level.\n","# linear.weight.data.sub_(0.01 * linear.weight.grad.data)\n","# linear.bias.data.sub_(0.01 * linear.bias.grad.data)\n","\n","# Print out the loss after 1-step gradient descent.\n","pred = linear(x)\n","loss = criterion(pred, y)\n","print('loss after 1 step optimization: ', loss.item())\n","\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["w: Parameter containing:\n","tensor([[-0.0474, 0.0154, 0.1638],\n"," [-0.1966, -0.2304, 0.4822]], requires_grad=True)\n","b: Parameter containing:\n","tensor([-0.2659, 0.1617], requires_grad=True)\n","loss: 0.8229060769081116\n","dL/dw: tensor([[ 0.3962, 0.3727, 0.3502],\n"," [ 0.1079, -0.2590, 0.1530]])\n","dL/db: tensor([-0.5923, -0.2813])\n","loss after 1 step optimization: 0.8134800791740417\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"KiJypsSQPzUC"},"source":["##3. Loading data from numpy"]},{"cell_type":"code","metadata":{"id":"RpFqOS4GQMeC"},"source":["# Create a numpy array.\n","x = np.array([[1, 2], [3, 4]])\n","\n","# Convert the numpy array to a torch tensor.\n","y = torch.from_numpy(x)\n","\n","# Convert the torch tensor to a numpy array.\n","z = y.numpy()\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Y8BGz1VxP2JY"},"source":["##4. Input pipeline"]},{"cell_type":"code","metadata":{"id":"hqXhTlCxQPwy"},"source":["# Download and construct CIFAR-10 dataset.\n","train_dataset = torchvision.datasets.CIFAR10(\n"," root='.',\n"," train=True, \n"," transform=transforms.ToTensor(),\n"," download=True)\n","\n","# Fetch one data pair (read data from disk).\n","image, label = train_dataset[0]\n","print (image.size())\n","print (label)\n","\n","# Data loader (this provides queues and threads in a very simple way).\n","train_loader = torch.utils.data.DataLoader(\n"," dataset=train_dataset,\n"," batch_size=64, \n"," shuffle=True)\n","\n","# When iteration starts, queue and thread start to load data from files.\n","data_iter = iter(train_loader)\n","\n","# Mini-batch images and labels.\n","images, labels = data_iter.next()\n","\n","# Actual usage of the data loader is as below.\n","for images, labels in train_loader:\n"," # Training code should be written here.\n"," pass\n","\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"22zI4Mi-P3nk"},"source":["##5. Input pipeline for custom dataset"]},{"cell_type":"code","metadata":{"id":"km9Ma1TTQcZB"},"source":["# You should build your custom dataset as below.\n","class CustomDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," # TODO\n"," # 1. Initialize file paths or a list of file names. \n"," pass\n"," def __getitem__(self, index):\n"," # TODO\n"," # 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).\n"," # 2. Preprocess the data (e.g. torchvision.Transform).\n"," # 3. Return a data pair (e.g. image and label).\n"," pass\n"," def __len__(self):\n"," # You should change 0 to the total size of your dataset.\n"," return 0 \n","\n","# You can then use the prebuilt data loader. \n","custom_dataset = CustomDataset()\n","train_loader = torch.utils.data.DataLoader(\n"," dataset=custom_dataset,\n"," batch_size=64, \n"," shuffle=True)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nyDUflRGP3v_"},"source":["##6. Pretrained model"]},{"cell_type":"code","metadata":{"id":"ez0wfFfTQg06"},"source":["# Download and load the pretrained ResNet-18.\n","resnet = torchvision.models.resnet18(pretrained=True)\n","\n","# If you want to finetune only the top layer of the model, set as below.\n","for param in resnet.parameters():\n"," param.requires_grad = False\n","\n","# Replace the top layer for finetuning.\n","resnet.fc = nn.Linear(resnet.fc.in_features, 100) # 100 is an example.\n","\n","# Forward pass.\n","images = torch.randn(64, 3, 224, 224)\n","outputs = resnet(images)\n","print (outputs.size()) # (64, 100)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"oX-SQZO0P36F"},"source":["##7. Save and load model"]},{"cell_type":"code","metadata":{"id":"zDpSgZKFQjKJ"},"source":["# Save and load the entire model.\n","torch.save(resnet, 'model.ckpt')\n","model = torch.load('model.ckpt')\n","\n","# Save and load only the model parameters (recommended).\n","torch.save(resnet.state_dict(), 'params.ckpt')\n","resnet.load_state_dict(torch.load('params.ckpt'))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"iK1YQYVgUurI"},"source":["##8. Train a simple MNIST Neural nets\n","Let's train a simple neural network to classify MNIST hand-written digit\n"]},{"cell_type":"markdown","metadata":{"id":"8WyC0tZ2U-rX"},"source":["####1. Download the dataset"]},{"cell_type":"code","metadata":{"id":"1TRyf_4lUwqC"},"source":["import torch\n","import torch.nn as nn\n","import torchvision\n","import torchvision.transforms as transforms\n","import matplotlib.pyplot as plt\n","\n","\n","# Device configuration\n","device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n","\n","# Hyper-parameters \n","input_size = 784\n","hidden_size = 500\n","num_classes = 10\n","num_epochs = 5\n","batch_size = 100\n","learning_rate = 0.001\n","\n","# MNIST dataset \n","train_dataset = torchvision.datasets.MNIST(root='.', \n"," train=True, \n"," transform=transforms.ToTensor(), \n"," download=True)\n","\n","test_dataset = torchvision.datasets.MNIST(root='.', \n"," train=False, \n"," transform=transforms.ToTensor())\n","\n","# Data loader\n","train_loader = torch.utils.data.DataLoader(dataset=train_dataset, \n"," batch_size=batch_size, \n"," shuffle=True)\n","\n","test_loader = torch.utils.data.DataLoader(dataset=test_dataset, \n"," batch_size=batch_size, \n"," shuffle=False)\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"7BEDqrM6VGNd","colab":{"base_uri":"https://localhost:8080/","height":299},"executionInfo":{"status":"ok","timestamp":1573205236173,"user_tz":-480,"elapsed":3175,"user":{"displayName":"Xuan Phi Nguyen","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBMe29t94knDbWFUTdAsYnOA_RwiofW5MdraN0v1TU=s64","userId":"15291677822697214066"}},"outputId":"79f68291-d9f5-4836-91c2-cd49e455e530"},"source":["image, label = test_dataset[0]\n","# reduce batch=1 to no batch\n","image = image[0]\n","print(f'{image.size()} , Label: {label}')\n","plt.imshow(image)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["torch.Size([28, 28]) , Label: 7\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":13},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANiklEQVR4nO3df4wc9XnH8c8n/kV8QGtDcF3j4ISQ\nqE4aSHWBRNDKESUFImSiJBRLtVyJ5lALElRRW0QVBalVSlEIok0aySluHESgaQBhJTSNa6W1UKlj\ng4yxgdaEmsau8QFOaxPAP/DTP24cHXD7vWNndmft5/2SVrs7z87Oo/F9PLMzO/t1RAjA8e9tbTcA\noD8IO5AEYQeSIOxAEoQdSGJ6Pxc207PiBA31c5FAKq/qZzoYBzxRrVbYbV8s6XZJ0yT9bUTcXHr9\nCRrSeb6wziIBFGyIdR1rXe/G254m6auSLpG0WNIy24u7fT8AvVXnM/u5kp6OiGci4qCkeyQtbaYt\nAE2rE/YFkn4y7vnOatrr2B6xvcn2pkM6UGNxAOro+dH4iFgZEcMRMTxDs3q9OAAd1An7LkkLxz0/\nvZoGYADVCftGSWfZfpftmZKulLSmmbYANK3rU28Rcdj2tZL+SWOn3lZFxLbGOgPQqFrn2SPiQUkP\nNtQLgB7i67JAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk\nCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiB\nJGoN2Wx7h6T9kl6TdDgihptoCkDzaoW98rGIeKGB9wHQQ+zGA0nUDXtI+oHtR2yPTPQC2yO2N9ne\ndEgHai4OQLfq7sZfEBG7bJ8maa3tpyJi/fgXRMRKSSsl6WTPjZrLA9ClWlv2iNhV3Y9Kul/SuU00\nBaB5XYfd9pDtk44+lvRxSVubagxAs+rsxs+TdL/to+/zrYj4fiNdAWhc12GPiGcknd1gLwB6iFNv\nQBKEHUiCsANJEHYgCcIOJNHEhTApvPjZj3asvXP508V5nxqdV6wfPDCjWF9wd7k+e+dLHWtHNj9R\nnBd5sGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4zz5Ff/xH3+pY+9TQT8szn1lz4UvK5R2HX+5Y\nu/35j9Vc+LHrR6NndKwN3foLxXmnr3uk6XZax5YdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRP8G\naTnZc+M8X9i35TXpZ58+r2PthQ+W/8+c82R5Hf/0V1ysz/zg/xbrt3zgvo61i97+SnHe7718YrH+\nidmdr5Wv65U4WKxvODBUrC854VDXy37P964u1t87srHr927ThlinfbF3wj8otuxAEoQdSIKwA0kQ\ndiAJwg4kQdiBJAg7kATXs0/R0Hc2FGr13vvkerPrr39pScfan5+/qLzsfy3/5v0tS97TRUdTM/2V\nI8X60Jbdxfop6+8t1n91Zuff25+9o/xb/MejSbfstlfZHrW9ddy0ubbX2t5e3c/pbZsA6prKbvw3\nJF38hmk3SFoXEWdJWlc9BzDAJg17RKyXtPcNk5dKWl09Xi3p8ob7AtCwbj+zz4uIox+onpPUcTAz\n2yOSRiTpBM3ucnEA6qp9ND7GrqTpeKVHRKyMiOGIGJ6hWXUXB6BL3YZ9j+35klTdjzbXEoBe6Dbs\nayStqB6vkPRAM+0A6JVJP7Pbvltjv1x+qu2dkr4g6WZJ37Z9laRnJV3RyyZRdvi5PR1rQ/d2rknS\na5O899B3Xuyio2bs+b2PFuvvn1n+8/3S3vd1rC36u2eK8x4uVo9Nk4Y9IpZ1KB2bv0IBJMXXZYEk\nCDuQBGEHkiDsQBKEHUiCS1zRmulnLCzWv3LjV4r1GZ5WrP/D7b/ZsXbK7oeL8x6P2LIDSRB2IAnC\nDiRB2IEkCDuQBGEHkiDsQBKcZ0drnvrDBcX6h2eVh7LedrA8HPXcJ15+yz0dz9iyA0kQdiAJwg4k\nQdiBJAg7kARhB5Ig7EASnGdHTx34xIc71h799G2TzF0eQej3r7uuWH/7v/1okvfPhS07kARhB5Ig\n7EAShB1IgrADSRB2IAnCDiTBeXb01H9f0nl7cqLL59GX/ddFxfrs7z9WrEexms+kW3bbq2yP2t46\nbtpNtnfZ3lzdLu1tmwDqmspu/DckXTzB9Nsi4pzq9mCzbQFo2qRhj4j1kvb2oRcAPVTnAN21trdU\nu/lzOr3I9ojtTbY3HdKBGosDUEe3Yf+apDMlnSNpt6RbO70wIlZGxHBEDM+Y5MIGAL3TVdgjYk9E\nvBYRRyR9XdK5zbYFoGldhd32/HFPPylpa6fXAhgMk55nt323pCWSTrW9U9IXJC2xfY7GTmXukHR1\nD3vEAHvbSScV68t//aGOtX1HXi3OO/rFdxfrsw5sLNbxepOGPSKWTTD5jh70AqCH+LoskARhB5Ig\n7EAShB1IgrADSXCJK2rZftP7i/Xvnvo3HWtLt3+qOO+sBzm11iS27EAShB1IgrADSRB2IAnCDiRB\n2IEkCDuQBOfZUfR/v/ORYn3Lb/9Vsf7jw4c61l76y9OL887S7mIdbw1bdiAJwg4kQdiBJAg7kARh\nB5Ig7EAShB1IgvPsyU1f8MvF+vWf//tifZbLf0JXPra8Y+0d/8j16v3Elh1IgrADSRB2IAnCDiRB\n2IEkCDuQBGEHkuA8+3HO08v/xGd/d2ex/pkTXyzW79p/WrE+7/OdtydHinOiaZNu2W0vtP1D20/Y\n3mb7umr6XNtrbW+v7uf0vl0A3ZrKbvxhSZ+LiMWSPiLpGtuLJd0gaV1EnCVpXfUcwICaNOwRsTsi\nHq0e75f0pKQFkpZKWl29bLWky3vVJID63tJndtuLJH1I0gZJ8yLi6I+EPSdpXod5RiSNSNIJmt1t\nnwBqmvLReNsnSrpX0vURsW98LSJCUkw0X0SsjIjhiBieoVm1mgXQvSmF3fYMjQX9roi4r5q8x/b8\nqj5f0mhvWgTQhEl3421b0h2SnoyIL48rrZG0QtLN1f0DPekQ9Zz9vmL5z067s9bbf/WLnynWf/Gx\nh2u9P5ozlc/s50taLulx25uraTdqLOTftn2VpGclXdGbFgE0YdKwR8RDktyhfGGz7QDoFb4uCyRB\n2IEkCDuQBGEHkiDsQBJc4nocmLb4vR1rI/fU+/rD4lXXFOuL7vz3Wu+P/mHLDiRB2IEkCDuQBGEH\nkiDsQBKEHUiCsANJcJ79OPDUH3T+Yd/LZu/rWJuK0//lYPkFMeEPFGEAsWUHkiDsQBKEHUiCsANJ\nEHYgCcIOJEHYgSQ4z34MePWyc4v1dZfdWqgy5BbGsGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSSm\nMj77QknflDRPUkhaGRG3275J0mclPV+99MaIeLBXjWb2P+dPK9bfOb37c+l37T+tWJ+xr3w9O1ez\nHzum8qWaw5I+FxGP2j5J0iO211a12yLiS71rD0BTpjI++25Ju6vH+20/KWlBrxsD0Ky39Jnd9iJJ\nH5K0oZp0re0ttlfZnvC3kWyP2N5ke9MhHajVLIDuTTnstk+UdK+k6yNin6SvSTpT0jka2/JP+AXt\niFgZEcMRMTxDsxpoGUA3phR22zM0FvS7IuI+SYqIPRHxWkQckfR1SeWrNQC0atKw27akOyQ9GRFf\nHjd9/riXfVLS1ubbA9CUqRyNP1/SckmP295cTbtR0jLb52js7MsOSVf3pEPU8hcvLi7WH/6tRcV6\n7H68wW7QpqkcjX9IkicocU4dOIbwDTogCcIOJEHYgSQIO5AEYQeSIOxAEo4+Drl7sufGeb6wb8sD\nstkQ67Qv9k50qpwtO5AFYQeSIOxAEoQdSIKwA0kQdiAJwg4k0dfz7Lafl/TsuEmnSnqhbw28NYPa\n26D2JdFbt5rs7YyIeMdEhb6G/U0LtzdFxHBrDRQMam+D2pdEb93qV2/sxgNJEHYgibbDvrLl5ZcM\nam+D2pdEb93qS2+tfmYH0D9tb9kB9AlhB5JoJey2L7b9H7aftn1DGz10YnuH7cdtb7a9qeVeVtke\ntb113LS5ttfa3l7dTzjGXku93WR7V7XuNtu+tKXeFtr+oe0nbG+zfV01vdV1V+irL+ut75/ZbU+T\n9J+SLpK0U9JGScsi4om+NtKB7R2ShiOi9S9g2P4NSS9J+mZEfKCadoukvRFxc/Uf5ZyI+JMB6e0m\nSS+1PYx3NVrR/PHDjEu6XNLvqsV1V+jrCvVhvbWxZT9X0tMR8UxEHJR0j6SlLfQx8CJivaS9b5i8\nVNLq6vFqjf2x9F2H3gZCROyOiEerx/slHR1mvNV1V+irL9oI+wJJPxn3fKcGa7z3kPQD24/YHmm7\nmQnMi4jd1ePnJM1rs5kJTDqMdz+9YZjxgVl33Qx/XhcH6N7sgoj4NUmXSLqm2l0dSDH2GWyQzp1O\naRjvfplgmPGfa3PddTv8eV1thH2XpIXjnp9eTRsIEbGruh+VdL8GbyjqPUdH0K3uR1vu5+cGaRjv\niYYZ1wCsuzaHP28j7BslnWX7XbZnSrpS0poW+ngT20PVgRPZHpL0cQ3eUNRrJK2oHq+Q9ECLvbzO\noAzj3WmYcbW87lof/jwi+n6TdKnGjsj/WNKfttFDh77eLemx6rat7d4k3a2x3bpDGju2cZWkUySt\nk7Rd0j9LmjtAvd0p6XFJWzQWrPkt9XaBxnbRt0jaXN0ubXvdFfrqy3rj67JAEhygA5Ig7EAShB1I\ngrADSRB2IAnCDiRB2IEk/h9BCfQTVPflJQAAAABJRU5ErkJggg==\n","text/plain":["

Google Colaboratory!