├── .gitignore ├── README.md ├── helper.py ├── linear_regression_project.ipynb ├── linear_regression_project_en.ipynb └── test.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Mac OS 2 | .DS_Store 3 | 4 | # Byte-compiled / optimized / DLL files 5 | __pycache__/ 6 | *.py[cod] 7 | *$py.class 8 | 9 | # C extensions 10 | *.so 11 | 12 | # Distribution / packaging 13 | .Python 14 | env/ 15 | build/ 16 | develop-eggs/ 17 | dist/ 18 | downloads/ 19 | eggs/ 20 | .eggs/ 21 | lib/ 22 | lib64/ 23 | parts/ 24 | sdist/ 25 | var/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *,cover 49 | .hypothesis/ 50 | 51 | # Translations 52 | *.mo 53 | *.pot 54 | 55 | # Django stuff: 56 | *.log 57 | 58 | # Sphinx documentation 59 | docs/_build/ 60 | 61 | # PyBuilder 62 | target/ 63 | 64 | #Ipython Notebook 65 | .ipynb_checkpoints 66 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 线性回归项目 2 | 3 | 欢迎来到线性回归项目，线性代数是很多机器学习算法的基础，在这个项目中，你将不借助任何库，用你之前所学来解决一个线性回归问题。 4 | 5 | # 项目内容 6 | 所有需要完成的任务都在 `linear_regression_project.ipynb` 中，其中包括编程题和证明题。 7 | 8 | **若项目中的题目有困难没完成也没关系，我们鼓励你带着问题提交项目，评审会给予你诸多帮助。** 9 | 10 | 11 | # 单元测试 12 | 项目已经部署了自动化测试，你能在每个需要你完成的函数下面看到形如以下测试代码： 13 | `%run -i -e test.py LinearRegressionTestCase.test_...` 14 | Ctrl + Enter 运行即可。 15 | 16 | 如果你的实现有问题，会有断言错误`AssertionError`被抛出。 17 | 请重新检查你的实现，并且修正bug，直到通过测试为止。 18 | 19 | 以下是一些带有特定反馈的断言错误说明： 20 | 21 | - AssertionError: Expected shape(M,N), but got shape(C,D)." 22 | + 返回的计算结果的形状不正确 23 | - AssertionError: Matrix A shouldn't be modified. 24 | + 你在实现augmentMatrix时修改了矩阵A 25 | - AssertionError: Matrix A is singular. 26 | + 你的gj_Solve实现在矩阵A是奇异矩阵时没有返回None 27 | - AssertionError: Matrix A is not singular. 28 | + 你的gj_Solve实现会在矩阵A不是奇异矩阵时返回None 29 | - AssertionError: Bad result. 30 | + 你的gj_Solve返回了不正确的计算结果 31 | 32 | # 项目提交 33 | 请在提交前确认你的项目已经满足所有[评审标准](https://review.udacity.com/#!/rubrics/871/view), 项目评审人会根据这份标准来给你相应的审阅。 34 | 35 | 你需要提交以下4个文件, 请注意每个文件的文件名和文件格式。 36 | 37 | 1. `linear_regression_project.ipynb`: 写有你代码及答案的 ipynb 文件 38 | 39 | 3. `linear_regression_project.html`: 由 Jupyter notebook 导出的 html 文件 40 | 41 | 3. `linear_regression_project.py`: 由 Jupyter notebook 导出的 python 文件 42 | 43 | 2. `proof.pdf`: 写有你的证明的 pdf 文件 （如果你在 ipython notebook中使用 LATEX 完成证明，则不需要提交此文件。） 44 | 45 | 5. 请不要提交其他任何文件。 46 | 47 | 你可以使用 Github 或上传 zip 压缩文档的方式提交项目。如果使用 Github 请将所有提交的文件放在你提交的repo中。 如果上传压缩文档，请将所有提交的文件放在压缩文档中，并命名为 `submit.zip` 后提交。 48 | 49 | --- 50 | 51 | # Linear Regression Project(in developing, not validable yet) 52 | 53 | Welcome to linear regression project. Linear regression is the basic of many machine learning algorithms. In this project, you will imply what you learn to solve a linear regression problem, without using any external libraries. 54 | 55 | # What to do 56 | All tasks are listed in `linear_regression_project.ipynb`，including coding and proving tasks. 57 | 58 | **You're encouragd to submit problem even if you haven't finished all tasks. You should submit with spefici questions and explain what you have tried and why it doesn't work. Reviewers will guide you accordingly.** 59 | 60 | 61 | # Unit test 62 | You can (and should) use unit tests to ensure all your implementations meet requirements. You can find the following code after every coding task. 63 | 64 | `%run -i -e test.py LinearRegressionTestCase.test_...` 65 | 66 | If there is an error in your implementation, the `AssertionError` will be thrown. Please modify your code accordingly, until you've passed all unit tests. 67 | 68 | The following are some examples of the assersion error. 69 | 70 | - AssertionError: Matrix A shouldn't be modified 71 | + Your augmentMatrix modifies matrix A. 72 | 73 | - AssertionError: Matrix A is singular 74 | + Your gj_Solve doesn't return None when A is singular. 75 | - AssertionError: Matrix A is not singular 76 | + Your gj_Solve returns None when A is not singular. 77 | - AssertionError: x have to be a two-dimensional Python list 78 | + Your gj_Solve returns with incorrect data structure. X should be a two a list of lists. 79 | 80 | - AssertionError: Regression result isn't good enough 81 | + Your gj_Solve has too much error. 82 | 83 | # Project submission 84 | Before submission, please ensure that your project meets all the requirements in this [rubric](https://review.udacity.com/#!/rubrics/854/view). Reviewer will give reviews according to this rubric. 85 | 86 | You should submit the follow four files. Please pay attention to the file names and file types. 87 | 88 | 1. `linear_regression_project.ipynb`: the ipynb file with your code and answers. 89 | 90 | 3. `linear_regression_project.html`: the html file exported by Jupyter notebook. 91 | 92 | 3. `linear_regression_project.py`: the python file exported by Jupyter notebook. 93 | 94 | 2. `proof.pdf`: the pdf file with your proof. （If you use LATEX in ipython notebook to write the proof, you don't need to submit this file. ） 95 | 96 | 5. Please DO NOT submit any other files. 97 | 98 | You can use Github or upload zip file to submit the project. If you use Github, please include all your files in the repo. If you submit with zip file, please compress all your files in `submit.zip` and then upload. 99 | -------------------------------------------------------------------------------- /helper.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib.pyplot as plt 3 | import matplotlib.cm as cm 4 | from mpl_toolkits.mplot3d import Axes3D 5 | 6 | def generateMatrix(rank,seed,singular=False): 7 | np.random.seed(seed) 8 | while True: 9 | matrix = np.random.randint(-10,10, size=(rank, rank)) 10 | if (np.linalg.matrix_rank(matrix) != rank) ^ (not singular): 11 | return matrix 12 | 13 | def printInMatrixFormat(Ab,padding=7,truncating=3): 14 | rank = len(Ab) 15 | rowFormat = ','.join(["{{:>{}.{}f}}".format(padding,truncating)] * rank) + " || {{:^{}.{}f}}".format(padding,truncating) 16 | matrixFormat = '\n'.join([rowFormat] * rank) 17 | 18 | flattern = [e for row in Ab for e in row] 19 | 20 | print(matrixFormat.format(*flattern)) 21 | 22 | 23 | def generatePoints2D(seed): 24 | np.random.seed(seed) 25 | 26 | num = np.random.randint(128,256) 27 | m = np.random.random() * 10 - 5 # -5 ~ 5 28 | b = np.random.random() * 10 + 5 # 5 ~ 15 29 | 30 | x = np.random.random(size=num) * 10 - 5 31 | y = x * m + b 32 | y += np.random.normal(size=num) 33 | 34 | return x.tolist(),y.tolist() 35 | 36 | def generatePoints3D(seed): 37 | np.random.seed(seed) 38 | 39 | num = np.random.randint(128,256) 40 | X = np.linspace(-5, 5, num) 41 | m = np.random.random(2) * 10 - 5 42 | b = np.random.random() * 15 + 5 43 | X = np.vstack([X]*2).T + np.random.randn(num, 2) 44 | Y = np.dot(X, m) + b + np.random.normal(size=num) 45 | return X.tolist(), Y.tolist() 46 | 47 | def vs_scatter_2d(X, Y, m=None, b=None): 48 | plt.figure() 49 | x_vals = (-5, 5) 50 | plt.xlim(x_vals) 51 | plt.xlabel('x',fontsize=18) 52 | plt.ylabel('y',fontsize=18) 53 | plt.scatter(X,Y,c='b') 54 | 55 | if m != None and b != None: 56 | y_vals = [m*x+b for x in x_vals] 57 | plt.plot(x_vals, y_vals, '-', color='r') 58 | 59 | plt.show() 60 | 61 | 62 | 63 | def vs_scatter_3d(X, Y, coeff=None): 64 | title = 'target points' 65 | cmap = cm.get_cmap('gist_rainbow') 66 | fig = plt.figure() 67 | axe3d = fig.gca(projection = '3d') 68 | axe3d.scatter(list(zip(*X))[0], list(zip(*X))[1], Y, linewidth = 0) 69 | 70 | if coeff: 71 | title = 'linear regression on target points' 72 | x = [-5 , 5] 73 | y = [-5, 5] 74 | z = np.dot(np.transpose([x, y]), coeff[:2]) + coeff[2] 75 | axe3d.plot(x, y, z, c='r') 76 | plt.show() 77 | -------------------------------------------------------------------------------- /linear_regression_project.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 欢迎来到线性回归项目\n", 8 | "\n", 9 | "若项目中的题目有困难没完成也没关系，我们鼓励你带着问题提交项目，评审人会给予你诸多帮助。\n", 10 | "\n", 11 | "所有选做题都可以不做，不影响项目通过。如果你做了，那么项目评审会帮你批改，也会因为选做部分做错而判定为不通过。\n", 12 | "\n", 13 | "其中非代码题可以提交手写后扫描的 pdf 文件，或使用 Latex 在文档中直接回答。" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "### 目录:\n", 21 | "[1 矩阵运算](#1-矩阵运算) \n", 22 | "[2 Gaussian Jordan 消元法](#2-Gaussian-Jordan-消元法) \n", 23 | "[3 线性回归](#3-线性回归) " 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": { 30 | "collapsed": true 31 | }, 32 | "outputs": [], 33 | "source": [ 34 | "# 任意选一个你喜欢的整数，这能帮你得到稳定的结果\n", 35 | "seed = # TODO" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "# 1 矩阵运算\n", 43 | "\n", 44 | "## 1.1 创建一个 4*4 的单位矩阵" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [ 55 | "# 这个项目设计来帮你熟悉 python list 和线性代数\n", 56 | "# 你不能调用任何NumPy以及相关的科学计算库来完成作业\n", 57 | "\n", 58 | "\n", 59 | "# 本项目要求矩阵统一使用二维列表表示，如下：\n", 60 | "A = [[1,2,3], \n", 61 | " [2,3,3], \n", 62 | " [1,2,5]]\n", 63 | "\n", 64 | "B = [[1,2,3,5], \n", 65 | " [2,3,3,5], \n", 66 | " [1,2,5,1]]\n", 67 | "\n", 68 | "# 向量也用二维列表表示\n", 69 | "C = [[1],\n", 70 | " [2],\n", 71 | " [3]]\n", 72 | "\n", 73 | "#TODO 创建一个 4*4 单位矩阵\n", 74 | "I = None" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "## 1.2 返回矩阵的行数和列数" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "collapsed": true 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "# TODO 返回矩阵的行数和列数\n", 93 | "def shape(M):\n", 94 | " return None" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": { 101 | "collapsed": true 102 | }, 103 | "outputs": [], 104 | "source": [ 105 | "# 运行以下代码测试你的 shape 函数\n", 106 | "%run -i -e test.py LinearRegressionTestCase.test_shape" 107 | ] 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": {}, 112 | "source": [ 113 | "## 1.3 每个元素四舍五入到特定小数数位" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": { 120 | "collapsed": true 121 | }, 122 | "outputs": [], 123 | "source": [ 124 | "# TODO 每个元素四舍五入到特定小数数位\n", 125 | "# 直接修改参数矩阵，无返回值\n", 126 | "def matxRound(M, decPts=4):\n", 127 | " pass" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": { 134 | "collapsed": true 135 | }, 136 | "outputs": [], 137 | "source": [ 138 | "# 运行以下代码测试你的 matxRound 函数\n", 139 | "%run -i -e test.py LinearRegressionTestCase.test_matxRound" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "## 1.4 计算矩阵的转置" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": { 153 | "collapsed": true 154 | }, 155 | "outputs": [], 156 | "source": [ 157 | "# TODO 计算矩阵的转置\n", 158 | "def transpose(M):\n", 159 | " return None" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": null, 165 | "metadata": { 166 | "collapsed": true 167 | }, 168 | "outputs": [], 169 | "source": [ 170 | "# 运行以下代码测试你的 transpose 函数\n", 171 | "%run -i -e test.py LinearRegressionTestCase.test_transpose" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "## 1.5 计算矩阵乘法 AB" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": { 185 | "collapsed": true 186 | }, 187 | "outputs": [], 188 | "source": [ 189 | "# TODO 计算矩阵乘法 AB，如果无法相乘则raise ValueError\n", 190 | "def matxMultiply(A, B):\n", 191 | " return None" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "metadata": { 198 | "collapsed": true 199 | }, 200 | "outputs": [], 201 | "source": [ 202 | "# 运行以下代码测试你的 matxMultiply 函数\n", 203 | "%run -i -e test.py LinearRegressionTestCase.test_matxMultiply" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "---\n", 211 | "\n", 212 | "# 2 Gaussian Jordan 消元法\n", 213 | "\n", 214 | "## 2.1 构造增广矩阵\n", 215 | "\n", 216 | "$ A = \\begin{bmatrix}\n", 217 | " a_{11} & a_{12} & ... & a_{1n}\\\\\n", 218 | " a_{21} & a_{22} & ... & a_{2n}\\\\\n", 219 | " a_{31} & a_{22} & ... & a_{3n}\\\\\n", 220 | " ... & ... & ... & ...\\\\\n", 221 | " a_{n1} & a_{n2} & ... & a_{nn}\\\\\n", 222 | "\\end{bmatrix} , b = \\begin{bmatrix}\n", 223 | " b_{1} \\\\\n", 224 | " b_{2} \\\\\n", 225 | " b_{3} \\\\\n", 226 | " ... \\\\\n", 227 | " b_{n} \\\\\n", 228 | "\\end{bmatrix}$\n", 229 | "\n", 230 | "返回 $ Ab = \\begin{bmatrix}\n", 231 | " a_{11} & a_{12} & ... & a_{1n} & b_{1}\\\\\n", 232 | " a_{21} & a_{22} & ... & a_{2n} & b_{2}\\\\\n", 233 | " a_{31} & a_{22} & ... & a_{3n} & b_{3}\\\\\n", 234 | " ... & ... & ... & ...& ...\\\\\n", 235 | " a_{n1} & a_{n2} & ... & a_{nn} & b_{n} \\end{bmatrix}$" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": { 242 | "collapsed": true 243 | }, 244 | "outputs": [], 245 | "source": [ 246 | "# TODO 构造增广矩阵，假设A，b行数相同\n", 247 | "def augmentMatrix(A, b):\n", 248 | " return None" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "metadata": { 255 | "collapsed": true 256 | }, 257 | "outputs": [], 258 | "source": [ 259 | "# 运行以下代码测试你的 augmentMatrix 函数\n", 260 | "%run -i -e test.py LinearRegressionTestCase.test_augmentMatrix" 261 | ] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": {}, 266 | "source": [ 267 | "## 2.2 初等行变换\n", 268 | "- 交换两行\n", 269 | "- 把某行乘以一个非零常数\n", 270 | "- 把某行加上另一行的若干倍：" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": null, 276 | "metadata": { 277 | "collapsed": true 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "# TODO r1 <---> r2\n", 282 | "# 直接修改参数矩阵，无返回值\n", 283 | "def swapRows(M, r1, r2):\n", 284 | " pass" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": null, 290 | "metadata": { 291 | "collapsed": true 292 | }, 293 | "outputs": [], 294 | "source": [ 295 | "# 运行以下代码测试你的 swapRows 函数\n", 296 | "%run -i -e test.py LinearRegressionTestCase.test_swapRows" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": null, 302 | "metadata": { 303 | "collapsed": true 304 | }, 305 | "outputs": [], 306 | "source": [ 307 | "# TODO r1 <--- r1 * scale\n", 308 | "# scale为0是非法输入，要求 raise ValueError\n", 309 | "# 直接修改参数矩阵，无返回值\n", 310 | "def scaleRow(M, r, scale):\n", 311 | " pass" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "metadata": { 318 | "collapsed": true 319 | }, 320 | "outputs": [], 321 | "source": [ 322 | "# 运行以下代码测试你的 scaleRow 函数\n", 323 | "%run -i -e test.py LinearRegressionTestCase.test_scaleRow" 324 | ] 325 | }, 326 | { 327 | "cell_type": "code", 328 | "execution_count": null, 329 | "metadata": { 330 | "collapsed": true 331 | }, 332 | "outputs": [], 333 | "source": [ 334 | "# TODO r1 <--- r1 + r2*scale\n", 335 | "# 直接修改参数矩阵，无返回值\n", 336 | "def addScaledRow(M, r1, r2, scale):\n", 337 | " pass" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": { 344 | "collapsed": true 345 | }, 346 | "outputs": [], 347 | "source": [ 348 | "# 运行以下代码测试你的 addScaledRow 函数\n", 349 | "%run -i -e test.py LinearRegressionTestCase.test_addScaledRow" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "## 2.3 Gaussian Jordan 消元法求解 Ax = b" 357 | ] 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "metadata": {}, 362 | "source": [ 363 | "### 2.3.1 算法\n", 364 | "\n", 365 | "步骤1 检查A，b是否行数相同\n", 366 | "\n", 367 | "步骤2 构造增广矩阵Ab\n", 368 | "\n", 369 | "步骤3 逐列转换Ab为化简行阶梯形矩阵 [中文维基链接](https://zh.wikipedia.org/wiki/%E9%98%B6%E6%A2%AF%E5%BD%A2%E7%9F%A9%E9%98%B5#.E5.8C.96.E7.AE.80.E5.90.8E.E7.9A.84-.7Bzh-hans:.E8.A1.8C.3B_zh-hant:.E5.88.97.3B.7D-.E9.98.B6.E6.A2.AF.E5.BD.A2.E7.9F.A9.E9.98.B5)\n", 370 | " \n", 371 | " 对于Ab的每一列（最后一列除外）\n", 372 | " 当前列为列c\n", 373 | " 寻找列c中 对角线以及对角线以下所有元素（行 c~N）的绝对值的最大值\n", 374 | " 如果绝对值最大值为0\n", 375 | " 那么A为奇异矩阵，返回None (你可以在选做问题2.4中证明为什么这里A一定是奇异矩阵)\n", 376 | " 否则\n", 377 | " 使用第一个行变换，将绝对值最大值所在行交换到对角线元素所在行（行c） \n", 378 | " 使用第二个行变换，将列c的对角线元素缩放为1\n", 379 | " 多次使用第三个行变换，将列c的其他元素消为0\n", 380 | " \n", 381 | "步骤4 返回Ab的最后一列\n", 382 | "\n", 383 | "**注：** 我们并没有按照常规方法先把矩阵转化为行阶梯形矩阵，再转换为化简行阶梯形矩阵，而是一步到位。如果你熟悉常规方法的话，可以思考一下两者的等价性。" 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": {}, 389 | "source": [ 390 | "### 2.3.2 算法推演\n", 391 | "\n", 392 | "为了充分了解Gaussian Jordan消元法的计算流程，请根据Gaussian Jordan消元法，分别手动推演矩阵A为***可逆矩阵***，矩阵A为***奇异矩阵***两种情况。" 393 | ] 394 | }, 395 | { 396 | "cell_type": "markdown", 397 | "metadata": {}, 398 | "source": [ 399 | "#### 推演示例 \n", 400 | "\n", 401 | "\n", 402 | "$Ab = \\begin{bmatrix}\n", 403 | " -7 & 5 & -1 & 1\\\\\n", 404 | " 1 & -3 & -8 & 1\\\\\n", 405 | " -10 & -2 & 9 & 1\\end{bmatrix}$\n", 406 | "\n", 407 | "$ --> $\n", 408 | "$\\begin{bmatrix}\n", 409 | " 1 & \\frac{1}{5} & -\\frac{9}{10} & -\\frac{1}{10}\\\\\n", 410 | " 0 & -\\frac{16}{5} & -\\frac{71}{10} & \\frac{11}{10}\\\\\n", 411 | " 0 & \\frac{32}{5} & -\\frac{73}{10} & \\frac{3}{10}\\end{bmatrix}$\n", 412 | "\n", 413 | "$ --> $\n", 414 | "$\\begin{bmatrix}\n", 415 | " 1 & 0 & -\\frac{43}{64} & -\\frac{7}{64}\\\\\n", 416 | " 0 & 1 & -\\frac{73}{64} & \\frac{3}{64}\\\\\n", 417 | " 0 & 0 & -\\frac{43}{4} & \\frac{5}{4}\\end{bmatrix}$\n", 418 | "\n", 419 | "$ --> $\n", 420 | "$\\begin{bmatrix}\n", 421 | " 1 & 0 & 0 & -\\frac{3}{16}\\\\\n", 422 | " 0 & 1 & 0 & -\\frac{59}{688}\\\\\n", 423 | " 0 & 0 & 1 & -\\frac{5}{43}\\end{bmatrix}$\n", 424 | " \n", 425 | "\n", 426 | "#### 推演有以下要求:\n", 427 | "1. 展示每一列的消元结果, 比如3*3的矩阵, 需要写三步\n", 428 | "2. 用分数来表示\n", 429 | "3. 分数不能再约分\n", 430 | "4. 我们已经给出了latex的语法,你只要把零改成你要的数字(或分数)即可\n", 431 | "5. 可以用[这个页面](http://www.math.odu.edu/~bogacki/cgi-bin/lat.cgi?c=sys)检查你的答案(注意只是答案, 推演步骤两者算法不一致)\n", 432 | "\n", 433 | "_你可以用python的 [fractions](https://docs.python.org/2/library/fractions.html) 模块辅助你的约分_" 434 | ] 435 | }, 436 | { 437 | "cell_type": "markdown", 438 | "metadata": {}, 439 | "source": [ 440 | "#### 分数的输入方法\n", 441 | "(双击这个区域就能看到语法啦) \n", 442 | " \n", 443 | "示例一: $\\frac{n}{m}$ \n", 444 | "\n", 445 | "示例二: $-\\frac{a}{b}$ " 446 | ] 447 | }, 448 | { 449 | "cell_type": "markdown", 450 | "metadata": {}, 451 | "source": [ 452 | "#### 以下开始你的尝试吧!" 453 | ] 454 | }, 455 | { 456 | "cell_type": "code", 457 | "execution_count": null, 458 | "metadata": { 459 | "collapsed": true 460 | }, 461 | "outputs": [], 462 | "source": [ 463 | "# 不要修改这里！\n", 464 | "from helper import *\n", 465 | "A = generateMatrix(3,seed,singular=False)\n", 466 | "b = np.ones(shape=(3,1),dtype=int) # it doesn't matter\n", 467 | "Ab = augmentMatrix(A.tolist(),b.tolist()) # 请确保你的增广矩阵已经写好了\n", 468 | "printInMatrixFormat(Ab,padding=3,truncating=0)" 469 | ] 470 | }, 471 | { 472 | "cell_type": "markdown", 473 | "metadata": {}, 474 | "source": [ 475 | "请按照算法的步骤3，逐步推演***可逆矩阵***的变换。\n", 476 | "\n", 477 | "在下面列出每一次循环体执行之后的增广矩阵(注意使用[分数语法](#分数的输入方法))\n", 478 | "\n", 479 | "$ Ab = \\begin{bmatrix}\n", 480 | " 0 & 0 & 0 & 0 \\\\\n", 481 | " 0 & 0 & 0 & 0 \\\\\n", 482 | " 0 & 0 & 0 & 0 \\end{bmatrix}$\n", 483 | "\n", 484 | "$ --> \\begin{bmatrix}\n", 485 | " 0 & 0 & 0 & 0 \\\\\n", 486 | " 0 & 0 & 0 & 0 \\\\\n", 487 | " 0 & 0 & 0 & 0 \\end{bmatrix}$\n", 488 | " \n", 489 | "$ --> \\begin{bmatrix}\n", 490 | " 0 & 0 & 0 & 0 \\\\\n", 491 | " 0 & 0 & 0 & 0 \\\\\n", 492 | " 0 & 0 & 0 & 0 \\end{bmatrix}$\n", 493 | " \n", 494 | "$...$" 495 | ] 496 | }, 497 | { 498 | "cell_type": "code", 499 | "execution_count": null, 500 | "metadata": { 501 | "collapsed": true 502 | }, 503 | "outputs": [], 504 | "source": [ 505 | "# 不要修改这里！\n", 506 | "A = generateMatrix(3,seed,singular=True)\n", 507 | "b = np.ones(shape=(3,1),dtype=int)\n", 508 | "Ab = augmentMatrix(A.tolist(),b.tolist()) # 请确保你的增广矩阵已经写好了\n", 509 | "printInMatrixFormat(Ab,padding=3,truncating=0)" 510 | ] 511 | }, 512 | { 513 | "cell_type": "markdown", 514 | "metadata": {}, 515 | "source": [ 516 | "请按照算法的步骤3，逐步推演***奇异矩阵***的变换。\n", 517 | "\n", 518 | "在下面列出每一次循环体执行之后的增广矩阵(注意使用[分数语法](#分数的输入方法))\n", 519 | "\n", 520 | "$ Ab = \\begin{bmatrix}\n", 521 | " 0 & 0 & 0 & 0 \\\\\n", 522 | " 0 & 0 & 0 & 0 \\\\\n", 523 | " 0 & 0 & 0 & 0 \\end{bmatrix}$\n", 524 | "\n", 525 | "$ --> \\begin{bmatrix}\n", 526 | " 0 & 0 & 0 & 0 \\\\\n", 527 | " 0 & 0 & 0 & 0 \\\\\n", 528 | " 0 & 0 & 0 & 0 \\end{bmatrix}$\n", 529 | " \n", 530 | "$ --> \\begin{bmatrix}\n", 531 | " 0 & 0 & 0 & 0 \\\\\n", 532 | " 0 & 0 & 0 & 0 \\\\\n", 533 | " 0 & 0 & 0 & 0 \\end{bmatrix}$\n", 534 | " \n", 535 | "$...$" 536 | ] 537 | }, 538 | { 539 | "cell_type": "markdown", 540 | "metadata": {}, 541 | "source": [ 542 | "### 2.3.3 实现 Gaussian Jordan 消元法" 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "execution_count": null, 548 | "metadata": { 549 | "collapsed": true 550 | }, 551 | "outputs": [], 552 | "source": [ 553 | "# TODO 实现 Gaussain Jordan 方法求解 Ax = b\n", 554 | "\n", 555 | "\"\"\" Gaussian Jordan 方法求解 Ax = b.\n", 556 | " 参数\n", 557 | " A: 方阵 \n", 558 | " b: 列向量\n", 559 | " decPts: 四舍五入位数，默认为4\n", 560 | " epsilon: 判读是否为0的阈值，默认 1.0e-16\n", 561 | " \n", 562 | " 返回列向量 x 使得 Ax = b \n", 563 | " 返回None，如果 A，b 高度不同\n", 564 | " 返回None，如果 A 为奇异矩阵\n", 565 | "\"\"\"\n", 566 | "def gj_Solve(A, b, decPts=4, epsilon=1.0e-16):\n", 567 | " return None" 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": null, 573 | "metadata": { 574 | "collapsed": true 575 | }, 576 | "outputs": [], 577 | "source": [ 578 | "# 运行以下代码测试你的 gj_Solve 函数\n", 579 | "%run -i -e test.py LinearRegressionTestCase.test_gj_Solve" 580 | ] 581 | }, 582 | { 583 | "cell_type": "markdown", 584 | "metadata": {}, 585 | "source": [ 586 | "## (选做) 2.4 算法正确判断了奇异矩阵：\n", 587 | "\n", 588 | "在算法的步骤3 中，如果发现某一列对角线和对角线以下所有元素都为0，那么则断定这个矩阵为奇异矩阵。\n", 589 | "\n", 590 | "我们用正式的语言描述这个命题，并证明为真。\n", 591 | "\n", 592 | "证明下面的命题：\n", 593 | "\n", 594 | "**如果方阵 A 可以被分为4个部分: ** \n", 595 | "\n", 596 | "$ A = \\begin{bmatrix}\n", 597 | " I & X \\\\\n", 598 | " Z & Y \\\\\n", 599 | "\\end{bmatrix} , \\text{其中 I 为单位矩阵，Z 为全0矩阵，Y 的第一列全0}$，\n", 600 | "\n", 601 | "**那么A为奇异矩阵。**\n", 602 | "\n", 603 | "提示：从多种角度都可以完成证明\n", 604 | "- 考虑矩阵 Y 和 矩阵 A 的秩\n", 605 | "- 考虑矩阵 Y 和 矩阵 A 的行列式\n", 606 | "- 考虑矩阵 A 的某一列是其他列的线性组合" 607 | ] 608 | }, 609 | { 610 | "cell_type": "markdown", 611 | "metadata": {}, 612 | "source": [ 613 | "TODO 证明：" 614 | ] 615 | }, 616 | { 617 | "cell_type": "markdown", 618 | "metadata": {}, 619 | "source": [ 620 | "# 3 线性回归" 621 | ] 622 | }, 623 | { 624 | "cell_type": "markdown", 625 | "metadata": {}, 626 | "source": [ 627 | "## 3.1 随机生成样本点" 628 | ] 629 | }, 630 | { 631 | "cell_type": "code", 632 | "execution_count": null, 633 | "metadata": {}, 634 | "outputs": [], 635 | "source": [ 636 | "# 不要修改这里！\n", 637 | "%matplotlib notebook\n", 638 | "from helper import *\n", 639 | "\n", 640 | "X,Y = generatePoints2D(seed)\n", 641 | "vs_scatter_2d(X, Y)" 642 | ] 643 | }, 644 | { 645 | "cell_type": "markdown", 646 | "metadata": {}, 647 | "source": [ 648 | "## 3.2 拟合一条直线\n", 649 | "\n", 650 | "### 3.2.1 猜测一条直线" 651 | ] 652 | }, 653 | { 654 | "cell_type": "code", 655 | "execution_count": null, 656 | "metadata": {}, 657 | "outputs": [], 658 | "source": [ 659 | "#TODO 请选择最适合的直线 y = mx + b\n", 660 | "m1 = 0.\n", 661 | "b1 = 0.\n", 662 | "\n", 663 | "# 不要修改这里！\n", 664 | "vs_scatter_2d(X, Y, m1, b1)" 665 | ] 666 | }, 667 | { 668 | "cell_type": "markdown", 669 | "metadata": {}, 670 | "source": [ 671 | "### 3.2.2 计算平均平方误差 (MSE)" 672 | ] 673 | }, 674 | { 675 | "cell_type": "markdown", 676 | "metadata": { 677 | "collapsed": true 678 | }, 679 | "source": [ 680 | "我们要编程计算所选直线的平均平方误差(MSE), 即数据集中每个点到直线的Y方向距离的平方的平均数，表达式如下：\n", 681 | "$$\n", 682 | "MSE = \\frac{1}{n}\\sum_{i=1}^{n}{(y_i - mx_i - b)^2}\n", 683 | "$$" 684 | ] 685 | }, 686 | { 687 | "cell_type": "code", 688 | "execution_count": null, 689 | "metadata": {}, 690 | "outputs": [], 691 | "source": [ 692 | "# TODO 实现以下函数并输出所选直线的MSE\n", 693 | "def calculateMSE2D(X,Y,m,b):\n", 694 | " return 0.\n", 695 | "\n", 696 | "# TODO 检查这里的结果, 如果你上面猜测的直线准确, 这里的输出会在1.5以内\n", 697 | "print(calculateMSE2D(X,Y,m1,b1))" 698 | ] 699 | }, 700 | { 701 | "cell_type": "markdown", 702 | "metadata": {}, 703 | "source": [ 704 | "### 3.2.3 调整参数 $m, b$ 来获得最小的平方平均误差\n", 705 | "\n", 706 | "你可以调整3.2.1中的参数 $m1,b1$ 让蓝点均匀覆盖在红线周围，然后微调 $m1, b1$ 让MSE最小。" 707 | ] 708 | }, 709 | { 710 | "cell_type": "markdown", 711 | "metadata": {}, 712 | "source": [ 713 | "## 3.3 (选做) 找到参数 $m, b$ 使得平方平均误差最小\n", 714 | "\n", 715 | "**这一部分需要简单的微积分知识( $ (x^2)' = 2x $ )。因为这是一个线性代数项目，所以设为选做。**\n", 716 | "\n", 717 | "刚刚我们手动调节参数，尝试找到最小的平方平均误差。下面我们要精确得求解 $m, b$ 使得平方平均误差最小。\n", 718 | "\n", 719 | "定义目标函数 $E$ 为\n", 720 | "$$\n", 721 | "E = \\frac{1}{2}\\sum_{i=1}^{n}{(y_i - mx_i - b)^2}\n", 722 | "$$\n", 723 | "\n", 724 | "因为 $E = \\frac{n}{2}MSE$, 所以 $E$ 取到最小值时，$MSE$ 也取到最小值。要找到 $E$ 的最小值，即要找到 $m, b$ 使得 $E$ 相对于 $m$, $E$ 相对于 $b$ 的偏导数等于0. \n", 725 | "\n", 726 | "因此我们要解下面的方程组。\n", 727 | "\n", 728 | "$$\n", 729 | "\\begin{cases}\n", 730 | "\\displaystyle\n", 731 | "\\frac{\\partial E}{\\partial m} =0 \\\\\n", 732 | "\\\\\n", 733 | "\\displaystyle\n", 734 | "\\frac{\\partial E}{\\partial b} =0 \\\\\n", 735 | "\\end{cases}\n", 736 | "$$\n", 737 | "\n", 738 | "### 3.3.1 计算目标函数相对于参数的导数\n", 739 | "首先我们计算两个式子左边的值\n", 740 | "\n", 741 | "证明/计算：\n", 742 | "$$\n", 743 | "\\frac{\\partial E}{\\partial m} = \\sum_{i=1}^{n}{-x_i(y_i - mx_i - b)}\n", 744 | "$$\n", 745 | "\n", 746 | "$$\n", 747 | "\\frac{\\partial E}{\\partial b} = \\sum_{i=1}^{n}{-(y_i - mx_i - b)}\n", 748 | "$$" 749 | ] 750 | }, 751 | { 752 | "cell_type": "markdown", 753 | "metadata": {}, 754 | "source": [ 755 | "TODO 证明:" 756 | ] 757 | }, 758 | { 759 | "cell_type": "markdown", 760 | "metadata": {}, 761 | "source": [ 762 | "### 3.3.2 实例推演\n", 763 | "\n", 764 | "现在我们有了一个二元二次方程组\n", 765 | "\n", 766 | "$$\n", 767 | "\\begin{cases}\n", 768 | "\\displaystyle\n", 769 | "\\sum_{i=1}^{n}{-x_i(y_i - mx_i - b)} =0 \\\\\n", 770 | "\\\\\n", 771 | "\\displaystyle\n", 772 | "\\sum_{i=1}^{n}{-(y_i - mx_i - b)} =0 \\\\\n", 773 | "\\end{cases}\n", 774 | "$$\n", 775 | "\n", 776 | "为了加强理解，我们用一个实际例子演练。\n", 777 | "\n", 778 | "我们要用三个点 $(1,1), (2,2), (3,2)$ 来拟合一条直线 y = m*x + b, 请写出\n", 779 | "\n", 780 | "- 目标函数 $E$, \n", 781 | "- 二元二次方程组，\n", 782 | "- 并求解最优参数 $m, b$" 783 | ] 784 | }, 785 | { 786 | "cell_type": "markdown", 787 | "metadata": {}, 788 | "source": [ 789 | "TODO 写出目标函数，方程组和最优参数" 790 | ] 791 | }, 792 | { 793 | "cell_type": "markdown", 794 | "metadata": {}, 795 | "source": [ 796 | "### 3.3.3 将方程组写成矩阵形式\n", 797 | "\n", 798 | "我们的二元二次方程组可以用更简洁的矩阵形式表达，将方程组写成矩阵形式更有利于我们使用 Gaussian Jordan 消元法求解。\n", 799 | "\n", 800 | "请证明 \n", 801 | "$$\n", 802 | "\\begin{bmatrix}\n", 803 | " \\frac{\\partial E}{\\partial m} \\\\\n", 804 | " \\frac{\\partial E}{\\partial b} \n", 805 | "\\end{bmatrix} = X^TXh - X^TY\n", 806 | "$$\n", 807 | "\n", 808 | "其中向量 $Y$, 矩阵 $X$ 和 向量 $h$ 分别为 :\n", 809 | "$$\n", 810 | "Y = \\begin{bmatrix}\n", 811 | " y_1 \\\\\n", 812 | " y_2 \\\\\n", 813 | " ... \\\\\n", 814 | " y_n\n", 815 | "\\end{bmatrix}\n", 816 | ",\n", 817 | "X = \\begin{bmatrix}\n", 818 | " x_1 & 1 \\\\\n", 819 | " x_2 & 1\\\\\n", 820 | " ... & ...\\\\\n", 821 | " x_n & 1 \\\\\n", 822 | "\\end{bmatrix},\n", 823 | "h = \\begin{bmatrix}\n", 824 | " m \\\\\n", 825 | " b \\\\\n", 826 | "\\end{bmatrix}\n", 827 | "$$" 828 | ] 829 | }, 830 | { 831 | "cell_type": "markdown", 832 | "metadata": {}, 833 | "source": [ 834 | "TODO 证明:" 835 | ] 836 | }, 837 | { 838 | "cell_type": "markdown", 839 | "metadata": {}, 840 | "source": [ 841 | "至此我们知道，通过求解方程 $X^TXh = X^TY$ 来找到最优参数。这个方程十分重要，他有一个名字叫做 **Normal Equation**，也有直观的几何意义。你可以在 [子空间投影](http://open.163.com/movie/2010/11/J/U/M6V0BQC4M_M6V2AJLJU.html) 和 [投影矩阵与最小二乘](http://open.163.com/movie/2010/11/P/U/M6V0BQC4M_M6V2AOJPU.html) 看到更多关于这个方程的内容。" 842 | ] 843 | }, 844 | { 845 | "cell_type": "markdown", 846 | "metadata": {}, 847 | "source": [ 848 | "### 3.4 求解 $X^TXh = X^TY$ \n", 849 | "\n", 850 | "在3.3 中，我们知道线性回归问题等价于求解 $X^TXh = X^TY$ (如果你选择不做3.3，就勇敢的相信吧，哈哈)" 851 | ] 852 | }, 853 | { 854 | "cell_type": "code", 855 | "execution_count": null, 856 | "metadata": { 857 | "collapsed": true 858 | }, 859 | "outputs": [], 860 | "source": [ 861 | "# TODO 实现线性回归\n", 862 | "'''\n", 863 | "参数：X, Y 存储着一一对应的横坐标与纵坐标的两个一维数组\n", 864 | "返回：线性回归的系数(如上面所说的 m, b)\n", 865 | "'''\n", 866 | "def linearRegression2D(X,Y):\n", 867 | " return 0.,0." 868 | ] 869 | }, 870 | { 871 | "cell_type": "code", 872 | "execution_count": null, 873 | "metadata": {}, 874 | "outputs": [], 875 | "source": [ 876 | "# 请不要修改下面的代码\n", 877 | "m2,b2 = linearRegression2D(X,Y)\n", 878 | "assert isinstance(m2,float),\"m is not a float\"\n", 879 | "assert isinstance(b2,float),\"b is not a float\"\n", 880 | "print(m2,b2)" 881 | ] 882 | }, 883 | { 884 | "cell_type": "markdown", 885 | "metadata": {}, 886 | "source": [ 887 | "你求得的回归结果是什么？\n", 888 | "请使用运行以下代码将它画出来。" 889 | ] 890 | }, 891 | { 892 | "cell_type": "code", 893 | "execution_count": null, 894 | "metadata": { 895 | "scrolled": false 896 | }, 897 | "outputs": [], 898 | "source": [ 899 | "## 请不要修改下面的代码\n", 900 | "vs_scatter_2d(X, Y, m2, b2)\n", 901 | "print(calculateMSE2D(X,Y,m2,b2))" 902 | ] 903 | }, 904 | { 905 | "cell_type": "markdown", 906 | "metadata": {}, 907 | "source": [ 908 | "## Bonus !!!\n", 909 | "如果你的高斯约当消元法通过了单元测试, 那么它将能够解决多维的回归问题 \n", 910 | "你将会在更高维度考验你的线性回归实现" 911 | ] 912 | }, 913 | { 914 | "cell_type": "code", 915 | "execution_count": null, 916 | "metadata": {}, 917 | "outputs": [], 918 | "source": [ 919 | "# 生成三维的数据点\n", 920 | "X_3d, Y_3d = generatePoints3D(seed)\n", 921 | "vs_scatter_3d(X_3d, Y_3d)" 922 | ] 923 | }, 924 | { 925 | "cell_type": "markdown", 926 | "metadata": {}, 927 | "source": [ 928 | "你的线性回归是否能够对付三维的情况?" 929 | ] 930 | }, 931 | { 932 | "cell_type": "code", 933 | "execution_count": null, 934 | "metadata": { 935 | "collapsed": true 936 | }, 937 | "outputs": [], 938 | "source": [ 939 | "def linearRegression(X,Y):\n", 940 | " return None" 941 | ] 942 | }, 943 | { 944 | "cell_type": "code", 945 | "execution_count": null, 946 | "metadata": {}, 947 | "outputs": [], 948 | "source": [ 949 | "coeff = linearRegression(X_3d, Y_3d)\n", 950 | "vs_scatter_3d(X_3d, Y_3d, coeff)" 951 | ] 952 | } 953 | ], 954 | "metadata": { 955 | "anaconda-cloud": {}, 956 | "celltoolbar": "Raw Cell Format", 957 | "kernelspec": { 958 | "display_name": "Python 3", 959 | "language": "python", 960 | "name": "python3" 961 | }, 962 | "language_info": { 963 | "codemirror_mode": { 964 | "name": "ipython", 965 | "version": 3 966 | }, 967 | "file_extension": ".py", 968 | "mimetype": "text/x-python", 969 | "name": "python", 970 | "nbconvert_exporter": "python", 971 | "pygments_lexer": "ipython3", 972 | "version": "3.6.2" 973 | } 974 | }, 975 | "nbformat": 4, 976 | "nbformat_minor": 2 977 | } 978 | -------------------------------------------------------------------------------- /linear_regression_project_en.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 1 Matrix operations\n", 8 | "\n", 9 | "## 1.1 Create a 4*4 identity matrix" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": { 16 | "collapsed": true 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "#This project is designed to get familiar with python list and linear algebra\n", 21 | "#You cannot use import any library yourself, especially numpy\n", 22 | "\n", 23 | "A = [[1,2,3], \n", 24 | " [2,3,3], \n", 25 | " [1,2,5]]\n", 26 | "\n", 27 | "B = [[1,2,3,5], \n", 28 | " [2,3,3,5], \n", 29 | " [1,2,5,1]]\n", 30 | "\n", 31 | "#TODO create a 4*4 identity matrix \n", 32 | "I = None" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "## 1.2 get the width and height of a matrix. " 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": { 46 | "collapsed": true 47 | }, 48 | "outputs": [], 49 | "source": [ 50 | "#TODO Get the height and weight of a matrix.\n", 51 | "def shape(M):\n", 52 | " return 0,0" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": { 59 | "collapsed": true 60 | }, 61 | "outputs": [], 62 | "source": [ 63 | "# run following code to test your shape function\n", 64 | "%run -i -e test.py LinearRegressionTestCase.test_shape" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "## 1.3 round all elements in M to certain decimal points" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": { 78 | "collapsed": true 79 | }, 80 | "outputs": [], 81 | "source": [ 82 | "# TODO in-place operation, no return value\n", 83 | "# TODO round all elements in M to decPts\n", 84 | "def matxRound(M, decPts=4):\n", 85 | " pass" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": { 92 | "collapsed": true 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "# run following code to test your matxRound function\n", 97 | "%run -i -e test.py LinearRegressionTestCase.test_matxRound" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "## 1.4 compute transpose of M" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": { 111 | "collapsed": true 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "#TODO compute transpose of M\n", 116 | "def transpose(M):\n", 117 | " return None" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": { 124 | "collapsed": true 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "# run following code to test your transpose function\n", 129 | "%run -i -e test.py LinearRegressionTestCase.test_transpose" 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "metadata": {}, 135 | "source": [ 136 | "## 1.5 compute AB. return None if the dimensions don't match" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": { 143 | "collapsed": true 144 | }, 145 | "outputs": [], 146 | "source": [ 147 | "#TODO compute matrix multiplication AB, return None if the dimensions don't match\n", 148 | "def matxMultiply(A, B):\n", 149 | " return None" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [ 160 | "# run following code to test your matxMultiply function\n", 161 | "%run -i -e test.py LinearRegressionTestCase.test_matxMultiply" 162 | ] 163 | }, 164 | { 165 | "cell_type": "markdown", 166 | "metadata": {}, 167 | "source": [ 168 | "---\n", 169 | "\n", 170 | "# 2 Gaussian Jordan Elimination\n", 171 | "\n", 172 | "## 2.1 Compute augmented Matrix \n", 173 | "\n", 174 | "$ A = \\begin{bmatrix}\n", 175 | " a_{11} & a_{12} & ... & a_{1n}\\\\\n", 176 | " a_{21} & a_{22} & ... & a_{2n}\\\\\n", 177 | " a_{31} & a_{22} & ... & a_{3n}\\\\\n", 178 | " ... & ... & ... & ...\\\\\n", 179 | " a_{n1} & a_{n2} & ... & a_{nn}\\\\\n", 180 | "\\end{bmatrix} , b = \\begin{bmatrix}\n", 181 | " b_{1} \\\\\n", 182 | " b_{2} \\\\\n", 183 | " b_{3} \\\\\n", 184 | " ... \\\\\n", 185 | " b_{n} \\\\\n", 186 | "\\end{bmatrix}$\n", 187 | "\n", 188 | "Return $ Ab = \\begin{bmatrix}\n", 189 | " a_{11} & a_{12} & ... & a_{1n} & b_{1}\\\\\n", 190 | " a_{21} & a_{22} & ... & a_{2n} & b_{2}\\\\\n", 191 | " a_{31} & a_{22} & ... & a_{3n} & b_{3}\\\\\n", 192 | " ... & ... & ... & ...& ...\\\\\n", 193 | " a_{n1} & a_{n2} & ... & a_{nn} & b_{n} \\end{bmatrix}$" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": { 200 | "collapsed": true 201 | }, 202 | "outputs": [], 203 | "source": [ 204 | "#TODO construct the augment matrix of matrix A and column vector b, assuming A and b have same number of rows\n", 205 | "def augmentMatrix(A, b):\n", 206 | " return None" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": { 213 | "collapsed": true 214 | }, 215 | "outputs": [], 216 | "source": [ 217 | "# run following code to test your augmentMatrix function\n", 218 | "%run -i -e test.py LinearRegressionTestCase.test_augmentMatrix" 219 | ] 220 | }, 221 | { 222 | "cell_type": "markdown", 223 | "metadata": {}, 224 | "source": [ 225 | "## 2.2 Basic row operations\n", 226 | "- exchange two rows\n", 227 | "- scale a row\n", 228 | "- add a scaled row to another" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": { 235 | "collapsed": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "# TODO r1 <---> r2\n", 240 | "# TODO in-place operation, no return value\n", 241 | "def swapRows(M, r1, r2):\n", 242 | " pass" 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "metadata": { 249 | "collapsed": true 250 | }, 251 | "outputs": [], 252 | "source": [ 253 | "# run following code to test your swapRows function\n", 254 | "%run -i -e test.py LinearRegressionTestCase.test_swapRows" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": null, 260 | "metadata": { 261 | "collapsed": true 262 | }, 263 | "outputs": [], 264 | "source": [ 265 | "# TODO r1 <--- r1 * scale\n", 266 | "# TODO in-place operation, no return value\n", 267 | "def scaleRow(M, r, scale):\n", 268 | " pass" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": { 275 | "collapsed": true 276 | }, 277 | "outputs": [], 278 | "source": [ 279 | "# run following code to test your scaleRow function\n", 280 | "%run -i -e test.py LinearRegressionTestCase.test_scaleRow" 281 | ] 282 | }, 283 | { 284 | "cell_type": "code", 285 | "execution_count": null, 286 | "metadata": { 287 | "collapsed": true 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "# TODO r1 <--- r1 + r2*scale\n", 292 | "# TODO in-place operation, no return value\n", 293 | "def addScaledRow(M, r1, r2, scale):\n", 294 | " pass" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": null, 300 | "metadata": { 301 | "collapsed": true 302 | }, 303 | "outputs": [], 304 | "source": [ 305 | "# run following code to test your addScaledRow function\n", 306 | "%run -i -e test.py LinearRegressionTestCase.test_addScaledRow" 307 | ] 308 | }, 309 | { 310 | "cell_type": "markdown", 311 | "metadata": {}, 312 | "source": [ 313 | "## 2.3 Gauss-jordan method to solve Ax = b\n", 314 | "\n", 315 | "### Hint：\n", 316 | "\n", 317 | "Step 1: Check if A and b have same number of rows\n", 318 | "Step 2: Construct augmented matrix Ab\n", 319 | "\n", 320 | "Step 3: Column by column, transform Ab to reduced row echelon form [wiki link](https://en.wikipedia.org/wiki/Row_echelon_form#Reduced_row_echelon_form)\n", 321 | " \n", 322 | " for every column of Ab (except the last one)\n", 323 | " column c is the current column\n", 324 | " Find in column c, at diagonal and under diagonal (row c ~ N) the maximum absolute value\n", 325 | " If the maximum absolute value is 0\n", 326 | " then A is singular, return None （Prove this proposition in Question 2.4）\n", 327 | " else\n", 328 | " Apply row operation 1, swap the row of maximum with the row of diagonal element (row c)\n", 329 | " Apply row operation 2, scale the diagonal element of column c to 1\n", 330 | " Apply row operation 3 mutiple time, eliminate every other element in column c\n", 331 | " \n", 332 | "Step 4: return the last column of Ab\n", 333 | "\n", 334 | "### Remark：\n", 335 | "We don't use the standard algorithm first transfering Ab to row echelon form and then to reduced row echelon form. Instead, we arrives directly at reduced row echelon form. If you are familiar with the stardard way, try prove to yourself that they are equivalent. " 336 | ] 337 | }, 338 | { 339 | "cell_type": "code", 340 | "execution_count": null, 341 | "metadata": { 342 | "collapsed": true 343 | }, 344 | "outputs": [], 345 | "source": [ 346 | "#TODO implement gaussian jordan method to solve Ax = b\n", 347 | "\n", 348 | "\"\"\" Gauss-jordan method to solve x such that Ax = b.\n", 349 | " A: square matrix, list of lists\n", 350 | " b: column vector, list of lists\n", 351 | " decPts: degree of rounding, default value 4\n", 352 | " epsilon: threshold for zero, default value 1.0e-16\n", 353 | " \n", 354 | " return x such that Ax = b, list of lists \n", 355 | " return None if A and b have same height\n", 356 | " return None if A is (almost) singular\n", 357 | "\"\"\"\n", 358 | "\n", 359 | "def gj_Solve(A, b, decPts=4, epsilon = 1.0e-16):\n", 360 | " return None" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": null, 366 | "metadata": { 367 | "collapsed": true 368 | }, 369 | "outputs": [], 370 | "source": [ 371 | "# run following code to test your addScaledRow function\n", 372 | "%run -i -e test.py LinearRegressionTestCase.test_gj_Solve" 373 | ] 374 | }, 375 | { 376 | "cell_type": "markdown", 377 | "metadata": {}, 378 | "source": [ 379 | "## 2.4 Prove the following proposition:\n", 380 | "\n", 381 | "**If square matrix A can be divided into four parts: ** \n", 382 | "\n", 383 | "$ A = \\begin{bmatrix}\n", 384 | " I & X \\\\\n", 385 | " Z & Y \\\\\n", 386 | "\\end{bmatrix} $, where I is the identity matrix, Z is all zero and the first column of Y is all zero, \n", 387 | "\n", 388 | "**then A is singular.**\n", 389 | "\n", 390 | "Hint: There are mutiple ways to prove this problem. \n", 391 | "- consider the rank of Y and A\n", 392 | "- consider the determinate of Y and A \n", 393 | "- consider certain column is the linear combination of other columns" 394 | ] 395 | }, 396 | { 397 | "cell_type": "markdown", 398 | "metadata": {}, 399 | "source": [ 400 | "TODO Please use latex （refering to the latex in problem may help）\n", 401 | "\n", 402 | "TODO Proof：" 403 | ] 404 | }, 405 | { 406 | "cell_type": "markdown", 407 | "metadata": {}, 408 | "source": [ 409 | "---\n", 410 | "\n", 411 | "# 3 Linear Regression: \n", 412 | "\n", 413 | "## 3.1 Compute the gradient of loss function with respect to parameters \n", 414 | "## (Choose one between two 3.1 questions)\n", 415 | "\n", 416 | "We define loss funtion $E$ as \n", 417 | "$$\n", 418 | "E(m, b) = \\sum_{i=1}^{n}{(y_i - mx_i - b)^2}\n", 419 | "$$\n", 420 | "and we define vertex $Y$, matrix $X$ and vertex $h$ :\n", 421 | "$$\n", 422 | "Y = \\begin{bmatrix}\n", 423 | " y_1 \\\\\n", 424 | " y_2 \\\\\n", 425 | " ... \\\\\n", 426 | " y_n\n", 427 | "\\end{bmatrix}\n", 428 | ",\n", 429 | "X = \\begin{bmatrix}\n", 430 | " x_1 & 1 \\\\\n", 431 | " x_2 & 1\\\\\n", 432 | " ... & ...\\\\\n", 433 | " x_n & 1 \\\\\n", 434 | "\\end{bmatrix},\n", 435 | "h = \\begin{bmatrix}\n", 436 | " m \\\\\n", 437 | " b \\\\\n", 438 | "\\end{bmatrix}\n", 439 | "$$\n", 440 | "\n", 441 | "\n", 442 | "Proves that \n", 443 | "$$\n", 444 | "\\frac{\\partial E}{\\partial m} = \\sum_{i=1}^{n}{-2x_i(y_i - mx_i - b)}\n", 445 | "$$\n", 446 | "\n", 447 | "$$\n", 448 | "\\frac{\\partial E}{\\partial b} = \\sum_{i=1}^{n}{-2(y_i - mx_i - b)}\n", 449 | "$$\n", 450 | "\n", 451 | "$$\n", 452 | "\\begin{bmatrix}\n", 453 | " \\frac{\\partial E}{\\partial m} \\\\\n", 454 | " \\frac{\\partial E}{\\partial b} \n", 455 | "\\end{bmatrix} = 2X^TXh - 2X^TY\n", 456 | "$$" 457 | ] 458 | }, 459 | { 460 | "cell_type": "markdown", 461 | "metadata": {}, 462 | "source": [ 463 | "TODO Please use latex （refering to the latex in problem may help）\n", 464 | "\n", 465 | "TODO Proof：" 466 | ] 467 | }, 468 | { 469 | "cell_type": "markdown", 470 | "metadata": {}, 471 | "source": [ 472 | "## 3.1 Compute the gradient of loss function with respect to parameters \n", 473 | "## (Choose one between two 3.1 questions)\n", 474 | "We define loss funtion $E$ as \n", 475 | "$$\n", 476 | "E(m, b) = \\sum_{i=1}^{n}{(y_i - mx_i - b)^2}\n", 477 | "$$\n", 478 | "and we define vertex $Y$, matrix $X$ and vertex $h$ :\n", 479 | "$$\n", 480 | "Y = \\begin{bmatrix}\n", 481 | " y_1 \\\\\n", 482 | " y_2 \\\\\n", 483 | " ... \\\\\n", 484 | " y_n\n", 485 | "\\end{bmatrix}\n", 486 | ",\n", 487 | "X = \\begin{bmatrix}\n", 488 | " x_1 & 1 \\\\\n", 489 | " x_2 & 1\\\\\n", 490 | " ... & ...\\\\\n", 491 | " x_n & 1 \\\\\n", 492 | "\\end{bmatrix},\n", 493 | "h = \\begin{bmatrix}\n", 494 | " m \\\\\n", 495 | " b \\\\\n", 496 | "\\end{bmatrix}\n", 497 | "$$\n", 498 | "\n", 499 | "Proves that \n", 500 | "$$\n", 501 | "E = Y^TY -2(Xh)^TY + (Xh)^TXh\n", 502 | "$$\n", 503 | "\n", 504 | "$$\n", 505 | "\\frac{\\partial E}{\\partial h} = 2X^TXh - 2X^TY\n", 506 | "$$" 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "metadata": {}, 512 | "source": [ 513 | "TODO Please use latex （refering to the latex in problem may help）\n", 514 | "\n", 515 | "TODO Proof：" 516 | ] 517 | }, 518 | { 519 | "cell_type": "markdown", 520 | "metadata": {}, 521 | "source": [ 522 | "## 3.2 Linear Regression\n", 523 | "### Solve equation $X^TXh = X^TY $ to compute the best parameter for linear regression." 524 | ] 525 | }, 526 | { 527 | "cell_type": "code", 528 | "execution_count": null, 529 | "metadata": { 530 | "collapsed": true 531 | }, 532 | "outputs": [], 533 | "source": [ 534 | "#TODO implement linear regression \n", 535 | "'''\n", 536 | "points: list of (x,y) tuple\n", 537 | "return m and b\n", 538 | "'''\n", 539 | "def linearRegression(points):\n", 540 | " return 0,0" 541 | ] 542 | }, 543 | { 544 | "cell_type": "markdown", 545 | "metadata": {}, 546 | "source": [ 547 | "## 3.3 Test your linear regression implementation" 548 | ] 549 | }, 550 | { 551 | "cell_type": "code", 552 | "execution_count": null, 553 | "metadata": { 554 | "collapsed": true 555 | }, 556 | "outputs": [], 557 | "source": [ 558 | "#TODO Construct the linear function\n", 559 | "\n", 560 | "#TODO Construct points with gaussian noise\n", 561 | "import random\n", 562 | "\n", 563 | "#TODO Compute m and b and compare with ground truth" 564 | ] 565 | } 566 | ], 567 | "metadata": { 568 | "anaconda-cloud": {}, 569 | "celltoolbar": "Raw Cell Format", 570 | "kernelspec": { 571 | "display_name": "Python 3", 572 | "language": "python", 573 | "name": "python3" 574 | }, 575 | "language_info": { 576 | "codemirror_mode": { 577 | "name": "ipython", 578 | "version": 3 579 | }, 580 | "file_extension": ".py", 581 | "mimetype": "text/x-python", 582 | "name": "python", 583 | "nbconvert_exporter": "python", 584 | "pygments_lexer": "ipython3", 585 | "version": "3.6.1" 586 | } 587 | }, 588 | "nbformat": 4, 589 | "nbformat_minor": 2 590 | } 591 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import numpy as np 3 | 4 | from decimal import * 5 | 6 | class LinearRegressionTestCase(unittest.TestCase): 7 | """Test for linear regression project""" 8 | 9 | def test_shape(self): 10 | 11 | for _ in range(10): 12 | r,c = np.random.randint(low=1,high=25,size=2) 13 | matrix = np.random.randint(low=-10,high=10,size=(r,c)) 14 | self.assertEqual(shape(matrix.tolist()),(r,c),'Wrong answer') 15 | 16 | 17 | def test_matxRound(self): 18 | 19 | for decpts in range(10): 20 | r,c = np.random.randint(low=1,high=25,size=2) 21 | matrix = np.random.random((r,c)) 22 | 23 | mat = matrix.tolist() 24 | dec_true = [[Decimal(str(round(num,decpts))) for num in row] for row in mat] 25 | 26 | matxRound(mat,decpts) 27 | dec_test = [[Decimal(str(num)) for num in row] for row in mat] 28 | 29 | res = Decimal('0') 30 | for i in range(len(mat)): 31 | for j in range(len(mat[0])): 32 | res += dec_test[i][j].compare_total(dec_true[i][j]) 33 | 34 | self.assertEqual(res,Decimal('0'),'Wrong answer') 35 | 36 | 37 | def test_transpose(self): 38 | for _ in range(100): 39 | r,c = np.random.randint(low=1,high=25,size=2) 40 | matrix = np.random.random((r,c)) 41 | 42 | mat = matrix.tolist() 43 | t = np.array(transpose(mat)) 44 | 45 | self.assertEqual(t.shape,(c,r),"Expected shape{}, but got shape{}".format((c,r),t.shape)) 46 | self.assertTrue((matrix.T == t).all(),'Wrong answer') 47 | 48 | 49 | def test_matxMultiply(self): 50 | 51 | for _ in range(100): 52 | r,d,c = np.random.randint(low=1,high=25,size=3) 53 | mat1 = np.random.randint(low=-10,high=10,size=(r,d)) 54 | mat2 = np.random.randint(low=-5,high=5,size=(d,c)) 55 | dotProduct = np.dot(mat1,mat2) 56 | 57 | dp = np.array(matxMultiply(mat1.tolist(),mat2.tolist())) 58 | self.assertEqual(dotProduct.shape, dp.shape, 59 | 'Wrong answer, expected shape{}, but got shape{}'.format(dotProduct.shape, dp.shape)) 60 | self.assertTrue((dotProduct == dp).all(),'Wrong answer') 61 | 62 | mat1 = np.random.randint(low=-10,high=10,size=(r,5)) 63 | mat2 = np.random.randint(low=-5,high=5,size=(4,c)) 64 | mat3 = np.random.randint(low=-5,high=5,size=(6,c)) 65 | with self.assertRaises(ValueError,msg="Matrix A\'s column number doesn\'t equal to Matrix b\'s row number"): 66 | matxMultiply(mat1.tolist(),mat2.tolist()) 67 | with self.assertRaises(ValueError,msg="Matrix A\'s column number doesn\'t equal to Matrix b\'s row number"): 68 | matxMultiply(mat1.tolist(),mat3.tolist()) 69 | 70 | 71 | def test_augmentMatrix(self): 72 | 73 | for _ in range(50): 74 | r,c = np.random.randint(low=1,high=25,size=2) 75 | A = np.random.randint(low=-10,high=10,size=(r,c)) 76 | b = np.random.randint(low=-10,high=10,size=(r,1)) 77 | Amat = A.tolist() 78 | bmat = b.tolist() 79 | 80 | Ab = np.array(augmentMatrix(Amat,bmat)) 81 | ab = np.hstack((A,b)) 82 | 83 | self.assertTrue(A.tolist() == Amat,"Matrix A shouldn't be modified") 84 | self.assertEqual(Ab.shape, ab.shape, 85 | 'Wrong answer, expected shape{}, but got shape{}'.format(ab.shape, Ab.shape)) 86 | self.assertTrue((Ab == ab).all(),'Wrong answer') 87 | 88 | def test_swapRows(self): 89 | for _ in range(10): 90 | r,c = np.random.randint(low=1,high=25,size=2) 91 | matrix = np.random.random((r,c)) 92 | 93 | mat = matrix.tolist() 94 | 95 | r1, r2 = np.random.randint(0,r, size = 2) 96 | swapRows(mat,r1,r2) 97 | 98 | matrix[[r1,r2]] = matrix[[r2,r1]] 99 | 100 | self.assertTrue((matrix == np.array(mat)).all(),'Wrong answer') 101 | 102 | def test_scaleRow(self): 103 | 104 | for _ in range(10): 105 | r,c = np.random.randint(low=1,high=25,size=2) 106 | matrix = np.random.random((r,c)) 107 | 108 | mat = matrix.tolist() 109 | 110 | rr = np.random.randint(0,r) 111 | with self.assertRaises(ValueError): 112 | scaleRow(mat,rr,0) 113 | 114 | scale = np.random.randint(low=1,high=10) 115 | scaleRow(mat,rr,scale) 116 | matrix[rr] *= scale 117 | 118 | self.assertTrue((matrix == np.array(mat)).all(),'Wrong answer') 119 | 120 | def test_addScaledRow(self): 121 | 122 | for _ in range(10): 123 | r,c = np.random.randint(low=1,high=25,size=2) 124 | matrix = np.random.random((r,c)) 125 | 126 | mat = matrix.tolist() 127 | 128 | r1,r2 = np.random.randint(0,r,size=2) 129 | 130 | scale = np.random.randint(low=1,high=10) 131 | addScaledRow(mat,r1,r2,scale) 132 | matrix[r1] += scale * matrix[r2] 133 | 134 | self.assertTrue((matrix == np.array(mat)).all(),'Wrong answer') 135 | 136 | 137 | def test_gj_Solve(self): 138 | 139 | for _ in range(9999): 140 | r = np.random.randint(low=3,high=10) 141 | A = np.random.randint(low=-10,high=10,size=(r,r)) 142 | b = np.arange(r).reshape((r,1)) 143 | 144 | x = gj_Solve(A.tolist(),b.tolist(),epsilon=1.0e-8) 145 | 146 | if np.linalg.matrix_rank(A) < r: 147 | self.assertEqual(x,None,"Matrix A is singular") 148 | else: 149 | self.assertNotEqual(x,None,"Matrix A is not singular") 150 | self.assertEqual(np.array(x).shape,(r,1),"Expected shape({},1), but got shape{}".format(r,np.array(x).shape)) 151 | Ax = np.dot(A,np.array(x)) 152 | loss = np.mean((Ax - b)**2) 153 | self.assertTrue(loss<0.1,"Bad result.") 154 | 155 | if __name__ == '__main__': 156 | unittest.main() 157 | --------------------------------------------------------------------------------