├── .gitignore ├── LICENSE ├── README.md ├── notebooks ├── Demo.ipynb ├── Machine-Learning-API.ipynb ├── assets │ ├── Pipeline.png │ ├── Pipeline.sketch │ └── ml_map.png ├── data │ ├── bitcoin │ │ └── btc.pkl │ ├── fashion-mnist │ │ └── fashion-mnist.pkl │ └── trump │ │ └── speeches.txt └── models │ └── trump │ ├── architecture.yaml │ ├── weights-01-2.796.hdf5 │ ├── weights-02-2.328.hdf5 │ └── weights-03-2.152.hdf5 ├── requirements.txt ├── scripts ├── clean.sh ├── remove.sh ├── setup.sh └── test.sh └── tests └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # General 2 | .DS_Store 3 | .AppleDouble 4 | .LSOverride 5 | .vscode 6 | 7 | # Icon must end with two \r 8 | Icon 9 | 10 | # Thumbnails 11 | ._* 12 | 13 | # Files that might appear in the root of a volume 14 | .DocumentRevisions-V100 15 | .fseventsd 16 | .Spotlight-V100 17 | .TemporaryItems 18 | .Trashes 19 | .VolumeIcon.icns 20 | .com.apple.timemachine.donotpresent 21 | 22 | # Directories potentially created on remote AFP share 23 | .AppleDB 24 | .AppleDesktop 25 | Network Trash Folder 26 | Temporary Items 27 | .apdisk 28 | 29 | # Byte-compiled / optimized / DLL files 30 | __pycache__/ 31 | *.py[cod] 32 | *$py.class 33 | 34 | # C extensions 35 | *.so 36 | 37 | # Distribution / packaging 38 | .Python 39 | build/ 40 | develop-eggs/ 41 | dist/ 42 | downloads/ 43 | eggs/ 44 | .eggs/ 45 | lib/ 46 | lib64/ 47 | parts/ 48 | sdist/ 49 | var/ 50 | wheels/ 51 | *.egg-info/ 52 | .installed.cfg 53 | *.egg 54 | MANIFEST 55 | 56 | # PyInstaller 57 | # Usually these files are written by a python script from a template 58 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 59 | *.manifest 60 | *.spec 61 | 62 | # Installer logs 63 | pip-log.txt 64 | pip-delete-this-directory.txt 65 | 66 | # Unit test / coverage reports 67 | htmlcov/ 68 | .tox/ 69 | .coverage 70 | .coverage.* 71 | .cache 72 | nosetests.xml 73 | coverage.xml 74 | *.cover 75 | .hypothesis/ 76 | 77 | # Translations 78 | *.mo 79 | *.pot 80 | 81 | # Django stuff: 82 | *.log 83 | .static_storage/ 84 | .media/ 85 | local_settings.py 86 | 87 | # Flask stuff: 88 | instance/ 89 | .webassets-cache 90 | 91 | # Scrapy stuff: 92 | .scrapy 93 | 94 | # Sphinx documentation 95 | docs/_build/ 96 | 97 | # PyBuilder 98 | target/ 99 | 100 | # Jupyter Notebook 101 | .ipynb_checkpoints 102 | 103 | # pyenv 104 | .python-version 105 | 106 | # celery beat schedule file 107 | celerybeat-schedule 108 | 109 | # SageMath parsed files 110 | *.sage.py 111 | 112 | # Environments 113 | .env 114 | .venv 115 | env/ 116 | venv/ 117 | ENV/ 118 | env.bak/ 119 | venv.bak/ 120 | 121 | # Spyder project settings 122 | .spyderproject 123 | .spyproject 124 | 125 | # Rope project settings 126 | .ropeproject 127 | 128 | # mkdocs documentation 129 | /site 130 | 131 | # mypy 132 | .mypy_cache/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 ICDSS 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Machine Learning API 2 | > ICDSS Machine Learning Workshop Series: Machine Learning API 3 | 4 | - [Prerequisites](#prerequisites) 5 | - [Overview](#overview) 6 | - [Notebook](notebooks/Machine-Learning-API.ipynb) 7 | - [License](#license) 8 | 9 | ## Getting Started 10 | 11 | ### macOS 12 | 13 | 1. `source scripts/setup.sh` 14 | 15 | 16 | 17 | ## Prerequisites 18 | 19 | * Linear Models 20 | * Scientific Python (NumPy & SciPy) 21 | * Neural Networks 22 | 23 | ## Overview 24 | 25 | In this workshop we will cover the most frequently used High-Level Machine Learning libraries, 26 | [`scikit-learn`](http://scikit-learn.org/) and [`keras`](https://keras.io/). 27 | The focus of this workshop will be on how to the implementation of abstract models in code, 28 | using the universal `fit`-`predict`-`score` pattern most machine learning APIs follow. 29 | Multiple sandbox projects will be attempted, including multivariate regression, classification and time-series forecasting. 30 | 31 | ## License 32 | 33 | MIT License 34 | 35 | Copyright (c) 2018 ICDSS 36 | 37 | Permission is hereby granted, free of charge, to any person obtaining a copy 38 | of this software and associated documentation files (the "Software"), to deal 39 | in the Software without restriction, including without limitation the rights 40 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 41 | copies of the Software, and to permit persons to whom the Software is 42 | furnished to do so, subject to the following conditions: 43 | 44 | The above copyright notice and this permission notice shall be included in all 45 | copies or substantial portions of the Software. 46 | 47 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 48 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 49 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 50 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 51 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 52 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 53 | SOFTWARE. 54 | -------------------------------------------------------------------------------- /notebooks/Demo.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Machine Learning API" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "> ICDSS Machine Learning Workshop Series: Coding Models on `scikit-learn`, `keras` & `fbprophet`" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "* [Pipeline](#pipeline)\n", 22 | " * [Preprocessing](#pipe:preprocessing)\n", 23 | " * [Estimation](#pipe:estimation)\n", 24 | " * [Supervised Learning](#pipe:supervised-learning)\n", 25 | " * [Unsupervised Learning](#pipe:unsupervised-learning)\n", 26 | " * [Evaluation](#pipe:evaluation)\n", 27 | "* [`scikit-learn`](#scikit-learn)\n", 28 | " * [Preprocessing](#sk:preprocessing)\n", 29 | " * [Estimation](#sk:estimation)\n", 30 | " * [Model Selection](#sk:model-selection)\n", 31 | " * [Hyperparameters](#sk:hyperparameters)\n", 32 | " * [`GridSearchCV`](#sk:GridSearchCV)\n", 33 | " * [Pipeline](#sk:pipeline)\n", 34 | "* [`keras`](#keras)\n", 35 | " * [Dense](#keras:dense)\n", 36 | " * [Iris](#keras:iris)\n", 37 | " * [CNN](#keras:cnn)\n", 38 | " * [Fashion MNIST](#keras:fashion-mnist)\n", 39 | " * [LSTM](#keras:lstm)\n", 40 | " * [President Trump Generator](#keras:president-trump-generator)\n", 41 | "* [`fbprophet`](#fbprophet)\n", 42 | " * [Bitcoin Capital Market](#fbprophet:bitcoin-capital-market)" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "## Pipeline " 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "\"Drawing\"" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "%reset -f\n", 66 | "\n", 67 | "import numpy as np\n", 68 | "import matplotlib.pyplot as plt\n", 69 | "plt.style.use('seaborn-muted')\n", 70 | "\n", 71 | "from sklearn import datasets" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "### Preprocessing " 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "class Preprocessor:\n", 88 | " \"\"\"Generic `preprocessing` transformer.\"\"\"\n", 89 | " \n", 90 | " def __init__(self, **hyperparameters):\n", 91 | " \"\"\"Initialise Hyperparameters\"\"\"\n", 92 | " raise NotImplementedError\n", 93 | " \n", 94 | " def fit(self, X_train):\n", 95 | " \"\"\"Set state of `preprocessor`.\"\"\"\n", 96 | " raise NotImplementedError\n", 97 | " \n", 98 | " def transform(self, X):\n", 99 | " \"\"\"Apply transformation.\"\"\"\n", 100 | " raise NotImplementedError\n", 101 | " \n", 102 | " def fit_transform(self, X):\n", 103 | " \"\"\"Reset state and apply transformer.\"\"\"\n", 104 | " self.fit(X)\n", 105 | " return self.transform(X)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": {}, 111 | "source": [ 112 | "#### Principle Component Analysis (PCA)" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "class PCA:\n", 122 | " \"\"\"Principle Component Analysis.\"\"\"\n", 123 | " \n", 124 | " def __init__(self, n_components=2):\n", 125 | " \"\"\"Contructor.\n", 126 | " \n", 127 | " Parameters\n", 128 | " ----------\n", 129 | " n_comps: int\n", 130 | " Number of principle components\n", 131 | " \"\"\"\n", 132 | " self.n_comps = n_components\n", 133 | " self.mu = None\n", 134 | " self.U = None\n", 135 | " self._fitted = False\n", 136 | " \n", 137 | " def fit(self, X):\n", 138 | " \"\"\"Fit PCA according to `X.cov()`.\n", 139 | " \n", 140 | " Parameters\n", 141 | " ----------\n", 142 | " X: numpy.ndarray\n", 143 | " Features matrix\n", 144 | " Returns\n", 145 | " -------\n", 146 | " array: numpy.ndarray\n", 147 | " Transformed features matrix\n", 148 | " \"\"\"\n", 149 | " self.D, N = X.shape\n", 150 | " self.mu = X.mean(axis=1).reshape(-1, 1)\n", 151 | " # center data\n", 152 | " A = X - self.mu\n", 153 | " # covariance matrix\n", 154 | " S = (1 / N) * np.dot(A.T, A)\n", 155 | " # eigendecomposition\n", 156 | " _l, _v = np.linalg.eig(S)\n", 157 | " _l = np.real(_l)\n", 158 | " _v = np.real(_v)\n", 159 | " # short eigenvalues\n", 160 | " _indexes = np.argsort(_l)[::-1]\n", 161 | " # sorted eigenvalues and eigenvectors\n", 162 | " l, v = _l[_indexes], _v[:, _indexes]\n", 163 | " # eigenvalues\n", 164 | " V = v[:, :self.n_comps]\n", 165 | " # unnormalised transformer\n", 166 | " _U = np.dot(A, V)\n", 167 | " # transformation matrix\n", 168 | " self.U = _U / np.apply_along_axis(np.linalg.norm, 0, _U)\n", 169 | " # unnormalised transformed features\n", 170 | " W = np.dot(self.U.T, A)\n", 171 | " # data statistics\n", 172 | " self.W_mu = np.mean(W, axis=1)\n", 173 | " self.W_std = np.std(W, axis=1)\n", 174 | "\n", 175 | " self._fitted = True\n", 176 | "\n", 177 | " return self\n", 178 | "\n", 179 | " def transform(self, X):\n", 180 | " \"\"\"Transform `X` by projecting it to PCA feature space.\n", 181 | " \n", 182 | " Parameters\n", 183 | " ----------\n", 184 | " X: numpy.ndarray\n", 185 | " Features matrix\n", 186 | " Returns\n", 187 | " -------\n", 188 | " array: numpy.ndarray\n", 189 | " Transformed features matrix\n", 190 | " \"\"\"\n", 191 | " if not self._fitted:\n", 192 | " raise AssertionError('Not fitted yet.')\n", 193 | " # centered data\n", 194 | " Phi = X - self.mu\n", 195 | " # unnormalised transformed features\n", 196 | " W = np.dot(self.U.T, Phi)\n", 197 | "\n", 198 | " return ((W.T - self.W_mu) / self.W_std).T\n", 199 | " \n", 200 | " def fit_transform(self, X):\n", 201 | " \"\"\"Fit PCA according to `X.cov()`\n", 202 | " and then transform `X` by\n", 203 | " projecting it to PCA feature space.\n", 204 | " \n", 205 | " Parameters\n", 206 | " ----------\n", 207 | " X: numpy.ndarray\n", 208 | " Features matrix\n", 209 | " Returns\n", 210 | " -------\n", 211 | " array: numpy.ndarray\n", 212 | " Transformed features matrix\n", 213 | " \"\"\"\n", 214 | " self.fit(X)\n", 215 | " return self.transform(X)" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": null, 221 | "metadata": {}, 222 | "outputs": [], 223 | "source": [ 224 | "# preprocessor\n", 225 | "pca = PCA(n_components=2)\n", 226 | "\n", 227 | "# fetch data\n", 228 | "iris = datasets.load_iris()\n", 229 | "# supervised-learning data\n", 230 | "X, y = iris.data, iris.target\n", 231 | "# transformed data\n", 232 | "X_transform = pca.fit_transform(X.T)\n", 233 | "\n", 234 | "_, ax = plt.subplots(figsize=(20.0, 3.0))\n", 235 | "for i in range(iris.target.max() + 1):\n", 236 | " I = iris.target == i\n", 237 | " ax.scatter(X_transform[0, I], X_transform[1, I], label=iris.target_names[i])\n", 238 | " ax.set_title('Iris Dataset PCA')\n", 239 | " ax.set_xlabel('Principle Component 1')\n", 240 | " ax.set_ylabel('Principle Component 2')\n", 241 | "ax.legend();" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": {}, 247 | "source": [ 248 | "#### `challenge` QuadraticFeatures\n", 249 | "\n", 250 | "Code the transformation $\\mathcal{G}$ such that:\n", 251 | "\n", 252 | "$$\\mathcal{G}: \\mathbb{R}^{2} \\rightarrow \\mathbb{R}^{6}$$\n", 253 | "\n", 254 | "according to the mapping:\n", 255 | "\n", 256 | "$$\\begin{bmatrix}x_{1} & x_{2}\\end{bmatrix} \\mapsto \\begin{bmatrix}1 & x_{1} & x_{2} & x_{1}x_{2} & x_{1}^{2} & x_{2}^{2}\\end{bmatrix}$$" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "class QuadraticFeatures:\n", 266 | " \"\"\"Generate Quadratic features.\"\"\"\n", 267 | " \n", 268 | " def fit(self, X_train):\n", 269 | " \"\"\"Set state of `preprocessor`.\"\"\"\n", 270 | " pass\n", 271 | " return self\n", 272 | " \n", 273 | " def transform(self, X):\n", 274 | " \"\"\"Apply transformation.\"\"\"\n", 275 | " # get dimensions\n", 276 | " N, D = X.shape\n", 277 | " # check number of input features\n", 278 | " assert(D==2)\n", 279 | " \n", 280 | " # get x_{1} column\n", 281 | " x_1 = X[:, 0]\n", 282 | " # get x_{2} column\n", 283 | " x_2 = X[:, 1]\n", 284 | " \n", 285 | " # initialise output matrix\n", 286 | " out = np.empty(shape=(N, 6))\n", 287 | " # column 1: constant\n", 288 | " out[:, 0] = \n", 289 | " # column 2: x_{1}\n", 290 | " out[:, 1] = \n", 291 | " # column 3: x_{2}\n", 292 | " out[:, 2] = \n", 293 | " # column 4: x_{1}x_{2}\n", 294 | " out[:, 3] = \n", 295 | " # column 5: x_{1}^{2}\n", 296 | " out[:, 4] = \n", 297 | " # column 6: x_{2}^{2}\n", 298 | " out[:, 5] = \n", 299 | " \n", 300 | " return out\n", 301 | " \n", 302 | " def fit_transform(self, X):\n", 303 | " \"\"\"Reset state and apply transformer.\"\"\"\n", 304 | " self.fit(X)\n", 305 | " return self.transform(X)" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": {}, 311 | "source": [ 312 | "##### Unit Test for `QuadraticFeatures`" 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": null, 318 | "metadata": {}, 319 | "outputs": [], 320 | "source": [ 321 | "# unit test function\n", 322 | "assert (QuadraticFeatures().fit_transform(np.array([[1, 2],\n", 323 | " [3, 4]])) == np.array([[ 1., 1., 2., 2., 1., 4.],\n", 324 | " [ 1., 3., 4., 12., 9., 16.]])).all(), \"Wrong implementation, try again!\"\n", 325 | "'Well Done!'" 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "metadata": {}, 331 | "source": [ 332 | "### Estimation " 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": null, 338 | "metadata": {}, 339 | "outputs": [], 340 | "source": [ 341 | "class Estimator:\n", 342 | " \"\"\"Generic `estimator` class.\"\"\"\n", 343 | " \n", 344 | " def __init__(self, **hyperparameters):\n", 345 | " \"\"\"Initialise Hyperparameters\"\"\"\n", 346 | " raise NotImplementedError\n", 347 | " \n", 348 | " def fit(self, X, y):\n", 349 | " \"\"\"Train model.\"\"\"\n", 350 | " return self\n", 351 | " \n", 352 | " def predict(self, X):\n", 353 | " \"\"\"Forward/Inference pass.\"\"\"\n", 354 | " return y_hat\n", 355 | " \n", 356 | " def score(self, X, y):\n", 357 | " \"\"\"Performance results.\"\"\"\n", 358 | " y_hat = self.predict(X)\n", 359 | " return self._loss(y, y_hat)\n", 360 | " \n", 361 | " def _loss(self, y, y_hat):\n", 362 | " \"\"\"Objective function for scoring.\"\"\"\n", 363 | " return L(y, y_hat)" 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "metadata": {}, 369 | "source": [ 370 | "#### `challenge` Linear Regression\n", 371 | "\n", 372 | "Code the estimator $\\mathcal{F}$ such that:\n", 373 | "\n", 374 | "$$\\mathbf{y} = \\mathbf{X} * \\mathbf{w}_{MLE}$$\n", 375 | "\n", 376 | "for the Maximum Likelihood estimation weights parameters:\n", 377 | "\n", 378 | "$$\\mathbf{w}_{MLE} = (\\mathbf{X}^{T} \\mathbf{X})^{-1} * \\mathbf{X}^{T} * \\mathbf{y}$$" 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": null, 384 | "metadata": {}, 385 | "outputs": [], 386 | "source": [ 387 | "class LinearRegression:\n", 388 | " \"\"\"Linear Regression `estimator` class.\"\"\"\n", 389 | " \n", 390 | " def __init__(self, dimensionality):\n", 391 | " \"\"\"Initialise Hyperparameters\"\"\"\n", 392 | " self.dimensionality = \n", 393 | " self.w_mle = \n", 394 | " \n", 395 | " def fit(self, X, y):\n", 396 | " \"\"\"Train model.\"\"\"\n", 397 | " self.w_mle = \n", 398 | " return self\n", 399 | " \n", 400 | " def predict(self, X):\n", 401 | " \"\"\"Forward/Inference pass.\"\"\"\n", 402 | " y_hat = \n", 403 | " return y_hat\n", 404 | " \n", 405 | " def score(self, X, y):\n", 406 | " \"\"\"Performance results.\"\"\"\n", 407 | " y_hat = self.predict(X)\n", 408 | " return self._loss(y, y_hat)\n", 409 | " \n", 410 | " def _loss(self, y, y_hat):\n", 411 | " \"\"\"Objective function for scoring.\"\"\"\n", 412 | " return ((y - y_hat) ** 2).mean(axis=None)" 413 | ] 414 | }, 415 | { 416 | "cell_type": "markdown", 417 | "metadata": {}, 418 | "source": [ 419 | "##### Unit Test for `LinearRegression`" 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": null, 425 | "metadata": {}, 426 | "outputs": [], 427 | "source": [ 428 | "# dummy data\n", 429 | "X, y = np.array([[1., 2.], [1., 3.], [1., 7.]]), np.array([2., 3., 7.])\n", 430 | "# input dimensions\n", 431 | "N, D = X.shape\n", 432 | "# estimator: init & fit\n", 433 | "lr = LinearRegression(dimensionality=D).fit(X, y)\n", 434 | "\n", 435 | "# unit test function\n", 436 | "assert np.isclose(lr.predict(X), y).all(), 'Wrong implementation, try again!'\n", 437 | "'Well Done!'" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": {}, 443 | "source": [ 444 | "## `scikit-learn` " 445 | ] 446 | }, 447 | { 448 | "cell_type": "markdown", 449 | "metadata": {}, 450 | "source": [ 451 | "### Preprocessing " 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": null, 457 | "metadata": {}, 458 | "outputs": [], 459 | "source": [ 460 | "%reset -f\n", 461 | "\n", 462 | "import numpy as np\n", 463 | "import pandas as pd\n", 464 | "import sklearn\n", 465 | "import sklearn.ensemble\n", 466 | "import sklearn.neural_network\n", 467 | "import sklearn.decomposition\n", 468 | "import sklearn.pipeline\n", 469 | "\n", 470 | "from sklearn import datasets\n", 471 | "\n", 472 | "import matplotlib.pyplot as plt\n", 473 | "import seaborn as sns\n", 474 | "plt.style.use('seaborn-muted')\n", 475 | "\n", 476 | "import warnings\n", 477 | "warnings.filterwarnings('ignore')\n", 478 | "\n", 479 | "np.random.seed(0)" 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": {}, 485 | "source": [ 486 | "#### Principle Component Analysis " 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "metadata": {}, 493 | "outputs": [], 494 | "source": [ 495 | "# preprocessor\n", 496 | "pca = sklearn.decomposition.PCA(n_components=2)\n", 497 | "\n", 498 | "# fetch data\n", 499 | "iris = datasets.load_iris()\n", 500 | "# supervised-learning data\n", 501 | "X, y = iris.data, iris.target\n", 502 | "# transformed data\n", 503 | "X_transform = pca.fit_transform(X)\n", 504 | "\n", 505 | "_, ax = plt.subplots(figsize=(20.0, 3.0))\n", 506 | "for i in range(iris.target.max() + 1):\n", 507 | " I = iris.target == i\n", 508 | " ax.scatter(X_transform[I, 0], X_transform[I, 1], label=iris.target_names[i])\n", 509 | " ax.set_title('Iris Dataset PCA')\n", 510 | " ax.set_xlabel('Principle Component 1')\n", 511 | " ax.set_ylabel('Principle Component 2')\n", 512 | "ax.legend();" 513 | ] 514 | }, 515 | { 516 | "cell_type": "markdown", 517 | "metadata": {}, 518 | "source": [ 519 | "### Estimation " 520 | ] 521 | }, 522 | { 523 | "cell_type": "markdown", 524 | "metadata": {}, 525 | "source": [ 526 | "\"Drawing\"" 527 | ] 528 | }, 529 | { 530 | "cell_type": "markdown", 531 | "metadata": {}, 532 | "source": [ 533 | "### Problem Statement\n", 534 | "\n", 535 | "Let's try to model a continous function:\n", 536 | "\n", 537 | "$$f(x) = x^{3} - 0.4x^{2} - x + 0.3 + \\epsilon, \\quad x \\in [-1, 1] \\text{ and } \\epsilon \\sim \\mathcal{N}(0, 0.05)$$" 538 | ] 539 | }, 540 | { 541 | "cell_type": "code", 542 | "execution_count": null, 543 | "metadata": {}, 544 | "outputs": [], 545 | "source": [ 546 | "# fetch data\n", 547 | "x = np.linspace(-1, 1, 500)\n", 548 | "# targets\n", 549 | "y = (x**3 - 0.4*x**2 - x + 0.3) + np.random.normal(0, 0.01, len(x))\n", 550 | "# features matrix\n", 551 | "X = x.reshape(-1, 1)" 552 | ] 553 | }, 554 | { 555 | "cell_type": "markdown", 556 | "metadata": {}, 557 | "source": [ 558 | "#### API" 559 | ] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "execution_count": null, 564 | "metadata": {}, 565 | "outputs": [], 566 | "source": [ 567 | "# initialize figure\n", 568 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 569 | "# true data\n", 570 | "ax.plot(x, y, label='Observations', lw=2)\n", 571 | "ax.fill_between(x, y-0.025, y+0.025)\n", 572 | "\n", 573 | "## (1) INIT - set hyperparameters\n", 574 | "estimators = [('Gradient Boosting Tree Regressor', sklearn.ensemble.GradientBoostingRegressor(n_estimators=25)),\n", 575 | " ('Support Vector Machine Regressor', sklearn.svm.SVR(C=1.0, kernel='rbf')),\n", 576 | " ('Ridge Regressor', sklearn.linear_model.Ridge(alpha=0.5)),\n", 577 | " ('K-Nearest Neighbors Regressor', sklearn.neighbors.KNeighborsRegressor(n_neighbors=3)),\n", 578 | " ('Multi-Layer Perceptron Regressor', sklearn.neural_network.MLPRegressor(hidden_layer_sizes=(15,),\n", 579 | " activation='relu'))\n", 580 | " ]\n", 581 | "\n", 582 | "for name, model in estimators:\n", 583 | " ## (2) FIT - train model\n", 584 | " model.fit(X, y)\n", 585 | " ## (3) PREDICT - make predictions\n", 586 | " y_hat = model.predict(X)\n", 587 | " ## (4) SCORE - evaluate model\n", 588 | " score = model.score(X, y)\n", 589 | " \n", 590 | " print('[Score] %s: %.3f' % (name, score))\n", 591 | " # figure settings\n", 592 | " ax.plot(x, y_hat, label=name, lw=3)\n", 593 | "\n", 594 | "ax.set_title('Estimators Comparison Matrix')\n", 595 | "ax.legend();" 596 | ] 597 | }, 598 | { 599 | "cell_type": "markdown", 600 | "metadata": {}, 601 | "source": [ 602 | "#### Support Vector Machine Regressor" 603 | ] 604 | }, 605 | { 606 | "cell_type": "markdown", 607 | "metadata": {}, 608 | "source": [ 609 | "##### Linear Features\n", 610 | "\n", 611 | "$f(x)$ is a highly non-linear function (cubic) and therefore it cannot be adequately modelled by a linear estimator,\n", 612 | "nonetheless we will fit a `Linear SVM Regressor` and ascess its performance (visually)." 613 | ] 614 | }, 615 | { 616 | "cell_type": "code", 617 | "execution_count": null, 618 | "metadata": {}, 619 | "outputs": [], 620 | "source": [ 621 | "# initialize figure\n", 622 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 623 | "# true data\n", 624 | "ax.plot(x, y, lw=3, label='Observations')\n", 625 | "\n", 626 | "## (1) INIT\n", 627 | "svr = sklearn.svm.SVR(C=1.0, kernel='linear')\n", 628 | "## (2) FIT\n", 629 | "svr.fit(X, y)\n", 630 | "## (3) PREDICT\n", 631 | "y_hat = svr.predict(X)\n", 632 | "## (4) SCORE\n", 633 | "score = svr.score(X, y)\n", 634 | "print('[Score] %s: %.3f' % ('Linear SVM', score))\n", 635 | "\n", 636 | "# figure settings\n", 637 | "ax.plot(x, y_hat, label='Model')\n", 638 | "ax.set_title('Linear SVM Regressor')\n", 639 | "ax.legend();" 640 | ] 641 | }, 642 | { 643 | "cell_type": "markdown", 644 | "metadata": {}, 645 | "source": [ 646 | "##### Cubic Features" 647 | ] 648 | }, 649 | { 650 | "cell_type": "code", 651 | "execution_count": null, 652 | "metadata": {}, 653 | "outputs": [], 654 | "source": [ 655 | "# preprocessor\n", 656 | "poly = sklearn.preprocessing.PolynomialFeatures(degree=3)\n", 657 | "# generate cubic features\n", 658 | "X_transform = poly.fit_transform(X)\n", 659 | "\n", 660 | "##### REPEAT THE SAME, X -> X_transform #####\n", 661 | "\n", 662 | "# initialize figure\n", 663 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 664 | "# true data\n", 665 | "ax.plot(x, y, lw=3, label='Observations')\n", 666 | "\n", 667 | "## (1) INIT\n", 668 | "svr = sklearn.svm.SVR(C=1.0, kernel='linear')\n", 669 | "## (2) FIT\n", 670 | "svr.fit(X_transform, y) # X -> X_transform #\n", 671 | "## (3) PREDICT\n", 672 | "y_hat = svr.predict(X_transform)\n", 673 | "## (4) SCORE\n", 674 | "score = svr.score(X_transform, y)\n", 675 | "print('[Score] %s: %.3f' % ('Linear SVM with Cubic Features', score))\n", 676 | "\n", 677 | "# figure settings\n", 678 | "ax.plot(x, y_hat, label='Model')\n", 679 | "ax.set_title('Linear SVM Regressor with Cubic Features')\n", 680 | "ax.legend();" 681 | ] 682 | }, 683 | { 684 | "cell_type": "markdown", 685 | "metadata": {}, 686 | "source": [ 687 | "#### `challenge` AdaBoost Regressor\n", 688 | "\n", 689 | "Model the continuous function using the Boosting Regressor `AdaBoostRegressor`:\n", 690 | "\n", 691 | "$$g(x) = -0.3x^{7} + 7x^{2} + \\epsilon, \\quad x \\in [-0.5, 1] \\text{ and } \\epsilon \\sim \\mathcal{N}(0, 0.5)$$" 692 | ] 693 | }, 694 | { 695 | "cell_type": "code", 696 | "execution_count": null, 697 | "metadata": {}, 698 | "outputs": [], 699 | "source": [ 700 | "# fetch data\n", 701 | "x_a = np.linspace(-0.5, 1, 500)\n", 702 | "# targets\n", 703 | "y_a = (-0.3*x_a**7 + 7*x_a**2) + np.random.normal(0, 0.5, len(x_a))\n", 704 | "# features matrix\n", 705 | "X_a = x_a.reshape(-1, 1)" 706 | ] 707 | }, 708 | { 709 | "cell_type": "code", 710 | "execution_count": null, 711 | "metadata": {}, 712 | "outputs": [], 713 | "source": [ 714 | "# preprocessor\n", 715 | "poly = sklearn.preprocessing.PolynomialFeatures(degree=7)\n", 716 | "# generate cubic features\n", 717 | "X_transform_a = poly.fit_transform(X_a)\n", 718 | "\n", 719 | "# initialize figure\n", 720 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 721 | "# true data\n", 722 | "ax.plot(x_a, y_a, label='Observations')\n", 723 | "\n", 724 | "## (1) INIT\n", 725 | "ada = \n", 726 | "## (2) FIT\n", 727 | "\n", 728 | "## (3) PREDICT\n", 729 | "y_hat = \n", 730 | "## (4) SCORE\n", 731 | "score = \n", 732 | "print('[Score] %s: %.3f' % ('AdaBoostRegressor with 7th Order Polynomial Features', score))\n", 733 | "\n", 734 | "# figure settings\n", 735 | "ax.plot(x_a, y_hat, lw=3, label='Model')\n", 736 | "ax.set_title('AdaBoostRegressor with 7th Order Polynomial Features')\n", 737 | "ax.legend();" 738 | ] 739 | }, 740 | { 741 | "cell_type": "markdown", 742 | "metadata": {}, 743 | "source": [ 744 | "##### Unit Test for `AdaBoostRegressor`" 745 | ] 746 | }, 747 | { 748 | "cell_type": "code", 749 | "execution_count": null, 750 | "metadata": {}, 751 | "outputs": [], 752 | "source": [ 753 | "# unit test function\n", 754 | "assert score > 0.9, 'Wrong implementation, low score!! Try again!'\n", 755 | "'Well Done!'" 756 | ] 757 | }, 758 | { 759 | "cell_type": "markdown", 760 | "metadata": {}, 761 | "source": [ 762 | "### Model Selection " 763 | ] 764 | }, 765 | { 766 | "cell_type": "markdown", 767 | "metadata": {}, 768 | "source": [ 769 | "#### Hyperparameters " 770 | ] 771 | }, 772 | { 773 | "cell_type": "code", 774 | "execution_count": null, 775 | "metadata": {}, 776 | "outputs": [], 777 | "source": [ 778 | "# initialize figure\n", 779 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 780 | "# true data\n", 781 | "ax.plot(x, y, label='original', lw=3)\n", 782 | "\n", 783 | "# parameters\n", 784 | "n_estimators_params = [1, 2, 10, 25]\n", 785 | "\n", 786 | "for n_estimators in n_estimators_params:\n", 787 | " # initialize estimator\n", 788 | " gbr = sklearn.ensemble.GradientBoostingRegressor(n_estimators=n_estimators).fit(X, y)\n", 789 | " # prediction\n", 790 | " y_hat = gbr.predict(X)\n", 791 | " \n", 792 | " # figure settings\n", 793 | " ax.plot(x, y_hat, label='n_estimators=%s' % n_estimators)\n", 794 | " ax.set_title('Gradient Boosting Tree Regressor')\n", 795 | "ax.legend();" 796 | ] 797 | }, 798 | { 799 | "cell_type": "code", 800 | "execution_count": null, 801 | "metadata": {}, 802 | "outputs": [], 803 | "source": [ 804 | "# preprocessor\n", 805 | "poly = sklearn.preprocessing.PolynomialFeatures(degree=3)\n", 806 | "# generate cubic features\n", 807 | "X_transform = poly.fit_transform(X)\n", 808 | "\n", 809 | "# initialize figure\n", 810 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 811 | "# true data\n", 812 | "ax.plot(x, y, label='original', lw=3)\n", 813 | "\n", 814 | "# parameters\n", 815 | "c_params = [1.0, 0.001]\n", 816 | "\n", 817 | "for c in c_params:\n", 818 | " # initialize estimator\n", 819 | " svr = sklearn.svm.SVR(C=c, kernel='linear').fit(X_transform, y)\n", 820 | " # prediction\n", 821 | " y_hat = svr.predict(X_transform)\n", 822 | " \n", 823 | " # figure settings\n", 824 | " ax.plot(x, y_hat, label='c=%s' % c)\n", 825 | " ax.set_title('SVM Regressor')\n", 826 | "ax.legend();" 827 | ] 828 | }, 829 | { 830 | "cell_type": "markdown", 831 | "metadata": {}, 832 | "source": [ 833 | "#### `GridSearchCV` " 834 | ] 835 | }, 836 | { 837 | "cell_type": "code", 838 | "execution_count": null, 839 | "metadata": {}, 840 | "outputs": [], 841 | "source": [ 842 | "# fetch data\n", 843 | "digits = datasets.load_digits()\n", 844 | "X_train_raw, X_test_raw, y_train, y_test = sklearn.model_selection.train_test_split(digits.data,\n", 845 | " digits.target,\n", 846 | " test_size=0.25)\n", 847 | "# use PCA to reduce input dimensionality\n", 848 | "pca = sklearn.decomposition.PCA(n_components=10)\n", 849 | "X_train = pca.fit_transform(X_train_raw)\n", 850 | "X_test = pca.transform(X_test_raw)\n", 851 | "\n", 852 | "# estimator hyperparameters grid\n", 853 | "param_grid = {'n_estimators': [1, 5, 10, 25, 50, 75, 100, 200],\n", 854 | " 'max_depth': [5, 7, 11, 15, 20, 45]\n", 855 | " }\n", 856 | "\n", 857 | "## (1) INIT - set hyperparameters **RANGES**, not single values\n", 858 | "search = sklearn.model_selection.GridSearchCV(sklearn.ensemble.RandomForestClassifier(), param_grid)\n", 859 | "\n", 860 | "## (2) FIT\n", 861 | "search.fit(X_train, y_train)\n", 862 | "\n", 863 | "## (**) REPORT - cross validation results\n", 864 | "results = pd.DataFrame(search.cv_results_).set_index(['param_' + key for key in param_grid.keys()])\n", 865 | "mean_test_score = results['mean_test_score'].unstack(0)\n", 866 | "# figure settings\n", 867 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 868 | "sns.heatmap(mean_test_score, annot=True, cmap=plt.cm.Reds, ax=ax)\n", 869 | "ax.set_title('Cross-Validation Accuracy')\n", 870 | "\n", 871 | "## (**) SELECT - pick the best model\n", 872 | "model = search.best_estimator_\n", 873 | "print('[Select] Best parameters: %s' % search.best_params_)\n", 874 | "\n", 875 | "## (3) PREDICT\n", 876 | "y_hat = model.predict(X_test)\n", 877 | "\n", 878 | "## (4) SCORE\n", 879 | "score = model.score(X_test, y_test)\n", 880 | "print('[Score] %s: %.3f' % ('Best Random Forest Classifier', score))\n", 881 | "\n", 882 | "# confusion matrix\n", 883 | "cm = sklearn.metrics.confusion_matrix(y_test, y_hat)\n", 884 | "\n", 885 | "# figure settings\n", 886 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 887 | "sns.heatmap(cm, ax=ax, annot=True, cmap=plt.cm.Blues)\n", 888 | "ax.set_title('Random Forest Classifier')\n", 889 | "ax.set_xlabel('Predicted Label')\n", 890 | "ax.set_ylabel('True Label');" 891 | ] 892 | }, 893 | { 894 | "cell_type": "markdown", 895 | "metadata": {}, 896 | "source": [ 897 | "### Pipeline " 898 | ] 899 | }, 900 | { 901 | "cell_type": "markdown", 902 | "metadata": {}, 903 | "source": [ 904 | "`sklearn.pipeline.Pipeline` is a container that put all the pieces:\n", 905 | "\n", 906 | "1. Preprocessing\n", 907 | "1. Estimation\n", 908 | "1. Model Selection\n", 909 | "\n", 910 | "together, using the common `fit`-`predict`-`score` API" 911 | ] 912 | }, 913 | { 914 | "cell_type": "code", 915 | "execution_count": null, 916 | "metadata": {}, 917 | "outputs": [], 918 | "source": [ 919 | "# fetch data\n", 920 | "digits = datasets.load_digits()\n", 921 | "X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(digits.data,\n", 922 | " digits.target,\n", 923 | " test_size=0.25)\n", 924 | "\n", 925 | "# data flow\n", 926 | "steps = [('pca', sklearn.decomposition.PCA(n_components=2)),\n", 927 | " ('rf', sklearn.ensemble.RandomForestClassifier())]\n", 928 | "\n", 929 | "# estimator hyperparameters grid\n", 930 | "# prepend the name of the step that the parameter corresponds to\n", 931 | "# (i.e 'pca' or 'rf') followed by two underscores '__'\n", 932 | "param_grid = {'pca__n_components': [5, 10, 20, 35, 45, 64],\n", 933 | " 'rf__n_estimators': [1, 25, 50, 75, 100, 200],\n", 934 | " 'rf__max_depth': [5, 11, 15, 20, 45]\n", 935 | " }\n", 936 | "\n", 937 | "## (1) INIT\n", 938 | "# pipeline\n", 939 | "pipe = sklearn.pipeline.Pipeline(steps=steps)\n", 940 | "# grid-search\n", 941 | "search = sklearn.model_selection.GridSearchCV(pipe, param_grid)\n", 942 | "\n", 943 | "## (2) FIT\n", 944 | "search.fit(X_train, y_train)\n", 945 | "\n", 946 | "## (**) SELECT - pick the best model\n", 947 | "model = search.best_estimator_\n", 948 | "print('[Select] Best parameters: %s' % search.best_params_)\n", 949 | "\n", 950 | "## (3) PREDICT\n", 951 | "y_hat = model.predict(X_test)\n", 952 | "\n", 953 | "## (4) SCORE\n", 954 | "score = model.score(X_test, y_test)\n", 955 | "print('[Score] %s: %.3f' % ('Best Random Forest Classifier', score))\n", 956 | "\n", 957 | "# confusion matrix\n", 958 | "cm = sklearn.metrics.confusion_matrix(y_test, y_hat)\n", 959 | "\n", 960 | "# figure settings\n", 961 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 962 | "sns.heatmap(cm, ax=ax, annot=True, cmap=plt.cm.Blues)\n", 963 | "ax.set_title('Pipeline of PCA and Random Forest Classifier')\n", 964 | "ax.set_xlabel('Predicted Label')\n", 965 | "ax.set_ylabel('True Label');" 966 | ] 967 | }, 968 | { 969 | "cell_type": "markdown", 970 | "metadata": {}, 971 | "source": [ 972 | "#### `challenge` Naive [`tpot`](https://github.com/EpistasisLab/tpot) Framework\n", 973 | "\n", 974 | "Code a `pipeline` that optimizes both the estimator type and its hyperparameters on the raw digits dataset." 975 | ] 976 | }, 977 | { 978 | "cell_type": "markdown", 979 | "metadata": {}, 980 | "source": [ 981 | "## `keras` " 982 | ] 983 | }, 984 | { 985 | "cell_type": "markdown", 986 | "metadata": {}, 987 | "source": [ 988 | "### Iris Dataset " 989 | ] 990 | }, 991 | { 992 | "cell_type": "code", 993 | "execution_count": null, 994 | "metadata": {}, 995 | "outputs": [], 996 | "source": [ 997 | "%reset -f\n", 998 | "\n", 999 | "from keras.models import Model\n", 1000 | "from keras.layers import Input\n", 1001 | "from keras.layers import Dense\n", 1002 | "from keras.layers import Activation\n", 1003 | "from keras.optimizers import RMSprop\n", 1004 | "from keras.utils import to_categorical\n", 1005 | "\n", 1006 | "import numpy as np\n", 1007 | "import matplotlib.pyplot as plt\n", 1008 | "import seaborn as sns\n", 1009 | "\n", 1010 | "from sklearn import datasets\n", 1011 | "from sklearn.model_selection import train_test_split\n", 1012 | "from sklearn.metrics import confusion_matrix" 1013 | ] 1014 | }, 1015 | { 1016 | "cell_type": "markdown", 1017 | "metadata": {}, 1018 | "source": [ 1019 | "#### Data & Preprocessing" 1020 | ] 1021 | }, 1022 | { 1023 | "cell_type": "code", 1024 | "execution_count": null, 1025 | "metadata": {}, 1026 | "outputs": [], 1027 | "source": [ 1028 | "# fetch data\n", 1029 | "iris = datasets.load_iris()\n", 1030 | "\n", 1031 | "# split to train/test datasets\n", 1032 | "X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target)\n", 1033 | "# one-hot encode categorical output\n", 1034 | "y_train_one_hot = to_categorical(y_train)\n", 1035 | "y_test_one_hot = to_categorical(y_test)\n", 1036 | "\n", 1037 | "# matrix shapes\n", 1038 | "N_train, D = X_train.shape\n", 1039 | "N_test, _ = X_test.shape\n", 1040 | "_, M = y_train_one_hot.shape" 1041 | ] 1042 | }, 1043 | { 1044 | "cell_type": "markdown", 1045 | "metadata": {}, 1046 | "source": [ 1047 | "#### Feedforward Neural Network" 1048 | ] 1049 | }, 1050 | { 1051 | "cell_type": "code", 1052 | "execution_count": null, 1053 | "metadata": {}, 1054 | "outputs": [], 1055 | "source": [ 1056 | "# input layer\n", 1057 | "X = Input(shape=(D,), name=\"X\")\n", 1058 | "\n", 1059 | "# Convolution Layer\n", 1060 | "A1 = Dense(16, name=\"A1\")(X)\n", 1061 | "# Non-Linearity\n", 1062 | "Z1 = Activation(\"relu\", name=\"Z1\")(A1)\n", 1063 | "# Affine Layer\n", 1064 | "A2 = Dense(M, name=\"A2\")(Z1)\n", 1065 | "# Multi-Class Classification\n", 1066 | "Y = Activation(\"softmax\", name=\"Y\")(A2)\n", 1067 | "\n", 1068 | "# Define Graph\n", 1069 | "model = Model(inputs=X, outputs=Y)\n", 1070 | "# Compile Graph\n", 1071 | "model.compile(optimizer=RMSprop(lr=0.04),\n", 1072 | " loss='categorical_crossentropy',\n", 1073 | " metrics=['accuracy'])\n", 1074 | "# Computational Graph Summary\n", 1075 | "model.summary()" 1076 | ] 1077 | }, 1078 | { 1079 | "cell_type": "code", 1080 | "execution_count": null, 1081 | "metadata": {}, 1082 | "outputs": [], 1083 | "source": [ 1084 | "# model training\n", 1085 | "history = model.fit(X_train, y_train_one_hot, epochs=100, validation_split=0.25, verbose=0)" 1086 | ] 1087 | }, 1088 | { 1089 | "cell_type": "markdown", 1090 | "metadata": {}, 1091 | "source": [ 1092 | "#### Evaluation" 1093 | ] 1094 | }, 1095 | { 1096 | "cell_type": "code", 1097 | "execution_count": null, 1098 | "metadata": {}, 1099 | "outputs": [], 1100 | "source": [ 1101 | "y_hat_one_hot = model.predict(X_test)\n", 1102 | "\n", 1103 | "# one-hot-encoded to raw data\n", 1104 | "y_hat = np.argmax(y_hat_one_hot, axis=1)\n", 1105 | "\n", 1106 | "# confusion matrix\n", 1107 | "cm = confusion_matrix(y_test, y_hat)\n", 1108 | "cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", 1109 | "_, ax = plt.subplots(figsize=(7.0, 6.0))\n", 1110 | "sns.heatmap(cm_norm, annot=True, cmap=plt.cm.Blues, ax=ax)\n", 1111 | "ax.set_xticklabels(iris.target_names, rotation=45)\n", 1112 | "ax.set_yticklabels(iris.target_names, rotation=45)\n", 1113 | "ax.set_title('Iris Dataset Confusion Matrix')\n", 1114 | "ax.set_xlabel('Predicted Class')\n", 1115 | "ax.set_ylabel('True Class');" 1116 | ] 1117 | }, 1118 | { 1119 | "cell_type": "markdown", 1120 | "metadata": {}, 1121 | "source": [ 1122 | "### Fashion MNIST " 1123 | ] 1124 | }, 1125 | { 1126 | "cell_type": "code", 1127 | "execution_count": null, 1128 | "metadata": {}, 1129 | "outputs": [], 1130 | "source": [ 1131 | "%reset -f\n", 1132 | "\n", 1133 | "from keras.models import Model\n", 1134 | "from keras.layers import Input\n", 1135 | "from keras.layers import Dense\n", 1136 | "from keras.layers import Conv2D\n", 1137 | "from keras.layers import Dropout\n", 1138 | "from keras.layers import MaxPooling2D\n", 1139 | "from keras.layers import Flatten\n", 1140 | "from keras.optimizers import RMSprop\n", 1141 | "from keras.utils import to_categorical\n", 1142 | "\n", 1143 | "from keras.datasets import fashion_mnist\n", 1144 | "\n", 1145 | "import numpy as np\n", 1146 | "import matplotlib.pyplot as plt\n", 1147 | "import seaborn as sns\n", 1148 | "\n", 1149 | "from sklearn.metrics import confusion_matrix\n", 1150 | "\n", 1151 | "np.random.seed(0)" 1152 | ] 1153 | }, 1154 | { 1155 | "cell_type": "markdown", 1156 | "metadata": {}, 1157 | "source": [ 1158 | "#### Data & Preprocessing" 1159 | ] 1160 | }, 1161 | { 1162 | "cell_type": "code", 1163 | "execution_count": null, 1164 | "metadata": {}, 1165 | "outputs": [], 1166 | "source": [ 1167 | "import pickle\n", 1168 | "\n", 1169 | "# fetch data\n", 1170 | "(X_train_raw, y_train), (X_test_raw, y_test) = pickle.load(open('data/fashion-mnist/fashion-mnist.pkl', 'rb'))\n", 1171 | "## ORIGINAL IMPLEMNTATION\n", 1172 | "# (X_train_raw, y_train), (X_test_raw, y_test) = fashion_mnist.load_data()\n", 1173 | "\n", 1174 | "# one-hot encode categorical output\n", 1175 | "y_train_one_hot = to_categorical(y_train)\n", 1176 | "y_test_one_hot = to_categorical(y_test)\n", 1177 | "\n", 1178 | "# tensor shape\n", 1179 | "N_train, h, w = X_train_raw.shape\n", 1180 | "N_test, _, _ = X_test_raw.shape\n", 1181 | "_, M = y_train_one_hot.shape\n", 1182 | "\n", 1183 | "# convert raw pixels to tensors\n", 1184 | "X_train = X_train_raw.reshape(-1, h, w, 1)\n", 1185 | "X_test = X_test_raw.reshape(-1, h, w, 1)" 1186 | ] 1187 | }, 1188 | { 1189 | "cell_type": "markdown", 1190 | "metadata": {}, 1191 | "source": [ 1192 | "#### Convolutional Neural Network" 1193 | ] 1194 | }, 1195 | { 1196 | "cell_type": "code", 1197 | "execution_count": null, 1198 | "metadata": {}, 1199 | "outputs": [], 1200 | "source": [ 1201 | "# input layer: shape=(height, width, number of channels)\n", 1202 | "X = Input(shape=(h, w, 1), name=\"X\")\n", 1203 | "\n", 1204 | "# Convolution Layer\n", 1205 | "CONV1 = Conv2D(filters=32, kernel_size=(3, 3), activation=\"relu\", name=\"CONV1\")(X)\n", 1206 | "# Max Pooling Layer\n", 1207 | "POOL1 = MaxPooling2D(pool_size=(2, 2), name=\"POOL1\")(CONV1)\n", 1208 | "# Convolution Layer\n", 1209 | "CONV2 = Conv2D(filters=32, kernel_size=(3, 3), activation=\"relu\", name=\"CONV2\")(POOL1)\n", 1210 | "# Max Pooling Layer\n", 1211 | "POOL2 = MaxPooling2D(pool_size=(2, 2), name=\"POOL2\")(CONV2)\n", 1212 | "# Convolution Layer\n", 1213 | "CONV3 = Conv2D(filters=64, kernel_size=(3, 3), activation=\"relu\", name=\"CONV3\")(POOL2)\n", 1214 | "# Max Pooling Layer\n", 1215 | "POOL3 = MaxPooling2D(pool_size=(2, 2), name=\"POOL3\")(CONV3)\n", 1216 | "# Convert 3D feature map to 1D\n", 1217 | "FLAT = Flatten()(POOL3)\n", 1218 | "# Fully Connected Layer\n", 1219 | "FC1 = Dense(units=64, name=\"FC1\")(FLAT)\n", 1220 | "# Dropout\n", 1221 | "DROP = Dropout(rate=0.5, name=\"DROP\")(FC1)\n", 1222 | "# Multi-Class Classification Output Layer\n", 1223 | "Y = Dense(M, activation=\"softmax\", name=\"Y\")(DROP)\n", 1224 | "\n", 1225 | "# Define Graph\n", 1226 | "model = Model(inputs=X, outputs=Y)\n", 1227 | "# Compile Graph\n", 1228 | "model.compile(optimizer=RMSprop(),\n", 1229 | " loss='categorical_crossentropy',\n", 1230 | " metrics=['accuracy'])\n", 1231 | "# Computational Graph Summary\n", 1232 | "model.summary()" 1233 | ] 1234 | }, 1235 | { 1236 | "cell_type": "code", 1237 | "execution_count": null, 1238 | "metadata": {}, 1239 | "outputs": [], 1240 | "source": [ 1241 | "# model training\n", 1242 | "history = model.fit(X_train, y_train_one_hot, batch_size=128, epochs=3, validation_split=0.25)" 1243 | ] 1244 | }, 1245 | { 1246 | "cell_type": "markdown", 1247 | "metadata": {}, 1248 | "source": [ 1249 | "#### Evaluation" 1250 | ] 1251 | }, 1252 | { 1253 | "cell_type": "code", 1254 | "execution_count": null, 1255 | "metadata": {}, 1256 | "outputs": [], 1257 | "source": [ 1258 | "y_hat_one_hot = model.predict(X_test)\n", 1259 | "\n", 1260 | "# one-hot-encoded to raw data\n", 1261 | "y_hat = np.argmax(y_hat_one_hot, axis=1)\n", 1262 | "\n", 1263 | "# confusion matrix\n", 1264 | "cm = confusion_matrix(y_test, y_hat)\n", 1265 | "cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", 1266 | "_, ax = plt.subplots(figsize=(20.0, 6.0))\n", 1267 | "sns.heatmap(cm_norm, annot=True, cmap=plt.cm.Greens, ax=ax)\n", 1268 | "ax.set_title('Fashion MNIST Dataset Confusion Matrix')\n", 1269 | "ax.set_xlabel('Predicted Class')\n", 1270 | "ax.set_ylabel('True Class');" 1271 | ] 1272 | }, 1273 | { 1274 | "cell_type": "markdown", 1275 | "metadata": {}, 1276 | "source": [ 1277 | "### President Trump Generator " 1278 | ] 1279 | }, 1280 | { 1281 | "cell_type": "code", 1282 | "execution_count": null, 1283 | "metadata": {}, 1284 | "outputs": [], 1285 | "source": [ 1286 | "%reset -f\n", 1287 | "\n", 1288 | "from keras.models import Sequential\n", 1289 | "from keras.layers import Dense\n", 1290 | "from keras.layers import LSTM\n", 1291 | "from keras.optimizers import Adam\n", 1292 | "from keras.callbacks import ModelCheckpoint\n", 1293 | "\n", 1294 | "from keras.datasets import fashion_mnist\n", 1295 | "\n", 1296 | "import numpy as np\n", 1297 | "\n", 1298 | "np.random.seed(0)" 1299 | ] 1300 | }, 1301 | { 1302 | "cell_type": "markdown", 1303 | "metadata": {}, 1304 | "source": [ 1305 | "#### Data & Preprocessing" 1306 | ] 1307 | }, 1308 | { 1309 | "cell_type": "code", 1310 | "execution_count": null, 1311 | "metadata": {}, 1312 | "outputs": [], 1313 | "source": [ 1314 | "# fetch data\n", 1315 | "with open(\"data/trump/speeches.txt\") as corpus_file:\n", 1316 | " corpus = corpus_file.read()\n", 1317 | "print(\"Loaded a corpus of {0} characters\".format(len(corpus)))\n", 1318 | "corpus_length = len(corpus)\n", 1319 | "\n", 1320 | "# Get a unique identifier for each char in the corpus\n", 1321 | "# then make some dicts to ease encoding and decoding\n", 1322 | "chars = sorted(list(set(corpus)))\n", 1323 | "num_chars = len(chars)\n", 1324 | "encoding = {c: i for i, c in enumerate(chars)}\n", 1325 | "decoding = {i: c for i, c in enumerate(chars)}\n", 1326 | "print(\"Our corpus contains {0} unique characters.\".format(num_chars))\n", 1327 | "\n", 1328 | "# it slices, it dices, it makes julienned datasets!\n", 1329 | "# chop up our data into X and y, slice into roughly (num_chars / skip) overlapping 'sentences'\n", 1330 | "# of length sentence_length, and encode the chars\n", 1331 | "sentence_length = 50\n", 1332 | "skip = 1\n", 1333 | "X_data = []\n", 1334 | "y_data = []\n", 1335 | "for i in range (0, len(corpus) - sentence_length, skip):\n", 1336 | " sentence = corpus[i:i + sentence_length]\n", 1337 | " next_char = corpus[i + sentence_length]\n", 1338 | " X_data.append([encoding[char] for char in sentence])\n", 1339 | " y_data.append(encoding[next_char])\n", 1340 | "\n", 1341 | "num_sentences = len(X_data)\n", 1342 | "print(\"Sliced our corpus into {0} sentences of length {1}\".format(num_sentences, sentence_length))\n", 1343 | "\n", 1344 | "# Vectorize our data and labels. We want everything in one-hot\n", 1345 | "# because smart data encoding cultivates phronesis and virtue.\n", 1346 | "print(\"Vectorizing X and y...\")\n", 1347 | "X = np.zeros((num_sentences, sentence_length, num_chars), dtype=np.bool)\n", 1348 | "y = np.zeros((num_sentences, num_chars), dtype=np.bool)\n", 1349 | "for i, sentence in enumerate(X_data):\n", 1350 | " for t, encoded_char in enumerate(sentence):\n", 1351 | " X[i, t, encoded_char] = 1\n", 1352 | " y[i, y_data[i]] = 1\n", 1353 | "\n", 1354 | "# Double check our vectorized data before we sink hours into fitting a model\n", 1355 | "print(\"Sanity check y. Dimension: {0} # Sentences: {1} Characters in corpus: {2}\".format(y.shape, num_sentences, len(chars)))\n", 1356 | "print(\"Sanity check X. Dimension: {0} Sentence length: {1}\".format(X.shape, sentence_length))" 1357 | ] 1358 | }, 1359 | { 1360 | "cell_type": "markdown", 1361 | "metadata": {}, 1362 | "source": [ 1363 | "#### Recurrent Neural Network" 1364 | ] 1365 | }, 1366 | { 1367 | "cell_type": "code", 1368 | "execution_count": null, 1369 | "metadata": {}, 1370 | "outputs": [], 1371 | "source": [ 1372 | "model = Sequential()\n", 1373 | "model.add(LSTM(units=256, input_shape=(sentence_length, num_chars), name=\"LSTM\"))\n", 1374 | "model.add(Dense(units=num_chars, activation=\"softmax\", name=\"Y\"))\n", 1375 | "\n", 1376 | "# Compile Graph\n", 1377 | "model.compile(optimizer=Adam(),\n", 1378 | " loss='categorical_crossentropy')\n", 1379 | "# Computational Graph Summary\n", 1380 | "model.summary()" 1381 | ] 1382 | }, 1383 | { 1384 | "cell_type": "code", 1385 | "execution_count": null, 1386 | "metadata": {}, 1387 | "outputs": [], 1388 | "source": [ 1389 | "# remove this line to train the model again\n", 1390 | "if None:\n", 1391 | " # Dump our model architecture to a file so we can load it elsewhere\n", 1392 | " architecture = model.to_yaml()\n", 1393 | " with open('models/trump/architecture.yaml', 'a') as model_file:\n", 1394 | " model_file.write(architecture)\n", 1395 | "\n", 1396 | " # Set up checkpoints\n", 1397 | " file_path=\"models/trump/weights-{epoch:02d}-{loss:.3f}.hdf5\"\n", 1398 | " checkpoint = ModelCheckpoint(file_path, monitor=\"loss\", verbose=1, save_best_only=True, mode=\"min\")\n", 1399 | " callbacks = [checkpoint]\n", 1400 | "\n", 1401 | " # model training\n", 1402 | " history = model.fit(X, y, epochs=30, batch_size=256, callbacks=callbacks)\n", 1403 | "else:\n", 1404 | " # load weights from checkpoint\n", 1405 | " model.load_weights(\"models/trump/weights-03-2.152.hdf5\")" 1406 | ] 1407 | }, 1408 | { 1409 | "cell_type": "markdown", 1410 | "metadata": {}, 1411 | "source": [ 1412 | "#### Helper Functions" 1413 | ] 1414 | }, 1415 | { 1416 | "cell_type": "code", 1417 | "execution_count": null, 1418 | "metadata": {}, 1419 | "outputs": [], 1420 | "source": [ 1421 | "def generate(seed_pattern):\n", 1422 | " X = np.zeros((1, sentence_length, num_chars), dtype=np.bool)\n", 1423 | " for i, character in enumerate(seed_pattern):\n", 1424 | " X[0, i, encoding[character]] = 1\n", 1425 | "\n", 1426 | " generated_text = \"\"\n", 1427 | " for i in range(10):\n", 1428 | " prediction = np.argmax(model.predict(X, verbose=0))\n", 1429 | "\n", 1430 | " generated_text += decoding[prediction]\n", 1431 | "\n", 1432 | " activations = np.zeros((1, 1, num_chars), dtype=np.bool)\n", 1433 | " activations[0, 0, prediction] = 1\n", 1434 | " X = np.concatenate((X[:, 1:, :], activations), axis=1)\n", 1435 | "\n", 1436 | " return generated_text\n", 1437 | "\n", 1438 | "def make_seed(seed_phrase=\"\"):\n", 1439 | " if seed_phrase:\n", 1440 | " phrase_length = len(seed_phrase)\n", 1441 | " pattern = \"\"\n", 1442 | " for i in range (0, sentence_length):\n", 1443 | " pattern += seed_phrase[i % phrase_length]\n", 1444 | " else:\n", 1445 | " seed = randint(0, corpus_length - sentence_length)\n", 1446 | " pattern = corpus[seed:seed + sentence_length]\n", 1447 | "\n", 1448 | " return pattern" 1449 | ] 1450 | }, 1451 | { 1452 | "cell_type": "markdown", 1453 | "metadata": {}, 1454 | "source": [ 1455 | "#### Evaluation" 1456 | ] 1457 | }, 1458 | { 1459 | "cell_type": "code", 1460 | "execution_count": null, 1461 | "metadata": {}, 1462 | "outputs": [], 1463 | "source": [ 1464 | "# seed letter\n", 1465 | "seed = \"b\"\n", 1466 | "# prediction\n", 1467 | "seed + generate(make_seed(seed))" 1468 | ] 1469 | }, 1470 | { 1471 | "cell_type": "markdown", 1472 | "metadata": {}, 1473 | "source": [ 1474 | "## `fbprophet` " 1475 | ] 1476 | }, 1477 | { 1478 | "cell_type": "code", 1479 | "execution_count": null, 1480 | "metadata": {}, 1481 | "outputs": [], 1482 | "source": [ 1483 | "%reset -f\n", 1484 | "\n", 1485 | "from fbprophet import Prophet\n", 1486 | "\n", 1487 | "import pandas_datareader.data as web\n", 1488 | "\n", 1489 | "import numpy as np\n", 1490 | "import pandas as pd\n", 1491 | "import matplotlib.pyplot as plt\n", 1492 | "import seaborn as sns\n", 1493 | "\n", 1494 | "np.random.seed(0)" 1495 | ] 1496 | }, 1497 | { 1498 | "cell_type": "markdown", 1499 | "metadata": {}, 1500 | "source": [ 1501 | "#### Data & Preprocessing" 1502 | ] 1503 | }, 1504 | { 1505 | "cell_type": "code", 1506 | "execution_count": null, 1507 | "metadata": {}, 1508 | "outputs": [], 1509 | "source": [ 1510 | "import pickle\n", 1511 | "\n", 1512 | "# fetch data\n", 1513 | "btc = pickle.load(open('data/bitcoin/btc.pkl', 'rb'))\n", 1514 | "## ORIGINAL IMPLEMNTATION\n", 1515 | "# btc = web.DataReader(\"BCHARTS/KRAKENUSD\", data_source=\"quandl\").dropna()\n", 1516 | "\n", 1517 | "# time-series DataFrame\n", 1518 | "df = pd.DataFrame(columns=[\"ds\", \"y\"])\n", 1519 | "df[\"ds\"] = btc.index\n", 1520 | "# market capital column\n", 1521 | "df[\"y\"] = (btc[\"Close\"] * btc[\"VolumeBTC\"]).values\n", 1522 | "\n", 1523 | "df.head()" 1524 | ] 1525 | }, 1526 | { 1527 | "cell_type": "markdown", 1528 | "metadata": {}, 1529 | "source": [ 1530 | "#### MAP Optimization and Hamiltonian Monte Carlo Inference" 1531 | ] 1532 | }, 1533 | { 1534 | "cell_type": "code", 1535 | "execution_count": null, 1536 | "metadata": {}, 1537 | "outputs": [], 1538 | "source": [ 1539 | "# define model\n", 1540 | "model = Prophet(daily_seasonality=True)\n", 1541 | "\n", 1542 | "# train model\n", 1543 | "model.fit(df);" 1544 | ] 1545 | }, 1546 | { 1547 | "cell_type": "markdown", 1548 | "metadata": {}, 1549 | "source": [ 1550 | "#### Evaluation" 1551 | ] 1552 | }, 1553 | { 1554 | "cell_type": "code", 1555 | "execution_count": null, 1556 | "metadata": {}, 1557 | "outputs": [], 1558 | "source": [ 1559 | "\n", 1560 | "future = model.make_future_dataframe(periods=365)\n", 1561 | "forecast = model.predict(future)\n", 1562 | "\n", 1563 | "# generate plots\n", 1564 | "model.plot(forecast);\n", 1565 | "model.plot_components(forecast);" 1566 | ] 1567 | }, 1568 | { 1569 | "cell_type": "markdown", 1570 | "metadata": {}, 1571 | "source": [ 1572 | "## Disclaimer\n", 1573 | "\n", 1574 | "Presentations are intended for educational purposes only and do not replace independent professional judgment.\n", 1575 | "Statements of fact and opinions expressed are those of the participants individually and,\n", 1576 | "unless expressly stated to the contrary, are not the opinion or position of the ICDSS, its cosponsors, or its committees.\n", 1577 | "The ICDSS does not endorse or approve, and assumes no responsibility for, the content,\n", 1578 | "accuracy or completeness of the information presented.\n", 1579 | "Attendees should note that sessions are video-recorded and may be published in various media, including print,\n", 1580 | "audio and video formats without further notice." 1581 | ] 1582 | } 1583 | ], 1584 | "metadata": { 1585 | "kernelspec": { 1586 | "display_name": "Python 3", 1587 | "language": "python", 1588 | "name": "python3" 1589 | }, 1590 | "language_info": { 1591 | "codemirror_mode": { 1592 | "name": "ipython", 1593 | "version": 3 1594 | }, 1595 | "file_extension": ".py", 1596 | "mimetype": "text/x-python", 1597 | "name": "python", 1598 | "nbconvert_exporter": "python", 1599 | "pygments_lexer": "ipython3", 1600 | "version": "3.6.4" 1601 | } 1602 | }, 1603 | "nbformat": 4, 1604 | "nbformat_minor": 2 1605 | } 1606 | -------------------------------------------------------------------------------- /notebooks/assets/Pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/assets/Pipeline.png -------------------------------------------------------------------------------- /notebooks/assets/Pipeline.sketch: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/assets/Pipeline.sketch -------------------------------------------------------------------------------- /notebooks/assets/ml_map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/assets/ml_map.png -------------------------------------------------------------------------------- /notebooks/data/bitcoin/btc.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/data/bitcoin/btc.pkl -------------------------------------------------------------------------------- /notebooks/data/fashion-mnist/fashion-mnist.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/data/fashion-mnist/fashion-mnist.pkl -------------------------------------------------------------------------------- /notebooks/data/trump/speeches.txt: -------------------------------------------------------------------------------- 1 | Chief Justice Roberts, President Carter, President Clinton, President Bush, President Obama, fellow Americans and people of the world, thank you.We, the citizens of America, are now joined in a great national effort to rebuild our country and restore its promise for all of our people.Together, we will determine the course of America and the world for many, many years to come. We will face challenges, we will confront hardships, but we will get the job done.Every four years, we gather on these steps to carry out the orderly and peaceful transfer of power, and we are grateful to President Obama and First Lady Michelle Obama for their gracious aid throughout this transition. They have been magnificent. Thank you.Today’s ceremony, however, has very special meaning because today, we are not merely transferring power from one administration to another or from one party to another, but we are transferring power from Washington, D.C., and giving it back to you, the people.For too long, a small group in our nation’s capital has reaped the rewards of government while the people have borne the cost. Washington flourished, but the people did not share in its wealth. Politicians prospered, but the jobs left and the factories closed. The establishment protected itself, but not the citizens of our country. Their victories have not been your victories. Their triumphs have not been your triumphs. And while they celebrated in our nation’s capital, there was little to celebrate for struggling families all across our land.That all changes starting right here and right now because this moment is your moment, it belongs to you.It belongs to everyone gathered here today and everyone watching all across America. This is your day. This is your celebration. And this, the United States of America, is your country.What truly matters is not which party controls our government, but whether our government is controlled by the people.January 20th, 2017, will be remembered as the day the people became the rulers of this nation again.The forgotten men and women of our country will be forgotten no longer.Everyone is listening to you now. You came by the tens of millions to become part of a historic movement, the likes of which the world has never seen before.At the center of this movement is a crucial conviction, that a nation exists to serve its citizens. Americans want great schools for their children, safe neighborhoods for their families, and good jobs for themselves. These are just and reasonable demands of righteous people and a righteous public.But for too many of our citizens, a different reality exists: mothers and children trapped in poverty in our inner cities; rusted-out factories scattered like tombstones across the landscape of our nation; an education system flush with cash, but which leaves our young and beautiful students deprived of all knowledge; and the crime and the gangs and the drugs that have stolen too many lives and robbed our country of so much unrealized potential.This American carnage stops right here and stops right now.We are one nation and their pain is our pain. Their dreams are our dreams. And their success will be our success. We share one heart, one home, and one glorious destiny. The oath of office I take today is an oath of allegiance to all Americans.For many decades, we’ve enriched foreign industry at the expense of American industry; subsidized the armies of other countries, while allowing for the very sad depletion of our military. We’ve defended other nations’ borders while refusing to defend our own.And spent trillions and trillions of dollars overseas while America’s infrastructure has fallen into disrepair and decay. We’ve made other countries rich, while the wealth, strength and confidence of our country has dissipated over the horizon.One by one, the factories shuttered and left our shores, with not even a thought about the millions and millions of American workers that were left behind. The wealth of our middle class has been ripped from their homes and then redistributed all across the world.But that is the past. And now, we are looking only to the future.We assembled here today are issuing a new decree to be heard in every city, in every foreign capital, and in every hall of power. From this day forward, a new vision will govern our land. From this day forward, it’s going to be only America first, America first.Every decision on trade, on taxes, on immigration, on foreign affairs will be made to benefit American workers and American families. We must protect our borders from the ravages of other countries making our products, stealing our companies and destroying our jobs.Protection will lead to great prosperity and strength. I will fight for you with every breath in my body and I will never ever let you down.America will start winning again, winning like never before.We will bring back our jobs. We will bring back our borders. We will bring back our wealth. And we will bring back our dreams.We will build new roads and highways and bridges and airports and tunnels and railways all across our wonderful nation. We will get our people off of welfare and back to work, rebuilding our country with American hands and American labor.We will follow two simple rules; buy American and hire American.We will seek friendship and goodwill with the nations of the world, but we do so with the understanding that it is the right of all nations to put their own interests first. We do not seek to impose our way of life on anyone, but rather to let it shine as an example. We will shine for everyone to follow.We will reinforce old alliances and form new ones and unite the civilized world against radical Islamic terrorism, which we will eradicate from the face of the Earth.At the bedrock of our politics will be a total allegiance to the United States of America, and through our loyalty to our country, we will rediscover our loyalty to each other. When you open your heart to patriotism, there is no room for prejudice.The bible tells us how good and pleasant it is when God’s people live together in unity. We must speak our minds openly, debate our disagreements honestly, but always pursue solidarity. When America is united, America is totally unstoppable.There should be no fear. We are protected and we will always be protected. We will be protected by the great men and women of our military and law enforcement. And most importantly, we will be protected by God.Finally, we must think big and dream even bigger. In America, we understand that a nation is only living as long as it is striving. We will no longer accept politicians who are all talk and no action, constantly complaining, but never doing anything about it.The time for empty talk is over. Now arrives the hour of action.Do not allow anyone to tell you that it cannot be done. No challenge can match the heart and fight and spirit of America. We will not fail. Our country will thrive and prosper again.We stand at the birth of a new millennium, ready to unlock the mysteries of space, to free the earth from the miseries of disease, and to harness the energies, industries and technologies of tomorrow. A new national pride will stir ourselves, lift our sights and heal our divisions.It’s time to remember that old wisdom our soldiers will never forget, that whether we are black or brown or white, we all bleed the same red blood of patriots.We all enjoy the same glorious freedoms and we all salute the same great American flag.And whether a child is born in the urban sprawl of Detroit or the wind-swept plains of Nebraska, they look up at the same night sky, they will their heart with the same dreams, and they are infused with the breath of life by the same almighty creator.So to all Americans in every city near and far, small and large, from mountain to mountain, from ocean to ocean, hear these words. You will never be ignored again.Your voice, your hopes, and your dreams will define our American destiny. And your courage and goodness and love will forever guide us along the way.Together, we will make America strong again. We will make America wealthy again. We will make America proud again. We will make America safe again. And yes, together we will make America great again.Thank you. God bless you. And God bless America.Thank you. God bless America. 2 | 3 | Wow. Thank you. That's a lot of people, Phoenix, that's a lot of people.Thank you very much.Thank you, Phoenix. I am so glad to be back in Arizona.The state that has a very, very special place in my heart. I love people of Arizona and together we are going to win the White House in November.Now, you know this is where it all began for me. Remember that massive crowd also. So, I said let's go and have some fun tonight. We're going to Arizona, OK?This will be a little bit different. This won't be a rally speech, per se. Instead, I'm going to deliver a detailed policy address on one of the greatest challenges facing our country today, illegal immigration.I've just landed having returned from a very important and special meeting with the President of Mexico, a man I like and respect very much. And a man who truly loves his country, Mexico.And, by the way, just like I am a man who loves my country, the United States.We agree on the importance of ending the illegal flow of drugs, cash, guns, and people across our border, and to put the cartels out of business.We also discussed the great contributions of Mexican-American citizens to our two countries, my love for the people of Mexico, and the leadership and friendship between Mexico and the United States. It was a thoughtful and substantive conversation and it will go on for awhile. And, in the end we're all going to win. Both countries, we're all going to win.This is the first of what I expect will be many, many conversations. And, in a Trump administration we're going to go about creating a new relationship between our two countries, but it's going to be a fair relationship. We want fairness.But to fix our immigration system, we must change our leadership in Washington and we must change it quickly.Sadly, sadly there is no other way. The truth is our immigration system is worse than anybody ever realized. But the facts aren't known because the media won't report on them. The politicians won't talk about them and the special interests spend a lot of money trying to cover them up because they are making an absolute fortune. That's the way it is.Today, on a very complicated and very difficult subject, you will get the truth. The fundamental problem with the immigration system in our country is that it serves the needs of wealthy donors, political activists and powerful, powerful politicians. It's all you can do. Thank you. Thank you.Let me tell you who it does not serve. It does not serve you the American people. Doesn't serve you. When politicians talk about immigration reform, they usually mean the following, amnesty, open borders, lower wages. Immigration reform should mean something else entirely. It should mean improvements to our laws and policies to make life better for American citizens.Thank you. But if we're going to make our immigration system work, then we have to be prepared to talk honestly and without fear about these important and very sensitive issues. For instance, we have to listen to the concerns that working people, our forgotten working people, have over the record pace of immigration and it's impact on their jobs, wages, housing, schools, tax bills and general living conditions.These are valid concerns expressed by decent and patriotic citizens from all backgrounds, all over. We also have to be honest about the fact that not everyone who seeks to join our country will be able to successfully assimilate. Sometimes it's just not going to work out. It's our right, as a sovereign nation to chose immigrants that we think are the likeliest to thrive and flourish and love us.Then there is the issue of security. Countless innocent American lives have been stolen because our politicians have failed in their duty to secure our borders and enforce our laws like they have to be enforced. I have met with many of the great parents who lost their children to sanctuary cities and open borders. So many people, so many, many people. So sad. They will be joining me on this stage in a little while and I look forward to introducing, these are amazing, amazing people.Countless Americans who have died in recent years would be alive today if not for the open border policies of this administration and the administration that causes this horrible, horrible thought process, called Hillary Clinton.This includes incredible Americans like 21 year old Sarah Root. The man who killed her arrived at the border, entered Federal custody and then was released into the U.S., think of it, into the U.S. community under the policies of the White House Barack Obama and Hillary Clinton. Weak, weak policies. Weak and foolish policies.He was released again after the crime, and now he's out there at large. Sarah had graduated from college with a 4.0, top student in her class one day before her death.Also among the victims of the Obama-Clinton open borders policy was Grant Ronnebeck, a 21-year-old convenience store clerk and a really good guy from Mesa, Arizona. A lot of you have known about Grant.He was murdered by an illegal immigrant gang member previously convicted of burglary, who had also been released from federal custody, and they knew it was going to happen again.Another victim is Kate Steinle. Gunned down in the sanctuary city of San Francisco, by an illegal immigrant, deported five previous times. And they knew he was no good.Then there is the case of 90-year-old Earl Olander, who was brutally beaten and left to bleed to death in his home, 90 years old and defenseless. The perpetrators were illegal immigrants with criminal records a mile long, who did not meet Obama administration standards for removal. And they knew it was going to happen.In California, a 64-year-old Air Force veteran, a great woman, according to everybody that knew her, Marilyn Pharis, was sexually assaulted and beaten to death with a hammer. Her killer had been arrested on multiple occasions but was never, ever deported, despite the fact that everybody wanted him out.A 2011 report from the Government Accountability Office found that illegal immigrants and other non-citizens, in our prisons and jails together, had around 25,000 homicide arrests to their names, 25,000.On top of that, illegal immigration costs our country more than $113 billion dollars a year.And this is what we get. For the money we are going to spend on illegal immigration over the next 10 years, we could provide 1 million at-risk students with a school voucher, which so many people are wanting.While there are many illegal immigrants in our country who are good people, many, many, this doesn't change the fact that most illegal immigrants are lower skilled workers with less education, who compete directly against vulnerable American workers, and that these illegal workers draw much more out from the system than they can ever possibly pay back.And they're hurting a lot of our people that cannot get jobs under any circumstances.But these facts are never reported. Instead, the media and my opponent discuss one thing and only one thing, the needs of people living here illegally. In many cases, by the way, they're treated better than our vets.Not going to happen anymore, folks. November 8th. Not going to happen anymore.The truth is, the central issue is not the needs of the 11 million illegal immigrants or however many there may be -- and honestly we've been hearing that number for years. It's always 11 million.Our government has no idea. It could be 3 million. It could be 30 million. They have no idea what the number is.Frankly our government has no idea what they're doing on many, many fronts, folks.But whatever the number, that's never really been the central issue. It will never be a central issue. It doesn't matter from that standpoint. Anyone who tells you that the core issue is the needs of those living here illegally has simply spent too much time in Washington.Only the out of touch media elites think the biggest problems facing America -- you know this, this is what they talk about, facing American society today is that there are 11 million illegal immigrants who don't have legal status. And, they also think the biggest thing, and you know this, it's not nuclear, and it's not ISIS, it's not Russia, it's not China, it's global warming.To all the politicians, donors, and special interests, hear these words from me and all of you today. There is only one core issue in the immigration debate, and that issue is the well being of the American people.Nothing even comes a close second. Hillary Clinton, for instance, talks constantly about her fears that families will be separated, but she's not talking about the American families who have been permanently separated from their loved ones because of a preventable homicide, because of a preventable death, because of murder.No, she's only talking about families who come here in violation of the law. We will treat everyone living or residing in our country with great dignity. So important.We will be fair, just, and compassionate to all, but our greatest compassion must be for our American citizens.Thank you.President Obama and Hillary Clinton have engaged in gross dereliction of duty by surrendering the safety of the American people to open borders, and you know it better than anybody right here in Arizona. You know it.President Obama and Hillary Clinton support sanctuary cities. They support catch and release on the border. they support visa overstays. They support the release of dangerous, dangerous, dangerous, criminals from detention. And, they support unconstitutional executive amnesty.Hillary Clinton has pledged amnesty in her first 100 days, and her plan will provide Obamacare, Social Security, and Medicare for illegal immigrants, breaking the federal budget.On top of that she promises uncontrolled, low-skilled immigration that continues to reduce jobs and wages for American workers, and especially for African-American and Hispanic workers within our country. Our citizens.Most incredibly, because to me this is unbelievable, we have no idea who these people are, where they come from. I always say Trojan Horse. Watch what's going to happen, folks. It's not going to be pretty.This includes her plan to bring in 620,000 new refugees from Syria and that region over a short period of time. And even yesterday, when you were watching the news, you saw thousands and thousands of people coming in from Syria. What is wrong with our politicians, our leaders if we can call them that. What the hell are we doing?Hard to believe. Hard to believe. Now that you've heard about Hillary Clinton's plan, about which she has not answered a single question, let me tell you about my plan. And do you notice --And do you notice all the time for weeks and weeks of debating my plan, debating, talking about it, what about this, what about that. They never even mentioned her plan on immigration because she doesn't want to get into the quagmire. It's a tough one, she doesn't know what she's doing except open borders and let everybody come in and destroy our country by the way.While Hillary Clinton meets only with donors and lobbyists, my plan was crafted with the input from Federal Immigration offices, very great people. Among the top immigration experts anywhere in this country, who represent workers, not corporations, very important to us.I also worked with lawmakers, who've led on this issue on behalf of American citizens for many years. And most importantly I've met with the people directly impacted by these policies. So important.Number one, are you ready? Are you ready?We will build a great wall along the southern border.And Mexico will pay for the wall.One hundred percent. They don't know it yet, but they're going to pay for it. And they're great people and great leaders but they're going to pay for the wall. On day one, we will begin working on intangible, physical, tall, power, beautiful southern border wall.We will use the best technology, including above and below ground sensors that's the tunnels. Remember that, above and below.Above and below ground sensors. Towers, aerial surveillance and manpower to supplement the wall, find and dislocate tunnels and keep out criminal cartels and Mexico you know that, will work with us. I really believe it. Mexico will work with us. I absolutely believe it. And especially after meeting with their wonderful, wonderful president today. I really believe they want to solve this problem along with us, and I'm sure they will.Number two, we are going to end catch and release. We catch them, oh go ahead. We catch them, go ahead.Under my administration, anyone who illegally crosses the border will be detained until they are removed out of our country and back to the country from which they came.And they'll be brought great distances. We're not dropping them right across. They learned that. President Eisenhower. They'd drop them across, right across, and they'd come back. And across.Then when they flew them to a long distance, all of a sudden that was the end. We will take them great distances. But we will take them to the country where they came from, OK?Number three. Number three, this is the one, I think it's so great. It's hard to believe, people don't even talk about it. Zero tolerance for criminal aliens. Zero. Zero.Zero. They don't come in here. They don't come in here.According to federal data, there are at least 2 million, 2 million, think of it, criminal aliens now inside of our country, 2 million people criminal aliens. We will begin moving them out day one. As soon as I take office. Day one. In joint operation with local, state, and federal law enforcement.Now, just so you understand, the police, who we all respect -- say hello to the police. Boy, they don't get the credit they deserve. I can tell you. They're great people. But the police and law enforcement, they know who these people are.They live with these people. They get mocked by these people. They can't do anything about these people, and they want to. They know who these people are. Day one, my first hour in office, those people are gone.And you can call it deported if you want. The press doesn't like that term. You can call it whatever the hell you want. They're gone.Beyond the 2 million, and there are vast numbers of additional criminal illegal immigrants who have fled, but their days have run out in this country. The crime will stop. They're going to be gone. It will be over.They're going out. They're going out fast.Moving forward. We will issue detainers for illegal immigrants who are arrested for any crime whatsoever, and they will be placed into immediate removal proceedings if we even have to do that.We will terminate the Obama administration's deadly, and it is deadly, non-enforcement policies that allow thousands of criminal aliens to freely roam our streets, walk around, do whatever they want to do, crime all over the place.That's over. That's over, folks. That's over.Since 2013 alone, the Obama administration has allowed 300,000 criminal aliens to return back into United States communities. These are individuals encountered or identified by ICE, but who were not detained or processed for deportation because it wouldn't have been politically correct.My plan also includes cooperating closely with local jurisdictions to remove criminal aliens immediately. We will restore the highly successful Secure Communities Program. Good program. We will expand and revitalize the popular 287(g) partnerships, which will help to identify hundreds of thousands of deportable aliens in local jails that we don't even know about.Both of these programs have been recklessly gutted by this administration. And those were programs that worked.This is yet one more area where we are headed in a totally opposite direction. There's no common sense, there's no brain power in our administration by our leader, or our leaders. None, none, none.Here's an interesting graphic from a study published this summer on voting attitudes among people who heard incumbents use restrictive rhetoric on immigration -- tough enforcement, no amnesty, deportations -- and its affect on their own opinions.On my first day in office I am also going to ask Congress to pass Kate's Law, named for Kate Steinle.To ensure that criminal aliens convicted of illegal reentry receive strong mandatory minimum sentences. Strong.And then we get them out.Another reform I'm proposing is the passage of legislation named for Detective Michael Davis and Deputy Sheriff Danny Oliver, to law enforcement officers recently killed by a previously deported illegal immigrant.The Davis-Oliver bill will enhance cooperation with state and local authorities to ensure that criminal immigrants and terrorists are swiftly, really swiftly, identified and removed. And they will go face, believe me. They're going to go.We're going to triple the number of ICE deportation officers.Within ICE I am going to create a new special deportation task force focused on identifying and quickly removing the most dangerous criminal illegal immigrants in America who have evaded justice just like Hillary Clinton has evaded justice, OK?Maybe they'll be able to deport her.The local police who know every one of these criminals, and they know each and every one by name, by crime, where they live, they will work so fast. And our local police will be so happy that they don't have to be abused by these thugs anymore. There's no great mystery to it, they've put up with it for years, and no finally we will turn the tables and law enforcement and our police will be allowed to clear up this dangerous and threatening mess.We're also going to hire 5,000 more Border Patrol agents.Who gave me their endorsement, 16,500 gave me their endorsement.And put more of them on the border instead of behind desks which is good. We will expand the number of border patrol stations significantly.I've had a chance to spend time with these incredible law enforcement officers, and I want to take a moment to thank them. What they do is incredible.And getting their endorsement means so much to me. More to me really than I can say. Means so much. First time they've ever endorsed a presidential candidate.Number four, block funding for sanctuary cities. We block the funding. No more funds.We will end the sanctuary cities that have resulted in so many needless deaths. Cities that refuse to cooperate with federal authorities will not receive taxpayer dollars, and we will work with Congress to pass legislation to protect those jurisdictions that do assist federal authorities.Number five, cancel unconstitutional executive orders and enforce all immigration laws.We will immediately terminate President Obama's two illegal executive amnesties in which he defied federal law and the Constitution to give amnesty to approximately five million illegal immigrants, five million.And how about all the millions that are waiting on line, going through the process legally? So unfair.Hillary Clinton has pledged to keep both of these illegal amnesty programs, including the 2014 amnesty which has been blocked by the United States Supreme Court. Great.Clinton has also pledged to add a third executive amnesty. And by the way, folks, she will be a disaster for our country, a disaster in so many other ways.And don't forget the Supreme Court of the United States. Don't forget that when you go to vote on November 8. And don't forget your Second Amendment. And don't forget the repeal and replacement of Obamacare.And don't forget building up our depleted military. And don't forget taking care of our vets. Don't forget our vets. They have been forgotten.Clinton's plan would trigger a constitutional crisis unlike almost anything we have ever seen before. In effect, she would be abolishing the lawmaking powers of Congress in order to write her own laws from the Oval Office. And you see what bad judgment she has. She has seriously bad judgment.Can you imagine? In a Trump administration all immigration laws will be enforced, will be enforced. As with any law enforcement activity, we will set priorities. But unlike this administration, no one will be immune or exempt from enforcement. And ICE and Border Patrol officers will be allowed to do their jobs the way their jobs are supposed to be done.Anyone who has entered the United States illegally is subject to deportation. That is what it means to have laws and to have a country. Otherwise we don't have a country.Our enforcement priorities will include removing criminals, gang members, security threats, visa overstays, public charges. That is those relying on public welfare or straining the safety net along with millions of recent illegal arrivals and overstays who've come here under this current corrupt administration.Number six, we are going to suspend the issuance of visas to any place where adequate screening cannot occur.According to data provided by the Senate Subcommittee on Immigration, and the national interest between 9/11 and the end of 2014, at least 380 foreign born individuals were convicted in terror cases inside the United States. And even right now the largest number of people are under investigation for exactly this that we've ever had in the history of our country.Our country is a mess. We don't even know what to look for anymore, folks. Our country has to straighten out. And we have to straighten out fast.The number is likely higher. But the administration refuses to provide this information, even to Congress. As soon as I enter office I am going to ask the Department of State, which has been brutalized by Hillary Clinton, brutalized.Homeland Security and the Department of Justice to begin a comprehensive review of these cases in order to develop a list of regions and countries from which immigration must be suspended until proven and effective vetting mechanisms can be put in place.I call it extreme vetting right? Extreme vetting. I want extreme. It's going to be so tough, and if somebody comes in that's fine but they're going to be good. It's extreme.And if people don't like it, we've got have a country folks. Got to have a country. Countries in which immigration will be suspended would include places like Syria and Libya. And we are going to stop the tens of thousands of people coming in from Syria. We have no idea who they are, where they come from. There's no documentation. There's no paperwork. It's going to end badly folks. It's going to end very, very badly.For the price of resettling, one refugee in the United States, 12 could be resettled in a safe zone in their home region. Which I agree with 100 percent. We have to build safe zones and we'll get the money from Gulf states. We don't want to put up the money. We owe almost $20 trillion. Doubled since Obama took office, our national debt.But we will get the money from Gulf states and others. We'll supervise it. We'll build safe zones which is something that I think all of us want to see.Another reform, involves new screening tests for all applicants that include, and this is so important, especially if you get the right people. And we will get the right people. An ideological certification to make sure that those we are admitting to our country share our values and love our people.Thank you. We're very proud of our country. Aren't we? Really? With all it's going through, we're very proud of our country. For instance, in the last five years, we've admitted nearly 100,000 immigrants from Iraq and Afghanistan. And these two countries according to Pew Research, a majority of residents say that the barbaric practice of honor killings against women are often or sometimes justified. That's what they say.That's what they say. They're justified. Right? And we're admitting them to our country. Applicants will be asked their views about honor killings, about respect for women and gays and minorities. Attitudes on radical Islam, which our President refuses to say and many other topics as part of this vetting procedure. And if we have the right people doing it, believe me, very, very few will slip through the cracks. Hopefully, none.Number seven, we will insure that other countries take their people back when they order them deported.There are at least 23 countries that refuse to take their people back after they've been ordered to leave the United States. Including large numbers of violent criminals, they won't take them back. So we say, OK, we'll keep them. Not going to happen with me, not going to happen with me.Due to a Supreme Court decision, if these violent offenders cannot be sent home, our law enforcement officers have to release them into your communities.And by the way, the results are horrific, horrific. There are often terrible consequences, such as Casey Chadwick's tragic death in Connecticut just last year. Yet despite the existence of a law that commands the Secretary of State to stop issuing visas to these countries.Secretary Hillary Clinton ignored this law and refused to use this powerful tool to bring nations into compliance. And, they would comply if we would act properly.In other words, if we had leaders that knew what they were doing, which we don't.The result of her misconduct was the release of thousands and thousands of dangerous criminal aliens who should have been sent home to their countries. Instead we have them all over the place. Probably a couple in this room as a matter of fact, but I hope not.According to a report for the Boston Globe from the year 2008 to 2014 nearly 13,000 criminal aliens were released back into U.S. communities because their home countries would not, under any circumstances, take them back. Hard to believe with the power we have. Hard to believe.We're like the big bully that keeps getting beat up. You ever see that? The big bully that keeps getting beat up.These 13,000 release occurred on Hillary Clinton's watch. She had the power and the duty to stop it cold, and she decided she would not do it.And, Arizona knows better than most exactly what I'm talking about.Those released include individuals convicted of killings, sexual assaults, and some of the most heinous crimes imaginable.The Boston Globe writes that a Globe review of 323 criminals released in New England from 2008 to 2012 found that as many as 30 percent committed new offenses, including rape, attempted murder, and child molestation. We take them, we take them.Number eight, we will finally complete the biometric entry-exit visa tracking system which we need desperately. For years Congress has required biometric entry-exit visa tracking systems, but it has never been completed. The politicians are all talk, no action, never happens. Never happens.Hillary Clinton, all talk. Unfortunately when there is action it's always the wrong decision. You ever notice? In my administration we will ensure that this system is in place. And, I will tell you, it will be on land, it will be on sea, it will be in air. We will have a proper tracking system.Approximately half of new illegal immigrants came on temporary visas and then never, ever left. Why should the? Nobody's telling them to leave. Stay as long as you want, we'll take care of you.Beyond violating our laws, visa overstays, pose -- and they really are a big problem, pose a substantial threat to national security. The 9/11 Commission said that this tracking system would be a high priority and would have assisted law enforcement and intelligence officials in august and September in 2001 in conducting a search for two of the 9/11 hijackers that were in the United States expired visas.And, you know what that would have meant, what that could have meant. Wouldn't that have been wonderful, right? What that could have meant?Last year alone nearly half a million individuals overstayed their temporary visas. Removing these overstays will be a top priority of my administration.If people around the world believe they can just come on a temporary visa and never, ever leave, the Obama-Clinton policy, that's what it is, then we have a completely open border, and we no longer have a country.We must send a message that visa expiration dates will be strongly enforced.Number nine, we will turn off the jobs and benefits magnet.We will ensure that E-Verify is used to the fullest extent possible under existing law, and we will work with Congress to strengthen and expand its use across the country.Immigration law doesn't exist for the purpose of keeping criminals out. It exists to protect all aspects of American life. The work site, the welfare office, the education system, and everything else.That is why immigration limits are established in the first place. If we only enforced the laws against crime, then we have an open border to the entire world. We will enforce all of our immigration laws.And the same goes for government benefits. The Center for Immigration Studies estimates that 62 percent of households headed by illegal immigrants use some form of cash or non-cash welfare programs like food stamps or housing assistance.Tremendous costs, by the way, to our country. Tremendous costs. This directly violates the federal public charge law designed to protect the United States Treasury. Those who abuse our welfare system will be priorities for immediate removal.Number 10, we will reform legal immigration to serve the best interests of America and its workers, the forgotten people. Workers. We're going to take care of our workers.And by the way, and by the way, we're going to make great trade deals. We're going to renegotiate trade deals. We're going to bring our jobs back home. We're going to bring our jobs back home.We have the most incompetently worked trade deals ever negotiated probably in the history of the world, and that starts with NAFTA. And now they want to go TPP, one of the great disasters.We're going to bring our jobs back home. And if companies want to leave Arizona and if they want to leave other states, there's going to be a lot of trouble for them. It's not going to be so easy. There will be consequence. Remember that. There will be consequence. They're not going to be leaving, go to another country, make the product, sell it into the United States, and all we end up with is no taxes and total unemployment. It's not going to happen. There will be consequences.We've admitted 59 million immigrants to the United States between 1965 and 2015. Many of these arrivals have greatly enriched our country. So true. But we now have an obligation to them and to their children to control future immigration as we are following, if you think, previous immigration waves.We've had some big waves. And tremendously positive things have happened. Incredible things have happened. To ensure assimilation we want to ensure that it works. Assimilation, an important word. Integration and upward mobility.Within just a few years immigration as a share of national population is set to break all historical records. The time has come for a new immigration commission to develop a new set of reforms to our legal immigration system in order to achieve the following goals.To keep immigration levels measured by population share within historical norms. To select immigrants based on their likelihood of success in U.S. society and their ability to be financially self- sufficient.We take anybody. Come on in, anybody. Just come on in. Not anymore.You know, folks, it's called a two-way street. It is a two-way street, right? We need a system that serves our needs, not the needs of others. Remember, under a Trump administration it's called America first. Remember that.To choose immigrants based on merit. Merit, skill, and proficiency. Doesn't that sound nice? And to establish new immigration controls to boost wages and to ensure that open jobs are offered to American workers first. And that in particular African- American and Latino workers who are being shut out in this process so unfairly.And Hillary Clinton is going to do nothing for the African- American worker, the Latino worker. She's going to do nothing. Give me your vote, she says, on November eighth. And then she'll say, so long, see you in four years. That's what it is.She is going to do nothing. And just look at the past. She's done nothing. She's been there for 35 years. She's done nothing. And I say what do you have to lose? Choose me. Watch how good we're going to do together. Watch.You watch. We want people to come into our country, but they have to come into our country legally and properly vetted, and in a manner that serves the national interest. We've been living under outdated immigration rules from decades ago. They're decades and decades old.To avoid this happening in the future, I believe we should sunset our visa laws so that Congress is forced to periodically revise and revisit them to bring them up to date. They're archaic. They're ancient. We wouldn't put our entire federal budget on auto pilot for decades, so why should we do the same for the very, very complex subject of immigration?So let's now talk about the big picture. These 10 steps, if rigorously followed and enforced, will accomplish more in a matter of months than our politicians have accomplished on this issue in the last 50 years. It's going to happen, folks. Because I am proudly not a politician, because I am not behold to any special interest, I've spent a lot of money on my campaign, I'll tell you. I write those checks. Nobody owns Trump.I will get this done for you and for your family. We'll do it right. You'll be proud of our country again. We'll do it right. We will accomplish all of the steps outlined above. And, when we do, peace and law and justice and prosperity will prevail. Crime will go down. Border crossings will plummet. Gangs will disappear.And the gangs are all over the place. And welfare use will decrease. We will have a peace dividend to spend on rebuilding America, beginning with our American inner cities. We're going to rebuild them, for once and for all.For those here illegally today, who are seeking legal status, they will have one route and one route only. To return home and apply for reentry like everybody else, under the rules of the new legal immigration system that I have outlined above. Those who have left to seek entry --Thank you. Thank you. Those who have left to seek entry under this new system -- and it will be an efficient system -- will not be awarded surplus visas, but will have to apply for entry under the immigration caps or limits that will be established in the future.We will break the cycle of amnesty and illegal immigration. We will break the cycle. There will be no amnesty.Our message to the world will be this. You cannot obtain legal status or become a citizen of the United States by illegally entering our country. Can't do it.This declaration alone will help stop the crisis of illegal crossings and illegal overstays, very importantly. People will know that you can't just smuggle in, hunker down and wait to be legalized. It's not going to work that way. Those days are over.Importantly, in several years when we have accomplished all of our enforcement and deportation goals and truly ended illegal immigration for good, including the construction of a great wall, which we will have built in record time. And at a reasonable cost, which you never hear from the government.And the establishment of our new lawful immigration system then and only then will we be in a position to consider the appropriate disposition of those individuals who remain.That discussion can take place only in an atmosphere in which illegal immigration is a memory of the past, no longer with us, allowing us to weigh the different options available based on the new circumstances at the time.Right now, however, we're in the middle of a jobs crisis, a border crisis and a terrorism crisis like never before. All energies of the federal government and the legislative process must now be focused on immigration security. That is the only conversation we should be having at this time, immigration security. Cut it off.Whether it's dangerous materials being smuggled across the border, terrorists entering on visas or Americans losing their jobs to foreign workers, these are the problems we must now focus on fixing. And the media needs to begin demanding to hear Hillary Clinton's answer on how her policies will affect Americans and their security.These are matters of life and death for our country and its people, and we deserve answers from Hillary Clinton. And do you notice, she doesn't answer.She didn't go to Louisiana. She didn't go to Mexico. She was invited.She doesn't have the strength or the stamina to make America great again. Believe me.What we do know, despite the lack of media curiosity, is that Hillary Clinton promises a radical amnesty combined with a radical reduction in immigration enforcement. Just ask the Border Patrol about Hillary Clinton. You won't like what you're hearing.The result will be millions more illegal immigrants; thousands of more violent, horrible crimes; and total chaos and lawlessness. That's what's going to happen, as sure as you're standing there.This election, and I believe this, is our last chance to secure the border, stop illegal immigration and reform our laws to make your life better. I really believe this is it. This is our last time. November 8. November 8. You got to get out and vote on November 8.It's our last chance. It's our last chance. And that includes Supreme Court justices and Second Amendment. Remember that.So I want to remind everyone what we're fighting for and who we are fighting for.I am going to ask -- these are really special people that I've gotten to know. I'm going to ask all the Angel Moms to come join me on the stage right now. These are amazing women.These are amazing people.Thank you.These are amazing people, and I am not asking for their endorsement, believe me that. I just think I've gotten to know so many of them, and many more, from our group. But they are incredible people and what they're going through is incredible, and there's just no reason for it. Let's give them a really tremendous hand.That's tough stuff, I will tell you. That is tough stuff. Incredible people.So, now is the time for these voices to be heard. Now is the time for the media to begin asking questions on their behalf. Now is the time for all of us as one country, Democrat, Republican, liberal, conservative to band together to deliver justice, and safety, and security for all Americans.Let's fix this horrible, horrible, problem. It can be fixed quickly. Let's our secure our border.Let's stop the drugs and the crime from pouring into our country. Let's protect our social security and Medicare. Let's get unemployed Americans off the welfare and back to work in their own country.This has been an incredible evening. We're going to remember this evening. November 8, we have to get everybody. This is such an important state. November 8 we have to get everybody to go out and vote.We're going to bring -- thank you, thank you. We're going to take our country back, folks. This is a movement. We're going to take our country back.Thank you.Thank you.This is an incredible movement. The world is talking about it. The world is talking about it and by the way, if you haven't been looking to what's been happening at the polls over the last three or four days I think you should start looking. You should start looking.Together we can save American lives, American jobs, and American futures. Together we can save America itself. Join me in this mission, we're going to make America great again.Thank you. I love you. God bless you, everybody. God bless you. God bless you, thank you. 4 | 5 | Thank you very much, ladies and gentlemen.When Justice Scalia passed away suddenly last February, I made a promise to the American people: If I were elected president, I would find the very best judge in the country for the Supreme Court. I promised to select someone who respects our laws and is representative of our Constitution and who loves our Constitution and someone who will interpret them as written.This may be the most transparent judicial selection process in history. Months ago as a candidate, I publicly presented a list of brilliant and accomplished people to the American electorate and pledged to make my choice from among that list. Millions of voters said this was the single most important issue to them when they voted for me for president.I am a man of my word. I will do as I say, something that the American people have been asking for from Washington for a very, very long time. Today...Thank you. Today I am keeping another promise to the American people by nominating Judge Neil Gorsuch of the United States Supreme Court to be of the United States Supreme Court. And I would like to ask Judge Gorsuch and his wonderful wife, Louise, to please step forward — please, Louise, Judge. Here they come. Here they come.I have always felt that after the defense of our nation, the most important decision a president of the United States can make is the appointment of a Supreme Court justice. Depending on their age, a justice can be active for 50 years and his or her decisions can last a century or more and can often be permanent.I took the task of this nomination very seriously. I have selected an individual whose qualities define — really, and I mean closely define — what we’re looking for. Judge Gorsuch has outstanding legal skills, a brilliant mind, tremendous discipline and has earned bipartisan support. When he was nominated to the 10th Circuit Court of Appeals, he was confirmed by the Senate unanimously. Also — that’s unanimous, can you believe that? Nowadays, with what’s going on?Does that happen anymore? Does it happen? I think it’s going to happen. Maybe it will.Also with us tonight is Maureen Scalia, a woman loved by her husband and deeply respected by all. I am so happy she’s with us. Where is Maureen? Please, stand up. Thank you, Maureen.Thank you, Maureen. She is really the ultimate representative of the late, great Justice Antonin Scalia, whose image and genius was in my mind throughout the decision-making process. Not only are we looking at the writings of the nominee — and I studied them closely — but he is said to be among the finest and most brilliant oftentimes the writings of any judge for a long, long time.And his academic credentials, something very important to me, in that education has always been a priority, are as good as I have ever seen. He received his undergraduate degree from Columbia with honors. He then received his law degree from Harvard, also with honors, where he was a Truman Scholar. After Harvard, he received his doctorate at Oxford, where he attended as a Marshall Scholar, one of the top academic honors anywhere in the world.After law school, he clerked on the Supreme Court for both Justices Byron White and Anthony Kennedy. It is an extraordinary résumé. As good as it gets. Judge Gorsuch was born and raised in Colorado and was taught the value of independence, hard work and public service. While in law school, he demonstrated a commitment to helping the less fortunate. He worked in both Harvard Prison Legal Assistance Projects and Harvard Defenders Program. Brilliance being assured, I studied every aspect of his life. He could have had any job at any law firm for any amount of money, but what he wanted to do with his career was to be a judge, to write decisions and to make an impact by upholding our laws and our Constitution.The qualifications of Judge Gorsuch are beyond dispute. He is the man of our country and a man who our country really needs and needs badly to ensure the rule of law and the rule of justice.I would like to thank Senate leadership. I only hope that both Democrats and Republicans can come together for once for the good of the country.Congratulations to you and your family. May God bless you, may God bless our glorious nation. Judge Gorsuch, the podium, sir, is yours. 6 | 7 | Last quarter, it was just announced, our gross domestic product - a sign of strength, right? But not for us.It was below zero. Who ever heard of this? It's never below zero.Our labor participation rate was the worst since 1978.But think of it, GDP below zero, horrible labor participation rate, and our real unemployment is anywhere from 18-20%. Don't believe the 5.6. Don't believe it.That's right - a lot of people up there can't get jobs. They can't get jobs because there are no jobs because China has our jobs and Mexico has our jobs. They all have our jobs. But the real number, the real number, is anywhere from 18-19 and maybe even 21% and nobody talks about it because it's a statistic that's full of nonsense.Our enemies are getting stronger and stronger by the day and we as a country are getting weaker. Even our nuclear arsenal doesn't work.It came out recently. They have equipment that's 30 years old and they don't even know if it works. And I thought it was horrible when it was broadcast on television because boy does that send signals to Putin and all of the other people that look at us and they say 'OK, that is a group of people and that is a nation that truly has no clue. They don't know what they're doing. They don't know what they're doing.'We have a disaster called the big lie - Obamacare, Obamacare.Yesterday it came out that costs are going, for people, up 39, 39, 49 and even 55%. And deductibles are through the roof. You have to get hit by a tractor, literally a tractor, to use it because the deductibles are so high it's virtually useless. It's a disaster.And remember the $5 billion website, 5 billion we spent on a website, and to this day it doesn't work. A $5 billion dollar website.I have so many websites. I have them all over the place. I hire people, they do a website. It costs me $3.$5 billion dollar website.Well you need somebody because politicians are all talk, no action. Nothing's going to get done. They will not bring us, believe me, to the promised land. They will not.As an example, I've been on the circuit making speeches and I hear my fellow Republicans and they're wonderful people. I like them. They all want me to support them.They don't know how to bring it about, they come up to my office. I'm meeting with three of them in the next week and they don't know: Are you running, are you not running, could we have your support, what do we do, how do we do it?And I like them. I hear their speeches. And they don't talk jobs. They don't talk China. When was the last time you heard 'China's killing us?' They're devaluing their currency to a level that you wouldn't believe it makes it impossible for our companies to compete. Impossible.They're killing us, but you don't hear that from anyone else. You don't hear that from anybody else.And I watch the speeches. I watch the speeches and they say 'the sun will rise. The moon will set. All sorts of wonderful things will happen.'And the people are saying 'What's going on? I just want a job. I don't need the rhetoric, I just want a job.'And it's going to get worse because remember, Obamacare really kicks in in 2016, 2016.Obama is going to be out playing golf. He might even be on one of my courses - I would invite him. I have the best courses in the world. So I say, you know what, if he wants to - I have one right next to the White House. Right on the Potomac. If he wants to, if he'd like to play, that's fine. In fact I'd love him to leave early and play. That would be a very good thing.But Obamacare kicks in in 2016, really bigly. It is going to be amazingly destructive.Doctors are quitting.I have a friend who's a doctor and he said to me the other day: 'Donald, I never saw anything like it. I have more accountants than I have nurses. It's a disaster. My patients are besides themselves. They had a plan that was good. They had a plan. They have no plan now.'We have to repeal Obamacare and it can be replaced with something much better for everybody. Let it be for everybody, but much better and much less expensive for people and for the government. And we can do it.So I've watched the politicians. I've dealt with them all my life. If you can't make a good deal with a politician, then there's something wrong with you. There's something certainly not very good and that's what we have representing us.They will never make America great again. They don't even have a chance. They are controlled fully, they are controlled fully by the lobbyists, by the donors and by the special interests. Fully. They control them.Hey, I have lobbyists. I have to tell you, I have lobbyists that can produce anything for me. They're great. But you know what? It won't happen. It won't happen because we have to stop doing things for some people, but for our country it's destroying this country.We have to stop and it has to stop now.Our country needs, our country needs a truly great leader and we need a truly great leader now.We need a leader that wrote the Art of the Deal. We need a leader that can bring back our jobs, can bring back our manufacturing, can bring back our military, can take care of our vets - our vets have been abandoned. And we also need a cheerleader.You know, when President Obama was elected I said 'Well, the one thing I think he'll do well - I think he'll be a great cheerleader for the country. I think he'd be a great spirit. He was vibrant. He was young. I really thought he would be a great cheerleader.He's not a leader, that's true. You're right about that. But he wasn't a cheerleader. He's actually a negative force. He's been a negative force. He wasn't a cheerleader, he was the opposite.We need somebody that can take the brand of the United States and make it great again. It's not great.We need, we need, we need somebody that literally will take this country and make it great again. We can do that.And, I will tell you, I love my life. I have a wonderful family. They're saying, 'Dad, you're going to do something that's so tough.'You know, all of my life I've heard that a truly successful person, a really, really successful person - and even modestly successful - cannot run for public office. Just can't happen.And yet, that's the kind of mindset that you need to make this country great again.So, ladies and gentlemen, I am officially running for President of the United States and we are going to make our country great again.It can happen. Our country has tremendous potential. We have tremendous potential.We have people that aren't working. We have people that have no incentive to work. But they're going to have incentive to work. Because the greatest social program is a job. And they'll be proud, and they'll love it, and they'll make much more money than they would have ever made. And they'll be doing so well, and we're going to be thriving as a country. Thriving. It can happen.I will be the greatest jobs president that god ever created, I tell you that.I'll bring back our jobs from China, from Mexico, from Japan, from so many places. I'll bring back our jobs, and I'll bring back our money.Right now, think of this - we owe China $1.3 trillion. We owe Japan more than that. So they come in, they take our jobs, they take our money and then they loan us back the money and we pay them in interest. And then the dollar goes up, so their deal's even better.How stupid are our leaders? How stupid are these politicians to allow this to happen? How stupid are they?I'm going to tell you a couple of stories about trade, because I'm totally against the trade bill for a number of reasons.Number one: the people negotiating it don't have a clue. Our president doesn't have a clue. He's a bad negotiator. He's the one that did Bergdahl. We get Bergdahl, they get five killer terrorists that everybody wanted over there. We get Bergdahl. We get a traitor. We get a no-good traitor and they get the five people that they wanted for years. And those people are now back on the battlefield trying to kill us. That's the negotiator we haveTake a look at the deal he's making with Iran. He makes that deal, Israel maybe won't exist very long. It's a disaster and we have to protect Israel.So we need people - I'm a free trader. But the problem with free trade is, you need really talented people to negotiate for you. If you don't have talented people, if you don't have great leadership, if you don't have people that know business - not just a political hack that got the job because he made a contribution to a campaign, which is the way all jobs just about are gotten, free trade is terrible.Free trade can be wonderful if you have smart people. But we have people that are stupid. We have people that aren't smart, and we have people that are controlled by special interests and it's just not going to work.So here's a couple of stories. Happened recently, a friend of mine is a great manufacturer, and you know, China comes over and they dump all their stuff.I buy it. I buy it because, frankly, I have an obligation to buy it, because they devalue their currency so brilliantly. They just did it recently and nobody thought they could do it again, but with all our problems with Russia, with all our problems with everything, everything, they got away with it again.And it's impossible for our people here to compete. So I want to tell you this story. Friend of mine if a great manufacturer. Calls me up a few weeks ago, he's very upset.I said, 'What's your problem?'He said, 'You know, I make a great product.'I said, 'I know, I know that, because I buy the product.'He said, 'I can't get it into China. They won't accept it. I sent a boat over and they actually sent it back. They talked about environmental, they talked about all sorts of crap that had nothing to do with it.'I said, 'Oh, wait a minute, that's terrible. Did anyone know this?'He said, 'They do it all the time with other people.'I said, 'They send it back?'He said, 'Yea, so I finally got it over there, and they charged me a big tariff.'They're not supposed to be doing that. I told him. Now they do charge you tariffs on trucks when we send trucks and other things over there.Ask Boeing. They wanted all their patents and secrets before they agreed to buy planes from Boeing.Hey, I'm not saying they're stupid. I like China. I just sold an apartment for $15 million to somebody from China. Am I supposed to dislike them?I own a big chunk of the Bank of America building at 1290 Avenue of Americas that I got from China in a war. Very valuable. I love China.The biggest bank in the world is from China. You know where their United States headquarters is located? In this building, in Trump Tower.I love China. People say, 'Oh, you don't like China.' No, I love them, but their leaders are much smarter than our leaders. And we can't sustain ourselves with that.There's too much - it's like, it's like take the New England Patriots and Tom Brady and have them play your high school football team. That's the difference between China's leaders and our leaders.They are ripping us. We are rebuilding China. We are rebuilding many countries.China, you got there now - roads, bridges, schools. You never saw anything like it. They have bridges that make the George Washington Bridge look like small potatoes.And they're all over the place. We have all the cards, but we don't know how to use them. We don't even know that we have the cards, because our leaders don't understand the game.We would turn off that spigot by charging them tax until they behave properly.Now they're going militarily. They're building a military island in the middle of the South China Sea - a military island. Now, our country could never do that because we'd have to get environmental clearance and the environmentalists wouldn't let our country - we would never be able to build in an ocean.They built it in about one year, this massive military port. They're building up their military to a point that is very scary.You have a problem with ISIS, you have a bigger problem with China.And in my opinion, the new China, believe it or not, in terms of trade is Mexico.So this man tells me about the manufacturing. I say, 'that's a terrible story, I hate to hear it.'But I have another one, Ford. So Mexico takes a company, car company, that was going to build in Tennessee, rips it out. Everybody thought the deal was dead. Reported in the 'Wall Street Journal' recently.Everybody said that it was a done deal. It's going in, and that's going to be it, going into Tennessee -. great state, great people. All of a sudden, at the last moment, this big car manufacturer, foreign, announces they're not going to Tennessee, they're going to spend their billion dollars in Mexico instead. Not good.Now Ford announces a few weeks ago that Ford is going to build a $2.5 billion car and truck and parts manufacturing plant in Mexico. $2.5 billion. It's going to be one of the largest in the world. Ford - good company.So I announced that I'm running for President. I would, one of the early things I would do, probably before I even got in, and I wouldn't even use - you know, I know the smartest negotiators in the world.I know the good ones, I know the bad ones, I know the overrated ones. You've got a lot that are overrated. They get good stories because the newspapers get buffaloed. But they're not good.But I know the best negotiators in the world and I'd put them one for each country. Believe me folks, we will do very, very well. Very, very well.But I wouldn't even waste my time with this one. I would call up the head of Ford, who I know. If I was President I'd say 'Congratulations, I understand that you're building a nice, $2.5 billion dollar factory in Mexico and that you're going to take your cars and sell them to the United States. Zero tax - just across the board.'And you say to yourself, 'How does that help us, right? Where is that good.' It's not.So I'd say 'Congratulations, that's the good news. Let me give you the bad news. Every car, and every truck and every part manufactured in this plant that comes across the border, we're going to charge you a 35% tax. Okay? And that tax is going to be paid simultaneously with the transaction, and that's it.'Now here's what's going to happen. If it's not me in the position, if it's one of these politicians that we're running against, you know, the 400 people that we're - and here's what going to happen. They're not so stupid. They know it's not a good thing. And they may even be upset by it,But then they're going to get a call from their donors or probably from the lobbyists for Ford and say 'you can't do that to Ford, because Ford takes care of me, and I take care of you, and you can't do that to Ford.'And you know what? No problem. They're going to build in Mexico, they're going to take away thousands of jobs. That's very bad for us. So under President Trump, here's what would happen: The head of Ford will call me back, I would say within an hour after I told him the bad news, but it could be he'd want to be cool and he'll wait until the next day. You know, they want to be a little cool.And he'll say, 'Please, please, please.'He'll beg for a little while, and I'll say, 'Sorry, no interest.'Then he'll call all sorts of political people and I'll say 'Sorry fellas, no interest.'Because I don't need anybody's money. It's nice. I don't need anybody's money. I'm using my own money. I'm not using lobbyists, I'm not using donors. I don't care. I'm really rich.And by the way, I'm not even saying that to brag. That's the kind of mindset, that's the kind of thinking you need for this country.So, because we've got to make the country rich. It sounds crass. Somebody said 'oh, that's crass.' It's not crass.We've got $18 trillion in debt, we've got nothing but problems.We've got a military that needs equipment all over the place. We've got nuclear weapons that are obsolete.We've got nothing.We've got social security that's going to be destroyed if somebody like me doesn't bring money into the country. All these other people want to cut the hell out of it. I'm not going to cut it at all. I'm going to bring money in, and we're going to save it.But here is what's going to happen. After I'm called by 30 friends of mine who contributed to different campaigns, after I'm called by all of the special interests and by the donors and by the lobbyists - and they have zero chance at convincing me. Zero. I'll get a call they next day from the head of Ford.He'll say, 'Please reconsider.'I'll say, 'No.'He'll say, 'Mr. President, we've decided to move the plant back to the United States. We're not going to build it in Mexico.'That's it. They'll have no choice. They have no choice. There are hundred of things like that.I'll give you another example: Saudi Arabia. They make a billion dollars a day, a billion dollars a day.I love the Saudis, many are in this building. They make a billion dollars a day. Whenever they have problems, we send over the ships. We send, we're going to protect - what are we doing? They got nothing but money.If the right person asked them, they'd pay a fortune. They wouldn't be there except for us.And believe me, you look at the border with Yemen - you remember Obama a year ago, Yemen was a great victory. Two weeks later the place was blown up. Everybody.And they kept our equipment. They always keep our equipment. We ought to send used equipment, right? They always keep our equipment, we ought to send some real junk because, frankly, it would be - we ought to send our surplus. We're always losing this gorgeous, brand-new stuff.But look at that border with Saudi Arabia. Do you really think that these people are interested in Yemen? Saudi Arabia without us is gone. They're gone.And I'm the one that made all of the right predictions about Iraq. You know, all of these politicians that I'm running against now, it's so nice to say I'm running as opposed to if I run, if I run - I'm running.But all of these politicians that I'm running against now, they're trying to dissociate. I mean, you look at Bush - it took him five days to answer the question on Iraq. He couldn't answer the question. He didn't know.I said, 'Is he intelligent?'And then I looked at Rubio. He was unable to answer the question. He didn't know.How are these people going to lead us? How are we going to go back and made it great again? We can't They don't have a clue. They can't lead us. They can't.They can't even answer simple questions. It was terrible, but Saudi Arabia is in big, big trouble.Now, thanks to fracking and other things, the oil is all over the place. And I used to say it, there are ships at sea, and this was during the worst crisis, that were loaded up with oil. And the cartel kept the prices up because, again, they were smarter than our leaders.They were smarter than our leaders. There is so much wealth out there that we can make our country so rich again and, therefore, make it great again.Because we need money. We're dying. We're dying. We need money. We have to do it and we need the right people.So Ford will come back. They'll all come back. And I will say this - this is going to be an election, in my opinion, that's based on competence.Somebody said to me the other day, a reporter, very nice reporter - 'But Mr. Trump, you're not a nice person.'But actually, I am. I think I'm a nice person. Does my family like me? I think so. Look at my family.I'm proud of my family by the way. Speaking of my family - Melania, Barron, Kai, Donny, Dunn, Vanessa, Tiffany, Ivanka did a great job. Did she do a great job? Jarrett, Laura and Eric. I'm very proud of my family. They're a great family.So the report said to me the other day 'But Mr. Trump, you're not a nice person. How can you get people to vote for you?'I said, 'I don't know. I think that, number one, I am a nice person. I give a lot of money away to charities and other things.'I think I'm actually a very nice person, but I said 'This is going to be an election that's based off competence. Because people are tired of these nice people and they're tired of being ripped of by everybody in the world and they're tired of spending more money on education than any nation in the world per capita. Than any nation in the world.'And we're 26th in the world. Twenty-five countries are better than us at education, and some of them are like, third-world countries.But we're becoming a third-world country because of our infrastructure, our airports, our roads, everything.So one of the things I did, and I said, you know what I'll do? I'll do it. And a lot of people said 'he'll never run. Number one, he won't want to give up his lifestyle.'They're right about that, but I'm doing it.Number two - I'm a private company, so nobody knows what I'm worth. And the one thing is, when you run, you have to announce and certify to all sorts of governmental authorities, your net worth.So I said, 'that's okay, I'm proud of my net worth.'I've done an amazing job. I started off in a small office with my father in Brooklyn and Queens. And my father said - and I love my father. I learned so much. He was a great negotiator.I learned so much just sitting as his feet playing with blocks, listening to him negotiate with subcontractors. But I learned a lot.But he used to say 'Donald, don't go into Manhattan. That's the big leagues. We don't know anything about that. Don't do it.'But I said, 'Dad, I gotta go into Manhattan. I gotta build those buildings. I've got to do it, Dad, I've got to do it.'And after four or five years in Brooklyn, I ventured into Manhattan and did a lot of great deals: the Grand Hyatt hotel, I was responsible for the convention center on the west side.I did a lot of great deals and I did them early and young, and now I'm building all over the world. And I love what I'm doing.But they all said, a lot of the pundits on television, 'well Donald will never run and one of the main reasons is, he's private, and he's probably not as successful as everybody thinks.'So I said to myself, 'you know, nobody's ever going to know unless I run because I'm really proud of my success, I really am.'I've employed tens of thousands of people over my lifetime. That means medical, that means education, that means everything.So a large accounting firm and my accountants have been working for months because I'm big and complex and they put together a statement, a financial statement. It's a summary, but everything will be filed eventually with the government. And we don't need extensions or anything, we'll be filing it right on time.We don't need anything. And it was even reported incorrectly yesterday, because they said he had assets of nine billion.I said, 'no, that the wrong number. That's the wrong number, not assets.'So they put together this, and before I say it, I have to say this: I made it the old-fashioned way. It's real estate. it's labor and it's union - good and some bad - and lots of people that aren't unions and it's all over the place and building all over the world.And I have assets, big accounting firm - one of the most highly respected - $9,240,000,000.And I have liabilities of about $500 - that's long-term debt, very low interest rates.In fact, one of the big banks came to me, said, 'Donald, you don't have enough borrowing, can we loan you $4 billion.'I said 'I don't need it. I don't want it. I've been there. I don't want it.'But in two seconds, they give me whatever I wanted. So I have a total net worth, and now with the increase, it'll be well-over $10 billion. But here, a total net worth of ' net worth, not assets, not ' a net worth, after all debt, after all expenses, the greatest assets ' Trump Tower, 1290 Avenue of the Americas, Bank of America building in San Francisco, 40 Wall Street, sometimes referred to as the Trump building right opposite the New York ' many other places all over the world.So the total is $8,737,540,000.Now I'm not doing that, I'm not doing that to brag, because you know what? I don't have to brag. I don't have to, believe it or not.I'm doing that to say that that's the kind of thinking our country needs. We need that thinking. We have the opposite thinking.We have losers. We have losers. We have people that don't have it. We have people that are morally corrupt. We have people that are selling this country down the drain.So I put together this statement, and the only reason I'm telling you about it today is because we really do have to get going, because if we have another three or four years ' you know, we're at $8 trillion now. We're soon going to be at $20 trillion.According to the economists, who I'm not big believers in, but, nevertheless, this is what they're saying, that $24 trillion. We're very close, that's the point of no return. $24 trillion.We will be there soon. That's when we become Greece. That's when we become a country that's unsalvageable. And we're gonna be there very soon. We're gonna be there very soon.So, just to sum up, I would do various things very quickly. I would repeal and replace the big lie, Obamacare.I would build a great wall, and nobody builds walls better than me, believe me, and I'll build them very inexpensively, I will build a great, great wall on our southern border. And I will have Mexico pay for that wall.Mark my words.Nobody would be tougher on ISIS than Donald Trump. Nobody.I will find, within our military, I will find the General Patton or I will find General MacArthur, I will find the right guy. I will find the guy that's going to take that military and make it really work. Nobody, nobody will be pushing us around.I will stop Iran from getting nuclear weapons. And we won't be using a man like Secretary Kerry that has absolutely no concept of negotiation, who's making a horrible and laughable deal, who's just being tapped along as they make weapons right now, and then goes into a bicycle race at 72 years old, and falls and breaks his leg.I won't be doing that. And I promise I will never be in a bicycle race. That I can tell you.I will immediately terminate President Obama's illegal executive order on immigration, immediately.Fully support and back up the Second Amendment.Now, it's very interesting. Today I heard it. Through stupidity, in a very, very hard core prison, interestingly named Clinton, two vicious murderers, two vicious people escaped, and nobody knows where they are.And a woman was on television this morning, and she said, 'You know, Mr. Trump,' and she was telling other people, and I actually called her, and she said, 'You know, Mr. Trump, I always was against guns. I didn't want guns. And now since this happened,' it's up in the prison area, 'my husband and I are finally in agreement, because he wanted the guns. We now have a gun on every table. We're ready to start shooting.'I said, 'Very interesting.'So protect the Second Amendment.End, end Common Core. Common Core should, it is a disaster. Bush is totally in favor of Common Core.I don't see how he can possibly get the nomination. He's weak on immigration. He's in favor of Common Core. How the hell can you vote for this guy? You just can't do it.We have to end, education has to be local.Rebuild the country's infrastructure. Nobody can do that like me. Believe me. It will be done on time, on budget, way below cost, way below what anyone ever thought.I look at the roads being built all over the country, and I say I can build those things for one-third. What they do is unbelievable, how bad.You know, we're building on Pennsylvania Avenue, the Old Post Office, we're converting it into one of the world's great hotels. It's gonna be the best hotel in Washington, D.C. We got it from the General Services Administration in Washington. The Obama administration. We got it. It was the most highly sought after ' or one of them, but I think the most highly sought after project in the history of General Services.We got it. People were shocked, Trump got it. Well, I got it for two reasons. Number one, we're really good. Number two, we had a really good plan. And I'll add in the third, we had a great financial statement. Because the General Services, who are terrific people, by the way, and talented people, they wanted to do a great job. And they wanted to make sure it got built.So we have to rebuild our infrastructure, our bridges, our roadways, our airports.You come into LaGuardia Airport, it's like we're in a third world country. You look at the patches and the 40-year-old floor. They throw down asphalt, and they throw.You look at these airports, we are like a third world country. And I come in from China and I come in from Qatar and I come in from different places, and they have the most incredible airports in the world. You come to back to this country and you have LAX, disaster. You have all of these disastrous airports. We have to rebuild our infrastructure.Save Medicare, Medicaid and Social Security without cuts. Have to do it.Get rid of the fraud. Get rid of the waste and abuse, but save it. People have been paying it for years. And now many of these candidates want to cut it.You save it by making the United States, by making us rich again, by taking back all of the money that's being lost.Renegotiate our foreign trade deals.Reduce our $18 trillion in debt, because, believe me, we're in a bubble. We have artificially low interest rates. We have a stock market that, frankly, has been good to me, but I still hate to see what's happening. We have a stock market that is so bloated.Be careful of a bubble because what you've seen in the past might be small potatoes compared to what happens. So be very, very careful.And strengthen our military and take care of our vets. So, so important.Sadly, the American dream is dead. But if I get elected president I will bring it back bigger and better and stronger than ever before, and we will make America great again.Thank you. Thank you very much. 8 | 9 | My fellow Americans,This week I nominated Neil Gorsuch for the United States Supreme Court.Judge Gorsuch is one of the most qualified people ever to be nominated for this post. He is a graduate of Columbia, Harvard and Oxford. He is a man of principle. He has an impeccable resume. He is widely respected by everyone. And, Judge Gorsuch's proven track record upholding the Constitution makes him the ideal person to fill the vacancy left by the late, great Antonin Scalia, a truly fabulous justice.Ten years ago, the Senate unanimously approved Judge Gorsuch's nomination to serve on the Tenth Circuit Court of Appeals. I urge members of both parties to support Judge Gorsuch and, in so doing, to protect our laws and our freedoms.This week we also took significant action to roll back the massive regulation that is devastating our economy and crippling American companies and jobs.That's why I have issued a new executive order to create a permanent structure of regulatory reduction. This order requires that for every 1 new regulation, 2 old regulations must—and I mean must—be eliminated. It’s out of control.The January employment report shows that the private sector added 237,000 jobs last month. A lot of that has to do with the spirit our country now has. Job growth far surpassed expectations in January, and the labor force participation also grew, so you can be encouraged about the progress of our economy. It’s going to be a whole new ball game.But there is still much work to do. That I can tell you.Also this week, on the first day of Black History Month, I was pleased to host African American leaders at the White House. We are determined to deliver more opportunity, jobs and safety for the African-American citizens of our country. America can really never, ever rest until children of every color are fully included in the American Dream—so important. I think, probably, one of my most and maybe my most important goal. It is our mutual duty and obligation to make sure this happens.At Dover Air Force Base on Wednesday I joined the family of Chief Special Warfare Operator William "Ryan" Owens as our fallen hero was returned home. A great man. Chief Owens gave his life for his country and for our people. Our debt to him and his family, a beautiful family, is eternal.God has truly blessed this nation to have given us such a brave and selfless patriot as Ryan. We will never forget him. We will never ever forget those who serve. Believe me.And I will never forget that my responsibility is to keep you—the American people—safe and free.That's why last week I signed an executive order to help keep terrorists out of our country. The executive order establishes a process to develop new vetting and mechanisms to ensure those coming into America love and support our people. That they have good intentions.On every single front, we are working to deliver for American workers and American families. You, the law-abiding citizens of this country, are my total priority. Your safety, your jobs and your wages guide our decisions.We are here to serve you, the great and loyal citizens of the United States of America.The forgotten men and women will be never be forgotten again. Because from now on, it's going to be America First. That’s how I got elected, that’s why you voted for me, and I will never forget it.God Bless You, and God Bless America. 10 | 11 | “Thank you very much. I am honored to have Prime Minister Theresa May here for our first official visit from a foreign leader. This is our first visit, so -- great honor.The special relationship between our two countries has been one of the great forces in history for justice and for peace. And, by the way, my mother was born in Scotland -- Stornoway -- which is serious Scotland.Today, the United States renews our deep bond with Britain -- military, financial, cultural, and political. We have one of the great bonds. We pledge our lasting support to this most special relationship. Together, America and the United Kingdom are a beacon for prosperity and the rule of law. That is why the United States respects the sovereignty of the British people and their right of self-determination. A free and independent Britain is a blessing to the world, and our relationship has never been stronger.Both America and Britain understand that governments must be responsive to everyday working people, that governments must represent their own citizens.Madam Prime Minister, we look forward to working closely with you as we strengthen our mutual ties in commerce, business and foreign affairs. Great days lie ahead for our two peoples and our two countries.On behalf of our nation, I thank you for joining us here today. It’s a really great honor. Thank you very much.” 12 | 13 | My fellow Americans, one week ago our administration assumed the enormous responsibility, that you, the American people have placed in us.There is much work to do in the days ahead, but I wanted to give you an update on what we have accomplished already.In my first few days as your president, I've met with the leaders of some our nation's top manufacturing companies and labor unions. My message was clear. We want to make things in America and we want to use American workers. Since my election many companies have announced they are no longer moving jobs out of our country but are instead keeping and creating jobs right here in America.Every day, we are fulfilling the promise we made to the American people. Here are just a few of the executive actions I have taken in the past few days.An order to prepare for repealing and replacing Obamacare - it's about time.The withdrawal from the Trans-Pacific Partnership, so that we can negotiate one-on-one deals that protect American workers that would have been a disastrous deal for our workers.An order to begin construction of the Keystone and Dakota Access pipelines following a renegotiation of terms with the requirement that pipelines installed in America be built with American steel and manufactured here.A directive to expedite permits for new infrastructure and new manufacturing plants.An order to immediately begin the Border Wall and to crack down on Sanctuary Cities. They are not safe, we have to take care of that horrible situation.This administration has hit the ground running at a record pace. Everybody's talking about it. We're doing it with speed, we're doing it with intelligence and we will never ever stop fighting on behalf of the American people.God Bless you and God bless America. 14 | 15 | Thank you.Thank you very much. Thank you, folks.Thank you, folks. It's great to be right here in Florida, which we love.In 26 days, we are going to win this great, great state and we are going to win the White House.Our movement is about replacing a failed and corrupt -- now, when I say "corrupt," I'm talking about totally corrupt -- political establishment, with a new government controlled by you, the American people.There is nothing the political establishment will not do -- no lie that they won't tell, to hold their prestige and power at your expense. And that's what's been happening.The Washington establishment and the financial and media corporations that fund it exist for only one reason: to protect and enrich itself.The establishment has trillions of dollars at stake in this election. As an example, just one single trade deal they'd like to pass involves trillions of dollars, controlled by many countries, corporations and lobbyists.For those who control the levers of power in Washington, and for the global special interests, they partner with these people that don't have your good in mind. Our campaign represents a true existential threat like they haven't seen before.This is not simply another four-year election. This is a crossroads in the history of our civilization that will determine whether or not we the people reclaim control over our government.The political establishment that is trying to stop us is the same group responsible for our disastrous trade deals, massive illegal immigration and economic and foreign policies that have bled our country dry.The political establishment has brought about the destruction of our factories, and our jobs, as they flee to Mexico, China and other countries all around the world. Our just-announced job numbers are anemic. Our gross domestic product, or GDP, is barely above 1 percent. And going down. Workers in the United States are making less than they were almost 20 years ago, and yet they are working harder.But so am I working harder, that I can tell you.It's a global power structure that is responsible for the economic decisions that have robbed our working class, stripped our country of its wealth and put that money into the pockets of a handful of large corporations and political entities.Just look at what this corrupt establishment has done to our cities like Detroit; Flint, Michigan; and rural towns in Pennsylvania, Ohio, North Carolina and all across our country. Take a look at what's going on. They stripped away these town bare. And raided the wealth for themselves and taken our jobs away out of our country never to return unless I'm elected president.The Clinton machine is at the center of this power structure. We've seen this first hand in the WikiLeaks documents, in which Hillary Clinton meets in secret with international banks to plot the destruction of U.S. sovereignty in order to enrich these global financial powers, her special interest friends and her donors.So true.Honestly, she should be locked up.Should be locked up.And likewise the e-mails show that the Clinton machine is so closely and irrevocably tied to the media organizations that she -- that she -- listen to this -- is given the questions and answers in advance of her debate performance with Bernie Sanders.Hillary Clinton is also given approval and veto power over quotes written about her in the New York Times.They definitely do not do that to me, that I can tell you.And the e-mails show the reporters collaborate and conspire directly with the Clinton campaign on helping her win the election all over.With their control over our government at stake, with trillions of dollars on the line, the Clinton machine is determined to achieve the destruction of our campaign, not gonna to happen.Which has now become a great, great movement, the likes of which our country has never seen before, never ever.They've never seen a movement like this in our country before. Yesterday in Florida, massive crowds, people lined up outside of big arenas, not able to get in. Never happened before. It's one of the phenomenas -- it's one of the great political phenomenas. The most powerful weapon deployed by the Clintons is the corporate media, the press.Let's be clear on one thing, the corporate media in our country is no longer involved in journalism. They're a political special interest no different than any lobbyist or other financial entity with a total political agenda, and the agenda is not for you, it's for themselves.And their agenda is to elect crooked Hillary Clinton at any cost, at any price, no matter how many lives they destroy.For them it's a war, and for them nothing at all is out of bounds. This is a struggle for the survival of our nation, believe me. And this will be our last chance to save it on November 8th, remember that.This election will determine whether we are a free nation or whether we have only the illusion of democracy, but are in fact controlled by a small handful of global special interests rigging the system, and our system is rigged. This is reality, you know it, they know it, I know it, and pretty much the whole world knows it. The establishment and their media enablers will control over this nation through means that are very well known. Anyone who challenges their control is deemed a sexist, a racist, a xenophobe, and morally deformed.They will attack you, they will slander you, they will seek to destroy your career and your family, they will seek to destroy everything about you, including your reputation. They will lie, lie, lie, and then again they will do worse than that, they will do whatever is necessary. The Clintons are criminals, remember that. They're criminals.This is well documented, and the establishment that protects them has engaged in a massive coverup of widespread criminal activity at the State Department and the Clinton Foundation in order to keep the Clintons in power.Never in history have we seen such a coverup as this, one that includes the total destruction of 33,000 e-mails; 13 iPhones, some by hammer; laptops; missing boxes of evidence; and many, many other things.People who are capable of such crimes against our nation are capable of anything. And so now we address the slander and libels that was just last night thrown at me by the Clinton machine and the New York Times and other media outlets, as part of a concerted, coordinated and vicious attack.It's not coincidence that these attacks come at the exact same moment, and all together at the same time as WikiLeaks releases documents exposing the massive international corruption of the Clinton machine, including 2,000 more e-mails just this morning.These vicious claims about me of inappropriate conduct with women are totally and absolutely false.And the Clintons know it, and they know it very well. These claims are all fabricated. They're pure fiction and they're outright lies. These events never, ever happened and the people said them meekly fully understand. You take a look at these people, you study these people, and you'll understand also.The claims are preposterous, ludicrous, and defy truth, common sense and logic. We already have substantial evidence to dispute these lies, and it will be made public in an appropriate way and at an appropriate time very soon.These lies come from outlets whose past stories and past claims have already been discredited. The media outlets did not even attempt to confirm the most basic facts because even a simple investigation would have shown that these were nothing more than false smears.Six months ago, the failing New York Times wrote a massive story attacking me, and the central witness they used said the story was false; that she was quoted inaccurately. She said that I was a great guy. She had great courage, I'll be honest with you. She was an amazing person. And never made those remarks -- that I was a great guy, and never made the remarks.And when I read the story, I was sort of surprised -- how could she say that? And she didn't say it.We demanded a retraction but, they refused to print it, just like they refused to pin the comments from another source who praised me in her book, or the words of another wonderful woman who said really nice things about me. They put other statements that she didn't say, they misrepresented. The story was a fraud and a big embarrassment to The New York Times and it was a big front page story. Front page, center, color picture, a disgrace. They were very embarrassed, it will be part of the lawsuit we are preparing against them.Now, today the same two discredited writers, who should've been fired from The New York Times for what the did, tell another totally fabricated and false story, that supposedly took place on an airplane more than 30 years ago. Another ridiculous tale, no witnesses, no nothing.Then, there was a writer from People Magazine, who wrote a story on Melania and myself on our first anniversary. The story was beautiful, it was beautiful, it was lovely. But, last night we hear that after 12 years -- this took place 12 years ago, this story -- a new claim that I made inappropriate advances during the interview to this writer...And I asked very simple question, why wasn't it part of the story that appeared 20, or 12 years ago? Why wasn't it a part of the story? Why didn't they make it part of the story? I was one of the biggest stars on television with The Apprentice and I would've been one of the biggest stories of the year. Think of it, she's doing this story on Melania, who was pregnant at the time. And Donald Trump, our one year anniversary and she said I made inappropriate advances, and by the way, the area was a public area, people all over the place.Take a look, you take a look. Look at her, look at her words. You tell me, what you think. I don't think so -- I don't think so.But, it is amazing, doing a story -- a love story -- on how great we are together -- and by the way, we're stronger today than we ever were before which is good, but...It's a love story -- it's a love story on our one year. And if I did that, she would've added that it would've been the headline. And who would've done that if you're doing this and you're one of the tops shows on television.These people are horrible people, they're horrible, horrible liars. And interestingly, it happens to appear 26 days before our very important election, isn't that amazing?This invented account has already been debunked by eyewitnesses who were there -- they were there. The very witness identified by the author has said the story is totally false.By the way, this is a room that everybody can see in. It's got glass walls. It's at Mar-A-Lago; it's got glass walls. Can you believe this? Why wasn't it in the story, biggest story of the year.This weekend the New York Times published a full-page hit piece with another claim from an individual who has been totally discredited based on the many, many, many e-mails and letters she has sent to our office over the years, looking for work, Donald is great, wanting go to my rallies. But, the New York Times -- and this was full op-ed piece -- refused to use the evidence that we presented -- refused to use it. If they used it, if they would have looked, they would have said, there's no story here.Others in the media, which almost surprises me because they're dishonest also, were presented with the story by this woman numerous times, and they got very excited. But, after seeing the evidence that we immediately give them, all of them refused to write the story. There was no story.The Times, though, didn't want to see it, they just wrote the story. And this was a full page opposite the editorials. This is part of a concerted effort, led by the New York Times and others. Now the New York Times is fighting desperately for its relevance and its financial survival. And it probably won't even be around in a few years, based on its financial outlook. Which wouldn't be a bad thing, if you want to know the truth.But, as it winds down its years and is becoming more and more problematic, it's gotten more and more vicious, more and more vile. And even the other mainstream media is talking about the single greatest pile on in history, and all between now and November 8th. And you have to see the stories they've written, it's one after another, after another, and facts mean nothing, third-rate journalism. The great editors of the past from the New York Times and others, ladies and gentlemen, are spinning in their grave.I will not allow the Clinton machine to turn our campaign into a discussion of their slanders and lies, but, will remain focused on the issues facing the American people.Thank you, thank you. But, let me state this as clearly as I can, these attacks are orchestrated by the Clintons and their media allies. The only thing Hillary Clinton has going for herself is the press, without the press, she is absolutely zero.And you saw that the other night in the debate, where some people said she made virtually a fool of herself. This is not presidential material, believe me. What they say is false and slanderous in virtually every respect. We are now less than a month from the most election of our lifetime. Indeed one of the most important elections in the history of our country. And the polls are showing us in a dead heat. Don't believe what you say (ph).The new, highly respected Rasmussen poll just came out this morning. Just released. Shows up nationally 2 points ahead, Trump. Beautiful (ph).Just came out. So now the Clinton machine has put forward a small handful of people out of tens of thousands of people over the years that I've met, that I've worked with, that I've employed, in order to make wild and false allegations that fail to meet even the most basic test of common sense. Not even common sense.Again, this is nothing more than an attempt to destroy our movement and for the Clintons to keep their corrupt control over our government. When I declared my candidacy, I knew what bad shape our country was in. And believe me all you have to do is look at world events. All you have to do is look at the $1.7 billion that we sent to Iran in cash. In cash.All you have to do is see the way ISIS was created in the vacuum left by Hillary Clinton and Barack Obama out of Iraq. All you have to do, all you have to do, is look at the 800 people that were very, very not good for our nation. They were going to be deported. Lo and behold, instead of being deported, they were made citizens of the United States. Just recently.And lo and behold, sadly, the 800 people is wrong. It turned out to be close to 1,800 people. Our president is incompetent. All he wants to do is campaign, and the last thing he wants to happen is to have Donald Trump terminate Obamacare and do all of the other things that are destroying ...He's led a very divided nation and it's only gotten worse. And the last thing our country needs is four more years of Barack Obama, believe me. I've seen first hand the corruption and the sickness that has taken over our politics. You've seen it and I've seen it and we're all watching together.They knew they would throw every lie they could at me and my family and my loved ones. They knew they would stop at nothing to try to stop me. But I never knew, as bad as it would be, I never knew it would be this vile, that it would be this bad, that it would be this vicious.Nevertheless, I take all of these slings and arrows gladly for you.I take them for our movement so that we can have our country back.Our great civilization, here in America and across the civilized world has come upon a moment of reckoning. We've seen it in the United Kingdom, where they voted to liberate themselves from global government and global trade deal, and global immigration deals that have destroyed their sovereignty and have destroyed many of those nations. But, the central base of world political power is right here in America, and it is our corrupt political establishment that is the greatest power behind the efforts at radical globalization and the disenfranchisement of working people. Their financial resources are virtually unlimited, their political resources are unlimited, their media resources are unmatched, and most importantly, the depths of their immorality is absolutely unlimited.They will allow radical Islamic terrorists to enter our country by the thousands.They will allow the great Trojan horse -- and I don't want people looking back in a hundred years and 200 years and have that story be told about us because we were led by inept, incompetent and corrupt people like Barack Obama and like Hillary Clinton. We don't want to be part of that history.And by the way, President Obama should stop campaigning and start working on creating jobs, start working on getting our GDP up, start working on strengthening our borders.The corrupt political establishment is a machine, it has no soul. I knew these false attacks would come. I knew this day would arrive, it's only a question of when. And I knew the American people would rise above it and vote for the future they deserve.The only thing that can stop this corrupt machine is you. The only force strong enough to save our country is us. The only people brave enough to vote out this corrupt establishment is you, the American peopleWe are going to have a policy, America first.They control incredibly, the Department of Justice.And they even secretly meet with the Attorney General of the United States.In the back of her airplane, while on the runway -- remember he was there -- he was going to play golf. There's the Attorney General. Let me go say hello -- plane's on the runway. Let me go say hello to the attorney general. He never got to play golf, I understand. And it was Arizona, a place I love, but, the weather was about a hundred and some odd degrees -- he's not gonna play. He was never there to play golf, folks, don't be foolish.They met for 39 minutes and most likely it was to discuss her re- appointment, in a Clinton administration, as the Attorney General, just prior to making a decision over whether or not to prosecute Hillary Clinton. OK? That's what happened, that's called real life and that's pretty sad.They met for 39 minutes. Remember he said, we talked golf, and we talked about our grandchildren.Three minutes for the grandchildren, two minutes for the golf, then they sat there and they twiddled their thumbs. Now I believe they talked about her remaining in her position under a crooked Hillary Clinton administration. That's what I believe -- that's what I believe folks. That's what I believe and I think that's what most people in this room believe.Likewise they've essentially corrupted the Director of the FBI to the point at which stories are already saying that the great -- and they are truly great -- men and women who work for the FBI are embarrassed and ashamed of what he's done to one of our truly great institutions, the FBI itself.Hillary Clinton is guilty, of all the things that Director Comey stated at his press conference and Congressional hearings, and far more. He stated many things, but it's far more and he knows that. And yet, after reading all of these items, where she's so guilty, he let her off the hook. While other lives, including General Petraeus and many others, have been destroyed for doing far, far less. This is a conspiracy against you the American people and we cannot let this happen or continue.This is our moment of reckoning as a society and as a civilization itself. I didn't need to do this, folks, believe me -- believe me. I built a great company, and I had a wonderful life. I could have enjoyed the fruits and benefits of years of successful business deals and businesses for myself and my family. Instead of going through this absolute horror show of lies, deceptions, malicious attacks -- who would have thought? I'm doing it because this country has given me so much, and I feel so strongly that it's my turn to give back to the country that I love.Many of my friends and many political experts warned me that this campaign would be a journey to hell. Said that. But they're wrong. It will be a journey to heaven, because we will help so many people that are so desperately in need of help.In my former life I was in insider, as much as anybody else. And I knew what it's like, and I still know what it's like to be an insider. It's not bad, it's not bad. Now I'm being punished for leaving the special club and revealing to you the terrible things that are going on having to do with our country. Because I used to be part of the club, I'm the only one that can fix it.I'm doing this for the people and for the movement, and we will take back this country for you and we will make America great again.The corrupt establishment knows that we are a great threat to their criminal enterprise. They know that if we win their power is gone, and it's returned to you, the people, will be. The dark clouds hanging over our government can be lifted and replaced with a bright future. But, it all depends on whether we let the corrupt media decide our future, or whether we let the American people decide our future.If this Clinton campaign of destruction is allowed to work, then no other highly success -- and this is so true -- I mean I've seen this so many times, and I've heard this all of my life -- I've heard it all of my life. If we let this happen, then no other highly- successful person, which is what our country needs -- it needs a certain thinking. When you look at our trade deals that are so bad. When you look, as an example, on trade, we're going to lose almost $800 billion this year, trade deficit, almost $800 billion. Our debt has doubled in seven and a half years to almost $20 trillion, under Obama.No other successful person, after watching this, and no other very successful person will ever again, ever -- and who can blame them? Even me, I'd say, you're right -- will ever again run for office. I've heard it for years, if you're very successful you can't run for high office, especially for President. I said, I don't care, I don't care. I've done so many deals, I've done so well. It's a certain mindset that we need in our country, at least for a period of time, we have to straighten our country out.I will not lie to you. These false attacks are absolutely hurtful. To be lied about, to be slandered, to be smeared so publicly, and before your family that you love, is very painful. What is going on is egregious beyond any words. People that know the story, people that see the story, people that know the facts, they can't even believe it. It's reprehensible beyond description, it's totally corrupt.But, I also know that it's not about me, it's about all of you and it's about our country, I know that, I fully understand that. That's why I got involved. It's about all of us together as a country. It's a movement the likes of which we have never in history in this country seen before, never in history. Even the pundits, even the media -- that truly dislikes Donald Trump for their own reasons -- will admit this is a movement the likes of which people have never seen before. And it's a movement about the veterans who need medical care.The mothers who've lost their beloved children to terrorism and to crime. It's about the inner cities and the border towns who desperately need our help. It's about the millions of jobless people in America. It's about the American workers who can't get jobs because our jobs have left for Mexico and so many other countries.This election is about the people being crushed by Obamacare. And it's about defeating ISIS and appointing a Supreme Court and a Supreme Court Justice -- it could be four or five -- who will defend and protect our Constitution.This election is also about, so importantly to me, African- American and Hispanic-American people whose communities have been plunged into crime, poverty and failing schools by the policies of crooked Hillary Clinton. Believe me, she's crooked.They've robbed these citizens of their future and, if we win, I will give them their hope, their jobs, their education. I will give them their security back. The inner cities, education is almost worthless, it's horrible. We're going to have Common Core ended.We're going to bring education local.But, you look at the inner cities and you see bad education, no jobs, no safety. You walk to the grocery store with your child and you get shot. You walk outside to look and see what's happening, and you get shot. In Chicago 3,000 people have been shot since January 1st. We're not going to let that happen. Our inner cities are almost at an all-time low, run by the Democrats for sometimes more than a hundred years, chain unbroken.So they have no jobs, they have horrible education, they have no safety or security, and I say to the African-American community, what the hell do you have to lose? I will fix it -- I will fix it, I will make it good, I'll bring back our jobs. We'll have good education, we'll have great safety in the inner city.And we will help the Hispanic-American people, who have been treated so badly and so unfairly in our nation, we will help them.I will deliver like you've never seen before, I deliver. Whether people like Donald Trump or not, they all say he delivers.Vote for Donald Trump. You're going to see something and you'll be so happy, you'll be so thrilled. This election is about every man, woman and child in our country who deserves to live in safety, prosperity and peace, so true. We will rise above the lies, the smears, the ludicrous slanders from ludicrous and very, very dishonest reporters.We will vote for the country we want; we will vote for the future we want; we will vote for the politics we want; and we will vote to put this corrupt government cartel out of business and out of business immediately.We will vote for the special interests and say lots of luck but, you're being voted out of power. They've betrayed our workers, they've betrayed our borders and, most of all, they've betrayed our freedoms. We will save our sovereign rights as a nation. We will end the politics of profit; we will end the rule of special interests; we will end the raiding of our jobs by other countries; we will end the total disenfranchisement of the American voter and the American worker. Our Independence Day is at hand, and it arrives finally on November 8th.Join me in taking back our country and creating a bright, glorious, and prosperous new future for our people. We will make America great again, and it will happen quickly.God bless you. God bless you. Thank you. Thank you. 16 | 17 | Well, the election, it came out really well. Next time we'll triple the number or quadruple it. We want to get it over 51, right? At least 51.Well this is Black History Month, so this is our little breakfast, our little get-together. Hi Lynn, how are you? Just a few notes. During this month, we honor the tremendous history of African-Americans throughout our country. Throughout the world, if you really think about it, right? And their story is one of unimaginable sacrifice, hard work, and faith in America. I've gotten a real glimpse'during the campaign, I'd go around with Ben to a lot of different places I wasn't so familiar with. They're incredible people. And I want to thank Ben Carson, who's gonna be heading up HUD. That's a big job. That's a job that's not only housing, but it's mind and spirit. Right, Ben? And you understand, nobody's gonna be better than Ben.Last month, we celebrated the life of Reverend Martin Luther King, Jr., whose incredible example is unique in American history. You read all about Dr. Martin Luther King a week ago when somebody said I took the statue out of my office. It turned out that that was fake news. Fake news. The statue is cherished, it's one of the favorite things in the'and we have some good ones. We have Lincoln, and we have Jefferson, and we have Dr. Martin Luther King. But they said the statue, the bust of Martin Luther King, was taken out of the office. And it was never even touched. So I think it was a disgrace, but that's the way the press is. Very unfortunate.I am very proud now that we have a museum on the National Mall where people can learn about Reverend King, so many other things. Frederick Douglass is an example of somebody who's done an amazing job and is being recognized more and more, I noticed. Harriet Tubman, Rosa Parks, and millions more black Americans who made America what it is today. Big impact.I'm proud to honor this heritage and will be honoring it more and more. The folks at the table in almost all cases have been great friends and supporters. Darrell'I met Darrell when he was defending me on television. And the people that were on the other side of the argument didn't have a chance, right? And Paris has done an amazing job in a very hostile CNN community. He's all by himself. You'll have seven people, and Paris. And I'll take Paris over the seven. But I don't watch CNN, so I don't get to see you as much as I used to. I don't like watching fake news. But Fox has treated me very nice. Wherever Fox is, thank you.We're gonna need better schools and we need them soon. We need more jobs, we need better wages, a lot better wages. We're gonna work very hard on the inner city. Ben is gonna be doing that, big league. That's one of the big things that you're gonna be looking at. We need safer communities and we're going to do that with law enforcement. We're gonna make it safe. We're gonna make it much better than it is right now. Right now it's terrible, and I saw you talking about it the other night, Paris, on something else that was really'you did a fantastic job the other night on a very unrelated show.I'm ready to do my part, and I will say this: We're gonna work together. This is a great group, this is a group that's been so special to me. You really helped me a lot. If you remember I wasn't going to do well with the African-American community, and after they heard me speaking and talking about the inner city and lots of other things, we ended up getting'and I won't go into details'but we ended up getting substantially more than other candidates who had run in the past years. And now we're gonna take that to new levels. I want to thank my television star over here'Omarosa's actually a very nice person, nobody knows that. I don't want to destroy her reputation but she's a very good person, and she's been helpful right from the beginning of the campaign, and I appreciate it. I really do. Very special.So I want to thank everybody for being here. 18 | 19 | Thank you.Well. I want to thank everybody. Very, very special people. And it is true: this is my first stop. Officially. We’re not talking about the balls, and we’re not talking about even the speeches. Although, they did treat me nicely on that speech yesterday [laughter].I always call them “the dishonest media”, but they treated me nicely.But, I want to say that there is nobody that feels stronger about the Intelligence Community and the CIA than Donald Trump. [applause]. There’s Nobody. Nobody.And the wall behind me is very very special. We’ve been touring for quite a while. And I’ll tell you what: twenty … nine? I can’t believe it.. No. Twenty eight. We’ve got to reduce it. That’s amazing. And we really appreciate it what you ‘ve done in terms of showing us something very special. And your whole group. These are really special, amazing people. Very. very few people could do the job you people do.And I want to just let you know: I am so behind you. And I know, maybe sometimes, you haven’t gotten the backing that you’ve wanted. And you’re going to get so much backing. Maybe you’re going to say “please don’t give us so much backing”. [laughter] “Mr President, please, we don’t need that much backing”.But you’re going to have that. And I think everybody in this room knows it.You know, the military, and the law-enforcement generally speaking, -- but, all of it -- but the military, gave us tremendous percentages of votes. We were unbelievably successful in the election with getting the vote of the military and probably almost everybody in this room voted for me, but I will not ask you to raise your hands if you did. [laughter]But I would guarantee a big portion. Because we’re all on the same wavelength, folks. We’re all on the same wavelength. [applause] Alight? [pointing to the crowd] He knows. Took Brian about 30 seconds to figure that one out, right? Because we know. We’re on the same wavelength.We’re going to do great things. We’re going to do great things. We’ve been fighting these wars for longer than any wars we’ve ever fought. We have not used the real abilities that we have. We’ve been restrained.We have to get rid of ISIS. We have to get rid of ISIS. We have no choice [applause]Radical Islamic terrorism - and I said it yesterday - has to be eradicated. Just off the face of the Earth. This is evil. This is evil.And you know, I can understand the other side. We can all understand the other side. There can be wars between countries. There can be wars. You can understand what happened. This is something nobody could even understand. This is a level of evil that we haven’t seen.You’re going to go to it, and you’re going to do a phenomenal job. But we’re going to end it. It’s time. It’s time right now to end it.You have somebody coming on who is extraordinary. You know for the different positions, of secretary of this and secretary of that and all of these great positions, I’d see five, six, seven, eight people.And we had a great transition. We had an amazing team of talent.And by the way, General Flynn is right over here. Put up your hand, Mike. What a good guy [applause]And Reince, and my whole group. Reince. You know Reince? They don’t care about Reince. He’s like, this political guy that turned out to be a superstar, right? We don’t have to talk about Reince.But, we did. We had just such a tremendous, tremendous success.So when I’m interviewing all of these candidates that Reince and his whole group is putting in front, it went very, very quickly, and in this case went so quickly. Because I would see six or seven or eight for secretary of agriculture, who we just named the other day. Sunny Perdue. Former Governor of Georgia. Fantastic guy. But I’d see six, seven, eight people for a certain position. Everybody wanted it.But I met Mike Pompeo, and he was the only guy I met. I didn’t want to meet anybody else. I said “cancel everybody else”. Cancel. Now he was approved, essentially. But they’re doing a little political games with me. You know, he was one of the three.Now, last night, as you know, General Mattis - fantastic guy - and General Kelly got approved [applause]And Mike Pompeo was supposed to be in that group; it was going to be the three of them. Can you imagine? All of these guys. People respect … they respect that military sense. All my political people? They’re not doing so well. The political people aren’t doing so well… but you … We’re going to get them all through. But some will take a little bit longer than others.But Mike was literally -- I had a group of, what, we had nine different people? -- Now. I must say, I didn’t mind cancelling eight appointments. That wasn’t the worst thing in the world.But I met him, and I said “he is so good”. Number one in his class at West Point. Now, I know a lot about West Point. I’m a person that very strongly believes in academics. In fact, every time I say, I had an uncle who was a great professor at MIT for 35 years, who did a fantastic job in so many different ways academically. He was an academic genius.And then they say: “is Donald Trump an intellectual?” Trust me. I’m like a smart person. [laughter] [pointing at Mike Pompeo] And I recognized immediately,So he was Number 1 at West Point. And he was also essentially number 1 at Harvard Law School. And then he decided to go into the military. And he ran for Congress. And everything he’s done has been a home run.People like him. But much more importantly to me, everybody respects him.When I told Paul Ryan that I want to do this, I would say, he may be the only person that was not totally thrilled, right, Mike? Because he said “I don’t want to lose this guy”.You will be getting a total star. You going to be getting a total gem. He is a gem. And I just …. [applause] You’ll see. You’ll see. And many of you know him anyway. But you’re going to see.And again: we have some great people going, but this one is something, going to be very special, because this is one of -- if I had to name the most important, this would certainly be, perhaps, you know, in certain ways, you could even say my most important.You do the job like everybody in this room is capable of doing.And the generals are wonderful and the fighting is wonderful. But if you give them the right direction? Boy does the fighting become easier. And boy do we lose so fewer lives, and win so … quickly.And that’s what we have to do. We have to start winning again.You know what? When I was young, And when I was … of course, I feel young. I feel like I’m 30. 35. 39. [laughter]. Somebody said “are you young?” I said “I think I’m young”.You know, I was stopping when we were in the final month of that campaign. Four stops, five stops. Seven stops. Speeches -- speeches -- in front of twenty five, thirty thousand people. Fifteen thousand, nineteen thousand, from stop to stop.I feel young.But when I was young -- and I think we’re all sort of young -- when I was young, we were always winning things in this country. We’d win with trade. We’d win with wars.At a certain age I remember hearing from one of my instructors “The United States has never lost a war”.And then, after that, it’s like, we haven’t won anything. We don’t win anymore.,The old expression: “to the victor belong the spoils” - you remember? You always used to say “keep the oil”. I wasn’t a fan of Iraq. I didn’t want to go into Iraq. But I will tell you. When we were in, we got out wrong.And I always said: “In addition to that, keep the oil”.Now I said it for economic reasons, but if you think about, Mike, if we kept the oil we would probably wouldn’t have ISIS, because that’s where they made their money in the first place. So we should have kept the oil.But okay. [laughter] Maybe we’ll have another chance.But the fact is: we should’ve kept the oil. I believe that this group is going to be one of the most important groups in this country towards making us safe, towards making us winners again. Towards ending all of the problems -- we have so many problems that are interrelated that we don’t even think of, but interrelated -- to the kind of havoc and fear that this sick group of people has caused.So I can only say that I am with you 1000%. And the reason you’re my first stop is that as you know, I have a running war with the media. They are among the most dishonest human beings on Earth. [laughter, applause]And they sort of made it sound like I had a feud with the Intelligence Community. And I just want to let you know, the reason you’re the number 1 stop is exactly the opposite. Exactly. And they understand that too.And I was explaining about the numbers. We did a thing yesterday, the speech, and everybody really liked the speech, you had to right? [applause]We had a massive field of people. You saw that. Packed.I get up this morning. I turn on one of the networks and they show an empty field. I say: “wait a minute. I made a speech. I looked out. The field was…. It looked like a million, a million and a half people.” They showed a field where there was practically nobody standing there. And they said “Donald Trump did not draw well”. And I said “well it was almost raining”. The rain should have scared them away. But God looked down and he said “we’re not going to let it rain on your speech”.In fact, when I first started I said “oh no”. First line, I got hit by a couple of drops. And i said “oh, this is too bad, but we’ll go right through it”. But the truth is: that it stopped immediately. It was amazing. And then it became really sudden, and then I walked off and it poured right after I left - it poured.But you know, we have something that’s amazing because, we had, it looked honestly, it looked like a million and a half people. Whatever it was. But it went all the way back to the Washington Monument.And I turn on, with my steak … and I get this network shows an empty field. And it said we drew 250,000 people.Now that’s not bad. But it’s a lie. We had 250,000 people literally around, you know, the little bowl that we constructed. That was 250,000 people. The rest of the 20 block area all the way back to the Washington Monument was packed.So we caught them. And we caught them in a beauty. And I think they’re going to pay a big price.They had another one yesterday which was interesting. In the Oval Office there’s a beautiful statue of Dr Martin Luther King. And I also happen to like Churchill. Winston Churchill. I think most of us like Churchill. He doesn’t come from our country. But he had lot to do with it. He helped us. A real ally.And as you know, the Churchill statue was taken out. The bust. And as you probably also have read, the Prime Minister is coming over to our country very shortly, and they wanted to know whether or not I’d like it back. And I said “absolutely, but in the meantime we have a bust of Churchill”.So a reporter for Time magazine. And I have been on their cover like 14 or 15 times. I think we have the all time record in the history of Time magazine. Like it Tom Brady is on the cover of Time magazine, it’s one time, because he won the Superbowl or something, right? [laughter]. I’ve been on for 15 times this year.I don’t think that’s a record, Mike, that they can ever be broken, do you agree with that? What do you think?But I will say that, he said something that was very interesting: that “Donald Trump took down the bust, the statue, of Dr Martin Luther King”. It was right there. But there was a cameraman that was in front of it.So Zeke - Zeke - from Time magazine writes a story about how I took it down. But I would never do that, because I have great respect for Dr Martin Luther King. But this is how dishonest the media is: a big story. And the retraction was like -- was it a line? Or did they even bother putting it in?So I only like to say that because I love honesty. I like honest reporting. I will tell you the final time: although I will say it, when you let in your thousands of other people that had been trying to come in, because I am coming back.We may have to get you a larger room. [laughter, applause] We may have to get you a larger room.And maybe - maybe - it’ll be built by somebody that knows how to build and we won’t have columns [laughter] You understand that? We’d get rid of the columns.I just wanted to really say that I love you. I respect you. There’s nobody that I respect more. You’re going to do a fantastic job. And we’re going to start winning again. And you’re going to be leading the charge.So thank you all very much. Thank you, beautiful. Thank you all very much.Have a good day.I’ll be back. I’ll be back. Thank you. 20 | 21 | -------------------------------------------------------------------------------- /notebooks/models/trump/architecture.yaml: -------------------------------------------------------------------------------- 1 | backend: tensorflow 2 | class_name: Model 3 | config: 4 | input_layers: 5 | - [X, 0, 0] 6 | layers: 7 | - class_name: InputLayer 8 | config: 9 | batch_input_shape: !!python/tuple [null, 50, 86] 10 | dtype: float32 11 | name: X 12 | sparse: false 13 | inbound_nodes: [] 14 | name: X 15 | - class_name: LSTM 16 | config: 17 | activation: tanh 18 | activity_regularizer: null 19 | bias_constraint: null 20 | bias_initializer: 21 | class_name: Zeros 22 | config: {} 23 | bias_regularizer: null 24 | dropout: 0.0 25 | go_backwards: false 26 | implementation: 1 27 | kernel_constraint: null 28 | kernel_initializer: 29 | class_name: VarianceScaling 30 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 31 | kernel_regularizer: null 32 | name: LSTM1 33 | recurrent_activation: hard_sigmoid 34 | recurrent_constraint: null 35 | recurrent_dropout: 0.0 36 | recurrent_initializer: 37 | class_name: Orthogonal 38 | config: {gain: 1.0, seed: null} 39 | recurrent_regularizer: null 40 | return_sequences: true 41 | return_state: false 42 | stateful: false 43 | trainable: true 44 | unit_forget_bias: true 45 | units: 256 46 | unroll: false 47 | use_bias: true 48 | inbound_nodes: 49 | - - - X 50 | - 0 51 | - 0 52 | - {} 53 | name: LSTM1 54 | - class_name: Dropout 55 | config: {name: DROP1, noise_shape: null, rate: 0.2, seed: null, trainable: true} 56 | inbound_nodes: 57 | - - - LSTM1 58 | - 0 59 | - 0 60 | - {} 61 | name: DROP1 62 | - class_name: Dense 63 | config: 64 | activation: softmax 65 | activity_regularizer: null 66 | bias_constraint: null 67 | bias_initializer: 68 | class_name: Zeros 69 | config: {} 70 | bias_regularizer: null 71 | kernel_constraint: null 72 | kernel_initializer: 73 | class_name: VarianceScaling 74 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 75 | kernel_regularizer: null 76 | name: Y 77 | trainable: true 78 | units: 86 79 | use_bias: true 80 | inbound_nodes: 81 | - - - DROP1 82 | - 0 83 | - 0 84 | - {} 85 | name: Y 86 | name: model_5 87 | output_layers: 88 | - [Y, 0, 0] 89 | keras_version: 2.1.4 90 | backend: tensorflow 91 | class_name: Model 92 | config: 93 | input_layers: 94 | - [X, 0, 0] 95 | layers: 96 | - class_name: InputLayer 97 | config: 98 | batch_input_shape: !!python/tuple [null, 50, 86] 99 | dtype: float32 100 | name: X 101 | sparse: false 102 | inbound_nodes: [] 103 | name: X 104 | - class_name: LSTM 105 | config: 106 | activation: tanh 107 | activity_regularizer: null 108 | bias_constraint: null 109 | bias_initializer: 110 | class_name: Zeros 111 | config: {} 112 | bias_regularizer: null 113 | dropout: 0.0 114 | go_backwards: false 115 | implementation: 1 116 | kernel_constraint: null 117 | kernel_initializer: 118 | class_name: VarianceScaling 119 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 120 | kernel_regularizer: null 121 | name: LSTM1 122 | recurrent_activation: hard_sigmoid 123 | recurrent_constraint: null 124 | recurrent_dropout: 0.0 125 | recurrent_initializer: 126 | class_name: Orthogonal 127 | config: {gain: 1.0, seed: null} 128 | recurrent_regularizer: null 129 | return_sequences: true 130 | return_state: false 131 | stateful: false 132 | trainable: true 133 | unit_forget_bias: true 134 | units: 256 135 | unroll: false 136 | use_bias: true 137 | inbound_nodes: 138 | - - - X 139 | - 0 140 | - 0 141 | - {} 142 | name: LSTM1 143 | - class_name: Dropout 144 | config: {name: DROP1, noise_shape: null, rate: 0.2, seed: null, trainable: true} 145 | inbound_nodes: 146 | - - - LSTM1 147 | - 0 148 | - 0 149 | - {} 150 | name: DROP1 151 | - class_name: Dense 152 | config: 153 | activation: softmax 154 | activity_regularizer: null 155 | bias_constraint: null 156 | bias_initializer: 157 | class_name: Zeros 158 | config: {} 159 | bias_regularizer: null 160 | kernel_constraint: null 161 | kernel_initializer: 162 | class_name: VarianceScaling 163 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 164 | kernel_regularizer: null 165 | name: Y 166 | trainable: true 167 | units: 86 168 | use_bias: true 169 | inbound_nodes: 170 | - - - DROP1 171 | - 0 172 | - 0 173 | - {} 174 | name: Y 175 | name: model_5 176 | output_layers: 177 | - [Y, 0, 0] 178 | keras_version: 2.1.4 179 | backend: tensorflow 180 | class_name: Model 181 | config: 182 | input_layers: 183 | - [X, 0, 0] 184 | layers: 185 | - class_name: InputLayer 186 | config: 187 | batch_input_shape: !!python/tuple [null, 50, 86] 188 | dtype: float32 189 | name: X 190 | sparse: false 191 | inbound_nodes: [] 192 | name: X 193 | - class_name: LSTM 194 | config: 195 | activation: tanh 196 | activity_regularizer: null 197 | bias_constraint: null 198 | bias_initializer: 199 | class_name: Zeros 200 | config: {} 201 | bias_regularizer: null 202 | dropout: 0.0 203 | go_backwards: false 204 | implementation: 1 205 | kernel_constraint: null 206 | kernel_initializer: 207 | class_name: VarianceScaling 208 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 209 | kernel_regularizer: null 210 | name: LSTM1 211 | recurrent_activation: hard_sigmoid 212 | recurrent_constraint: null 213 | recurrent_dropout: 0.0 214 | recurrent_initializer: 215 | class_name: Orthogonal 216 | config: {gain: 1.0, seed: null} 217 | recurrent_regularizer: null 218 | return_sequences: true 219 | return_state: false 220 | stateful: false 221 | trainable: true 222 | unit_forget_bias: true 223 | units: 256 224 | unroll: false 225 | use_bias: true 226 | inbound_nodes: 227 | - - - X 228 | - 0 229 | - 0 230 | - {} 231 | name: LSTM1 232 | - class_name: Dropout 233 | config: {name: DROP1, noise_shape: null, rate: 0.2, seed: null, trainable: true} 234 | inbound_nodes: 235 | - - - LSTM1 236 | - 0 237 | - 0 238 | - {} 239 | name: DROP1 240 | - class_name: Dense 241 | config: 242 | activation: softmax 243 | activity_regularizer: null 244 | bias_constraint: null 245 | bias_initializer: 246 | class_name: Zeros 247 | config: {} 248 | bias_regularizer: null 249 | kernel_constraint: null 250 | kernel_initializer: 251 | class_name: VarianceScaling 252 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 253 | kernel_regularizer: null 254 | name: Y 255 | trainable: true 256 | units: 86 257 | use_bias: true 258 | inbound_nodes: 259 | - - - DROP1 260 | - 0 261 | - 0 262 | - {} 263 | name: Y 264 | name: model_5 265 | output_layers: 266 | - [Y, 0, 0] 267 | keras_version: 2.1.4 268 | backend: tensorflow 269 | class_name: Model 270 | config: 271 | input_layers: 272 | - [X, 0, 0] 273 | layers: 274 | - class_name: InputLayer 275 | config: 276 | batch_input_shape: !!python/tuple [null, 50, 86] 277 | dtype: float32 278 | name: X 279 | sparse: false 280 | inbound_nodes: [] 281 | name: X 282 | - class_name: LSTM 283 | config: 284 | activation: tanh 285 | activity_regularizer: null 286 | bias_constraint: null 287 | bias_initializer: 288 | class_name: Zeros 289 | config: {} 290 | bias_regularizer: null 291 | dropout: 0.0 292 | go_backwards: false 293 | implementation: 1 294 | kernel_constraint: null 295 | kernel_initializer: 296 | class_name: VarianceScaling 297 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 298 | kernel_regularizer: null 299 | name: LSTM1 300 | recurrent_activation: hard_sigmoid 301 | recurrent_constraint: null 302 | recurrent_dropout: 0.0 303 | recurrent_initializer: 304 | class_name: Orthogonal 305 | config: {gain: 1.0, seed: null} 306 | recurrent_regularizer: null 307 | return_sequences: true 308 | return_state: false 309 | stateful: false 310 | trainable: true 311 | unit_forget_bias: true 312 | units: 256 313 | unroll: false 314 | use_bias: true 315 | inbound_nodes: 316 | - - - X 317 | - 0 318 | - 0 319 | - {} 320 | name: LSTM1 321 | - class_name: Dropout 322 | config: {name: DROP1, noise_shape: null, rate: 0.2, seed: null, trainable: true} 323 | inbound_nodes: 324 | - - - LSTM1 325 | - 0 326 | - 0 327 | - {} 328 | name: DROP1 329 | - class_name: Dense 330 | config: 331 | activation: softmax 332 | activity_regularizer: null 333 | bias_constraint: null 334 | bias_initializer: 335 | class_name: Zeros 336 | config: {} 337 | bias_regularizer: null 338 | kernel_constraint: null 339 | kernel_initializer: 340 | class_name: VarianceScaling 341 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 342 | kernel_regularizer: null 343 | name: Y 344 | trainable: true 345 | units: 86 346 | use_bias: true 347 | inbound_nodes: 348 | - - - DROP1 349 | - 0 350 | - 0 351 | - {} 352 | name: Y 353 | name: model_5 354 | output_layers: 355 | - [Y, 0, 0] 356 | keras_version: 2.1.4 357 | backend: tensorflow 358 | class_name: Model 359 | config: 360 | input_layers: 361 | - [INPUT, 0, 0] 362 | layers: 363 | - class_name: InputLayer 364 | config: 365 | batch_input_shape: !!python/tuple [null, 50, 86] 366 | dtype: float32 367 | name: INPUT 368 | sparse: false 369 | inbound_nodes: [] 370 | name: INPUT 371 | - class_name: LSTM 372 | config: 373 | activation: tanh 374 | activity_regularizer: null 375 | bias_constraint: null 376 | bias_initializer: 377 | class_name: Zeros 378 | config: {} 379 | bias_regularizer: null 380 | dropout: 0.0 381 | go_backwards: false 382 | implementation: 1 383 | kernel_constraint: null 384 | kernel_initializer: 385 | class_name: VarianceScaling 386 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 387 | kernel_regularizer: null 388 | name: LSTM1 389 | recurrent_activation: hard_sigmoid 390 | recurrent_constraint: null 391 | recurrent_dropout: 0.0 392 | recurrent_initializer: 393 | class_name: Orthogonal 394 | config: {gain: 1.0, seed: null} 395 | recurrent_regularizer: null 396 | return_sequences: true 397 | return_state: false 398 | stateful: false 399 | trainable: true 400 | unit_forget_bias: true 401 | units: 256 402 | unroll: false 403 | use_bias: true 404 | inbound_nodes: 405 | - - - INPUT 406 | - 0 407 | - 0 408 | - {} 409 | name: LSTM1 410 | - class_name: Dropout 411 | config: {name: DROP1, noise_shape: null, rate: 0.2, seed: null, trainable: true} 412 | inbound_nodes: 413 | - - - LSTM1 414 | - 0 415 | - 0 416 | - {} 417 | name: DROP1 418 | - class_name: Dense 419 | config: 420 | activation: softmax 421 | activity_regularizer: null 422 | bias_constraint: null 423 | bias_initializer: 424 | class_name: Zeros 425 | config: {} 426 | bias_regularizer: null 427 | kernel_constraint: null 428 | kernel_initializer: 429 | class_name: VarianceScaling 430 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 431 | kernel_regularizer: null 432 | name: OUTPUT 433 | trainable: true 434 | units: 86 435 | use_bias: true 436 | inbound_nodes: 437 | - - - DROP1 438 | - 0 439 | - 0 440 | - {} 441 | name: OUTPUT 442 | name: model_6 443 | output_layers: 444 | - [OUTPUT, 0, 0] 445 | keras_version: 2.1.4 446 | backend: tensorflow 447 | class_name: Sequential 448 | config: 449 | - class_name: LSTM 450 | config: 451 | activation: tanh 452 | activity_regularizer: null 453 | batch_input_shape: !!python/tuple [null, 50, 86] 454 | bias_constraint: null 455 | bias_initializer: 456 | class_name: Zeros 457 | config: {} 458 | bias_regularizer: null 459 | dropout: 0.0 460 | dtype: float32 461 | go_backwards: false 462 | implementation: 1 463 | kernel_constraint: null 464 | kernel_initializer: 465 | class_name: VarianceScaling 466 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 467 | kernel_regularizer: null 468 | name: LSTM 469 | recurrent_activation: hard_sigmoid 470 | recurrent_constraint: null 471 | recurrent_dropout: 0.0 472 | recurrent_initializer: 473 | class_name: Orthogonal 474 | config: {gain: 1.0, seed: null} 475 | recurrent_regularizer: null 476 | return_sequences: true 477 | return_state: false 478 | stateful: false 479 | trainable: true 480 | unit_forget_bias: true 481 | units: 256 482 | unroll: false 483 | use_bias: true 484 | - class_name: Dense 485 | config: 486 | activation: softmax 487 | activity_regularizer: null 488 | bias_constraint: null 489 | bias_initializer: 490 | class_name: Zeros 491 | config: {} 492 | bias_regularizer: null 493 | kernel_constraint: null 494 | kernel_initializer: 495 | class_name: VarianceScaling 496 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 497 | kernel_regularizer: null 498 | name: Y 499 | trainable: true 500 | units: 86 501 | use_bias: true 502 | keras_version: 2.1.4 503 | backend: tensorflow 504 | class_name: Sequential 505 | config: 506 | - class_name: LSTM 507 | config: 508 | activation: tanh 509 | activity_regularizer: null 510 | batch_input_shape: !!python/tuple [null, 50, 86] 511 | bias_constraint: null 512 | bias_initializer: 513 | class_name: Zeros 514 | config: {} 515 | bias_regularizer: null 516 | dropout: 0.0 517 | dtype: float32 518 | go_backwards: false 519 | implementation: 1 520 | kernel_constraint: null 521 | kernel_initializer: 522 | class_name: VarianceScaling 523 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 524 | kernel_regularizer: null 525 | name: lstm_3 526 | recurrent_activation: hard_sigmoid 527 | recurrent_constraint: null 528 | recurrent_dropout: 0.0 529 | recurrent_initializer: 530 | class_name: Orthogonal 531 | config: {gain: 1.0, seed: null} 532 | recurrent_regularizer: null 533 | return_sequences: false 534 | return_state: false 535 | stateful: false 536 | trainable: true 537 | unit_forget_bias: true 538 | units: 256 539 | unroll: false 540 | use_bias: true 541 | - class_name: Dense 542 | config: 543 | activation: softmax 544 | activity_regularizer: null 545 | bias_constraint: null 546 | bias_initializer: 547 | class_name: Zeros 548 | config: {} 549 | bias_regularizer: null 550 | kernel_constraint: null 551 | kernel_initializer: 552 | class_name: VarianceScaling 553 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 554 | kernel_regularizer: null 555 | name: dense_3 556 | trainable: true 557 | units: 86 558 | use_bias: true 559 | keras_version: 2.1.4 560 | backend: tensorflow 561 | class_name: Sequential 562 | config: 563 | - class_name: LSTM 564 | config: 565 | activation: tanh 566 | activity_regularizer: null 567 | batch_input_shape: !!python/tuple [null, 50, 86] 568 | bias_constraint: null 569 | bias_initializer: 570 | class_name: Zeros 571 | config: {} 572 | bias_regularizer: null 573 | dropout: 0.0 574 | dtype: float32 575 | go_backwards: false 576 | implementation: 1 577 | kernel_constraint: null 578 | kernel_initializer: 579 | class_name: VarianceScaling 580 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 581 | kernel_regularizer: null 582 | name: LSTM 583 | recurrent_activation: hard_sigmoid 584 | recurrent_constraint: null 585 | recurrent_dropout: 0.0 586 | recurrent_initializer: 587 | class_name: Orthogonal 588 | config: {gain: 1.0, seed: null} 589 | recurrent_regularizer: null 590 | return_sequences: false 591 | return_state: false 592 | stateful: false 593 | trainable: true 594 | unit_forget_bias: true 595 | units: 256 596 | unroll: false 597 | use_bias: true 598 | - class_name: Dense 599 | config: 600 | activation: softmax 601 | activity_regularizer: null 602 | bias_constraint: null 603 | bias_initializer: 604 | class_name: Zeros 605 | config: {} 606 | bias_regularizer: null 607 | kernel_constraint: null 608 | kernel_initializer: 609 | class_name: VarianceScaling 610 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 611 | kernel_regularizer: null 612 | name: Y 613 | trainable: true 614 | units: 86 615 | use_bias: true 616 | keras_version: 2.1.4 617 | backend: tensorflow 618 | class_name: Sequential 619 | config: 620 | - class_name: LSTM 621 | config: 622 | activation: tanh 623 | activity_regularizer: null 624 | batch_input_shape: !!python/tuple [null, 50, 86] 625 | bias_constraint: null 626 | bias_initializer: 627 | class_name: Zeros 628 | config: {} 629 | bias_regularizer: null 630 | dropout: 0.0 631 | dtype: float32 632 | go_backwards: false 633 | implementation: 1 634 | kernel_constraint: null 635 | kernel_initializer: 636 | class_name: VarianceScaling 637 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 638 | kernel_regularizer: null 639 | name: LSTM 640 | recurrent_activation: hard_sigmoid 641 | recurrent_constraint: null 642 | recurrent_dropout: 0.0 643 | recurrent_initializer: 644 | class_name: Orthogonal 645 | config: {gain: 1.0, seed: null} 646 | recurrent_regularizer: null 647 | return_sequences: false 648 | return_state: false 649 | stateful: false 650 | trainable: true 651 | unit_forget_bias: true 652 | units: 256 653 | unroll: false 654 | use_bias: true 655 | - class_name: Dense 656 | config: 657 | activation: softmax 658 | activity_regularizer: null 659 | bias_constraint: null 660 | bias_initializer: 661 | class_name: Zeros 662 | config: {} 663 | bias_regularizer: null 664 | kernel_constraint: null 665 | kernel_initializer: 666 | class_name: VarianceScaling 667 | config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null} 668 | kernel_regularizer: null 669 | name: Y 670 | trainable: true 671 | units: 86 672 | use_bias: true 673 | keras_version: 2.1.4 674 | -------------------------------------------------------------------------------- /notebooks/models/trump/weights-01-2.796.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/models/trump/weights-01-2.796.hdf5 -------------------------------------------------------------------------------- /notebooks/models/trump/weights-02-2.328.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/models/trump/weights-02-2.328.hdf5 -------------------------------------------------------------------------------- /notebooks/models/trump/weights-03-2.152.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Imperial-College-Data-Science-Society/Machine-Learning-API/4e27b154f22435143700d66018d73ea0d8929694/notebooks/models/trump/weights-03-2.152.hdf5 -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | pandas 3 | scipy 4 | matplotlib 5 | seaborn 6 | scikit-learn 7 | keras 8 | tensorflow 9 | fbprophet 10 | jupyter 11 | h5py 12 | pandas-datareader 13 | -------------------------------------------------------------------------------- /scripts/clean.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # clean environemnt from objects and cache files 4 | find . -name "*.pyc" -type f -delete 5 | find . -name "__pycache__" -type d -delete 6 | find . -name "*.ipynb*" -type d -delete -------------------------------------------------------------------------------- /scripts/remove.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # virtual environemnt directory name 4 | ENV=".env" 5 | # python version 6 | # python3 --> 3.6 | python2 --> 2.7 7 | PYTHON="python3" 8 | # pip version 9 | # pip3 --> python3 | pip2 --> python2 10 | PIP="pip3" 11 | 12 | # remove virtual environment 13 | if [ -d ${ENV} ] ; then 14 | rm -rf ${ENV} ; 15 | fi -------------------------------------------------------------------------------- /scripts/setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # virtual environemnt directory name 4 | ENV=".env" 5 | # python version 6 | # python3 --> 3.6 | python2 --> 2.7 7 | PYTHON="python3" 8 | # pip version 9 | # pip3 --> python3 | pip2 --> python2 10 | PIP="pip3" 11 | 12 | # remove virtual environment 13 | source scripts/remove.sh 14 | # install `virtualenv` to global pip 15 | ${PIP} install virtualenv 16 | # create virtual environment 17 | virtualenv -p ${PYTHON} ${ENV} 18 | # activate virtual environment 19 | source "./${ENV}/bin/activate" 20 | # install dependencies to virtual environment 21 | pip install -r requirements.txt 22 | # run tests 23 | source scripts/test.sh -------------------------------------------------------------------------------- /scripts/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # virtual environemnt directory name 4 | ENV=".env" 5 | # python version 6 | # python3 --> 3.6 | python2 --> 2.7 7 | PYTHON="python3" 8 | # pip version 9 | # pip3 --> python3 | pip2 --> python2 10 | PIP="pip3" 11 | 12 | # check if virtual environment is setup 13 | if [ ! -d ${ENV} ] 14 | then 15 | # setup virtual environemnt 16 | source scripts/setup.sh 17 | fi 18 | # activate virtual environment 19 | source "./${ENV}/bin/activate" 20 | # run tests using `unittest` 21 | python -m unittest tests/* 22 | # clean environemnt 23 | source scripts/clean.sh -------------------------------------------------------------------------------- /tests/setup.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | 4 | class TestSetup(unittest.TestCase): 5 | """Test Dependencies from `requirements.txt`.""" 6 | 7 | def test__sys(self): 8 | import sys 9 | return self.assertIsNotNone(sys) 10 | 11 | 12 | if __name__ == '__main__': 13 | unittest.main() 14 | --------------------------------------------------------------------------------