├── README.md ├── 10_Final_Thoughts_and_Further_Practice.ipynb ├── Practice ├── Hard_Titanic_Kaggle.ipynb ├── Moderate_Intel_Image_Classification.ipynb ├── Moderate_Wine.ipynb └── Easy_Iris_Starter.ipynb ├── 03b_Baysian.ipynb ├── 04c_Permutation_Importance.ipynb ├── DeployingModels.ipynb ├── 08a_Callbacks.ipynb └── 05b_State_of_the_Art.ipynb /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Practical Deep Learning for Coders 4 | Material for my Proctor of Fast.AI's course. This course is originally done by Jeremy Howard and Rachel Thomas, and is taught at the University by course alumni. 5 | 6 | The course can be found here: https://course.fast.ai 7 | 8 | The Fast.AI forums can be found at: https://forums.fast.ai 9 | -------------------------------------------------------------------------------- /10_Final_Thoughts_and_Further_Practice.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "10_Final_Thoughts_and_Further_Practice.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | } 14 | }, 15 | "cells": [ 16 | { 17 | "cell_type": "markdown", 18 | "metadata": { 19 | "id": "pUR2kjYnTcAH", 20 | "colab_type": "text" 21 | }, 22 | "source": [ 23 | "# Final Thoughts and Further Practice\n", 24 | "\n", 25 | "Ideas for further workshops:\n", 26 | "\n", 27 | "* Productionizing Models?\n", 28 | "* Tabular Models?\n", 29 | "\n", 30 | "In the github is a 'Practice' folder: [practice](https://github.com/UWF-AIRG/Practical-Deep-Learning-For-Coders/tree/master/Practice) where a few notebooks are stored containing setups in order to do practice problems.\n", 31 | "\n", 32 | "The problems are split up like so:\n", 33 | "\n", 34 | "**Tabular:**\n", 35 | "\n", 36 | "* Iris - Easy, \"The Iris Flowers Dataset involves predicting the flower species given measurements of iris flowers.\"\n", 37 | "* Wine - Moderate, \"This is a regression problem. There are 4,898 observations with 11 input variables and one output variable\"\n", 38 | "* Titanic - Hard, \"With this dataset, you will need to use to the best of your ability feature engineering in order to help figure out who survived the titanic.\"\n", 39 | "\n", 40 | "**Image Classification:**\n", 41 | "* Intel Image Classification - Moderate, \"This is image data of Natural Scenes around the world.\"\n", 42 | "\n", 43 | "**Image Segmentation:**\n", 44 | "* Kaggle: [SIIM ACR Pneumothorax Segmentation](https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation/data)\n", 45 | "\n", 46 | "**NLP**\n", 47 | "* Kaggle: [Jigsaw Toxic Comment Classification Challange](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)\n", 48 | "\n", 49 | "\n", 50 | "## From Here:\n", 51 | "\n", 52 | "First and foremost, are you apart of the [forums](forums.fast.ai)? If not, you should be! There is plenty of reading material on here and is\n", 53 | "a wonderful community full of ML practitioners from very beginner to Jeremy-Howard level. Join, start reading and looking into some topics you want to do,\n", 54 | "and I guarentee you there's at least one post from one person who's interested in the same thing!\n", 55 | "\n", 56 | "Now, class-wise. You have a few options from here. You can call this enough, and you have the skills to do some cool things. If you want more,\n", 57 | "Fast.AI has even more courses:\n", 58 | "* [Practical Deep Learning for Coders](https://course.fast.ai/#) (has a few topics I could not cover, and goes into much more depth and complexity in them)\n", 59 | "* [Deep Learning from the Foundations](https://course.fast.ai/part2) **new and will be taught here in the Fall**\n", 60 | "* [Matrix Calculus for Deep Learning](https://explained.ai/matrix-calculus/index.html)\n", 61 | "* [Computational Linear Algebra](https://github.com/fastai/numerical-linear-algebra)\n", 62 | "* [Natural Langage Processing Course](https://github.com/fastai/course-nlp) **new**\n", 63 | "\n", 64 | "Also, Jeremy recommends you start blogging! Whatever work you do! Find a small project, do something, write it. Learn some chunk fo the library really well,\n", 65 | "write a detailed blog with what it does and how it works. It helps cement your knowledge of the subject and helps others in their understanding of it!\n", 66 | "\n", 67 | "There is a *lot* you can learn by digging into the source code and the reasoning behind a few of the papers:\n", 68 | "\n", 69 | "[Leslie Smith Fast.AI Forums Article](https://forums.fast.ai/t/jupyter-notebook-explaining-the-4-papers-by-leslie-n-smith/47243)\n", 70 | "\n", 71 | "## Some things I have done personally to get more situated with the library:\n", 72 | "* People post ideas or things they would like to see all the time on the forums, which can be solved fairly easily with the standard Fast.AI library and restructuring the code a bit\n", 73 | "For an example, see [this discussion](https://forums.fast.ai/t/wanting-to-plot-most-confused-classes/49899)\n", 74 | "\n", 75 | "\n" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "metadata": { 81 | "id": "Uj3iyva8rudw", 82 | "colab_type": "code", 83 | "colab": {} 84 | }, 85 | "source": [ 86 | "" 87 | ], 88 | "execution_count": 0, 89 | "outputs": [] 90 | } 91 | ] 92 | } -------------------------------------------------------------------------------- /Practice/Hard_Titanic_Kaggle.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Hard: Titanic - Kaggle.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "collapsed_sections": [] 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "accelerator": "GPU" 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "CiXYiVBh9bDV", 22 | "colab_type": "text" 23 | }, 24 | "source": [ 25 | "# Titanic\n", 26 | "\n", 27 | "With this dataset, you will need to use to the best of your ability feature engineering in order to help figure out who survived the titanic. The data is available here: https://www.kaggle.com/c/titanic/data\n", 28 | "\n", 29 | "You need to make a Kaggle account to download the train and test.csv documents" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": { 35 | "id": "OkUWolBm90KI", 36 | "colab_type": "text" 37 | }, 38 | "source": [ 39 | "# Get Data" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": { 45 | "id": "CglRbTM89_YP", 46 | "colab_type": "text" 47 | }, 48 | "source": [ 49 | "# Training" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "metadata": { 55 | "id": "aXcdzue6Elhm", 56 | "colab_type": "code", 57 | "colab": {} 58 | }, 59 | "source": [ 60 | "from sklearn.model_selection import train_test_split\n", 61 | "train, test = train_test_split(train_df, test_size=0.1)" 62 | ], 63 | "execution_count": 0, 64 | "outputs": [] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": { 69 | "id": "LhaawGxxEqYH", 70 | "colab_type": "text" 71 | }, 72 | "source": [ 73 | "# Results" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "metadata": { 79 | "id": "Uo_AyQGzCzik", 80 | "colab_type": "code", 81 | "colab": {} 82 | }, 83 | "source": [ 84 | "def getAvg(learn:Learner, test_df:DataFrame):\n", 85 | " val = 0\n", 86 | " arr = []\n", 87 | " cat_vars = learn.data.train_dl.x.cat_names\n", 88 | " cont_vars = learn.data.train_dl.x.cont_names\n", 89 | " procs = learn.data.train_dl.x.procs\n", 90 | " dep_var = 'Survived'\n", 91 | " data_test = (TabularList.from_df(test_df, cat_names=cat_vars, \n", 92 | " cont_names=cont_vars, procs=procs, \n", 93 | " processor=learn.data.processor)\n", 94 | " .split_none()\n", 95 | " .label_from_df(cols=dep_var)\n", 96 | " .databunch())\n", 97 | " learn.data.valid_dl = data_test.train_dl\n", 98 | " for x in range(10):\n", 99 | " arr.append(float(learn.validate()[1]))\n", 100 | " for x in range(len(arr)):\n", 101 | " val = val + arr[x]\n", 102 | " return print(f'My Average Accuracy: {val/10}%')" 103 | ], 104 | "execution_count": 0, 105 | "outputs": [] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "metadata": { 110 | "id": "ZtKCzQ7zC_mg", 111 | "colab_type": "code", 112 | "colab": { 113 | "base_uri": "https://localhost:8080/", 114 | "height": 34 115 | }, 116 | "outputId": "1a589e7c-76b0-4808-9ac5-0f01688b05ba" 117 | }, 118 | "source": [ 119 | "getAvg(learn, test)" 120 | ], 121 | "execution_count": 24, 122 | "outputs": [ 123 | { 124 | "output_type": "stream", 125 | "text": [ 126 | "My Average Accuracy: 0.8765625%\n" 127 | ], 128 | "name": "stdout" 129 | } 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "metadata": { 135 | "id": "0yZ6TmgWEsKQ", 136 | "colab_type": "text" 137 | }, 138 | "source": [ 139 | "Now that you're here, why not try getting the predictions on the test set too! Go save this model, make a new databunch using the `.add_test()` function, and use `.get_preds()` to generate predictions\n", 140 | "\n", 141 | "You can submit predictions by hitting the \"Submit Predictions\" button on the kaggle competition page, and upload the CSV document generated by the function below." 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "metadata": { 147 | "id": "TXzrtqNSDH3m", 148 | "colab_type": "code", 149 | "colab": {} 150 | }, 151 | "source": [ 152 | "def TitanicPreds(learn:Learner):\n", 153 | " predictions, *_ = learn.get_preds(DatasetType.Test)\n", 154 | " labels = np.argmax(predictions, 1)\n", 155 | " sub_df = pd.DataFrame({'PassengerId': test_df['PassengerId'], 'Survived': labels})\n", 156 | " sub_df.to_csv('submission.csv', index=False)" 157 | ], 158 | "execution_count": 0, 159 | "outputs": [] 160 | } 161 | ] 162 | } -------------------------------------------------------------------------------- /Practice/Moderate_Intel_Image_Classification.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Moderate: Intel Image Classification.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "toc_visible": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "accelerator": "GPU" 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "Yw4xr2EiF7aA", 22 | "colab_type": "text" 23 | }, 24 | "source": [ 25 | "# Intel Image Classification\n", 26 | "\n", 27 | "\"**Context**\n", 28 | "\n", 29 | "This is image data of Natural Scenes around the world.\n", 30 | "\n", 31 | "**Content**\n", 32 | "\n", 33 | "This Data contains around 25k images of size 150x150 distributed under 6 categories. {'buildings' -> 0, 'forest' -> 1, 'glacier' -> 2, 'mountain' -> 3, 'sea' -> 4, 'street' -> 5 }\n", 34 | "\n", 35 | "The Train, Test and Prediction data is separated in each zip files. There are around 14k images in Train, 3k in Test and 7k in Prediction. This data was initially published on https://datahack.analyticsvidhya.com by Intel to host a Image classification Challenge.\n", 36 | "\n", 37 | "**Acknowledgements**\n", 38 | "\n", 39 | "Thanks to https://datahack.analyticsvidhya.com for the challenge and Intel for the Data\n", 40 | "\n", 41 | "Photo by Jan Böttinger on Unsplash\"" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": { 47 | "id": "ZhfA7TZwGKOx", 48 | "colab_type": "text" 49 | }, 50 | "source": [ 51 | "With this dataset, I want you to try to be able to identify one of five classes on a tiny image. The kaggle data is available here: https://www.kaggle.com/puneet6060/intel-image-classification\n", 52 | "\n", 53 | "You need to make a Kaggle account to download the data" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": { 59 | "id": "tR7crTYJJYDN", 60 | "colab_type": "text" 61 | }, 62 | "source": [ 63 | "# Get Data\n", 64 | "\n", 65 | "The following will set you up with a folder structure like so after you download the data and upload it to Google Colab:\n", 66 | "```\n", 67 | "-- images\n", 68 | " -- train\n", 69 | " -- valid\n", 70 | " -- test\n", 71 | "```" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "metadata": { 77 | "id": "kdAOJMl4GrRh", 78 | "colab_type": "code", 79 | "colab": {} 80 | }, 81 | "source": [ 82 | "!unzip intel-image-classification.zip\n", 83 | "!unzip seg_train.zip\n", 84 | "!unzip seg_pred.zip\n", 85 | "!unzip seg_test.zip\n", 86 | "!mkdir images\n", 87 | "!cp -a ./seg_train ./images/train\n", 88 | "!cp -a ./seg_test ./images/valid\n", 89 | "!cp -a ./seg_pred ./images/test" 90 | ], 91 | "execution_count": 0, 92 | "outputs": [] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": { 97 | "id": "-ZpW_oudJyXx", 98 | "colab_type": "text" 99 | }, 100 | "source": [ 101 | "# Train" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": { 107 | "id": "QRceVVXnQgyC", 108 | "colab_type": "text" 109 | }, 110 | "source": [ 111 | "Hint: ImageDataBunch can take a train, valid, and test folder: https://docs.fast.ai/vision.data.html#ImageDataBunch" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": { 117 | "id": "0drg4WLBSo8u", 118 | "colab_type": "text" 119 | }, 120 | "source": [ 121 | "# Results" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "metadata": { 127 | "id": "kSFNItPUPE3w", 128 | "colab_type": "code", 129 | "colab": {} 130 | }, 131 | "source": [ 132 | "def getAvg(learn:Learner):\n", 133 | " val = 0\n", 134 | " arr = []\n", 135 | " for x in range(10):\n", 136 | " arr.append(float(learn.validate()[1]))\n", 137 | " for x in range(len(arr)):\n", 138 | " val = val + arr[x]\n", 139 | " return print(f'My Average Accuracy: {val/10}%')" 140 | ], 141 | "execution_count": 0, 142 | "outputs": [] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "metadata": { 147 | "id": "20cwwNTVQTha", 148 | "colab_type": "code", 149 | "colab": { 150 | "base_uri": "https://localhost:8080/", 151 | "height": 34 152 | }, 153 | "outputId": "7d4b3f8e-8bc4-4cfd-c626-b241ac77e427" 154 | }, 155 | "source": [ 156 | "getAvg(learn)" 157 | ], 158 | "execution_count": 29, 159 | "outputs": [ 160 | { 161 | "output_type": "stream", 162 | "text": [ 163 | "My Average Accuracy: 0.9326666593551636%\n" 164 | ], 165 | "name": "stdout" 166 | } 167 | ] 168 | } 169 | ] 170 | } -------------------------------------------------------------------------------- /Practice/Moderate_Wine.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Moderate: Wine.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "gzPqqGec5CWn", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Wine\n", 25 | "\n", 26 | "This is a regression problem. There are 4,898 observations with 11 input variables and one output variable. The variable names are as follows:\n", 27 | "\n", 28 | "1. Fixed acidity.\n", 29 | "2. Volatile acidity.\n", 30 | "3. Citric acid.\n", 31 | "4. Residual sugar.\n", 32 | "5. Chlorides.\n", 33 | "6. Free sulfur dioxide.\n", 34 | "7. Total sulfur dioxide.\n", 35 | "8. Density.\n", 36 | "9. pH.\n", 37 | "10. Sulphates.\n", 38 | "11. Alcohol.\n", 39 | "12. Quality (score between 0 and 10).\n", 40 | "\n", 41 | "The overall performance is in RMSE." 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": { 47 | "id": "zVKDkygi5e3W", 48 | "colab_type": "text" 49 | }, 50 | "source": [ 51 | "# Get Data" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "metadata": { 57 | "id": "qpS-fCoA44hO", 58 | "colab_type": "code", 59 | "colab": { 60 | "base_uri": "https://localhost:8080/", 61 | "height": 204 62 | }, 63 | "outputId": "f8c6c85f-3f7c-443d-9c44-e2f03a859029" 64 | }, 65 | "source": [ 66 | "!wget http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv" 67 | ], 68 | "execution_count": 1, 69 | "outputs": [ 70 | { 71 | "output_type": "stream", 72 | "text": [ 73 | "--2019-07-09 08:42:44-- http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv\n", 74 | "Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\n", 75 | "Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:80... connected.\n", 76 | "HTTP request sent, awaiting response... 200 OK\n", 77 | "Length: 264426 (258K) [application/x-httpd-php]\n", 78 | "Saving to: ‘winequality-white.csv’\n", 79 | "\n", 80 | "winequality-white.c 100%[===================>] 258.23K 464KB/s in 0.6s \n", 81 | "\n", 82 | "2019-07-09 08:42:45 (464 KB/s) - ‘winequality-white.csv’ saved [264426/264426]\n", 83 | "\n" 84 | ], 85 | "name": "stdout" 86 | } 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": { 92 | "id": "K-OdhPGj56--", 93 | "colab_type": "text" 94 | }, 95 | "source": [ 96 | "# Training" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "metadata": { 102 | "id": "Q4es3Qo498sV", 103 | "colab_type": "code", 104 | "colab": {} 105 | }, 106 | "source": [ 107 | "from sklearn.model_selection import train_test_split\n", 108 | "train, test = train_test_split(df, test_size=0.1)" 109 | ], 110 | "execution_count": 0, 111 | "outputs": [] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": { 116 | "id": "i_ZaY4GR8nkn", 117 | "colab_type": "text" 118 | }, 119 | "source": [ 120 | "# Results\n", 121 | "\n", 122 | "Run the following function to get your overall average results out of ten runs on the test set. My best was an RMSE value of 0.12995" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "metadata": { 128 | "id": "Q_Z8uIWV8PWQ", 129 | "colab_type": "code", 130 | "colab": {} 131 | }, 132 | "source": [ 133 | "def getAvg(learn:Learner, test_df:DataFrame):\n", 134 | " val = 0\n", 135 | " arr = []\n", 136 | " cat_vars = learn.data.train_dl.x.cat_names\n", 137 | " cont_vars = learn.data.train_dl.x.cont_names\n", 138 | " dep_var = 'quality'\n", 139 | " data_test = (TabularList.from_df(test, cat_names=[], cont_names=cont_vars, procs=[Normalize], processor=learn.data.processor)\n", 140 | " .split_none()\n", 141 | " .label_from_df(cols=dep_var, label_cls=FloatList, log=True)\n", 142 | " .databunch())\n", 143 | " learn.data.valid_dl = data_test.train_dl\n", 144 | " for x in range(10):\n", 145 | " arr.append(float(learn.validate()[1]))\n", 146 | " for x in range(len(arr)):\n", 147 | " val = val + arr[x]\n", 148 | " return print(f'My Average RMSE: {val/10}')" 149 | ], 150 | "execution_count": 0, 151 | "outputs": [] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "metadata": { 156 | "id": "d2LIpQuy8bQE", 157 | "colab_type": "code", 158 | "colab": { 159 | "base_uri": "https://localhost:8080/", 160 | "height": 34 161 | }, 162 | "outputId": "bed297ca-59b6-438a-ac92-c2f9ae2ddcc7" 163 | }, 164 | "source": [ 165 | "getAvg(learn, test)" 166 | ], 167 | "execution_count": 42, 168 | "outputs": [ 169 | { 170 | "output_type": "stream", 171 | "text": [ 172 | "My Average RMSE:0.12995133250951768\n" 173 | ], 174 | "name": "stdout" 175 | } 176 | ] 177 | } 178 | ] 179 | } -------------------------------------------------------------------------------- /Practice/Easy_Iris_Starter.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Easy: Iris_Starter.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "D6EAY3LqutxR", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Iris - A simple small dataset\n", 25 | "\n", 26 | "\"The Iris Flowers Dataset involves predicting the flower species given measurements of iris flowers.\n", 27 | "\n", 28 | "It is a multi-class classification problem. The number of observations for each class is balanced. There are 150 observations with 4 input variables and 1 output variable. The variable names are as follows:\n", 29 | "\n", 30 | "1. Sepal length in cm.\n", 31 | "2. Sepal width in cm.\n", 32 | "3. Petal length in cm.\n", 33 | "4. Petal width in cm.\n", 34 | "5. Class (Iris Setosa, Iris Versicolour, Iris Virginica).\n", 35 | "\n", 36 | "The baseline performance of predicting the most prevalent class is a classification accuracy of approximately 26%.\"\n", 37 | "\n", 38 | "I have attached some starter code below, with my highest accuracy at the bottom. See if you can get to it, if not beat it!" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "id": "obJslte234qi", 45 | "colab_type": "text" 46 | }, 47 | "source": [ 48 | "# Get Data" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": { 54 | "id": "vA1VCmE83777", 55 | "colab_type": "text" 56 | }, 57 | "source": [ 58 | "All I will include here is the libraries we are using. (A few hints: you can read in a .data to pandas through .csv. Figure out how to make it skip the header though. Also, feature engineering will come in handy here. Be creative. You have measurements. Try anything.)" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "metadata": { 64 | "id": "PzGMPr0EusN9", 65 | "colab_type": "code", 66 | "colab": {} 67 | }, 68 | "source": [ 69 | "!wget http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data" 70 | ], 71 | "execution_count": 0, 72 | "outputs": [] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "metadata": { 77 | "id": "sMQ_wWwrvfSn", 78 | "colab_type": "code", 79 | "colab": {} 80 | }, 81 | "source": [ 82 | "import pandas as pd\n", 83 | "from fastai import *\n", 84 | "from fastai.tabular import *" 85 | ], 86 | "execution_count": 0, 87 | "outputs": [] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": { 92 | "id": "wjK-lVW131b2", 93 | "colab_type": "text" 94 | }, 95 | "source": [ 96 | "# Training" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "metadata": { 102 | "id": "5igHE9qp6SLd", 103 | "colab_type": "code", 104 | "colab": {} 105 | }, 106 | "source": [ 107 | "from sklearn.model_selection import train_test_split\n", 108 | "train, test = train_test_split(df, test_size=0.1)" 109 | ], 110 | "execution_count": 0, 111 | "outputs": [] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": { 116 | "id": "igA7KgdL32l_", 117 | "colab_type": "text" 118 | }, 119 | "source": [ 120 | "# Results" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "metadata": { 126 | "id": "iWp4zk3t3l77", 127 | "colab_type": "text" 128 | }, 129 | "source": [ 130 | "Run the following function to get your overall average results out of ten runs on the test set. My best was 61%" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "metadata": { 136 | "id": "uutdCaEO20k1", 137 | "colab_type": "code", 138 | "colab": {} 139 | }, 140 | "source": [ 141 | "def getAvg(learn:Learner, test_df:DataFrame):\n", 142 | " val = 0\n", 143 | " arr = []\n", 144 | " cat_vars = learn.data.train_dl.x.cat_names\n", 145 | " cont_vars = learn.data.train_dl.x.cont_names\n", 146 | " dep_var = 'Class'\n", 147 | " data_test = (TabularList.from_df(test_df, cat_vars, cont_vars, procs=procs, processor=learn.data.processor)\n", 148 | " .split_none()\n", 149 | " .label_from_df(dep_var)\n", 150 | " .databunch(bs=10))\n", 151 | " learn.data.valid_dl = data_test.train_dl\n", 152 | " for x in range(10):\n", 153 | " arr.append(float(learn.validate()[1]))\n", 154 | " for x in range(len(arr)):\n", 155 | " val = val + arr[x]\n", 156 | " return print(f'My Average Accuracy: {val/10}')" 157 | ], 158 | "execution_count": 0, 159 | "outputs": [] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "metadata": { 164 | "id": "QiJvjHby3hbo", 165 | "colab_type": "code", 166 | "colab": { 167 | "base_uri": "https://localhost:8080/", 168 | "height": 34 169 | }, 170 | "outputId": "16dac54f-1720-4ea0-e7e7-5ae26af9c8a9" 171 | }, 172 | "source": [ 173 | "getAvg(learn, test)" 174 | ], 175 | "execution_count": 41, 176 | "outputs": [ 177 | { 178 | "output_type": "stream", 179 | "text": [ 180 | "My Average Accuracy:0.610000005364418\n" 181 | ], 182 | "name": "stdout" 183 | } 184 | ] 185 | } 186 | ] 187 | } -------------------------------------------------------------------------------- /03b_Baysian.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "03_Baysian.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "kt3CQFwQ3IE_", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Baysian Optimization" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": { 30 | "id": "93zkQS0j3Lx1", 31 | "colab_type": "text" 32 | }, 33 | "source": [ 34 | "Form of hyper-parameter tuning.\n", 35 | "\n", 36 | "Example of using something designed for pytorch in fastai\n", 37 | "\n", 38 | "Repository for Today: [BayesianOptimization](https://github.com/fmfn/BayesianOptimization)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "id": "LNvrk43w6ip4", 45 | "colab_type": "text" 46 | }, 47 | "source": [ 48 | "## How does it work?\n", 49 | "\n", 50 | "Bayesian optimization works by constructing a posterior distribution of functions (gaussian process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not, as seen in the picture below.\n", 51 | "\n", 52 | "![](https://github.com/fmfn/BayesianOptimization/raw/master/examples/bo_example.png)\n", 53 | "\n", 54 | "- Taken from their github readme" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "metadata": { 60 | "id": "RiD9STjY20F7", 61 | "colab_type": "code", 62 | "colab": { 63 | "base_uri": "https://localhost:8080/", 64 | "height": 274 65 | }, 66 | "outputId": "d12563da-25ab-4b9a-8f3a-6147d01e30e6" 67 | }, 68 | "source": [ 69 | "!pip install bayesian-optimization" 70 | ], 71 | "execution_count": 2, 72 | "outputs": [ 73 | { 74 | "output_type": "stream", 75 | "text": [ 76 | "Collecting bayesian-optimization\n", 77 | " Downloading https://files.pythonhosted.org/packages/72/0c/173ac467d0a53e33e41b521e4ceba74a8ac7c7873d7b857a8fbdca88302d/bayesian-optimization-1.0.1.tar.gz\n", 78 | "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from bayesian-optimization) (1.16.4)\n", 79 | "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from bayesian-optimization) (1.3.1)\n", 80 | "Requirement already satisfied: scikit-learn>=0.18.0 in /usr/local/lib/python3.6/dist-packages (from bayesian-optimization) (0.21.3)\n", 81 | "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.18.0->bayesian-optimization) (0.13.2)\n", 82 | "Building wheels for collected packages: bayesian-optimization\n", 83 | " Building wheel for bayesian-optimization (setup.py) ... \u001b[?25l\u001b[?25hdone\n", 84 | " Created wheel for bayesian-optimization: filename=bayesian_optimization-1.0.1-cp36-none-any.whl size=10031 sha256=bf519bbd7268a6d30ad9b7ca7c245ebfb97b3d7f8e6c0b98f5cfe700ae7791d1\n", 85 | " Stored in directory: /root/.cache/pip/wheels/1d/0d/3b/6b9d4477a34b3905f246ff4e7acf6aafd4cc9b77d473629b77\n", 86 | "Successfully built bayesian-optimization\n", 87 | "Installing collected packages: bayesian-optimization\n", 88 | "Successfully installed bayesian-optimization-1.0.1\n" 89 | ], 90 | "name": "stdout" 91 | } 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "metadata": { 97 | "id": "SFlERfSP3TWl", 98 | "colab_type": "code", 99 | "colab": {} 100 | }, 101 | "source": [ 102 | "from fastai import *\n", 103 | "from fastai.tabular import *\n", 104 | "from bayes_opt import BayesianOptimization\n", 105 | "from fastprogress import *\n", 106 | "from fastai.utils.mod_display import progress_disabled_ctx" 107 | ], 108 | "execution_count": 0, 109 | "outputs": [] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": { 114 | "id": "4D8nzBRi3f0l", 115 | "colab_type": "text" 116 | }, 117 | "source": [ 118 | "For today's example, we will use the Adults problem and adjust weight decay, learning rate, and drop out" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "metadata": { 124 | "id": "UpovbsU13e9M", 125 | "colab_type": "code", 126 | "colab": {} 127 | }, 128 | "source": [ 129 | "path = untar_data(URLs.ADULT_SAMPLE)\n", 130 | "df = pd.read_csv(path/'adult.csv')" 131 | ], 132 | "execution_count": 0, 133 | "outputs": [] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "metadata": { 138 | "id": "u5v_MqFz3oCw", 139 | "colab_type": "code", 140 | "colab": {} 141 | }, 142 | "source": [ 143 | "dep_var = 'salary'\n", 144 | "cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race']\n", 145 | "cont_names = ['age', 'fnlwgt', 'education-num']\n", 146 | "procs = [FillMissing, Categorify, Normalize]" 147 | ], 148 | "execution_count": 0, 149 | "outputs": [] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "metadata": { 154 | "id": "ia4XvVKP3rUo", 155 | "colab_type": "code", 156 | "colab": {} 157 | }, 158 | "source": [ 159 | "data = (TabularList.from_df(df, path=path, cat_names=cat_names, cont_names=cont_names, procs=procs)\n", 160 | " .split_by_idx(list(range(800,1000)))\n", 161 | " .label_from_df(cols=dep_var)\n", 162 | " .databunch())" 163 | ], 164 | "execution_count": 0, 165 | "outputs": [] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": { 170 | "id": "i-P3m3Nt3wde", 171 | "colab_type": "text" 172 | }, 173 | "source": [ 174 | "Next we need to define a `fit_with` function, where our inputs will be whatever we want to test our hyperparemeters on, essentially you can adjust *anything* in here. For us, we only care about how hyperparemeters dealing with our model do." 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "metadata": { 180 | "id": "XA88zN693tVX", 181 | "colab_type": "code", 182 | "colab": {} 183 | }, 184 | "source": [ 185 | "def fit_with(lr:float, wd:float, dp:float):\n", 186 | " # create a Learner\n", 187 | " learn = tabular_learner(data, layers=[200,100], metrics=accuracy, emb_drop=dp, wd=wd)\n", 188 | " \n", 189 | " # Train for x epochs\n", 190 | " with progress_disabled_ctx(learn) as learn:\n", 191 | " learn.fit_one_cycle(3, lr)\n", 192 | " \n", 193 | " # Save, print, and return the overall accuracy\n", 194 | " acc = float(learn.validate()[1])\n", 195 | " \n", 196 | " return acc" 197 | ], 198 | "execution_count": 0, 199 | "outputs": [] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": { 204 | "id": "g2Achkku4cCn", 205 | "colab_type": "text" 206 | }, 207 | "source": [ 208 | "Finally, we need to determine what our ranges for our hyperparameters need to be as a dict" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "metadata": { 214 | "id": "sWUdC42Z4bqG", 215 | "colab_type": "code", 216 | "colab": {} 217 | }, 218 | "source": [ 219 | "hps = {'lr': (1e-05, 1e-02),\n", 220 | " 'wd': (4e-4, 0.4),\n", 221 | " 'dp': (0.01, 0.5)}" 222 | ], 223 | "execution_count": 0, 224 | "outputs": [] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "metadata": { 229 | "id": "C7Lkcl0b42gB", 230 | "colab_type": "text" 231 | }, 232 | "source": [ 233 | "Now we can build our optimizer" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "metadata": { 239 | "id": "PKT9Zlp74zjO", 240 | "colab_type": "code", 241 | "colab": {} 242 | }, 243 | "source": [ 244 | "optim = BayesianOptimization( \n", 245 | " f = fit_with, # our function\n", 246 | " pbounds=hps, # our boundaries\n", 247 | " verbose=2, # 1 prints a maximum only when observed, 0 is silent\n", 248 | " random_state=1)" 249 | ], 250 | "execution_count": 0, 251 | "outputs": [] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": { 256 | "id": "TD1nhhk46UOb", 257 | "colab_type": "text" 258 | }, 259 | "source": [ 260 | "And now we do a search!" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "metadata": { 266 | "id": "rBfEqU6H5Qy6", 267 | "colab_type": "code", 268 | "colab": { 269 | "base_uri": "https://localhost:8080/", 270 | "height": 90 271 | }, 272 | "outputId": "cbbd653c-73ae-46c6-8c00-678fddee00a4" 273 | }, 274 | "source": [ 275 | "%time optim.maximize(n_iter=10)" 276 | ], 277 | "execution_count": 23, 278 | "outputs": [ 279 | { 280 | "output_type": "stream", 281 | "text": [ 282 | "| \u001b[0m 21 \u001b[0m | \u001b[0m 0.835 \u001b[0m | \u001b[0m 0.2231 \u001b[0m | \u001b[0m 0.01 \u001b[0m | \u001b[0m 0.3484 \u001b[0m |\n", 283 | "=============================================================\n", 284 | "CPU times: user 3min 33s, sys: 30.2 s, total: 4min 3s\n", 285 | "Wall time: 3min 43s\n" 286 | ], 287 | "name": "stdout" 288 | } 289 | ] 290 | }, 291 | { 292 | "cell_type": "markdown", 293 | "metadata": { 294 | "id": "NHAkjSfd62zJ", 295 | "colab_type": "text" 296 | }, 297 | "source": [ 298 | "Now let's look at our best results" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "metadata": { 304 | "id": "WclmLG2z5USl", 305 | "colab_type": "code", 306 | "colab": { 307 | "base_uri": "https://localhost:8080/", 308 | "height": 35 309 | }, 310 | "outputId": "28d65e45-65df-4e53-c3e9-8bd230ac39d2" 311 | }, 312 | "source": [ 313 | "print(optim.max)" 314 | ], 315 | "execution_count": 25, 316 | "outputs": [ 317 | { 318 | "output_type": "stream", 319 | "text": [ 320 | "{'target': 0.8349999785423279, 'params': {'dp': 0.2740201996616449, 'lr': 0.004197753198888915, 'wd': 0.27421371235854514}}\n" 321 | ], 322 | "name": "stdout" 323 | } 324 | ] 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": { 329 | "id": "aFkiYg5e7C6O", 330 | "colab_type": "text" 331 | }, 332 | "source": [ 333 | "We can also look at all of our results" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "metadata": { 339 | "id": "VOfGifnm68Zu", 340 | "colab_type": "code", 341 | "colab": { 342 | "base_uri": "https://localhost:8080/", 343 | "height": 781 344 | }, 345 | "outputId": "9f9b2f49-00e7-41fb-c2e2-9630e593b6ad" 346 | }, 347 | "source": [ 348 | "for i, res in enumerate(optim.res):\n", 349 | " print('Iteration {} \\n\\t{}'.format(i, res))" 350 | ], 351 | "execution_count": 27, 352 | "outputs": [ 353 | { 354 | "output_type": "stream", 355 | "text": [ 356 | "Iteration 0 \n", 357 | "\t{'target': 0.824999988079071, 'params': {'dp': 0.21434078230426126, 'lr': 0.007206041689487159, 'wd': 0.00044570417701101674}}\n", 358 | "Iteration 1 \n", 359 | "\t{'target': 0.8149999976158142, 'params': {'dp': 0.1581429605896015, 'lr': 0.0014760913492629594, 'wd': 0.0372985024696116}}\n", 360 | "Iteration 2 \n", 361 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.10126750357505873, 'lr': 0.0034621516631600474, 'wd': 0.15894828270257572}}\n", 362 | "Iteration 3 \n", 363 | "\t{'target': 0.8349999785423279, 'params': {'dp': 0.2740201996616449, 'lr': 0.004197753198888915, 'wd': 0.27421371235854514}}\n", 364 | "Iteration 4 \n", 365 | "\t{'target': 0.824999988079071, 'params': {'dp': 0.11018160236844353, 'lr': 0.008782393189545545, 'wd': 0.011344082241891295}}\n", 366 | "Iteration 5 \n", 367 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.5, 'lr': 0.01, 'wd': 0.4}}\n", 368 | "Iteration 6 \n", 369 | "\t{'target': 0.824999988079071, 'params': {'dp': 0.3385290799874171, 'lr': 0.004178874975647598, 'wd': 0.2236524554469224}}\n", 370 | "Iteration 7 \n", 371 | "\t{'target': 0.8100000023841858, 'params': {'dp': 0.07878959991166454, 'lr': 0.0019890338759579393, 'wd': 0.32037752964274446}}\n", 372 | "Iteration 8 \n", 373 | "\t{'target': 0.8149999976158142, 'params': {'dp': 0.4844481721025048, 'lr': 0.003141107539810836, 'wd': 0.27705211722145795}}\n", 374 | "Iteration 9 \n", 375 | "\t{'target': 0.824999988079071, 'params': {'dp': 0.4394306846250588, 'lr': 0.008947120568403435, 'wd': 0.03438366686336325}}\n", 376 | "Iteration 10 \n", 377 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.029136843784112354, 'lr': 0.0017066058914500435, 'wd': 0.3513057443703935}}\n", 378 | "Iteration 11 \n", 379 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.3109234184704589, 'lr': 0.01, 'wd': 0.4}}\n", 380 | "Iteration 12 \n", 381 | "\t{'target': 0.6600000262260437, 'params': {'dp': 0.01, 'lr': 1e-05, 'wd': 0.0004}}\n", 382 | "Iteration 13 \n", 383 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.5, 'lr': 0.01, 'wd': 0.0004}}\n", 384 | "Iteration 14 \n", 385 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.06992043366184122, 'lr': 0.01, 'wd': 0.4}}\n", 386 | "Iteration 15 \n", 387 | "\t{'target': 0.8299999833106995, 'params': {'dp': 0.5, 'lr': 0.01, 'wd': 0.13414236271019292}}\n", 388 | "Iteration 16 \n", 389 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.2109149821872044, 'lr': 0.01, 'wd': 0.16439313383002094}}\n", 390 | "Iteration 17 \n", 391 | "\t{'target': 0.8299999833106995, 'params': {'dp': 0.01, 'lr': 0.01, 'wd': 0.2502516509945325}}\n", 392 | "Iteration 18 \n", 393 | "\t{'target': 0.8199999928474426, 'params': {'dp': 0.37616126312596776, 'lr': 0.01, 'wd': 0.31599268719972023}}\n", 394 | "Iteration 19 \n", 395 | "\t{'target': 0.824999988079071, 'params': {'dp': 0.01, 'lr': 0.01, 'wd': 0.4}}\n", 396 | "Iteration 20 \n", 397 | "\t{'target': 0.8349999785423279, 'params': {'dp': 0.2231423431471948, 'lr': 0.01, 'wd': 0.3484051374580442}}\n" 398 | ], 399 | "name": "stdout" 400 | } 401 | ] 402 | } 403 | ] 404 | } -------------------------------------------------------------------------------- /04c_Permutation_Importance.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "04_Feature_Engineering.ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "uNJYgFkBlt90", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Feature Importance\n", 25 | "\n", 26 | "| | Tuesday 4-5:15pm | Friday 4-5:30pm |\n", 27 | "|:------:|:-------------------------------------------:|:--------------------------------------------------------------------------:|\n", 28 | "| **Week 1** | Introduction | Introduction |\n", 29 | "| **Week 2** | Custom computer vision tasks | State of the art in Computer Vision |\n", 30 | "| **Week 3** | Introduction to Tabular modeling and pandas | Pandas workshop and feature engineering |\n", 31 | "| **Week 4** | Tabular and Image Regression | **Feature importance and advanced feature engineering** |\n", 32 | "| **Week 5** | Natural Language Processing | State of the art in NLP |\n", 33 | "| **Week 6** | Segmentation and Kaggle | Audio |\n", 34 | "| **Week 7** | Computer vision from scratch | NLP from scratch |\n", 35 | "| **Week 8** | Callbacks | Optimizers |\n", 36 | "| **Week 9** | Generative Adversarial Networks | Research time / presentations |\n", 37 | "| **Week 10** | Putting models into production | Putting models into production |\n", 38 | "\n", 39 | "## What is it?\n", 40 | "\n", 41 | "* A way to examine how particular variables or features are being used or the most helpful for a model\n", 42 | "\n", 43 | "## Why is it important?\n", 44 | "\n", 45 | "* Feature pruning\n", 46 | "* Can help explain our models\n", 47 | "\n", 48 | "## How do we do it?\n", 49 | "\n", 50 | "* Permutation importance\n", 51 | "* Measure the change in accuracy of a **fully trained model** on a seperate set\n", 52 | "\n", 53 | "For example, instead of our 70% Train, 20% Validation, 10% Test, we could now have:\n", 54 | "\n", 55 | "We have 10% Test, 10% Feature Importance, 64% Train, 16% Validation" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "metadata": { 61 | "id": "9oGN5YITxw9T", 62 | "colab_type": "code", 63 | "colab": {} 64 | }, 65 | "source": [ 66 | "from sklearn.model_selection import train_test_split" 67 | ], 68 | "execution_count": 0, 69 | "outputs": [] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "metadata": { 74 | "id": "9PyPftGa-VSF", 75 | "colab_type": "code", 76 | "colab": {} 77 | }, 78 | "source": [ 79 | "import pandas as pd" 80 | ], 81 | "execution_count": 0, 82 | "outputs": [] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "metadata": { 87 | "id": "3mWRgCnno2as", 88 | "colab_type": "code", 89 | "colab": {} 90 | }, 91 | "source": [ 92 | "def create_sets(df:pd.DataFrame, is_feat:bool):\n", 93 | " train, test = train_test_split(df, test_size=0.1)\n", 94 | " if is_feat: \n", 95 | " train, feat = train_test_split(train, test_size=0.1)\n", 96 | " return train, test, feat\n", 97 | " else: return train, test" 98 | ], 99 | "execution_count": 0, 100 | "outputs": [] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "metadata": { 105 | "id": "QNJF-q1Hk6v_", 106 | "colab_type": "code", 107 | "colab": {} 108 | }, 109 | "source": [ 110 | "from fastai.tabular import *" 111 | ], 112 | "execution_count": 0, 113 | "outputs": [] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "metadata": { 118 | "id": "n40lLE1umVaT", 119 | "colab_type": "code", 120 | "colab": {} 121 | }, 122 | "source": [ 123 | "path = untar_data(URLs.ADULT_SAMPLE)" 124 | ], 125 | "execution_count": 0, 126 | "outputs": [] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "metadata": { 131 | "id": "xJatTs45mZOr", 132 | "colab_type": "code", 133 | "colab": {} 134 | }, 135 | "source": [ 136 | "df = pd.read_csv(path/'adult.csv')" 137 | ], 138 | "execution_count": 0, 139 | "outputs": [] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "metadata": { 144 | "id": "-CkIjY-wmccN", 145 | "colab_type": "code", 146 | "colab": {} 147 | }, 148 | "source": [ 149 | "dep_var = 'salary'\n", 150 | "cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race']\n", 151 | "cont_names = ['age', 'fnlwgt', 'education-num']" 152 | ], 153 | "execution_count": 0, 154 | "outputs": [] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "metadata": { 159 | "id": "7ilon33hmdQk", 160 | "colab_type": "code", 161 | "colab": {} 162 | }, 163 | "source": [ 164 | "procs = [FillMissing, Categorify, Normalize]" 165 | ], 166 | "execution_count": 0, 167 | "outputs": [] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "metadata": { 172 | "id": "dRK1z1wpp6ts", 173 | "colab_type": "code", 174 | "outputId": "7942d56c-b9ed-4dfd-d981-69bb6b140c97", 175 | "colab": { 176 | "base_uri": "https://localhost:8080/", 177 | "height": 34 178 | } 179 | }, 180 | "source": [ 181 | "train, test, feat = create_sets(df, True)\n", 182 | "len(train), len(test), len(feat)" 183 | ], 184 | "execution_count": 10, 185 | "outputs": [ 186 | { 187 | "output_type": "execute_result", 188 | "data": { 189 | "text/plain": [ 190 | "(26373, 3257, 2931)" 191 | ] 192 | }, 193 | "metadata": { 194 | "tags": [] 195 | }, 196 | "execution_count": 10 197 | } 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": { 203 | "id": "1uric2N2qNAa", 204 | "colab_type": "text" 205 | }, 206 | "source": [ 207 | "# Initial Training" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "metadata": { 213 | "id": "VtXHWHSXnXCZ", 214 | "colab_type": "code", 215 | "colab": {} 216 | }, 217 | "source": [ 218 | "data = (TabularList.from_df(train, path=path, cat_names=cat_names, \n", 219 | " cont_names=cont_names, procs=procs)\n", 220 | " .split_by_rand_pct()\n", 221 | " .label_from_df(cols=dep_var)\n", 222 | " .databunch())" 223 | ], 224 | "execution_count": 0, 225 | "outputs": [] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "metadata": { 230 | "id": "_0Y_v2iNqZsz", 231 | "colab_type": "code", 232 | "colab": {} 233 | }, 234 | "source": [ 235 | "learn = tabular_learner(data, layers=[200,100], metrics=accuracy)" 236 | ], 237 | "execution_count": 0, 238 | "outputs": [] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "metadata": { 243 | "id": "S3L9TkH6qdAe", 244 | "colab_type": "code", 245 | "outputId": "804e6523-ef79-47ca-922a-eac5660966ff", 246 | "colab": { 247 | "base_uri": "https://localhost:8080/", 248 | "height": 204 249 | } 250 | }, 251 | "source": [ 252 | "learn.fit_one_cycle(5, 3e-02)" 253 | ], 254 | "execution_count": 13, 255 | "outputs": [ 256 | { 257 | "output_type": "display_data", 258 | "data": { 259 | "text/html": [ 260 | "\n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | "
epochtrain_lossvalid_lossaccuracytime
00.3914390.3703380.82802400:03
10.3639340.3705370.82290500:03
20.3629870.3717700.82480100:03
30.3502300.3661460.82688700:03
40.3408990.3640210.82992000:03
" 308 | ], 309 | "text/plain": [ 310 | "" 311 | ] 312 | }, 313 | "metadata": { 314 | "tags": [] 315 | } 316 | } 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "metadata": { 322 | "id": "O6NWDmGdqgpe", 323 | "colab_type": "code", 324 | "outputId": "2f5c0d46-1149-4dcb-b21c-f3c255b5bab3", 325 | "colab": { 326 | "base_uri": "https://localhost:8080/", 327 | "height": 80 328 | } 329 | }, 330 | "source": [ 331 | "learn.fit_one_cycle(1, 3e-3)" 332 | ], 333 | "execution_count": 14, 334 | "outputs": [ 335 | { 336 | "output_type": "display_data", 337 | "data": { 338 | "text/html": [ 339 | "\n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | "
epochtrain_lossvalid_lossaccuracytime
00.3428070.3650480.82897200:03
" 359 | ], 360 | "text/plain": [ 361 | "" 362 | ] 363 | }, 364 | "metadata": { 365 | "tags": [] 366 | } 367 | } 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": { 373 | "id": "YnvjgU5_qplG", 374 | "colab_type": "text" 375 | }, 376 | "source": [ 377 | "# Permutation Selection Algorithm" 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "metadata": { 383 | "id": "n1-NpoeLrN9x", 384 | "colab_type": "code", 385 | "outputId": "02e2996b-f15a-48f4-a6d5-337f250a6ef4", 386 | "colab": { 387 | "base_uri": "https://localhost:8080/", 388 | "height": 136 389 | } 390 | }, 391 | "source": [ 392 | "learn.data.cat_names" 393 | ], 394 | "execution_count": 15, 395 | "outputs": [ 396 | { 397 | "output_type": "execute_result", 398 | "data": { 399 | "text/plain": [ 400 | "['workclass',\n", 401 | " 'education',\n", 402 | " 'marital-status',\n", 403 | " 'occupation',\n", 404 | " 'relationship',\n", 405 | " 'race',\n", 406 | " 'education-num_na']" 407 | ] 408 | }, 409 | "metadata": { 410 | "tags": [] 411 | }, 412 | "execution_count": 15 413 | } 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "metadata": { 419 | "id": "JIF75IVgtgaj", 420 | "colab_type": "code", 421 | "outputId": "f12ede3f-1b1c-43a5-8d13-799738a87f1e", 422 | "colab": { 423 | "base_uri": "https://localhost:8080/", 424 | "height": 136 425 | } 426 | }, 427 | "source": [ 428 | "learn.data.cat_names" 429 | ], 430 | "execution_count": 16, 431 | "outputs": [ 432 | { 433 | "output_type": "execute_result", 434 | "data": { 435 | "text/plain": [ 436 | "['workclass',\n", 437 | " 'education',\n", 438 | " 'marital-status',\n", 439 | " 'occupation',\n", 440 | " 'relationship',\n", 441 | " 'race',\n", 442 | " 'education-num_na']" 443 | ] 444 | }, 445 | "metadata": { 446 | "tags": [] 447 | }, 448 | "execution_count": 16 449 | } 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "metadata": { 455 | "id": "UXP3UPywqmZ-", 456 | "colab_type": "code", 457 | "colab": {} 458 | }, 459 | "source": [ 460 | "def feature_importance(learn:Learner, dep_var:str, test:DataFrame):\n", 461 | " pd.options.mode.chained_assignment= None # Gets rid of annoying warning\n", 462 | " \n", 463 | " data = learn.data\n", 464 | " cats = [x for x in data.cat_names if '_na' not in x]\n", 465 | " conts = data.cont_names\n", 466 | " procs = data.procs\n", 467 | " \n", 468 | " dt = (TabularList.from_df(test, path='', cat_names=cats.copy(), cont_names=conts.copy(),\n", 469 | " procs=procs)\n", 470 | " .split_none()\n", 471 | " .label_from_df(cols=dep_var))\n", 472 | " dt.valid = dt.train\n", 473 | " dt = dt.databunch()\n", 474 | " \n", 475 | " learn.data.valid_dl = dt.valid_dl\n", 476 | " loss0 = float(learn.validate()[1])\n", 477 | " \n", 478 | " types = [cats, conts]\n", 479 | " \n", 480 | " fi = dict()\n", 481 | " for j, t in enumerate(types):\n", 482 | " for i, c in enumerate(t):\n", 483 | " print(c)\n", 484 | " base = test.copy()\n", 485 | " base[c] = base[c].sample(n=len(base), replace=True).reset_index(drop=True)\n", 486 | " \n", 487 | " \n", 488 | " dt = (TabularList.from_df(base, path='', cat_names=cats.copy(), cont_names=conts.copy(),\n", 489 | " procs=procs)\n", 490 | " .split_none()\n", 491 | " .label_from_df(cols=dep_var))\n", 492 | " dt.valid = dt.train\n", 493 | " dt = dt.databunch()\n", 494 | " \n", 495 | " learn.data.valid_dl = dt.valid_dl\n", 496 | " fi[c] = float(learn.validate()[1]) - loss0\n", 497 | " \n", 498 | " d = sorted(fi.items(), key=lambda kv: kv[1])\n", 499 | " df = pd.DataFrame({'Variable': [l for l, v in d], u'Δ Accuracy': [v for l, v in d]})\n", 500 | " df['Type'] = ''\n", 501 | " for x in range(len(df)):\n", 502 | " if df['Variable'].iloc[x] in cats:\n", 503 | " df['Type'].iloc[x] = 'categorical'\n", 504 | " else:\n", 505 | " df['Type'].iloc[x] = 'continuous'\n", 506 | " return df" 507 | ], 508 | "execution_count": 0, 509 | "outputs": [] 510 | }, 511 | { 512 | "cell_type": "code", 513 | "metadata": { 514 | "id": "r0_9oGY0tMzW", 515 | "colab_type": "code", 516 | "colab": {} 517 | }, 518 | "source": [ 519 | "res = feature_importance(learn, dep_var, feat)" 520 | ], 521 | "execution_count": 0, 522 | "outputs": [] 523 | }, 524 | { 525 | "cell_type": "markdown", 526 | "metadata": { 527 | "id": "vMGquCaN-1cT", 528 | "colab_type": "text" 529 | }, 530 | "source": [ 531 | "Let's look at how we are measuring. New - Original percentage. The lower the value the more important a value is" 532 | ] 533 | }, 534 | { 535 | "cell_type": "code", 536 | "metadata": { 537 | "id": "a13jB22JtSxM", 538 | "colab_type": "code", 539 | "outputId": "ac5bb447-2874-4ec5-e375-4d19d7601bfb", 540 | "colab": { 541 | "base_uri": "https://localhost:8080/", 542 | "height": 328 543 | } 544 | }, 545 | "source": [ 546 | "res" 547 | ], 548 | "execution_count": 19, 549 | "outputs": [ 550 | { 551 | "output_type": "execute_result", 552 | "data": { 553 | "text/html": [ 554 | "
\n", 555 | "\n", 568 | "\n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | "
VariableΔ AccuracyType
0marital-status-0.043330categorical
1age-0.024224continuous
2occupation-0.015694categorical
3education-num-0.011259continuous
4education-0.007165categorical
5race-0.004435categorical
6fnlwgt-0.003071continuous
7workclass-0.002729categorical
8relationship-0.002729categorical
\n", 634 | "
" 635 | ], 636 | "text/plain": [ 637 | " Variable Δ Accuracy Type\n", 638 | "0 marital-status -0.043330 categorical\n", 639 | "1 age -0.024224 continuous\n", 640 | "2 occupation -0.015694 categorical\n", 641 | "3 education-num -0.011259 continuous\n", 642 | "4 education -0.007165 categorical\n", 643 | "5 race -0.004435 categorical\n", 644 | "6 fnlwgt -0.003071 continuous\n", 645 | "7 workclass -0.002729 categorical\n", 646 | "8 relationship -0.002729 categorical" 647 | ] 648 | }, 649 | "metadata": { 650 | "tags": [] 651 | }, 652 | "execution_count": 19 653 | } 654 | ] 655 | }, 656 | { 657 | "cell_type": "markdown", 658 | "metadata": { 659 | "id": "sH2KMQrp_AtT", 660 | "colab_type": "text" 661 | }, 662 | "source": [ 663 | "In this case if we wanted to prune features we could drop `relationship`, `workclass`, `fnlwgt`, `race`, and `education`. If anything was positive or zero then dropping the column would **increase** our accuracy so we should definitely drop them" 664 | ] 665 | } 666 | ] 667 | } -------------------------------------------------------------------------------- /DeployingModels.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "DeployingModels.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "f8LDqp4HfVjw", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Model Deployment:\n", 25 | "\n", 26 | "Today we will be going over an easier way to get your models into production using Render. Render is used by most of the Fast.AI students to get their models up, and we can run the files locally ourselves to visually see it. To do this, I have provided download links to all of the standard models Fast.AI have used. Also we will bbe working out of the 'Supplementary' folder in Google Drive. Check your shared files to get access to it.\n", 27 | "\n", 28 | "We will explore productioninzing the following models:\n", 29 | "* **Computer Vision:** Cats vs Dogs\n", 30 | "* **Tabular:** > or <= $50k\n", 31 | "* **NLP:** IMDB" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": { 37 | "id": "gfGpxHKbga3t", 38 | "colab_type": "text" 39 | }, 40 | "source": [ 41 | "![alt text](https://course.fast.ai/images/render/render-logo.svg)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": { 47 | "id": "Fq8NC2ZAglcT", 48 | "colab_type": "text" 49 | }, 50 | "source": [ 51 | "To use the Render template, you will need docker installed. You can run the following below to get it running locally and test your changes." 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "metadata": { 57 | "id": "iWDp94yMfQFJ", 58 | "colab_type": "code", 59 | "colab": {} 60 | }, 61 | "source": [ 62 | "docker build -t fastai-v3 . && docker run --rm -it -p 5000:5000 fastai-v3" 63 | ], 64 | "execution_count": 0, 65 | "outputs": [] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": { 70 | "id": "I1j9qe0Ce6HM", 71 | "colab_type": "text" 72 | }, 73 | "source": [ 74 | "**OR** Simply run `python3 app/server.py serve`.\n", 75 | "\n", 76 | "Before you do this though, do a `pip3 install -r requirements.txt`" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "metadata": { 82 | "id": "kAgi1K5Ue4OZ", 83 | "colab_type": "code", 84 | "colab": {} 85 | }, 86 | "source": [ 87 | "* Or simply run " 88 | ], 89 | "execution_count": 0, 90 | "outputs": [] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": { 95 | "id": "s4KiwVwyhhqu", 96 | "colab_type": "text" 97 | }, 98 | "source": [ 99 | "## Repo\n", 100 | "\n", 101 | "Jeremy has a render example repo that we will be working off of [here](https://github.com/render-examples/fastai-v3). First thing we need is our models. Let's look at the Pets notebook first!" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": { 107 | "id": "sleaLlyj9FyM", 108 | "colab_type": "text" 109 | }, 110 | "source": [ 111 | "## Cats vs Dogs" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": { 117 | "id": "dCCC3-W49IRK", 118 | "colab_type": "text" 119 | }, 120 | "source": [ 121 | "### Training\n", 122 | "\n", 123 | "This should take ~5 minutes to run on your own. For todays purposes though we *just* want the models." 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "metadata": { 129 | "id": "HkGZhLqW8_XY", 130 | "colab_type": "code", 131 | "colab": {} 132 | }, 133 | "source": [ 134 | "from fastai.vision import *\n", 135 | "path = untar_data(URLs.PETS)\n", 136 | "path_img = path/'images'\n", 137 | "fnames = get_image_files(path_img)\n", 138 | "np.random.seed(2)\n", 139 | "pat = re.compile(r'/([^/]+)_\\d+.jpg$')\n", 140 | "data = ImageDataBunch.from_name_re(path_img, fnames, pat, ds_tfms=get_transforms(), size=224)\n", 141 | "data = data.normalize(imagenet_stats);\n", 142 | "learn = create_cnn(data, models.resnet34, pretrained=True, metrics=error_rate)\n", 143 | "learn.fit_one_cycle(2);\n", 144 | "learn.unfreeze()\n", 145 | "learn.fit_one_cycle(1)" 146 | ], 147 | "execution_count": 0, 148 | "outputs": [] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": { 153 | "id": "fmtLPnNf_aei", 154 | "colab_type": "text" 155 | }, 156 | "source": [ 157 | "### Exporting the Model" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "metadata": { 163 | "id": "y8ez7Vh6-6Ah", 164 | "colab_type": "code", 165 | "outputId": "0b6c943f-1f34-46ce-e55c-868799148d15", 166 | "colab": { 167 | "base_uri": "https://localhost:8080/", 168 | "height": 35 169 | } 170 | }, 171 | "source": [ 172 | "learn.path" 173 | ], 174 | "execution_count": 0, 175 | "outputs": [ 176 | { 177 | "output_type": "execute_result", 178 | "data": { 179 | "text/plain": [ 180 | "PosixPath('/root/.fastai/data/oxford-iiit-pet/images')" 181 | ] 182 | }, 183 | "metadata": { 184 | "tags": [] 185 | }, 186 | "execution_count": 2 187 | } 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": { 193 | "id": "pil1gpVQ_JpM", 194 | "colab_type": "text" 195 | }, 196 | "source": [ 197 | "Well, that's not very easy to get to. Let's change that to Colab's root working directory!" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "metadata": { 203 | "id": "2zSl6cJW9jhE", 204 | "colab_type": "code", 205 | "colab": {} 206 | }, 207 | "source": [ 208 | "learn.path = Path('')" 209 | ], 210 | "execution_count": 0, 211 | "outputs": [] 212 | }, 213 | { 214 | "cell_type": "markdown", 215 | "metadata": { 216 | "id": "YypSD1ii_QzA", 217 | "colab_type": "text" 218 | }, 219 | "source": [ 220 | "Now we can run `learn.export()` and get a pkl file with everything we need! (This will also change where learn.save() points to as well)" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "metadata": { 226 | "id": "RxlboB81_P6M", 227 | "colab_type": "code", 228 | "colab": {} 229 | }, 230 | "source": [ 231 | "learn.export('pets.pkl')" 232 | ], 233 | "execution_count": 0, 234 | "outputs": [] 235 | }, 236 | { 237 | "cell_type": "markdown", 238 | "metadata": { 239 | "id": "EWcKsQpAMsQW", 240 | "colab_type": "text" 241 | }, 242 | "source": [ 243 | "**NOTE** When we want to load the model, we can now run `load_learner(path, fname)`. This is **different** than a simple `learn.save()` and `learn.load()` combination." 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": { 249 | "id": "1v8KrpkN_9ha", 250 | "colab_type": "text" 251 | }, 252 | "source": [ 253 | "### Downloading the model\n", 254 | "\n", 255 | "To download my already-built model, run the following" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "metadata": { 261 | "id": "FS_qaHfDAlBK", 262 | "colab_type": "code", 263 | "colab": {} 264 | }, 265 | "source": [ 266 | "!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1oIn2_DxTJIYBWofQFOQQFcyO8yxJ3H8G' -O 'pets.pkl'" 267 | ], 268 | "execution_count": 0, 269 | "outputs": [] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "metadata": { 274 | "id": "l5bFtXHxBN4D", 275 | "colab_type": "text" 276 | }, 277 | "source": [ 278 | "### Modifying the files we need to\n", 279 | "\n", 280 | "Most of what we will be modifying is the 'server.py' file, which is inside the `app` directory. Now first, I recommend either uploading that file to your own google drive, or to dropbox. Then (if you use google drive for your model) change your sharing settings to 'Public', copy the share URL, and find the 'ID=' section. Copy that bit, and replace 'YOURID' in the below:\n", 281 | "\n", 282 | "'https://docs.google.com/uc?export=download&id=YOURID'\n" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": { 288 | "id": "Iurgq_NvB9QH", 289 | "colab_type": "text" 290 | }, 291 | "source": [ 292 | "Now let's look at the server.py. I'll post the whole thing below:" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "metadata": { 298 | "id": "0JB60N-QAnTA", 299 | "colab_type": "code", 300 | "colab": {} 301 | }, 302 | "source": [ 303 | "import aiohttp\n", 304 | "import asyncio\n", 305 | "import uvicorn\n", 306 | "from fastai import *\n", 307 | "from fastai.vision import *\n", 308 | "from io import BytesIO\n", 309 | "from starlette.applications import Starlette\n", 310 | "from starlette.middleware.cors import CORSMiddleware\n", 311 | "from starlette.responses import HTMLResponse, JSONResponse\n", 312 | "from starlette.staticfiles import StaticFiles\n", 313 | "\n", 314 | "export_file_url = 'https://www.dropbox.com/s/6bgq8t6yextloqp/export.pkl?raw=1'\n", 315 | "export_file_name = 'export.pkl'\n", 316 | "\n", 317 | "classes = ['black', 'grizzly', 'teddys']\n", 318 | "path = Path(__file__).parent\n", 319 | "\n", 320 | "app = Starlette()\n", 321 | "app.add_middleware(CORSMiddleware, allow_origins=['*'], allow_headers=['X-Requested-With', 'Content-Type'])\n", 322 | "app.mount('/static', StaticFiles(directory='app/static'))\n", 323 | "\n", 324 | "\n", 325 | "async def download_file(url, dest):\n", 326 | " if dest.exists(): return\n", 327 | " async with aiohttp.ClientSession() as session:\n", 328 | " async with session.get(url) as response:\n", 329 | " data = await response.read()\n", 330 | " with open(dest, 'wb') as f:\n", 331 | " f.write(data)\n", 332 | "\n", 333 | "\n", 334 | "async def setup_learner():\n", 335 | " await download_file(export_file_url, path / export_file_name)\n", 336 | " try:\n", 337 | " learn = load_learner(path, export_file_name)\n", 338 | " return learn\n", 339 | " except RuntimeError as e:\n", 340 | " if len(e.args) > 0 and 'CPU-only machine' in e.args[0]:\n", 341 | " print(e)\n", 342 | " message = \"\\n\\nThis model was trained with an old version of fastai and will not work in a CPU environment.\\n\\nPlease update the fastai library in your training environment and export your model again.\\n\\nSee instructions for 'Returning to work' at https://course.fast.ai.\"\n", 343 | " raise RuntimeError(message)\n", 344 | " else:\n", 345 | " raise\n", 346 | "\n", 347 | "\n", 348 | "loop = asyncio.get_event_loop()\n", 349 | "tasks = [asyncio.ensure_future(setup_learner())]\n", 350 | "learn = loop.run_until_complete(asyncio.gather(*tasks))[0]\n", 351 | "loop.close()\n", 352 | "\n", 353 | "\n", 354 | "@app.route('/')\n", 355 | "async def homepage(request):\n", 356 | " html_file = path / 'view' / 'index.html'\n", 357 | " return HTMLResponse(html_file.open().read())\n", 358 | "\n", 359 | "\n", 360 | "@app.route('/analyze', methods=['POST'])\n", 361 | "async def analyze(request):\n", 362 | " return JSONResponse({'result': str(prediction)})\n", 363 | "\n", 364 | "\n", 365 | "if __name__ == '__main__':\n", 366 | " if 'serve' in sys.argv:\n", 367 | " uvicorn.run(app=app, host='0.0.0.0', port=5000, log_level=\"info\")" 368 | ], 369 | "execution_count": 0, 370 | "outputs": [] 371 | }, 372 | { 373 | "cell_type": "markdown", 374 | "metadata": { 375 | "id": "XZjenLyaCFOL", 376 | "colab_type": "text" 377 | }, 378 | "source": [ 379 | "Wow! Lot's to unpack here! Let's go through it bit by bit.\n", 380 | "\n", 381 | "For our app to work, we need the following libraries:" 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "metadata": { 387 | "id": "xTkiytmmCOCD", 388 | "colab_type": "code", 389 | "colab": {} 390 | }, 391 | "source": [ 392 | "import aiohttp\n", 393 | "import asyncio\n", 394 | "import uvicorn\n", 395 | "from fastai import *\n", 396 | "from fastai.vision import *\n", 397 | "from io import BytesIO\n", 398 | "from starlette.applications import Starlette\n", 399 | "from starlette.middleware.cors import CORSMiddleware\n", 400 | "from starlette.responses import HTMLResponse, JSONResponse\n", 401 | "from starlette.staticfiles import StaticFiles" 402 | ], 403 | "execution_count": 0, 404 | "outputs": [] 405 | }, 406 | { 407 | "cell_type": "markdown", 408 | "metadata": { 409 | "id": "t3JaimLiCOf9", 410 | "colab_type": "text" 411 | }, 412 | "source": [ 413 | "* [aiohttp](https://aiohttp.readthedocs.io/en/stable/): HTTP client/server for asyncio\n", 414 | "* [asyncio](https://docs.python.org/3/library/asyncio.html): A python framework for good IO performances\n", 415 | "* fastai - We *need* the library for our model\n", 416 | "* [starlette](https://www.starlette.io/): \"A lightweight ASGI framework for high performance asyncio services\"" 417 | ] 418 | }, 419 | { 420 | "cell_type": "markdown", 421 | "metadata": { 422 | "id": "TlGX9T84FYSH", 423 | "colab_type": "text" 424 | }, 425 | "source": [ 426 | "### The *non* functions:\n", 427 | "\n", 428 | "By non-functions I mean the base variables. As you can see, we have a few things. First, we need a link to download our model, then some filename to give it, and lastly what classes we expect things to fall into." 429 | ] 430 | }, 431 | { 432 | "cell_type": "code", 433 | "metadata": { 434 | "id": "XcTwyU5TIeGu", 435 | "colab_type": "code", 436 | "colab": {} 437 | }, 438 | "source": [ 439 | "export_file_url = 'https://www.dropbox.com/s/6bgq8t6yextloqp/export.pkl?raw=1'\n", 440 | "export_file_name = 'export.pkl'\n", 441 | "\n", 442 | "classes = ['Abyssinian', 'Bengal', 'Birman', 'Bombay', 'British_Shorthair',\n", 443 | " 'Egyptian_Mau', 'Maine_Coon', 'Persian', 'Ragdoll', 'Russian_Blue',\n", 444 | " 'Siamese', 'Sphynx', 'american_bulldog', 'american_pit_bull_terrier',\n", 445 | " 'basset_hound', 'beagle', 'boxer', 'chihuahua', 'english_cocker_spaniel',\n", 446 | " 'english_setter', 'german_shorthaired', 'great_pyrenees', 'havanese',\n", 447 | " 'japanese_chin', 'keeshond', 'leonberger', 'miniature_pinscher',\n", 448 | " 'newfoundland', 'pomeranian', 'pug', 'saint_bernard', 'samoyed',\n", 449 | " 'scottish_terrier', 'shiba_inu', 'staffordshire_bull_terrier', 'wheaten_terrier',\n", 450 | " 'yorkshire_terrier']" 451 | ], 452 | "execution_count": 0, 453 | "outputs": [] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": { 458 | "id": "seOo5-0kDXMJ", 459 | "colab_type": "text" 460 | }, 461 | "source": [ 462 | "### The Functions:\n", 463 | "\n", 464 | "We have a few functions, 'analyze', 'homepage', 'download_files', and 'setup_learner'. They all do pretty much what we need to get our model' up and running. Analyze we can adjust to however we want our input data to come in. In our case, we expect the user to upload an image file, ad we extract those bytes, utilize the `open_image` function, and run `learn.predict()`. This will be one of the main functions we will change for our use cases." 465 | ] 466 | }, 467 | { 468 | "cell_type": "markdown", 469 | "metadata": { 470 | "id": "QXjmtEydExFt", 471 | "colab_type": "text" 472 | }, 473 | "source": [ 474 | "Now what is an app_route? Those mean seperate web-pages on our server." 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": { 480 | "id": "ZaNvUgDtFAJS", 481 | "colab_type": "text" 482 | }, 483 | "source": [ 484 | "#### analyze" 485 | ] 486 | }, 487 | { 488 | "cell_type": "code", 489 | "metadata": { 490 | "id": "NeGczuyMDAFx", 491 | "colab_type": "code", 492 | "colab": {} 493 | }, 494 | "source": [ 495 | "@app.route('/analyze', methods=['POST'])\n", 496 | "async def analyze(request):\n", 497 | " img_data = await request.form()\n", 498 | " img_bytes = await (img_data['file'].read())\n", 499 | " img = open_image(BytesIO(img_bytes))\n", 500 | " prediction = learn.predict(img)[0]\n", 501 | " return JSONResponse({'result': str(prediction)})" 502 | ], 503 | "execution_count": 0, 504 | "outputs": [] 505 | }, 506 | { 507 | "cell_type": "markdown", 508 | "metadata": { 509 | "id": "Mig9fKCNGvvC", 510 | "colab_type": "text" 511 | }, 512 | "source": [ 513 | "#### setup_learner and download_file\n", 514 | "\n", 515 | "We don't really need to mess with these as they do exactly what we need, regardless of the model, and they're pretty self-explanitory." 516 | ] 517 | }, 518 | { 519 | "cell_type": "markdown", 520 | "metadata": { 521 | "id": "s9VfgilwHa3x", 522 | "colab_type": "text" 523 | }, 524 | "source": [ 525 | "## Running our server!\n", 526 | "\n", 527 | "To run your server locally, navigate to your directory you cloned and run the following **if** you have Docker installed:\n", 528 | "\n", 529 | "`docker build -t fastai-v3 . && docker run --rm -it -p 5000:5000 fastai-v3`\n", 530 | "\n", 531 | "Else run: `python app/serve.py serve`\n", 532 | "\n", 533 | "Now we're running on http://localhost:5000/ !" 534 | ] 535 | }, 536 | { 537 | "cell_type": "markdown", 538 | "metadata": { 539 | "id": "oLNGjNPlH6ps", 540 | "colab_type": "text" 541 | }, 542 | "source": [ 543 | "Try it! Currently the HTML page is set up for Jeremy's lesson 2 where he built a bear classifier, but in our lesson today I won't go over HTML and web-page development, you guys can get that experience on your own." 544 | ] 545 | }, 546 | { 547 | "cell_type": "markdown", 548 | "metadata": { 549 | "id": "gj9qUBngJgYC", 550 | "colab_type": "text" 551 | }, 552 | "source": [ 553 | "# Other Model Types\n", 554 | "\n", 555 | "The next one we will look at is NLP-based models. I made an app for CodeFest last year called 'Suspecto' where we utilized an NLP model to help grade a models reliability. Using the standard Fast.AI approach we won first place. We'll borrow some of the website code for that setup today." 556 | ] 557 | }, 558 | { 559 | "cell_type": "markdown", 560 | "metadata": { 561 | "id": "rOPNpeUzKIIS", 562 | "colab_type": "text" 563 | }, 564 | "source": [ 565 | "## What do we change?\n", 566 | "\n", 567 | "First, change the imports to use whatever libraries you used. In my case, this involved the `fastai.text` library.\n", 568 | "\n", 569 | "Next, we changed analyze. Here, we wanted our model to take the results from `learn.predict()`, and based on a formula we had made as a reliability score, report this back to the user." 570 | ] 571 | }, 572 | { 573 | "cell_type": "code", 574 | "metadata": { 575 | "id": "LsqhtYNOKgqr", 576 | "colab_type": "code", 577 | "colab": {} 578 | }, 579 | "source": [ 580 | "@app.route('/analyze', methods=['POST'])\n", 581 | "async def analyze(request):\n", 582 | " data = await request.form()\n", 583 | " content = data['content']\n", 584 | " prediction = learn.predict(content)[2]\n", 585 | " reliability = prediction[7]-(prediction[6]*prediction[0]) - prediction[0] - prediction[4] - prediction[5] - prediction[8] - (prediction[1]-prediction[11])\n", 586 | " ReliabilityScore = ((reliability.item())*50)+50\n", 587 | " ReliabilityScore = int(ReliabilityScore)\n", 588 | " return JSONResponse({'result': ReliabilityScore})" 589 | ], 590 | "execution_count": 0, 591 | "outputs": [] 592 | }, 593 | { 594 | "cell_type": "markdown", 595 | "metadata": { 596 | "id": "yh-TomxzKsz0", 597 | "colab_type": "text" 598 | }, 599 | "source": [ 600 | "Here, notice instead we get ['content'] instead. This is due to our request form changing to a text box. To do this, we adjusted index.html. You see we have a special 'content' checker? This lives inside:" 601 | ] 602 | }, 603 | { 604 | "cell_type": "code", 605 | "metadata": { 606 | "id": "uZw2xQr2LheQ", 607 | "colab_type": "code", 608 | "colab": {} 609 | }, 610 | "source": [ 611 | "
\n", 612 | " \n", 613 | "
\n", 614 | " \n", 615 | "
\n", 616 | "
\n", 617 | " \n", 618 | "
\n", 619 | "
" 620 | ], 621 | "execution_count": 0, 622 | "outputs": [] 623 | }, 624 | { 625 | "cell_type": "markdown", 626 | "metadata": { 627 | "id": "xma07FLCLlHf", 628 | "colab_type": "text" 629 | }, 630 | "source": [ 631 | "So depending on what you want, see what the HTML equivalent is, and so long as you put it in, you can pass it to our analyze function! Now in our case, we want to instead do a movie review, so let's modify a few things!" 632 | ] 633 | }, 634 | { 635 | "cell_type": "code", 636 | "metadata": { 637 | "id": "26_5nXaAL0CE", 638 | "colab_type": "code", 639 | "colab": {} 640 | }, 641 | "source": [ 642 | "@app.route('/analyze', methods=['POST'])\n", 643 | "async def analyze(request):\n", 644 | " data = await request.form()\n", 645 | " content = data['content']\n", 646 | " prediction = learn.predict(content)[0]\n", 647 | " return JSONResponse({'Review Rating': str(prediction)})" 648 | ], 649 | "execution_count": 0, 650 | "outputs": [] 651 | }, 652 | { 653 | "cell_type": "markdown", 654 | "metadata": { 655 | "id": "E7ZMBQXfL-8l", 656 | "colab_type": "text" 657 | }, 658 | "source": [ 659 | "Looks pretty close to what we had before, doesn't it! " 660 | ] 661 | }, 662 | { 663 | "cell_type": "markdown", 664 | "metadata": { 665 | "id": "ionPEPDgMERm", 666 | "colab_type": "text" 667 | }, 668 | "source": [ 669 | "# Advanced Ideas and Tabular Production\n", 670 | "\n", 671 | "\n", 672 | "\n", 673 | "In the *real* world, we need to ask a few questions. How will I get my data? How will it be easiest for my customer to send me their data? For example. Take the image problem. If I am dealing with large sets of data they want me to analyze, I'm not going to expect my customer to sit there and upload 100's of files one by one and then we run them! That would be absurd. Instead, we can tell them to provide us with links to the relevant image websites they want us to run predictions with in the form of a CSV, *or* upload a zip document that contains all of their images into a nice folder structure that we can specify.\n" 674 | ] 675 | }, 676 | { 677 | "cell_type": "code", 678 | "metadata": { 679 | "id": "xdN-qLi0OodA", 680 | "colab_type": "code", 681 | "colab": {} 682 | }, 683 | "source": [ 684 | "import zipfile\n", 685 | "import csv\n", 686 | "\n", 687 | "@app.route('/analyze', methods=['POST'])\n", 688 | "async def analyze(request):\n", 689 | " data = await request.form()\n", 690 | " content = data['content']\n", 691 | " zip_ref = zipfile.ZipFile(content, 'r')\n", 692 | " mkdir('Downloaded_Images')\n", 693 | " zip_ref.extractall('Downloaded_Images')\n", 694 | " zip_ref.close()\n", 695 | " path2 = Path('Downloaded_Images')\n", 696 | " data = ImageList.from_folder(path)\n", 697 | " learn = load_learner(path, export_file_name, test=data)\n", 698 | " y, _ = learn.get_preds(DatasetType.Test)\n", 699 | " y = torch.argmax(y, dim=1)\n", 700 | " preds = [learn.data.classes[int(x)] for x in y]\n", 701 | " rm -r 'Downloaded_Images'\n", 702 | " resultsFile = open('results.csv', 'wb')\n", 703 | " wr = csv.writer(resultsFile)\n", 704 | " wr.writerows([preds])\n", 705 | " return FileResponse('results.csv')" 706 | ], 707 | "execution_count": 0, 708 | "outputs": [] 709 | }, 710 | { 711 | "cell_type": "markdown", 712 | "metadata": { 713 | "id": "FttEfV4WPeQT", 714 | "colab_type": "text" 715 | }, 716 | "source": [ 717 | "Now, lets parse this CSV when uploaded and download all of our images. We're going to use the `download_images()` function back from lesson 2 to help with this" 718 | ] 719 | }, 720 | { 721 | "cell_type": "code", 722 | "metadata": { 723 | "id": "lnrmj7-Nb575", 724 | "colab_type": "code", 725 | "colab": {} 726 | }, 727 | "source": [ 728 | "import csv\n", 729 | "import StringIO\n", 730 | "\n", 731 | "@app.route('/analyze', methods=['POST'])\n", 732 | "async def analyze(request):\n", 733 | " data = await request.form()\n", 734 | " content = await (data['file'].read())\n", 735 | " s = str(content, 'utf-8')\n", 736 | " data = StringIO(s)\n", 737 | " !mkdir('Downloaded_Images')\n", 738 | " download_images(data, 'Downloaded_Images')\n", 739 | " path2 = Path('Downloaded_Images')\n", 740 | " data = ImageList.from_folder(path)\n", 741 | " learn = load_learner(path, export_file_name, test=data)\n", 742 | " y, _ = learn.get_preds(DatasetType.Test)\n", 743 | " y = torch.argmax(y, dim=1)\n", 744 | " preds = [learn.data.classes[int(x)] for x in y]\n", 745 | " rm -r 'Downloaded_Images'\n", 746 | " resultsFile = open('results.csv', 'wb')\n", 747 | " wr = csv.writer(resultsFile)\n", 748 | " wr.writerows([preds])\n", 749 | " return FileResponse('results.csv')" 750 | ], 751 | "execution_count": 0, 752 | "outputs": [] 753 | }, 754 | { 755 | "cell_type": "markdown", 756 | "metadata": { 757 | "id": "j8mqJZ7dOvDu", 758 | "colab_type": "text" 759 | }, 760 | "source": [ 761 | "Now, tabular is a pretty special case. In the business world, most work will be done by sending in large chunks of data for analysis on a served model *somewhere*. The company may have it hooked into a GPU to account for the faster time, and having it shut off and on based on when the server gets a request. The models we export are generally CPU based models, but we can adjust for this if needed.\n", 762 | "\n", 763 | "Now let's recreate what we did above for tabular data. Instead we will load it into pandas." 764 | ] 765 | }, 766 | { 767 | "cell_type": "code", 768 | "metadata": { 769 | "id": "DNuwpWKGeMpV", 770 | "colab_type": "code", 771 | "colab": {} 772 | }, 773 | "source": [ 774 | "import StringIO\n", 775 | "import csv\n", 776 | "\n", 777 | "@app.route('/analyze', methods=['POST'])\n", 778 | "async def analyze(request):\n", 779 | " data = await request.form()\n", 780 | " content = await (data['file'].read())\n", 781 | " s = str(content, 'utf-8')\n", 782 | " data = StringIO(s)\n", 783 | " df = pd.read_csv(data)\n", 784 | " data = TabularList.from_df(df, path='', cat_names = cat_names,\n", 785 | " cont_names = cont_names, procs = procs)\n", 786 | " learn = load_learner(path, export_file_name, test=data)\n", 787 | " y, _ = learn.get_preds(DatasetType.Test)\n", 788 | " y = torch.argmax(y, dim=1)\n", 789 | " preds = [learn.data.classes[int(x)] for x in y]\n", 790 | " df['Predictions'] = preds\n", 791 | " \n", 792 | " path3 = Path('app/static/')\n", 793 | " df.to_csv(path3/'results.csv')\n", 794 | " \n", 795 | " return FileRespose('results.csv', media_type='csv')" 796 | ], 797 | "execution_count": 0, 798 | "outputs": [] 799 | }, 800 | { 801 | "cell_type": "markdown", 802 | "metadata": { 803 | "id": "1hUg6nIEeohZ", 804 | "colab_type": "text" 805 | }, 806 | "source": [ 807 | "We also need to adjust our JavaScript a little bit too. Specifically the end bit of the Anlyze function:" 808 | ] 809 | }, 810 | { 811 | "cell_type": "code", 812 | "metadata": { 813 | "id": "NWBTJ3cgex2v", 814 | "colab_type": "code", 815 | "colab": {} 816 | }, 817 | "source": [ 818 | " xhr.onload = function(e) {\n", 819 | " if (this.readyState === 4) {\n", 820 | " el(\"result-label\").innerHTML = `Result = Good`;\n", 821 | " \n", 822 | " download('results.csv', 'results.csv');\n", 823 | " xhr.send();\n", 824 | " }\n", 825 | " el(\"analyze-button\").innerHTML = \"Analyze\";\n", 826 | " };\n", 827 | "\n", 828 | " var fileData = new FormData();\n", 829 | " fileData.append(\"file\", uploadFiles[0]);\n", 830 | " xhr.send(fileData);\n", 831 | "}" 832 | ], 833 | "execution_count": 0, 834 | "outputs": [] 835 | }, 836 | { 837 | "cell_type": "markdown", 838 | "metadata": { 839 | "id": "io4oP-QBey6j", 840 | "colab_type": "text" 841 | }, 842 | "source": [ 843 | "As we are now taking FormData" 844 | ] 845 | } 846 | ] 847 | } -------------------------------------------------------------------------------- /08a_Callbacks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "CallBacks.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "collapsed_sections": [ 10 | "GBiBDXdPXKpv", 11 | "6Dnk5fpjZafV", 12 | "yOmj0ITGZ-R2" 13 | ] 14 | }, 15 | "kernelspec": { 16 | "name": "python3", 17 | "display_name": "Python 3" 18 | } 19 | }, 20 | "cells": [ 21 | { 22 | "cell_type": "markdown", 23 | "metadata": { 24 | "id": "temuo4OuSodZ", 25 | "colab_type": "text" 26 | }, 27 | "source": [ 28 | "# Callbacks" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": { 34 | "id": "bKWd_DWrzbP5", 35 | "colab_type": "text" 36 | }, 37 | "source": [ 38 | "| | Tuesday 4-5:15pm | Friday 4-5:30pm |\n", 39 | "|:------:|:-------------------------------------------:|:--------------------------------------------------------------------------:|\n", 40 | "| **Week 1** | Introduction | Introduction |\n", 41 | "| **Week 2** | Custom computer vision tasks | State of the art in Computer Vision |\n", 42 | "| **Week 3** | Introduction to Tabular modeling and pandas | Pandas workshop and feature engineering |\n", 43 | "| **Week 4** | Tabular and Image Regression | Feature importance and advanced feature engineering |\n", 44 | "| **Week 5** | Natural Language Processing | State of the art in NLP |\n", 45 | "| **Week 6** | Segmentation and Kaggle | Audio |\n", 46 | "| **Week 7** | Computer vision from scratch | NLP from scratch |\n", 47 | "| **Week 8** | **Callbacks** | Optimizers |\n", 48 | "| **Week 9** | Generative Adversarial Networks | Research time / presentations |\n", 49 | "| **Week 10** | Putting models into production | Putting models into production |" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": { 55 | "id": "eqZlXmgdcNSl", 56 | "colab_type": "text" 57 | }, 58 | "source": [ 59 | "

Schedule for the next few weeks:

\n", 60 | "\n", 61 | "**Week 5:** Callbacks\n", 62 | "\n", 63 | "**Week 6:** Optimizers\n", 64 | "\n", 65 | "**Week 7:** Loss Functions\n", 66 | "\n", 67 | "**Week 8:** Custom Architecture Introduction" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "id": "i4ogQYd1TMex", 74 | "colab_type": "text" 75 | }, 76 | "source": [ 77 | "

Key Terms and Definitions:

\n", 78 | "\n", 79 | "**Model**: 'A mathematical representation of a real-world-process,' in our case consists of an architecture and we call these Deep Learning Models\n", 80 | "\n", 81 | "**Loss**: The loss function associated with grading performance to the model.\n", 82 | "\n", 83 | "**Metric**: How we grade the workings of our model in a 'pretty' way. Eg: accuracy vs Cross Entropy Loss\n", 84 | "\n", 85 | "**Gradient Decent**: Optimization algorithm used to update the parameters of our model, help 'fit'\n", 86 | "\n", 87 | "**Callback**: A function that occurs in case something happens\n" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": { 93 | "id": "WCSvb6iWS-0g", 94 | "colab_type": "text" 95 | }, 96 | "source": [ 97 | "Everything in Fast.AI is done during the training loop. Every single step inside of it can be altered via a callback.\n", 98 | "\n", 99 | "

Examples of useful callbacks, some you already know!

\n", 100 | "\n", 101 | "* OneCycleScheduler\n", 102 | "\n", 103 | "* LRFinder\n", 104 | "\n", 105 | "* SaveModelCallback\n", 106 | "* CSVLogger\n", 107 | "* MixedPrecision\n", 108 | "\n", 109 | "Today we are going to go through how each of these work, what it's like building callbacks within fastai, and what all can be done\n", 110 | "\n", 111 | "\n" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": { 117 | "id": "ioTFxNhdWkhB", 118 | "colab_type": "text" 119 | }, 120 | "source": [ 121 | "

The basic training loop

" 122 | ] 123 | }, 124 | { 125 | "cell_type": "markdown", 126 | "metadata": { 127 | "id": "cXmaYCdWWdWw", 128 | "colab_type": "text" 129 | }, 130 | "source": [ 131 | "Here is the code" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "metadata": { 137 | "id": "DGng3fOOSmSd", 138 | "colab_type": "code", 139 | "colab": {} 140 | }, 141 | "source": [ 142 | "# train_dl is the dataloader for your training set in the databunch. \n", 143 | "# It is a link to where your databunch set is saved\n", 144 | "# opt_fn is the optimization function, there are a few. We use Adam.\n", 145 | "# Standard Gradient Decent is also commonly found in papers and models\n", 146 | "def train(train_dl, epochs, opt_fn, loss_func):\n", 147 | " for _ in range(epoch):\n", 148 | " # Well what is _? _ is a placeholder for some increment we dont care about modifying, \n", 149 | " # see Code Examples section for more\n", 150 | " model.train() # set the model to train mode\n", 151 | " for xb,yb in train_dl: # for each batch of x and y in the dataset\n", 152 | " out = model(xb) # This is our forward pass, we get our models output\n", 153 | " loss = loss_func(out, yb) # We take the models output and our label and grade\n", 154 | " loss.backward() # Our model's backward pass, accumulates gradients (graph)\n", 155 | " opt_fn.step() # Updates a parameter on the current gradient\n", 156 | " opt_fn.zero_grad() # zeros our gradients" 157 | ], 158 | "execution_count": 0, 159 | "outputs": [] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": { 164 | "id": "ZjNFBkHMWoQh", 165 | "colab_type": "text" 166 | }, 167 | "source": [ 168 | "

Pytorch for comparison

" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "metadata": { 174 | "id": "8_UJlOtyWq_6", 175 | "colab_type": "code", 176 | "colab": {} 177 | }, 178 | "source": [ 179 | "def pyTrain(train_dl, model, opt_func, loss_func):\n", 180 | " losses = AverageMeter('Loss', ':.4e') # how we want to see the losses\n", 181 | " top1 = AverageMeter('Acc@1', ':6.2f')\n", 182 | " top5 = AverageMeter('Acc@5, '6.2f)\n", 183 | " \n", 184 | " model.train() # set model to train\n", 185 | " for i, (images, target) in enumerate(train_dl): # for every item in the training loader\n", 186 | " target = target.cuda(args.gpu, non_blocking=True) # load the target tensor to the GPU\n", 187 | " \n", 188 | " output = model(images) # compute output and loss\n", 189 | " loss = criterion(output, target)\n", 190 | " \n", 191 | " acc1, acc5 = accuracy(output, target, topk=(1,5)) # topk returns top 'x' predictions\n", 192 | " losses.update(loss.item(), images.size(0))\n", 193 | " top1.update(acc1[0], images.size(0))\n", 194 | " top5.update(acc5[0], images.size(0))\n", 195 | " \n", 196 | " \n", 197 | " optimizer.zero_grad() # zero our gradient\n", 198 | " loss.backward() # backward pass\n", 199 | " optimizer.step() # Update parameters" 200 | ], 201 | "execution_count": 0, 202 | "outputs": [] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": { 207 | "id": "9FDoSVmgS5O-", 208 | "colab_type": "text" 209 | }, 210 | "source": [ 211 | "![alt text](https://pouannes.github.io/fastai-callbacks/fastai_training_loop_vanilla.png#center)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": { 217 | "id": "3Z7o3TafYKKZ", 218 | "colab_type": "text" 219 | }, 220 | "source": [ 221 | "Now lets view where the callback functionalities can apply" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "metadata": { 227 | "id": "8Ohxf6ZxYPJb", 228 | "colab_type": "code", 229 | "colab": {} 230 | }, 231 | "source": [ 232 | "def train(train_dl, epoch, opt_fn, loss_func):\n", 233 | " callbacks.on_train_begin() # we can call even before we do any epochs\n", 234 | " \n", 235 | " for _ in range(epochs):\n", 236 | " callbacks.on_epoch_begin() # we can call here\n", 237 | " model.train()\n", 238 | " \n", 239 | " for xb,yb in train_dl:\n", 240 | " callbacks.on_batch_begin() # even before we get our output!\n", 241 | " out = model(xb)\n", 242 | " \n", 243 | " callbacks.on_loss_begin() # here we have the output and need to modify it\n", 244 | " loss = loss_func(out, yb)\n", 245 | " \n", 246 | " callback.on_backward_begin() # our start of backward pass\n", 247 | " loss.backward()\n", 248 | " callbacks.on_backward_end() # end of our backward pass\n", 249 | " \n", 250 | " opt_fn.step()\n", 251 | " callbacks.on_step_end() # after we update a parameter\n", 252 | " \n", 253 | " opt.zero_grad()\n", 254 | " callbacks.on_batch_end() # after we finish a batch\n", 255 | " \n", 256 | " callbacks.on_epoch_end() # end of our epoch\n", 257 | " \n", 258 | " callbacks.on_train_end() # end of our training" 259 | ], 260 | "execution_count": 0, 261 | "outputs": [] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": { 266 | "id": "-HIrIiQvZPOz", 267 | "colab_type": "text" 268 | }, 269 | "source": [ 270 | "![alt text](https://pouannes.github.io/fastai-callbacks/fastai_training_loop_callbacks.png#center)" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "id": "SS8STNQcZzae", 277 | "colab_type": "text" 278 | }, 279 | "source": [ 280 | "

Callback Handler

\n", 281 | "\n", 282 | "We can do most of whatever we want but we're missing something. How do they handle ANYTHING without access to the state of training itself? (see they're around but not within?)\n", 283 | "\n", 284 | "Welcome to the CallbackHandler\n", 285 | "\n", 286 | "The CallbackHandler takes relevent data and transmits it to the callbacks and returns value depending on behaviour" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "metadata": { 292 | "id": "6yPCWwnaafbV", 293 | "colab_type": "text" 294 | }, 295 | "source": [ 296 | "Let's have an example. We can use callbacks to send the new training data, *xb* and and *yb* for the batch.\n", 297 | "Here we will have a CallbackHandler called cb_handler:" 298 | ] 299 | }, 300 | { 301 | "cell_type": "code", 302 | "metadata": { 303 | "id": "Ec3JBX7zaV3x", 304 | "colab_type": "code", 305 | "colab": {} 306 | }, 307 | "source": [ 308 | "cb_handler.on_batch_begin(xb,yb)" 309 | ], 310 | "execution_count": 0, 311 | "outputs": [] 312 | }, 313 | { 314 | "cell_type": "markdown", 315 | "metadata": { 316 | "id": "N-wS0bQZat_4", 317 | "colab_type": "text" 318 | }, 319 | "source": [ 320 | "Now lets think of another example. I want to skip the backward pass if my loss is too high. Also, if my loss is too small, perhaps scale it up!" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "metadata": { 326 | "id": "VAiAMOVJa3JP", 327 | "colab_type": "code", 328 | "colab": {} 329 | }, 330 | "source": [ 331 | "loss, skip_backward = cb_handler.on_backward_begin(loss)\n", 332 | "if not skip_backward: loss.backward() # if we decide we don't want to skip backward" 333 | ], 334 | "execution_count": 0, 335 | "outputs": [] 336 | }, 337 | { 338 | "cell_type": "markdown", 339 | "metadata": { 340 | "id": "EXfhiy4IbLCV", 341 | "colab_type": "text" 342 | }, 343 | "source": [ 344 | "The new training loop" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "metadata": { 350 | "id": "V8I5n5rJbNFL", 351 | "colab_type": "code", 352 | "colab": {} 353 | }, 354 | "source": [ 355 | "def train(learn, epochs, callbacks, metrics):\n", 356 | " cb_handler = CallbackHandler(callbacks)\n", 357 | " cb_handler.on_train_begin(epochs, learn, metrics)\n", 358 | " \n", 359 | " for epoch in range(epochs):\n", 360 | " learn.model.train()\n", 361 | " cb_handler.on_epoch_begin(epoch)\n", 362 | " \n", 363 | " for xb,yb in train_dl:\n", 364 | " xb,yb = cb_handler.on_batch_begin(xb,yb)\n", 365 | " \n", 366 | " out = learn.model(xb)\n", 367 | " out = cb_handler.on_loss_begin(out)\n", 368 | " \n", 369 | " loss = learn.loss_func(out, yb)\n", 370 | " \n", 371 | " loss, skip_backward = cb_handler.on_backward_begin(loss)\n", 372 | " if not skip_backward: loss.backward()\n", 373 | " if not cb_handler.on_backward_end(): learn.opt.step()\n", 374 | " \n", 375 | " if not cb_handler.on_step_end(): learn.opt.zero_grad()\n", 376 | " if not cb_handler.on_batch_end(): break\n", 377 | " \n", 378 | " val_loss, mets = validate(learn.data.valid_dl, model, metrics)\n", 379 | " if not cb_handler.on_epoch_end(val_loss, mets): break\n", 380 | " \n", 381 | " cb_handler.on_train_end()" 382 | ], 383 | "execution_count": 0, 384 | "outputs": [] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": { 389 | "id": "ceBsPGc8W6nT", 390 | "colab_type": "text" 391 | }, 392 | "source": [ 393 | "

Let's build a few

\n", 394 | "\n", 395 | "I will have 3 examples for us to follow today, OneCycleScheduler, LRFinder, and CSV Logger." 396 | ] 397 | }, 398 | { 399 | "cell_type": "markdown", 400 | "metadata": { 401 | "id": "0DbE30InXD0P", 402 | "colab_type": "text" 403 | }, 404 | "source": [ 405 | "What is OneCycleScheduler? Based on this [paper](https://arxiv.org/pdf/1803.09820.pdf) by Leslie Smith. \n", 406 | "From the documentation: \n", 407 | "\n", 408 | "\"Create a Callback that handles the hyperparameters settings following the 1cycle policy for learn. lr_max should be picked with the lr_find test. In phase 1, the learning rates goes from lr_max/div_factor to lr_max linearly while the momentum goes from moms[0] to moms[1] linearly. In phase 2, the learning rates follows a cosine annealing from lr_max to 0, as the momentum goes from moms[1] to moms[0] with the same annealing.\"\"\n", 409 | "\n", 410 | "https://docs.fast.ai/callbacks.one_cycle.html#OneCycleScheduler\n", 411 | "\n", 412 | "https://docs.fast.ai/callbacks.one_cycle.html#What-is-1cycle?\n", 413 | "\n", 414 | "\n", 415 | "* One difference to note: in pytorch to break we pass False, in fastai we pass True " 416 | ] 417 | }, 418 | { 419 | "cell_type": "code", 420 | "metadata": { 421 | "id": "Chp6UIKmiBaY", 422 | "colab_type": "code", 423 | "colab": {} 424 | }, 425 | "source": [ 426 | "from fastai.torch_core import *\n", 427 | "from fastai.basic_data import DataBunch\n", 428 | "from fastai.callback import *\n", 429 | "from fastai.basic_train import Learner, LearnerCallback" 430 | ], 431 | "execution_count": 0, 432 | "outputs": [] 433 | }, 434 | { 435 | "cell_type": "markdown", 436 | "metadata": { 437 | "id": "6MU7iCfZfB7J", 438 | "colab_type": "text" 439 | }, 440 | "source": [ 441 | "

OneCycleScheduler

" 442 | ] 443 | }, 444 | { 445 | "cell_type": "code", 446 | "metadata": { 447 | "id": "x4KHdBqKd0NQ", 448 | "colab_type": "code", 449 | "colab": {} 450 | }, 451 | "source": [ 452 | "class OneCycleScheduler(LearnerCallback): # We give it a name and say we inherit from the LearnerCallback class\n", 453 | " def __init__(self, learn:Learner, lr_max:float, moms:Floats=(0.95,0.85), # pass in learner, a max lr, and momentums\n", 454 | " div_factor:float=25., pct_start:float=0.3, final_div:float=None, # pct_start is how far into training we start slowing the speed\n", 455 | " tot_epochs:int=None, start_epoch:int=None):\n", 456 | " super().__init__(learn)\n", 457 | " self.lr_max = lr_max\n", 458 | " self.div_factor = div_factor\n", 459 | " self.pct_start = pct_start\n", 460 | " self.final_div = final_div\n", 461 | " \n", 462 | " if self.final_div is None: self.final_div = div_factor * 1e4\n", 463 | " \n", 464 | " self.moms = tuple(listify(moms,2)) # see below what listify is\n", 465 | " if is_listy(self.lr_max): self.lr_max = np.array(self.lr_max)\n", 466 | " \n", 467 | " self.start_epoch, self.tot_epochs = start_epoch, tot_epoch" 468 | ], 469 | "execution_count": 0, 470 | "outputs": [] 471 | }, 472 | { 473 | "cell_type": "code", 474 | "metadata": { 475 | "id": "zB7Z11Dgd2tq", 476 | "colab_type": "code", 477 | "colab": {} 478 | }, 479 | "source": [ 480 | " def steps(self, *steps_cfg:StartOptEnd):\n", 481 | " 'Build a learning rate schedule for all parameters'\n", 482 | " return [Scheduler(step, n_iter, func=func)\n", 483 | " for (step, (n_iter,func)) in zip(steps_cfg, self.phases)]" 484 | ], 485 | "execution_count": 0, 486 | "outputs": [] 487 | }, 488 | { 489 | "cell_type": "code", 490 | "metadata": { 491 | "id": "Gy9MYqFcd3Pe", 492 | "colab_type": "code", 493 | "colab": {} 494 | }, 495 | "source": [ 496 | " def on_train_begin(self, n_epochs:int, epoch:int, **kwargs:Any)->None:\n", 497 | " 'Initialize the parameters bsed on the schedule above'\n", 498 | " res = {'epoch':self.start_epoch} if self.start_epoch is not None else None\n", 499 | " self.start_epoch = ifnone(self.start_epoch, epoch)\n", 500 | " self.tot_epochs = ifnone(self.tot_epochs, n_epochs)\n", 501 | " \n", 502 | " n = len(self.learn.data.train_dl) * self.tot_epochs\n", 503 | " a1 = int(n*self.pct_start)\n", 504 | " a2 = n-a1\n", 505 | " \n", 506 | " self.phases = ((a1, annealing_cos), (a2, annealing_cos))\n", 507 | " low_lr = self.lr_max/self.div_factor # sets our lower LR\n", 508 | " \n", 509 | " self.lr_scheds = self.steps((low_lr, self.lr_max),\n", 510 | " (self.lr_max, self.lr_max/self.final_div))\n", 511 | " \n", 512 | " self.moms_scheds = self.steps(self.moms, (self.moms[1], self.moms[0]))\n", 513 | " \n", 514 | " self.opt = self.learn.opt\n", 515 | " self.opt.lr, self.opt.mom = self.lr_scheds[0].start, self.mom_scheds[0].start\n", 516 | " self.idx_s = 0\n", 517 | " return res" 518 | ], 519 | "execution_count": 0, 520 | "outputs": [] 521 | }, 522 | { 523 | "cell_type": "code", 524 | "metadata": { 525 | "id": "NdioqzqJd6jg", 526 | "colab_type": "code", 527 | "colab": {} 528 | }, 529 | "source": [ 530 | " def jump_to_epoch(self, epoch:int)->None:\n", 531 | " for _ in range(len(self.learn.data.train_dl) * epoch):\n", 532 | " self.on_batch_end(True)" 533 | ], 534 | "execution_count": 0, 535 | "outputs": [] 536 | }, 537 | { 538 | "cell_type": "code", 539 | "metadata": { 540 | "id": "74Q-YhdVeFdz", 541 | "colab_type": "code", 542 | "colab": {} 543 | }, 544 | "source": [ 545 | " def on_batch_end(self, train, **kwargs:Any)->None:\n", 546 | " 'Take one step forward on the schedule for optim parameters'\n", 547 | " if train:\n", 548 | " if self.idx_s >= len(self.lr_scheds): return {'stop_training': True, 'stop_epoch': True}\n", 549 | " self.opt.lr = self.lr_scheds[self.idx_s].step()\n", 550 | " self.opt.mom = self.moms_scheds[self.idx_s].step()\n", 551 | " \n", 552 | " # When one scheduler is done, move to the next. \n", 553 | " # (in one cycle we have two)\n", 554 | " \n", 555 | " if self.lr_scheds[self.idx_s].is_done:\n", 556 | " self.idx_s += 1" 557 | ], 558 | "execution_count": 0, 559 | "outputs": [] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "metadata": { 564 | "id": "UfeYJBMGenO6", 565 | "colab_type": "code", 566 | "colab": {} 567 | }, 568 | "source": [ 569 | " def on_epoch_end(self, epoch, **kwargs:Any)->None:\n", 570 | " 'Tell the learner to stop training on finish'\n", 571 | " if epoch > self.tot_epochs: return {'stop_training': True}" 572 | ], 573 | "execution_count": 0, 574 | "outputs": [] 575 | }, 576 | { 577 | "cell_type": "markdown", 578 | "metadata": { 579 | "id": "8Q-hAitDfFkj", 580 | "colab_type": "text" 581 | }, 582 | "source": [ 583 | "

LRFinder

" 584 | ] 585 | }, 586 | { 587 | "cell_type": "code", 588 | "metadata": { 589 | "id": "Pd-xvAEhfHyU", 590 | "colab_type": "code", 591 | "colab": {} 592 | }, 593 | "source": [ 594 | "class LRFinder(LearnerCallback):\n", 595 | " \"Mock training from `start_lr` to `end_lr` for `num_it` iterations.\"\n", 596 | " def __init__(self, learn:Learner, start_lr:float=1e-7, end_lr:float=10, \n", 597 | " num_it:int=100, stop_div:bool=True):\n", 598 | " super().__init__(learn)\n", 599 | " self.data = learn.data\n", 600 | " self.stop_div = stop_div\n", 601 | " self.sched = Scheduler((start_lr, end_lr), num_it, annealing_exp) # annealing exp is a function" 602 | ], 603 | "execution_count": 0, 604 | "outputs": [] 605 | }, 606 | { 607 | "cell_type": "code", 608 | "metadata": { 609 | "id": "xvcpqyVRf4pQ", 610 | "colab_type": "code", 611 | "colab": {} 612 | }, 613 | "source": [ 614 | " def on_train_begin(self, pbar, **kwargs:Any)->None:\n", 615 | " 'Initialize optimizer and hyperparameters'\n", 616 | " setattr(pbar, 'clean_on_interrupt', True)\n", 617 | " # pbar.clean_on_interrupt = True\n", 618 | " \n", 619 | " self.learn.save('tmp')\n", 620 | " self.opt = self.learn.opt\n", 621 | " self.opt.lr = self.sched.start\n", 622 | " self.stop = False\n", 623 | " self.best_loss = 0.\n", 624 | " return {'skip_validate': True}" 625 | ], 626 | "execution_count": 0, 627 | "outputs": [] 628 | }, 629 | { 630 | "cell_type": "code", 631 | "metadata": { 632 | "id": "lv8cwo2xguZJ", 633 | "colab_type": "code", 634 | "colab": {} 635 | }, 636 | "source": [ 637 | " def on_batch_end(self, iteration:int, smooth_loss:TensorOrNumber, **kwargs:Any)->None:\n", 638 | " 'Determine if loss is getting exponentially worse quickly'\n", 639 | " if iteration == 0 or smooth_loss < self.best_loss: self.best_loss = smooth_loss\n", 640 | " \n", 641 | " self.opt.lr = self.sched.step() # go to the next LR available in our two steps\n", 642 | " \n", 643 | " if self.sched.is_done or (self.stop_div and (smooth_loss > 4*self.best_loss or torch.isnan(smooth_loss))):\n", 644 | " 'Use the smooth loss to decide if stopping since it is less shaky'\n", 645 | " return {'stop_epoch': True, 'stop_training': True} # end epoch early and stop the training" 646 | ], 647 | "execution_count": 0, 648 | "outputs": [] 649 | }, 650 | { 651 | "cell_type": "code", 652 | "metadata": { 653 | "id": "SiN9Cirkherg", 654 | "colab_type": "code", 655 | "colab": {} 656 | }, 657 | "source": [ 658 | " def on_train_end(self, **kwargs:Any)->None:\n", 659 | " 'Cleanup the weights'\n", 660 | " self.learn.load('tmp', purge=False)\n", 661 | " if hasattr(self.learn.model, 'reset'): self.learn.model.reset()\n", 662 | " for cb in self.callbacks:\n", 663 | " if hasattr(cb, 'reset'): cb.reset()\n", 664 | " print('LR Finder is complete, type {learner_name}.recorder.plot() to see the graph')" 665 | ], 666 | "execution_count": 0, 667 | "outputs": [] 668 | }, 669 | { 670 | "cell_type": "markdown", 671 | "metadata": { 672 | "id": "nPqKub6HsGCS", 673 | "colab_type": "text" 674 | }, 675 | "source": [ 676 | "

Accuracy

" 677 | ] 678 | }, 679 | { 680 | "cell_type": "markdown", 681 | "metadata": { 682 | "id": "7O1bW7CDshrs", 683 | "colab_type": "text" 684 | }, 685 | "source": [ 686 | "

Metric

" 687 | ] 688 | }, 689 | { 690 | "cell_type": "code", 691 | "metadata": { 692 | "id": "rc58u5v1ska1", 693 | "colab_type": "code", 694 | "colab": {} 695 | }, 696 | "source": [ 697 | "def accuracy(input:Tensor, targs:Tensor)->Rank0Tensor:\n", 698 | " \"Compute accuracy with `targs` when `input` is bs * n_classes.\"\n", 699 | " n = targs.shape[0]\n", 700 | " input = input.argmax(dim=-1).view(n,-1)\n", 701 | " targs = targs.view(n,-1)\n", 702 | " return (input==targs).float().mean()" 703 | ], 704 | "execution_count": 0, 705 | "outputs": [] 706 | }, 707 | { 708 | "cell_type": "markdown", 709 | "metadata": { 710 | "id": "4xXlxuirsZjJ", 711 | "colab_type": "text" 712 | }, 713 | "source": [ 714 | "

Callback

" 715 | ] 716 | }, 717 | { 718 | "cell_type": "code", 719 | "metadata": { 720 | "id": "kMmW3QYzsNRs", 721 | "colab_type": "code", 722 | "colab": {} 723 | }, 724 | "source": [ 725 | "class AverageMetric(Callback):\n", 726 | " \"Wrap a `func` in a callback for metrics computation.\"\n", 727 | " def __init__(self, func):\n", 728 | " # If it's a partial, use func.func\n", 729 | " name = getattr(func,'func',func).__name__\n", 730 | " self.func, self.name = func, name\n", 731 | "\n", 732 | " def on_epoch_begin(self, **kwargs):\n", 733 | " \"Set the inner value to 0.\"\n", 734 | " self.val, self.count = 0.,0\n", 735 | "\n", 736 | " def on_batch_end(self, last_output, last_target, **kwargs):\n", 737 | " \"Update metric computation with `last_output` and `last_target`.\"\n", 738 | " if not is_listy(last_target): last_target=[last_target]\n", 739 | " self.count += last_target[0].size(0)\n", 740 | " val = self.func(last_output, *last_target)\n", 741 | " self.val += last_target[0].size(0) * val.detach().cpu()\n", 742 | "\n", 743 | " def on_epoch_end(self, last_metrics, **kwargs):\n", 744 | " \"Set the final result in `last_metrics`.\"\n", 745 | " return add_metrics(last_metrics, self.val/self.count)" 746 | ], 747 | "execution_count": 0, 748 | "outputs": [] 749 | }, 750 | { 751 | "cell_type": "code", 752 | "metadata": { 753 | "id": "vJuJ0SZ4sN3c", 754 | "colab_type": "code", 755 | "colab": {} 756 | }, 757 | "source": [ 758 | "{'last_metrics': last_metrics + [self.val/self.count]}" 759 | ], 760 | "execution_count": 0, 761 | "outputs": [] 762 | }, 763 | { 764 | "cell_type": "markdown", 765 | "metadata": { 766 | "id": "LoaJ8ZhAbQMw", 767 | "colab_type": "text" 768 | }, 769 | "source": [ 770 | "# Useful Links" 771 | ] 772 | }, 773 | { 774 | "cell_type": "markdown", 775 | "metadata": { 776 | "id": "oAMW4FN7bZR6", 777 | "colab_type": "text" 778 | }, 779 | "source": [ 780 | "https://docs.fast.ai/callbacks.html\n", 781 | "\n", 782 | "https://docs.fast.ai/callback.html\n", 783 | "\n", 784 | "https://forums.fast.ai/t/using-the-callback-system-in-fastai/16216\n" 785 | ] 786 | }, 787 | { 788 | "cell_type": "markdown", 789 | "metadata": { 790 | "id": "4DkMkRmTXc4d", 791 | "colab_type": "text" 792 | }, 793 | "source": [ 794 | "# Code Examples" 795 | ] 796 | }, 797 | { 798 | "cell_type": "markdown", 799 | "metadata": { 800 | "id": "GBiBDXdPXKpv", 801 | "colab_type": "text" 802 | }, 803 | "source": [ 804 | "### for _ in range(epoch)" 805 | ] 806 | }, 807 | { 808 | "cell_type": "code", 809 | "metadata": { 810 | "id": "M4LYiUj3XNE5", 811 | "colab_type": "code", 812 | "outputId": "9bfe1d8b-b890-4a32-f1e1-71e706aa0192", 813 | "colab": { 814 | "base_uri": "https://localhost:8080/", 815 | "height": 68 816 | } 817 | }, 818 | "source": [ 819 | "for _ in range(3):\n", 820 | " print('hello')" 821 | ], 822 | "execution_count": 0, 823 | "outputs": [ 824 | { 825 | "output_type": "stream", 826 | "text": [ 827 | "hello\n", 828 | "hello\n", 829 | "hello\n" 830 | ], 831 | "name": "stdout" 832 | } 833 | ] 834 | }, 835 | { 836 | "cell_type": "markdown", 837 | "metadata": { 838 | "id": "6Dnk5fpjZafV", 839 | "colab_type": "text" 840 | }, 841 | "source": [ 842 | "### Listify" 843 | ] 844 | }, 845 | { 846 | "cell_type": "code", 847 | "metadata": { 848 | "id": "SGNx5KPAZs8E", 849 | "colab_type": "code", 850 | "colab": {} 851 | }, 852 | "source": [ 853 | "from fastai.core import *\n", 854 | "from fastai.callback import *\n", 855 | "from fastai.basic_train import *" 856 | ], 857 | "execution_count": 0, 858 | "outputs": [] 859 | }, 860 | { 861 | "cell_type": "code", 862 | "metadata": { 863 | "id": "esfggNDnZdGv", 864 | "colab_type": "code", 865 | "outputId": "ceb0b411-2aaa-4166-b5dd-d83bdb09ba2b", 866 | "colab": { 867 | "base_uri": "https://localhost:8080/", 868 | "height": 34 869 | } 870 | }, 871 | "source": [ 872 | "moms = (0.95,0.85)\n", 873 | "listify(moms,2)" 874 | ], 875 | "execution_count": 0, 876 | "outputs": [ 877 | { 878 | "output_type": "execute_result", 879 | "data": { 880 | "text/plain": [ 881 | "[0.95, 0.85]" 882 | ] 883 | }, 884 | "metadata": { 885 | "tags": [] 886 | }, 887 | "execution_count": 4 888 | } 889 | ] 890 | }, 891 | { 892 | "cell_type": "code", 893 | "metadata": { 894 | "id": "iP-0entAZhgh", 895 | "colab_type": "code", 896 | "outputId": "6cff9dd5-0ec7-4f24-9c2d-a48aabb64310", 897 | "colab": { 898 | "base_uri": "https://localhost:8080/", 899 | "height": 34 900 | } 901 | }, 902 | "source": [ 903 | "tuple(listify(moms,2))" 904 | ], 905 | "execution_count": 0, 906 | "outputs": [ 907 | { 908 | "output_type": "execute_result", 909 | "data": { 910 | "text/plain": [ 911 | "(0.95, 0.85)" 912 | ] 913 | }, 914 | "metadata": { 915 | "tags": [] 916 | }, 917 | "execution_count": 5 918 | } 919 | ] 920 | }, 921 | { 922 | "cell_type": "markdown", 923 | "metadata": { 924 | "id": "yOmj0ITGZ-R2", 925 | "colab_type": "text" 926 | }, 927 | "source": [ 928 | "### Is Listy" 929 | ] 930 | }, 931 | { 932 | "cell_type": "code", 933 | "metadata": { 934 | "id": "Bsh5prL3Z_mm", 935 | "colab_type": "code", 936 | "outputId": "469739e0-5b81-4b8d-fe04-5da2ed4ef361", 937 | "colab": { 938 | "base_uri": "https://localhost:8080/", 939 | "height": 34 940 | } 941 | }, 942 | "source": [ 943 | "moms = listify(slice(0.95,0.85)) # if we pass in a slice as lr_max\n", 944 | "is_listy(moms)" 945 | ], 946 | "execution_count": 0, 947 | "outputs": [ 948 | { 949 | "output_type": "execute_result", 950 | "data": { 951 | "text/plain": [ 952 | "True" 953 | ] 954 | }, 955 | "metadata": { 956 | "tags": [] 957 | }, 958 | "execution_count": 11 959 | } 960 | ] 961 | }, 962 | { 963 | "cell_type": "code", 964 | "metadata": { 965 | "id": "HaJobGSRaEVf", 966 | "colab_type": "code", 967 | "outputId": "8aa9b1c3-5e7c-42c2-9175-066f8c25ae7d", 968 | "colab": { 969 | "base_uri": "https://localhost:8080/", 970 | "height": 34 971 | } 972 | }, 973 | "source": [ 974 | "if is_listy(moms): moms = np.array(moms)\n", 975 | "print(moms)" 976 | ], 977 | "execution_count": 0, 978 | "outputs": [ 979 | { 980 | "output_type": "stream", 981 | "text": [ 982 | "[slice(0.95, 0.85, None)]\n" 983 | ], 984 | "name": "stdout" 985 | } 986 | ] 987 | } 988 | ] 989 | } -------------------------------------------------------------------------------- /05b_State_of_the_Art.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "05b_State_of_the_Art.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "p6Lp4jrmYZci", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# State of the Art in NLP\n", 25 | "\n", 26 | "| | Tuesday 4-5:15pm | Friday 4-5:30pm |\n", 27 | "|:------:|:-------------------------------------------:|:--------------------------------------------------------------------------:|\n", 28 | "| **Week 1** | Introduction | Introduction |\n", 29 | "| **Week 2** | Custom computer vision tasks | State of the art in Computer Vision |\n", 30 | "| **Week 3** | Introduction to Tabular modeling and pandas | Pandas workshop and feature engineering |\n", 31 | "| **Week 4** | Tabular and Image Regression | Feature importance and advanced feature engineering |\n", 32 | "| **Week 5** | Natural Language Processing | **State of the art in NLP** |\n", 33 | "| **Week 6** | Segmentation and Kaggle | Audio |\n", 34 | "| **Week 7** | Computer vision from scratch | NLP from scratch |\n", 35 | "| **Week 8** | Callbacks | Optimizers |\n", 36 | "| **Week 9** | Generative Adversarial Networks | Research time / presentations |\n", 37 | "| **Week 10** | Putting models into production | Putting models into production |\n", 38 | "\n", 39 | "Today will be run just like it was for last week, we will use the sample to run it within class time, but you should run the bottom part to see teh full extent.\n", 40 | "\n", 41 | "Topics we will be looking at:\n", 42 | "\n", 43 | "* Backwards models\n", 44 | "* Combining models" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": { 50 | "id": "XmiU2_kpYpGI", 51 | "colab_type": "text" 52 | }, 53 | "source": [ 54 | "# What is a backwards model?\n", 55 | "\n", 56 | "It does exactly as you would think. It reads the langauge backwards. Let's show a databunch" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "metadata": { 62 | "id": "cIQ3xEJKYxAz", 63 | "colab_type": "code", 64 | "colab": {} 65 | }, 66 | "source": [ 67 | "from fastai.text import *" 68 | ], 69 | "execution_count": 0, 70 | "outputs": [] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "metadata": { 75 | "id": "fPRBU5hYYVQ9", 76 | "colab_type": "code", 77 | "colab": {} 78 | }, 79 | "source": [ 80 | "path = untar_data(URLs.IMDB_SAMPLE)" 81 | ], 82 | "execution_count": 0, 83 | "outputs": [] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "metadata": { 88 | "id": "rdexf1nJg8AQ", 89 | "colab_type": "code", 90 | "colab": {} 91 | }, 92 | "source": [ 93 | "df = pd.read_csv(path/'texts.csv')" 94 | ], 95 | "execution_count": 0, 96 | "outputs": [] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "metadata": { 101 | "id": "9RJRAVilg_Qi", 102 | "colab_type": "code", 103 | "colab": {} 104 | }, 105 | "source": [ 106 | "df_train = df[:800]\n", 107 | "df_valid = df[800:]" 108 | ], 109 | "execution_count": 0, 110 | "outputs": [] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "metadata": { 115 | "id": "Q8oE9EziY0w4", 116 | "colab_type": "code", 117 | "colab": {} 118 | }, 119 | "source": [ 120 | "data_lm_bwd = TextLMDataBunch.from_df(path, train_df = df_train, valid_df=df_valid, backwards=True)\n", 121 | "data_lm_fwd = TextLMDataBunch.from_df(path, train_df = df_train, valid_df=df_valid, backwards=False)" 122 | ], 123 | "execution_count": 0, 124 | "outputs": [] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "metadata": { 129 | "id": "txas-ajAZRMj", 130 | "colab_type": "code", 131 | "outputId": "f60ea1b6-8cb2-4636-ff05-bbe867f0da18", 132 | "colab": { 133 | "base_uri": "https://localhost:8080/", 134 | "height": 459 135 | } 136 | }, 137 | "source": [ 138 | "data_lm_fwd.show_batch()" 139 | ], 140 | "execution_count": 0, 141 | "outputs": [ 142 | { 143 | "output_type": "display_data", 144 | "data": { 145 | "text/html": [ 146 | "\n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | "
idxtext
0! ! ! xxmaj finally this was directed by the guy who did xxmaj big xxmaj xxunk ? xxmaj must be a replay of xxmaj jonestown - hollywood style . xxmaj xxunk ! xxbos xxmaj this is a extremely well - made film . xxmaj the acting , script and camera - work are all first - rate . xxmaj the music is good , too , though it is
1first loved him and then later hated him because he was xxunk . xxmaj he tries to explain to us the reasons he did what he did , but it 's really really so hard to xxunk . xxmaj such sad and unusual self destruction . xxmaj was it supposed to be funny ? xxmaj what was it all about really ? xxbos ( aka : xxup blood xxup castle
2sleeping was funny . xxmaj the xxunk bubble , and when xxmaj pumba leaves , the xxunk stop . xxmaj it 's all harmless fun , good for kids and some adults . i think this movie will last for a while because it is rather good for a straight to xxmaj video and xxup dvd movie . xxmaj while the movie does seem a little odd and kind of
3most people have said was rooting for the homeless people to make it , specially the guy , he gave me a few cheap laughs here and there . i think this film could have really been something special instead it became what every other horror nowadays are ! xxmaj just boring and well not worth the money . \\n \\n if you are looking for a cheap scare
4xxmaj the director is unable to resist showing the destruction of a major landmark ( xxmaj big xxmaj ben ) , but at least does n't dwell xxunk on the xxunk of xxmaj london . \\n \\n xxmaj the victory of the xxmaj martians is hardly a surprise , despite the destruction by xxunk of some of their machines . xxmaj the xxmaj narrator , traveling about to seek
" 176 | ], 177 | "text/plain": [ 178 | "" 179 | ] 180 | }, 181 | "metadata": { 182 | "tags": [] 183 | } 184 | } 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "metadata": { 190 | "id": "bH5UBOAdZW8Y", 191 | "colab_type": "code", 192 | "outputId": "df43cbf5-f2ed-4b04-ef6a-1d3c3dc3ebf1", 193 | "colab": { 194 | "base_uri": "https://localhost:8080/", 195 | "height": 459 196 | } 197 | }, 198 | "source": [ 199 | "data_lm_bwd.show_batch()" 200 | ], 201 | "execution_count": 0, 202 | "outputs": [ 203 | { 204 | "output_type": "display_data", 205 | "data": { 206 | "text/html": [ 207 | "\n", 208 | " \n", 209 | " \n", 210 | " \n", 211 | " \n", 212 | " \n", 213 | " \n", 214 | " \n", 215 | " \n", 216 | " \n", 217 | " \n", 218 | " \n", 219 | " \n", 220 | " \n", 221 | " \n", 222 | " \n", 223 | " \n", 224 | " \n", 225 | " \n", 226 | " \n", 227 | " \n", 228 | " \n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | "
idxtext
0convincing more make makeup without queens drag seen have i . completely fail they , now by guessed have will you as xxmaj . women white specific two as themselves disguise to required are brothers wayans xxmaj the however xxmaj . xxunk white to change a was script the by required was that all if enough bad be would it xxmaj . screen on put ever xxunk convincing least the
1as ( xxunk xxmaj robert xxmaj xxunk which in ) 1932 ( ' zombie xxmaj white xxmaj ' film earlier , better brothers xxunk xxmaj the from themes basic its of lot a borrows story the xxmaj . ) 1958 ( ' creole xxmaj king xxmaj ' and ) 1956 ( ' unknown xxmaj the xxmaj x ' , ) xxunk ( ' christmas xxmaj white xxmaj ' in roles
2late the in popular so became that formula horror teen stereotypical the takes , legend xxmaj urban xxmaj 's xxunk for responsible also was who , blanks xxmaj jamie xxmaj director xxmaj . horror / comedy a almost is valentine xxmaj . get to something is there but , this like film a get n't did i saying ridiculous sound might it xxmaj \\n \\n . \" a \"
3. end its toward moves it when it in quality redeeming some have does movie does enough surprisingly xxmaj xxbos . forever \" xxunk - xxunk \" of land the in lost being of consequences the suffer might she or fast , it find to needs she xxmaj . lost still is vampire xxmaj the with interview xxmaj in saw i that xxunk the , however character 's pegg xxmaj
4boy every , beautiful is melissa xxmaj since xxmaj . week one in up coming is xxunk her and old years fifteen 's she , town in girl new a is melissa xxmaj = plot xxmaj xxbos . director the sack xxmaj . better been have could xxmaj . pity xxmaj . day rainy a on entertaining - xxunk is it : general xxmaj \\n \\n . watch to
" 237 | ], 238 | "text/plain": [ 239 | "" 240 | ] 241 | }, 242 | "metadata": { 243 | "tags": [] 244 | } 245 | } 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": { 251 | "id": "PjJyNqCqZrou", 252 | "colab_type": "text" 253 | }, 254 | "source": [ 255 | "It's as simple as that! One model that goes forwards, one model that goes backwards. However this is now what Jeremy et al used to achieve state of the art results on the IMDB dataset. Now that's a lot of models to train, let's automate it" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "metadata": { 261 | "id": "L-lzWIKnaINk", 262 | "colab_type": "code", 263 | "colab": {} 264 | }, 265 | "source": [ 266 | "fwd = language_model_learner(data_lm_fwd, AWD_LSTM, drop_mult=0.3).to_fp16()\n", 267 | "bwd = language_model_learner(data_lm_bwd, AWD_LSTM, drop_mult=0.3).to_fp16()" 268 | ], 269 | "execution_count": 0, 270 | "outputs": [] 271 | }, 272 | { 273 | "cell_type": "code", 274 | "metadata": { 275 | "id": "t3Ff_QHwZpu2", 276 | "colab_type": "code", 277 | "colab": {} 278 | }, 279 | "source": [ 280 | "def train_models(fwd:Learner, bwd:Learner):\n", 281 | " models = [fwd, bwd]\n", 282 | " bs = fwd.data.batch_size\n", 283 | " names = ['forward', 'backward']\n", 284 | " x = 0\n", 285 | " for model in models:\n", 286 | " lr = 1e-2\n", 287 | " lr *= bs/48\n", 288 | " model.fit_one_cycle(1, lr, moms=(0.8, 0.7))\n", 289 | " model.unfreeze()\n", 290 | " model.fit_one_cycle(1, lr/10, moms=(0.8, 0.7))\n", 291 | " model.save(f'{names[x]}_fine_tuned_10')\n", 292 | " model.save_encoder(f'{names[x]}_fine_tuned_enc_10')\n", 293 | " x += 1" 294 | ], 295 | "execution_count": 0, 296 | "outputs": [] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": { 301 | "id": "Kj38VOwJbz4p", 302 | "colab_type": "text" 303 | }, 304 | "source": [ 305 | "We're going to overfit terribly here, but just remember it's a small dataset. Normally we also want to run for many more epochs" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "metadata": { 311 | "id": "4L0jQdl_bh6Q", 312 | "colab_type": "code", 313 | "outputId": "66f69f68-934c-4027-c283-5f0389e6989c", 314 | "colab": { 315 | "base_uri": "https://localhost:8080/", 316 | "height": 269 317 | } 318 | }, 319 | "source": [ 320 | "train_models(fwd, bwd)" 321 | ], 322 | "execution_count": 0, 323 | "outputs": [ 324 | { 325 | "output_type": "display_data", 326 | "data": { 327 | "text/html": [ 328 | "\n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | "
epochtrain_lossvalid_lossaccuracytime
04.2283863.8716150.29022300:04
" 348 | ], 349 | "text/plain": [ 350 | "" 351 | ] 352 | }, 353 | "metadata": { 354 | "tags": [] 355 | } 356 | }, 357 | { 358 | "output_type": "display_data", 359 | "data": { 360 | "text/html": [ 361 | "\n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | " \n", 375 | " \n", 376 | " \n", 377 | " \n", 378 | " \n", 379 | " \n", 380 | "
epochtrain_lossvalid_lossaccuracytime
03.7557963.8130860.29575900:05
" 381 | ], 382 | "text/plain": [ 383 | "" 384 | ] 385 | }, 386 | "metadata": { 387 | "tags": [] 388 | } 389 | }, 390 | { 391 | "output_type": "display_data", 392 | "data": { 393 | "text/html": [ 394 | "\n", 395 | " \n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | "
epochtrain_lossvalid_lossaccuracytime
04.2644833.9186740.31437500:04
" 414 | ], 415 | "text/plain": [ 416 | "" 417 | ] 418 | }, 419 | "metadata": { 420 | "tags": [] 421 | } 422 | }, 423 | { 424 | "output_type": "display_data", 425 | "data": { 426 | "text/html": [ 427 | "\n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | "
epochtrain_lossvalid_lossaccuracytime
03.7870223.8487520.32125000:05
" 447 | ], 448 | "text/plain": [ 449 | "" 450 | ] 451 | }, 452 | "metadata": { 453 | "tags": [] 454 | } 455 | } 456 | ] 457 | }, 458 | { 459 | "cell_type": "markdown", 460 | "metadata": { 461 | "id": "FJ8_sttVcJm2", 462 | "colab_type": "text" 463 | }, 464 | "source": [ 465 | "Great! Now we have our langauge model. Let's train the classifier now" 466 | ] 467 | }, 468 | { 469 | "cell_type": "code", 470 | "metadata": { 471 | "id": "95PFIeWFblXt", 472 | "colab_type": "code", 473 | "colab": {} 474 | }, 475 | "source": [ 476 | "data_cls_bwd = TextClasDataBunch.from_df(path, train_df = df_train, valid_df=df_valid, backwards=True, \n", 477 | " vocab=data_lm_bwd.vocab)\n", 478 | "data_cls_fwd = TextClasDataBunch.from_df(path, train_df = df_train, valid_df=df_valid, backwards=False,\n", 479 | " vocab=data_lm_fwd.vocab)" 480 | ], 481 | "execution_count": 0, 482 | "outputs": [] 483 | }, 484 | { 485 | "cell_type": "code", 486 | "metadata": { 487 | "id": "MFv1Ux4Ucd-f", 488 | "colab_type": "code", 489 | "outputId": "31dce0ec-0483-4e55-9eda-71130c0343aa", 490 | "colab": { 491 | "base_uri": "https://localhost:8080/", 492 | "height": 442 493 | } 494 | }, 495 | "source": [ 496 | "data_cls_bwd.show_batch()" 497 | ], 498 | "execution_count": 0, 499 | "outputs": [ 500 | { 501 | "output_type": "display_data", 502 | "data": { 503 | "text/html": [ 504 | "\n", 505 | " \n", 506 | " \n", 507 | " \n", 508 | " \n", 509 | " \n", 510 | " \n", 511 | " \n", 512 | " \n", 513 | " \n", 514 | " \n", 515 | " \n", 516 | " \n", 517 | " \n", 518 | " \n", 519 | " \n", 520 | " \n", 521 | " \n", 522 | " \n", 523 | " \n", 524 | " \n", 525 | " \n", 526 | " \n", 527 | " \n", 528 | " \n", 529 | " \n", 530 | " \n", 531 | " \n", 532 | " \n", 533 | "
texttarget
\\n \\n . grits of bowl a into hands my stick go to going 'm i , me excuse 'll you if so xxmaj ? him about film a like i can how ultimately so , him like n't do i way either xxmaj ' ? xxunk xxmaj - xxunk xxmaj xxunk my into get to want ju , bee - bay hey xxmaj ` , car his fromnegative
. could possibly very we that feeling it from away come we that is film this about thing best the ; film this in depicted romance the have to as lucky so be all should we xxmaj . role his in impeccable is stewart xxmaj and , charming , funny , sweet 's it xxmaj . instead christmas xxmaj this corner xxup the xxup around xxup shop xxup the xxuppositive
) 10 / xxunk the about xxunk a just even know who all of hearts the break to bound is it xxmaj . costs all at avoided be to , film soderbergh xxmaj a for , shockingly -- misery irritating , confusing just is xxunk cinema interminable this of rest the xxmaj \\n \\n . subject 's film the of execution and capture -- tragic yet -- dramatic thenegative
. end xxup the xxup . other each at look they xxmaj . window the down xxunk , up drives movie the in earlier her pay n't did who contractor the xxmaj . business does normally she where corner the at xxunk and , street the down walking prostitute the see we and , off goes gun the , eventually xxmaj . aim to where her telling , mouth hispositive
10 / 8 . games 3d xxup of fans all to this recommended i . appeared first he after years fifteen nearly ) ? especially even maybe xxunk , recognition the deserves he ... shoes 's xxunk xxup into step and bunker the enter to door the open , xxunk xxmaj the up load so xxmaj . one this to existence their owe , genre the of rest the aspositive
" 534 | ], 535 | "text/plain": [ 536 | "" 537 | ] 538 | }, 539 | "metadata": { 540 | "tags": [] 541 | } 542 | } 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "metadata": { 548 | "id": "zWVSz_7ecgrM", 549 | "colab_type": "code", 550 | "outputId": "fdf6b1f1-22e6-413d-b3de-ecdad0f34b38", 551 | "colab": { 552 | "base_uri": "https://localhost:8080/", 553 | "height": 459 554 | } 555 | }, 556 | "source": [ 557 | "data_cls_fwd.show_batch()" 558 | ], 559 | "execution_count": 0, 560 | "outputs": [ 561 | { 562 | "output_type": "display_data", 563 | "data": { 564 | "text/html": [ 565 | "\n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | "
texttarget
xxbos xxmaj raising xxmaj victor xxmaj vargas : a xxmaj review \\n \\n xxmaj you know , xxmaj raising xxmaj victor xxmaj vargas is like sticking your hands into a big , steaming bowl of xxunk . xxmaj it 's warm and gooey , but you 're not sure if it feels right . xxmaj try as i might , no matter how warm and gooey xxmaj raising xxmajnegative
xxbos xxup the xxup shop xxup around xxup the xxup corner is one of the sweetest and most feel - good romantic comedies ever made . xxmaj there 's just no getting around that , and it 's hard to actually put one 's feeling for this film into words . xxmaj it 's not one of those films that tries too hard , nor does it come up withpositive
xxbos xxmaj now that xxmaj che(2008 ) has finished its relatively short xxmaj australian cinema run ( extremely limited xxunk screen in xxmaj sydney , after xxunk ) , i can xxunk join both xxunk of \" xxmaj at xxmaj the xxmaj movies \" in taking xxmaj steven xxmaj soderbergh to task . \\n \\n xxmaj it 's usually satisfying to watch a film director change his style /negative
xxbos xxmaj this film sat on my xxmaj xxunk for weeks before i watched it . i dreaded a self - indulgent xxunk flick about relationships gone bad . i was wrong ; this was an xxunk xxunk into the screwed - up xxunk of xxmaj new xxmaj yorkers . \\n \\n xxmaj the format is the same as xxmaj max xxmaj xxunk ' \" xxmaj la xxmaj rondepositive
xxbos xxmaj many neglect that this is n't just a classic due to the fact that it 's the first xxup 3d game , or even the first xxunk - up . xxmaj it 's also one of the first xxunk games , one of the xxunk definitely the first ) truly claustrophobic games , and just a pretty well - xxunk gaming experience in general . xxmaj with graphicspositive
" 595 | ], 596 | "text/plain": [ 597 | "" 598 | ] 599 | }, 600 | "metadata": { 601 | "tags": [] 602 | } 603 | } 604 | ] 605 | }, 606 | { 607 | "cell_type": "code", 608 | "metadata": { 609 | "id": "N-bv2QE6crBg", 610 | "colab_type": "code", 611 | "colab": {} 612 | }, 613 | "source": [ 614 | "learn_cls_fwd = text_classifier_learner(data_cls_fwd, AWD_LSTM, drop_mult=0.5).to_fp16()\n", 615 | "learn_cls_bwd = text_classifier_learner(data_cls_bwd, AWD_LSTM, drop_mult=0.5).to_fp16()" 616 | ], 617 | "execution_count": 0, 618 | "outputs": [] 619 | }, 620 | { 621 | "cell_type": "code", 622 | "metadata": { 623 | "id": "vMau3Pbrc431", 624 | "colab_type": "code", 625 | "colab": {} 626 | }, 627 | "source": [ 628 | "learn_cls_fwd.load_encoder('forward_fine_tuned_enc_10')\n", 629 | "learn_cls_bwd.load_encoder('backward_fine_tuned_enc_10')\n", 630 | "learn_cls_fwd.freeze()\n", 631 | "learn_cls_bwd.freeze()" 632 | ], 633 | "execution_count": 0, 634 | "outputs": [] 635 | }, 636 | { 637 | "cell_type": "markdown", 638 | "metadata": { 639 | "id": "E9W7qULRcl-m", 640 | "colab_type": "text" 641 | }, 642 | "source": [ 643 | "Now here we are going to return our models, so we can ensemble them" 644 | ] 645 | }, 646 | { 647 | "cell_type": "code", 648 | "metadata": { 649 | "id": "HhBD21packB9", 650 | "colab_type": "code", 651 | "colab": {} 652 | }, 653 | "source": [ 654 | "def train_models(fwd:Learner, bwd:Learner):\n", 655 | " models = [fwd, bwd]\n", 656 | " bs = fwd.data.batch_size\n", 657 | " names = ['forward', 'backward']\n", 658 | " x = 0\n", 659 | " for model in models:\n", 660 | " lr = 1e-2\n", 661 | " lr *= bs/48\n", 662 | " model.fit_one_cycle(1, lr, moms=(0.8, 0.7))\n", 663 | " \n", 664 | " model.freeze_to(-2)\n", 665 | " model.fit_one_cycle(1, slice(lr/(2.6**4),lr), moms=(0.8,0.7))\n", 666 | " \n", 667 | " model.freeze_to(-3)\n", 668 | " model.fit_one_cycle(1, slice(lr/2/(2.6**4),lr/2), moms=(0.8,0.7))\n", 669 | " \n", 670 | " model.unfreeze()\n", 671 | " model.fit_one_cycle(2, slice(lr/10/(2.6**4),lr/10), moms=(0.8,0.7))\n", 672 | " return fwd, bwd" 673 | ], 674 | "execution_count": 0, 675 | "outputs": [] 676 | }, 677 | { 678 | "cell_type": "code", 679 | "metadata": { 680 | "id": "JQbi-fLYfzT3", 681 | "colab_type": "code", 682 | "outputId": "b3bf4040-1519-4970-8748-479e2cb79c6a", 683 | "colab": { 684 | "base_uri": "https://localhost:8080/", 685 | "height": 583 686 | } 687 | }, 688 | "source": [ 689 | "learn_cls_fwd, learn_cls_bwd = train_models(learn_cls_fwd, learn_cls_bwd)" 690 | ], 691 | "execution_count": 0, 692 | "outputs": [ 693 | { 694 | "output_type": "display_data", 695 | "data": { 696 | "text/html": [ 697 | "\n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | " \n", 716 | "
epochtrain_lossvalid_lossaccuracytime
00.5837560.6492100.75500000:03
" 717 | ], 718 | "text/plain": [ 719 | "" 720 | ] 721 | }, 722 | "metadata": { 723 | "tags": [] 724 | } 725 | }, 726 | { 727 | "output_type": "display_data", 728 | "data": { 729 | "text/html": [ 730 | "\n", 731 | " \n", 732 | " \n", 733 | " \n", 734 | " \n", 735 | " \n", 736 | " \n", 737 | " \n", 738 | " \n", 739 | " \n", 740 | " \n", 741 | " \n", 742 | " \n", 743 | " \n", 744 | " \n", 745 | " \n", 746 | " \n", 747 | " \n", 748 | " \n", 749 | "
epochtrain_lossvalid_lossaccuracytime
00.4853010.5974410.63500000:03
" 750 | ], 751 | "text/plain": [ 752 | "" 753 | ] 754 | }, 755 | "metadata": { 756 | "tags": [] 757 | } 758 | }, 759 | { 760 | "output_type": "display_data", 761 | "data": { 762 | "text/html": [ 763 | "\n", 764 | " \n", 765 | " \n", 766 | " \n", 767 | " \n", 768 | " \n", 769 | " \n", 770 | " \n", 771 | " \n", 772 | " \n", 773 | " \n", 774 | " \n", 775 | " \n", 776 | " \n", 777 | " \n", 778 | " \n", 779 | " \n", 780 | " \n", 781 | " \n", 782 | "
epochtrain_lossvalid_lossaccuracytime
00.3931360.4654170.81500000:04
" 783 | ], 784 | "text/plain": [ 785 | "" 786 | ] 787 | }, 788 | "metadata": { 789 | "tags": [] 790 | } 791 | }, 792 | { 793 | "output_type": "display_data", 794 | "data": { 795 | "text/html": [ 796 | "\n", 797 | " \n", 798 | " \n", 799 | " \n", 800 | " \n", 801 | " \n", 802 | " \n", 803 | " \n", 804 | " \n", 805 | " \n", 806 | " \n", 807 | " \n", 808 | " \n", 809 | " \n", 810 | " \n", 811 | " \n", 812 | " \n", 813 | " \n", 814 | " \n", 815 | " \n", 816 | " \n", 817 | " \n", 818 | " \n", 819 | " \n", 820 | " \n", 821 | " \n", 822 | "
epochtrain_lossvalid_lossaccuracytime
00.2787820.4143690.81000000:05
10.2477070.4093600.81000000:05
" 823 | ], 824 | "text/plain": [ 825 | "" 826 | ] 827 | }, 828 | "metadata": { 829 | "tags": [] 830 | } 831 | }, 832 | { 833 | "output_type": "display_data", 834 | "data": { 835 | "text/html": [ 836 | "\n", 837 | " \n", 838 | " \n", 839 | " \n", 840 | " \n", 841 | " \n", 842 | " \n", 843 | " \n", 844 | " \n", 845 | " \n", 846 | " \n", 847 | " \n", 848 | " \n", 849 | " \n", 850 | " \n", 851 | " \n", 852 | " \n", 853 | " \n", 854 | " \n", 855 | "
epochtrain_lossvalid_lossaccuracytime
00.6092810.6528330.59500000:03
" 856 | ], 857 | "text/plain": [ 858 | "" 859 | ] 860 | }, 861 | "metadata": { 862 | "tags": [] 863 | } 864 | }, 865 | { 866 | "output_type": "display_data", 867 | "data": { 868 | "text/html": [ 869 | "\n", 870 | " \n", 871 | " \n", 872 | " \n", 873 | " \n", 874 | " \n", 875 | " \n", 876 | " \n", 877 | " \n", 878 | " \n", 879 | " \n", 880 | " \n", 881 | " \n", 882 | " \n", 883 | " \n", 884 | " \n", 885 | " \n", 886 | " \n", 887 | " \n", 888 | "
epochtrain_lossvalid_lossaccuracytime
00.5767360.5830280.69500000:03
" 889 | ], 890 | "text/plain": [ 891 | "" 892 | ] 893 | }, 894 | "metadata": { 895 | "tags": [] 896 | } 897 | }, 898 | { 899 | "output_type": "display_data", 900 | "data": { 901 | "text/html": [ 902 | "\n", 903 | " \n", 904 | " \n", 905 | " \n", 906 | " \n", 907 | " \n", 908 | " \n", 909 | " \n", 910 | " \n", 911 | " \n", 912 | " \n", 913 | " \n", 914 | " \n", 915 | " \n", 916 | " \n", 917 | " \n", 918 | " \n", 919 | " \n", 920 | " \n", 921 | "
epochtrain_lossvalid_lossaccuracytime
00.4582960.5209960.71500000:04
" 922 | ], 923 | "text/plain": [ 924 | "" 925 | ] 926 | }, 927 | "metadata": { 928 | "tags": [] 929 | } 930 | }, 931 | { 932 | "output_type": "display_data", 933 | "data": { 934 | "text/html": [ 935 | "\n", 936 | " \n", 937 | " \n", 938 | " \n", 939 | " \n", 940 | " \n", 941 | " \n", 942 | " \n", 943 | " \n", 944 | " \n", 945 | " \n", 946 | " \n", 947 | " \n", 948 | " \n", 949 | " \n", 950 | " \n", 951 | " \n", 952 | " \n", 953 | " \n", 954 | " \n", 955 | " \n", 956 | " \n", 957 | " \n", 958 | " \n", 959 | " \n", 960 | " \n", 961 | "
epochtrain_lossvalid_lossaccuracytime
00.3867450.5315370.72500000:06
10.3656910.4969640.76000000:05
" 962 | ], 963 | "text/plain": [ 964 | "" 965 | ] 966 | }, 967 | "metadata": { 968 | "tags": [] 969 | } 970 | } 971 | ] 972 | }, 973 | { 974 | "cell_type": "markdown", 975 | "metadata": { 976 | "id": "RQ_t1W1ogGDv", 977 | "colab_type": "text" 978 | }, 979 | "source": [ 980 | "Now let's ensemble!" 981 | ] 982 | }, 983 | { 984 | "cell_type": "markdown", 985 | "metadata": { 986 | "id": "GFA_iCtygMNW", 987 | "colab_type": "text" 988 | }, 989 | "source": [ 990 | "# Ensemble" 991 | ] 992 | }, 993 | { 994 | "cell_type": "code", 995 | "metadata": { 996 | "id": "X8u1M_Bhf5mv", 997 | "colab_type": "code", 998 | "colab": {} 999 | }, 1000 | "source": [ 1001 | "preds_a, targs_a = learn_cls_fwd.get_preds(ordered=True)\n", 1002 | "preds_b, targs_b = learn_cls_bwd.get_preds(ordered=True)" 1003 | ], 1004 | "execution_count": 0, 1005 | "outputs": [] 1006 | }, 1007 | { 1008 | "cell_type": "code", 1009 | "metadata": { 1010 | "id": "QXK1RDE-gdap", 1011 | "colab_type": "code", 1012 | "outputId": "6c717308-cdb4-49f0-8893-c01eda6ec2f9", 1013 | "colab": { 1014 | "base_uri": "https://localhost:8080/", 1015 | "height": 34 1016 | } 1017 | }, 1018 | "source": [ 1019 | "accuracy(preds_a, targs_b), accuracy(preds_b, targs_b)" 1020 | ], 1021 | "execution_count": 0, 1022 | "outputs": [ 1023 | { 1024 | "output_type": "execute_result", 1025 | "data": { 1026 | "text/plain": [ 1027 | "(tensor(0.8100), tensor(0.7600))" 1028 | ] 1029 | }, 1030 | "metadata": { 1031 | "tags": [] 1032 | }, 1033 | "execution_count": 75 1034 | } 1035 | ] 1036 | }, 1037 | { 1038 | "cell_type": "code", 1039 | "metadata": { 1040 | "id": "TIzwENomghBF", 1041 | "colab_type": "code", 1042 | "colab": {} 1043 | }, 1044 | "source": [ 1045 | "preds_avg = (preds_a + preds_b)/2" 1046 | ], 1047 | "execution_count": 0, 1048 | "outputs": [] 1049 | }, 1050 | { 1051 | "cell_type": "code", 1052 | "metadata": { 1053 | "id": "FHajaPHXgul-", 1054 | "colab_type": "code", 1055 | "outputId": "338069bc-4ad2-4b13-b1c9-1be03d95c2ab", 1056 | "colab": { 1057 | "base_uri": "https://localhost:8080/", 1058 | "height": 34 1059 | } 1060 | }, 1061 | "source": [ 1062 | "accuracy(preds_avg, targs_b)" 1063 | ], 1064 | "execution_count": 0, 1065 | "outputs": [ 1066 | { 1067 | "output_type": "execute_result", 1068 | "data": { 1069 | "text/plain": [ 1070 | "tensor(0.8150)" 1071 | ] 1072 | }, 1073 | "metadata": { 1074 | "tags": [] 1075 | }, 1076 | "execution_count": 77 1077 | } 1078 | ] 1079 | }, 1080 | { 1081 | "cell_type": "markdown", 1082 | "metadata": { 1083 | "id": "cyp1dMzeh2Cb", 1084 | "colab_type": "text" 1085 | }, 1086 | "source": [ 1087 | "See? We got some improvement!" 1088 | ] 1089 | }, 1090 | { 1091 | "cell_type": "code", 1092 | "metadata": { 1093 | "id": "8PwXoyYcgwb7", 1094 | "colab_type": "code", 1095 | "colab": {} 1096 | }, 1097 | "source": [ 1098 | "" 1099 | ], 1100 | "execution_count": 0, 1101 | "outputs": [] 1102 | } 1103 | ] 1104 | } --------------------------------------------------------------------------------