├── .gitignore ├── 2021 Edition ├── 1. Intro to ML pt. 1 │ ├── Intro_to_ML1.ipynb │ └── Intro_to_ML1.pdf ├── 2. Intro to ML pt. 2 │ ├── Introduction_ML_II.pdf │ ├── Nature-COVID.pdf │ ├── Wprowadzenie_ML_2_regression.R │ ├── Wprowadzenie_ML_2_tree.ipynb │ └── extract.csv ├── 3. Intro to DL │ ├── img │ │ ├── colab.png │ │ ├── graph.jpg │ │ └── logo.png │ ├── introduction_to_deep_learning.ipynb │ └── lecture.pdf ├── 4. CNN │ ├── 2020 │ │ ├── Introduction to CNNs.pdf │ │ └── hands-on │ │ │ ├── CNNs_Training_Session_CIFAR_10_hands_on.ipynb │ │ │ └── completed_CNNs_Training_Session_CIFAR_10_hands_on_1.ipynb │ ├── img │ │ ├── colab.png │ │ ├── conv.gif │ │ ├── graph.jpg │ │ ├── logo.png │ │ ├── nn.jpg │ │ └── pool.gif │ └── introduction_to_cnns.ipynb ├── 5. RNN │ └── lecture.pdf ├── 6. ML Tools │ └── lecture.pdf └── 7. ML Deployment │ └── lecture.pdf ├── 2022 Edition ├── 1. Kick-off meeting │ └── slides.pdf ├── 2. Basics of Machine Learning │ ├── .gitkeep │ └── 2. Introduction to ML.pdf ├── 3. Classical Algorithms │ ├── 3. Algorytmy Klasyczne.pdf │ └── Algorytmy_Klasyczne.ipynb ├── 4. Introduction to Deep Learning │ ├── .gitkeep │ └── 4. Introduction to Deep Learning.pdf ├── 5. Computer Vision │ ├── .gitkeep │ ├── 5. Computer vision.pdf │ └── ComputerVision.ipynb ├── 6. Recurrent Neural Networks │ ├── .gitkeep │ ├── 6.Sequential Data.pdf │ └── 6.Sequential Data.pptx ├── 7. Reinforcement Learning │ ├── .gitkeep │ ├── 7. Reinforcement Learning.pdf │ └── 7. Reinforcement Learning.pptx └── 8. Technical Aspects in ML │ ├── .gitkeep │ └── slides.pdf ├── 2023_Edition ├── Deep_Learning_course │ ├── Classical Algorithms │ │ ├── Algorytmy Klasyczne.pdf │ │ ├── Algorytmy_Klasyczne.ipynb │ │ ├── Regression.ipynb │ │ └── datasets │ │ │ ├── car_data.csv │ │ │ ├── winequality-red.csv │ │ │ └── winequality-white.csv │ ├── Deep_Learning_in_Computer_Vision │ │ ├── 4. Deep Learning in Computer Vision-1.pdf │ │ └── ComputerVision.ipynb │ ├── Implementation_of_NN │ │ ├── CNN_notebook.ipynb │ │ ├── Implementation.ipynb │ │ ├── mlp.png │ │ └── relu.png │ └── Intro_to_Deep_Learning │ │ ├── 3. Introduction to Deep Learning.pdf │ │ └── main.ipynb └── Workshops │ ├── 1 - Intro to Neural Networks │ ├── mlp.png │ ├── notatnik.ipynb │ └── relu.png │ └── 2 - Hyperparameters Tuning and CNN │ ├── CNN_notebook.ipynb │ ├── Choosing Parameters.ipynb │ ├── Tensorflow Image Classification.ipynb │ ├── assets │ ├── nnvis.png │ └── tf_playground.png │ └── utils │ ├── __init__.py │ ├── parameters.py │ └── wplot.py └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.bundle.* 2 | lib/ 3 | node_modules/ 4 | *.egg-info/ 5 | .ipynb_checkpoints 6 | *.tsbuildinfo 7 | 8 | # Created by https://www.gitignore.io/api/python 9 | # Edit at https://www.gitignore.io/?templates=python 10 | 11 | ### Python ### 12 | # Byte-compiled / optimized / DLL files 13 | __pycache__/ 14 | *.py[cod] 15 | *$py.class 16 | 17 | # C extensions 18 | *.so 19 | 20 | # Distribution / packaging 21 | .Python 22 | build/ 23 | develop-eggs/ 24 | dist/ 25 | downloads/ 26 | eggs/ 27 | .eggs/ 28 | lib/ 29 | lib64/ 30 | parts/ 31 | sdist/ 32 | var/ 33 | wheels/ 34 | pip-wheel-metadata/ 35 | share/python-wheels/ 36 | .installed.cfg 37 | *.egg 38 | MANIFEST 39 | 40 | # PyInstaller 41 | # Usually these files are written by a python script from a template 42 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 43 | *.manifest 44 | *.spec 45 | 46 | # Installer logs 47 | pip-log.txt 48 | pip-delete-this-directory.txt 49 | 50 | # Unit test / coverage reports 51 | htmlcov/ 52 | .tox/ 53 | .nox/ 54 | .coverage 55 | .coverage.* 56 | .cache 57 | nosetests.xml 58 | coverage.xml 59 | *.cover 60 | .hypothesis/ 61 | .pytest_cache/ 62 | 63 | # Translations 64 | *.mo 65 | *.pot 66 | 67 | # Scrapy stuff: 68 | .scrapy 69 | 70 | # Sphinx documentation 71 | docs/_build/ 72 | 73 | # PyBuilder 74 | target/ 75 | 76 | # pyenv 77 | .python-version 78 | 79 | # celery beat schedule file 80 | celerybeat-schedule 81 | 82 | # SageMath parsed files 83 | *.sage.py 84 | 85 | # Spyder project settings 86 | .spyderproject 87 | .spyproject 88 | 89 | # Rope project settings 90 | .ropeproject 91 | 92 | # Mr Developer 93 | .mr.developer.cfg 94 | .project 95 | .pydevproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | .dmypy.json 103 | dmypy.json 104 | 105 | # Pyre type checker 106 | .pyre/ 107 | 108 | # OS X stuff 109 | *.DS_Store 110 | 111 | # End of https://www.gitignore.io/api/python 112 | 113 | _temp_extension 114 | junit.xml 115 | [uU]ntitled* 116 | notebook/static/* 117 | !notebook/static/favicons 118 | notebook/labextension 119 | notebook/schemas 120 | docs/source/changelog.md 121 | docs/source/contributing.md 122 | 123 | # playwright 124 | ui-tests/test-results 125 | ui-tests/playwright-report 126 | 127 | # VSCode 128 | .vscode 129 | 130 | # RTC 131 | .jupyter_ystore.db 132 | 133 | # yarn >=2.x local files 134 | .yarn/* 135 | .pnp.* 136 | ui-tests/.yarn/* 137 | ui-tests/.pnp.* -------------------------------------------------------------------------------- /2021 Edition/1. Intro to ML pt. 1/Intro_to_ML1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/1. Intro to ML pt. 1/Intro_to_ML1.pdf -------------------------------------------------------------------------------- /2021 Edition/2. Intro to ML pt. 2/Introduction_ML_II.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/2. Intro to ML pt. 2/Introduction_ML_II.pdf -------------------------------------------------------------------------------- /2021 Edition/2. Intro to ML pt. 2/Nature-COVID.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/2. Intro to ML pt. 2/Nature-COVID.pdf -------------------------------------------------------------------------------- /2021 Edition/2. Intro to ML pt. 2/Wprowadzenie_ML_2_regression.R: -------------------------------------------------------------------------------- 1 | #install.packages("ggplot2") 2 | #install.packages("plyr") 3 | #install.packages("Hmisc") 4 | library(ggplot2) 5 | library(plyr) 6 | library(Hmisc) 7 | 8 | 9 | extract<-read.csv("C:/Users/dawid/OneDrive/Pulpit/Dawid/Gradient/Github/gradient-live-session/lab2/extract.csv") 10 | 11 | to_corr<-na.omit(data.frame(extract$class,extract$age,extract$education,extract$income,extract$height)) 12 | rcorr(as.matrix(to_corr)) 13 | count(extract,"sex") 14 | count(extract,"education") 15 | extract$woje<-as.factor(extract$woje) 16 | extract$class<-as.factor(extract$class) 17 | extract$education<-as.factor(extract$education) 18 | extract$class = revalue(extract$class, c('1'= ' city more than 500 thou.','2'=' city 200-500 thou.','3'=' city 100-200 thou.','4'=' city 20-100 thou.','5'=' city less than 20 thou.','6'=' village')) 19 | extract$education = revalue(extract$education, c('70'=" without",'60'=' primary','50' = ' basic vocational', '51'=' middle school','40'=' secondary','30'=' vocational','20'=' postsecondary','12'=' bachelor','11'=' master','10'=' doctor')) 20 | ######################## 21 | extract$sex<-(extract$sex==1)*1 22 | extract$weight_m<-extract$sex*extract$weight 23 | extract$weight_f<-extract$weight-extract$weight_m 24 | extract$height_m<-extract$sex*extract$height 25 | extract$height_f<-extract$height-extract$height_m 26 | ############################# 27 | 28 | extract$height_ms<-(extract$height-mean(extract$height[extract$sex==1]))*(extract$sex) 29 | extract$weight_ms<-(extract$weight-mean(extract$weight[extract$sex==1]))*(extract$sex) 30 | extract$height_fs<-(extract$height-mean(extract$height[extract$sex==0]))*(extract$sex==0) 31 | extract$weight_fs<-(extract$weight-mean(extract$weight[extract$sex==0]))*(extract$sex==0) 32 | 33 | 34 | ############ model for age #################### 35 | 36 | tmp<-summary( 37 | lm(income~sex+as.factor(age)+class+height_ms+height_fs + education, data=extract, weights=waga) 38 | ) 39 | lata <- c(17:96,98,99,100,103) 40 | lata_income<-data.frame(lata,tmp$coefficients[3:86,1]) 41 | ggplot(lata_income, aes(x=lata,y=tmp$coefficients[3:86,1])) + 42 | geom_point() + geom_smooth()+ labs(x = "Age", y = "Beta coefficient") 43 | 44 | ###### age transformation ###### 45 | extract$ageabove16upto40<-(extract$age<=40)*(extract$age-16) 46 | extract$ageabove16upto40cap<-(extract$age<=40)*(extract$age-16)+(extract$age>40)*(40-16) 47 | extract$age40flag<-(extract$age>40)&(extract$age<=60) 48 | extract$ageafter40<-extract$age40flag*(extract$age-40) 49 | extract$age60plusflag<-(extract$age>60) 50 | extract$ageafter60<-(extract$age60plusflag)*(extract$age-60) 51 | 52 | ########### creating model ########## 53 | #Step 1 54 | summary( 55 | lm(income~sex, data=extract, weights=waga) 56 | ) 57 | #Step 2 58 | summary( 59 | lm(income~sex+ageabove16upto40+age40flag+ageafter40+age60plusflag+ 60 | +height_ms+height_fs, data=extract, weights=waga) 61 | ) 62 | #Step 3 63 | summary( 64 | lm(income~sex+ageabove16upto40+age40flag+ageafter40+age60plusflag+height_ms+ height_fs+relevel(education, ref = " without") 65 | + relevel(class, ref = " village"), data=extract, weights=waga) 66 | ) 67 | 68 | -------------------------------------------------------------------------------- /2021 Edition/3. Intro to DL/img/colab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/3. Intro to DL/img/colab.png -------------------------------------------------------------------------------- /2021 Edition/3. Intro to DL/img/graph.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/3. Intro to DL/img/graph.jpg -------------------------------------------------------------------------------- /2021 Edition/3. Intro to DL/img/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/3. Intro to DL/img/logo.png -------------------------------------------------------------------------------- /2021 Edition/3. Intro to DL/introduction_to_deep_learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "lx6_sK4Ihs5y", 7 | "pycharm": { 8 | "name": "#%% md\n" 9 | } 10 | }, 11 | "source": [ 12 | "
\n", 13 | "\n", 14 | "

Introduction to Deep Learning

\n", 15 | "

Gradient PG, 2020

\n", 16 | "

Bazyli Polednia

\n", 17 | "\n", 18 | "---\n", 19 | "\n", 20 | "\n", 21 | "Run in Google Colab" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": { 27 | "id": "rnqZcoDTkTzh" 28 | }, 29 | "source": [ 30 | "# MNIST Digit Classifier\n", 31 | "\n", 32 | "In this notebook you will get to know the very basics of Tensorflow library and learn how to use it to build a basic network classifying handwritten digits. It is a well-known task, sometimes considered as a \"Hello world\" problem in AI field. We will mostly use Keras - it is a high-level API coming as submodule of Tensorflow. I highly encourage you to explore more stuff from those libraries - we will deal only with a small part of what they are capable of.\n", 33 | "\n", 34 | "Apart from text sections below, you will also find pre-prepared snippets of code, some of them with \\#TODO sections - fill them!" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": { 40 | "id": "XWbYezRWoWCb" 41 | }, 42 | "source": [ 43 | "# 1. Prepare the environment\n", 44 | "\n", 45 | "Before you start executing the code, change runtime time of this notebook. Go to \"Runtime\" → \"Change runtime type\" and select \"TPU\". This will enable you to run the notebook faster using the hardware accelerator. If you want to run the whole notebook from the beginning, press Ctrl+F9. If you only want to rerun a certain cell, press Ctrl+Enter while your cursor is in the cell.\n", 46 | "\n", 47 | "Afterwards, let's install necessary libraries which we'll be using." 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": { 54 | "colab": { 55 | "base_uri": "https://localhost:8080/" 56 | }, 57 | "id": "kiUUhDClhlKq", 58 | "outputId": "86c64710-d26e-460b-9deb-4f0c8734c455", 59 | "vscode": { 60 | "languageId": "python" 61 | } 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "%tensorflow_version 2.x\n", 66 | "import tensorflow as tf\n", 67 | "import matplotlib.pyplot as plt\n", 68 | "import numpy as np\n", 69 | "from tqdm import tqdm\n", 70 | "\n", 71 | "!pip install mitdeeplearning\n", 72 | "import mitdeeplearning as mdl" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": { 78 | "id": "fLIgBBVnqRob" 79 | }, 80 | "source": [ 81 | "#2. Explore the dataset\n", 82 | "\n", 83 | "Our task is to classify handwritten digits from MNIST dataset provided in Keras library. We will import it dividing the images and labels into train set and test set. Train set is the bunch of data we will be using to train the network on, meanwhile test set will be used to evaluate how well our network performs. It's important to remember that **you can't use test set to train the network** - it should only be used to evaluate the permorfance. " 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": { 90 | "id": "7XStzSPyrdmn", 91 | "vscode": { 92 | "languageId": "python" 93 | } 94 | }, 95 | "outputs": [], 96 | "source": [ 97 | "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": { 103 | "id": "M__yke0Vs-LH" 104 | }, 105 | "source": [ 106 | "Remember - the first thing you should always do before jumping straight to designing neural networks is explore the data you are using. Let's start with seeing in what format it is stored and how many images there are." 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": { 113 | "colab": { 114 | "base_uri": "https://localhost:8080/" 115 | }, 116 | "id": "RZ2f81k_tF2W", 117 | "outputId": "a898bbd1-5fae-42a2-9cb8-17c3d14ea01f", 118 | "vscode": { 119 | "languageId": "python" 120 | } 121 | }, 122 | "outputs": [], 123 | "source": [ 124 | "print(f\"Train images shape: {train_images.shape}\")\n", 125 | "print(f\"Test images shape: {test_images.shape}\")\n", 126 | "print(f\"Minimal value: {np.min(train_images)}\")\n", 127 | "print(f\"Maximal value: {np.max(train_images)}\")\n", 128 | "print(\"-\"*20)\n", 129 | "print(f\"Train labels: {train_labels}\")\n", 130 | "print(f\"Test labels: {test_labels}\")\n", 131 | "print(f\"Train labels length: {len(train_labels)}\")\n", 132 | "print(f\"Test labels length: {len(test_labels)}\")" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": { 139 | "colab": { 140 | "base_uri": "https://localhost:8080/", 141 | "height": 265 142 | }, 143 | "id": "6katIjV7tSqM", 144 | "outputId": "8d164f6d-bfa7-443f-ff6b-c5bcd3b264a3", 145 | "vscode": { 146 | "languageId": "python" 147 | } 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "plt.figure()\n", 152 | "plt.imshow(train_images[0])\n", 153 | "plt.colorbar()\n", 154 | "plt.grid(False)\n", 155 | "plt.show()" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": { 161 | "id": "uc6gl8BhuMil" 162 | }, 163 | "source": [ 164 | "As you can see, each image is represented as 28x28 pixel map, where each pixel takes value from 0.0 to 255.0. Label is simply the digit presented on the corresponding image. \n", 165 | "\n", 166 | "In order to easily pass the image to the network, let's scale the values in each image to range from 0.0 to 1.0 and store them as *float32* type." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": { 173 | "id": "oCw1pXCdwREp", 174 | "vscode": { 175 | "languageId": "python" 176 | } 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "train_images = train_images.astype('float32') / 255.0\n", 181 | "test_images = test_images.astype('float32') / 255.0" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": { 188 | "colab": { 189 | "base_uri": "https://localhost:8080/", 190 | "height": 589 191 | }, 192 | "id": "qBEbQwWcwkvH", 193 | "outputId": "5276a5d7-3f6f-4b29-8d84-2ad900c278f4", 194 | "vscode": { 195 | "languageId": "python" 196 | } 197 | }, 198 | "outputs": [], 199 | "source": [ 200 | "plt.figure(figsize=(10,10))\n", 201 | "for i in range(25):\n", 202 | " plt.subplot(5, 5, i+1)\n", 203 | " plt.xticks([])\n", 204 | " plt.yticks([])\n", 205 | " plt.grid(False)\n", 206 | " plt.imshow(train_images[i], cmap=plt.cm.binary)\n", 207 | " plt.xlabel(train_labels[i])\n", 208 | "plt.show()" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": { 214 | "id": "o5VvIvadydz7" 215 | }, 216 | "source": [ 217 | "#3. Create the network\n", 218 | "\n", 219 | "After we've done some very basic preprocessing and data exploration, let's get to the coolest thing - creating the network! There is no universal rule of thumb when it comes to which network architecture is best for a given problem. Picking the right one is more an art than a science and you can only get needed intuition by solving tens, hundreds and thousands of deep learning problems.\n", 220 | "\n", 221 | "This time we'll go with a very basic network design:\n", 222 | "\n", 223 | "" 224 | ] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "metadata": { 229 | "id": "yu6Xc8pwfo60" 230 | }, 231 | "source": [ 232 | "Let's take a moment to analyse what we are going to do. First, we create an empty network model. We use \"Sequential\" type - it means that all layers of the network will be connected one after another. In the next line, we add our first layer of newly created network. It is a dense layer taking a few arguments - number of nodes (512), activation function (ReLU) and shape of the input it receives - here it is a 1D vector of length 784 (28*28), so each input corresponds to a single pixel. Now go ahead and add the second dense layer! It should have 10 nodes and use softmax as activation function - this way n-th node will output the probability of input being \"n\" e.g. 4th node being equal to 0.4 will mean that there is a 40% that the digit on the image is a \"4\".\n", 233 | "\n" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": null, 239 | "metadata": { 240 | "id": "tt_fC15d1kkM", 241 | "vscode": { 242 | "languageId": "python" 243 | } 244 | }, 245 | "outputs": [], 246 | "source": [ 247 | "def create_model():\n", 248 | " model = tf.keras.models.Sequential()\n", 249 | " model.add(tf.keras.layers.Dense(512, \n", 250 | " activation=\"relu\", \n", 251 | " input_shape=(28*28,)))\n", 252 | " #TODO: Add next layer\n", 253 | " \n", 254 | " return model" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": { 260 | "id": "O3e4LnBGibiY" 261 | }, 262 | "source": [ 263 | "#4. Train the network\n", 264 | "\n", 265 | "Good job, it looks like we have our network ready! Let's print a summary of it." 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": { 272 | "colab": { 273 | "base_uri": "https://localhost:8080/" 274 | }, 275 | "id": "iSrEEPlGfnK6", 276 | "outputId": "daf93251-33c7-45cb-b3f7-eb55edb00a5c", 277 | "vscode": { 278 | "languageId": "python" 279 | } 280 | }, 281 | "outputs": [], 282 | "source": [ 283 | "print(create_model().summary())" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "metadata": { 289 | "id": "9Afi2IFQitv6" 290 | }, 291 | "source": [ 292 | "In this case, \"params\" mean actual weigth parameters that we will be optimizing during network training. 407,050 parameters for which we have to calculate the gradient - that's quite a lot... Good thing Tensorflow already has it implemented! Let's learn some basics of automatic differentiation before applying it to our network." 293 | ] 294 | }, 295 | { 296 | "cell_type": "markdown", 297 | "metadata": { 298 | "id": "nflzDtY0W0c_" 299 | }, 300 | "source": [ 301 | "##4.1 How to use tf.GradientTape()\n", 302 | "\n", 303 | "GradientTape is one of the core tools in Tensorflow. It allows you to record operations happening during automatic differentiation. Let's compute the gradient for function $y = x^2$ at $x=3$. " 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "metadata": { 310 | "colab": { 311 | "base_uri": "https://localhost:8080/" 312 | }, 313 | "id": "aodbuLoLXzrZ", 314 | "outputId": "b057e922-0721-4de5-b051-0bdf56419eac", 315 | "vscode": { 316 | "languageId": "python" 317 | } 318 | }, 319 | "outputs": [], 320 | "source": [ 321 | "x = tf.Variable(3.0)\n", 322 | "with tf.GradientTape() as tape:\n", 323 | " y = x * x\n", 324 | "dy_dx = tape.gradient(y, x)\n", 325 | "\n", 326 | "print(f\"dy/dx at x=3.0: {dy_dx}\")" 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "metadata": { 332 | "id": "bare03ErYS_K" 333 | }, 334 | "source": [ 335 | "It's super easy, isn't it? Now try to compute multiple gradient - we have two functions: $y = x^2$ and $z = y^2$, our goal is to compute $\\frac{\\partial z}{\\partial x}$ at $x=3$." 336 | ] 337 | }, 338 | { 339 | "cell_type": "code", 340 | "execution_count": null, 341 | "metadata": { 342 | "colab": { 343 | "base_uri": "https://localhost:8080/" 344 | }, 345 | "id": "_iTZZgjkZHyw", 346 | "outputId": "b2b03597-6696-41ad-af78-19d97a7b2c28", 347 | "vscode": { 348 | "languageId": "python" 349 | } 350 | }, 351 | "outputs": [], 352 | "source": [ 353 | "x = tf.Variable(3.0)\n", 354 | "with tf.GradientTape() as tape:\n", 355 | " #TODO: Create y and z functions\n", 356 | "dz_dx = #TODO: Calculate the gradient\n", 357 | "print(f\"dz/dx at x=3.0: {dz_dx}\")\n" 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "metadata": { 363 | "id": "VfU4lmvOZdmz" 364 | }, 365 | "source": [ 366 | "Let's check: $\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} * \\frac{\\partial y}{\\partial x} = 2y(x) \\cdot 2x = 4 \\cdot x^3 \\vert ^{x=3} = 4 \\cdot 27 = 108$\n", 367 | "\n", 368 | "Now you see how easy it is to compute the gradients using Tensorflow - you just tell which gradient you want to get and the library does everything for you." 369 | ] 370 | }, 371 | { 372 | "cell_type": "markdown", 373 | "metadata": { 374 | "id": "dM4aJ6BfeFhx" 375 | }, 376 | "source": [ 377 | "##4.2 Train the network using GradientTape()\n", 378 | "\n" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "metadata": { 384 | "id": "dzUfnn-nfgE9" 385 | }, 386 | "source": [ 387 | "Because we engineered our network to take a 1D vector as an input, we have to convert our 28x28 images to match this format. Next, we will create a dataset using `tf.data.Dataset.from_tensor_slices()` - this function divides our images and labels using their first dimension (in our case this dimension is 60000 - dataset size). This way we will have a dataset in which every element will be a 1D vector of size 784 (image) with a label corresponding to it.\n", 388 | "\n", 389 | " " 390 | ] 391 | }, 392 | { 393 | "cell_type": "code", 394 | "execution_count": null, 395 | "metadata": { 396 | "id": "pSfG94z3shti", 397 | "vscode": { 398 | "languageId": "python" 399 | } 400 | }, 401 | "outputs": [], 402 | "source": [ 403 | "original_images = {\"train\": train_images, \"test\": test_images}\n", 404 | "\n", 405 | "train_images = train_images.reshape(60000, 28 * 28)\n", 406 | "test_images = test_images.reshape(10000, 28 * 28)\n", 407 | "\n", 408 | "train_dataset = tf.data.Dataset.from_tensor_slices((\"\"\"TODO: Pass the data\"\"\"))" 409 | ] 410 | }, 411 | { 412 | "cell_type": "markdown", 413 | "metadata": { 414 | "id": "Jix26WAegsHD" 415 | }, 416 | "source": [ 417 | "Theoretically, we could say that our dataset is now ready - but there is one thing we could do to improve training performance. Instead of passing the whole dataset to the network at each iteration, we could split it into smaller parts called batches which we will pass to the model one at a time. This approach, called mini-batch gradient descent, will improve efficiency and enable faster convergence. Actually, it's quite a complicated concept - I encourage you to Google it later!" 418 | ] 419 | }, 420 | { 421 | "cell_type": "code", 422 | "execution_count": null, 423 | "metadata": { 424 | "id": "qz3AVneyjasR", 425 | "vscode": { 426 | "languageId": "python" 427 | } 428 | }, 429 | "outputs": [], 430 | "source": [ 431 | "batch_size = 256\n", 432 | "train_dataset = train_dataset.batch(batch_size)" 433 | ] 434 | }, 435 | { 436 | "cell_type": "markdown", 437 | "metadata": { 438 | "id": "-yVH3vKrjh1N" 439 | }, 440 | "source": [ 441 | "When the network is trained, it needs a reliable metric to know if it's doing better or worse - it's called the loss function. In our example, we'll be using `tf.keras.losses.sparse_categorical_crossentropy` function, which is very commonly used when dealing with classification models with probabilities as outputs. It uses standard cross entropy formula:\n", 442 | "\n", 443 | "$J(W)=\\frac{1}{n} \\cdot \\sum_{i=1}^{n} y_i log(\\hat{y}_i) + (1-y_i)log(1-\\hat{y}_i)$\n", 444 | "\n", 445 | "where $y_i$ is the actual label for i-th image and $\\hat{y}_i$ is prediction of the model. " 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": null, 451 | "metadata": { 452 | "id": "Q5hBpmwxmELC", 453 | "vscode": { 454 | "languageId": "python" 455 | } 456 | }, 457 | "outputs": [], 458 | "source": [ 459 | "def compute_loss(true_labels, predictions):\n", 460 | " loss = #TODO: Pass labels and predictions to crossentropy loss function\n", 461 | " return loss" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": { 467 | "id": "G-EVN-xCmTb-" 468 | }, 469 | "source": [ 470 | "One of the most crucial things during training the network is setting the correct parameters. We've already chosen one - batch size. Now let's add a few more:\n", 471 | "* Learning rate - size of steps we take in gradient descent. It can't be too small, because we will find ourselves in local minimum, but it can't be too big either - we will jump all across the loss function and never converge.\n", 472 | "* Number of training epochs - how many times we will pass the entire dataset to the model during training.\n", 473 | "* Optimizer - determines how the network will be updated after each iteration based on loss value, it basically implements some variant of Stochastic Gradient Descent. For now, we will use Adam optimizer - a relatively new invention which is often used with great results.\n", 474 | "\n", 475 | "I filled the parameters for you, but don't worry - you will experiment with them later! " 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": null, 481 | "metadata": { 482 | "id": "vlOpLAEpo6t_", 483 | "vscode": { 484 | "languageId": "python" 485 | } 486 | }, 487 | "outputs": [], 488 | "source": [ 489 | "learning_rate = 5e-3\n", 490 | "training_epochs = 5\n", 491 | "optimizer = tf.optimizers.Adam(learning_rate)" 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": { 497 | "id": "F3RRTZFRo-GB" 498 | }, 499 | "source": [ 500 | "We have defined both our model and loss function, so let's move to backpropagating the gradients. First we will create the model and then define a single train step used during the training. At each step, we want to feed the input batch ($x$) to the model, calculate the loss based on actual labels ($y$) and predictions ($\\hat{y}$) and compute the gradients of loss functions with respect to model parameters. The last step will be using our optimizer to adjust weights in the model and return loss value for current step. This task might be the hardest one so far, but when you're done - it will almost be the end!\n", 501 | "\n", 502 | "Hints:\n", 503 | "* `tf.keras.models.Sequential()` is callable, so you can pass the data and get predictions simply by calling it and providing input batch as the argument\n", 504 | "* You can calculate all gradients at once using a single GradientTape - all model parameters can be found at `model.trainable_variables`\n", 505 | "* To backpropagate the gradients, use `apply_gradients` method of our optimizer\n", 506 | "\n", 507 | "If you're not sure what `tf.function` decorator stands for - it takes care of changing our function into Tensorflow graph. You don't have to worry about it for now." 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": null, 513 | "metadata": { 514 | "id": "Vv9Nwdohs9iu", 515 | "vscode": { 516 | "languageId": "python" 517 | } 518 | }, 519 | "outputs": [], 520 | "source": [ 521 | "model = create_model()\n", 522 | "\n", 523 | "@tf.function\n", 524 | "def train_step(x, y):\n", 525 | " with tf.GradientTape() as tape:\n", 526 | " y_hat = #TODO: Use the model to get predictions\n", 527 | "\n", 528 | " loss = #TODO: Calculate loss\n", 529 | "\n", 530 | " gradients = #TODO: Calculate gradients\n", 531 | " \n", 532 | " #TODO: Backpropagate gradients\n", 533 | "\n", 534 | " return loss" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": { 540 | "id": "dKGaaCr-ANyG" 541 | }, 542 | "source": [ 543 | "Good job! Now it's finally time to train the network. We will take a batch of images and labels from training dataset and pass it to our train_step function. We will also plot loss value for each step. Before running the cell, try to guess the answer - what will happen with loss value during the training?" 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": null, 549 | "metadata": { 550 | "colab": { 551 | "base_uri": "https://localhost:8080/", 552 | "height": 541 553 | }, 554 | "id": "nX7rLFNipE9y", 555 | "outputId": "dfa2b4fa-3219-4e9f-f587-7f962000ab86", 556 | "vscode": { 557 | "languageId": "python" 558 | } 559 | }, 560 | "outputs": [], 561 | "source": [ 562 | "history = []\n", 563 | "plotter = mdl.util.PeriodicPlotter(sec=2, xlabel='Iterations', ylabel='Loss')\n", 564 | "\n", 565 | "for epoch in range(training_epochs):\n", 566 | " for step, (\"\"\"TODO: Get image and label batch\"\"\") in enumerate(train_dataset):\n", 567 | " loss = #TODO: Use our train_step function\n", 568 | " history.append(loss.numpy().mean())\n", 569 | " plotter.plot(history)\n" 570 | ] 571 | }, 572 | { 573 | "cell_type": "markdown", 574 | "metadata": { 575 | "id": "mD9bu95FBOPp" 576 | }, 577 | "source": [ 578 | "As we can see on the plot - loss value decreases over time. It means that our optimizer is correctly using computed gradients to change weights of the network to make it better and better at recognizing handwritten digits. But now we can ask ourselves - how good this model actually is? We can see that the loss decreases, but that doesn't tell us anything about the accuracy of our predictions. \n", 579 | "\n", 580 | "That's where we will use our test set. Earlier in the notebook we made an assumption - test set can't be used during training. This is really important, because it allows us to check how our model performs on data that it has never seen before.\n", 581 | "\n", 582 | "As our test metric we will use `tf.keras.metrics.SparseCategoricalAccuracy` - it calculates how often actual labels match our predictions. You just need to pass them as arguments - let's try it." 583 | ] 584 | }, 585 | { 586 | "cell_type": "code", 587 | "execution_count": null, 588 | "metadata": { 589 | "colab": { 590 | "base_uri": "https://localhost:8080/" 591 | }, 592 | "id": "8mxJyRByS2F6", 593 | "outputId": "cdebc2e9-8863-49c0-fe9e-91c7271ebe07", 594 | "vscode": { 595 | "languageId": "python" 596 | } 597 | }, 598 | "outputs": [], 599 | "source": [ 600 | "def evaluate_model(model):\n", 601 | " test_metric = #TODO: Pick test metric\n", 602 | " test_acc = #TODO: Calculate test accuracy\n", 603 | " print(f\"Test accuracy: {test_acc*100:.2f}%\")\n", 604 | "\n", 605 | "evaluate_model(model)" 606 | ] 607 | }, 608 | { 609 | "cell_type": "markdown", 610 | "metadata": { 611 | "id": "_Io4Xzy4Cnup" 612 | }, 613 | "source": [ 614 | "If your accuracy is somewhere over 95% - you did it! You've just built your own neural network with the ability to recognize digits. If you are the type of person who doesn't believe something until you see it - let's visualize our predictions. " 615 | ] 616 | }, 617 | { 618 | "cell_type": "code", 619 | "execution_count": null, 620 | "metadata": { 621 | "colab": { 622 | "base_uri": "https://localhost:8080/", 623 | "height": 589 624 | }, 625 | "id": "3zSgKEf1DO6j", 626 | "outputId": "2cd0b023-810a-41a9-e838-65cc43f05676", 627 | "vscode": { 628 | "languageId": "python" 629 | } 630 | }, 631 | "outputs": [], 632 | "source": [ 633 | "def plot_predictions(predictions_model):\n", 634 | " predictions = predictions_model.predict(test_images)\n", 635 | " plt.figure(figsize=(10,10))\n", 636 | " for i in range(25):\n", 637 | " plt.subplot(5, 5, i+1)\n", 638 | " plt.xticks([])\n", 639 | " plt.yticks([])\n", 640 | " plt.grid(False)\n", 641 | " plt.imshow(original_images[\"test\"][i], cmap=plt.cm.binary)\n", 642 | " plt.xlabel(np.argmax(predictions[i]))\n", 643 | " plt.show()\n", 644 | "\n", 645 | "plot_predictions(model)" 646 | ] 647 | }, 648 | { 649 | "cell_type": "markdown", 650 | "metadata": { 651 | "id": "agQCr-eQDdoZ" 652 | }, 653 | "source": [ 654 | "#4.3 Train the network using model compilation\n", 655 | "\n", 656 | "You successfully created your network but there was quite a lot of coding, wasn't it? We used GradientTape() to see how Tensorflow uses it during the training, but there is a quicker way to implement simple models like this - using model compilation. Instead of creating our own loss function, train step etc., we simply provide the most important parameters to `model.compile()`. In our case it will be the name of the optimizer, loss function and metrics (what metrics we want to evaluate during training). " 657 | ] 658 | }, 659 | { 660 | "cell_type": "code", 661 | "execution_count": null, 662 | "metadata": { 663 | "id": "D1AYJ3CXTwVQ", 664 | "vscode": { 665 | "languageId": "python" 666 | } 667 | }, 668 | "outputs": [], 669 | "source": [ 670 | "model = create_model()\n", 671 | "\n", 672 | "model.compile(optimizer=\"\"\"TODO: Pick optimizer\"\"\",\n", 673 | " loss=\"\"\"TODO: Pick loss function\"\"\",\n", 674 | " metrics=[\"accuracy\"])" 675 | ] 676 | }, 677 | { 678 | "cell_type": "markdown", 679 | "metadata": { 680 | "id": "YoUYO6bqHuIo" 681 | }, 682 | "source": [ 683 | "Before we feed the data to the model, we have to change our labels a bit - instead of a single label, we need to convert it to a 1D vector, where all values will be 0 except the value at index equal to the label - it will be 1. Such representation of data is also called one-hot encoding. " 684 | ] 685 | }, 686 | { 687 | "cell_type": "code", 688 | "execution_count": null, 689 | "metadata": { 690 | "colab": { 691 | "base_uri": "https://localhost:8080/" 692 | }, 693 | "id": "uiW7coCxIGKV", 694 | "outputId": "3e3d2ecd-6b89-4b60-a634-75b0d99ea6be", 695 | "vscode": { 696 | "languageId": "python" 697 | } 698 | }, 699 | "outputs": [], 700 | "source": [ 701 | "print(f\"Label before encoding: {train_labels[0]}\")\n", 702 | "train_labels = tf.keras.utils.to_categorical(train_labels)\n", 703 | "test_labels = tf.keras.utils.to_categorical(test_labels)\n", 704 | "print(f\"Label after encoding: {train_labels[0]}\")" 705 | ] 706 | }, 707 | { 708 | "cell_type": "markdown", 709 | "metadata": { 710 | "id": "ovxInLb1F_xy" 711 | }, 712 | "source": [ 713 | "After compilation of the model, we can train the network using `model.fit()`, once again providing necessary arguments. Try to find yourself what you need to pass to it!" 714 | ] 715 | }, 716 | { 717 | "cell_type": "code", 718 | "execution_count": null, 719 | "metadata": { 720 | "colab": { 721 | "base_uri": "https://localhost:8080/" 722 | }, 723 | "id": "RVk2XWCDGEf9", 724 | "outputId": "d57bd08a-490c-45a9-c346-98df0a0ee4bc", 725 | "vscode": { 726 | "languageId": "python" 727 | } 728 | }, 729 | "outputs": [], 730 | "source": [ 731 | "model.fit(\"\"\"TODO: Pass arguments\"\"\")" 732 | ] 733 | }, 734 | { 735 | "cell_type": "markdown", 736 | "metadata": { 737 | "id": "Uer4iEUuGFEh" 738 | }, 739 | "source": [ 740 | "And finally, to evaluate how our model performs on new data, we will use `model.evaluate()`." 741 | ] 742 | }, 743 | { 744 | "cell_type": "code", 745 | "execution_count": null, 746 | "metadata": { 747 | "colab": { 748 | "base_uri": "https://localhost:8080/" 749 | }, 750 | "id": "K3Q-edlEGWZy", 751 | "outputId": "754c0e47-cb4a-43ae-b18f-9fe930f6da5d", 752 | "vscode": { 753 | "languageId": "python" 754 | } 755 | }, 756 | "outputs": [], 757 | "source": [ 758 | "test_loss, test_acc = model.evaluate(\"\"\"TODO: Pass arguments\"\"\")\n", 759 | "print(f\"Test accuracy: {test_acc*100:.2f}%\")" 760 | ] 761 | }, 762 | { 763 | "cell_type": "markdown", 764 | "metadata": { 765 | "id": "x8K_rR75GXqd" 766 | }, 767 | "source": [ 768 | "As you could see, using model compilation was much faster than writing the whole training implementation yourself. However, bear in mind that sometimes, especially when dealing with more complex network architectures and algorithms, you might find this approach a bit too simplified and that's when writing your own train_step or loss function might come in handy. " 769 | ] 770 | }, 771 | { 772 | "cell_type": "markdown", 773 | "metadata": { 774 | "id": "XoEu_bSZJXP1" 775 | }, 776 | "source": [ 777 | "#5. Experiment!\n", 778 | "\n", 779 | "As I said earlier, there are no strict rules in creating neural networks. Nothing limits you, the model will always give you some answers - in the worst case they will just be less accurate than picking them at random. The task for you is to tweak the parameters and see how they affect performance of the model, maybe you will get better accuracy than now? This is also a great moment to explore things that were omitted in this notebook and during the lecture. I would suggest starting with those things:\n", 780 | "* Change the number of units in the dense layer\n", 781 | "* Add more layers\n", 782 | "* Change activation functions\n", 783 | "* Change optimizers\n", 784 | "* ... and much more!" 785 | ] 786 | } 787 | ], 788 | "metadata": { 789 | "accelerator": "TPU", 790 | "colab": { 791 | "collapsed_sections": [], 792 | "name": "TODO: Introduction to Deep Learning.ipynb", 793 | "provenance": [] 794 | }, 795 | "kernelspec": { 796 | "display_name": "Python 3", 797 | "name": "python3" 798 | }, 799 | "pycharm": { 800 | "stem_cell": { 801 | "cell_type": "raw", 802 | "metadata": { 803 | "collapsed": false 804 | }, 805 | "source": [] 806 | } 807 | } 808 | }, 809 | "nbformat": 4, 810 | "nbformat_minor": 0 811 | } 812 | -------------------------------------------------------------------------------- /2021 Edition/3. Intro to DL/lecture.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/3. Intro to DL/lecture.pdf -------------------------------------------------------------------------------- /2021 Edition/4. CNN/2020/Introduction to CNNs.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/2020/Introduction to CNNs.pdf -------------------------------------------------------------------------------- /2021 Edition/4. CNN/img/colab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/img/colab.png -------------------------------------------------------------------------------- /2021 Edition/4. CNN/img/conv.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/img/conv.gif -------------------------------------------------------------------------------- /2021 Edition/4. CNN/img/graph.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/img/graph.jpg -------------------------------------------------------------------------------- /2021 Edition/4. CNN/img/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/img/logo.png -------------------------------------------------------------------------------- /2021 Edition/4. CNN/img/nn.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/img/nn.jpg -------------------------------------------------------------------------------- /2021 Edition/4. CNN/img/pool.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/4. CNN/img/pool.gif -------------------------------------------------------------------------------- /2021 Edition/4. CNN/introduction_to_cnns.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "TlydNN4HBcJT" 7 | }, 8 | "source": [ 9 | "
\n", 10 | "\n", 11 | "

Introduction to CNNs

\n", 12 | "

Gradient PG, 2021

\n", 13 | "

Marcin Walkowski

\n", 14 | "\n", 15 | "---\n", 16 | "\n", 17 | "\n", 18 | "Run in Google Colab\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "metadata": { 24 | "id": "oeeVdlhoBy-y" 25 | }, 26 | "source": [ 27 | "# Fashion MNIST Classifier\n", 28 | "\n", 29 | "In this notebook, we will return to the problem of image classification. This time, however, we will use the popular Fashion MNIST dataset, which consists of photos of clothing items. We will try an approach based on the architecture typical for computer vision problems. Using the Keras library, we will build simple, convolutional neural networks based classifier. In this notebook, we will use the knowledge gained from the previous hands-on.\n", 30 | "\n", 31 | "Remember to complete all sections marked with #TODO!" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": { 37 | "id": "82WkFLABCDEw" 38 | }, 39 | "source": [ 40 | "# 1. Prepare the environment\n", 41 | "\n", 42 | "Before you get to work, go to ```Runtime``` → ```Change runtime type``` tab and select ```GPU```. Hardware acceleration will speed up code execution. This is particularly important in computer vision problems, where data are images and deep architectures are used.\n", 43 | "\n", 44 | "If you want to start the whole notebook from scratch, press ```Ctrl+F9```. Pressing ```Ctrl+Enter``` will only run the cell in which the cursor is currently located.\n", 45 | "\n", 46 | "Let's start by downloading the packages we need. Like last time, we will use Keras to build our classifier model. We will use NumPy for mathematical operations, and the PyPlot package for visualization." 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": { 53 | "id": "65FTue07CRB3" 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "%tensorflow_version 2.x\n", 58 | "import tensorflow as tf\n", 59 | "from tensorflow import keras\n", 60 | "import matplotlib.pyplot as plt\n", 61 | "import numpy as np" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "id": "UvQY5kI5CTJJ" 68 | }, 69 | "source": [ 70 | "#2. Explore the dataset\n", 71 | "\n", 72 | "The Fashion MNIST dataset comes with the Keras library. The dataset consists of pictures of clothing items and labels in the form of numbers representing specific categories of clothing. Keras provides a dataset already split into training and test subsets.\n", 73 | "\n", 74 | "**Reminder: The training set is used only to train our model. The test set is intended for the evaluation of the obtained model. It cannot be used for training!**" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "metadata": { 81 | "id": "zTCUn5tDC33H" 82 | }, 83 | "outputs": [], 84 | "source": [ 85 | "(train_images, train_labels), (test_images, test_labels) = keras.datasets.fashion_mnist.load_data()" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": { 92 | "id": "uMxFrtl7PEbq" 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "class_names = [\n", 97 | " 'T-shirt/top',\n", 98 | " 'Trouser',\n", 99 | " 'Pullover',\n", 100 | " 'Dress',\n", 101 | " 'Coat',\n", 102 | " 'Sandal',\n", 103 | " 'Shirt',\n", 104 | " 'Sneaker',\n", 105 | " 'Bag',\n", 106 | " 'Ankle boot'\n", 107 | "]" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": { 113 | "id": "6IiHFYEhReth" 114 | }, 115 | "source": [ 116 | "Let's begin with some data exploration." 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": { 123 | "id": "gReClt0PRoIH" 124 | }, 125 | "outputs": [], 126 | "source": [ 127 | "print(f\"Train dataset shape: {train_images.shape}\")\n", 128 | "print(f\"Test dataset shape: {test_images.shape}\")\n", 129 | "print(f\"Minimal value: {np.min(train_images)}\")\n", 130 | "print(f\"Maximal value: {np.max(train_images)}\")\n", 131 | "print(\"-\"*20)\n", 132 | "print(f\"Train labels: {train_labels}\")\n", 133 | "print(f\"Test labels: {test_labels}\")\n", 134 | "print(f\"Train examples: {len(train_labels)}\")\n", 135 | "print(f\"Test examples: {len(test_labels)}\")" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": { 142 | "id": "eAjQwSAcTQeO" 143 | }, 144 | "outputs": [], 145 | "source": [ 146 | "plt.figure(figsize=(10,10))\n", 147 | "for i in range(25):\n", 148 | " plt.subplot(5,5,i+1)\n", 149 | " plt.xticks([])\n", 150 | " plt.yticks([])\n", 151 | " plt.grid(False)\n", 152 | " plt.imshow(train_images[i], cmap=plt.cm.binary)\n", 153 | " plt.xlabel(class_names[train_labels[i]])\n", 154 | "plt.show()" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": { 160 | "id": "eGf6xrdWUUc2" 161 | }, 162 | "source": [ 163 | "As you can see, the data structure is similar to the original MNIST dataset. The images are represented as 28x28 monochrome bitmaps, and the labels are numbers between 0 and 9.\n", 164 | "\n", 165 | "\n", 166 | "Like the last time, we normalize the pixel values ​​to the range from 0.0 to 1.0 and write them as float32." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": { 173 | "id": "3YD2U4GQUu1N" 174 | }, 175 | "outputs": [], 176 | "source": [ 177 | "train_images = #TODO normalize train images\n", 178 | "test_images = #TODO normalize test images\n", 179 | "\n", 180 | "print(f\"New minimal value: {np.min(train_images)}\")\n", 181 | "print(f\"New maximal value: {np.max(train_images)}\")" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": { 188 | "id": "hl8FJ-zzSVwQ" 189 | }, 190 | "outputs": [], 191 | "source": [ 192 | "plt.figure()\n", 193 | "plt.imshow(train_images[0], cmap=plt.cm.binary)\n", 194 | "plt.colorbar()\n", 195 | "plt.grid(False)\n", 196 | "plt.title(class_names[train_labels[0]])\n", 197 | "plt.show()" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": { 203 | "id": "6Nzv262aUxza" 204 | }, 205 | "source": [ 206 | "To pass the images as input to the convolutional layer, we need to extend them with an additional dimension representing depth. This is because the implementation of convolutional layers in Keras is adapted to processing, more complex data, like colour images (RGB), where the pixel is encoded with three values. The images from the Fashion MNIST set are monochromatic, so the number of channels will be one.\n" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": { 213 | "id": "i64hOltIqp5p" 214 | }, 215 | "outputs": [], 216 | "source": [ 217 | "train_images = np.expand_dims(train_images, -1)\n", 218 | "test_images = np.expand_dims(test_images, -1)\n", 219 | "\n", 220 | "print(f\"New train image shape: {train_images[0].shape}\")\n", 221 | "print(f\"New test image shape: {test_images[0].shape}\")" 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "metadata": { 227 | "id": "5doMFT5trfXO" 228 | }, 229 | "source": [ 230 | "The size one extra dimension is often called the dummy dimension." 231 | ] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "metadata": { 236 | "id": "lQagNU2RDNpQ" 237 | }, 238 | "source": [ 239 | "# 3. Create the network\n", 240 | "Our model will consist of two parts. The first part is responsible for creating the image representation vector using the convolution operation. The second part is a dense classifier, which task is to predict the label for the obtained representation.\n", 241 | "\n", 242 | "\n", 243 | "\n", 244 | "We will use the ```keras.Sequential``` type to build our model." 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": null, 250 | "metadata": { 251 | "id": "N2FZwZXq6ZH2" 252 | }, 253 | "outputs": [], 254 | "source": [ 255 | "model = keras.Sequential()\n", 256 | "\n", 257 | "input_image_shape = (28, 28, 1)\n", 258 | "model.add(keras.Input(shape=input_image_shape))" 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": { 264 | "id": "oJUITijP2SJY" 265 | }, 266 | "source": [ 267 | "We explicitly define the size of the input data by adding the ```keras.Input()``` object to the model.\n", 268 | "\n" 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "metadata": { 274 | "id": "PqTK4CGWDWkv" 275 | }, 276 | "source": [ 277 | "## 3.1 Convolution layer\n", 278 | "\n", 279 | "The convolution operation applies a filter/kernel to the input data. A kernel multiplies the input data area by the learned weights and then sums them up. The purpose of the operation is to extract information about high-level features, such as edges, from the image. By shifting the kernel by some step, we create a new data representation containing information about high-level features. By applying many different filters we can create multiple feature maps of the input image.\n", 280 | "\n", 281 | "\n", 282 | "\n", 283 | "Image source: A Comprehensive Guide to Convolutional Neural Networks\n", 284 | "\n", 285 | "Keras provides the implementation of the convolutional layer. Our task is to provide the correct parameters. We will use 32 filters of 3x3 size. As the default filter step is one, the operation will produce 32 feature maps of the input data with a size of 26x26 each. We will use the ReLU activation function. It is currently the most popular activation function used in convolutional networks" 286 | ] 287 | }, 288 | { 289 | "cell_type": "code", 290 | "execution_count": null, 291 | "metadata": { 292 | "id": "rL0_klC08XQf" 293 | }, 294 | "outputs": [], 295 | "source": [ 296 | "conv_layer_1 = keras.layers.Conv2D(32, kernel_size=(3, 3), activation=\"relu\")\n", 297 | "\n", 298 | "model.add(conv_layer_1)" 299 | ] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "metadata": { 304 | "id": "n2dycrcZD2fq" 305 | }, 306 | "source": [ 307 | "## 3.2 Pooling layer\n", 308 | "\n", 309 | "The purpose of the pooling layer is to reduce the spatial size of data. We want to reduce the computation time and extract only the dominant image features. Similarly to the convolution operation, a filter/kernel is shifted over the data. For Max-Pooling, the highest activation is selected from the kernel area. For Average-Pooling, we use the average activation value.\n", 310 | "\n", 311 | "\n", 312 | "\n", 313 | "Image source: A Comprehensive Guide to Convolutional Neural Networks\n", 314 | "\n", 315 | "We will use the Max-Pooling operation available in Keras. We set the kernel size to 2x2. The default step is equal to the kernel width/height. The feature map size will be reduced from 26x26 to 13x13. Remembering that we are operating on 32 feature maps, the shape of the output data will be 13x13x32." 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "metadata": { 322 | "id": "4kB2YAmb8w27" 323 | }, 324 | "outputs": [], 325 | "source": [ 326 | "pool_layer_1 = keras.layers.MaxPooling2D(pool_size=(2, 2))\n", 327 | "\n", 328 | "model.add(pool_layer_1)" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": { 334 | "id": "Nh6haosT9M-W" 335 | }, 336 | "source": [ 337 | "Now let's add an additional convolution and pooling layer to our model." 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": { 344 | "id": "Qqa39k7j9YS9" 345 | }, 346 | "outputs": [], 347 | "source": [ 348 | "#TODO Add Conv2D layer (I suggest 3x3 kernel and 64 filters, but fill free to experiment)\n", 349 | "#TODO Add MaxPool2D layer (I suggest 2x2 kernel)\n", 350 | "\n", 351 | "# model.add(conv_layer_2)\n", 352 | "# model.add(pool_layer_2)" 353 | ] 354 | }, 355 | { 356 | "cell_type": "markdown", 357 | "metadata": { 358 | "id": "uNw4y01uD_bM" 359 | }, 360 | "source": [ 361 | "## 3.3 Dense classifier\n", 362 | "\n", 363 | "After convolutional operations, we obtained a new high-level representation of the data. We will use the fully connected network to classify it. To pass processed data to the dense layer, we have to transform it into a vector. Operation Flatten available in Keras \"flattens\" the matrix to one dimension.\n", 364 | "\n" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "metadata": { 371 | "id": "mWFaFvdH92C5" 372 | }, 373 | "outputs": [], 374 | "source": [ 375 | "model.add(keras.layers.Flatten())" 376 | ] 377 | }, 378 | { 379 | "cell_type": "markdown", 380 | "metadata": { 381 | "id": "WgpoLuTgwSBf" 382 | }, 383 | "source": [ 384 | "\n", 385 | "The Dense layer should consist of 10 nodes and use the Softmax activation function so that the output on the nth node corresponds to the probability of the nth class." 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "metadata": { 392 | "id": "lQyyBsIMwSR0" 393 | }, 394 | "outputs": [], 395 | "source": [ 396 | "num_classes = 10\n", 397 | "#TODO Add Dense layer\n", 398 | "# model.add(dense_layer)" 399 | ] 400 | }, 401 | { 402 | "cell_type": "markdown", 403 | "metadata": { 404 | "id": "dYePr8Zz_6og" 405 | }, 406 | "source": [ 407 | "Our simple classifier model is now ready! As you may have noticed, even building more complex architectures with ML libraries can be fast. For a better understanding of CNN's architecture, I recommend reading thru the article \"A Comprehensive Guide to Convolutional Neural Networks\"." 408 | ] 409 | }, 410 | { 411 | "cell_type": "code", 412 | "execution_count": null, 413 | "metadata": { 414 | "id": "PdXlTnADAFMd" 415 | }, 416 | "outputs": [], 417 | "source": [ 418 | "model.summary()" 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "metadata": { 424 | "id": "vZ1LBtgJFlDS" 425 | }, 426 | "source": [ 427 | "# 4. Train the model\n", 428 | "\n", 429 | "To speed up the training process we will use the ```model.compile()``` method. As we are dealing with a classification problem, we will use the Categorical Crossentropy loss function. Due to the format of our labels, which are encoded as integers, we will use the ```keras.losses.sparse_categorical_crossentropy``` implementation." 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": null, 435 | "metadata": { 436 | "id": "5OovL01mAnX3" 437 | }, 438 | "outputs": [], 439 | "source": [ 440 | "loss_fn = #TODO Create loss function\n", 441 | "optimizer = #TODO Pick optimizer (Adam is a common choice)\n", 442 | "\n", 443 | "model.compile(loss=loss_fn, optimizer=optimizer, metrics=[\"accuracy\"])" 444 | ] 445 | }, 446 | { 447 | "cell_type": "markdown", 448 | "metadata": { 449 | "id": "iiQnDssY1x1B" 450 | }, 451 | "source": [ 452 | "It's time to train our model! Adjust the number of epochs and the mini-batch size (15 epochs and batch size 128 is a good starting point)." 453 | ] 454 | }, 455 | { 456 | "cell_type": "code", 457 | "execution_count": null, 458 | "metadata": { 459 | "id": "WfQP77uWA0hd" 460 | }, 461 | "outputs": [], 462 | "source": [ 463 | "batch_size = #TODO \n", 464 | "epochs = #TODO\n", 465 | "model.fit(train_images, train_labels, batch_size=batch_size, epochs=epochs)" 466 | ] 467 | }, 468 | { 469 | "cell_type": "markdown", 470 | "metadata": { 471 | "id": "t9GtjNlo2dO-" 472 | }, 473 | "source": [ 474 | "Finally, let's use the ```model.evaluate()``` method to see how our network deals with the test set." 475 | ] 476 | }, 477 | { 478 | "cell_type": "code", 479 | "execution_count": null, 480 | "metadata": { 481 | "id": "HNDF0PtxDMGW" 482 | }, 483 | "outputs": [], 484 | "source": [ 485 | "score = #TODO Evalutae model on the test set\n", 486 | "print(f\"Test loss: {score[0]}\")\n", 487 | "print(f\"Test accuracy: {score[1]*100:.2f}%\")" 488 | ] 489 | }, 490 | { 491 | "cell_type": "markdown", 492 | "metadata": { 493 | "id": "M9utgT1f2usZ" 494 | }, 495 | "source": [ 496 | "If you have completed all the #TODO sections correctly, you should get an accuracy of around 90% on the test set. It is a good result but leaves room for improvement. Try different network parameter configurations and see if you can improve this score." 497 | ] 498 | }, 499 | { 500 | "cell_type": "markdown", 501 | "metadata": { 502 | "id": "KkbDqdN5RlQl" 503 | }, 504 | "source": [ 505 | "# 5. Use the trained model\n", 506 | "\n", 507 | "Use the code below to see how your trained model works. Each time you execute the cell below, the image is sampled from the test set, and the model predicts the class." 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": null, 513 | "metadata": { 514 | "id": "vUMmQOZ94dCR" 515 | }, 516 | "outputs": [], 517 | "source": [ 518 | "# Sample image index\n", 519 | "index = np.random.randint(0, len(test_images))\n", 520 | "img = np.expand_dims(test_images[index], 0)\n", 521 | "# Predict label\n", 522 | "predictions_array = model.predict(img).squeeze()\n", 523 | "predicted_label = np.argmax(predictions_array)\n", 524 | "true_label = test_labels[index]\n", 525 | "\n", 526 | "plt.figure(figsize=(10,5))\n", 527 | "\n", 528 | "# Plot image\n", 529 | "plt.subplot(1,2,1)\n", 530 | "plt.grid(False)\n", 531 | "plt.xticks([])\n", 532 | "plt.yticks([])\n", 533 | "\n", 534 | "plt.imshow(img.squeeze(), cmap=plt.cm.binary)\n", 535 | "if predicted_label == true_label:\n", 536 | " color = 'blue'\n", 537 | "else:\n", 538 | " color = 'red'\n", 539 | "\n", 540 | "plt.xlabel(\n", 541 | " f\"{class_names[predicted_label]} \" + \n", 542 | " f\"{100*np.max(predictions_array):2.0f}% \" +\n", 543 | " f\"({class_names[true_label]})\",\n", 544 | " color=color\n", 545 | " )\n", 546 | "\n", 547 | "# Plot probabilities\n", 548 | "plt.subplot(1,2,2)\n", 549 | "plt.grid(False)\n", 550 | "plt.xticks(range(10), class_names, rotation=45)\n", 551 | "plt.yticks([])\n", 552 | "thisplot = plt.bar(range(10), predictions_array, color=\"#777777\")\n", 553 | "plt.ylim([0, 1])\n", 554 | "predicted_label = np.argmax(predictions_array)\n", 555 | "\n", 556 | "thisplot[predicted_label].set_color('red')\n", 557 | "thisplot[true_label].set_color('blue')\n", 558 | "\n", 559 | "plt.show()" 560 | ] 561 | }, 562 | { 563 | "cell_type": "markdown", 564 | "metadata": { 565 | "id": "jYq2MeK-FwsC" 566 | }, 567 | "source": [ 568 | "# 6. Experiment... with more complex data!\n", 569 | "\n", 570 | "Now that you know how to quickly build an image classification model try your hand with more complex data. The CIFAR10 dataset, available in Keras, contains pictures representing one of the ten types of objects." 571 | ] 572 | }, 573 | { 574 | "cell_type": "code", 575 | "execution_count": null, 576 | "metadata": { 577 | "id": "3o3iS2pXPpiT" 578 | }, 579 | "outputs": [], 580 | "source": [ 581 | "(cifar_train_images, cifar_train_labels), (cifar_test_images, cifar_test_labels) = keras.datasets.cifar10.load_data()\n", 582 | "\n", 583 | "cifar_class_names = [\n", 584 | " 'Airplane',\n", 585 | " 'Automobile',\n", 586 | " 'Bird',\n", 587 | " 'Cat',\n", 588 | " 'Deer',\n", 589 | " 'Dog',\n", 590 | " 'Frog',\n", 591 | " 'Horse',\n", 592 | " 'Ship',\n", 593 | " 'Truck'\n", 594 | "]" 595 | ] 596 | }, 597 | { 598 | "cell_type": "markdown", 599 | "metadata": { 600 | "id": "YhuRPtPmAHYK" 601 | }, 602 | "source": [ 603 | "The dataset format is similar to Fashion MNIST, except that the images are 32x32 and are in colour (pixel encoded with three values)." 604 | ] 605 | }, 606 | { 607 | "cell_type": "code", 608 | "execution_count": null, 609 | "metadata": { 610 | "id": "uYsow1SuP9fj" 611 | }, 612 | "outputs": [], 613 | "source": [ 614 | "plt.figure(figsize=(10,10))\n", 615 | "for i in range(25):\n", 616 | " plt.subplot(5,5,i+1)\n", 617 | " plt.xticks([])\n", 618 | " plt.yticks([])\n", 619 | " plt.grid(False)\n", 620 | " plt.imshow(cifar_train_images[i], cmap=plt.cm.binary)\n", 621 | " plt.xlabel(cifar_class_names[cifar_train_labels[i][0]])\n", 622 | "plt.show()" 623 | ] 624 | } 625 | ], 626 | "metadata": { 627 | "accelerator": "GPU", 628 | "colab": { 629 | "collapsed_sections": [], 630 | "name": "Introduction to CNNs.ipynb", 631 | "provenance": [] 632 | }, 633 | "kernelspec": { 634 | "display_name": "Python 3", 635 | "name": "python3" 636 | }, 637 | "language_info": { 638 | "name": "python" 639 | } 640 | }, 641 | "nbformat": 4, 642 | "nbformat_minor": 0 643 | } 644 | -------------------------------------------------------------------------------- /2021 Edition/5. RNN/lecture.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/5. RNN/lecture.pdf -------------------------------------------------------------------------------- /2021 Edition/6. ML Tools/lecture.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/6. ML Tools/lecture.pdf -------------------------------------------------------------------------------- /2021 Edition/7. ML Deployment/lecture.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2021 Edition/7. ML Deployment/lecture.pdf -------------------------------------------------------------------------------- /2022 Edition/1. Kick-off meeting/slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/1. Kick-off meeting/slides.pdf -------------------------------------------------------------------------------- /2022 Edition/2. Basics of Machine Learning/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/2. Basics of Machine Learning/.gitkeep -------------------------------------------------------------------------------- /2022 Edition/2. Basics of Machine Learning/2. Introduction to ML.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/2. Basics of Machine Learning/2. Introduction to ML.pdf -------------------------------------------------------------------------------- /2022 Edition/3. Classical Algorithms/3. Algorytmy Klasyczne.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/3. Classical Algorithms/3. Algorytmy Klasyczne.pdf -------------------------------------------------------------------------------- /2022 Edition/4. Introduction to Deep Learning/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/4. Introduction to Deep Learning/.gitkeep -------------------------------------------------------------------------------- /2022 Edition/4. Introduction to Deep Learning/4. Introduction to Deep Learning.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/4. Introduction to Deep Learning/4. Introduction to Deep Learning.pdf -------------------------------------------------------------------------------- /2022 Edition/5. Computer Vision/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/5. Computer Vision/.gitkeep -------------------------------------------------------------------------------- /2022 Edition/5. Computer Vision/5. Computer vision.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/5. Computer Vision/5. Computer vision.pdf -------------------------------------------------------------------------------- /2022 Edition/6. Recurrent Neural Networks/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/6. Recurrent Neural Networks/.gitkeep -------------------------------------------------------------------------------- /2022 Edition/6. Recurrent Neural Networks/6.Sequential Data.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/6. Recurrent Neural Networks/6.Sequential Data.pdf -------------------------------------------------------------------------------- /2022 Edition/6. Recurrent Neural Networks/6.Sequential Data.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/6. Recurrent Neural Networks/6.Sequential Data.pptx -------------------------------------------------------------------------------- /2022 Edition/7. Reinforcement Learning/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/7. Reinforcement Learning/.gitkeep -------------------------------------------------------------------------------- /2022 Edition/7. Reinforcement Learning/7. Reinforcement Learning.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/7. Reinforcement Learning/7. Reinforcement Learning.pdf -------------------------------------------------------------------------------- /2022 Edition/7. Reinforcement Learning/7. Reinforcement Learning.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/7. Reinforcement Learning/7. Reinforcement Learning.pptx -------------------------------------------------------------------------------- /2022 Edition/8. Technical Aspects in ML/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/8. Technical Aspects in ML/.gitkeep -------------------------------------------------------------------------------- /2022 Edition/8. Technical Aspects in ML/slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2022 Edition/8. Technical Aspects in ML/slides.pdf -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Classical Algorithms/Algorytmy Klasyczne.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Deep_Learning_course/Classical Algorithms/Algorytmy Klasyczne.pdf -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Classical Algorithms/datasets/car_data.csv: -------------------------------------------------------------------------------- 1 | Car_Name,Year,Selling_Price,Present_Price,Kms_Driven,Fuel_Type,Seller_Type,Transmission,Owner 2 | ritz,2014,3.35,5.59,27000,Petrol,Dealer,Manual,0 3 | sx4,2013,4.75,9.54,43000,Diesel,Dealer,Manual,0 4 | ciaz,2017,7.25,9.85,6900,Petrol,Dealer,Manual,0 5 | wagon r,2011,2.85,4.15,5200,Petrol,Dealer,Manual,0 6 | swift,2014,4.6,6.87,42450,Diesel,Dealer,Manual,0 7 | vitara brezza,2018,9.25,9.83,2071,Diesel,Dealer,Manual,0 8 | ciaz,2015,6.75,8.12,18796,Petrol,Dealer,Manual,0 9 | s cross,2015,6.5,8.61,33429,Diesel,Dealer,Manual,0 10 | ciaz,2016,8.75,8.89,20273,Diesel,Dealer,Manual,0 11 | ciaz,2015,7.45,8.92,42367,Diesel,Dealer,Manual,0 12 | alto 800,2017,2.85,3.6,2135,Petrol,Dealer,Manual,0 13 | ciaz,2015,6.85,10.38,51000,Diesel,Dealer,Manual,0 14 | ciaz,2015,7.5,9.94,15000,Petrol,Dealer,Automatic,0 15 | ertiga,2015,6.1,7.71,26000,Petrol,Dealer,Manual,0 16 | dzire,2009,2.25,7.21,77427,Petrol,Dealer,Manual,0 17 | ertiga,2016,7.75,10.79,43000,Diesel,Dealer,Manual,0 18 | ertiga,2015,7.25,10.79,41678,Diesel,Dealer,Manual,0 19 | ertiga,2016,7.75,10.79,43000,Diesel,Dealer,Manual,0 20 | wagon r,2015,3.25,5.09,35500,CNG,Dealer,Manual,0 21 | sx4,2010,2.65,7.98,41442,Petrol,Dealer,Manual,0 22 | alto k10,2016,2.85,3.95,25000,Petrol,Dealer,Manual,0 23 | ignis,2017,4.9,5.71,2400,Petrol,Dealer,Manual,0 24 | sx4,2011,4.4,8.01,50000,Petrol,Dealer,Automatic,0 25 | alto k10,2014,2.5,3.46,45280,Petrol,Dealer,Manual,0 26 | wagon r,2013,2.9,4.41,56879,Petrol,Dealer,Manual,0 27 | swift,2011,3,4.99,20000,Petrol,Dealer,Manual,0 28 | swift,2013,4.15,5.87,55138,Petrol,Dealer,Manual,0 29 | swift,2017,6,6.49,16200,Petrol,Individual,Manual,0 30 | alto k10,2010,1.95,3.95,44542,Petrol,Dealer,Manual,0 31 | ciaz,2015,7.45,10.38,45000,Diesel,Dealer,Manual,0 32 | ritz,2012,3.1,5.98,51439,Diesel,Dealer,Manual,0 33 | ritz,2011,2.35,4.89,54200,Petrol,Dealer,Manual,0 34 | swift,2014,4.95,7.49,39000,Diesel,Dealer,Manual,0 35 | ertiga,2014,6,9.95,45000,Diesel,Dealer,Manual,0 36 | dzire,2014,5.5,8.06,45000,Diesel,Dealer,Manual,0 37 | sx4,2011,2.95,7.74,49998,CNG,Dealer,Manual,0 38 | dzire,2015,4.65,7.2,48767,Petrol,Dealer,Manual,0 39 | 800,2003,0.35,2.28,127000,Petrol,Individual,Manual,0 40 | alto k10,2016,3,3.76,10079,Petrol,Dealer,Manual,0 41 | sx4,2003,2.25,7.98,62000,Petrol,Dealer,Manual,0 42 | baleno,2016,5.85,7.87,24524,Petrol,Dealer,Automatic,0 43 | alto k10,2014,2.55,3.98,46706,Petrol,Dealer,Manual,0 44 | sx4,2008,1.95,7.15,58000,Petrol,Dealer,Manual,0 45 | dzire,2014,5.5,8.06,45780,Diesel,Dealer,Manual,0 46 | omni,2012,1.25,2.69,50000,Petrol,Dealer,Manual,0 47 | ciaz,2014,7.5,12.04,15000,Petrol,Dealer,Automatic,0 48 | ritz,2013,2.65,4.89,64532,Petrol,Dealer,Manual,0 49 | wagon r,2006,1.05,4.15,65000,Petrol,Dealer,Manual,0 50 | ertiga,2015,5.8,7.71,25870,Petrol,Dealer,Manual,0 51 | ciaz,2017,7.75,9.29,37000,Petrol,Dealer,Automatic,0 52 | fortuner,2012,14.9,30.61,104707,Diesel,Dealer,Automatic,0 53 | fortuner,2015,23,30.61,40000,Diesel,Dealer,Automatic,0 54 | innova,2017,18,19.77,15000,Diesel,Dealer,Automatic,0 55 | fortuner,2013,16,30.61,135000,Diesel,Individual,Automatic,0 56 | innova,2005,2.75,10.21,90000,Petrol,Individual,Manual,0 57 | corolla altis,2009,3.6,15.04,70000,Petrol,Dealer,Automatic,0 58 | etios cross,2015,4.5,7.27,40534,Petrol,Dealer,Manual,0 59 | corolla altis,2010,4.75,18.54,50000,Petrol,Dealer,Manual,0 60 | etios g,2014,4.1,6.8,39485,Petrol,Dealer,Manual,1 61 | fortuner,2014,19.99,35.96,41000,Diesel,Dealer,Automatic,0 62 | corolla altis,2013,6.95,18.61,40001,Petrol,Dealer,Manual,0 63 | etios cross,2015,4.5,7.7,40588,Petrol,Dealer,Manual,0 64 | fortuner,2014,18.75,35.96,78000,Diesel,Dealer,Automatic,0 65 | fortuner,2015,23.5,35.96,47000,Diesel,Dealer,Automatic,0 66 | fortuner,2017,33,36.23,6000,Diesel,Dealer,Automatic,0 67 | etios liva,2014,4.75,6.95,45000,Diesel,Dealer,Manual,0 68 | innova,2017,19.75,23.15,11000,Petrol,Dealer,Automatic,0 69 | fortuner,2010,9.25,20.45,59000,Diesel,Dealer,Manual,0 70 | corolla altis,2011,4.35,13.74,88000,Petrol,Dealer,Manual,0 71 | corolla altis,2016,14.25,20.91,12000,Petrol,Dealer,Manual,0 72 | etios liva,2014,3.95,6.76,71000,Diesel,Dealer,Manual,0 73 | corolla altis,2011,4.5,12.48,45000,Diesel,Dealer,Manual,0 74 | corolla altis,2013,7.45,18.61,56001,Petrol,Dealer,Manual,0 75 | etios liva,2011,2.65,5.71,43000,Petrol,Dealer,Manual,0 76 | etios cross,2014,4.9,8.93,83000,Diesel,Dealer,Manual,0 77 | etios g,2015,3.95,6.8,36000,Petrol,Dealer,Manual,0 78 | corolla altis,2013,5.5,14.68,72000,Petrol,Dealer,Manual,0 79 | corolla,2004,1.5,12.35,135154,Petrol,Dealer,Automatic,0 80 | corolla altis,2010,5.25,22.83,80000,Petrol,Dealer,Automatic,0 81 | fortuner,2012,14.5,30.61,89000,Diesel,Dealer,Automatic,0 82 | corolla altis,2016,14.73,14.89,23000,Diesel,Dealer,Manual,0 83 | etios gd,2015,4.75,7.85,40000,Diesel,Dealer,Manual,0 84 | innova,2017,23,25.39,15000,Diesel,Dealer,Automatic,0 85 | innova,2015,12.5,13.46,38000,Diesel,Dealer,Manual,0 86 | innova,2005,3.49,13.46,197176,Diesel,Dealer,Manual,0 87 | camry,2006,2.5,23.73,142000,Petrol,Individual,Automatic,3 88 | land cruiser,2010,35,92.6,78000,Diesel,Dealer,Manual,0 89 | corolla altis,2012,5.9,13.74,56000,Petrol,Dealer,Manual,0 90 | etios liva,2013,3.45,6.05,47000,Petrol,Dealer,Manual,0 91 | etios g,2014,4.75,6.76,40000,Petrol,Dealer,Manual,0 92 | corolla altis,2009,3.8,18.61,62000,Petrol,Dealer,Manual,0 93 | innova,2014,11.25,16.09,58242,Diesel,Dealer,Manual,0 94 | innova,2005,3.51,13.7,75000,Petrol,Dealer,Manual,0 95 | fortuner,2015,23,30.61,40000,Diesel,Dealer,Automatic,0 96 | corolla altis,2008,4,22.78,89000,Petrol,Dealer,Automatic,0 97 | corolla altis,2012,5.85,18.61,72000,Petrol,Dealer,Manual,0 98 | innova,2016,20.75,25.39,29000,Diesel,Dealer,Automatic,0 99 | corolla altis,2017,17,18.64,8700,Petrol,Dealer,Manual,0 100 | corolla altis,2013,7.05,18.61,45000,Petrol,Dealer,Manual,0 101 | fortuner,2010,9.65,20.45,50024,Diesel,Dealer,Manual,0 102 | Royal Enfield Thunder 500,2016,1.75,1.9,3000,Petrol,Individual,Manual,0 103 | UM Renegade Mojave,2017,1.7,1.82,1400,Petrol,Individual,Manual,0 104 | KTM RC200,2017,1.65,1.78,4000,Petrol,Individual,Manual,0 105 | Bajaj Dominar 400,2017,1.45,1.6,1200,Petrol,Individual,Manual,0 106 | Royal Enfield Classic 350,2017,1.35,1.47,4100,Petrol,Individual,Manual,0 107 | KTM RC390,2015,1.35,2.37,21700,Petrol,Individual,Manual,0 108 | Hyosung GT250R,2014,1.35,3.45,16500,Petrol,Individual,Manual,1 109 | Royal Enfield Thunder 350,2013,1.25,1.5,15000,Petrol,Individual,Manual,0 110 | Royal Enfield Thunder 350,2016,1.2,1.5,18000,Petrol,Individual,Manual,0 111 | Royal Enfield Classic 350,2017,1.2,1.47,11000,Petrol,Individual,Manual,0 112 | KTM RC200,2016,1.2,1.78,6000,Petrol,Individual,Manual,0 113 | Royal Enfield Thunder 350,2016,1.15,1.5,8700,Petrol,Individual,Manual,0 114 | KTM 390 Duke ,2014,1.15,2.4,7000,Petrol,Individual,Manual,0 115 | Mahindra Mojo XT300,2016,1.15,1.4,35000,Petrol,Individual,Manual,0 116 | Royal Enfield Classic 350,2015,1.15,1.47,17000,Petrol,Individual,Manual,0 117 | Royal Enfield Classic 350,2015,1.11,1.47,17500,Petrol,Individual,Manual,0 118 | Royal Enfield Classic 350,2013,1.1,1.47,33000,Petrol,Individual,Manual,0 119 | Royal Enfield Thunder 500,2015,1.1,1.9,14000,Petrol,Individual,Manual,0 120 | Royal Enfield Classic 350,2015,1.1,1.47,26000,Petrol,Individual,Manual,0 121 | Royal Enfield Thunder 500,2013,1.05,1.9,5400,Petrol,Individual,Manual,0 122 | Bajaj Pulsar RS200,2016,1.05,1.26,5700,Petrol,Individual,Manual,0 123 | Royal Enfield Thunder 350,2011,1.05,1.5,6900,Petrol,Individual,Manual,0 124 | Royal Enfield Bullet 350,2016,1.05,1.17,6000,Petrol,Individual,Manual,0 125 | Royal Enfield Classic 350,2013,1,1.47,46500,Petrol,Individual,Manual,0 126 | Royal Enfield Classic 500,2012,0.95,1.75,11500,Petrol,Individual,Manual,0 127 | Royal Enfield Classic 500,2009,0.9,1.75,40000,Petrol,Individual,Manual,0 128 | Bajaj Avenger 220,2017,0.9,0.95,1300,Petrol,Individual,Manual,0 129 | Bajaj Avenger 150,2016,0.75,0.8,7000,Petrol,Individual,Manual,0 130 | Honda CB Hornet 160R,2017,0.8,0.87,3000,Petrol,Individual,Manual,0 131 | Yamaha FZ S V 2.0,2017,0.78,0.84,5000,Petrol,Individual,Manual,0 132 | Honda CB Hornet 160R,2017,0.75,0.87,11000,Petrol,Individual,Manual,0 133 | Yamaha FZ 16,2015,0.75,0.82,18000,Petrol,Individual,Manual,0 134 | Bajaj Avenger 220,2017,0.75,0.95,3500,Petrol,Individual,Manual,0 135 | Bajaj Avenger 220,2016,0.72,0.95,500,Petrol,Individual,Manual,0 136 | TVS Apache RTR 160,2017,0.65,0.81,11800,Petrol,Individual,Manual,0 137 | Bajaj Pulsar 150,2015,0.65,0.74,5000,Petrol,Individual,Manual,0 138 | Honda CBR 150,2014,0.65,1.2,23500,Petrol,Individual,Manual,0 139 | Hero Extreme,2013,0.65,0.787,16000,Petrol,Individual,Manual,0 140 | Honda CB Hornet 160R,2016,0.6,0.87,15000,Petrol,Individual,Manual,0 141 | Bajaj Avenger 220 dtsi,2015,0.6,0.95,16600,Petrol,Individual,Manual,0 142 | Honda CBR 150,2013,0.6,1.2,32000,Petrol,Individual,Manual,0 143 | Bajaj Avenger 150 street,2016,0.6,0.8,20000,Petrol,Individual,Manual,0 144 | Yamaha FZ v 2.0,2015,0.6,0.84,29000,Petrol,Individual,Manual,0 145 | Yamaha FZ v 2.0,2016,0.6,0.84,25000,Petrol,Individual,Manual,0 146 | Bajaj Pulsar NS 200,2014,0.6,0.99,25000,Petrol,Individual,Manual,0 147 | TVS Apache RTR 160,2012,0.6,0.81,19000,Petrol,Individual,Manual,0 148 | Hero Extreme,2014,0.55,0.787,15000,Petrol,Individual,Manual,0 149 | Yamaha FZ S V 2.0,2015,0.55,0.84,58000,Petrol,Individual,Manual,0 150 | Bajaj Pulsar 220 F,2010,0.52,0.94,45000,Petrol,Individual,Manual,0 151 | Bajaj Pulsar 220 F,2016,0.51,0.94,24000,Petrol,Individual,Manual,0 152 | TVS Apache RTR 180,2011,0.5,0.826,6000,Petrol,Individual,Manual,0 153 | Hero Passion X pro,2016,0.5,0.55,31000,Petrol,Individual,Manual,0 154 | Bajaj Pulsar NS 200,2012,0.5,0.99,13000,Petrol,Individual,Manual,0 155 | Bajaj Pulsar NS 200,2013,0.5,0.99,45000,Petrol,Individual,Manual,0 156 | Yamaha Fazer ,2014,0.5,0.88,8000,Petrol,Individual,Manual,0 157 | Honda Activa 4G,2017,0.48,0.51,4300,Petrol,Individual,Automatic,0 158 | TVS Sport ,2017,0.48,0.52,15000,Petrol,Individual,Manual,0 159 | Yamaha FZ S V 2.0,2015,0.48,0.84,23000,Petrol,Individual,Manual,0 160 | Honda Dream Yuga ,2017,0.48,0.54,8600,Petrol,Individual,Manual,0 161 | Honda Activa 4G,2017,0.45,0.51,4000,Petrol,Individual,Automatic,0 162 | Bajaj Avenger Street 220,2011,0.45,0.95,24000,Petrol,Individual,Manual,0 163 | TVS Apache RTR 180,2014,0.45,0.826,23000,Petrol,Individual,Manual,0 164 | Bajaj Pulsar NS 200,2012,0.45,0.99,14500,Petrol,Individual,Manual,0 165 | Bajaj Avenger 220 dtsi,2010,0.45,0.95,27000,Petrol,Individual,Manual,0 166 | Hero Splender iSmart,2016,0.45,0.54,14000,Petrol,Individual,Manual,0 167 | Activa 3g,2016,0.45,0.54,500,Petrol,Individual,Automatic,0 168 | Hero Passion Pro,2016,0.45,0.55,1000,Petrol,Individual,Manual,0 169 | TVS Apache RTR 160,2014,0.42,0.81,42000,Petrol,Individual,Manual,0 170 | Honda CB Trigger,2013,0.42,0.73,12000,Petrol,Individual,Manual,0 171 | Hero Splender iSmart,2015,0.4,0.54,14000,Petrol,Individual,Manual,0 172 | Yamaha FZ S ,2012,0.4,0.83,5500,Petrol,Individual,Manual,0 173 | Hero Passion Pro,2015,0.4,0.55,6700,Petrol,Individual,Manual,0 174 | Bajaj Pulsar 135 LS,2014,0.4,0.64,13700,Petrol,Individual,Manual,0 175 | Activa 4g,2017,0.4,0.51,1300,Petrol,Individual,Automatic,0 176 | Honda CB Unicorn,2015,0.38,0.72,38600,Petrol,Individual,Manual,0 177 | Hero Honda CBZ extreme,2011,0.38,0.787,75000,Petrol,Individual,Manual,0 178 | Honda Karizma,2011,0.35,1.05,30000,Petrol,Individual,Manual,0 179 | Honda Activa 125,2016,0.35,0.57,24000,Petrol,Individual,Automatic,0 180 | TVS Jupyter,2014,0.35,0.52,19000,Petrol,Individual,Automatic,0 181 | Honda Karizma,2010,0.31,1.05,213000,Petrol,Individual,Manual,0 182 | Hero Honda Passion Pro,2012,0.3,0.51,60000,Petrol,Individual,Manual,0 183 | Hero Splender Plus,2016,0.3,0.48,50000,Petrol,Individual,Manual,0 184 | Honda CB Shine,2013,0.3,0.58,30000,Petrol,Individual,Manual,0 185 | Bajaj Discover 100,2013,0.27,0.47,21000,Petrol,Individual,Manual,0 186 | Bajaj Pulsar 150,2008,0.25,0.75,26000,Petrol,Individual,Manual,1 187 | Suzuki Access 125,2008,0.25,0.58,1900,Petrol,Individual,Automatic,0 188 | TVS Wego,2010,0.25,0.52,22000,Petrol,Individual,Automatic,0 189 | Honda CB twister,2013,0.25,0.51,32000,Petrol,Individual,Manual,0 190 | Hero Glamour,2013,0.25,0.57,18000,Petrol,Individual,Manual,0 191 | Hero Super Splendor,2005,0.2,0.57,55000,Petrol,Individual,Manual,0 192 | Bajaj Pulsar 150,2008,0.2,0.75,60000,Petrol,Individual,Manual,0 193 | Bajaj Discover 125,2012,0.2,0.57,25000,Petrol,Individual,Manual,1 194 | Hero Hunk,2007,0.2,0.75,49000,Petrol,Individual,Manual,1 195 | Hero Ignitor Disc,2013,0.2,0.65,24000,Petrol,Individual,Manual,1 196 | Hero CBZ Xtreme,2008,0.2,0.787,50000,Petrol,Individual,Manual,0 197 | Bajaj ct 100,2015,0.18,0.32,35000,Petrol,Individual,Manual,0 198 | Activa 3g,2008,0.17,0.52,500000,Petrol,Individual,Automatic,0 199 | Honda CB twister,2010,0.16,0.51,33000,Petrol,Individual,Manual,0 200 | Bajaj Discover 125,2011,0.15,0.57,35000,Petrol,Individual,Manual,1 201 | Honda CB Shine,2007,0.12,0.58,53000,Petrol,Individual,Manual,0 202 | Bajaj Pulsar 150,2006,0.1,0.75,92233,Petrol,Individual,Manual,0 203 | i20,2010,3.25,6.79,58000,Diesel,Dealer,Manual,1 204 | grand i10,2015,4.4,5.7,28200,Petrol,Dealer,Manual,0 205 | i10,2011,2.95,4.6,53460,Petrol,Dealer,Manual,0 206 | eon,2015,2.75,4.43,28282,Petrol,Dealer,Manual,0 207 | grand i10,2016,5.25,5.7,3493,Petrol,Dealer,Manual,1 208 | xcent,2017,5.75,7.13,12479,Petrol,Dealer,Manual,0 209 | grand i10,2015,5.15,5.7,34797,Petrol,Dealer,Automatic,0 210 | i20,2017,7.9,8.1,3435,Petrol,Dealer,Manual,0 211 | grand i10,2015,4.85,5.7,21125,Diesel,Dealer,Manual,0 212 | i10,2012,3.1,4.6,35775,Petrol,Dealer,Manual,0 213 | elantra,2015,11.75,14.79,43535,Diesel,Dealer,Manual,0 214 | creta,2016,11.25,13.6,22671,Petrol,Dealer,Manual,0 215 | i20,2011,2.9,6.79,31604,Petrol,Dealer,Manual,0 216 | grand i10,2017,5.25,5.7,20114,Petrol,Dealer,Manual,0 217 | verna,2012,4.5,9.4,36100,Petrol,Dealer,Manual,0 218 | eon,2016,2.9,4.43,12500,Petrol,Dealer,Manual,0 219 | eon,2016,3.15,4.43,15000,Petrol,Dealer,Manual,0 220 | verna,2014,6.45,9.4,45078,Petrol,Dealer,Manual,0 221 | verna,2012,4.5,9.4,36000,Petrol,Dealer,Manual,0 222 | eon,2017,3.5,4.43,38488,Petrol,Dealer,Manual,0 223 | i20,2013,4.5,6.79,32000,Petrol,Dealer,Automatic,0 224 | i20,2014,6,7.6,77632,Diesel,Dealer,Manual,0 225 | verna,2015,8.25,9.4,61381,Diesel,Dealer,Manual,0 226 | verna,2013,5.11,9.4,36198,Petrol,Dealer,Automatic,0 227 | i10,2011,2.7,4.6,22517,Petrol,Dealer,Manual,0 228 | grand i10,2015,5.25,5.7,24678,Petrol,Dealer,Manual,0 229 | i10,2011,2.55,4.43,57000,Petrol,Dealer,Manual,0 230 | verna,2012,4.95,9.4,60000,Diesel,Dealer,Manual,0 231 | i20,2012,3.1,6.79,52132,Diesel,Dealer,Manual,0 232 | verna,2013,6.15,9.4,45000,Diesel,Dealer,Manual,0 233 | verna,2017,9.25,9.4,15001,Petrol,Dealer,Manual,0 234 | elantra,2015,11.45,14.79,12900,Petrol,Dealer,Automatic,0 235 | grand i10,2013,3.9,5.7,53000,Diesel,Dealer,Manual,0 236 | grand i10,2015,5.5,5.7,4492,Petrol,Dealer,Manual,0 237 | verna,2017,9.1,9.4,15141,Petrol,Dealer,Manual,0 238 | eon,2016,3.1,4.43,11849,Petrol,Dealer,Manual,0 239 | creta,2015,11.25,13.6,68000,Diesel,Dealer,Manual,0 240 | verna,2013,4.8,9.4,60241,Petrol,Dealer,Manual,0 241 | eon,2012,2,4.43,23709,Petrol,Dealer,Manual,0 242 | verna,2012,5.35,9.4,32322,Diesel,Dealer,Manual,0 243 | xcent,2015,4.75,7.13,35866,Petrol,Dealer,Manual,1 244 | xcent,2014,4.4,7.13,34000,Petrol,Dealer,Manual,0 245 | i20,2016,6.25,7.6,7000,Petrol,Dealer,Manual,0 246 | verna,2013,5.95,9.4,49000,Diesel,Dealer,Manual,0 247 | verna,2012,5.2,9.4,71000,Diesel,Dealer,Manual,0 248 | i20,2012,3.75,6.79,35000,Petrol,Dealer,Manual,0 249 | verna,2015,5.95,9.4,36000,Petrol,Dealer,Manual,0 250 | i10,2013,4,4.6,30000,Petrol,Dealer,Manual,0 251 | i20,2016,5.25,7.6,17000,Petrol,Dealer,Manual,0 252 | creta,2016,12.9,13.6,35934,Diesel,Dealer,Manual,0 253 | city,2013,5,9.9,56701,Petrol,Dealer,Manual,0 254 | brio,2015,5.4,6.82,31427,Petrol,Dealer,Automatic,0 255 | city,2014,7.2,9.9,48000,Diesel,Dealer,Manual,0 256 | city,2013,5.25,9.9,54242,Petrol,Dealer,Manual,0 257 | brio,2012,3,5.35,53675,Petrol,Dealer,Manual,0 258 | city,2016,10.25,13.6,49562,Petrol,Dealer,Manual,0 259 | city,2015,8.5,13.6,40324,Petrol,Dealer,Manual,0 260 | city,2015,8.4,13.6,25000,Petrol,Dealer,Manual,0 261 | amaze,2014,3.9,7,36054,Petrol,Dealer,Manual,0 262 | city,2016,9.15,13.6,29223,Petrol,Dealer,Manual,0 263 | brio,2016,5.5,5.97,5600,Petrol,Dealer,Manual,0 264 | amaze,2015,4,5.8,40023,Petrol,Dealer,Manual,0 265 | jazz,2016,6.6,7.7,16002,Petrol,Dealer,Manual,0 266 | amaze,2015,4,7,40026,Petrol,Dealer,Manual,0 267 | jazz,2017,6.5,8.7,21200,Petrol,Dealer,Manual,0 268 | amaze,2014,3.65,7,35000,Petrol,Dealer,Manual,0 269 | city,2016,8.35,9.4,19434,Diesel,Dealer,Manual,0 270 | brio,2017,4.8,5.8,19000,Petrol,Dealer,Manual,0 271 | city,2015,6.7,10,18828,Petrol,Dealer,Manual,0 272 | city,2011,4.1,10,69341,Petrol,Dealer,Manual,0 273 | city,2009,3,10,69562,Petrol,Dealer,Manual,0 274 | city,2015,7.5,10,27600,Petrol,Dealer,Manual,0 275 | jazz,2010,2.25,7.5,61203,Petrol,Dealer,Manual,0 276 | brio,2014,5.3,6.8,16500,Petrol,Dealer,Manual,0 277 | city,2016,10.9,13.6,30753,Petrol,Dealer,Automatic,0 278 | city,2015,8.65,13.6,24800,Petrol,Dealer,Manual,0 279 | city,2015,9.7,13.6,21780,Petrol,Dealer,Manual,0 280 | jazz,2016,6,8.4,4000,Petrol,Dealer,Manual,0 281 | city,2014,6.25,13.6,40126,Petrol,Dealer,Manual,0 282 | brio,2015,5.25,5.9,14465,Petrol,Dealer,Manual,0 283 | city,2006,2.1,7.6,50456,Petrol,Dealer,Manual,0 284 | city,2014,8.25,14,63000,Diesel,Dealer,Manual,0 285 | city,2016,8.99,11.8,9010,Petrol,Dealer,Manual,0 286 | brio,2013,3.5,5.9,9800,Petrol,Dealer,Manual,0 287 | jazz,2016,7.4,8.5,15059,Petrol,Dealer,Automatic,0 288 | jazz,2016,5.65,7.9,28569,Petrol,Dealer,Manual,0 289 | amaze,2015,5.75,7.5,44000,Petrol,Dealer,Automatic,0 290 | city,2015,8.4,13.6,34000,Petrol,Dealer,Manual,0 291 | city,2016,10.11,13.6,10980,Petrol,Dealer,Manual,0 292 | amaze,2014,4.5,6.4,19000,Petrol,Dealer,Manual,0 293 | brio,2015,5.4,6.1,31427,Petrol,Dealer,Manual,0 294 | jazz,2016,6.4,8.4,12000,Petrol,Dealer,Manual,0 295 | city,2010,3.25,9.9,38000,Petrol,Dealer,Manual,0 296 | amaze,2014,3.75,6.8,33019,Petrol,Dealer,Manual,0 297 | city,2015,8.55,13.09,60076,Diesel,Dealer,Manual,0 298 | city,2016,9.5,11.6,33988,Diesel,Dealer,Manual,0 299 | brio,2015,4,5.9,60000,Petrol,Dealer,Manual,0 300 | city,2009,3.35,11,87934,Petrol,Dealer,Manual,0 301 | city,2017,11.5,12.5,9000,Diesel,Dealer,Manual,0 302 | brio,2016,5.3,5.9,5464,Petrol,Dealer,Manual,0 303 | -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Deep_Learning_in_Computer_Vision/4. Deep Learning in Computer Vision-1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Deep_Learning_course/Deep_Learning_in_Computer_Vision/4. Deep Learning in Computer Vision-1.pdf -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Implementation_of_NN/CNN_notebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "source": [ 6 | "Run in Colab: https://colab.research.google.com/drive/1K1aK-q2XbFLHzU08xVBtnb36HleR5mG_?usp=sharing" 7 | ], 8 | "metadata": { 9 | "id": "ph0sATzpIzmw" 10 | }, 11 | "id": "ph0sATzpIzmw" 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "dd8bc65e", 16 | "metadata": { 17 | "id": "dd8bc65e" 18 | }, 19 | "source": [ 20 | "# Convolutional Neural Networks\n", 21 | " This code defines a Convolutional Neural Network (CNN) model using PyTorch. The model is used for image classification tasks." 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "id": "aa160f82", 27 | "metadata": { 28 | "id": "aa160f82" 29 | }, 30 | "source": [ 31 | "# Model architecture\n", 32 | "## Convolutional Layers\n", 33 | " The CNN consists of two convolutional layers with ReLU activation functions and max pooling layers. The first convolutional layer takes a 3-channel input image and outputs 6 feature maps, using a 4x4 kernel. The second convolutional layer takes the output of the first convolutional layer as input and outputs 9 feature maps, using a 5x5 kernel.\n", 34 | "## Linear Layers\n", 35 | " The output from the last convolutional layer is flattened into a 1D tensor and passed through two fully connected (linear) layers with ReLU activation functions. The first linear layer has 80 neurons and the second has 10 neurons (the number of classes in the classification task)." 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "id": "c43039c6", 41 | "metadata": { 42 | "id": "c43039c6" 43 | }, 44 | "source": [ 45 | "# Model Definition" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "id": "b11c4293", 52 | "metadata": { 53 | "id": "b11c4293" 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "import torch.nn as nn\n", 58 | "\n", 59 | "\n", 60 | "class ConvModel(nn.Module):\n", 61 | " def __init__(self):\n", 62 | " super().__init__()\n", 63 | " self.conv = nn.Sequential(\n", 64 | " nn.Conv2d(3, 6, 4),\n", 65 | " nn.ReLU(),\n", 66 | " nn.MaxPool2d(2, 2),\n", 67 | "\n", 68 | " nn.Conv2d(6, 9, 5),\n", 69 | " nn.ReLU(),\n", 70 | " nn.MaxPool2d(2, 2),\n", 71 | " )\n", 72 | " self.linear = nn.Sequential(\n", 73 | " nn.Linear(5 * 5 * 9, 80),\n", 74 | " nn.ReLU(),\n", 75 | " nn.Linear(80, 10),\n", 76 | " )\n", 77 | "\n", 78 | " def forward(self, x):\n", 79 | " x = self.conv(x)\n", 80 | " x = x.view(-1, 5 * 5 * 9)\n", 81 | " x = self.linear(x)\n", 82 | " return x\n" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "id": "96b69466", 88 | "metadata": { 89 | "id": "96b69466" 90 | }, 91 | "source": [ 92 | "In the **forward** method, the input tensor is first passed through the convolutional layers and then flattened into a 1D tensor. The flattened tensor is then passed through the linear layers to produce the final output." 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "id": "d3c16e60", 98 | "metadata": { 99 | "id": "d3c16e60" 100 | }, 101 | "source": [ 102 | "# CNN Training" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "id": "f4621864", 109 | "metadata": { 110 | "id": "f4621864", 111 | "outputId": "d7bc42aa-22d4-4f89-817c-9d361ab35fe0" 112 | }, 113 | "outputs": [ 114 | { 115 | "name": "stdout", 116 | "output_type": "stream", 117 | "text": [ 118 | "Files already downloaded and verified\n", 119 | "Files already downloaded and verified\n", 120 | "Epoch: 1/20 | Step: 2000;250000 | Loss: 1.868149995803833\n", 121 | "Epoch: 1/20 | Step: 4000;250000 | Loss: 1.808663249015808\n", 122 | "Epoch: 1/20 | Step: 6000;250000 | Loss: 2.0486741065979004\n", 123 | "Epoch: 1/20 | Step: 8000;250000 | Loss: 2.148817539215088\n", 124 | "Epoch: 1/20 | Step: 10000;250000 | Loss: 1.7136116027832031\n", 125 | "Epoch: 1/20 | Step: 12000;250000 | Loss: 1.2580480575561523\n", 126 | "Epoch: 2/20 | Step: 14500;250000 | Loss: 0.9803611636161804\n", 127 | "Epoch: 2/20 | Step: 16500;250000 | Loss: 0.6896300911903381\n", 128 | "Epoch: 2/20 | Step: 18500;250000 | Loss: 0.8863891959190369\n", 129 | "Epoch: 2/20 | Step: 20500;250000 | Loss: 1.2686976194381714\n", 130 | "Epoch: 2/20 | Step: 22500;250000 | Loss: 0.6618356704711914\n", 131 | "Epoch: 2/20 | Step: 24500;250000 | Loss: 1.5075292587280273\n", 132 | "Epoch: 3/20 | Step: 27000;250000 | Loss: 1.0405328273773193\n", 133 | "Epoch: 3/20 | Step: 29000;250000 | Loss: 1.1788005828857422\n", 134 | "Epoch: 3/20 | Step: 31000;250000 | Loss: 0.28276947140693665\n", 135 | "Epoch: 3/20 | Step: 33000;250000 | Loss: 0.5511508584022522\n", 136 | "Epoch: 3/20 | Step: 35000;250000 | Loss: 1.1681625843048096\n", 137 | "Epoch: 3/20 | Step: 37000;250000 | Loss: 0.7604244351387024\n", 138 | "Epoch: 4/20 | Step: 39500;250000 | Loss: 1.8094203472137451\n", 139 | "Epoch: 4/20 | Step: 41500;250000 | Loss: 1.0180308818817139\n", 140 | "Epoch: 4/20 | Step: 43500;250000 | Loss: 1.2012836933135986\n", 141 | "Epoch: 4/20 | Step: 45500;250000 | Loss: 1.326040267944336\n", 142 | "Epoch: 4/20 | Step: 47500;250000 | Loss: 1.4490337371826172\n", 143 | "Epoch: 4/20 | Step: 49500;250000 | Loss: 0.5611826777458191\n", 144 | "Epoch: 5/20 | Step: 52000;250000 | Loss: 1.1142256259918213\n", 145 | "Epoch: 5/20 | Step: 54000;250000 | Loss: 1.4296016693115234\n", 146 | "Epoch: 5/20 | Step: 56000;250000 | Loss: 2.276676654815674\n", 147 | "Epoch: 5/20 | Step: 58000;250000 | Loss: 0.9492870569229126\n", 148 | "Epoch: 5/20 | Step: 60000;250000 | Loss: 2.0789132118225098\n", 149 | "Epoch: 5/20 | Step: 62000;250000 | Loss: 1.1576035022735596\n", 150 | "Epoch: 6/20 | Step: 64500;250000 | Loss: 0.4269680380821228\n", 151 | "Epoch: 6/20 | Step: 66500;250000 | Loss: 0.7921403646469116\n", 152 | "Epoch: 6/20 | Step: 68500;250000 | Loss: 2.279670000076294\n", 153 | "Epoch: 6/20 | Step: 70500;250000 | Loss: 0.6963077783584595\n", 154 | "Epoch: 6/20 | Step: 72500;250000 | Loss: 0.6398743987083435\n", 155 | "Epoch: 6/20 | Step: 74500;250000 | Loss: 1.271297812461853\n", 156 | "Epoch: 7/20 | Step: 77000;250000 | Loss: 0.3938380777835846\n", 157 | "Epoch: 7/20 | Step: 79000;250000 | Loss: 1.205625295639038\n", 158 | "Epoch: 7/20 | Step: 81000;250000 | Loss: 0.7136484980583191\n", 159 | "Epoch: 7/20 | Step: 83000;250000 | Loss: 0.8042351007461548\n", 160 | "Epoch: 7/20 | Step: 85000;250000 | Loss: 1.6126559972763062\n", 161 | "Epoch: 7/20 | Step: 87000;250000 | Loss: 0.28250449895858765\n", 162 | "Epoch: 8/20 | Step: 89500;250000 | Loss: 1.3294621706008911\n", 163 | "Epoch: 8/20 | Step: 91500;250000 | Loss: 1.0522974729537964\n", 164 | "Epoch: 8/20 | Step: 93500;250000 | Loss: 0.924340009689331\n", 165 | "Epoch: 8/20 | Step: 95500;250000 | Loss: 0.8075780868530273\n", 166 | "Epoch: 8/20 | Step: 97500;250000 | Loss: 2.1963303089141846\n", 167 | "Epoch: 8/20 | Step: 99500;250000 | Loss: 0.9101738333702087\n", 168 | "Epoch: 9/20 | Step: 102000;250000 | Loss: 1.7532811164855957\n", 169 | "Epoch: 9/20 | Step: 104000;250000 | Loss: 0.9450191259384155\n", 170 | "Epoch: 9/20 | Step: 106000;250000 | Loss: 0.8171242475509644\n", 171 | "Epoch: 9/20 | Step: 108000;250000 | Loss: 0.6594465374946594\n", 172 | "Epoch: 9/20 | Step: 110000;250000 | Loss: 1.352130651473999\n", 173 | "Epoch: 9/20 | Step: 112000;250000 | Loss: 1.8082475662231445\n", 174 | "Epoch: 10/20 | Step: 114500;250000 | Loss: 0.705681562423706\n", 175 | "Epoch: 10/20 | Step: 116500;250000 | Loss: 0.6882866621017456\n", 176 | "Epoch: 10/20 | Step: 118500;250000 | Loss: 0.9365450143814087\n", 177 | "Epoch: 10/20 | Step: 120500;250000 | Loss: 1.0160834789276123\n", 178 | "Epoch: 10/20 | Step: 122500;250000 | Loss: 1.345630407333374\n", 179 | "Epoch: 10/20 | Step: 124500;250000 | Loss: 1.0211046934127808\n", 180 | "Epoch: 11/20 | Step: 127000;250000 | Loss: 1.4727071523666382\n", 181 | "Epoch: 11/20 | Step: 129000;250000 | Loss: 1.2301959991455078\n", 182 | "Epoch: 11/20 | Step: 131000;250000 | Loss: 1.3269494771957397\n", 183 | "Epoch: 11/20 | Step: 133000;250000 | Loss: 0.6285428404808044\n", 184 | "Epoch: 11/20 | Step: 135000;250000 | Loss: 1.069046139717102\n", 185 | "Epoch: 11/20 | Step: 137000;250000 | Loss: 0.8192417621612549\n", 186 | "Epoch: 12/20 | Step: 139500;250000 | Loss: 0.7842008471488953\n", 187 | "Epoch: 12/20 | Step: 141500;250000 | Loss: 0.4815007448196411\n", 188 | "Epoch: 12/20 | Step: 143500;250000 | Loss: 1.2153191566467285\n", 189 | "Epoch: 12/20 | Step: 145500;250000 | Loss: 1.181143045425415\n", 190 | "Epoch: 12/20 | Step: 147500;250000 | Loss: 0.9448485970497131\n", 191 | "Epoch: 12/20 | Step: 149500;250000 | Loss: 0.8398048281669617\n", 192 | "Epoch: 13/20 | Step: 152000;250000 | Loss: 0.9905456900596619\n", 193 | "Epoch: 13/20 | Step: 154000;250000 | Loss: 0.5820574760437012\n", 194 | "Epoch: 13/20 | Step: 156000;250000 | Loss: 1.0987743139266968\n", 195 | "Epoch: 13/20 | Step: 158000;250000 | Loss: 0.553002119064331\n", 196 | "Epoch: 13/20 | Step: 160000;250000 | Loss: 1.3437292575836182\n", 197 | "Epoch: 13/20 | Step: 162000;250000 | Loss: 0.636999249458313\n", 198 | "Epoch: 14/20 | Step: 164500;250000 | Loss: 0.6099785566329956\n", 199 | "Epoch: 14/20 | Step: 166500;250000 | Loss: 1.0579653978347778\n", 200 | "Epoch: 14/20 | Step: 168500;250000 | Loss: 1.0641443729400635\n", 201 | "Epoch: 14/20 | Step: 170500;250000 | Loss: 0.658701479434967\n", 202 | "Epoch: 14/20 | Step: 172500;250000 | Loss: 3.1185717582702637\n", 203 | "Epoch: 14/20 | Step: 174500;250000 | Loss: 0.8587619066238403\n", 204 | "Epoch: 15/20 | Step: 177000;250000 | Loss: 1.8661249876022339\n", 205 | "Epoch: 15/20 | Step: 179000;250000 | Loss: 0.6939230561256409\n", 206 | "Epoch: 15/20 | Step: 181000;250000 | Loss: 0.5399549603462219\n", 207 | "Epoch: 15/20 | Step: 183000;250000 | Loss: 0.7700042128562927\n", 208 | "Epoch: 15/20 | Step: 185000;250000 | Loss: 0.7247531414031982\n", 209 | "Epoch: 15/20 | Step: 187000;250000 | Loss: 0.746134877204895\n", 210 | "Epoch: 16/20 | Step: 189500;250000 | Loss: 0.4735276401042938\n", 211 | "Epoch: 16/20 | Step: 191500;250000 | Loss: 0.6705207824707031\n", 212 | "Epoch: 16/20 | Step: 193500;250000 | Loss: 0.863497793674469\n", 213 | "Epoch: 16/20 | Step: 195500;250000 | Loss: 0.7962812185287476\n", 214 | "Epoch: 16/20 | Step: 197500;250000 | Loss: 0.5657091736793518\n", 215 | "Epoch: 16/20 | Step: 199500;250000 | Loss: 1.288136601448059\n", 216 | "Epoch: 17/20 | Step: 202000;250000 | Loss: 0.8295432925224304\n", 217 | "Epoch: 17/20 | Step: 204000;250000 | Loss: 0.4508132338523865\n", 218 | "Epoch: 17/20 | Step: 206000;250000 | Loss: 1.2814412117004395\n", 219 | "Epoch: 17/20 | Step: 208000;250000 | Loss: 0.27081042528152466\n", 220 | "Epoch: 17/20 | Step: 210000;250000 | Loss: 0.9623105525970459\n", 221 | "Epoch: 17/20 | Step: 212000;250000 | Loss: 0.9066767692565918\n", 222 | "Epoch: 18/20 | Step: 214500;250000 | Loss: 1.0262855291366577\n", 223 | "Epoch: 18/20 | Step: 216500;250000 | Loss: 0.7027924060821533\n", 224 | "Epoch: 18/20 | Step: 218500;250000 | Loss: 1.3816018104553223\n", 225 | "Epoch: 18/20 | Step: 220500;250000 | Loss: 1.6824274063110352\n", 226 | "Epoch: 18/20 | Step: 222500;250000 | Loss: 1.5017560720443726\n", 227 | "Epoch: 18/20 | Step: 224500;250000 | Loss: 0.2840704023838043\n", 228 | "Epoch: 19/20 | Step: 227000;250000 | Loss: 0.837134063243866\n", 229 | "Epoch: 19/20 | Step: 229000;250000 | Loss: 0.6496121287345886\n", 230 | "Epoch: 19/20 | Step: 231000;250000 | Loss: 0.29826340079307556\n", 231 | "Epoch: 19/20 | Step: 233000;250000 | Loss: 1.4169983863830566\n", 232 | "Epoch: 19/20 | Step: 235000;250000 | Loss: 2.216799736022949\n", 233 | "Epoch: 19/20 | Step: 237000;250000 | Loss: 0.7554011344909668\n", 234 | "Epoch: 20/20 | Step: 239500;250000 | Loss: 0.653556227684021\n", 235 | "Epoch: 20/20 | Step: 241500;250000 | Loss: 0.38188064098358154\n", 236 | "Epoch: 20/20 | Step: 243500;250000 | Loss: 0.9678006172180176\n", 237 | "Epoch: 20/20 | Step: 245500;250000 | Loss: 1.0471148490905762\n", 238 | "Epoch: 20/20 | Step: 247500;250000 | Loss: 1.0283788442611694\n", 239 | "Epoch: 20/20 | Step: 249500;250000 | Loss: 0.3411109149456024\n", 240 | "Training data accuracy is: 0.68204\n", 241 | "Test data accuracy is: 0.6101\n" 242 | ] 243 | } 244 | ], 245 | "source": [ 246 | "import torch\n", 247 | "import torch.nn as nn\n", 248 | "import torchvision.datasets as datasets\n", 249 | "from torch.utils.data import DataLoader\n", 250 | "from model import ConvModel\n", 251 | "import torchvision.transforms as tf\n", 252 | "\n", 253 | "\n", 254 | "def train():\n", 255 | " device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 256 | "\n", 257 | " learning_rate = 3e-4\n", 258 | " batch_size = 4\n", 259 | " num_epochs = 20\n", 260 | "\n", 261 | " model = ConvModel().to(device)\n", 262 | "\n", 263 | " criterion = nn.CrossEntropyLoss()\n", 264 | " optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n", 265 | "\n", 266 | " transforms = tf.Compose([\n", 267 | " tf.ToTensor(),\n", 268 | " tf.Normalize((0.5,), (0.5,), (0.5,))\n", 269 | " ])\n", 270 | "\n", 271 | " train_dataset = datasets.CIFAR10(\"dataset/\", train=True, transform=transforms, download=True)\n", 272 | " train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n", 273 | "\n", 274 | " test_dataset = datasets.CIFAR10(\"dataset/\", train=False, transform=transforms, download=True)\n", 275 | " test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\n", 276 | "\n", 277 | " step = 0\n", 278 | " total_steps = len(train_loader) * num_epochs\n", 279 | "\n", 280 | " for epoch in range(num_epochs):\n", 281 | " for idx, (images, labels) in enumerate(train_loader):\n", 282 | " images = images.to(device)\n", 283 | " labels = labels.to(device)\n", 284 | "\n", 285 | " outputs = model(images)\n", 286 | "\n", 287 | " loss = criterion(outputs, labels)\n", 288 | " optimizer.zero_grad()\n", 289 | " loss.backward()\n", 290 | " optimizer.step()\n", 291 | "\n", 292 | " if (idx + 1) % 2000 == 0:\n", 293 | " print(f'Epoch: {epoch + 1}/{num_epochs} | Step: {step + 1};{total_steps} | Loss: {loss}')\n", 294 | "\n", 295 | " step += 1\n", 296 | "\n", 297 | " with torch.no_grad():\n", 298 | " n_correct = 0\n", 299 | " n_samples = 0\n", 300 | " n_class_correct = [0 for _ in range(10)]\n", 301 | " n_class_samples = [0 for _ in range(10)]\n", 302 | "\n", 303 | " for images, labels in train_loader:\n", 304 | " images = images.to(device)\n", 305 | " labels = labels.to(device)\n", 306 | "\n", 307 | " output = model(images)\n", 308 | " _, predicted = torch.max(output, 1)\n", 309 | " n_samples += labels.size(0)\n", 310 | " n_correct += (predicted == labels).sum().item()\n", 311 | "\n", 312 | " for i in range(batch_size):\n", 313 | " label = labels[i]\n", 314 | " pred = predicted[i]\n", 315 | "\n", 316 | " if label == pred:\n", 317 | " n_class_correct[label.item()] += 1\n", 318 | "\n", 319 | " n_class_samples[label.item()] += 1\n", 320 | "\n", 321 | " print(f'Training data accuracy is: {n_correct / n_samples}')\n", 322 | "\n", 323 | " with torch.no_grad():\n", 324 | " n_correct = 0\n", 325 | " n_samples = 0\n", 326 | " n_class_correct = [0 for _ in range(10)]\n", 327 | " n_class_samples = [0 for _ in range(10)]\n", 328 | "\n", 329 | " for images, labels in test_loader:\n", 330 | " images = images.to(device)\n", 331 | " labels = labels.to(device)\n", 332 | "\n", 333 | " output = model(images)\n", 334 | " _, predicted = torch.max(output, 1)\n", 335 | " n_samples += labels.size(0)\n", 336 | " n_correct += (predicted == labels).sum().item()\n", 337 | "\n", 338 | " for i in range(batch_size):\n", 339 | " label = labels[i]\n", 340 | " pred = predicted[i]\n", 341 | "\n", 342 | " if label == pred:\n", 343 | " n_class_correct[label.item()] += 1\n", 344 | "\n", 345 | " n_class_samples[label.item()] += 1\n", 346 | "\n", 347 | " print(f'Test data accuracy is: {n_correct / n_samples}')\n", 348 | "\n", 349 | "\n", 350 | "if __name__ == \"__main__\":\n", 351 | " train()\n" 352 | ] 353 | } 354 | ], 355 | "metadata": { 356 | "kernelspec": { 357 | "display_name": "Python 3 (ipykernel)", 358 | "language": "python", 359 | "name": "python3" 360 | }, 361 | "language_info": { 362 | "codemirror_mode": { 363 | "name": "ipython", 364 | "version": 3 365 | }, 366 | "file_extension": ".py", 367 | "mimetype": "text/x-python", 368 | "name": "python", 369 | "nbconvert_exporter": "python", 370 | "pygments_lexer": "ipython3", 371 | "version": "3.10.10" 372 | }, 373 | "colab": { 374 | "provenance": [] 375 | } 376 | }, 377 | "nbformat": 4, 378 | "nbformat_minor": 5 379 | } -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Implementation_of_NN/Implementation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "source": [ 6 | "Link do edytowania i sprawdzenia kodu: https://colab.research.google.com/drive/1tl_tbeHeuM9OX7mBDTKc7_QxseY6Alef?usp=sharing" 7 | ], 8 | "metadata": { 9 | "id": "viKsE1rbIaVK" 10 | } 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": { 15 | "id": "Mh6UJUk--i7o" 16 | }, 17 | "source": [ 18 | "# Importowane biblioteki\n", 19 | "\n", 20 | "torch - główna biblioteka projektu znanego jako PyTorch, obecnie jedna z najpopularniejszych bibliotek do Uczenia Głębokiego\n", 21 | "\n", 22 | "torch.nn - moduł z biblioteki torch zawierający klasy (funkcje) odpowiadające za implementację wszystkich bloków funkcjonalnych sieci neuronowych takich jak warstwy, funkcje aktywacyjne czy funkcje kosztu.\n", 23 | "\n", 24 | "torch.optim - moduł biblioteki torch zawierający implementację optymalizatorów takich jak SGD, Ada czy Adam.\n", 25 | "\n", 26 | "torchvision - inna z bibliotek projektu PyTorch, zawierająca popularne datasety, modele czy transormacje z dziedziny wizji komputerowej.\n", 27 | "\n", 28 | "torch.utils.data.Dataloader - klasa, która odpowiada za załadowanie zbioru danych w celu przeprowadzenia treningu modeli\n", 29 | "\n", 30 | "matplotlib - biblioteka umożliwiająca wizualizację danych" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": { 37 | "id": "H5zUdRo_agIu" 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "import torch\n", 42 | "import torch.nn as nn\n", 43 | "import torch.optim as optim\n", 44 | "import torchvision\n", 45 | "import torchvision.datasets as datasets\n", 46 | "import torchvision.transforms as tf\n", 47 | "from torch.utils.data import DataLoader\n", 48 | "import matplotlib.pyplot as plt" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": { 54 | "id": "L9qSEZWy-hp8" 55 | }, 56 | "source": [ 57 | "# Definiowanie własnego modelu\n", 58 | "\n", 59 | "PyTorch umożliwia oczywiście definicję swoich własnych modeli. Zostały do tego przygotowane funkcje z wyżej wymienionych modułów.\n", 60 | "\n", 61 | "W celu zdefiniowania własnego modelu sieci neuronowej należy się trzymać kilku reguł\n", 62 | "\n", 63 | "* Klasa naszego modelu musi dziedziczyć po klasie abstrakcyjnej `nn.Module`\n", 64 | "* W tym celu klasa musi implementować metody abstrakcyjne `__init__` oraz `forward`\n", 65 | "* Na początku metody `__init__` wywołujemy konstruktor \"rodzica\".\n", 66 | "\n", 67 | "Dla przykładu w poniższej komórce zdefiniujemy prostą sieć neuronową.\n", 68 | "\n" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "id": "dSyvBxZNbbze" 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "class ExampleModel(nn.Module):\n", 80 | " def __init__(self):\n", 81 | " super().__init__()\n", 82 | "\n", 83 | " self.layer1 = nn.Linear(28*28, 64)\n", 84 | " self.layer2 = nn.Linear(64, 32)\n", 85 | " self.layer3 = nn.Linear(32, 10)\n", 86 | " self.activation = nn.ReLU()\n", 87 | " self.Softmax = nn.Softmax(dim=0)\n", 88 | "\n", 89 | " def forward(self, x):\n", 90 | " out = x.view(-1, 28*28)\n", 91 | " out = self.activation(self.layer1(out))\n", 92 | " out = self.activation(self.layer2(out))\n", 93 | " out = self.Softmax(self.layer3(out))\n", 94 | " return out" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": { 100 | "id": "Pgo9jFEGPqG7" 101 | }, 102 | "source": [ 103 | "Powyższy model jest typowym przykładem sieci typu Feed Forward, nazywanym również Multilayer Perceptron (MLP).\n", 104 | "\n", 105 | "Zaimplementowana przez nas sieć składa się z 3 warst liniowych:\n", 106 | "* warstwy wejściowej\n", 107 | "* warstwy ukrytej\n", 108 | "* warstwy wyjściowej\n", 109 | "\n", 110 | "Pierwsze dwie warstwy aktywowane są za pomocą funkcji ReLu, natomiast warstwa wyjściowa korzysta z funkcji Softmax.\n", 111 | "Zgodnie z zasadą sieci MLP, rozmiar danych wejściowych do warstwy **L** musi być równy rozmiarowi warstwy wyjściowej **L - 1** np.\n", 112 | "\n", 113 | "```\n", 114 | " self.layer1 = nn.Linear(28*28, 64)\n", 115 | " self.layer2 = nn.Linear(64, 32)\n", 116 | "```" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": { 122 | "id": "i6DSXPn1PqG8" 123 | }, 124 | "source": [ 125 | "### Funkcja aktywacji ReLu\n", 126 | "\n", 127 | "\n", 128 | "Funkcja ReLu obliczana jest zgodnie ze wzorem\n", 129 | "\n", 130 | "$$\n", 131 | " ReLu(x) = max(x, 0)\n", 132 | "$$\n", 133 | "\n", 134 | "i jej celem jest zapewnienie, że pochodna liczona przez metodę Gradientu Prostego będzie przyjmowała 2 wartości: 0 i 1.\n", 135 | "Pozwala to na zapobieżenie zjawisku zanikających gradientów." 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": { 141 | "id": "OBR7Oa8DPqG8" 142 | }, 143 | "source": [ 144 | "### Funkcja Softmax\n", 145 | "\n", 146 | "Celem funkcji **Softmax** jest zapewnienie, aby wartości podanego wektora sumowały się do jedności." 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": { 152 | "id": "RLJgdw6hPqG8" 153 | }, 154 | "source": [ 155 | "Wracając do kodu:\n", 156 | "\n", 157 | "W konstruktorze klasy `__init__` definiowana jest architektura sieci:\n", 158 | "```\n", 159 | " self.layer1 = nn.Linear(28*28, 64)\n", 160 | " self.layer2 = nn.Linear(64, 32)\n", 161 | " self.layer3 = nn.Linear(32, 10)\n", 162 | " self.activation = nn.ReLU()\n", 163 | " self.Softmax = nn.Softmax(dim=0)\n", 164 | "```\n", 165 | "\n", 166 | "natomiast w funkcji `forward()` implementowany jest przepływ danych przez sieć." 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": { 172 | "id": "prCWK7saPqG9" 173 | }, 174 | "source": [ 175 | "W poniższym kodzie możemy zauważyć inny sposób definiowania sieci neuronowej. W tym celu wykorzystana została klasa `Sequential()` która odpowiada za implementację tzw. mdoelu sekwencyjnego, czyli takiego, w którym przepływ danych może się odbywać jedynie według zdefiniowanej w konstruktorze sekwencji." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": null, 181 | "metadata": { 182 | "id": "RkxAV8WTPqG9" 183 | }, 184 | "outputs": [], 185 | "source": [ 186 | "class ExampleModel2(nn.Module):\n", 187 | " def __init__(self):\n", 188 | " super().__init__()\n", 189 | " self.net = nn.Sequential(\n", 190 | " nn.Linear(28*28, 128),\n", 191 | " nn.ReLU(),\n", 192 | " nn.Linear(128, 64),\n", 193 | " nn.ReLU(),\n", 194 | " nn.Linear(64, 10),\n", 195 | " nn.Softmax()\n", 196 | " )\n", 197 | "\n", 198 | " def forward(self, x):\n", 199 | " out = x.view(-1, 28*28)\n", 200 | " return self.net(out)\n" 201 | ] 202 | }, 203 | { 204 | "cell_type": "markdown", 205 | "metadata": { 206 | "id": "INSxBTJwF8H3" 207 | }, 208 | "source": [ 209 | "# Hiperparametry\n", 210 | "\n", 211 | "Do szybszego uczenia się modelu stosujemy parametry takie jak \"szybkość dostosywania wag\".\n", 212 | "\n", 213 | "Proces treningu modelu sieci neuronowej jest definiowany za pomocą tzw. hiperparametrów.\n", 214 | "\n", 215 | "Do hiperparametrów zaliczamy takie zmienne jak:\n", 216 | "- wielkość kroku uczenia się *(ang. learning rate)*\n", 217 | "- wielkość batcha, czyli tzw. paczki danych*\n", 218 | "- ilość epok treningu, czyli kolejnych iteracji po zbiorze treningowym\n", 219 | "- jak mogą zmieniać się wagi, na przykład, że nie mogą przekraczać wyznaczonej wartości\n", 220 | "- rozmiar wyjściowego obrazu\n", 221 | "\n", 222 | "Zestaw hiperparametrów będzie się różnić w zależności od trenowanego modelu, ale najcześciej występującymu są właśnie wielkość kroku uczenia się, rozmiar batcha i liczba epok treningu.\n", 223 | "\n", 224 | "**paczka (batch) to zestaw danych wejściowych które zostaną wprowadzone na wejście do modelu/sieci neuronowej, rozmiar batcha oznacza ile danych wejściowych dostanie przetworzonych zanim model zaktualizuje wagi*" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "metadata": { 231 | "id": "4w2jkToqdRcp" 232 | }, 233 | "outputs": [], 234 | "source": [ 235 | "learning_rate = 1e-2\n", 236 | "batch_size = 128\n", 237 | "num_epochs = 20" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": { 243 | "id": "b3J0ehN5HTHE" 244 | }, 245 | "source": [ 246 | "# Inicjalizacja modelu\n", 247 | "\n", 248 | "Zanim zaczniemy trening musimy określić na jakim urządzeniu będą realizowane obliczenia. Możemy wybrać jedną z dwóch opcji - trenować model za pomocą naszej jednostki GPU (ang. graphics processing unit) potocznie nazywaną kartą graficzną albo na wykonywać obliczenia za pomocą CPU (ang. central processing unit). Do obsługi karty graficznej potrzebny jest sterownik CUDA (ang. Compute Unified Device Architecture) jeśli korzysta z kart firmy NVIDIA lub ROCm (ang. Radeon Open Compute platforM) w przypadku sprzętu firmy AMD.\n", 249 | "\n", 250 | "\n", 251 | "Pierwsza linijka kodu sprawdza czy oprogramowanie cuda jest dostępne i można go używać, a jeżeli nie, to wykorzystywany jest CPU.\n", 252 | "\n", 253 | "W następnej linicje kodu tworzony jest obiekt naszej klasy modelu i lokowany jest on naszym urządzeniu.\n", 254 | "\n", 255 | "W kolejnych dwóch linijkach ustalamy sposób liczenia funkcji straty oraz algorytm optymalizacji wag modelu.\n", 256 | "\n", 257 | "(Kilka słów tłumaczących jakie funkcje strat służą do jakich modeli oraz czym mogą się różnić poszczególne optymalizatory)\n", 258 | "\n", 259 | "\n", 260 | "Przykładowe funkcje strat i ich użycia:\n", 261 | "- MSE (mean square error) - używane do regresji liniowej\n", 262 | "$$\n", 263 | "MSE = \\frac{1}{n} ∑_{i=1}^{n}(y_i - ŷ_i)^2\n", 264 | "$$\n", 265 | "\n", 266 | "- L1 loss (mean absolute error) - używane do regresji liniowej\n", 267 | "\n", 268 | "$$\n", 269 | "MAE = \\frac{1}{n} ∑_{i=1}^{n}|y_i - ŷ_i|\n", 270 | "$$\n", 271 | "\n", 272 | "- cross entropy loss - używane do klasyfikacji (dwie klasy lub wiele klas)\n", 273 | "\n", 274 | "$$\n", 275 | "BinaryCrossEntropyLoss = -\\frac{1}{n}∑_{i=1}^{n}(log(ŷ_i) + (1 - y_i)log(1-ŷ_i))\n", 276 | "$$\n", 277 | "\n", 278 | "$$\n", 279 | "CrossEntropyLoss = -\\frac{1}{n}∑_{i=1}^{n}(y_i * log(ŷ_i))\n", 280 | "$$\n", 281 | "\n", 282 | "Optimizery:\n", 283 | "\n", 284 | "- SGD (Stochastic gradient descent) - aktualizuje wagi sieci na podstawie wyliczonego gradientu i learning rate'u.\n", 285 | "\n", 286 | "- Adam - aktualizuje wagi sieci na podstawie informacji o poprzednich gradientach, wyliczonego gradientu i learning rate'u.\n", 287 | "https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam" 288 | ] 289 | }, 290 | { 291 | "cell_type": "code", 292 | "execution_count": null, 293 | "metadata": { 294 | "id": "UC6_fPk7ckM0" 295 | }, 296 | "outputs": [], 297 | "source": [ 298 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 299 | "\n", 300 | "model = ExampleModel2().to(device)\n", 301 | "\n", 302 | "criterion = nn.CrossEntropyLoss()\n", 303 | "optimizer = optim.SGD(model.parameters(), lr=learning_rate)" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "source": [ 309 | "print(device)" 310 | ], 311 | "metadata": { 312 | "colab": { 313 | "base_uri": "https://localhost:8080/" 314 | }, 315 | "id": "MUPh_Jc3Bcaz", 316 | "outputId": "9160a12d-4f65-4985-e851-beeef45e9330" 317 | }, 318 | "execution_count": null, 319 | "outputs": [ 320 | { 321 | "output_type": "stream", 322 | "name": "stdout", 323 | "text": [ 324 | "cpu\n" 325 | ] 326 | } 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "metadata": { 332 | "id": "ci3pa77aIzjw" 333 | }, 334 | "source": [ 335 | "# Dane treningowe i testowe\n", 336 | "\n", 337 | "Podczas procesu uczenia modelu wyróżniamy 3 typy danych biorących udział w tym procesie:\n", 338 | "- dane treningowe - na nich uczony jest model\n", 339 | "- dane walidacyjne - na nich walidowany jest co epokę model i testowane jest, czy model nie zaczyna być przeuczony\n", 340 | "- dane testowe - finalnie wykorzystane do testu modelu, który miał najlepszy wynik na zbiorze walidacyjnym.\n", 341 | "\n", 342 | "Sposób podziału zbioru danych na powyższe podzbiory zależy od zadania, samego zbioru i założenia, jakie przyjął autor.\n", 343 | "\n", 344 | "Zanim zaczniemy trenować nasz model musimy przeprowadzić transformację danych. Transfromacja danych służy zmianie struktury danych do postaci wykorzystywanej przez modele w bibliotece torch.\n", 345 | "\n", 346 | "W tym celu korzystamy z klasy Compose z biblioteki torchvision, definiując pakiet transformacji naszych danych. W poniższym kodzie jest to:\n", 347 | "- przekształcenie formatu danych do klasy torch.Tensor()\n", 348 | "- stosujemy normalizacje, czyli przetwarzamy dane do wartości z przedziału [0, 1], aby zapewnić większą stabilność obliczeniową podczas treningu\n", 349 | "\n", 350 | "W tym przykładzie wykorzystamy zbiór danych MNIST, dostępny w module torch.datasets wczytując jego podzbiory treningowy i testowy.\n", 351 | "\n", 352 | "W dalszej części kodu tworzymy obiekty klasy DataLoader, służacej do załadowywania batchów danych do treningu modelu." 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": null, 358 | "metadata": { 359 | "id": "sVcePhToeSlW" 360 | }, 361 | "outputs": [], 362 | "source": [ 363 | "transforms = tf.Compose([\n", 364 | " tf.ToTensor(),\n", 365 | " tf.Normalize((0.5,), (0.5,))\n", 366 | "])\n", 367 | "\n", 368 | "train_dataset = datasets.MNIST(root=\"./dataset\", train=True, transform=transforms,download=True)\n", 369 | "test_dataset = datasets.MNIST(root=\"./dataset\", train=False, transform=transforms,download=True)\n", 370 | "\n", 371 | "train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)\n", 372 | "test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)\n" 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": { 379 | "colab": { 380 | "base_uri": "https://localhost:8080/", 381 | "height": 482 382 | }, 383 | "id": "t9xZralXd6al", 384 | "outputId": "37d20534-9dd8-4b71-a658-15b74e650a1a" 385 | }, 386 | "outputs": [ 387 | { 388 | "output_type": "stream", 389 | "name": "stdout", 390 | "text": [ 391 | "(28, 28)\n" 392 | ] 393 | }, 394 | { 395 | "output_type": "stream", 396 | "name": "stderr", 397 | "text": [ 398 | "/usr/local/lib/python3.10/dist-packages/torchvision/datasets/mnist.py:75: UserWarning: train_data has been renamed data\n", 399 | " warnings.warn(\"train_data has been renamed data\")\n" 400 | ] 401 | }, 402 | { 403 | "output_type": "display_data", 404 | "data": { 405 | "text/plain": [ 406 | "
" 407 | ], 408 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAbe0lEQVR4nO3df2xV9f3H8dflR6+I7e1KbW8rPyygsIlgxqDrVMRRKd1G5McWdS7BzWhwrRGYuNRM0W2uDqczbEz5Y4GxCSjJgEEWNi22ZLNgQBgxbg0l3VpGWyZb7y2FFmw/3z+I98uVFjyXe/u+vTwfySeh955378fjtU9vezn1OeecAADoZ4OsNwAAuDIRIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYGKI9QY+qaenR8eOHVN6erp8Pp/1dgAAHjnn1N7ervz8fA0a1PfrnKQL0LFjxzRq1CjrbQAALlNTU5NGjhzZ5/1J9y249PR06y0AAOLgUl/PExag1atX6/rrr9dVV12lwsJCvfvuu59qjm+7AUBquNTX84QE6PXXX9eyZcu0YsUKvffee5oyZYpKSkp0/PjxRDwcAGAgcgkwffp0V1ZWFvm4u7vb5efnu8rKykvOhkIhJ4nFYrFYA3yFQqGLfr2P+yugM2fOaP/+/SouLo7cNmjQIBUXF6u2tvaC47u6uhQOh6MWACD1xT1AH374obq7u5Wbmxt1e25urlpaWi44vrKyUoFAILJ4BxwAXBnM3wVXUVGhUCgUWU1NTdZbAgD0g7j/PaDs7GwNHjxYra2tUbe3trYqGAxecLzf75ff74/3NgAASS7ur4DS0tI0depUVVVVRW7r6elRVVWVioqK4v1wAIABKiFXQli2bJkWLVqkL3zhC5o+fbpefvlldXR06Nvf/nYiHg4AMAAlJED33HOP/vOf/+jpp59WS0uLbrnlFu3cufOCNyYAAK5cPuecs97E+cLhsAKBgPU2AACXKRQKKSMjo8/7zd8FBwC4MhEgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmhlhvAEgmgwcP9jwTCAQSsJP4KC8vj2nu6quv9jwzYcIEzzNlZWWeZ372s595nrnvvvs8z0hSZ2en55nnn3/e88yzzz7reSYV8AoIAGCCAAEATMQ9QM8884x8Pl/UmjhxYrwfBgAwwCXkZ0A33XST3nrrrf9/kCH8qAkAEC0hZRgyZIiCwWAiPjUAIEUk5GdAhw8fVn5+vsaOHav7779fjY2NfR7b1dWlcDgctQAAqS/uASosLNS6deu0c+dOvfLKK2poaNDtt9+u9vb2Xo+vrKxUIBCIrFGjRsV7SwCAJBT3AJWWluob3/iGJk+erJKSEv3xj39UW1ub3njjjV6Pr6ioUCgUiqympqZ4bwkAkIQS/u6AzMxM3Xjjjaqvr+/1fr/fL7/fn+htAACSTML/HtDJkyd15MgR5eXlJfqhAAADSNwD9Pjjj6umpkb//Oc/9c4772j+/PkaPHhwzJfCAACkprh/C+7o0aO67777dOLECV177bW67bbbtGfPHl177bXxfigAwAAW9wBt2rQp3p8SSWr06NGeZ9LS0jzPfOlLX/I8c9ttt3mekc79zNKrhQsXxvRYqebo0aOeZ1atWuV5Zv78+Z5n+noX7qX87W9/8zxTU1MT02NdibgWHADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgwuecc9abOF84HFYgELDexhXllltuiWlu165dnmf4dzsw9PT0eJ75zne+43nm5MmTnmdi0dzcHNPc//73P88zdXV1MT1WKgqFQsrIyOjzfl4BAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwMQQ6w3AXmNjY0xzJ06c8DzD1bDP2bt3r+eZtrY2zzN33nmn5xlJOnPmjOeZ3/72tzE9Fq5cvAICAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAExwMVLov//9b0xzy5cv9zzzta99zfPMgQMHPM+sWrXK80ysDh486Hnmrrvu8jzT0dHheeamm27yPCNJjz32WExzgBe8AgIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATPicc856E+cLh8MKBALW20CCZGRkeJ5pb2/3PLNmzRrPM5L04IMPep751re+5Xlm48aNnmeAgSYUCl30v3leAQEATBAgAIAJzwHavXu35s6dq/z8fPl8Pm3dujXqfuecnn76aeXl5WnYsGEqLi7W4cOH47VfAECK8Bygjo4OTZkyRatXr+71/pUrV2rVqlV69dVXtXfvXg0fPlwlJSXq7Oy87M0CAFKH59+IWlpaqtLS0l7vc87p5Zdf1g9+8APdfffdkqT169crNzdXW7du1b333nt5uwUApIy4/gyooaFBLS0tKi4ujtwWCARUWFio2traXme6uroUDoejFgAg9cU1QC0tLZKk3NzcqNtzc3Mj931SZWWlAoFAZI0aNSqeWwIAJCnzd8FVVFQoFApFVlNTk/WWAAD9IK4BCgaDkqTW1tao21tbWyP3fZLf71dGRkbUAgCkvrgGqKCgQMFgUFVVVZHbwuGw9u7dq6Kiong+FABggPP8LriTJ0+qvr4+8nFDQ4MOHjyorKwsjR49WkuWLNGPf/xj3XDDDSooKNBTTz2l/Px8zZs3L577BgAMcJ4DtG/fPt15552Rj5ctWyZJWrRokdatW6cnnnhCHR0devjhh9XW1qbbbrtNO3fu1FVXXRW/XQMABjwuRoqU9MILL8Q09/H/UHlRU1Pjeeb8v6rwafX09HieASxxMVIAQFIiQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACa6GjZQ0fPjwmOa2b9/ueeaOO+7wPFNaWup55s9//rPnGcASV8MGACQlAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEFyMFzjNu3DjPM++9957nmba2Ns8zb7/9tueZffv2eZ6RpNWrV3ueSbIvJUgCXIwUAJCUCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATXIwUuEzz58/3PLN27VrPM+np6Z5nYvXkk096nlm/fr3nmebmZs8zGDi4GCkAICkRIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACa4GClgYNKkSZ5nXnrpJc8zs2bN8jwTqzVr1nieee655zzP/Pvf//Y8AxtcjBQAkJQIEADAhOcA7d69W3PnzlV+fr58Pp+2bt0adf8DDzwgn88XtebMmROv/QIAUoTnAHV0dGjKlClavXp1n8fMmTNHzc3NkbVx48bL2iQAIPUM8TpQWlqq0tLSix7j9/sVDAZj3hQAIPUl5GdA1dXVysnJ0YQJE/TII4/oxIkTfR7b1dWlcDgctQAAqS/uAZozZ47Wr1+vqqoq/fSnP1VNTY1KS0vV3d3d6/GVlZUKBAKRNWrUqHhvCQCQhDx/C+5S7r333sifb775Zk2ePFnjxo1TdXV1r38noaKiQsuWLYt8HA6HiRAAXAES/jbssWPHKjs7W/X19b3e7/f7lZGREbUAAKkv4QE6evSoTpw4oby8vEQ/FABgAPH8LbiTJ09GvZppaGjQwYMHlZWVpaysLD377LNauHChgsGgjhw5oieeeELjx49XSUlJXDcOABjYPAdo3759uvPOOyMff/zzm0WLFumVV17RoUOH9Jvf/EZtbW3Kz8/X7Nmz9aMf/Uh+vz9+uwYADHhcjBQYIDIzMz3PzJ07N6bHWrt2recZn8/neWbXrl2eZ+666y7PM7DBxUgBAEmJAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJrgaNoALdHV1eZ4ZMsTzb3fRRx995Hkmlt8tVl1d7XkGl4+rYQMAkhIBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYML71QMBXLbJkyd7nvn617/ueWbatGmeZ6TYLiwaiw8++MDzzO7duxOwE1jgFRAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIKLkQLnmTBhgueZ8vJyzzMLFizwPBMMBj3P9Kfu7m7PM83NzZ5nenp6PM8gOfEKCABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwwcVIkfRiuQjnfffdF9NjxXJh0euvvz6mx0pm+/bt8zzz3HPPeZ75wx/+4HkGqYNXQAAAEwQIAGDCU4AqKys1bdo0paenKycnR/PmzVNdXV3UMZ2dnSorK9OIESN0zTXXaOHChWptbY3rpgEAA5+nANXU1KisrEx79uzRm2++qbNnz2r27Nnq6OiIHLN06VJt375dmzdvVk1NjY4dOxbTL98CAKQ2T29C2LlzZ9TH69atU05Ojvbv368ZM2YoFArp17/+tTZs2KAvf/nLkqS1a9fqs5/9rPbs2aMvfvGL8ds5AGBAu6yfAYVCIUlSVlaWJGn//v06e/asiouLI8dMnDhRo0ePVm1tba+fo6urS+FwOGoBAFJfzAHq6enRkiVLdOutt2rSpEmSpJaWFqWlpSkzMzPq2NzcXLW0tPT6eSorKxUIBCJr1KhRsW4JADCAxBygsrIyvf/++9q0adNlbaCiokKhUCiympqaLuvzAQAGhpj+Imp5ebl27Nih3bt3a+TIkZHbg8Ggzpw5o7a2tqhXQa2trX3+ZUK/3y+/3x/LNgAAA5inV0DOOZWXl2vLli3atWuXCgoKou6fOnWqhg4dqqqqqshtdXV1amxsVFFRUXx2DABICZ5eAZWVlWnDhg3atm2b0tPTIz/XCQQCGjZsmAKBgB588EEtW7ZMWVlZysjI0KOPPqqioiLeAQcAiOIpQK+88ookaebMmVG3r127Vg888IAk6ec//7kGDRqkhQsXqqurSyUlJfrVr34Vl80CAFKHzznnrDdxvnA4rEAgYL0NfAq5ubmeZz73uc95nvnlL3/peWbixImeZ5Ld3r17Pc+88MILMT3Wtm3bPM/09PTE9FhIXaFQSBkZGX3ez7XgAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYCKm34iK5JWVleV5Zs2aNTE91i233OJ5ZuzYsTE9VjJ75513PM+8+OKLnmf+9Kc/eZ45ffq05xmgv/AKCABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwwcVI+0lhYaHnmeXLl3uemT59uueZ6667zvNMsjt16lRMc6tWrfI885Of/MTzTEdHh+cZINXwCggAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMMHFSPvJ/Pnz+2WmP33wwQeeZ3bs2OF55qOPPvI88+KLL3qekaS2traY5gB4xysgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMCEzznnrDdxvnA4rEAgYL0NAMBlCoVCysjI6PN+XgEBAEwQIACACU8Bqqys1LRp05Senq6cnBzNmzdPdXV1UcfMnDlTPp8vai1evDiumwYADHyeAlRTU6OysjLt2bNHb775ps6ePavZs2ero6Mj6riHHnpIzc3NkbVy5cq4bhoAMPB5+o2oO3fujPp43bp1ysnJ0f79+zVjxozI7VdffbWCwWB8dggASEmX9TOgUCgkScrKyoq6/bXXXlN2drYmTZqkiooKnTp1qs/P0dXVpXA4HLUAAFcAF6Pu7m731a9+1d16661Rt69Zs8bt3LnTHTp0yP3ud79z1113nZs/f36fn2fFihVOEovFYrFSbIVCoYt2JOYALV682I0ZM8Y1NTVd9LiqqionydXX1/d6f2dnpwuFQpHV1NRkftJYLBaLdfnrUgHy9DOgj5WXl2vHjh3avXu3Ro4cedFjCwsLJUn19fUaN27cBff7/X75/f5YtgEAGMA8Bcg5p0cffVRbtmxRdXW1CgoKLjlz8OBBSVJeXl5MGwQApCZPASorK9OGDRu0bds2paenq6WlRZIUCAQ0bNgwHTlyRBs2bNBXvvIVjRgxQocOHdLSpUs1Y8YMTZ48OSH/AACAAcrLz33Ux/f51q5d65xzrrGx0c2YMcNlZWU5v9/vxo8f75YvX37J7wOeLxQKmX/fksVisViXvy71tZ+LkQIAEoKLkQIAkhIBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwETSBcg5Z70FAEAcXOrredIFqL293XoLAIA4uNTXc59LspccPT09OnbsmNLT0+Xz+aLuC4fDGjVqlJqampSRkWG0Q3uch3M4D+dwHs7hPJyTDOfBOaf29nbl5+dr0KC+X+cM6cc9fSqDBg3SyJEjL3pMRkbGFf0E+xjn4RzOwzmch3M4D+dYn4dAIHDJY5LuW3AAgCsDAQIAmBhQAfL7/VqxYoX8fr/1VkxxHs7hPJzDeTiH83DOQDoPSfcmBADAlWFAvQICAKQOAgQAMEGAAAAmCBAAwMSACdDq1at1/fXX66qrrlJhYaHeffdd6y31u2eeeUY+ny9qTZw40XpbCbd7927NnTtX+fn58vl82rp1a9T9zjk9/fTTysvL07Bhw1RcXKzDhw/bbDaBLnUeHnjggQueH3PmzLHZbIJUVlZq2rRpSk9PV05OjubNm6e6urqoYzo7O1VWVqYRI0bommuu0cKFC9Xa2mq048T4NOdh5syZFzwfFi9ebLTj3g2IAL3++utatmyZVqxYoffee09TpkxRSUmJjh8/br21fnfTTTepubk5sv7yl79YbynhOjo6NGXKFK1evbrX+1euXKlVq1bp1Vdf1d69ezV8+HCVlJSos7Ozn3eaWJc6D5I0Z86cqOfHxo0b+3GHiVdTU6OysjLt2bNHb775ps6ePavZs2ero6MjcszSpUu1fft2bd68WTU1NTp27JgWLFhguOv4+zTnQZIeeuihqOfDypUrjXbcBzcATJ8+3ZWVlUU+7u7udvn5+a6ystJwV/1vxYoVbsqUKdbbMCXJbdmyJfJxT0+PCwaD7oUXXojc1tbW5vx+v9u4caPBDvvHJ8+Dc84tWrTI3X333Sb7sXL8+HEnydXU1Djnzv27Hzp0qNu8eXPkmL///e9OkqutrbXaZsJ98jw459wdd9zhHnvsMbtNfQpJ/wrozJkz2r9/v4qLiyO3DRo0SMXFxaqtrTXcmY3Dhw8rPz9fY8eO1f3336/GxkbrLZlqaGhQS0tL1PMjEAiosLDwinx+VFdXKycnRxMmTNAjjzyiEydOWG8poUKhkCQpKytLkrR//36dPXs26vkwceJEjR49OqWfD588Dx977bXXlJ2drUmTJqmiokKnTp2y2F6fku5ipJ/04Ycfqru7W7m5uVG35+bm6h//+IfRrmwUFhZq3bp1mjBhgpqbm/Xss8/q9ttv1/vvv6/09HTr7ZloaWmRpF6fHx/fd6WYM2eOFixYoIKCAh05ckRPPvmkSktLVVtbq8GDB1tvL+56enq0ZMkS3XrrrZo0aZKkc8+HtLQ0ZWZmRh2bys+H3s6DJH3zm9/UmDFjlJ+fr0OHDun73/++6urq9Pvf/95wt9GSPkD4f6WlpZE/T548WYWFhRozZozeeOMNPfjgg4Y7QzK49957I3+++eabNXnyZI0bN07V1dWaNWuW4c4So6ysTO+///4V8XPQi+nrPDz88MORP998883Ky8vTrFmzdOTIEY0bN66/t9mrpP8WXHZ2tgYPHnzBu1haW1sVDAaNdpUcMjMzdeONN6q+vt56K2Y+fg7w/LjQ2LFjlZ2dnZLPj/Lycu3YsUNvv/121K9vCQaDOnPmjNra2qKOT9XnQ1/noTeFhYWSlFTPh6QPUFpamqZOnaqqqqrIbT09PaqqqlJRUZHhzuydPHlSR44cUV5envVWzBQUFCgYDEY9P8LhsPbu3XvFPz+OHj2qEydOpNTzwzmn8vJybdmyRbt27VJBQUHU/VOnTtXQoUOjng91dXVqbGxMqefDpc5Dbw4ePChJyfV8sH4XxKexadMm5/f73bp169wHH3zgHn74YZeZmelaWlqst9avvve977nq6mrX0NDg/vrXv7ri4mKXnZ3tjh8/br21hGpvb3cHDhxwBw4ccJLcSy+95A4cOOD+9a9/Oeece/75511mZqbbtm2bO3TokLv77rtdQUGBO336tPHO4+ti56G9vd09/vjjrra21jU0NLi33nrLff7zn3c33HCD6+zstN563DzyyCMuEAi46upq19zcHFmnTp2KHLN48WI3evRot2vXLrdv3z5XVFTkioqKDHcdf5c6D/X19e6HP/yh27dvn2toaHDbtm1zY8eOdTNmzDDeebQBESDnnPvFL37hRo8e7dLS0tz06dPdnj17rLfU7+655x6Xl5fn0tLS3HXXXefuueceV19fb72thHv77bedpAvWokWLnHPn3or91FNPudzcXOf3+92sWbNcXV2d7aYT4GLn4dSpU2727Nnu2muvdUOHDnVjxoxxDz30UMr9T1pv//yS3Nq1ayPHnD592n33u991n/nMZ9zVV1/t5s+f75qbm+02nQCXOg+NjY1uxowZLisry/n9fjd+/Hi3fPlyFwqFbDf+Cfw6BgCAiaT/GRAAIDURIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACb+Dwuo74MxItlsAAAAAElFTkSuQmCC\n" 409 | }, 410 | "metadata": {} 411 | } 412 | ], 413 | "source": [ 414 | "print(train_dataset.train_data[0].numpy().shape)\n", 415 | "plt.imshow(train_dataset.train_data[0].numpy(), cmap='gray')\n", 416 | "plt.show()" 417 | ] 418 | }, 419 | { 420 | "cell_type": "markdown", 421 | "metadata": { 422 | "id": "zdvZuX9SQEBL" 423 | }, 424 | "source": [ 425 | "# Pętla treningowa\n", 426 | "\n", 427 | "Definiujemy pętlę for iterującą po każdej epoce treningu, wewnątrz której iterujemy po paczkach (batches) danych treningowych i w każdej iteracji:\n", 428 | "\n", 429 | "- ładujemy dane wejściowe i etykiety (labels) na urządzenie obliczeniowe\n", 430 | "- model generuje output z danych wejściowych\n", 431 | "- obliczamy różnice między wynikami modelu a etykietami\n", 432 | "- zerujemy gradient który był wcześniej obliczany\n", 433 | "- obliczamy gradient z funkcji straty\n", 434 | "- aktualizujemy wagi w naszym modelu" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": null, 440 | "metadata": { 441 | "id": "nGcRZHmLPqG_", 442 | "outputId": "5fc64985-d31b-4eb4-b29d-b443832af2d4", 443 | "colab": { 444 | "base_uri": "https://localhost:8080/" 445 | } 446 | }, 447 | "outputs": [ 448 | { 449 | "output_type": "stream", 450 | "name": "stdout", 451 | "text": [ 452 | "cpu\n" 453 | ] 454 | } 455 | ], 456 | "source": [ 457 | "print(device)" 458 | ] 459 | }, 460 | { 461 | "cell_type": "code", 462 | "execution_count": null, 463 | "metadata": { 464 | "colab": { 465 | "base_uri": "https://localhost:8080/" 466 | }, 467 | "id": "JsnwIdJlgnKg", 468 | "outputId": "79b3004a-bc48-4c3c-a990-5eb2e96a6b20" 469 | }, 470 | "outputs": [ 471 | { 472 | "output_type": "stream", 473 | "name": "stderr", 474 | "text": [ 475 | "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1518: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n", 476 | " return self._call_impl(*args, **kwargs)\n" 477 | ] 478 | }, 479 | { 480 | "output_type": "stream", 481 | "name": "stdout", 482 | "text": [ 483 | "Epoch: 0/20 | Step: 0/9380 | Loss: 2.3019766807556152\n", 484 | "Epoch: 1/20 | Step: 469/9380 | Loss: 2.292313575744629\n", 485 | "Epoch: 2/20 | Step: 938/9380 | Loss: 2.2423782348632812\n", 486 | "Epoch: 3/20 | Step: 1407/9380 | Loss: 2.079305410385132\n", 487 | "Epoch: 4/20 | Step: 1876/9380 | Loss: 2.0324323177337646\n", 488 | "Epoch: 5/20 | Step: 2345/9380 | Loss: 1.919705867767334\n", 489 | "Epoch: 6/20 | Step: 2814/9380 | Loss: 1.845484733581543\n", 490 | "Epoch: 7/20 | Step: 3283/9380 | Loss: 1.816897988319397\n", 491 | "Epoch: 8/20 | Step: 3752/9380 | Loss: 1.7392539978027344\n", 492 | "Epoch: 9/20 | Step: 4221/9380 | Loss: 1.7449798583984375\n", 493 | "Epoch: 10/20 | Step: 4690/9380 | Loss: 1.7332803010940552\n", 494 | "Epoch: 11/20 | Step: 5159/9380 | Loss: 1.7056034803390503\n", 495 | "Epoch: 12/20 | Step: 5628/9380 | Loss: 1.700762391090393\n", 496 | "Epoch: 13/20 | Step: 6097/9380 | Loss: 1.7639660835266113\n", 497 | "Epoch: 14/20 | Step: 6566/9380 | Loss: 1.7692153453826904\n", 498 | "Epoch: 15/20 | Step: 7035/9380 | Loss: 1.7374157905578613\n", 499 | "Epoch: 16/20 | Step: 7504/9380 | Loss: 1.7156481742858887\n", 500 | "Epoch: 17/20 | Step: 7973/9380 | Loss: 1.737479567527771\n", 501 | "Epoch: 18/20 | Step: 8442/9380 | Loss: 1.6465409994125366\n", 502 | "Epoch: 19/20 | Step: 8911/9380 | Loss: 1.711321234703064\n" 503 | ] 504 | } 505 | ], 506 | "source": [ 507 | "step = 0\n", 508 | "total_steps = len(train_loader) * num_epochs\n", 509 | "\n", 510 | "for epoch in range(num_epochs):\n", 511 | " for batch_idx, (images, labels) in enumerate(train_loader):\n", 512 | " images = images.to(device)\n", 513 | " labels = labels.to(device)\n", 514 | "\n", 515 | " outputs = model(images)\n", 516 | " loss = criterion(outputs, labels)\n", 517 | " #Resets the gradients of all optimized torch.Tensor\n", 518 | " optimizer.zero_grad()\n", 519 | "\n", 520 | " #This is where the model learns by backpropagating\n", 521 | " loss.backward()\n", 522 | " #And optimizes its weights here\n", 523 | " optimizer.step()\n", 524 | "\n", 525 | " if batch_idx % 500 == 0:\n", 526 | " print(f'Epoch: {epoch}/{num_epochs} | Step: {step}/{total_steps} | Loss: {loss}')\n", 527 | " step += 1" 528 | ] 529 | }, 530 | { 531 | "cell_type": "code", 532 | "execution_count": null, 533 | "metadata": { 534 | "id": "zGGGxyO22qQB", 535 | "outputId": "68502d29-03db-4c2c-f8db-7e7b357903ce", 536 | "colab": { 537 | "base_uri": "https://localhost:8080/", 538 | "height": 656 539 | } 540 | }, 541 | "outputs": [ 542 | { 543 | "output_type": "display_data", 544 | "data": { 545 | "text/plain": [ 546 | "
" 547 | ], 548 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAb2UlEQVR4nO3dfWxV9R3H8c/loRfE9rJS29srD5aCsIhgZNJ1aIejoXSbEyQLOrPh4iS44ob4sNUp6KbpxpJpVIb7Y4Jm4gPZACVLN6y2xNliKBBCdB0lda2BFmnGvVBoYfS3P4h3XimUc7m339vyfiW/hHvO+fZ8+Xnaj+few68+55wTAAB9bJB1AwCASxMBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABNDrBv4ou7ubh04cEDp6eny+XzW7QAAPHLO6ejRowqFQho06Nz3OSkXQAcOHNCYMWOs2wAAXKSWlhaNHj36nPtT7i249PR06xYAAAnQ28/zpAXQ6tWrddVVV2nYsGEqKCjQBx98cEF1vO0GAANDbz/PkxJAr7/+upYvX66VK1dq586dmjZtmkpKSnTo0KFknA4A0B+5JJgxY4YrKyuLvj59+rQLhUKuoqKi19pwOOwkMRgMBqOfj3A4fN6f9wm/Azp58qTq6+tVXFwc3TZo0CAVFxertrb2rOO7uroUiURiBgBg4Et4AB0+fFinT59WTk5OzPacnBy1traedXxFRYUCgUB08AQcAFwazJ+CKy8vVzgcjo6WlhbrlgAAfSDh/w4oKytLgwcPVltbW8z2trY2BYPBs473+/3y+/2JbgMAkOISfgeUlpam6dOnq6qqKrqtu7tbVVVVKiwsTPTpAAD9VFJWQli+fLkWLVqkr3zlK5oxY4aeeeYZdXR06Ic//GEyTgcA6IeSEkALFy7Up59+qhUrVqi1tVXXXXedKisrz3owAQBw6fI555x1E58XiUQUCASs2wAAXKRwOKyMjIxz7jd/Cg4AcGkigAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYGKIdQNAKpk1a1afnOfdd9/1XFNZWem5Zs6cOZ5rJOnpp5/2XNPe3h7Xubz685//7LnmX//6VxI6wcXiDggAYIIAAgCYSHgAPf744/L5fDFj8uTJiT4NAKCfS8pnQNdcc43efvvt/59kCB81AQBiJSUZhgwZomAwmIwvDQAYIJLyGdC+ffsUCoU0fvx43XnnnWpubj7nsV1dXYpEIjEDADDwJTyACgoKtG7dOlVWVmrNmjVqamrSTTfdpKNHj/Z4fEVFhQKBQHSMGTMm0S0BAFJQwgOotLRU3/3udzV16lSVlJTor3/9q44cOaI33nijx+PLy8sVDoejo6WlJdEtAQBSUNKfDhg5cqSuvvpqNTY29rjf7/fL7/cnuw0AQIpJ+r8DOnbsmPbv36/c3NxknwoA0I8kPIAefPBB1dTU6OOPP9b777+v+fPna/DgwbrjjjsSfSoAQD+W8LfgPvnkE91xxx1qb2/XFVdcoRtvvFF1dXW64oorEn0qAEA/5nPOOesmPi8SiSgQCFi3cUm5884746r7+c9/7rkm1Z9yTEtL65PzxPO5Z4p9q5pZsGCB55rNmzcnoRP0JhwOKyMj45z7WQsOAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACRYjHWCeeuopzzUPPPBAXOcaMsT7Yuo7d+70XHP99dd7rkl1Pp/Pc02Kfaua+eCDDzzXfO1rX0tCJ+gNi5ECAFISAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMCE9+WMkdK+//3ve66JZ1XreH300UeeayZOnOi5pqqqynNNvP7+9797rhk2bJjnmr/97W+ea/rSihUrPNcsXLjQc811113nueY73/mO5xpJevPNN+Oqw4XhDggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJn3POWTfxeZFIRIFAwLqNfqu5udlzTSgUSkInPVu6dKnnmvfee89zzd69ez3X4OJkZGR4rrn66quT0MnZPv7447jqDh8+nNhGLjHhcPi81wV3QAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwMsW4AifXiiy96rnn00UeT0EnPGhoaPNewsGj/EIlEPNfs2LEjCZ2gv+AOCABgggACAJjwHEDbtm3TLbfcolAoJJ/Pp02bNsXsd85pxYoVys3N1fDhw1VcXKx9+/Ylql8AwADhOYA6Ojo0bdo0rV69usf9q1at0rPPPqsXXnhB27dv14gRI1RSUqLOzs6LbhYAMHB4fgihtLRUpaWlPe5zzumZZ57Ro48+qltvvVWS9PLLLysnJ0ebNm3S7bfffnHdAgAGjIR+BtTU1KTW1lYVFxdHtwUCARUUFKi2trbHmq6uLkUikZgBABj4EhpAra2tkqScnJyY7Tk5OdF9X1RRUaFAIBAdY8aMSWRLAIAUZf4UXHl5ucLhcHS0tLRYtwQA6AMJDaBgMChJamtri9ne1tYW3fdFfr9fGRkZMQMAMPAlNIDy8vIUDAZVVVUV3RaJRLR9+3YVFhYm8lQAgH7O81Nwx44dU2NjY/R1U1OTdu/erczMTI0dO1bLli3Tk08+qYkTJyovL0+PPfaYQqGQ5s2bl8i+AQD9nOcA2rFjh26++ebo6+XLl0uSFi1apHXr1unhhx9WR0eHFi9erCNHjujGG29UZWWlhg0blriuAQD9ns8556yb+LxIJKJAIGDdRr/1gx/8wHNNPAuYxutHP/qR55q6urokdNKzTz/9tE/O097e3ifnASyFw+Hzfq5v/hQcAODSRAABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwwWrYA0x+fr7nmjfffDOuc02aNCmuulS2a9euPjnPhx9+6LmmtrbWc83rr7/uuUaS/vOf/8RVB3weq2EDAFISAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEyxGCs2fPz+uug0bNiS4k0uHz+fzXBPPt2pDQ4PnGkl6/vnnPde89NJLnmuOHz/uuQb9B4uRAgBSEgEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABNDrBsAkDyTJk2Kq+65557zXDNv3jzPNSUlJZ5rMHBwBwQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMCEzznnrJv4vEgkokAgYN0GLsCECRM81zzyyCOea7797W97rnnttdc810jSxIkTPddkZGR4rjl58qTnmqKiIs81qe7JJ5/0XLNy5cokdIJkCIfD5/3+4A4IAGCCAAIAmPAcQNu2bdMtt9yiUCgkn8+nTZs2xey/66675PP5YsbcuXMT1S8AYIDwHEAdHR2aNm2aVq9efc5j5s6dq4MHD0bHq6++elFNAgAGHs+/EbW0tFSlpaXnPcbv9ysYDMbdFABg4EvKZ0DV1dXKzs7WpEmTdO+996q9vf2cx3Z1dSkSicQMAMDAl/AAmjt3rl5++WVVVVXpN7/5jWpqalRaWqrTp0/3eHxFRYUCgUB0jBkzJtEtAQBSkOe34Hpz++23R/987bXXaurUqcrPz1d1dbVmz5591vHl5eVavnx59HUkEiGEAOASkPTHsMePH6+srCw1Njb2uN/v9ysjIyNmAAAGvqQH0CeffKL29nbl5uYm+1QAgH7E81twx44di7mbaWpq0u7du5WZmanMzEw98cQTWrBggYLBoPbv36+HH35YEyZMUElJSUIbBwD0b54DaMeOHbr55pujrz/7/GbRokVas2aN9uzZo5deeklHjhxRKBTSnDlz9Ktf/Up+vz9xXQMA+j0WI0WfGjx4sOeaYcOGea45ceKE5xpJGjLE+3M5Pp/Pc00833ZPPfWU55rFixd7rpGkESNGxFXn1bmejj2fBQsWeK7ZsmWL5xpcPBYjBQCkJAIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACVbDBgawnJycuOq2bdvmuSY/Pz+uc3n15JNPeq55/PHHE98IesVq2ACAlEQAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMDEEOsGACRPW1tbXHX//e9/E9xJ4jz88MOea3bu3BnXud5888246nBhuAMCAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABggsVIgQHsJz/5SVx1Y8eOTXAniXPgwAHPNe+//34SOsHF4g4IAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACRYjRdwCgYDnGuec55pIJOK5Jl5XXXWV55rs7GzPNU899ZTnmnjmbubMmZ5rJMnv98dV1xdOnDjhuebw4cNJ6AQXizsgAIAJAggAYMJTAFVUVOiGG25Qenq6srOzNW/ePDU0NMQc09nZqbKyMo0aNUqXX365FixYoLa2toQ2DQDo/zwFUE1NjcrKylRXV6etW7fq1KlTmjNnjjo6OqLH3H///Xrrrbe0YcMG1dTU6MCBA7rtttsS3jgAoH/z9BBCZWVlzOt169YpOztb9fX1KioqUjgc1h//+EetX79e3/jGNyRJa9eu1Ze//GXV1dXpq1/9auI6BwD0axf1GVA4HJYkZWZmSpLq6+t16tQpFRcXR4+ZPHmyxo4dq9ra2h6/RldXlyKRSMwAAAx8cQdQd3e3li1bppkzZ2rKlCmSpNbWVqWlpWnkyJExx+bk5Ki1tbXHr1NRUaFAIBAdY8aMibclAEA/EncAlZWVae/evXrttdcuqoHy8nKFw+HoaGlpuaivBwDoH+L6h6hLly7Vli1btG3bNo0ePTq6PRgM6uTJkzpy5EjMXVBbW5uCwWCPX8vv96f0P3oDACSHpzsg55yWLl2qjRs36p133lFeXl7M/unTp2vo0KGqqqqKbmtoaFBzc7MKCwsT0zEAYEDwdAdUVlam9evXa/PmzUpPT49+rhMIBDR8+HAFAgHdfffdWr58uTIzM5WRkaH77rtPhYWFPAEHAIjhKYDWrFkjSZo1a1bM9rVr1+quu+6SJD399NMaNGiQFixYoK6uLpWUlOj3v/99QpoFAAwcPhfPCodJFIlE4lrkEn3vxRdf9FyTn5/vuaYvF5KcOnWq55ovvhV9IXw+n+eaFPtWTYh4FhYtLy/3XPP88897rsHFC4fDysjIOOd+1oIDAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJiI6zeiApK0a9cuzzULFy70XMNvzO0furq6PNewsvWljTsgAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJnzOOWfdxOdFIhEFAgHrNpAkmzdv9lwza9YszzUjRozwXNOXfD6f55p4Fvvs7Oz0XBOvX/ziF55r1qxZk4ROkCrC4bAyMjLOuZ87IACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACZYjBQpb9SoUZ5rli1bFte5ysvL46rz6tFHH/VcU19f77lm69atnmuARGExUgBASiKAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCxUgBAEnBYqQAgJREAAEATHgKoIqKCt1www1KT09Xdna25s2bp4aGhphjZs2aJZ/PFzOWLFmS0KYBAP2fpwCqqalRWVmZ6urqtHXrVp06dUpz5sxRR0dHzHH33HOPDh48GB2rVq1KaNMAgP5viJeDKysrY16vW7dO2dnZqq+vV1FRUXT7ZZddpmAwmJgOAQAD0kV9BhQOhyVJmZmZMdtfeeUVZWVlacqUKSovL9fx48fP+TW6uroUiURiBgDgEuDidPr0afetb33LzZw5M2b7H/7wB1dZWen27Nnj/vSnP7krr7zSzZ8//5xfZ+XKlU4Sg8FgMAbYCIfD582RuANoyZIlbty4ca6lpeW8x1VVVTlJrrGxscf9nZ2dLhwOR0dLS4v5pDEYDAbj4kdvAeTpM6DPLF26VFu2bNG2bds0evTo8x5bUFAgSWpsbFR+fv5Z+/1+v/x+fzxtAAD6MU8B5JzTfffdp40bN6q6ulp5eXm91uzevVuSlJubG1eDAICByVMAlZWVaf369dq8ebPS09PV2toqSQoEAho+fLj279+v9evX65vf/KZGjRqlPXv26P7771dRUZGmTp2alL8AAKCf8vK5j87xPt/atWudc841Nze7oqIil5mZ6fx+v5swYYJ76KGHen0f8PPC4bD5+5YMBoPBuPjR289+FiMFACQFi5ECAFISAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMBEygWQc866BQBAAvT28zzlAujo0aPWLQAAEqC3n+c+l2K3HN3d3Tpw4IDS09Pl8/li9kUiEY0ZM0YtLS3KyMgw6tAe83AG83AG83AG83BGKsyDc05Hjx5VKBTSoEHnvs8Z0oc9XZBBgwZp9OjR5z0mIyPjkr7APsM8nME8nME8nME8nGE9D4FAoNdjUu4tOADApYEAAgCY6FcB5Pf7tXLlSvn9futWTDEPZzAPZzAPZzAPZ/SneUi5hxAAAJeGfnUHBAAYOAggAIAJAggAYIIAAgCY6DcBtHr1al111VUaNmyYCgoK9MEHH1i31Ocef/xx+Xy+mDF58mTrtpJu27ZtuuWWWxQKheTz+bRp06aY/c45rVixQrm5uRo+fLiKi4u1b98+m2aTqLd5uOuuu866PubOnWvTbJJUVFTohhtuUHp6urKzszVv3jw1NDTEHNPZ2amysjKNGjVKl19+uRYsWKC2tjajjpPjQuZh1qxZZ10PS5YsMeq4Z/0igF5//XUtX75cK1eu1M6dOzVt2jSVlJTo0KFD1q31uWuuuUYHDx6Mjvfee8+6paTr6OjQtGnTtHr16h73r1q1Ss8++6xeeOEFbd++XSNGjFBJSYk6Ozv7uNPk6m0eJGnu3Lkx18err77ahx0mX01NjcrKylRXV6etW7fq1KlTmjNnjjo6OqLH3H///Xrrrbe0YcMG1dTU6MCBA7rtttsMu068C5kHSbrnnntirodVq1YZdXwOrh+YMWOGKysri74+ffq0C4VCrqKiwrCrvrdy5Uo3bdo06zZMSXIbN26Mvu7u7nbBYND99re/jW47cuSI8/v97tVXXzXosG98cR6cc27RokXu1ltvNenHyqFDh5wkV1NT45w7899+6NChbsOGDdFjPvroIyfJ1dbWWrWZdF+cB+ec+/rXv+5++tOf2jV1AVL+DujkyZOqr69XcXFxdNugQYNUXFys2tpaw85s7Nu3T6FQSOPHj9edd96p5uZm65ZMNTU1qbW1Neb6CAQCKigouCSvj+rqamVnZ2vSpEm699571d7ebt1SUoXDYUlSZmamJKm+vl6nTp2KuR4mT56ssWPHDujr4Yvz8JlXXnlFWVlZmjJlisrLy3X8+HGL9s4p5RYj/aLDhw/r9OnTysnJidmek5Ojf/7zn0Zd2SgoKNC6des0adIkHTx4UE888YRuuukm7d27V+np6dbtmWhtbZWkHq+Pz/ZdKubOnavbbrtNeXl52r9/vx555BGVlpaqtrZWgwcPtm4v4bq7u7Vs2TLNnDlTU6ZMkXTmekhLS9PIkSNjjh3I10NP8yBJ3/ve9zRu3DiFQiHt2bNHP/vZz9TQ0KC//OUvht3GSvkAwv+VlpZG/zx16lQVFBRo3LhxeuONN3T33XcbdoZUcPvtt0f/fO2112rq1KnKz89XdXW1Zs+ebdhZcpSVlWnv3r2XxOeg53OueVi8eHH0z9dee61yc3M1e/Zs7d+/X/n5+X3dZo9S/i24rKwsDR48+KynWNra2hQMBo26Sg0jR47U1VdfrcbGRutWzHx2DXB9nG38+PHKysoakNfH0qVLtWXLFr377rsxv74lGAzq5MmTOnLkSMzxA/V6ONc89KSgoECSUup6SPkASktL0/Tp01VVVRXd1t3draqqKhUWFhp2Zu/YsWPav3+/cnNzrVsxk5eXp2AwGHN9RCIRbd++/ZK/Pj755BO1t7cPqOvDOaelS5dq48aNeuedd5SXlxezf/r06Ro6dGjM9dDQ0KDm5uYBdT30Ng892b17tySl1vVg/RTEhXjttdec3+9369atcx9++KFbvHixGzlypGttbbVurU898MADrrq62jU1Nbl//OMfrri42GVlZblDhw5Zt5ZUR48edbt27XK7du1yktzvfvc7t2vXLvfvf//bOefcr3/9azdy5Ei3efNmt2fPHnfrrbe6vLw8d+LECePOE+t883D06FH34IMPutraWtfU1OTefvttd/3117uJEye6zs5O69YT5t5773WBQMBVV1e7gwcPRsfx48ejxyxZssSNHTvWvfPOO27Hjh2usLDQFRYWGnadeL3NQ2Njo/vlL3/pduzY4ZqamtzmzZvd+PHjXVFRkXHnsfpFADnn3HPPPefGjh3r0tLS3IwZM1xdXZ11S31u4cKFLjc316Wlpbkrr7zSLVy40DU2Nlq3lXTvvvuuk3TWWLRokXPuzKPYjz32mMvJyXF+v9/Nnj3bNTQ02DadBOebh+PHj7s5c+a4K664wg0dOtSNGzfO3XPPPQPuf9J6+vtLcmvXro0ec+LECffjH//YfelLX3KXXXaZmz9/vjt48KBd00nQ2zw0Nze7oqIil5mZ6fx+v5swYYJ76KGHXDgctm38C/h1DAAAEyn/GRAAYGAigAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABg4n/L0w5wQTyhPgAAAABJRU5ErkJggg==\n" 549 | }, 550 | "metadata": {} 551 | }, 552 | { 553 | "output_type": "stream", 554 | "name": "stdout", 555 | "text": [ 556 | "Szansa na:\n", 557 | "0 - 0.0011\n", 558 | "1 - 0.0000\n", 559 | "2 - 0.0000\n", 560 | "3 - 0.7762\n", 561 | "4 - 0.0000\n", 562 | "5 - 0.0000\n", 563 | "6 - 0.0000\n", 564 | "7 - 0.0000\n", 565 | "8 - 0.2227\n", 566 | "9 - 0.0000\n", 567 | "Prawdziwa liczba\n", 568 | "tensor(5)\n" 569 | ] 570 | } 571 | ], 572 | "source": [ 573 | "import random\n", 574 | "i = random.randint(1, batch_size)\n", 575 | "examples = enumerate(test_loader) # używamy danych testowych\n", 576 | "batch_idx, (example_image, example_labels) = next(examples)\n", 577 | "plt.imshow(example_image.numpy()[i][0], cmap='gray')\n", 578 | "plt.show()\n", 579 | "result = model(example_image[i][0])\n", 580 | "print(f'Szansa na:')\n", 581 | "print(f'0 - {result[0][0]:.4f}')\n", 582 | "print(f'1 - {result[0][1]:.4f}')\n", 583 | "print(f'2 - {result[0][2]:.4f}')\n", 584 | "print(f'3 - {result[0][3]:.4f}')\n", 585 | "print(f'4 - {result[0][4]:.4f}')\n", 586 | "print(f'5 - {result[0][5]:.4f}')\n", 587 | "print(f'6 - {result[0][6]:.4f}')\n", 588 | "print(f'7 - {result[0][7]:.4f}')\n", 589 | "print(f'8 - {result[0][8]:.4f}')\n", 590 | "print(f'9 - {result[0][9]:.4f}')\n", 591 | "print('Prawdziwa liczba')\n", 592 | "print(example_labels[i])" 593 | ] 594 | }, 595 | { 596 | "cell_type": "code", 597 | "source": [ 598 | "torch.save(model, './mnist_model.pt')" 599 | ], 600 | "metadata": { 601 | "id": "5DACiECftpKa" 602 | }, 603 | "execution_count": null, 604 | "outputs": [] 605 | } 606 | ], 607 | "metadata": { 608 | "colab": { 609 | "provenance": [] 610 | }, 611 | "kernelspec": { 612 | "display_name": "Python 3 (ipykernel)", 613 | "language": "python", 614 | "name": "python3" 615 | }, 616 | "language_info": { 617 | "codemirror_mode": { 618 | "name": "ipython", 619 | "version": 3 620 | }, 621 | "file_extension": ".py", 622 | "mimetype": "text/x-python", 623 | "name": "python", 624 | "nbconvert_exporter": "python", 625 | "pygments_lexer": "ipython3", 626 | "version": "3.10.6" 627 | } 628 | }, 629 | "nbformat": 4, 630 | "nbformat_minor": 0 631 | } -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Implementation_of_NN/mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Deep_Learning_course/Implementation_of_NN/mlp.png -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Implementation_of_NN/relu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Deep_Learning_course/Implementation_of_NN/relu.png -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Intro_to_Deep_Learning/3. Introduction to Deep Learning.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Deep_Learning_course/Intro_to_Deep_Learning/3. Introduction to Deep Learning.pdf -------------------------------------------------------------------------------- /2023_Edition/Deep_Learning_course/Intro_to_Deep_Learning/main.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Introduction to deep learning with pytorch\n", 8 | "\n", 9 | "## Sources:\n", 10 | "\n", 11 | "https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "# Imports\n", 21 | "\n", 22 | "import torch\n", 23 | "torch.manual_seed(0) # For reproducibility of results\n", 24 | "from torch import nn\n", 25 | "from torch.utils.data import DataLoader\n", 26 | "from torchvision import datasets\n", 27 | "from torchvision.transforms import ToTensor" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": {}, 34 | "outputs": [ 35 | { 36 | "name": "stdout", 37 | "output_type": "stream", 38 | "text": [ 39 | "Shape of X [N, C, H, W]: torch.Size([64, 1, 28, 28])\n", 40 | "Shape of y: torch.Size([64]) torch.int64\n" 41 | ] 42 | } 43 | ], 44 | "source": [ 45 | "# Download training data from open datasets.\n", 46 | "## Dataset view: https://www.kaggle.com/datasets/zalando-research/fashionmnist\n", 47 | "\n", 48 | "training_data = datasets.FashionMNIST(\n", 49 | " root=\"data\",\n", 50 | " train=True,\n", 51 | " download=True,\n", 52 | " transform=ToTensor(),\n", 53 | ")\n", 54 | "\n", 55 | "# Download test data from open datasets.\n", 56 | "test_data = datasets.FashionMNIST(\n", 57 | " root=\"data\",\n", 58 | " train=False,\n", 59 | " download=True,\n", 60 | " transform=ToTensor(),\n", 61 | ")\n", 62 | "\n", 63 | "batch_size = 64\n", 64 | "\n", 65 | "# Create data loaders.\n", 66 | "\n", 67 | "# We pass the Dataset as an argument to DataLoader. This wraps an iterable over our dataset, and supports automatic batching, sampling, shuffling and multiprocess data loading. Here we define a batch size of 64, i.e. each element in the dataloader iterable will return a batch of 64 features and labels.\n", 68 | "\n", 69 | "train_dataloader = DataLoader(training_data, batch_size=batch_size)\n", 70 | "test_dataloader = DataLoader(test_data, batch_size=batch_size)\n", 71 | "\n", 72 | "for X, y in test_dataloader:\n", 73 | " print(f\"Shape of X [N, C, H, W]: {X.shape}\")\n", 74 | " print(f\"Shape of y: {y.shape} {y.dtype}\")\n", 75 | " break" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "Image shape (28, 28)\n" 88 | ] 89 | }, 90 | { 91 | "data": { 92 | "text/plain": [ 93 | "" 94 | ] 95 | }, 96 | "execution_count": 3, 97 | "metadata": {}, 98 | "output_type": "execute_result" 99 | }, 100 | { 101 | "data": { 102 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAdR0lEQVR4nO3df2xV9f3H8deltJdS2qu1tPd2lKYqqLOMbKAg8Ucx2thkZIouqMkCyWZ0/EhINWaMP2z2BzUuEv5gsmxZGESY/KPOBCJ2wZYZxlIIBsaMw1GlDq4VhN7Sllvafr5/EG6+pfzwc7y37972+UhOwj33vHo+HA599fSe+7kh55wTAAAGJlgPAAAwflFCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMDPRegBXGhwc1MmTJ1VYWKhQKGQ9HACAJ+ecurq6VF5ergkTrn+tM+pK6OTJk6qoqLAeBgDgO2pvb9e0adOuu82oK6HCwkJJlwZfVFRkPBqk24kTJ7wz27Zt886UlJR4ZyQpEol4Z3Jzc70zX331lXcmyG8Gbr31Vu+MJB04cMA709HR4Z35+uuvvTNvvvmmdwYjK5FIqKKiIvX9/HoyVkJvvPGGfvvb3+rUqVO6++67tWHDBj3wwAM3zF3+j1ZUVEQJjUHf5qS80qRJk7wz+fn53hlJmjx5sncmSAkFGV+QEiooKPDOSFI4HPbO5OXleWeCHDu+L2SPb3POZuTGhB07dmj16tVau3atDh06pAceeEB1dXWBfgoGAIxdGSmh9evX6+c//7l+8Ytf6K677tKGDRtUUVGhTZs2ZWJ3AIAslfYS6uvr08GDB1VbWztkfW1trfbt2zds+2QyqUQiMWQBAIwPaS+h06dPa2BgQGVlZUPWl5WVKR6PD9u+sbFRkUgktXBnHACMHxl7s+qVL0g55676ItWaNWvU2dmZWtrb2zM1JADAKJP2u+NKSkqUk5Mz7Kqno6Nj2NWRdOkunCB34gAAsl/ar4Ty8vI0Z84cNTU1DVnf1NSkBQsWpHt3AIAslpH3CdXX1+tnP/uZ5s6dq/vuu09/+MMfdOLECb3wwguZ2B0AIEtlpISWLFmiM2fO6De/+Y1OnTql6upq7dq1S5WVlZnYHQAgS2VsxoTly5dr+fLlmfryyFJ79+71zhw+fNg7c6NJE9O5r87OTu9MkGl7gkxFVFxc7J2RdNXXb28kFot5Z44cOeKdwdjCRzkAAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAwk7EJTIGr6erq8s7cdttt3pnTp097ZyTpzjvv9M4MDAwE2pevnp4e74xzLtC+pk6d6p255ZZbvDPJZNI7c+7cOe/MTTfd5J3ByOBKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmKGEAABmKCEAgBlKCABghlm0MaI++eQT70xHR4d3prOz0zsjBZuhOZFIeGemT5/unenv7/fO9Pb2emckacIE/59PBwcHvTN9fX3emUOHDnlnFi5c6J3ByOBKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmKGEAABmKCEAgBkmMMWI+vrrr70zZ8+e9c4EncA0yPhisZh35uLFi96ZgYEB70yQyVWlYBOY9vT0eGeCTMp6+vRp7wxGL66EAABmKCEAgBlKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmGECU4yoM2fOeGcqKiq8M5WVld4ZSWptbfXOxONx70xBQYF3Joggk54GFYlEvDPOOe/Mxx9/7J356U9/6p3ByOBKCABghhICAJhJewk1NDQoFAoNWaLRaLp3AwAYAzLymtDdd9+tv/3tb6nHOTk5mdgNACDLZaSEJk6cyNUPAOCGMvKa0LFjx1ReXq6qqio9/fTTOn78+DW3TSaTSiQSQxYAwPiQ9hKaN2+etm7dqt27d+uPf/yj4vG4FixYcM1bcxsbGxWJRFJLkNtxAQDZKe0lVFdXpyeffFKzZs3SI488op07d0qStmzZctXt16xZo87OztTS3t6e7iEBAEapjL9ZtaCgQLNmzdKxY8eu+nw4HFY4HM70MAAAo1DG3yeUTCb1ySefKBaLZXpXAIAsk/YSeumll9TS0qK2tjb985//1FNPPaVEIqGlS5eme1cAgCyX9l/Hffnll3rmmWd0+vRpTZ06VfPnz9f+/fsDz+UFABi70l5Cb731Vrq/JEap/v5+70yQCUx/+MMfemfy8vK8M5J07733emfOnTvnnfnPf/4zIvspKSnxzkhSUVGRd6asrMw7M23aNO/M//73P+8MRi/mjgMAmKGEAABmKCEAgBlKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGAm4x9qh7HrwoUL3pmbb77ZO1NaWuqd6ejo8M5Ilz7/ylcikfDOTJjg//Nfb2+vdybIhKxSsPHl5OR4ZwoKCrwzzjnvDEYvroQAAGYoIQCAGUoIAGCGEgIAmKGEAABmKCEAgBlKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGaYRRuBdXd3e2fy8/O9M0FmdO7r6/POSFI4HPbOXLx40Ttz8OBB70xlZaV35vjx494ZSbr99tu9M0H+bUtKSrwzubm53hmMXlwJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMMMEpggsyCShhYWFGRjJcEEmFZWkrq4u70wsFvPOhEIh70yQyT4LCgq8M5L03//+1zszdepU70yQyUiDTk6L0YkrIQCAGUoIAGCGEgIAmKGEAABmKCEAgBlKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGaYwBSBBZmEMz8/PwMjGW5wcDBQ7qabbvLOHDhwINC+fEWjUe/MlClTAu1r5syZ3pkvvvjCO9PT0+OdGalJcDEyuBICAJihhAAAZrxLaO/evVq0aJHKy8sVCoX07rvvDnneOaeGhgaVl5crPz9fNTU1Onr0aLrGCwAYQ7xLqLu7W7Nnz9bGjRuv+vxrr72m9evXa+PGjWptbVU0GtWjjz4a6MPCAABjm/eNCXV1daqrq7vqc845bdiwQWvXrtXixYslSVu2bFFZWZm2b9+u559//ruNFgAwpqT1NaG2tjbF43HV1tam1oXDYT300EPat2/fVTPJZFKJRGLIAgAYH9JaQvF4XJJUVlY2ZH1ZWVnquSs1NjYqEomkloqKinQOCQAwimXk7rgr3z/inLvme0rWrFmjzs7O1NLe3p6JIQEARqG0vln18pvp4vG4YrFYan1HR8ewq6PLwuGwwuFwOocBAMgSab0SqqqqUjQaVVNTU2pdX1+fWlpatGDBgnTuCgAwBnhfCZ0/f16fffZZ6nFbW5s+/vhjFRcXa/r06Vq9erXWrVunGTNmaMaMGVq3bp0mT56sZ599Nq0DBwBkP+8SOnDggBYuXJh6XF9fL0launSp/vznP+vll19Wb2+vli9frrNnz2revHn64IMPmO8JADCMdwnV1NTIOXfN50OhkBoaGtTQ0PBdxoUscL3z4FomTBiZmaKC7ufChQvembNnzwbal6877rjDO7N///5A+7rrrru8MwUFBd6Zb775xjuTk5PjncHoxdxxAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzaf1kVYwv1/rI9uvJzc31zkyZMsU7k5+f752RpK+//to7U1RUFGhfvr7//e97Z3bv3h1oX5MmTfLO/P9PU/622tvbvTMDAwPeGYxeXAkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAwwwSmGFFBJj29ePGidyboBKZ9fX3embKyskD78vWDH/xgRPYjBZvIdXBw0DsTjUa9M0HOIYxeXAkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAwwwSmCMw5551JJpPemd7eXu9MXl6ed0aScnJyvDORSCTQvnzddddd3pmBgYFA++rv7/fOTJjg/zNtkPNh8uTJ3hmMXlwJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMMMEpghscHDQOxNk0tMgk2l+88033hkp2N+puro60L58hcPhEdmPJIVCIe9M0MlSfQUZG0YvroQAAGYoIQCAGe8S2rt3rxYtWqTy8nKFQiG9++67Q55ftmyZQqHQkGX+/PnpGi8AYAzxLqHu7m7Nnj1bGzduvOY2jz32mE6dOpVadu3a9Z0GCQAYm7xvTKirq1NdXd11twmHw4pGo4EHBQAYHzLymlBzc7NKS0s1c+ZMPffcc+ro6LjmtslkUolEYsgCABgf0l5CdXV12rZtm/bs2aPXX39dra2tevjhh6/5WfKNjY2KRCKppaKiIt1DAgCMUml/n9CSJUtSf66urtbcuXNVWVmpnTt3avHixcO2X7Nmjerr61OPE4kERQQA40TG36wai8VUWVmpY8eOXfX5cDg8om/CAwCMHhl/n9CZM2fU3t6uWCyW6V0BALKM95XQ+fPn9dlnn6Uet7W16eOPP1ZxcbGKi4vV0NCgJ598UrFYTJ9//rl+/etfq6SkRE888URaBw4AyH7eJXTgwAEtXLgw9fjy6zlLly7Vpk2bdOTIEW3dulXnzp1TLBbTwoULtWPHDhUWFqZv1ACAMcG7hGpqaq47CeXu3bu/04CQPYJMWNnb2+udmTp1qnfmyy+/9M5IwcZ36623BtqXryCvnebl5QXaV5BJY4NM/jpxov/L0ryGPLYwdxwAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwEzGP1kV+P9OnjzpnSkqKvLOJJNJ70zQ3MyZMwPtayQUFxcHyvX19Xln8vPzvTMXLlzwzkyZMsU7g9GLKyEAgBlKCABghhICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmKGEAABmmMAUgU2Y4P8zTCKR8M60tbV5Zy5evOidkaTe3l7vzPTp0wPtayREo9FAubNnz3pnCgsLvTOhUGhEMhi9uBICAJihhAAAZighAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmKGEAABmKCEAgBlKCABghglMMeqdP39+xPblnPPOFBQUZGAk6VFRUREod/jwYe9MOBz2zvT19XlncnNzvTMYvbgSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmKGEAABmKCEAgBlKCABghhICAJihhAAAZighAIAZSggAYIYJTDGiBgYGvDPd3d3emZ6eHu+MJE2aNGlEMiMlGo0Gyv3rX//yzpw7d84789VXX3lngk7KitGJKyEAgBlKCABgxquEGhsbdc8996iwsFClpaV6/PHH9emnnw7ZxjmnhoYGlZeXKz8/XzU1NTp69GhaBw0AGBu8SqilpUUrVqzQ/v371dTUpP7+ftXW1g75nf1rr72m9evXa+PGjWptbVU0GtWjjz6qrq6utA8eAJDdvG5MeP/994c83rx5s0pLS3Xw4EE9+OCDcs5pw4YNWrt2rRYvXixJ2rJli8rKyrR9+3Y9//zz6Rs5ACDrfafXhDo7OyVJxcXFkqS2tjbF43HV1tamtgmHw3rooYe0b9++q36NZDKpRCIxZAEAjA+BS8g5p/r6et1///2qrq6WJMXjcUlSWVnZkG3LyspSz12psbFRkUgktXD7JQCMH4FLaOXKlTp8+LD+8pe/DHsuFAoNeeycG7busjVr1qizszO1tLe3Bx0SACDLBHqz6qpVq/Tee+9p7969mjZtWmr95TfGxeNxxWKx1PqOjo5hV0eXhcNhhcPhIMMAAGQ5rysh55xWrlypt99+W3v27FFVVdWQ56uqqhSNRtXU1JRa19fXp5aWFi1YsCA9IwYAjBleV0IrVqzQ9u3b9de//lWFhYWp13kikYjy8/MVCoW0evVqrVu3TjNmzNCMGTO0bt06TZ48Wc8++2xG/gIAgOzlVUKbNm2SJNXU1AxZv3nzZi1btkyS9PLLL6u3t1fLly/X2bNnNW/ePH3wwQcqLCxMy4ABAGOHVwk55264TSgUUkNDgxoaGoKOCWNYXl6ed2ZwcNA7E/TN0ZffbuAjJycn0L5GQmlpaaBckElZJ070f4n5woULI7IfjF7MHQcAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMMN0tAisr6/POxOJRLwz32b29islEgnvjCRVVFQEyo1Wt99+e6BcMpn0zuTn5wfal6/c3NwR2Q9GBldCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzDCBKQLr7+/3zuTl5XlngkyU2tvb652RpJtvvjlQbrS65ZZbAuVCodCIZIJMlDphAj87jyX8awIAzFBCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADDDBKYI7OLFi96ZIBOYBjE4OBgoN3ny5DSP5Oqcc96ZIBOEBj3e+fn53pmcnBzvTElJiXcmyNgwenElBAAwQwkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwTmGJEFRcXj8h+IpFIoNxYm8C0qKjIOyNJEyf6f2sIMr4gGSYwHVu4EgIAmKGEAABmvEqosbFR99xzjwoLC1VaWqrHH39cn3766ZBtli1bplAoNGSZP39+WgcNABgbvEqopaVFK1as0P79+9XU1KT+/n7V1taqu7t7yHaPPfaYTp06lVp27dqV1kEDAMYGr1cf33///SGPN2/erNLSUh08eFAPPvhgan04HFY0Gk3PCAEAY9Z3ek2os7NT0vA7npqbm1VaWqqZM2fqueeeU0dHxzW/RjKZVCKRGLIAAMaHwCXknFN9fb3uv/9+VVdXp9bX1dVp27Zt2rNnj15//XW1trbq4YcfVjKZvOrXaWxsVCQSSS0VFRVBhwQAyDKB3ye0cuVKHT58WB999NGQ9UuWLEn9ubq6WnPnzlVlZaV27typxYsXD/s6a9asUX19fepxIpGgiABgnAhUQqtWrdJ7772nvXv3atq0adfdNhaLqbKyUseOHbvq8+FwWOFwOMgwAABZzquEnHNatWqV3nnnHTU3N6uqquqGmTNnzqi9vV2xWCzwIAEAY5PXa0IrVqzQm2++qe3bt6uwsFDxeFzxeFy9vb2SpPPnz+ull17SP/7xD33++edqbm7WokWLVFJSoieeeCIjfwEAQPbyuhLatGmTJKmmpmbI+s2bN2vZsmXKycnRkSNHtHXrVp07d06xWEwLFy7Ujh07VFhYmLZBAwDGBu9fx11Pfn6+du/e/Z0GBAAYP5hFG4F988033pl4PO6dKS0t9c5cOYvHtzVSMzQHmUU7iJycnEC5CxcueGeCzIgdZD9dXV3eGYxeTGAKADBDCQEAzFBCAAAzlBAAwAwlBAAwQwkBAMxQQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADBOYIrAZM2Z4Z5566invzMWLF70zt9xyi3dGkh555JFAOV9BJvsMYsqUKYFyd9xxh3cmyIS206dP987MmTPHO4PRiyshAIAZSggAYIYSAgCYoYQAAGYoIQCAGUoIAGCGEgIAmKGEAABmKCEAgBlKCABghhICAJgZdXPHOeckSYlEwngkuJGuri7vTG9vr3cmyNxxyWTSOyNJ58+f984EOVcHBwe9MxMm+P/M2N/f752Rgh2/vr4+70yQf9uenh7vDN9PRtbl4335+/n1hNy32WoEffnll6qoqLAeBgDgO2pvb9e0adOuu82oK6HBwUGdPHlShYWFw2YaTiQSqqioUHt7u4qKioxGaI/jcAnH4RKOwyUch0tGw3Fwzqmrq0vl5eU3vIIfdb+OmzBhwg2bs6ioaFyfZJdxHC7hOFzCcbiE43CJ9XGIRCLfajtuTAAAmKGEAABmsqqEwuGwXnnlFYXDYeuhmOI4XMJxuITjcAnH4ZJsOw6j7sYEAMD4kVVXQgCAsYUSAgCYoYQAAGYoIQCAmawqoTfeeENVVVWaNGmS5syZo7///e/WQxpRDQ0NCoVCQ5ZoNGo9rIzbu3evFi1apPLycoVCIb377rtDnnfOqaGhQeXl5crPz1dNTY2OHj1qM9gMutFxWLZs2bDzY/78+TaDzZDGxkbdc889KiwsVGlpqR5//HF9+umnQ7YZD+fDtzkO2XI+ZE0J7dixQ6tXr9batWt16NAhPfDAA6qrq9OJEyeshzai7r77bp06dSq1HDlyxHpIGdfd3a3Zs2dr48aNV33+tdde0/r167Vx40a1trYqGo3q0UcfDTTB6mh2o+MgSY899tiQ82PXrl0jOMLMa2lp0YoVK7R//341NTWpv79ftbW16u7uTm0zHs6Hb3McpCw5H1yWuPfee90LL7wwZN2dd97pfvWrXxmNaOS98sorbvbs2dbDMCXJvfPOO6nHg4ODLhqNuldffTW17sKFCy4Sibjf//73BiMcGVceB+ecW7p0qfvJT35iMh4rHR0dTpJraWlxzo3f8+HK4+Bc9pwPWXEl1NfXp4MHD6q2tnbI+traWu3bt89oVDaOHTum8vJyVVVV6emnn9bx48eth2Sqra1N8Xh8yLkRDof10EMPjbtzQ5Kam5tVWlqqmTNn6rnnnlNHR4f1kDKqs7NTklRcXCxp/J4PVx6Hy7LhfMiKEjp9+rQGBgZUVlY2ZH1ZWZni8bjRqEbevHnztHXrVu3evVt//OMfFY/HtWDBAp05c8Z6aGYu//uP93NDkurq6rRt2zbt2bNHr7/+ulpbW/Xwww8H/myl0c45p/r6et1///2qrq6WND7Ph6sdByl7zodRN4v29Vz50Q7OuWHrxrK6urrUn2fNmqX77rtPt912m7Zs2aL6+nrDkdkb7+eGJC1ZsiT15+rqas2dO1eVlZXauXOnFi9ebDiyzFi5cqUOHz6sjz76aNhz4+l8uNZxyJbzISuuhEpKSpSTkzPsJ5mOjo5hP/GMJwUFBZo1a5aOHTtmPRQzl+8O5NwYLhaLqbKyckyeH6tWrdJ7772nDz/8cMhHv4y38+Fax+FqRuv5kBUllJeXpzlz5qipqWnI+qamJi1YsMBoVPaSyaQ++eQTxWIx66GYqaqqUjQaHXJu9PX1qaWlZVyfG5J05swZtbe3j6nzwzmnlStX6u2339aePXtUVVU15Pnxcj7c6Dhczag9HwxvivDy1ltvudzcXPenP/3J/fvf/3arV692BQUF7vPPP7ce2oh58cUXXXNzszt+/Ljbv3+/+/GPf+wKCwvH/DHo6upyhw4dcocOHXKS3Pr1692hQ4fcF1984Zxz7tVXX3WRSMS9/fbb7siRI+6ZZ55xsVjMJRIJ45Gn1/WOQ1dXl3vxxRfdvn37XFtbm/vwww/dfffd5773ve+NqePwy1/+0kUiEdfc3OxOnTqVWnp6elLbjIfz4UbHIZvOh6wpIeec+93vfucqKytdXl6e+9GPfjTkdsTxYMmSJS4Wi7nc3FxXXl7uFi9e7I4ePWo9rIz78MMPnaRhy9KlS51zl27LfeWVV1w0GnXhcNg9+OCD7siRI7aDzoDrHYeenh5XW1vrpk6d6nJzc9306dPd0qVL3YkTJ6yHnVZX+/tLcps3b05tMx7Ohxsdh2w6H/goBwCAmax4TQgAMDZRQgAAM5QQAMAMJQQAMEMJAQDMUEIAADOUEADADCUEADBDCQEAzFBCAAAzlBAAwAwlBAAw838byrwPlMm3yQAAAABJRU5ErkJggg==", 103 | "text/plain": [ 104 | "
" 105 | ] 106 | }, 107 | "metadata": {}, 108 | "output_type": "display_data" 109 | } 110 | ], 111 | "source": [ 112 | "# Displaying sample the data using matplotlib\n", 113 | "\n", 114 | "import matplotlib.pyplot as plt\n", 115 | "import numpy as np\n", 116 | "\n", 117 | "# matplotlib inline\n", 118 | "sample_data = training_data[16]\n", 119 | "image = sample_data[0].numpy()[0]\n", 120 | "print(f\"Image shape {image.shape}\")\n", 121 | "plt.imshow(image * 255, cmap='Greys')" 122 | ] 123 | }, 124 | { 125 | "cell_type": "markdown", 126 | "metadata": {}, 127 | "source": [ 128 | "# Model" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 4, 134 | "metadata": {}, 135 | "outputs": [ 136 | { 137 | "name": "stdout", 138 | "output_type": "stream", 139 | "text": [ 140 | "Using mps device\n", 141 | "NeuralNetwork(\n", 142 | " (flatten): Flatten(start_dim=1, end_dim=-1)\n", 143 | " (linear_relu_stack): Sequential(\n", 144 | " (0): Linear(in_features=784, out_features=512, bias=True)\n", 145 | " (1): ReLU()\n", 146 | " (2): Linear(in_features=512, out_features=512, bias=True)\n", 147 | " (3): ReLU()\n", 148 | " (4): Linear(in_features=512, out_features=10, bias=True)\n", 149 | " )\n", 150 | ")\n" 151 | ] 152 | } 153 | ], 154 | "source": [ 155 | "# Get cpu, gpu or mps device for training.\n", 156 | "device = (\n", 157 | " \"cuda\"\n", 158 | " if torch.cuda.is_available()\n", 159 | " else \"mps\"\n", 160 | " if torch.backends.mps.is_available()\n", 161 | " else \"cpu\"\n", 162 | ")\n", 163 | "print(f\"Using {device} device\")\n", 164 | "\n", 165 | "# Define model\n", 166 | "class NeuralNetwork(nn.Module):\n", 167 | " def __init__(self):\n", 168 | " super().__init__()\n", 169 | " self.flatten = nn.Flatten() # convert each 2D 28x28 image into a contiguous array of 784 pixel values (the minibatch dimension (at dim=0) is maintained).\n", 170 | " self.linear_relu_stack = nn.Sequential(\n", 171 | " nn.Linear(28*28, 512), # images are 28x28 pixels\n", 172 | " nn.ReLU(),\n", 173 | " nn.Linear(512, 512),\n", 174 | " nn.ReLU(),\n", 175 | " nn.Linear(512, 10)\n", 176 | " )\n", 177 | "\n", 178 | " def forward(self, x):\n", 179 | " x = self.flatten(x)\n", 180 | " logits = self.linear_relu_stack(x)\n", 181 | " return logits\n", 182 | "\n", 183 | "model = NeuralNetwork().to(device)\n", 184 | "print(model)" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 5, 190 | "metadata": {}, 191 | "outputs": [ 192 | { 193 | "name": "stdout", 194 | "output_type": "stream", 195 | "text": [ 196 | "Predicted class: Ankle boot, Actual class: Trouser\n" 197 | ] 198 | } 199 | ], 200 | "source": [ 201 | "# Forward propagation on first data sample\n", 202 | "\n", 203 | "X = sample_data[0].to(device)\n", 204 | "logits = model(X)\n", 205 | "pred_probab = nn.Softmax(dim=1)(logits)\n", 206 | "y_pred = pred_probab.argmax(1).cpu().item()\n", 207 | "\n", 208 | "print(f\"Predicted class: {datasets.FashionMNIST.classes[y_pred]}, Actual class: {datasets.FashionMNIST.classes[sample_data[1]]}\")\n", 209 | "\n", 210 | "# Why wrong prediction? (depends on random initial weights) -> We havent trained our model yet" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": 6, 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "loss_fn = nn.CrossEntropyLoss() # More about loss functions: https://pytorch.org/docs/stable/nn.html#loss-functions\n", 220 | "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) # More about optimizers: https://pytorch.org/docs/stable/optim.html\n", 221 | "\n", 222 | "def train(dataloader, model, loss_fn, optimizer):\n", 223 | " size = len(dataloader.dataset)\n", 224 | " model.train() # set model to training mode\n", 225 | " for batch, (X, y) in enumerate(dataloader):\n", 226 | " X, y = X.to(device), y.to(device)\n", 227 | "\n", 228 | " # Compute prediction error\n", 229 | " pred = model(X)\n", 230 | " loss = loss_fn(pred, y)\n", 231 | "\n", 232 | " # Backpropagation\n", 233 | " loss.backward()\n", 234 | " optimizer.step()\n", 235 | " optimizer.zero_grad() # reset gradients\n", 236 | "\n", 237 | " if batch % 100 == 0:\n", 238 | " loss, current = loss.item(), (batch + 1) * len(X)\n", 239 | " print(f\"loss: {loss:>7f} [{current:>5d}/{size:>5d}]\")\n", 240 | "\n", 241 | "def test(dataloader, model, loss_fn):\n", 242 | " size = len(dataloader.dataset)\n", 243 | " num_batches = len(dataloader)\n", 244 | " model.eval() # set model to evaluation mode\n", 245 | " test_loss, correct = 0, 0\n", 246 | " with torch.no_grad():\n", 247 | " for X, y in dataloader:\n", 248 | " X, y = X.to(device), y.to(device)\n", 249 | " pred = model(X)\n", 250 | " test_loss += loss_fn(pred, y).item()\n", 251 | " correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n", 252 | " test_loss /= num_batches\n", 253 | " correct /= size\n", 254 | " print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": 7, 260 | "metadata": {}, 261 | "outputs": [ 262 | { 263 | "name": "stdout", 264 | "output_type": "stream", 265 | "text": [ 266 | "Test Error: \n", 267 | " Accuracy: 8.9%, Avg loss: 2.304297 \n", 268 | "\n", 269 | "Epoch 1\n", 270 | "-------------------------------\n", 271 | "loss: 2.314872 [ 64/60000]\n", 272 | "loss: 0.555549 [ 6464/60000]\n", 273 | "loss: 0.385013 [12864/60000]\n", 274 | "loss: 0.497228 [19264/60000]\n", 275 | "loss: 0.427932 [25664/60000]\n", 276 | "loss: 0.448676 [32064/60000]\n", 277 | "loss: 0.364214 [38464/60000]\n", 278 | "loss: 0.523146 [44864/60000]\n", 279 | "loss: 0.470050 [51264/60000]\n", 280 | "loss: 0.540722 [57664/60000]\n", 281 | "Test Error: \n", 282 | " Accuracy: 85.0%, Avg loss: 0.420927 \n", 283 | "\n", 284 | "Epoch 2\n", 285 | "-------------------------------\n", 286 | "loss: 0.263710 [ 64/60000]\n", 287 | "loss: 0.368335 [ 6464/60000]\n", 288 | "loss: 0.279124 [12864/60000]\n", 289 | "loss: 0.384675 [19264/60000]\n", 290 | "loss: 0.406725 [25664/60000]\n", 291 | "loss: 0.424508 [32064/60000]\n", 292 | "loss: 0.303943 [38464/60000]\n", 293 | "loss: 0.490382 [44864/60000]\n", 294 | "loss: 0.412547 [51264/60000]\n", 295 | "loss: 0.454745 [57664/60000]\n", 296 | "Test Error: \n", 297 | " Accuracy: 85.4%, Avg loss: 0.396090 \n", 298 | "\n", 299 | "Epoch 3\n", 300 | "-------------------------------\n", 301 | "loss: 0.217470 [ 64/60000]\n", 302 | "loss: 0.339472 [ 6464/60000]\n", 303 | "loss: 0.233601 [12864/60000]\n", 304 | "loss: 0.326719 [19264/60000]\n", 305 | "loss: 0.404295 [25664/60000]\n", 306 | "loss: 0.359180 [32064/60000]\n", 307 | "loss: 0.276142 [38464/60000]\n", 308 | "loss: 0.434472 [44864/60000]\n", 309 | "loss: 0.332469 [51264/60000]\n", 310 | "loss: 0.402681 [57664/60000]\n", 311 | "Test Error: \n", 312 | " Accuracy: 85.9%, Avg loss: 0.380580 \n", 313 | "\n", 314 | "Epoch 4\n", 315 | "-------------------------------\n", 316 | "loss: 0.221186 [ 64/60000]\n", 317 | "loss: 0.299829 [ 6464/60000]\n", 318 | "loss: 0.210987 [12864/60000]\n", 319 | "loss: 0.264386 [19264/60000]\n", 320 | "loss: 0.346169 [25664/60000]\n", 321 | "loss: 0.315576 [32064/60000]\n", 322 | "loss: 0.226186 [38464/60000]\n", 323 | "loss: 0.423226 [44864/60000]\n", 324 | "loss: 0.292712 [51264/60000]\n", 325 | "loss: 0.362410 [57664/60000]\n", 326 | "Test Error: \n", 327 | " Accuracy: 87.1%, Avg loss: 0.355349 \n", 328 | "\n", 329 | "Epoch 5\n", 330 | "-------------------------------\n", 331 | "loss: 0.188006 [ 64/60000]\n", 332 | "loss: 0.248344 [ 6464/60000]\n", 333 | "loss: 0.221260 [12864/60000]\n", 334 | "loss: 0.246449 [19264/60000]\n", 335 | "loss: 0.364733 [25664/60000]\n", 336 | "loss: 0.299162 [32064/60000]\n", 337 | "loss: 0.228872 [38464/60000]\n", 338 | "loss: 0.357628 [44864/60000]\n", 339 | "loss: 0.284954 [51264/60000]\n", 340 | "loss: 0.307330 [57664/60000]\n", 341 | "Test Error: \n", 342 | " Accuracy: 87.4%, Avg loss: 0.340698 \n", 343 | "\n", 344 | "Done!\n" 345 | ] 346 | } 347 | ], 348 | "source": [ 349 | "# Initial accuracy\n", 350 | "test(test_dataloader, model, loss_fn)\n", 351 | "\n", 352 | "epochs = 5\n", 353 | "for t in range(epochs):\n", 354 | " print(f\"Epoch {t+1}\\n-------------------------------\")\n", 355 | " train(train_dataloader, model, loss_fn, optimizer)\n", 356 | " test(test_dataloader, model, loss_fn)\n", 357 | "print(\"Done!\")" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": 8, 363 | "metadata": {}, 364 | "outputs": [ 365 | { 366 | "name": "stdout", 367 | "output_type": "stream", 368 | "text": [ 369 | "Predicted class: Trouser, Actual class: Trouser\n" 370 | ] 371 | } 372 | ], 373 | "source": [ 374 | "# Forward propagation on first data sample\n", 375 | "\n", 376 | "X = sample_data[0].to(device)\n", 377 | "logits = model(X)\n", 378 | "pred_probab = nn.Softmax(dim=1)(logits)\n", 379 | "y_pred = pred_probab.argmax(1).cpu().item()\n", 380 | "\n", 381 | "print(f\"Predicted class: {datasets.FashionMNIST.classes[y_pred]}, Actual class: {datasets.FashionMNIST.classes[sample_data[1]]}\")" 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "execution_count": null, 387 | "metadata": {}, 388 | "outputs": [], 389 | "source": [] 390 | } 391 | ], 392 | "metadata": { 393 | "kernelspec": { 394 | "display_name": "Python Pytorch", 395 | "language": "python", 396 | "name": "pytorch" 397 | }, 398 | "language_info": { 399 | "codemirror_mode": { 400 | "name": "ipython", 401 | "version": 3 402 | }, 403 | "file_extension": ".py", 404 | "mimetype": "text/x-python", 405 | "name": "python", 406 | "nbconvert_exporter": "python", 407 | "pygments_lexer": "ipython3", 408 | "version": "3.11.5" 409 | } 410 | }, 411 | "nbformat": 4, 412 | "nbformat_minor": 2 413 | } 414 | -------------------------------------------------------------------------------- /2023_Edition/Workshops/1 - Intro to Neural Networks/mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Workshops/1 - Intro to Neural Networks/mlp.png -------------------------------------------------------------------------------- /2023_Edition/Workshops/1 - Intro to Neural Networks/relu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Workshops/1 - Intro to Neural Networks/relu.png -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/CNN_notebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "dd8bc65e", 6 | "metadata": {}, 7 | "source": [ 8 | "# Convolutional Neural Networks\n", 9 | " This code defines a Convolutional Neural Network (CNN) model using PyTorch. The model is used for image classification tasks." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "id": "aa160f82", 15 | "metadata": {}, 16 | "source": [ 17 | "# Model architecture\n", 18 | "## Convolutional Layers\n", 19 | " The CNN consists of two convolutional layers with ReLU activation functions and max pooling layers. The first convolutional layer takes a 3-channel input image and outputs 6 feature maps, using a 4x4 kernel. The second convolutional layer takes the output of the first convolutional layer as input and outputs 9 feature maps, using a 5x5 kernel.\n", 20 | "## Linear Layers\n", 21 | " The output from the last convolutional layer is flattened into a 1D tensor and passed through two fully connected (linear) layers with ReLU activation functions. The first linear layer has 80 neurons and the second has 10 neurons (the number of classes in the classification task)." 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "id": "c43039c6", 27 | "metadata": {}, 28 | "source": [ 29 | "# Model Definition" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 2, 35 | "id": "b11c4293", 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "import torch.nn as nn\n", 40 | "\n", 41 | "\n", 42 | "class ConvModel(nn.Module):\n", 43 | " def __init__(self):\n", 44 | " super().__init__()\n", 45 | " self.conv = nn.Sequential(\n", 46 | " nn.Conv2d(3, 6, 4),\n", 47 | " nn.ReLU(),\n", 48 | " nn.MaxPool2d(2, 2),\n", 49 | "\n", 50 | " nn.Conv2d(6, 9, 5),\n", 51 | " nn.ReLU(),\n", 52 | " nn.MaxPool2d(2, 2),\n", 53 | " )\n", 54 | " self.linear = nn.Sequential(\n", 55 | " nn.Linear(5 * 5 * 9, 80),\n", 56 | " nn.ReLU(),\n", 57 | " nn.Linear(80, 10),\n", 58 | " )\n", 59 | "\n", 60 | " def forward(self, x):\n", 61 | " x = self.conv(x)\n", 62 | " x = x.view(-1, 5 * 5 * 9)\n", 63 | " x = self.linear(x)\n", 64 | " return x\n", 65 | " " 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "id": "96b69466", 71 | "metadata": {}, 72 | "source": [ 73 | "In the **forward** method, the input tensor is first passed through the convolutional layers and then flattened into a 1D tensor. The flattened tensor is then passed through the linear layers to produce the final output." 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "id": "d3c16e60", 79 | "metadata": {}, 80 | "source": [ 81 | "# CNN Training" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 3, 87 | "id": "f4621864", 88 | "metadata": {}, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Files already downloaded and verified\n", 95 | "Files already downloaded and verified\n", 96 | "Epoch: 1/20 | Step: 2000;250000 | Loss: 1.868149995803833\n", 97 | "Epoch: 1/20 | Step: 4000;250000 | Loss: 1.808663249015808\n", 98 | "Epoch: 1/20 | Step: 6000;250000 | Loss: 2.0486741065979004\n", 99 | "Epoch: 1/20 | Step: 8000;250000 | Loss: 2.148817539215088\n", 100 | "Epoch: 1/20 | Step: 10000;250000 | Loss: 1.7136116027832031\n", 101 | "Epoch: 1/20 | Step: 12000;250000 | Loss: 1.2580480575561523\n", 102 | "Epoch: 2/20 | Step: 14500;250000 | Loss: 0.9803611636161804\n", 103 | "Epoch: 2/20 | Step: 16500;250000 | Loss: 0.6896300911903381\n", 104 | "Epoch: 2/20 | Step: 18500;250000 | Loss: 0.8863891959190369\n", 105 | "Epoch: 2/20 | Step: 20500;250000 | Loss: 1.2686976194381714\n", 106 | "Epoch: 2/20 | Step: 22500;250000 | Loss: 0.6618356704711914\n", 107 | "Epoch: 2/20 | Step: 24500;250000 | Loss: 1.5075292587280273\n", 108 | "Epoch: 3/20 | Step: 27000;250000 | Loss: 1.0405328273773193\n", 109 | "Epoch: 3/20 | Step: 29000;250000 | Loss: 1.1788005828857422\n", 110 | "Epoch: 3/20 | Step: 31000;250000 | Loss: 0.28276947140693665\n", 111 | "Epoch: 3/20 | Step: 33000;250000 | Loss: 0.5511508584022522\n", 112 | "Epoch: 3/20 | Step: 35000;250000 | Loss: 1.1681625843048096\n", 113 | "Epoch: 3/20 | Step: 37000;250000 | Loss: 0.7604244351387024\n", 114 | "Epoch: 4/20 | Step: 39500;250000 | Loss: 1.8094203472137451\n", 115 | "Epoch: 4/20 | Step: 41500;250000 | Loss: 1.0180308818817139\n", 116 | "Epoch: 4/20 | Step: 43500;250000 | Loss: 1.2012836933135986\n", 117 | "Epoch: 4/20 | Step: 45500;250000 | Loss: 1.326040267944336\n", 118 | "Epoch: 4/20 | Step: 47500;250000 | Loss: 1.4490337371826172\n", 119 | "Epoch: 4/20 | Step: 49500;250000 | Loss: 0.5611826777458191\n", 120 | "Epoch: 5/20 | Step: 52000;250000 | Loss: 1.1142256259918213\n", 121 | "Epoch: 5/20 | Step: 54000;250000 | Loss: 1.4296016693115234\n", 122 | "Epoch: 5/20 | Step: 56000;250000 | Loss: 2.276676654815674\n", 123 | "Epoch: 5/20 | Step: 58000;250000 | Loss: 0.9492870569229126\n", 124 | "Epoch: 5/20 | Step: 60000;250000 | Loss: 2.0789132118225098\n", 125 | "Epoch: 5/20 | Step: 62000;250000 | Loss: 1.1576035022735596\n", 126 | "Epoch: 6/20 | Step: 64500;250000 | Loss: 0.4269680380821228\n", 127 | "Epoch: 6/20 | Step: 66500;250000 | Loss: 0.7921403646469116\n", 128 | "Epoch: 6/20 | Step: 68500;250000 | Loss: 2.279670000076294\n", 129 | "Epoch: 6/20 | Step: 70500;250000 | Loss: 0.6963077783584595\n", 130 | "Epoch: 6/20 | Step: 72500;250000 | Loss: 0.6398743987083435\n", 131 | "Epoch: 6/20 | Step: 74500;250000 | Loss: 1.271297812461853\n", 132 | "Epoch: 7/20 | Step: 77000;250000 | Loss: 0.3938380777835846\n", 133 | "Epoch: 7/20 | Step: 79000;250000 | Loss: 1.205625295639038\n", 134 | "Epoch: 7/20 | Step: 81000;250000 | Loss: 0.7136484980583191\n", 135 | "Epoch: 7/20 | Step: 83000;250000 | Loss: 0.8042351007461548\n", 136 | "Epoch: 7/20 | Step: 85000;250000 | Loss: 1.6126559972763062\n", 137 | "Epoch: 7/20 | Step: 87000;250000 | Loss: 0.28250449895858765\n", 138 | "Epoch: 8/20 | Step: 89500;250000 | Loss: 1.3294621706008911\n", 139 | "Epoch: 8/20 | Step: 91500;250000 | Loss: 1.0522974729537964\n", 140 | "Epoch: 8/20 | Step: 93500;250000 | Loss: 0.924340009689331\n", 141 | "Epoch: 8/20 | Step: 95500;250000 | Loss: 0.8075780868530273\n", 142 | "Epoch: 8/20 | Step: 97500;250000 | Loss: 2.1963303089141846\n", 143 | "Epoch: 8/20 | Step: 99500;250000 | Loss: 0.9101738333702087\n", 144 | "Epoch: 9/20 | Step: 102000;250000 | Loss: 1.7532811164855957\n", 145 | "Epoch: 9/20 | Step: 104000;250000 | Loss: 0.9450191259384155\n", 146 | "Epoch: 9/20 | Step: 106000;250000 | Loss: 0.8171242475509644\n", 147 | "Epoch: 9/20 | Step: 108000;250000 | Loss: 0.6594465374946594\n", 148 | "Epoch: 9/20 | Step: 110000;250000 | Loss: 1.352130651473999\n", 149 | "Epoch: 9/20 | Step: 112000;250000 | Loss: 1.8082475662231445\n", 150 | "Epoch: 10/20 | Step: 114500;250000 | Loss: 0.705681562423706\n", 151 | "Epoch: 10/20 | Step: 116500;250000 | Loss: 0.6882866621017456\n", 152 | "Epoch: 10/20 | Step: 118500;250000 | Loss: 0.9365450143814087\n", 153 | "Epoch: 10/20 | Step: 120500;250000 | Loss: 1.0160834789276123\n", 154 | "Epoch: 10/20 | Step: 122500;250000 | Loss: 1.345630407333374\n", 155 | "Epoch: 10/20 | Step: 124500;250000 | Loss: 1.0211046934127808\n", 156 | "Epoch: 11/20 | Step: 127000;250000 | Loss: 1.4727071523666382\n", 157 | "Epoch: 11/20 | Step: 129000;250000 | Loss: 1.2301959991455078\n", 158 | "Epoch: 11/20 | Step: 131000;250000 | Loss: 1.3269494771957397\n", 159 | "Epoch: 11/20 | Step: 133000;250000 | Loss: 0.6285428404808044\n", 160 | "Epoch: 11/20 | Step: 135000;250000 | Loss: 1.069046139717102\n", 161 | "Epoch: 11/20 | Step: 137000;250000 | Loss: 0.8192417621612549\n", 162 | "Epoch: 12/20 | Step: 139500;250000 | Loss: 0.7842008471488953\n", 163 | "Epoch: 12/20 | Step: 141500;250000 | Loss: 0.4815007448196411\n", 164 | "Epoch: 12/20 | Step: 143500;250000 | Loss: 1.2153191566467285\n", 165 | "Epoch: 12/20 | Step: 145500;250000 | Loss: 1.181143045425415\n", 166 | "Epoch: 12/20 | Step: 147500;250000 | Loss: 0.9448485970497131\n", 167 | "Epoch: 12/20 | Step: 149500;250000 | Loss: 0.8398048281669617\n", 168 | "Epoch: 13/20 | Step: 152000;250000 | Loss: 0.9905456900596619\n", 169 | "Epoch: 13/20 | Step: 154000;250000 | Loss: 0.5820574760437012\n", 170 | "Epoch: 13/20 | Step: 156000;250000 | Loss: 1.0987743139266968\n", 171 | "Epoch: 13/20 | Step: 158000;250000 | Loss: 0.553002119064331\n", 172 | "Epoch: 13/20 | Step: 160000;250000 | Loss: 1.3437292575836182\n", 173 | "Epoch: 13/20 | Step: 162000;250000 | Loss: 0.636999249458313\n", 174 | "Epoch: 14/20 | Step: 164500;250000 | Loss: 0.6099785566329956\n", 175 | "Epoch: 14/20 | Step: 166500;250000 | Loss: 1.0579653978347778\n", 176 | "Epoch: 14/20 | Step: 168500;250000 | Loss: 1.0641443729400635\n", 177 | "Epoch: 14/20 | Step: 170500;250000 | Loss: 0.658701479434967\n", 178 | "Epoch: 14/20 | Step: 172500;250000 | Loss: 3.1185717582702637\n", 179 | "Epoch: 14/20 | Step: 174500;250000 | Loss: 0.8587619066238403\n", 180 | "Epoch: 15/20 | Step: 177000;250000 | Loss: 1.8661249876022339\n", 181 | "Epoch: 15/20 | Step: 179000;250000 | Loss: 0.6939230561256409\n", 182 | "Epoch: 15/20 | Step: 181000;250000 | Loss: 0.5399549603462219\n", 183 | "Epoch: 15/20 | Step: 183000;250000 | Loss: 0.7700042128562927\n", 184 | "Epoch: 15/20 | Step: 185000;250000 | Loss: 0.7247531414031982\n", 185 | "Epoch: 15/20 | Step: 187000;250000 | Loss: 0.746134877204895\n", 186 | "Epoch: 16/20 | Step: 189500;250000 | Loss: 0.4735276401042938\n", 187 | "Epoch: 16/20 | Step: 191500;250000 | Loss: 0.6705207824707031\n", 188 | "Epoch: 16/20 | Step: 193500;250000 | Loss: 0.863497793674469\n", 189 | "Epoch: 16/20 | Step: 195500;250000 | Loss: 0.7962812185287476\n", 190 | "Epoch: 16/20 | Step: 197500;250000 | Loss: 0.5657091736793518\n", 191 | "Epoch: 16/20 | Step: 199500;250000 | Loss: 1.288136601448059\n", 192 | "Epoch: 17/20 | Step: 202000;250000 | Loss: 0.8295432925224304\n", 193 | "Epoch: 17/20 | Step: 204000;250000 | Loss: 0.4508132338523865\n", 194 | "Epoch: 17/20 | Step: 206000;250000 | Loss: 1.2814412117004395\n", 195 | "Epoch: 17/20 | Step: 208000;250000 | Loss: 0.27081042528152466\n", 196 | "Epoch: 17/20 | Step: 210000;250000 | Loss: 0.9623105525970459\n", 197 | "Epoch: 17/20 | Step: 212000;250000 | Loss: 0.9066767692565918\n", 198 | "Epoch: 18/20 | Step: 214500;250000 | Loss: 1.0262855291366577\n", 199 | "Epoch: 18/20 | Step: 216500;250000 | Loss: 0.7027924060821533\n", 200 | "Epoch: 18/20 | Step: 218500;250000 | Loss: 1.3816018104553223\n", 201 | "Epoch: 18/20 | Step: 220500;250000 | Loss: 1.6824274063110352\n", 202 | "Epoch: 18/20 | Step: 222500;250000 | Loss: 1.5017560720443726\n", 203 | "Epoch: 18/20 | Step: 224500;250000 | Loss: 0.2840704023838043\n", 204 | "Epoch: 19/20 | Step: 227000;250000 | Loss: 0.837134063243866\n", 205 | "Epoch: 19/20 | Step: 229000;250000 | Loss: 0.6496121287345886\n", 206 | "Epoch: 19/20 | Step: 231000;250000 | Loss: 0.29826340079307556\n", 207 | "Epoch: 19/20 | Step: 233000;250000 | Loss: 1.4169983863830566\n", 208 | "Epoch: 19/20 | Step: 235000;250000 | Loss: 2.216799736022949\n", 209 | "Epoch: 19/20 | Step: 237000;250000 | Loss: 0.7554011344909668\n", 210 | "Epoch: 20/20 | Step: 239500;250000 | Loss: 0.653556227684021\n", 211 | "Epoch: 20/20 | Step: 241500;250000 | Loss: 0.38188064098358154\n", 212 | "Epoch: 20/20 | Step: 243500;250000 | Loss: 0.9678006172180176\n", 213 | "Epoch: 20/20 | Step: 245500;250000 | Loss: 1.0471148490905762\n", 214 | "Epoch: 20/20 | Step: 247500;250000 | Loss: 1.0283788442611694\n", 215 | "Epoch: 20/20 | Step: 249500;250000 | Loss: 0.3411109149456024\n", 216 | "Training data accuracy is: 0.68204\n", 217 | "Test data accuracy is: 0.6101\n" 218 | ] 219 | } 220 | ], 221 | "source": [ 222 | "import torch\n", 223 | "import torch.nn as nn\n", 224 | "import torchvision.datasets as datasets\n", 225 | "from torch.utils.data import DataLoader\n", 226 | "from model import ConvModel\n", 227 | "import torchvision.transforms as tf\n", 228 | "\n", 229 | "\n", 230 | "def train():\n", 231 | " device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 232 | "\n", 233 | " learning_rate = 3e-4\n", 234 | " batch_size = 4\n", 235 | " num_epochs = 20\n", 236 | "\n", 237 | " model = ConvModel().to(device)\n", 238 | "\n", 239 | " criterion = nn.CrossEntropyLoss()\n", 240 | " optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n", 241 | "\n", 242 | " transforms = tf.Compose([\n", 243 | " tf.ToTensor(),\n", 244 | " tf.Normalize((0.5,), (0.5,), (0.5,))\n", 245 | " ])\n", 246 | "\n", 247 | " train_dataset = datasets.CIFAR10(\"dataset/\", train=True, transform=transforms, download=True)\n", 248 | " train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n", 249 | "\n", 250 | " test_dataset = datasets.CIFAR10(\"dataset/\", train=False, transform=transforms, download=True)\n", 251 | " test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\n", 252 | "\n", 253 | " step = 0\n", 254 | " total_steps = len(train_loader) * num_epochs\n", 255 | "\n", 256 | " for epoch in range(num_epochs):\n", 257 | " for idx, (images, labels) in enumerate(train_loader):\n", 258 | " images = images.to(device)\n", 259 | " labels = labels.to(device)\n", 260 | "\n", 261 | " outputs = model(images)\n", 262 | "\n", 263 | " loss = criterion(outputs, labels)\n", 264 | " optimizer.zero_grad()\n", 265 | " loss.backward()\n", 266 | " optimizer.step()\n", 267 | "\n", 268 | " if (idx + 1) % 2000 == 0:\n", 269 | " print(f'Epoch: {epoch + 1}/{num_epochs} | Step: {step + 1};{total_steps} | Loss: {loss}')\n", 270 | " \n", 271 | " step += 1\n", 272 | "\n", 273 | " with torch.no_grad():\n", 274 | " n_correct = 0\n", 275 | " n_samples = 0\n", 276 | " n_class_correct = [0 for _ in range(10)]\n", 277 | " n_class_samples = [0 for _ in range(10)]\n", 278 | "\n", 279 | " for images, labels in train_loader:\n", 280 | " images = images.to(device)\n", 281 | " labels = labels.to(device)\n", 282 | "\n", 283 | " output = model(images)\n", 284 | " _, predicted = torch.max(output, 1)\n", 285 | " n_samples += labels.size(0)\n", 286 | " n_correct += (predicted == labels).sum().item()\n", 287 | "\n", 288 | " for i in range(batch_size):\n", 289 | " label = labels[i]\n", 290 | " pred = predicted[i]\n", 291 | "\n", 292 | " if label == pred:\n", 293 | " n_class_correct[label.item()] += 1\n", 294 | "\n", 295 | " n_class_samples[label.item()] += 1\n", 296 | "\n", 297 | " print(f'Training data accuracy is: {n_correct / n_samples}') \n", 298 | " \n", 299 | " with torch.no_grad():\n", 300 | " n_correct = 0\n", 301 | " n_samples = 0\n", 302 | " n_class_correct = [0 for _ in range(10)]\n", 303 | " n_class_samples = [0 for _ in range(10)]\n", 304 | "\n", 305 | " for images, labels in test_loader:\n", 306 | " images = images.to(device)\n", 307 | " labels = labels.to(device)\n", 308 | "\n", 309 | " output = model(images)\n", 310 | " _, predicted = torch.max(output, 1)\n", 311 | " n_samples += labels.size(0)\n", 312 | " n_correct += (predicted == labels).sum().item()\n", 313 | "\n", 314 | " for i in range(batch_size):\n", 315 | " label = labels[i]\n", 316 | " pred = predicted[i]\n", 317 | "\n", 318 | " if label == pred:\n", 319 | " n_class_correct[label.item()] += 1\n", 320 | "\n", 321 | " n_class_samples[label.item()] += 1\n", 322 | "\n", 323 | " print(f'Test data accuracy is: {n_correct / n_samples}')\n", 324 | "\n", 325 | " \n", 326 | "if __name__ == \"__main__\":\n", 327 | " train()\n" 328 | ] 329 | } 330 | ], 331 | "metadata": { 332 | "kernelspec": { 333 | "display_name": "Python 3 (ipykernel)", 334 | "language": "python", 335 | "name": "python3" 336 | }, 337 | "language_info": { 338 | "codemirror_mode": { 339 | "name": "ipython", 340 | "version": 3 341 | }, 342 | "file_extension": ".py", 343 | "mimetype": "text/x-python", 344 | "name": "python", 345 | "nbconvert_exporter": "python", 346 | "pygments_lexer": "ipython3", 347 | "version": "3.10.10" 348 | } 349 | }, 350 | "nbformat": 4, 351 | "nbformat_minor": 5 352 | } 353 | -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/Choosing Parameters.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "latest-joshua", 7 | "metadata": {}, 8 | "source": [ 9 | "# Advice for Applying Machine Learning" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "id": "transsexual-worry", 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "from sklearn.model_selection import train_test_split\n", 20 | "from sklearn.metrics import mean_squared_error\n", 21 | "import matplotlib.pyplot as plt\n", 22 | "import numpy as np\n", 23 | "from utils import generate_data, LinearModel\n", 24 | "\n", 25 | "import warnings\n", 26 | "warnings.filterwarnings(\"ignore\")" 27 | ] 28 | }, 29 | { 30 | "attachments": {}, 31 | "cell_type": "markdown", 32 | "id": "understanding-comment", 33 | "metadata": {}, 34 | "source": [ 35 | "## Generating data" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "id": "separated-grace", 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "X, y, X_ideal, y_ideal = generate_data(18, 3, 0.7)" 46 | ] 47 | }, 48 | { 49 | "attachments": {}, 50 | "cell_type": "markdown", 51 | "id": "after-anatomy", 52 | "metadata": {}, 53 | "source": [ 54 | "## Plotting data" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "id": "narrow-gateway", 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "plt.scatter(X, y, label=\"Data\")\n", 65 | "plt.plot(X_ideal, y_ideal, 'r', label=\"Ideal\")\n", 66 | "plt.legend()\n", 67 | "plt.show()" 68 | ] 69 | }, 70 | { 71 | "attachments": {}, 72 | "cell_type": "markdown", 73 | "id": "about-adrian", 74 | "metadata": {}, 75 | "source": [ 76 | "## Splitting dataset" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "id": "threatened-macro", 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.33, random_state=1)\n", 87 | "print(\"X_train.shape\", X_train.shape, \"y_train.shape\", y_train.shape)\n", 88 | "print(\"X_test.shape\", X_test.shape, \"y_test.shape\", y_test.shape)" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "id": "adolescent-split", 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "plt.scatter(X_train, y_train, label=\"Training\")\n", 99 | "plt.scatter(X_test, y_test, label=\"Test\")\n", 100 | "plt.legend()\n", 101 | "plt.show()" 102 | ] 103 | }, 104 | { 105 | "attachments": {}, 106 | "cell_type": "markdown", 107 | "id": "adverse-english", 108 | "metadata": {}, 109 | "source": [ 110 | "# Compare performance on training and test data" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "id": "informal-layout", 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "# create a model in sklearn, train on training data\n", 121 | "degree = 10\n", 122 | "\n", 123 | "model = LinearModel(degree)\n", 124 | "\n", 125 | "# Training\n", 126 | "model.fit(X_train, y_train)\n", 127 | "\n", 128 | "# MSE\n", 129 | "\n", 130 | "y_hat = model.predict(X_train)\n", 131 | "mse_train = mean_squared_error(y_train, y_hat)\n", 132 | "\n", 133 | "print(f\"MSE X Train: {mse_train}\")\n", 134 | "\n", 135 | "y_hat = model.predict(X_test)\n", 136 | "mse_test = mean_squared_error(y_test, y_hat)\n", 137 | "\n", 138 | "print(f\"MSE X Train: {mse_test}\")" 139 | ] 140 | }, 141 | { 142 | "attachments": {}, 143 | "cell_type": "markdown", 144 | "id": "wicked-banner", 145 | "metadata": {}, 146 | "source": [ 147 | "## Plotting" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "id": "settled-piano", 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "plt.plot(X_ideal, model.predict(X_ideal), label=\"degree=10\")\n", 158 | "plt.scatter(X_test, y_test,color='r',label=\"Test\")\n", 159 | "plt.scatter(X_train, y_train, color='b',label=\"Train\")\n", 160 | "plt.plot(X_ideal, y_ideal, label=\"ideal\")\n", 161 | "plt.legend()\n", 162 | "plt.show()" 163 | ] 164 | }, 165 | { 166 | "attachments": {}, 167 | "cell_type": "markdown", 168 | "id": "wooden-saturn", 169 | "metadata": {}, 170 | "source": [ 171 | "# How to choose the right degree?\n", 172 | "\n", 173 | "| data | % of total | Description |\n", 174 | "|------------------|:----------:|:---------|\n", 175 | "| training | 60 | Data used to tune model parameters $w$ and $b$ in training or fitting |\n", 176 | "| cross-validation | 20 | Data used to tune other model parameters like degree of polynomial, regularization or the architecture of a neural network.|\n", 177 | "| test | 20 | Data used to test the model after tuning to gauge performance on new data |" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": null, 183 | "id": "mexican-union", 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "# Generate data\n", 188 | "X,y, X_ideal,y_ideal = generate_data(40, 5, 0.7)\n", 189 | "print(\"X.shape\", X.shape, \"y.shape\", y.shape)\n", 190 | "\n", 191 | "#split the data using sklearn routine \n", 192 | "X_train, X_, y_train, y_ = train_test_split(X,y,test_size=0.40, random_state=1)\n", 193 | "X_cv, X_test, y_cv, y_test = train_test_split(X_,y_,test_size=0.50, random_state=1)\n", 194 | "print(\"X_train.shape\", X_train.shape, \"y_train.shape\", y_train.shape)\n", 195 | "print(\"X_cv.shape\", X_cv.shape, \"y_cv.shape\", y_cv.shape)\n", 196 | "print(\"X_test.shape\", X_test.shape, \"y_test.shape\", y_test.shape)" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": null, 202 | "id": "demanding-segment", 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [ 206 | "plt.scatter(X_train, y_train, color = \"red\", label=\"train\")\n", 207 | "plt.scatter(X_cv, y_cv, color = \"blue\", label=\"cv\")\n", 208 | "plt.scatter(X_test, y_test, color = \"orange\", label=\"test\")\n", 209 | "plt.legend(loc='upper left')\n", 210 | "plt.show()" 211 | ] 212 | }, 213 | { 214 | "attachments": {}, 215 | "cell_type": "markdown", 216 | "id": "respected-publication", 217 | "metadata": {}, 218 | "source": [ 219 | "# Finding optimal degree" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "id": "necessary-period", 226 | "metadata": {}, 227 | "outputs": [], 228 | "source": [ 229 | "max_degree = 11\n", 230 | "err_train = np.zeros(max_degree) \n", 231 | "err_cv = np.zeros(max_degree) \n", 232 | "\n", 233 | "y_pred = np.zeros((max_degree, len(X_ideal)))\n", 234 | "\n", 235 | "for degree in range(1, max_degree):\n", 236 | "\n", 237 | " model = LinearModel(degree)\n", 238 | " model.fit(X_train, y_train)\n", 239 | " \n", 240 | " y_hat = model.predict(X_train)\n", 241 | " err_train[degree] = mean_squared_error(y_train, y_hat)\n", 242 | "\n", 243 | " y_hat = model.predict(X_cv)\n", 244 | " err_cv[degree] = mean_squared_error(y_cv, y_hat)\n", 245 | " \n", 246 | " # Just for plotting results\n", 247 | " \n", 248 | " y_hat = model.predict(X_ideal)\n", 249 | " y_pred[degree] = y_hat\n", 250 | "\n", 251 | "optimal_degree = np.argmin(err_cv[1:])+1\n", 252 | "print(optimal_degree)" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": null, 258 | "id": "general-slide", 259 | "metadata": {}, 260 | "outputs": [], 261 | "source": [ 262 | "plt.plot(np.arange(1,max_degree), err_train[1:])\n", 263 | "plt.scatter(np.arange(1,max_degree), err_train[1:])\n", 264 | "plt.plot(np.arange(1,max_degree), err_cv[1:])\n", 265 | "plt.scatter(np.arange(1,max_degree), err_cv[1:])\n", 266 | "plt.xlabel(\"Degree\")\n", 267 | "plt.ylabel(\"MSE\")\n", 268 | "plt.show()" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "id": "personal-uniform", 275 | "metadata": {}, 276 | "outputs": [], 277 | "source": [ 278 | "# plt.plot(X_ideal, y_pred.T)\n", 279 | "plt.plot(X_ideal, y_pred[optimal_degree])\n", 280 | "plt.scatter(X_train, y_train)\n", 281 | "plt.scatter(X_cv, y_cv)\n", 282 | "plt.show()" 283 | ] 284 | }, 285 | { 286 | "attachments": {}, 287 | "cell_type": "markdown", 288 | "id": "green-browser", 289 | "metadata": {}, 290 | "source": [ 291 | "# Tuning in lambda\n", 292 | "\n", 293 | "$$ J(\\mathbf{w},b) = \n", 294 | " \\frac{1}{2m}\\sum_{i=0}^{m-1} ( f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})^2 + \\frac{\\lambda}{2m} \\sum_{j=0}^{n-1} (\\mathbf{w}_j)^2\n", 295 | "$$\n" 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": null, 301 | "id": "contrary-lawrence", 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "lambda_range = np.array([0.0, 1e-6, 1e-5, 1e-4,1e-3,1e-2, 1e-1,1,10,100])\n", 306 | "num_steps = len(lambda_range)\n", 307 | "degree = 10\n", 308 | "\n", 309 | "err_train = np.zeros(num_steps) \n", 310 | "err_cv = np.zeros(num_steps)\n", 311 | "\n", 312 | "y_pred = np.zeros((num_steps, len(X_ideal)))\n", 313 | "\n", 314 | "for i in range(num_steps):\n", 315 | " lambda_= lambda_range[i]\n", 316 | "\n", 317 | " model = LinearModel(degree, lambda_)\n", 318 | " model.fit(X_train, y_train)\n", 319 | " \n", 320 | " y_hat = model.predict(X_train)\n", 321 | " err_train[i] = mean_squared_error(y_hat, y_train)\n", 322 | "\n", 323 | " y_hat = model.predict(X_cv)\n", 324 | " err_cv[i] = mean_squared_error(y_hat, y_cv)\n", 325 | "\n", 326 | " # Just for plotting results\n", 327 | "\n", 328 | " y_hat = model.predict(X_ideal)\n", 329 | " y_pred[i] = y_hat\n", 330 | " \n", 331 | " \n", 332 | "optimal_reg_idx = np.argmin(err_cv) \n", 333 | "print(lambda_range[optimal_reg_idx])" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "id": "engaged-reduction", 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [ 343 | "# plt.plot(X_ideal, y_pred.T)\n", 344 | "plt.plot(X_ideal, y_pred[optimal_reg_idx], color='r')\n", 345 | "plt.scatter(X_train, y_train)\n", 346 | "plt.scatter(X_cv, y_cv)\n", 347 | "plt.show()" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "id": "positive-singapore", 354 | "metadata": {}, 355 | "outputs": [], 356 | "source": [ 357 | "x = np.arange(num_steps)\n", 358 | "plt.plot(x, err_cv, label=\"cv\")\n", 359 | "plt.plot(x, err_train, label=\"train\")\n", 360 | "plt.xticks(x, lambda_range)\n", 361 | "\n", 362 | "plt.scatter(x, err_cv)\n", 363 | "plt.scatter(x, err_train)\n", 364 | "plt.legend()\n", 365 | "\n", 366 | "plt.xlabel(\"Lambda\")\n", 367 | "plt.ylabel(\"MSE\")\n", 368 | "\n", 369 | "plt.show()" 370 | ] 371 | } 372 | ], 373 | "metadata": { 374 | "kernelspec": { 375 | "display_name": "tensorflow", 376 | "language": "python", 377 | "name": "tensorflow" 378 | }, 379 | "language_info": { 380 | "codemirror_mode": { 381 | "name": "ipython", 382 | "version": 3 383 | }, 384 | "file_extension": ".py", 385 | "mimetype": "text/x-python", 386 | "name": "python", 387 | "nbconvert_exporter": "python", 388 | "pygments_lexer": "ipython3", 389 | "version": "3.9.16" 390 | } 391 | }, 392 | "nbformat": 4, 393 | "nbformat_minor": 5 394 | } 395 | -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/Tensorflow Image Classification.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "double-thunder", 7 | "metadata": {}, 8 | "source": [ 9 | "# Image classification in tensorflow" 10 | ] 11 | }, 12 | { 13 | "attachments": {}, 14 | "cell_type": "markdown", 15 | "id": "southwest-heaven", 16 | "metadata": {}, 17 | "source": [ 18 | "## Imports" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "id": "bulgarian-sperm", 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "import tensorflow as tf\n", 29 | "import matplotlib.pyplot as plt\n", 30 | "import numpy as np\n", 31 | "from utils import plot_data_with_predictions, plot_data\n", 32 | "import datetime\n", 33 | "\n", 34 | "import warnings\n", 35 | "warnings.filterwarnings(\"ignore\")" 36 | ] 37 | }, 38 | { 39 | "attachments": {}, 40 | "cell_type": "markdown", 41 | "id": "artistic-spain", 42 | "metadata": {}, 43 | "source": [ 44 | "## Loading dataset" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "id": "pretty-trunk", 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n", 55 | "\n", 56 | "# Normalizing data from 0...255 to 0...1\n", 57 | "\n", 58 | "(x_train, x_test) = (x_train/255, x_test/255) \n", 59 | "\n", 60 | "# Reshaping\n", 61 | "\n", 62 | "img_size = x_train.shape[1]\n", 63 | "\n", 64 | "print(\"x_train shape: \", x_train.shape)\n", 65 | "print(f\"Amount of training examples: {x_train.shape[0]}\")\n", 66 | "print(f\"Amount of test examples: {x_test.shape[0]}\")\n", 67 | "print(f\"Images sizes: {x_train.shape[1]}x{x_train.shape[2]}\")" 68 | ] 69 | }, 70 | { 71 | "attachments": {}, 72 | "cell_type": "markdown", 73 | "id": "vocal-surfing", 74 | "metadata": {}, 75 | "source": [ 76 | "# Visualizing data" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "id": "floppy-office", 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "plot_data(x_train, y_train, 10)" 87 | ] 88 | }, 89 | { 90 | "attachments": {}, 91 | "cell_type": "markdown", 92 | "id": "conceptual-fossil", 93 | "metadata": {}, 94 | "source": [ 95 | "# Initialize model" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "id": "regulated-error", 102 | "metadata": {}, 103 | "outputs": [], 104 | "source": [ 105 | "from tensorflow.keras.layers import Flatten, Dense\n", 106 | "from tensorflow.keras.models import Sequential\n", 107 | "\n", 108 | "model = Sequential([\n", 109 | " Flatten(input_shape=(28, 28), name='input_layer'),\n", 110 | " Dense(128, activation='relu', name='layer_dense'),\n", 111 | " Dense(10, activation='softmax', name='output_layer')\n", 112 | "])\n", 113 | "\n", 114 | "optimizer = tf.keras.optimizers.Adam()\n", 115 | "loss = tf.keras.losses.SparseCategoricalCrossentropy()\n", 116 | "\n", 117 | "model.compile(optimizer=optimizer,\n", 118 | " loss=loss,\n", 119 | " metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n", 120 | "\n", 121 | "model.summary()" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "id": "emerging-blame", 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "model.fit(x_train, y_train, epochs=5, batch_size=32, validation_data=(x_test, y_test), shuffle=True)" 132 | ] 133 | }, 134 | { 135 | "attachments": {}, 136 | "cell_type": "markdown", 137 | "id": "varied-belgium", 138 | "metadata": {}, 139 | "source": [ 140 | "# Predictions on test data" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "id": "neither-recruitment", 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "print(\"Predicting\")\n", 151 | "predictions = model.predict(x_test)\n", 152 | "\n", 153 | "sample_prediction = predictions[0]\n", 154 | "correct_label = y_test[0]\n", 155 | "\n", 156 | "print()\n", 157 | "print(f\"Prediction: {sample_prediction}\\n\")\n", 158 | "print(f\"Correct label: {correct_label}\")\n", 159 | "print(f\"Max argument of sample: {np.argmax(sample_prediction)}\")" 160 | ] 161 | }, 162 | { 163 | "attachments": {}, 164 | "cell_type": "markdown", 165 | "id": "statewide-lexington", 166 | "metadata": {}, 167 | "source": [ 168 | "# Visualize predictions" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "id": "adjusted-saint", 175 | "metadata": {}, 176 | "outputs": [], 177 | "source": [ 178 | "plot_data_with_predictions(x_test, y_test, predictions, 10)" 179 | ] 180 | }, 181 | { 182 | "attachments": {}, 183 | "cell_type": "markdown", 184 | "id": "union-christianity", 185 | "metadata": {}, 186 | "source": [ 187 | "# Check bad predictions" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "id": "spoken-checklist", 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "predicted_labels = np.argmax(predictions, axis=1)\n", 198 | "bad_indexes = predicted_labels != y_test\n", 199 | "\n", 200 | "wrong_x_test = x_test[bad_indexes]\n", 201 | "wrong_y_test = y_test[bad_indexes]\n", 202 | "wrong_predictions = predictions[bad_indexes]\n", 203 | "\n", 204 | "plot_data_with_predictions(wrong_x_test, wrong_y_test, wrong_predictions, 10)" 205 | ] 206 | }, 207 | { 208 | "attachments": {}, 209 | "cell_type": "markdown", 210 | "id": "twelve-cleveland", 211 | "metadata": {}, 212 | "source": [ 213 | "# Tensorboard" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": null, 219 | "id": "developed-davis", 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "log_dir = \"logs/fit/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", 224 | "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n", 225 | "\n", 226 | "model.fit(x_train, y_train, epochs=5, validation_data=(x_test, y_test), callbacks=[tensorboard_callback])" 227 | ] 228 | }, 229 | { 230 | "attachments": {}, 231 | "cell_type": "markdown", 232 | "id": "purple-movie", 233 | "metadata": {}, 234 | "source": [ 235 | "# Additional resources" 236 | ] 237 | }, 238 | { 239 | "attachments": {}, 240 | "cell_type": "markdown", 241 | "id": "supreme-offering", 242 | "metadata": {}, 243 | "source": [ 244 | "## Tensorflow Neural Network Playground\n", 245 | "\n", 246 | "[https://playground.tensorflow.org](https://playground.tensorflow.org)\n", 247 | "\n", 248 | "![Tensorflow Neural Network Playground](assets/tf_playground.png)\n" 249 | ] 250 | }, 251 | { 252 | "attachments": {}, 253 | "cell_type": "markdown", 254 | "id": "fabulous-attempt", 255 | "metadata": {}, 256 | "source": [ 257 | "## 3b1b\n", 258 | "\n", 259 | "[Deep learning YouTube playlist](https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)" 260 | ] 261 | }, 262 | { 263 | "attachments": {}, 264 | "cell_type": "markdown", 265 | "id": "celtic-physiology", 266 | "metadata": {}, 267 | "source": [ 268 | "# Neural network visualizations\n", 269 | "\n", 270 | "https://adamharley.com/nn_vis/\n", 271 | "\n", 272 | "![NN Visualization](assets/nnvis.png)" 273 | ] 274 | } 275 | ], 276 | "metadata": { 277 | "kernelspec": { 278 | "display_name": "tensorflow", 279 | "language": "python", 280 | "name": "tensorflow" 281 | }, 282 | "language_info": { 283 | "codemirror_mode": { 284 | "name": "ipython", 285 | "version": 3 286 | }, 287 | "file_extension": ".py", 288 | "mimetype": "text/x-python", 289 | "name": "python", 290 | "nbconvert_exporter": "python", 291 | "pygments_lexer": "ipython3", 292 | "version": "3.9.16" 293 | } 294 | }, 295 | "nbformat": 4, 296 | "nbformat_minor": 5 297 | } 298 | -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/assets/nnvis.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/assets/nnvis.png -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/assets/tf_playground.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gradient-PG/gradient-live-session/24e6bc25f663f1ae9a542cc368b582f72054c1d3/2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/assets/tf_playground.png -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/utils/__init__.py: -------------------------------------------------------------------------------- 1 | from .parameters import generate_data, LinearModel 2 | from .wplot import plot_data, plot_data_with_predictions -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/utils/parameters.py: -------------------------------------------------------------------------------- 1 | from sklearn.linear_model import LinearRegression, Ridge 2 | from sklearn.preprocessing import PolynomialFeatures, StandardScaler 3 | import numpy as np 4 | 5 | class LinearModel: 6 | 7 | regressor: Ridge 8 | degree: int 9 | 10 | def __init__(self, degree: int, lambda_: float = 0) -> None: 11 | if lambda_ == 0: 12 | self.regressor = LinearRegression() 13 | else: 14 | self.regressor = Ridge(alpha=lambda_, random_state=1) 15 | 16 | self.degree = degree 17 | self.scaler = StandardScaler() 18 | self.poly = PolynomialFeatures(degree, include_bias=False) 19 | 20 | def fit(self, X, y): 21 | poly_features = self.poly.fit_transform(X.reshape(-1, 1)) 22 | poly_scaled = self.scaler.fit_transform(poly_features) 23 | 24 | self.regressor.fit(poly_scaled, y) 25 | 26 | def predict(self, X): 27 | poly_features = self.poly.fit_transform(X.reshape(-1, 1)) 28 | poly_scaled = self.scaler.transform(poly_features) 29 | 30 | return self.regressor.predict(poly_scaled) 31 | 32 | 33 | def generate_data(m, seed=1, scale=0.7): 34 | """ generate a data set based on a x^2 with added noise """ 35 | c = 0 36 | x_train = np.linspace(0,49,m) 37 | np.random.seed(seed) 38 | y_ideal = x_train**2 + c 39 | y_train = y_ideal + scale * y_ideal*(np.random.sample((m,))-0.5) 40 | x_ideal = np.linspace(0,49,200) 41 | y_ideal = x_ideal**2 + c 42 | 43 | return x_train, y_train, x_ideal, y_ideal -------------------------------------------------------------------------------- /2023_Edition/Workshops/2 - Hyperparameters Tuning and CNN/utils/wplot.py: -------------------------------------------------------------------------------- 1 | import matplotlib.pyplot as plt 2 | import numpy as np 3 | 4 | def plot_data(x, y, amount): 5 | plt.figure(figsize=(10,10)) 6 | 7 | for i in range(amount): 8 | plt.subplot(5,5,i+1) 9 | plt.xticks([]) 10 | plt.yticks([]) 11 | plt.grid(False) 12 | plt.imshow(x[i], cmap=plt.cm.binary) 13 | plt.xlabel(y[i]) 14 | plt.show() 15 | 16 | 17 | def plot_data_with_predictions(x, y, predictions, amount): 18 | plt.figure(figsize=(10,10)) 19 | 20 | for i in range(amount): 21 | plt.subplot(5,5,i+1) 22 | plt.xticks([]) 23 | plt.yticks([]) 24 | plt.grid(False) 25 | plt.imshow(x[i], cmap=plt.cm.binary) 26 | plt.xlabel(str(y[i]) + f", P={np.argmax(predictions[i])}") 27 | plt.show() -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Gradient Live Session 2 | 3 | [![Visit us](https://img.shields.io/badge/Gradient%20PG-visit%20us%21-blue.svg)](https://gradient.eti.pg.gda.pl/) 4 | 5 | ### Materials for live sessions following lectures 6 | 7 | #### 2022 Edition coming soon! 8 | 9 |
10 | 2021 Edition 11 | 12 | * [1. Intro to ML pt. 1](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/1.%20Intro%20to%20ML%20pt.%201) 13 | * [2. Intro to ML pt. 2](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/2.%20Intro%20to%20ML%20pt.%202) 14 | * [3. Intro to DL](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/3.%20Intro%20to%20DL) 15 | * [4. CNN](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/4.%20CNN) 16 | * [5. RNN](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/5.%20RNN) 17 | * [6. ML Tools](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/6.%20ML%20Tools) 18 | * [7. ML Deployment](https://github.com/Gradient-PG/gradient-live-session/tree/main/2021%20Edition/7.%20ML%20Deployment) 19 |
20 | 21 | ### What is a Jupyter Notebook? 22 | 23 | Jupyter notebooks are great tools allowing you to develop highly readable and easy to understand code in Python. It is mostly used when creating proof of concepts where you want to be able to merge code snippets together with comments, images, graphs, LaTeX equations etc. 24 | 25 | Jupyter notebooks are divided into cells. Each cell can be a code cell or a text cell (well, [actually](https://i.kym-cdn.com/entries/icons/original/000/021/665/DpQ9YJl.png) there are also two other types, but it's not important for now). Code cells let you add executable Python code to them and text cells are places where you can add all description you need. Code cells can be executed one by one or all at once (i.e. one after the other automatically). Each cell can also be rerun without rerunning whole notebook - it is very useful when you want to, for example, rewrite a function responsible for creating statistics, without retraining the whole neural network model, which could take a lot of time. You can find more detailed info about Jupyter Notebooks [here](https://realpython.com/jupyter-notebook-introduction/). 26 | 27 | ### What is Google Colaboratory? 28 | 29 | Google Colaboratory, or “Colab” for short, is a product from Google Research. Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning, data analysis and education. More technically, Colab is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources including GPUs and TPUs. See more [in this video](https://www.youtube.com/watch?v=inN8seMm7UI). 30 | 31 | Each of the notebooks on this repository contains a link at the top, which redirects you to Colab where you can run and experiment with the notebook without having to worry about installing Jupyter or any other software. 32 | --------------------------------------------------------------------------------