├── requirements.txt ├── .github └── workflows │ └── colab_badge_workflow.yml ├── README.md ├── .gitignore ├── environment.yml ├── Week 06 - Advanced TensorFlow └── 2. Custom Training Loops.ipynb ├── Week 02 - Optimization and Regularization ├── 2. Introduction to Gradio.ipynb ├── 1. Regularization Techniques.ipynb └── 3. Reproducibility, Callbacks, and Tensorboard.ipynb ├── Week 04 - Introduction to Sequence Modelling └── 2. Text Data Pipelines.ipynb ├── LICENSE └── Week 01 - Review and Intro to Deep Learning └── 1. Introduction.ipynb /requirements.txt: -------------------------------------------------------------------------------- 1 | jupyter 2 | numpy 3 | matplotlib 4 | seaborn 5 | pandas 6 | scikit-learn 7 | gradio 8 | tensorflow 9 | tqdm 10 | transformers 11 | -------------------------------------------------------------------------------- /.github/workflows/colab_badge_workflow.yml: -------------------------------------------------------------------------------- 1 | name: colab-badge-workflow 2 | on: [push] 3 | jobs: 4 | add-colab-badge: 5 | runs-on: ubuntu-latest 6 | steps: 7 | - name: Checkout first 8 | id: checkout 9 | uses: actions/checkout@v2 10 | - name: Add/Update badges 11 | id: badges 12 | uses: trsvchn/colab-badge-action@v3 13 | with: 14 | check: 'all' 15 | update: true 16 | target_branch: main 17 | target_repository: TheAIDojo/AI_4_Climate_Bootcamp 18 | - name: Commit changes 19 | uses: EndBug/add-and-commit@v7 20 | with: 21 | author_name: ${{ github.repository_owner }} 22 | author_email: accounts@sabri.io 23 | message: 'Added Colab Badges' 24 | add: '*.ipynb' 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AI 4 Climate Bootcamp - Module 3: Deep Learning & TensorFlow 2 | 3 | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/TheAIDojo/AI-for-Climate-Change/HEAD) 4 | [![Jupyter Notebook](https://img.shields.io/badge/Launch-Jupyter%20Notebook-blue.svg)](https://nbviewer.org/github/TheAIDojo/AI-for-Climate-Change/tree/main/) 5 | [![TensorFlow](https://img.shields.io/badge/TensorFlow-2.x-green.svg)](https://www.tensorflow.org/) 6 | [![Python](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org/) 7 | [![Transformers](https://img.shields.io/badge/Transformers-2.8.0-orange)](https://huggingface.co/transformers/) 8 | [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) 9 | 10 | 11 | Welcome to the AI 4 Climate Bootcamp - Module 3: Deep Learning & TensorFlow! This repository contains materials to help you understand the fundamental concepts of deep learning and its practical implementation using TensorFlow. 12 | 13 | ## Lessons Outline 14 | 15 | ### [Week 1: Introduction to Deep Learning and TensorFlow](./Week%2001%20-%20Review%20and%20Intro%20to%20Deep%20Learning/) 16 | 17 | In the first week, we reviewed data science and machine learning concepts and tools, and we introduced deep learning concepts and basic TensorFlow syntax. We covered the following topics: 18 | 19 | - Introduction to deep learning. 20 | - Building models using TensorFlow's Sequential API. 21 | - Dense layers. 22 | - Activation functions. 23 | - Loss functions and optimizers. 24 | 25 | ### [Week 2: Building User Interfaces and Reproducibility](./Week%2002%20-%20Optimization%20and%20Regularization/) 26 | 27 | In the second week, we went deeper into TensorFlow by learning about regularization techniques, building user interfaces for our models, and working on training monitoring and reproduction using Tensorboard. We covered the following topics: 28 | 29 | - Regularization techniques (dropout and early stopping). 30 | - Building user interfaces for models with Gradio. 31 | - Model reproducibility. 32 | - Training monitoring with Tensorboard. 33 | 34 | ### [Week 3: Computer Vision](./Week%2003%20-%20Introduction%20to%20Computer%20Vision/) 35 | 36 | In the third week, we learned about computer vision and how to build models for image classification and object detection. We covered the following topics: 37 | 38 | - Introduction to computer vision. 39 | - Image classification using CNNs. 40 | - CNN optimization and regularization techniques. 41 | - Transfer learning. 42 | - Data pipelines. 43 | 44 | 45 | ### [Week 4: Introduction to Sequence Modelling](./Week%2004%20-%20Introduction%20to%20Sequence%20Modelling/) 46 | 47 | In the fourth week, we learned about sequence modelling and how to build models for text classification and forecasting. We covered the following topics: 48 | 49 | - Introduction to NLP. 50 | - Sequence modelling with Recurrent Neural Networks (RNNs). 51 | - Text Data Pipelines. 52 | 53 | ### [Week 5: Advanced ML Applications](./Week%2005%20-%20Advanced%20ML%20Applications/) 54 | 55 | In the fifth week, we learned about advanced ML applications and how to build model and we covered the following topics: 56 | - Forecasting with RNNs. 57 | - Chatbots with Transformers. 58 | - Object Detection with YOLOv5. 59 | 60 | ### [Week 6: Advanced TensorFlow](./Week%2006%20-%20Advanced%20TensorFlow/) -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ### JetBrains template 2 | # Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider 3 | # Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839 4 | 5 | # User-specific stuff 6 | .idea/**/workspace.xml 7 | .idea/**/tasks.xml 8 | .idea/**/usage.statistics.xml 9 | .idea/**/dictionaries 10 | .idea/**/shelf 11 | 12 | # AWS User-specific 13 | .idea/**/aws.xml 14 | 15 | # Generated files 16 | .idea/**/contentModel.xml 17 | 18 | # Sensitive or high-churn files 19 | .idea/**/dataSources/ 20 | .idea/**/dataSources.ids 21 | .idea/**/dataSources.local.xml 22 | .idea/**/sqlDataSources.xml 23 | .idea/**/dynamic.xml 24 | .idea/**/uiDesigner.xml 25 | .idea/**/dbnavigator.xml 26 | 27 | # Gradle 28 | .idea/**/gradle.xml 29 | .idea/**/libraries 30 | 31 | # Gradle and Maven with auto-import 32 | # When using Gradle or Maven with auto-import, you should exclude module files, 33 | # since they will be recreated, and may cause churn. Uncomment if using 34 | # auto-import. 35 | # .idea/artifacts 36 | # .idea/compiler.xml 37 | # .idea/jarRepositories.xml 38 | # .idea/modules.xml 39 | # .idea/*.iml 40 | # .idea/modules 41 | # *.iml 42 | # *.ipr 43 | 44 | # CMake 45 | cmake-build-*/ 46 | 47 | # Mongo Explorer plugin 48 | .idea/**/mongoSettings.xml 49 | 50 | # File-based project format 51 | *.iws 52 | 53 | # IntelliJ 54 | out/ 55 | 56 | # mpeltonen/sbt-idea plugin 57 | .idea_modules/ 58 | 59 | # JIRA plugin 60 | atlassian-ide-plugin.xml 61 | 62 | # Cursive Clojure plugin 63 | .idea/replstate.xml 64 | 65 | # SonarLint plugin 66 | .idea/sonarlint/ 67 | 68 | # Crashlytics plugin (for Android Studio and IntelliJ) 69 | com_crashlytics_export_strings.xml 70 | crashlytics.properties 71 | crashlytics-build.properties 72 | fabric.properties 73 | 74 | # Editor-based Rest Client 75 | .idea/httpRequests 76 | 77 | # Android studio 3.1+ serialized cache file 78 | .idea/caches/build_file_checksums.ser 79 | 80 | ### Python template 81 | # Byte-compiled / optimized / DLL files 82 | __pycache__/ 83 | *.py[cod] 84 | *$py.class 85 | 86 | # C extensions 87 | *.so 88 | 89 | # Distribution / packaging 90 | .Python 91 | build/ 92 | develop-eggs/ 93 | dist/ 94 | downloads/ 95 | eggs/ 96 | .eggs/ 97 | lib/ 98 | lib64/ 99 | parts/ 100 | sdist/ 101 | var/ 102 | wheels/ 103 | share/python-wheels/ 104 | *.egg-info/ 105 | .installed.cfg 106 | *.egg 107 | MANIFEST 108 | 109 | # PyInstaller 110 | # Usually these files are written by a python script from a template 111 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 112 | *.manifest 113 | *.spec 114 | 115 | # Installer logs 116 | pip-log.txt 117 | pip-delete-this-directory.txt 118 | 119 | # Unit test / coverage reports 120 | htmlcov/ 121 | .tox/ 122 | .nox/ 123 | .coverage 124 | .coverage.* 125 | .cache 126 | nosetests.xml 127 | coverage.xml 128 | *.cover 129 | *.py,cover 130 | .hypothesis/ 131 | .pytest_cache/ 132 | cover/ 133 | 134 | # Translations 135 | *.mo 136 | *.pot 137 | 138 | # Django stuff: 139 | *.log 140 | local_settings.py 141 | db.sqlite3 142 | db.sqlite3-journal 143 | 144 | # Flask stuff: 145 | instance/ 146 | .webassets-cache 147 | 148 | # Scrapy stuff: 149 | .scrapy 150 | 151 | # Sphinx documentation 152 | docs/_build/ 153 | 154 | # PyBuilder 155 | .pybuilder/ 156 | target/ 157 | 158 | # Jupyter Notebook 159 | .ipynb_checkpoints 160 | 161 | # IPython 162 | profile_default/ 163 | ipython_config.py 164 | 165 | # pyenv 166 | # For a library or package, you might want to ignore these files since the code is 167 | # intended to run in multiple environments; otherwise, check them in: 168 | # .python-version 169 | 170 | # pipenv 171 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 172 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 173 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 174 | # install all needed dependencies. 175 | #Pipfile.lock 176 | 177 | # poetry 178 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 179 | # This is especially recommended for binary packages to ensure reproducibility, and is more 180 | # commonly ignored for libraries. 181 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 182 | #poetry.lock 183 | 184 | # pdm 185 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 186 | #pdm.lock 187 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 188 | # in version control. 189 | # https://pdm.fming.dev/#use-with-ide 190 | .pdm.toml 191 | 192 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 193 | __pypackages__/ 194 | 195 | # Celery stuff 196 | celerybeat-schedule 197 | celerybeat.pid 198 | 199 | # SageMath parsed files 200 | *.sage.py 201 | 202 | # Environments 203 | .env 204 | .venv 205 | env/ 206 | venv/ 207 | ENV/ 208 | env.bak/ 209 | venv.bak/ 210 | 211 | # Spyder project settings 212 | .spyderproject 213 | .spyproject 214 | 215 | # Rope project settings 216 | .ropeproject 217 | 218 | # mkdocs documentation 219 | /site 220 | 221 | # mypy 222 | .mypy_cache/ 223 | .dmypy.json 224 | dmypy.json 225 | 226 | # Pyre type checker 227 | .pyre/ 228 | 229 | # pytype static type analyzer 230 | .pytype/ 231 | 232 | # Cython debug symbols 233 | cython_debug/ 234 | 235 | # PyCharm 236 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 237 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 238 | # and can be added to the global gitignore or merged into this file. For a more nuclear 239 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 240 | #.idea/ 241 | 242 | /.idea/ 243 | /.idea/AI 4 Climate Bootcamp.iml 244 | /.idea/misc.xml 245 | /.idea/modules.xml 246 | /.idea/inspectionProfiles/profiles_settings.xml 247 | /.idea/vcs.xml 248 | 249 | # ignore folder "Assignments" and all its content 250 | /Assignments/ 251 | 252 | # ignore .vscode folder and all its content 253 | /.vscode/ 254 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: ML-Lectures 2 | channels: 3 | - defaults 4 | dependencies: 5 | - _tflow_select=2.3.0=mkl 6 | - absl-py=1.3.0=py37hecd8cb5_0 7 | - astor=0.8.1=py37hecd8cb5_0 8 | - blas=1.0=mkl 9 | - bottleneck=1.3.5=py37h67323c0_0 10 | - brotli=1.0.9=hca72f7f_7 11 | - brotli-bin=1.0.9=hca72f7f_7 12 | - brotlipy=0.7.0=py37h9ed2024_1003 13 | - c-ares=1.18.1=hca72f7f_0 14 | - ca-certificates=2022.10.11=hecd8cb5_0 15 | - certifi=2022.12.7=py37hecd8cb5_0 16 | - cffi=1.15.1=py37h6c40b1e_3 17 | - charset-normalizer=2.0.4=pyhd3eb1b0_0 18 | - cryptography=38.0.4=py37hf6deb26_0 19 | - cycler=0.11.0=pyhd3eb1b0_0 20 | - fftw=3.3.9=h9ed2024_1 21 | - filelock=3.9.0=py37hecd8cb5_0 22 | - flit-core=3.6.0=pyhd3eb1b0_0 23 | - fonttools=4.25.0=pyhd3eb1b0_0 24 | - freetype=2.12.1=hd8bbffd_0 25 | - future=0.18.2=py37_1 26 | - gast=0.2.2=py37_0 27 | - giflib=5.2.1=haf1e3a3_0 28 | - google-pasta=0.2.0=pyhd3eb1b0_0 29 | - grpcio=1.42.0=py37ha29bfda_0 30 | - h5py=3.7.0=py37h4a1dd59_0 31 | - hdf5=1.10.6=h10fe05b_1 32 | - huggingface_hub=0.10.1=py37hecd8cb5_0 33 | - idna=3.4=py37hecd8cb5_0 34 | - importlib-metadata=4.11.3=py37hecd8cb5_0 35 | - intel-openmp=2021.4.0=hecd8cb5_3538 36 | - joblib=1.1.1=py37hecd8cb5_0 37 | - jpeg=9e=hca72f7f_0 38 | - keras-applications=1.0.8=py_1 39 | - keras-preprocessing=1.1.2=pyhd3eb1b0_0 40 | - kiwisolver=1.4.4=py37hcec6c5f_0 41 | - lcms2=2.12=hf1fd2bf_0 42 | - lerc=3.0=he9d5cce_0 43 | - libbrotlicommon=1.0.9=hca72f7f_7 44 | - libbrotlidec=1.0.9=hca72f7f_7 45 | - libbrotlienc=1.0.9=hca72f7f_7 46 | - libcxx=14.0.6=h9765a3e_0 47 | - libdeflate=1.8=h9ed2024_5 48 | - libffi=3.4.2=hecd8cb5_6 49 | - libgfortran=5.0.0=11_3_0_hecd8cb5_28 50 | - libgfortran5=11.3.0=h9dfd629_28 51 | - libpng=1.6.37=ha441bb4_0 52 | - libprotobuf=3.20.1=h8346a28_0 53 | - libtiff=4.5.0=h2cd0358_0 54 | - libwebp=1.2.4=h56c3ce4_0 55 | - libwebp-base=1.2.4=hca72f7f_0 56 | - llvm-openmp=14.0.6=h0dcd299_0 57 | - lz4-c=1.9.4=hcec6c5f_0 58 | - markdown=3.4.1=py37hecd8cb5_0 59 | - markupsafe=2.1.1=py37hca72f7f_0 60 | - matplotlib=3.5.2=py37hecd8cb5_0 61 | - matplotlib-base=3.5.2=py37hfb0c5b7_0 62 | - mkl=2021.4.0=hecd8cb5_637 63 | - mkl-service=2.4.0=py37h9ed2024_0 64 | - mkl_fft=1.3.1=py37h4ab4a9b_0 65 | - mkl_random=1.2.2=py37hb2f4e1b_0 66 | - munkres=1.1.4=py_0 67 | - ncurses=6.3=hca72f7f_3 68 | - ninja=1.10.2=hecd8cb5_5 69 | - ninja-base=1.10.2=haf03e11_5 70 | - numexpr=2.8.4=py37he696674_0 71 | - numpy=1.21.5=py37h2e5f0a9_3 72 | - numpy-base=1.21.5=py37h3b1a694_3 73 | - openssl=1.1.1s=hca72f7f_0 74 | - opt_einsum=3.3.0=pyhd3eb1b0_1 75 | - packaging=22.0=py37hecd8cb5_0 76 | - pandas=1.3.5=py37h743cdd8_0 77 | - pillow=9.3.0=py37h81888ad_1 78 | - pip=22.3.1=py37hecd8cb5_0 79 | - protobuf=3.20.1=py37he9d5cce_0 80 | - pycparser=2.21=pyhd3eb1b0_0 81 | - pyopenssl=22.0.0=pyhd3eb1b0_0 82 | - pyparsing=3.0.9=py37hecd8cb5_0 83 | - pysocks=1.7.1=py37hecd8cb5_0 84 | - python=3.7.16=h218abb5_0 85 | - python-dateutil=2.8.2=pyhd3eb1b0_0 86 | - pytorch=1.12.1=cpu_py37ha26b6ec_0 87 | - pytz=2022.7=py37hecd8cb5_0 88 | - pyyaml=6.0=py37h6c40b1e_1 89 | - readline=8.2=hca72f7f_0 90 | - regex=2022.7.9=py37hca72f7f_0 91 | - requests=2.28.1=py37hecd8cb5_0 92 | - scikit-learn=1.0.2=py37hae1ba45_1 93 | - scipy=1.7.3=py37h214d14d_2 94 | - seaborn=0.12.2=py37hecd8cb5_0 95 | - setuptools=65.6.3=py37hecd8cb5_0 96 | - six=1.16.0=pyhd3eb1b0_1 97 | - sqlite=3.40.1=h880c91c_0 98 | - tensorboard=2.0.0=pyhb38c66f_1 99 | - tensorflow=2.0.0=mkl_py37hda344b4_0 100 | - tensorflow-base=2.0.0=mkl_py37h66b1bf0_0 101 | - tensorflow-estimator=2.0.0=pyh2649769_0 102 | - termcolor=2.1.0=py37hecd8cb5_0 103 | - threadpoolctl=2.2.0=pyh0d69192_0 104 | - tk=8.6.12=h5d9f67b_0 105 | - tokenizers=0.11.4=py37h8776b5c_1 106 | - tornado=6.2=py37hca72f7f_0 107 | - tqdm=4.64.1=py37hecd8cb5_0 108 | - transformers=4.24.0=py37hecd8cb5_0 109 | - typing-extensions=4.4.0=py37hecd8cb5_0 110 | - typing_extensions=4.4.0=py37hecd8cb5_0 111 | - urllib3=1.26.14=py37hecd8cb5_0 112 | - werkzeug=2.2.2=py37hecd8cb5_0 113 | - wheel=0.37.1=pyhd3eb1b0_0 114 | - wrapt=1.14.1=py37hca72f7f_0 115 | - xz=5.2.10=h6c40b1e_1 116 | - yaml=0.2.5=haf1e3a3_0 117 | - zipp=3.11.0=py37hecd8cb5_0 118 | - zlib=1.2.13=h4dc903c_0 119 | - zstd=1.5.2=hcb37349_0 120 | - pip: 121 | - aiofiles==22.1.0 122 | - aiohttp==3.8.3 123 | - aiosignal==1.3.1 124 | - alembic==1.9.2 125 | - altair==4.2.0 126 | - anyio==3.6.2 127 | - appnope==0.1.3 128 | - argon2-cffi==21.3.0 129 | - argon2-cffi-bindings==21.2.0 130 | - async-timeout==4.0.2 131 | - asynctest==0.13.0 132 | - attrs==22.2.0 133 | - babel==2.11.0 134 | - backcall==0.2.0 135 | - beautifulsoup4==4.11.1 136 | - black==22.12.0 137 | - bleach==5.0.1 138 | - click==8.1.3 139 | - debugpy==1.6.5 140 | - decorator==5.1.1 141 | - defusedxml==0.7.1 142 | - entrypoints==0.4 143 | - fastapi==0.89.1 144 | - fastjsonschema==2.16.2 145 | - ffmpy==0.3.0 146 | - frozenlist==1.3.3 147 | - fsspec==2023.1.0 148 | - gradio==3.16.2 149 | - greenlet==2.0.1 150 | - h11==0.14.0 151 | - httpcore==0.16.3 152 | - httpx==0.23.3 153 | - importlib-resources==5.10.2 154 | - ipykernel==6.16.2 155 | - ipython==7.34.0 156 | - ipython-genutils==0.2.0 157 | - ipywidgets==8.0.4 158 | - jedi==0.18.2 159 | - jinja2==3.1.2 160 | - json5==0.9.11 161 | - jsonschema==4.17.3 162 | - jupyter==1.0.0 163 | - jupyter-client==7.4.9 164 | - jupyter-console==6.4.4 165 | - jupyter-core==4.12.0 166 | - jupyter-server==1.23.5 167 | - jupyterlab==3.5.2 168 | - jupyterlab-pygments==0.2.2 169 | - jupyterlab-server==2.19.0 170 | - jupyterlab-widgets==3.0.5 171 | - linkify-it-py==1.0.3 172 | - mako==1.2.4 173 | - markdown-it-py==2.1.0 174 | - matplotlib-inline==0.1.6 175 | - mdit-py-plugins==0.3.3 176 | - mdurl==0.1.2 177 | - mistune==2.0.4 178 | - multidict==6.0.4 179 | - mypy-extensions==0.4.3 180 | - nbclassic==0.3.7 181 | - nbclient==0.6.3 182 | - nbconvert==7.2.8 183 | - nbformat==5.7.3 184 | - nbgrader==0.8.1 185 | - nest-asyncio==1.5.6 186 | - notebook==6.4.12 187 | - notebook-shim==0.2.2 188 | - orjson==3.8.5 189 | - pandocfilters==1.5.0 190 | - parso==0.8.3 191 | - pathspec==0.10.3 192 | - pexpect==4.8.0 193 | - pickleshare==0.7.5 194 | - pkgutil-resolve-name==1.3.10 195 | - platformdirs==2.6.2 196 | - prometheus-client==0.15.0 197 | - prompt-toolkit==3.0.36 198 | - psutil==5.9.4 199 | - ptyprocess==0.7.0 200 | - pycryptodome==3.16.0 201 | - pydantic==1.10.4 202 | - pydub==0.25.1 203 | - pygments==2.14.0 204 | - pyrsistent==0.19.3 205 | - python-multipart==0.0.5 206 | - pyzmq==25.0.0 207 | - qtconsole==5.4.0 208 | - qtpy==2.3.0 209 | - rapidfuzz==2.13.7 210 | - rfc3986==1.5.0 211 | - send2trash==1.8.0 212 | - sniffio==1.3.0 213 | - soupsieve==2.3.2.post1 214 | - sqlalchemy==1.4.46 215 | - starlette==0.22.0 216 | - terminado==0.17.1 217 | - tinycss2==1.2.1 218 | - tokenize-rt==5.0.0 219 | - tomli==2.0.1 220 | - toolz==0.12.0 221 | - traitlets==5.1.1 222 | - typed-ast==1.5.4 223 | - uc-micro-py==1.0.1 224 | - uvicorn==0.20.0 225 | - wcwidth==0.2.6 226 | - webencodings==0.5.1 227 | - websocket-client==1.4.2 228 | - websockets==10.4 229 | - widgetsnbextension==4.0.5 230 | - yarl==1.8.2 231 | prefix: /Applications/Anaconda/anaconda3/envs/ML-Lectures 232 | -------------------------------------------------------------------------------- /Week 06 - Advanced TensorFlow/2. Custom Training Loops.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "# Custom Training Loops\n", 9 | "\n", 10 | "Custom training loops are a powerful tool for training deep learning models. They are used when the standard `fit` method provided by `Keras` or other high-level libraries is not enough to achieve the desired behavior.\n", 11 | "\n", 12 | "Custom training loops provide full control over the training process, including the forward and backward pass, the optimization step, and the calculation of metrics. This allows for the implementation of complex training procedures, such as multi-stage training, transfer learning, or custom regularization methods.\n", 13 | "\n", 14 | "The steps to create a custom training loop after preparing the data and defining the model are:\n", 15 | "\n", 16 | "1. Define essential variables like the optimizer, loss function, and metrics that will be used to evaluate the model's performance.\n", 17 | "2. Create a training step function that will be used to perform a single training step. This function will be called for each batch of data in the training set. The training step function should perform the following steps:\n", 18 | " - Calculate the forward pass passing the input data to the model and calculating the predictions.\n", 19 | " - Calculate the loss using the predictions and the true labels.\n", 20 | " - Calculate the gradients using the loss and the model's variables.\n", 21 | " - Apply parameters update using the optimizer and the gradients.\n", 22 | " - Calculate the metrics using the predictions and the true labels.\n", 23 | "3. Create a test step function that will be used to perform a single test step. This function will be called for each batch of data in the test set. The test step function should perform the following steps:\n", 24 | " - Calculate the forward pass passing the input data to the model and calculating the predictions.\n", 25 | " - Calculate the loss using the predictions and the true labels.\n", 26 | " - Calculate the metrics using the predictions and the true labels.\n", 27 | "4. Create a training loop that will be used to perform the training and test steps for each epoch. The training loop should perform the following steps:\n", 28 | " - Iterate over the training set and call the training step function for each batch.\n", 29 | " - Iterate over the test set and call the test step function for each batch.\n", 30 | " - Print the loss and metrics for the current epoch.\n", 31 | "5. Profit!\n", 32 | "\n", 33 | "By using custom training loops, you can achieve better control and flexibility over the training process and achieve better results for your specific use case.\n", 34 | "\n", 35 | "## Table of Contents\n", 36 | "- [Dataset Preparation](#dataset-preparation)\n", 37 | "- [Model Definition](#model-definition)\n", 38 | "- [Custom Training Loop](#custom-training-loop)" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 1, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "import numpy as np\n", 48 | "import pandas as pd\n", 49 | "import matplotlib.pyplot as plt\n", 50 | "import tensorflow as tf\n", 51 | "import tensorflow_datasets as tfds # datasets\n", 52 | "from tqdm.notebook import tqdm # progress bar" 53 | ] 54 | }, 55 | { 56 | "attachments": {}, 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "## Dataset Preparation \n", 61 | "[Back to Top](#toc)\n", 62 | "\n", 63 | "We'll use the same dataset setup as in the previous notebook." 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "dataset, info = tfds.load(\n", 73 | " \"fashion_mnist\", # name of the dataset\n", 74 | " as_supervised=True, # returns (image, label)\n", 75 | " with_info=True, # returns info about the dataset\n", 76 | ")\n", 77 | "\n", 78 | "# prepare index labels\n", 79 | "labels_index = info.features[\"label\"].names" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 4, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "def preprocess(image, label):\n", 89 | " # preprocess images\n", 90 | " image = tf.reshape(image, (28, 28, 1))\n", 91 | " image = tf.cast(image, tf.float32)\n", 92 | " image = image / 255.0\n", 93 | " # preprocess labels\n", 94 | " label = tf.one_hot(label, 10)\n", 95 | " return image, label\n", 96 | "\n", 97 | "\n", 98 | "# we will create a function to prepare the dataset for training\n", 99 | "def dataset_prep(dataset):\n", 100 | " dataset = dataset.map(preprocess, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n", 101 | " # shuffle the dataset\n", 102 | " dataset = dataset.shuffle(1000)\n", 103 | "\n", 104 | " # batch the dataset\n", 105 | " dataset = dataset.batch(32)\n", 106 | "\n", 107 | " # prefetch the dataset\n", 108 | " dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n", 109 | "\n", 110 | " return dataset\n", 111 | "\n", 112 | "\n", 113 | "train_dataset = dataset_prep(dataset[\"train\"])\n", 114 | "test_dataset = dataset_prep(dataset[\"test\"])" 115 | ] 116 | }, 117 | { 118 | "attachments": {}, 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "## Model Definition \n", 123 | "[Back to top](#toc)" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 5, 129 | "metadata": {}, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "Model: \"sequential\"\n", 136 | "_________________________________________________________________\n", 137 | " Layer (type) Output Shape Param # \n", 138 | "=================================================================\n", 139 | " conv2d (Conv2D) (None, 28, 28, 32) 320 \n", 140 | " \n", 141 | " max_pooling2d (MaxPooling2D (None, 14, 14, 32) 0 \n", 142 | " ) \n", 143 | " \n", 144 | " conv2d_1 (Conv2D) (None, 14, 14, 64) 18496 \n", 145 | " \n", 146 | " max_pooling2d_1 (MaxPooling (None, 7, 7, 64) 0 \n", 147 | " 2D) \n", 148 | " \n", 149 | " flatten (Flatten) (None, 3136) 0 \n", 150 | " \n", 151 | " dense (Dense) (None, 128) 401536 \n", 152 | " \n", 153 | " dense_1 (Dense) (None, 10) 1290 \n", 154 | " \n", 155 | "=================================================================\n", 156 | "Total params: 421,642\n", 157 | "Trainable params: 421,642\n", 158 | "Non-trainable params: 0\n", 159 | "_________________________________________________________________\n" 160 | ] 161 | } 162 | ], 163 | "source": [ 164 | "model = tf.keras.Sequential(\n", 165 | " [\n", 166 | " tf.keras.layers.Conv2D(\n", 167 | " 32, 3, padding=\"same\", activation=\"relu\", input_shape=(28, 28, 1)\n", 168 | " ),\n", 169 | " tf.keras.layers.MaxPooling2D(),\n", 170 | " tf.keras.layers.Conv2D(64, 3, padding=\"same\", activation=\"relu\"),\n", 171 | " tf.keras.layers.MaxPooling2D(),\n", 172 | " tf.keras.layers.Flatten(),\n", 173 | " tf.keras.layers.Dense(128, activation=\"relu\"),\n", 174 | " tf.keras.layers.Dense(10, activation=\"softmax\"),\n", 175 | " ]\n", 176 | ")\n", 177 | "\n", 178 | "model.summary()" 179 | ] 180 | }, 181 | { 182 | "attachments": {}, 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "## Custom Training Loop \n", 187 | "[Back to top](#toc)\n", 188 | "\n", 189 | "Let's start applying the steps described above to create a custom training loop." 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 6, 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "# Step 1: Define essential variables like the optimizer, loss function, and metrics that will be used to evaluate the model's performance.\n", 199 | "\n", 200 | "optimizer = tf.keras.optimizers.Adam()\n", 201 | "loss_fn = tf.keras.losses.CategoricalCrossentropy()\n", 202 | "\n", 203 | "train_metric = tf.keras.metrics.CategoricalAccuracy()\n", 204 | "test_metric = tf.keras.metrics.CategoricalAccuracy()" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 11, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "# Step 2: Define the training and testing steps. These steps will be executed in the training loop.\n", 214 | "@tf.function # this decorator will convert the function to a graph which will be executed in the GPU\n", 215 | "def train_step(images, labels):\n", 216 | " # forward propagation starts\n", 217 | " with tf.GradientTape() as tape: # this will record all the operations performed inside the block\n", 218 | " predictions = model(images, training=True) # pass the images to the model\n", 219 | " loss = loss_fn(labels, predictions) # calculate the loss value\n", 220 | "\n", 221 | " # forward propagation ends, now we will start the backward propagation\n", 222 | "\n", 223 | " parameters = (\n", 224 | " model.trainable_variables\n", 225 | " ) # get all the trainable parameters of the model\n", 226 | "\n", 227 | " gradients = tape.gradient(\n", 228 | " loss, parameters\n", 229 | " ) # calculate the gradients of the loss with respect to the parameters\n", 230 | "\n", 231 | " gradients_parameters_tuple = zip(\n", 232 | " gradients, parameters\n", 233 | " ) # zip the gradients and weights together\n", 234 | "\n", 235 | " optimizer.apply_gradients(\n", 236 | " gradients_parameters_tuple\n", 237 | " ) # apply the gradients to the weights\n", 238 | "\n", 239 | " # backward propagation ends\n", 240 | "\n", 241 | " # update the metrics using the labels and predictions\n", 242 | " train_metric.update_state(labels, predictions)\n", 243 | "\n", 244 | " # return the loss value to be used in the training loop\n", 245 | " return loss\n", 246 | "\n", 247 | "\n", 248 | "# Step 3: Define the testing step. This step will be executed in the training loop using the test dataset.\n", 249 | "@tf.function\n", 250 | "def test_step(images, labels):\n", 251 | " predictions = model(images, training=True) # pass the images to the model\n", 252 | " loss = loss_fn(labels, predictions) # calculate the loss value\n", 253 | " test_metric.update_state(\n", 254 | " labels, predictions\n", 255 | " ) # update the metrics using the labels and predictions\n", 256 | " return loss # return the loss value to be used in the training loop" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": 12, 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "epochs_trained = 0 # this will be used to keep track of the current epoch, this will be useful when we resume training from a checkpoint or when we want to train the model for more epochs\n", 266 | "epochs = 10 # number of epochs" 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "execution_count": 13, 272 | "metadata": {}, 273 | "outputs": [ 274 | { 275 | "data": { 276 | "application/vnd.jupyter.widget-view+json": { 277 | "model_id": "cab61805412842638d795ab880ac217e", 278 | "version_major": 2, 279 | "version_minor": 0 280 | }, 281 | "text/plain": [ 282 | " 0%| | 0/10 [00:00-->\"Open\n", 11 | "\n", 12 | "Gradio is a powerful, open-source library for building user interfaces (UI) in Python. It allows developers to easily create web-based interfaces for machine learning models, data visualization, and other Python scripts. With Gradio, you can easily share your work with a wider audience and make your models accessible to non-technical users.\n", 13 | "\n", 14 | "Here are some of the key features of Gradio:\n", 15 | "- **Easy to use**: Gradio has a simple and intuitive API that makes it easy to get started.\n", 16 | "- **Built-in support for popular ML libraries**: Gradio has built-in support for popular ML libraries like TensorFlow, Keras, and PyTorch, allowing you to easily deploy your models.\n", 17 | "- **Customizable**: Gradio provides options for adding custom HTML, JavaScript, and CSS to the interface, allowing you to customize the look and feel of your application.\n", 18 | "- **Advanced features**: Gradio provides advanced features such as authentication and access control, logging, monitoring, and more.\n", 19 | "\n", 20 | "You can easily install Gradio using `pip` and start building your interface with as little as few lines of code. The library is well-documented and there are plenty of resources and tutorials available to help you get started.\n", 21 | "\n", 22 | "For more information, you can check out the [official documentation](https://gradio.app/docs) and the [GitHub repository](https://github.com/gradio-app/gradio)\n" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "collapsed": false 29 | }, 30 | "source": [ 31 | "## Table of Contents \n", 32 | "* [Concepts](#concepts)\n", 33 | "* [Components](#components)\n", 34 | "* [Code Examples](#code-examples)\n", 35 | " * [Greetings](#greetings)\n", 36 | " * [Calculator](#calculator)\n", 37 | " * [MNIST Classifier](#mnist-classifier)\n" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "source": [ 46 | "## Concepts \n", 47 | "[Back to Top](#toc)\n", 48 | "\n", 49 | "- **Interfaces**: In Gradio, an interface is a web-based UI that allows users to interact with a machine learning model or other Python script. An interface can be built using the `gr.Interface` class and can consist of inputs, outputs, and custom HTML, JavaScript and CSS.\n", 50 | "\n", 51 | "- **Run function**: The run function is a python function that takes the inputs as inputs and returns the outputs. It is the backbone of the interface, it is the function that will be executed when the user interacts with the interface.\n", 52 | "\n", 53 | "- **Inputs**: An input is a parameter that the user can interact with to generate the output. For example, an image classifier would have an input of an image. In Gradio, inputs can be created using the `gr.inputs` function.\n", 54 | "\n", 55 | "- **Outputs**: An output is the result generated by the model or script based on the inputs. In Gradio, outputs can be created using the `gr.outputs` function.\n", 56 | "\n", 57 | "- **Customization**: Gradio provides options for customizing the look and feel of the interface using HTML, CSS and JavaScript. Additionally, it also provides options for customizing the functionality and behavior of the interface.\n", 58 | "\n", 59 | "- **Launching**: Once the interface is created, it can be launched by calling the `launch()` function on the interface object. This will open the interface in a new browser tab.\n", 60 | "\n", 61 | "- **Sharing**: Once the interface is launched, it can be shared with others via a shareable link. This allows others to use the interface without having to run the code locally." 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "collapsed": false 68 | }, 69 | "source": [ 70 | "## Components \n", 71 | "[Back to Top](#toc)\n", 72 | "\n", 73 | "Gradio has built-in support for several types of inputs and outputs, which can be used to create a wide range of interfaces, some of the most common ones are listed below:\n", 74 | "\n", 75 | "### Inputs\n", 76 | "- **Textbox**: Accepts a single line of text input from the user. Can be customized with options such as `label`, `placeholder`, `default`.\n", 77 | "- **Checkbox**: Accepts a boolean input from the user. Can be customized with options such as `label`, `default`.\n", 78 | "- **Radio**: Accepts one option from a set of options provided by the user. Can be customized with options such as `label`, `options`, `default`.\n", 79 | "- **Select**: Accepts one option from a set of options provided by the user. Can be customized with options such as `label`, `options`, `default`.\n", 80 | "- **Slider**: Accepts a numeric input from the user within a range provided by the user. Can be customized with options such as `label`, `min`, `max`, `step`, `default`.\n", 81 | "- **File**: Accepts a file input from the user. Can be customized with options such as `label`, `allowed_extensions`.\n", 82 | "- **Image**: Accepts an image input from the user. Can be customized with options such as `shape`, `label`.\n", 83 | "\n", 84 | "### Outputs\n", 85 | "- **Textbox**: Displays a single line of text. Can be customized with options such as `label`.\n", 86 | "- **Label**: Displays probability scores for a set of classes. Can be customized with options such as `label`, `num_top_classes`.\n", 87 | "- **Plot**: Displays a plot. Can be customized with options such as `label`.\n", 88 | "- **Chatbot**: Displays a chatbot interface. Can be customized with options such as `label`.\n", 89 | "\n", 90 | "\n", 91 | "All of these components can be easily used by importing them from the `gradio.inputs` and `gradio.outputs` modules. For more information, you can check out the [official documentation](https://gradio.app/docs/#components)" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "source": [ 100 | "## Code Examples \n", 101 | "[Back to Top](#toc)\n" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": null, 107 | "metadata": { 108 | "collapsed": false, 109 | "pycharm": { 110 | "is_executing": true 111 | } 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "# As of writing this tutorial, Gradio is not available by default on Google Colab and needs to be installed manually, to install it run this cell\n", 116 | "!pip install gradio" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": 4, 122 | "metadata": { 123 | "collapsed": false 124 | }, 125 | "outputs": [], 126 | "source": [ 127 | "import numpy as np\n", 128 | "import pandas as pd\n", 129 | "import matplotlib.pyplot as plt\n", 130 | "import tensorflow as tf\n", 131 | "import gradio as gr" 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": { 137 | "collapsed": false 138 | }, 139 | "source": [ 140 | "### Greetings \n", 141 | "[Back to Top](#toc)\n", 142 | "\n", 143 | "A simple interface that takes a name as input and returns a greeting as output." 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "metadata": { 150 | "collapsed": false 151 | }, 152 | "outputs": [], 153 | "source": [ 154 | "def greeting(name):\n", 155 | " return f\"Hello {name}!\"\n", 156 | "\n", 157 | "\n", 158 | "interface = gr.Interface(\n", 159 | " fn=greeting,\n", 160 | " inputs=gr.inputs.Textbox(label=\"Your Name\", placeholder=\"Enter your name here\"),\n", 161 | " outputs=gr.inputs.Textbox(label=\"Greeting\"),\n", 162 | ")\n", 163 | "\n", 164 | "interface.launch()" 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": { 170 | "collapsed": false 171 | }, 172 | "source": [ 173 | "### Calculator \n", 174 | "[Back to Top](#toc)\n", 175 | "\n", 176 | "A simple interface that takes two numbers and operation as input and returns the result as output." 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": { 183 | "collapsed": false 184 | }, 185 | "outputs": [], 186 | "source": [ 187 | "def calculate(num1, num2, op):\n", 188 | " output = \"\"\n", 189 | "\n", 190 | " if op == \"+\":\n", 191 | " output = num1 + num2\n", 192 | " elif op == \"-\":\n", 193 | " output = num1 - num2\n", 194 | " elif op == \"*\":\n", 195 | " output = num1 * num2\n", 196 | " elif op == \"/\":\n", 197 | " if num2 != 0:\n", 198 | " output = num1 / num2\n", 199 | " else:\n", 200 | " output = \"Division by zero is not allowed\"\n", 201 | " else:\n", 202 | " output = \"Unsupported operation\"\n", 203 | "\n", 204 | " return output\n", 205 | "\n", 206 | "\n", 207 | "interface = gr.Interface(\n", 208 | " fn=calculate,\n", 209 | " inputs=[\n", 210 | " gr.inputs.Number(label=\"Number 1\"),\n", 211 | " gr.inputs.Number(label=\"Number 2\"),\n", 212 | " gr.inputs.Radio(\n", 213 | " choices=[\"+\", \"-\", \"*\", \"/\"], label=\"Choose the Operation\", default=\"+\"\n", 214 | " ),\n", 215 | " ],\n", 216 | " outputs=gr.inputs.Textbox(),\n", 217 | " title=\"Super Awesome Calculator\",\n", 218 | " description=\"This calculator takes in any two numbers and does all four major operations on them, isn't that awesome?!\",\n", 219 | " live=True,\n", 220 | ")\n", 221 | "\n", 222 | "interface.launch(debug=True, share=True)" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": { 228 | "collapsed": false 229 | }, 230 | "source": [ 231 | "### MNIST Classifier \n", 232 | "[Back to Top](#toc)\n", 233 | "\n", 234 | "A simple interface that takes an image as input and returns the predicted digit as output." 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": { 240 | "collapsed": false 241 | }, 242 | "source": [ 243 | "#### Prepare Model" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": null, 249 | "metadata": { 250 | "collapsed": false 251 | }, 252 | "outputs": [], 253 | "source": [ 254 | "# Load and preprocess the data\n", 255 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n", 256 | "\n", 257 | "x_train.shape, y_train.shape, x_test.shape, y_train.shape" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": null, 263 | "metadata": { 264 | "collapsed": false 265 | }, 266 | "outputs": [], 267 | "source": [ 268 | "# Pick a random image index\n", 269 | "index = np.random.randint(0, x_train.shape[0])\n", 270 | "\n", 271 | "# Plot the image\n", 272 | "plt.imshow(x_train[index], cmap=\"gray\")\n", 273 | "plt.title(\"Label: \" + str(y_train[index]))\n", 274 | "plt.show()" 275 | ] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": null, 280 | "metadata": { 281 | "collapsed": false 282 | }, 283 | "outputs": [], 284 | "source": [ 285 | "# reshape the data to be in the format (batch_size, input_dim) because Dense layers require one dimensional input\n", 286 | "x_train = x_train.reshape(-1, 28 * 28)\n", 287 | "x_test = x_test.reshape(-1, 28 * 28)\n", 288 | "\n", 289 | "x_train.shape, x_test.shape" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "metadata": { 296 | "collapsed": false 297 | }, 298 | "outputs": [], 299 | "source": [ 300 | "# Normalize the data\n", 301 | "x_train = x_train / 255.0\n", 302 | "x_test = x_test / 255.0" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": null, 308 | "metadata": { 309 | "collapsed": false 310 | }, 311 | "outputs": [], 312 | "source": [ 313 | "# convert the labels from integers to categorical using one hot encoding\n", 314 | "depth = len(set(y_train)) # calculate the number of classes to use as depth\n", 315 | "y_train = tf.one_hot(y_train, depth=depth)\n", 316 | "y_test = tf.one_hot(y_test, depth=depth)\n", 317 | "\n", 318 | "y_train.shape, y_test.shape" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "metadata": { 325 | "collapsed": false 326 | }, 327 | "outputs": [], 328 | "source": [ 329 | "# fetch the input shape, i.e. the number of features\n", 330 | "input_shape = x_train.shape[1] # input_shape = 784\n", 331 | "\n", 332 | "model = tf.keras.Sequential(\n", 333 | " [\n", 334 | " # input layer\n", 335 | " tf.keras.layers.Input(shape=(input_shape)),\n", 336 | " # hidden layers\n", 337 | " tf.keras.layers.Dense(256, activation=\"relu\"),\n", 338 | " tf.keras.layers.Dropout(0.2),\n", 339 | " tf.keras.layers.Dense(128, activation=\"relu\"),\n", 340 | " tf.keras.layers.Dropout(0.3),\n", 341 | " # output layer\n", 342 | " tf.keras.layers.Dense(10, activation=\"softmax\"),\n", 343 | " ]\n", 344 | ")\n", 345 | "\n", 346 | "model.summary()" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": null, 352 | "metadata": { 353 | "collapsed": false 354 | }, 355 | "outputs": [], 356 | "source": [ 357 | "loss_fn = tf.losses.CategoricalCrossentropy()\n", 358 | "optimizer = tf.optimizers.Adam(learning_rate=0.001)\n", 359 | "metrics = [tf.keras.metrics.CategoricalAccuracy()]\n", 360 | "\n", 361 | "# Compile the model\n", 362 | "model.compile(optimizer=optimizer, loss=loss_fn, metrics=metrics)" 363 | ] 364 | }, 365 | { 366 | "cell_type": "code", 367 | "execution_count": null, 368 | "metadata": { 369 | "collapsed": false 370 | }, 371 | "outputs": [], 372 | "source": [ 373 | "model.fit(x_train, y_train, batch_size=32, epochs=10, validation_data=(x_test, y_test))" 374 | ] 375 | }, 376 | { 377 | "cell_type": "markdown", 378 | "metadata": { 379 | "collapsed": false 380 | }, 381 | "source": [ 382 | "#### Create Interface #1\n", 383 | "This interface takes an image as input and returns the predicted digit as output." 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": null, 389 | "metadata": { 390 | "collapsed": false 391 | }, 392 | "outputs": [], 393 | "source": [ 394 | "def predict_mnist(img):\n", 395 | " # check if no image is passed\n", 396 | " if img is None:\n", 397 | " return \"You didn't pass an image\"\n", 398 | "\n", 399 | " # reduce the three color channel to single grayscale value\n", 400 | " img = tf.image.rgb_to_grayscale(img)\n", 401 | " # reshape/flatten the image\n", 402 | " img = tf.reshape(img, (1, 28 * 28))\n", 403 | " # cast tensor values to float32\n", 404 | " img = tf.cast(img, tf.float32)\n", 405 | " # normalize pixel values between 0 and 1\n", 406 | " img = img / 255.0\n", 407 | "\n", 408 | " # getting the predictions of the model\n", 409 | " prediction = model.predict(img)\n", 410 | " prediction = prediction[0]\n", 411 | " # getting the index of the highest probability\n", 412 | " index = tf.argmax(prediction)\n", 413 | "\n", 414 | " return int(index)\n", 415 | "\n", 416 | "\n", 417 | "interface = gr.Interface(\n", 418 | " # function that will be called when the user inputs an image\n", 419 | " fn=predict_mnist,\n", 420 | " # the input the user will use to interact with the function\n", 421 | " inputs=gr.inputs.Image(shape=(28, 28)),\n", 422 | " # the output the user will see after interacting with the function\n", 423 | " outputs=gr.inputs.Textbox(label=\"Predicted Digit\"),\n", 424 | " # title of the interface\n", 425 | " title=\"Hand-written digits recognizer\",\n", 426 | " # description of the interface\n", 427 | " description=\"Neural network to accurately predict a hand-written digit based on an image\",\n", 428 | ")\n", 429 | "# launch the interface and allow sharing\n", 430 | "interface.launch()" 431 | ] 432 | }, 433 | { 434 | "cell_type": "markdown", 435 | "metadata": { 436 | "collapsed": false 437 | }, 438 | "source": [ 439 | "#### Create Interface #2\n", 440 | "This interface takes an image as input and returns the predicted digit as output. It also shows the probability of each digit." 441 | ] 442 | }, 443 | { 444 | "cell_type": "code", 445 | "execution_count": null, 446 | "metadata": { 447 | "collapsed": false 448 | }, 449 | "outputs": [], 450 | "source": [ 451 | "def predict_mnist(img):\n", 452 | " # check if no image is passed\n", 453 | " if img is None:\n", 454 | " return \"You didn't pass an image\"\n", 455 | "\n", 456 | " # reduce the three color channel to single grayscale value\n", 457 | " img = tf.image.rgb_to_grayscale(img)\n", 458 | " # reshape/flatten the image\n", 459 | " img = tf.reshape(img, (1, 28 * 28))\n", 460 | " # cast tensor values to float32\n", 461 | " img = tf.cast(img, tf.float32)\n", 462 | " # normalize pixel values between 0 and 1\n", 463 | " img = img / 255.0\n", 464 | "\n", 465 | " # getting the predictions of the model\n", 466 | " prediction = model.predict(img)\n", 467 | " prediction = prediction[0]\n", 468 | "\n", 469 | " labels = dict()\n", 470 | "\n", 471 | " for i in range(len(prediction)): # loop through all the predictions\n", 472 | " labels[f\"{i}\"] = float(prediction[i]) # add the prediction to the dictionary\n", 473 | "\n", 474 | " return labels\n", 475 | "\n", 476 | "\n", 477 | "interface = gr.Interface(\n", 478 | " # function that will be called when the user inputs an image\n", 479 | " fn=predict_mnist,\n", 480 | " # the input the user will use to interact with the function\n", 481 | " inputs=gr.inputs.Image(shape=(28, 28)),\n", 482 | " # the output the user will see after interacting with the function\n", 483 | " outputs=gr.outputs.Label(label=\"Predicted Digit\", num_top_classes=3),\n", 484 | " # title of the interface\n", 485 | " title=\"Hand-written digits recognizer\",\n", 486 | " # description of the interface\n", 487 | " description=\"Neural network to accuratly predict a hand-written digit based on an image\",\n", 488 | " # options for flagging\n", 489 | " flagging_options=[\"Incorrect Prediction\", \"Program Error\"],\n", 490 | " # directory to store flags in, if not specified, flags will be stored in a folder called flags in the current directory\n", 491 | " # it's a good idea to specify a different directory for each interface\n", 492 | " flagging_dir=\"mnist_experiment_2\",\n", 493 | ")\n", 494 | "# launch the interface and allow sharing\n", 495 | "interface.launch(share=True)" 496 | ] 497 | } 498 | ], 499 | "metadata": { 500 | "kernelspec": { 501 | "display_name": "Python 3", 502 | "language": "python", 503 | "name": "python3" 504 | }, 505 | "language_info": { 506 | "codemirror_mode": { 507 | "name": "ipython", 508 | "version": 2 509 | }, 510 | "file_extension": ".py", 511 | "mimetype": "text/x-python", 512 | "name": "python", 513 | "nbconvert_exporter": "python", 514 | "pygments_lexer": "ipython2", 515 | "version": "2.7.6" 516 | } 517 | }, 518 | "nbformat": 4, 519 | "nbformat_minor": 0 520 | } 521 | -------------------------------------------------------------------------------- /Week 04 - Introduction to Sequence Modelling/2. Text Data Pipelines.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": {}, 7 | "source": [ 8 | "# Text Data Pipelines\n", 9 | "\n", 10 | "\"Open\n", 11 | "\n", 12 | "In this notebook, we'll cover the basics of building data pipelines for text data. This is an important step in processing text data efficiently and effectively for tasks such as sentiment analysis or machine translation.\n", 13 | "\n", 14 | "We'll start by reading text data from directories, preprocessing the data to clean and prepare it for modeling, and then building a data pipeline to efficiently process the data and feed it into a machine learning model.\n", 15 | "\n", 16 | "By the end of this notebook, you will have a good understanding of how to build and use text data pipelines in practice." 17 | ] 18 | }, 19 | { 20 | "attachments": {}, 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "## Table of Contents \n", 25 | "- [Text Dataset from Directory](#text-dataset-from-directory)\n", 26 | "- [Text Vectorization Layer](#text-vectorization-layer)\n", 27 | "- [Model Training](#model-training)" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 1, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "import numpy as np\n", 37 | "import pandas as pd\n", 38 | "import matplotlib.pyplot as plt\n", 39 | "import tensorflow as tf\n", 40 | "import re\n", 41 | "import string" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": 2, 47 | "metadata": {}, 48 | "outputs": [ 49 | { 50 | "name": "stdout", 51 | "output_type": "stream", 52 | "text": [ 53 | "Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n", 54 | "84125825/84125825 [==============================] - 35s 0us/step\n" 55 | ] 56 | } 57 | ], 58 | "source": [ 59 | "# Download the dataset\n", 60 | "url = \"https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n", 61 | "\n", 62 | "# we will be using the tf.keras.utils.get_file method to download the dataset and extract it automatically\n", 63 | "dataset = tf.keras.utils.get_file(\n", 64 | " \"aclImdb\", url, untar=True, cache_dir=\".\", cache_subdir=\"\"\n", 65 | ")\n", 66 | "\n", 67 | "# remove extra class that we will not be using\n", 68 | "!rm -rf aclImdb/train/unsup" 69 | ] 70 | }, 71 | { 72 | "attachments": {}, 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "Note that the data is structured as follows:\n", 77 | "\n", 78 | "```\n", 79 | "aclImdb\n", 80 | "\u251c\u2500\u2500 test\n", 81 | "\u2502 \u251c\u2500\u2500 neg\n", 82 | "| | \u251c\u2500\u2500 0_2.txt\n", 83 | "| | \u251c\u2500\u2500 10000_4.txt\n", 84 | "| | \u251c\u2500\u2500 ...\n", 85 | "\u2502 \u2514\u2500\u2500 pos\n", 86 | "| \u251c\u2500\u2500 0_10.txt\n", 87 | "| \u251c\u2500\u2500 10000_7.txt\n", 88 | "| \u251c\u2500\u2500 ...\n", 89 | "\u2514\u2500\u2500 train\n", 90 | " \u251c\u2500\u2500 neg\n", 91 | " | \u251c\u2500\u2500 0_3.txt\n", 92 | " | \u251c\u2500\u2500 10000_4.txt\n", 93 | " | \u251c\u2500\u2500 ...\n", 94 | " \u2514\u2500\u2500 pos\n", 95 | " \u251c\u2500\u2500 0_8.txt\n", 96 | " \u251c\u2500\u2500 10000_7.txt\n", 97 | " \u251c\u2500\u2500 ...\n", 98 | "```\n", 99 | "\n", 100 | "This is similar to cats vs dogs dataset structure we used in the previous week, but instead of images, we have text files where each file contains a review." 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 2, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "# first, we will set some parameters\n", 110 | "vocab_size = 8000 # number of words in the vocabulary, we will use the top 8000 most common words\n", 111 | "max_length = 120 # maximum length of a review, we will truncate reviews longer than 120 words and pad reviews shorter than 120 words\n", 112 | "embedding_dim = 50 # dimension of the embedding vector, we will use 50-dimensional embedding vectors\n", 113 | "batch_size = 32 # number of reviews in each batch\n", 114 | "seed = 42 # random seed" 115 | ] 116 | }, 117 | { 118 | "attachments": {}, 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "## Text Dataset from Directory \n", 123 | "[Back to top](#toc)\n", 124 | "\n", 125 | "The `tf.keras.utils.text_dataset_from_directory` is a function that provides the ability to read and process text data from a directory. It's ideal for working with large datasets of text data that are organized in a directory structure.\n", 126 | "\n", 127 | "This class has the following key parameters:\n", 128 | "\n", 129 | "- `directory`: The directory containing the text data. The files in this directory will be processed as text data.\n", 130 | "- `labels`: A list of labels, one for each text file. The labels should correspond to the text files in the directory. The default value is `infered` which will infer the labels from the directory structure.\n", 131 | "- `label_mode`: The type of label to return. The default value is `int` which will return an integer label for each text file. The other options are `binary` which will return a binary label for each text file, and `categorical` which will return a categorical label for each text file.\n", 132 | "\n", 133 | "For full documentation, see the [tf.keras.utils.text_dataset_from_directory doc](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text_dataset_from_directory)." 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 4, 139 | "metadata": {}, 140 | "outputs": [ 141 | { 142 | "name": "stdout", 143 | "output_type": "stream", 144 | "text": [ 145 | "Found 25000 files belonging to 2 classes.\n", 146 | "Found 25000 files belonging to 2 classes.\n" 147 | ] 148 | } 149 | ], 150 | "source": [ 151 | "# read the train and test datasets\n", 152 | "raw_train_ds = tf.keras.utils.text_dataset_from_directory(\n", 153 | " \"aclImdb/train\", batch_size=32, seed=seed\n", 154 | ")\n", 155 | "\n", 156 | "raw_test_ds = tf.keras.utils.text_dataset_from_directory(\n", 157 | " \"aclImdb/test\", batch_size=32, seed=seed\n", 158 | ")" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": 5, 164 | "metadata": {}, 165 | "outputs": [ 166 | { 167 | "name": "stdout", 168 | "output_type": "stream", 169 | "text": [ 170 | "Review: tf.Tensor(b'\"Pandemonium\" is a horror movie spoof that comes off more stupid than funny. Believe me when I tell you, I love comedies. Especially comedy spoofs. \"Airplane\", \"The Naked Gun\" trilogy, \"Blazing Saddles\", \"High Anxiety\", and \"Spaceballs\" are some of my favorite comedies that spoof a particular genre. \"Pandemonium\" is not up there with those films. Most of the scenes in this movie had me sitting there in stunned silence because the movie wasn\\'t all that funny. There are a few laughs in the film, but when you watch a comedy, you expect to laugh a lot more than a few times and that\\'s all this film has going for it. Geez, \"Scream\" had more laughs than this film and that was more of a horror film. How bizarre is that?

*1/2 (out of four)', shape=(), dtype=string)\n", 171 | "Label: tf.Tensor(0, shape=(), dtype=int32)\n" 172 | ] 173 | }, 174 | { 175 | "name": "stderr", 176 | "output_type": "stream", 177 | "text": [ 178 | "2023-02-05 02:43:40.935256: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n" 179 | ] 180 | } 181 | ], 182 | "source": [ 183 | "# preview samples from the training dataset\n", 184 | "for text_batch, label_batch in raw_train_ds.take(1):\n", 185 | " print(\"Review:\", text_batch[0])\n", 186 | " print(\"Label:\", label_batch[0])" 187 | ] 188 | }, 189 | { 190 | "attachments": {}, 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "## Text Vectorization Layer \n", 195 | "[Back to top](#toc)\n", 196 | "\n", 197 | "Text Vectorization is a preprocessing step in NLP where we convert the raw text data into numerical representations or embeddings. This is a crucial step as most machine learning models work with numerical data, not with text.\n", 198 | "\n", 199 | "In TensorFlow and Keras, there are several layers that we can use for text vectorization. We have already used `tf.keras.preprocessing.text.Tokenizer` in the previous lesson which only tokenizes the text data. In this lesson, we will use `tf.keras.layers.experimental.preprocessing.TextVectorization` which is a more advanced layer which will take care of the whole preprocessing pipeline.\n", 200 | "\n", 201 | "The `TextVectorization` layer has the following key parameters:\n", 202 | "- `max_tokens`: The maximum number of words to keep, based on word frequency. Only the most common `max_tokens` words will be kept.\n", 203 | "- `output_mode`: The output mode of the layer. The default value is `int` which will return an integer representation of the words. The other options are `binary` which will return a binary representation of the words, and `count` which will return the count of each word.\n", 204 | "- `output_sequence_length`: The length of the output sequences. If the input sequence is shorter than this value, the output sequence will be padded. If the input sequence is longer than this value, the output sequence will be truncated.\n", 205 | "- `standardize`: The standardization to apply to the text. The default value is `lower_and_strip_punctuation` which will convert the text to lowercase and strip punctuation. This parameter can be set to a custom function to apply a custom standardization.\n" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": 6, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "# let's create a custom standardization function similar to the one we used in the previous notebook but using TensorFlow operations\n", 215 | "def custom_standardization(text):\n", 216 | " # change all text to lowercase\n", 217 | " text = tf.strings.lower(text)\n", 218 | "\n", 219 | " # remove HTML tags\n", 220 | " text = tf.strings.regex_replace(text, r\"<.*?>\", \"\")\n", 221 | "\n", 222 | " # remove numbers\n", 223 | " text = tf.strings.regex_replace(text, r\"\\d+\", \"\")\n", 224 | "\n", 225 | " # remove words with numbers\n", 226 | " text = tf.strings.regex_replace(text, r\"\\w*\\d\\w*\", \"\")\n", 227 | "\n", 228 | " # remove URLs\n", 229 | " text = tf.strings.regex_replace(text, r\"https?://\\S+\", \"\")\n", 230 | "\n", 231 | " # remove emails\n", 232 | " text = tf.strings.regex_replace(text, r\"\\S+@\\S+\", \"\")\n", 233 | "\n", 234 | " # remove mentions (@username)\n", 235 | " text = tf.strings.regex_replace(text, r\"@\\S+\", \"\")\n", 236 | "\n", 237 | " # remove hashtags (#)\n", 238 | " text = tf.strings.regex_replace(text, r\"#\", \"\")\n", 239 | "\n", 240 | " # remove Punctuation\n", 241 | " text = tf.strings.regex_replace(text, f\"[{re.escape(string.punctuation)}]\", \" \")\n", 242 | "\n", 243 | " # remove extra spaces\n", 244 | " text = tf.strings.regex_replace(text, r\"\\s+\", \" \")\n", 245 | "\n", 246 | " return text" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 7, 252 | "metadata": {}, 253 | "outputs": [ 254 | { 255 | "name": "stderr", 256 | "output_type": "stream", 257 | "text": [ 258 | "2023-02-05 02:43:45.637438: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.\n" 259 | ] 260 | } 261 | ], 262 | "source": [ 263 | "# create a TextVectorization layer with our custom standardization function and other parameters\n", 264 | "vectorize_layer = tf.keras.layers.TextVectorization(\n", 265 | " standardize=custom_standardization,\n", 266 | " max_tokens=vocab_size,\n", 267 | " output_mode=\"int\",\n", 268 | " output_sequence_length=max_length,\n", 269 | ")\n", 270 | "\n", 271 | "# make a text-only dataset (without labels), then call adapt to build the vocabulary\n", 272 | "train_text = raw_train_ds.map(lambda x, y: x)\n", 273 | "vectorize_layer.adapt(train_text)" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": 8, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "# create a pipeline mapping function to vectorize the text and label\n", 283 | "def vectorize_text(text, label):\n", 284 | " # expand the dimensions of the text to make it into a batch\n", 285 | " text = tf.expand_dims(text, -1)\n", 286 | " # apply the vectorization layer to the text\n", 287 | " text = vectorize_layer(text)\n", 288 | " return text, label\n", 289 | "\n", 290 | "\n", 291 | "# create a pipeline mapping function to vectorize the text and label\n", 292 | "def dataset_creator(dataset):\n", 293 | " # create a dataset of text and labels\n", 294 | " dataset = dataset.map(\n", 295 | " vectorize_text, num_parallel_calls=tf.data.experimental.AUTOTUNE\n", 296 | " )\n", 297 | "\n", 298 | " # prefetch the dataset to improve latency\n", 299 | " dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n", 300 | "\n", 301 | " return dataset\n", 302 | "\n", 303 | "\n", 304 | "# create the training and test datasets\n", 305 | "train_ds = dataset_creator(raw_train_ds)\n", 306 | "test_ds = dataset_creator(raw_test_ds)" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": 9, 312 | "metadata": {}, 313 | "outputs": [ 314 | { 315 | "name": "stdout", 316 | "output_type": "stream", 317 | "text": [ 318 | "X batch shape: (32, 120) Y batch shape: (32,)\n", 319 | "X: tf.Tensor(\n", 320 | "[ 84 18 256 2 223 1 566 32 232 11 2436 1 54 22\n", 321 | " 28 413 254 12 315 278 0 0 0 0 0 0 0 0\n", 322 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 323 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 324 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 325 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 326 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 327 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 328 | " 0 0 0 0 0 0 0 0], shape=(120,), dtype=int64)\n", 329 | "Y: tf.Tensor(0, shape=(), dtype=int32)\n" 330 | ] 331 | } 332 | ], 333 | "source": [ 334 | "# preview samples from the training dataset\n", 335 | "for x_batch, y_batch in train_ds.take(1):\n", 336 | " print(\"X batch shape:\", x_batch.shape, \"Y batch shape:\", y_batch.shape)\n", 337 | " print(\"X:\", x_batch[0])\n", 338 | " print(\"Y:\", y_batch[0])" 339 | ] 340 | }, 341 | { 342 | "attachments": {}, 343 | "cell_type": "markdown", 344 | "metadata": {}, 345 | "source": [ 346 | "## Model Training \n", 347 | "[Back to top](#toc)\n", 348 | "\n", 349 | "In this section, we will build a model to classify the text data using the data pipeline we built in the previous section." 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": 10, 355 | "metadata": {}, 356 | "outputs": [ 357 | { 358 | "name": "stdout", 359 | "output_type": "stream", 360 | "text": [ 361 | "Model: \"sequential\"\n", 362 | "_________________________________________________________________\n", 363 | " Layer (type) Output Shape Param # \n", 364 | "=================================================================\n", 365 | " embedding (Embedding) (None, 120, 50) 400000 \n", 366 | " \n", 367 | " gru (GRU) (None, 64) 22272 \n", 368 | " \n", 369 | " dense (Dense) (None, 64) 4160 \n", 370 | " \n", 371 | " dense_1 (Dense) (None, 32) 2080 \n", 372 | " \n", 373 | " dense_2 (Dense) (None, 1) 33 \n", 374 | " \n", 375 | "=================================================================\n", 376 | "Total params: 428,545\n", 377 | "Trainable params: 428,545\n", 378 | "Non-trainable params: 0\n", 379 | "_________________________________________________________________\n" 380 | ] 381 | } 382 | ], 383 | "source": [ 384 | "gru_model = tf.keras.Sequential(\n", 385 | " [\n", 386 | " tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n", 387 | " tf.keras.layers.GRU(64, activation=\"tanh\"),\n", 388 | " tf.keras.layers.Dense(64, activation=\"relu\"),\n", 389 | " tf.keras.layers.Dense(32, activation=\"relu\"),\n", 390 | " tf.keras.layers.Dense(1, activation=\"sigmoid\"),\n", 391 | " ]\n", 392 | ")\n", 393 | "\n", 394 | "gru_model.summary()" 395 | ] 396 | }, 397 | { 398 | "cell_type": "code", 399 | "execution_count": null, 400 | "metadata": {}, 401 | "outputs": [], 402 | "source": [ 403 | "# compile the model\n", 404 | "gru_model.compile(optimizer=\"adam\", loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\n", 405 | "\n", 406 | "# train the model\n", 407 | "gru_model.fit(train_ds, epochs=10, validation_data=test_ds)" 408 | ] 409 | } 410 | ], 411 | "metadata": { 412 | "kernelspec": { 413 | "display_name": "ML-Training", 414 | "language": "python", 415 | "name": "python3" 416 | }, 417 | "language_info": { 418 | "codemirror_mode": { 419 | "name": "ipython", 420 | "version": 3 421 | }, 422 | "file_extension": ".py", 423 | "mimetype": "text/x-python", 424 | "name": "python", 425 | "nbconvert_exporter": "python", 426 | "pygments_lexer": "ipython3", 427 | "version": "3.8.15" 428 | }, 429 | "orig_nbformat": 4, 430 | "vscode": { 431 | "interpreter": { 432 | "hash": "effefe004be52d4cd3a12856ff0d4a1b800b83fc4bd48cce66e2ad043e78af0c" 433 | } 434 | } 435 | }, 436 | "nbformat": 4, 437 | "nbformat_minor": 2 438 | } -------------------------------------------------------------------------------- /Week 02 - Optimization and Regularization/1. Regularization Techniques.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "rlLPYUfJGcSK" 7 | }, 8 | "source": [ 9 | "# Regularization in TensorFlow\n", 10 | "\n", 11 | "\"Open\n", 12 | "Regularization is a technique used to prevent overfitting in deep learning models. Overfitting occurs when a model is trained too well on the training data and performs poorly on unseen data. Regularization helps to reduce the complexity of the model and make it more generalizable to new data.\n", 13 | "\n" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": { 19 | "id": "3iv_1SilGRHu" 20 | }, 21 | "source": [ 22 | "## Table of Contents \n", 23 | "* [Preparing the Example](#prep)\n", 24 | "* [Dropout](#dropout)\n", 25 | "* [Early Stopping](#early-stopping)\n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": { 31 | "id": "t4NGHPVHGmIH" 32 | }, 33 | "source": [ 34 | "## Preparing the Example \n", 35 | "[Back to Top](#toc)" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 1, 41 | "metadata": { 42 | "id": "aEngIJPwGJRr" 43 | }, 44 | "outputs": [], 45 | "source": [ 46 | "import numpy as np\n", 47 | "import pandas as pd\n", 48 | "import matplotlib.pyplot as plt\n", 49 | "import tensorflow as tf" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 2, 55 | "metadata": { 56 | "colab": { 57 | "base_uri": "https://localhost:8080/" 58 | }, 59 | "id": "jW_W_xzxEHJo", 60 | "outputId": "5b54fa01-4a27-423f-8673-dd3f977c2436" 61 | }, 62 | "outputs": [ 63 | { 64 | "data": { 65 | "text/plain": [ 66 | "((60000, 28, 28), (60000,), (10000, 28, 28), (60000,))" 67 | ] 68 | }, 69 | "execution_count": 2, 70 | "metadata": {}, 71 | "output_type": "execute_result" 72 | } 73 | ], 74 | "source": [ 75 | "# Load and preprocess the data\n", 76 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n", 77 | "\n", 78 | "x_train.shape, y_train.shape, x_test.shape, y_train.shape" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 3, 84 | "metadata": { 85 | "colab": { 86 | "base_uri": "https://localhost:8080/", 87 | "height": 281 88 | }, 89 | "id": "Fv4ewQxzESI5", 90 | "outputId": "d1f4047d-049f-4470-e4b5-9ff6f719d6fa" 91 | }, 92 | "outputs": [ 93 | { 94 | "data": { 95 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPvklEQVR4nO3dfcyV9X3H8fdHxYfiQ0EcYyA+EFjAmkgFNdE4lq4tmhDtH5qa1LFsE6N1auLmiNsiMTNpllXWxowE4wOyzq5IRSKk02orq9XOG0MVNK2uQgURpLQ+NuPpuz/ORXdze851bs65zgP39/NKTu5zX99znet7H/3wO9fDOT9FBGY28h3V6wbMrDscdrMkHHazJBx2syQcdrMkHHazJBz2xCT9UNJfdntd6w2HfQSQtFnSn/S6j0YkHSdpsaS3Jf1a0r9KGtXrvrJx2K0bFgKzgM8A04DPAn/f044ScthHMEljJD0h6d1iRH1C0qQhD5si6b8lvS/pcUljB61/kaQfS/qNpJ9KmtNiK/OAb0bE7oh4F/gm8OctPpe1yGEf2Y4CHgTOACYDvwXuHfKYP6UWvAnAPmpBRNJEYA3wj8BY4K+BlZJOG7oRSZOLfxAml/SiIfcnSTqllT/KWuOwj2AR8auIWBkRH0fEB8DdwB8NedjyiNgYER8B/wBcLelo4CvA2ohYGxEHIuIpYAC4vM52fhkRn46IXzZo5XvALZJOk/T7wM3F8k9V8GfaMB3T6wascyR9ClgMzAXGFItPknR0ROwvfn9r0CpbgFHAOGrvBq6SNG9QfRTwgxZauRv4NLAB+F/gPmAmsKOF57IWeWQf2W4D/hC4MCJOBi4tlg9+S336oPuTgb3ALmr/CCwvRuyDt9ER8bXDbSIifhsRN0XExIg4G/gVsD4iDrTyR1lrHPaRY5Sk4wfdjgFOoraf/pviwNudddb7iqQZxbuAu4BHi1H/34B5kr4o6ejiOefUOcDXlKSJkv5ANRdR212o14t1kMM+cqylFuyDt0XAvwAnUBupX6C27zzUcuAh4B3geIr96Yh4C7gCuAN4l9pI/zfU+X+mOED3YckBuinAj4GPgGXAwoh4soW/0dogf3mFWQ4e2c2ScNjNknDYzZJw2M2S6OpFNZJ8NNCswyJC9Za3NbJLmivpZ5LekLSwnecys85q+dRbcf30z4HPA1uBF4FrIuLVknU8spt1WCdG9guANyLiFxGxB/g2tYswzKwPtRP2iRz6IYqtxbJDSFogaUDSQBvbMrM2dfwAXUQsBZaC38ab9VI7I/s2Dv3E1KRimZn1oXbC/iIwVdJZko4FvgysrqYtM6tay2/jI2KfpJuA/wSOBh6IiE2VdWZmlerqp968z27WeR25qMbMjhwOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRIOu1kSDrtZEi3Pzw4gaTPwAbAf2BcRs6poysyq11bYC38cEbsqeB4z6yC/jTdLot2wB/CkpPWSFtR7gKQFkgYkDbS5LTNrgyKi9ZWliRGxTdLvAU8BfxUR60oe3/rGzGxYIkL1lrc1skfEtuLnTuAx4IJ2ns/MOqflsEsaLemkg/eBLwAbq2rMzKrVztH48cBjkg4+z79HxPcq6crMKtfWPvthb8z77GYd15F9djM7cjjsZkk47GZJOOxmSTjsZklU8UEYa9Npp51WWt+0aVNpfdy4cQ1rxanRhlatWlVa37JlS2l9xYoVpfXnnnuutG7d45HdLAmH3SwJh90sCYfdLAmH3SwJh90sCYfdLAl/6q0PPPPMM6X1OXPmdKeRFuzdu7e0vnjx4oa1hQsXlq578sknl9Y/+uij0vr+/ftL6yOVP/VmlpzDbpaEw26WhMNuloTDbpaEw26WhMNuloQ/z94FZ5xxRml99uzZbT3/gw8+2LD28ccfl6575ZVXltZPPPHE0vopp5xSWr/lllsa1ppd43HjjTeW1pcsWVJab3YePxuP7GZJOOxmSTjsZkk47GZJOOxmSTjsZkk47GZJ+PPsXXDPPfeU1m+99dbS+vLly0vr1113XcPanj17Stdtptl32i9atKi0fsMNN7S1/TLNPs8+bdq0hrXt27dX3U7faPnz7JIekLRT0sZBy8ZKekrS68XPMVU2a2bVG87b+IeAuUOWLQSejoipwNPF72bWx5qGPSLWAbuHLL4CWFbcXwaUX3NpZj3X6rXx4yPi4E7PO8D4Rg+UtABY0OJ2zKwibX8QJiKi7MBbRCwFlkLeA3Rm/aDVU287JE0AKH7urK4lM+uEVsO+Gphf3J8PPF5NO2bWKU3Ps0t6BJgDjAN2AHcCq4DvAJOBLcDVETH0IF6950r5Nn79+vWl9ZkzZ5bWzz777NL65s2bD7elyhxzTPme4P3339+wdu2111bdziHKrj8o6+tI1+g8e9N99oi4pkHpc211ZGZd5ctlzZJw2M2ScNjNknDYzZJw2M2S8FdJV2DSpEml9enTp5fWN23aVFp/++23D7unbtm3b19p/e67725Y6/SpNzuUR3azJBx2syQcdrMkHHazJBx2syQcdrMkHHazJHyevQLnn39+af34448vra9ataq03u7XQXdSs4+43n777R3b9rZt20rrjz76aMe2fSTyyG6WhMNuloTDbpaEw26WhMNuloTDbpaEw26WhM+zV+Dmm29ua/2VK1dW1MnhO+ecc0rrs2fPLq1fddVVpfXLLrvssHs6qNmUzM2mi37vvfda3vZI5JHdLAmH3SwJh90sCYfdLAmH3SwJh90sCYfdLAmfZ69As8+rN3PhhReW1idOnFhanzt3bsPaxRdfXLrujBkzSuvHHntsab2T7r333tL6SJ52uROajuySHpC0U9LGQcsWSdomaUNxu7yzbZpZu4bzNv4hoN7QsTgizitua6tty8yq1jTsEbEO2N2FXsysg9o5QHeTpJeLt/ljGj1I0gJJA5IG2tiWmbWp1bAvAaYA5wHbga83emBELI2IWRExq8VtmVkFWgp7ROyIiP0RcQC4D7ig2rbMrGothV3ShEG/fgnY2OixZtYfmp5nl/QIMAcYJ2krcCcwR9J5QACbges72GPfa/b95c0sWbKkok6qd+DAgdL6mjVrSuvz5s1redv+3vdqNQ17RFxTZ7GvZjA7wvhyWbMkHHazJBx2syQcdrMkHHazJBQR3duY1L2NddHo0aNL6wMD5VcKT5s2ra3tb926tWFt3bp1peuuXr26tL5ixYrS+rhx40rrO3fubFh78803S9edMmVKad3qiwjVW+6R3SwJh90sCYfdLAmH3SwJh90sCYfdLAmH3SwJf5V0BZpNLTx9+vTS+qWXXlpa37NnT2n9hRdeKK13UrMpm8s0+7usWh7ZzZJw2M2ScNjNknDYzZJw2M2ScNjNknDYzZLwefY+0Owz57103HHHldbvuuuulp/7iSeeaHldO3we2c2ScNjNknDYzZJw2M2ScNjNknDYzZJw2M2SGM6UzacDDwPjqU3RvDQiviFpLPAfwJnUpm2+OiJ+3blWrReOOqp8PDj11FNbfu5nn3225XXt8A1nZN8H3BYRM4CLgK9KmgEsBJ6OiKnA08XvZtanmoY9IrZHxEvF/Q+A14CJwBXAsuJhy4ArO9WkmbXvsPbZJZ0JzAR+AoyPiO1F6R1qb/PNrE8N+9p4SScCK4FbI+J96f+nk4qIaDSPm6QFwIJ2GzWz9gxrZJc0ilrQvxUR3y0W75A0oahPAOrO4BcRSyNiVkTMqqJhM2tN07CrNoTfD7wWEfcMKq0G5hf35wOPV9+emVWl6ZTNki4B/gt4BThQLL6D2n77d4DJwBZqp952N3muETll80h2wgknlNabfY32rl27GtbOOuustp7b6ms0ZXPTffaI+BFQd2Xgc+00ZWbd4yvozJJw2M2ScNjNknDYzZJw2M2ScNjNkvBXSVtH7d27t2HN59G7yyO7WRIOu1kSDrtZEg67WRIOu1kSDrtZEg67WRI+z26lzj333F63YBXxyG6WhMNuloTDbpaEw26WhMNuloTDbpaEw26WhM+zW6mpU6e2tf6aNWsq6sTa5ZHdLAmH3SwJh90sCYfdLAmH3SwJh90sCYfdLImm59klnQ48DIwHAlgaEd+QtAi4Dni3eOgdEbG2U41ab1x//fVtrf/8889X1Im1azgX1ewDbouIlySdBKyX9FRRWxwR/9y59sysKk3DHhHbge3F/Q8kvQZM7HRjZlatw9pnl3QmMBP4SbHoJkkvS3pA0pgG6yyQNCBpoK1Ozawtww67pBOBlcCtEfE+sASYApxHbeT/er31ImJpRMyKiFkV9GtmLRpW2CWNohb0b0XEdwEiYkdE7I+IA8B9wAWda9PM2tU07JIE3A+8FhH3DFo+YdDDvgRsrL49M6vKcI7GXwxcC7wiaUOx7A7gGknnUTsdtxlo7xyN9aUNGzaU1mfOnFlaX7vWZ2P7xXCOxv8IUJ2S/yuaHUF8BZ1ZEg67WRIOu1kSDrtZEg67WRIOu1kSiojubUzq3sbMkoqIeqfKPbKbZeGwmyXhsJsl4bCbJeGwmyXhsJsl4bCbJdHtKZt3AVsG/T6uWNaP+rW3fu0L3FurquztjEaFrl5U84mNSwP9+t10/dpbv/YF7q1V3erNb+PNknDYzZLoddiX9nj7Zfq1t37tC9xbq7rSW0/32c2se3o9sptZlzjsZkn0JOyS5kr6maQ3JC3sRQ+NSNos6RVJG3o9P10xh95OSRsHLRsr6SlJrxc/686x16PeFknaVrx2GyRd3qPeTpf0A0mvStok6ZZieU9fu5K+uvK6dX2fXdLRwM+BzwNbgReBayLi1a420oCkzcCsiOj5BRiSLgU+BB6OiM8Uy/4J2B0RXyv+oRwTEX/bJ70tAj7s9TTexWxFEwZPMw5cCfwZPXztSvq6mi68br0Y2S8A3oiIX0TEHuDbwBU96KPvRcQ6YPeQxVcAy4r7y6j9z9J1DXrrCxGxPSJeKu5/ABycZrynr11JX13Ri7BPBN4a9PtW+mu+9wCelLRe0oJeN1PH+IjYXtx/Bxjfy2bqaDqNdzcNmWa8b167VqY/b5cP0H3SJRHxWeAy4KvF29W+FLV9sH46dzqsaby7pc4047/Ty9eu1enP29WLsG8DTh/0+6RiWV+IiG3Fz53AY/TfVNQ7Ds6gW/zc2eN+fqefpvGuN804ffDa9XL6816E/UVgqqSzJB0LfBlY3YM+PkHS6OLACZJGA1+g/6aiXg3ML+7PBx7vYS+H6JdpvBtNM06PX7ueT38eEV2/AZdTOyL/P8Df9aKHBn2dDfy0uG3qdW/AI9Te1u2ldmzjL4BTgaeB14HvA2P7qLflwCvAy9SCNaFHvV1C7S36y8CG4nZ5r1+7kr668rr5clmzJHyAziwJh90sCYfdLAmH3SwJh90sCYfdLAmH3SyJ/wN/C+dtTHlErQAAAABJRU5ErkJggg==\n", 96 | "text/plain": [ 97 | "
" 98 | ] 99 | }, 100 | "metadata": { 101 | "needs_background": "light" 102 | }, 103 | "output_type": "display_data" 104 | } 105 | ], 106 | "source": [ 107 | "# Pick a random image index\n", 108 | "index = np.random.randint(0, x_train.shape[0])\n", 109 | "\n", 110 | "# Plot the image\n", 111 | "plt.imshow(x_train[index], cmap=\"gray\")\n", 112 | "plt.title(\"Label: \" + str(y_train[index]))\n", 113 | "plt.show()" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": 4, 119 | "metadata": { 120 | "colab": { 121 | "base_uri": "https://localhost:8080/" 122 | }, 123 | "id": "z0iSgwwxELmV", 124 | "outputId": "54d4ba04-875e-4ac5-c7b1-3ecae18e16f6" 125 | }, 126 | "outputs": [ 127 | { 128 | "data": { 129 | "text/plain": [ 130 | "((60000, 784), (10000, 784))" 131 | ] 132 | }, 133 | "execution_count": 4, 134 | "metadata": {}, 135 | "output_type": "execute_result" 136 | } 137 | ], 138 | "source": [ 139 | "# reshape the data to be in the format (batch_size, input_dim) because Dense layers require one dimensional input\n", 140 | "x_train = x_train.reshape(-1, 28 * 28)\n", 141 | "x_test = x_test.reshape(-1, 28 * 28)\n", 142 | "\n", 143 | "x_train.shape, x_test.shape" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 5, 149 | "metadata": { 150 | "id": "E8sS-G2aEmUr" 151 | }, 152 | "outputs": [], 153 | "source": [ 154 | "# Normalize the data\n", 155 | "x_train = x_train / 255.0\n", 156 | "x_test = x_test / 255.0" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": 6, 162 | "metadata": { 163 | "colab": { 164 | "base_uri": "https://localhost:8080/" 165 | }, 166 | "id": "kONnV_mDEw6x", 167 | "outputId": "ba00e7ad-0a1b-4708-ef3d-ef233a402df4" 168 | }, 169 | "outputs": [ 170 | { 171 | "data": { 172 | "text/plain": [ 173 | "(TensorShape([60000, 10]), TensorShape([10000, 10]))" 174 | ] 175 | }, 176 | "execution_count": 6, 177 | "metadata": {}, 178 | "output_type": "execute_result" 179 | } 180 | ], 181 | "source": [ 182 | "# convert the labels from integers to categorical using one hot encoding\n", 183 | "depth = len(set(y_train)) # calculate the number of classes to use as depth\n", 184 | "y_train = tf.one_hot(y_train, depth=depth)\n", 185 | "y_test = tf.one_hot(y_test, depth=depth)\n", 186 | "\n", 187 | "y_train.shape, y_test.shape" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": { 193 | "id": "gVWMcjkEIlqj" 194 | }, 195 | "source": [ 196 | "## Dropout \n", 197 | "[Back to Top](#toc)\n", 198 | "\n", 199 | "Dropout is a regularization technique that helps to prevent overfitting in deep learning models. It works by randomly \"dropping out\" or ignoring a subset of neurons during training. This is done by setting a probability, usually between 0 and 1, for each neuron to be dropped out.\n", 200 | "\n", 201 | "When dropout is applied, each neuron in the network has a probability of p of being \"dropped out\" during training. This means that for each training iteration, the neuron will not be used to make any computations or contribute to the final output. By dropping out neurons at random during training, the model is forced to learn to rely on multiple neurons rather than just a few specific ones. This makes the model more robust to noise in the data and less reliant on any one neuron.\n", 202 | "\n", 203 | "Dropout can be applied to any layer of a neural network, but it is most commonly applied to the fully connected layers. In TensorFlow, dropout can be easily applied to a layer using the `tf.keras.layers.Dropout` layer, where you can specify the dropout rate(probability) as a parameter.\n", 204 | "\n", 205 | "Here is an example of how to apply dropout to a dense layer in TensorFlow:\n", 206 | "\n", 207 | "```\n", 208 | "model = tf.keras.Sequential([\n", 209 | " ...\n", 210 | " tf.keras.layers.Dense(64, activation='relu'),\n", 211 | " tf.keras.layers.Dropout(rate=0.2)\n", 212 | " ...\n", 213 | "])\n", 214 | "```\n", 215 | "This will apply a dropout rate of 0.2 (20%) to the dense layer, meaning that 20% of the neurons in this layer will be dropped out during training. It's worth noting that the dropout is only used during training time, during the inference time all neurons are used.\n", 216 | "\n", 217 | "\n", 218 | "\n" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 7, 224 | "metadata": { 225 | "colab": { 226 | "base_uri": "https://localhost:8080/" 227 | }, 228 | "id": "xNBIUnOhFCVs", 229 | "outputId": "d95f52d3-afcf-42c0-9cdd-2ac3f9997057" 230 | }, 231 | "outputs": [ 232 | { 233 | "name": "stdout", 234 | "output_type": "stream", 235 | "text": [ 236 | "Model: \"sequential\"\n", 237 | "_________________________________________________________________\n", 238 | " Layer (type) Output Shape Param # \n", 239 | "=================================================================\n", 240 | " dense (Dense) (None, 256) 200960 \n", 241 | " \n", 242 | " dropout (Dropout) (None, 256) 0 \n", 243 | " \n", 244 | " dense_1 (Dense) (None, 128) 32896 \n", 245 | " \n", 246 | " dropout_1 (Dropout) (None, 128) 0 \n", 247 | " \n", 248 | " dense_2 (Dense) (None, 10) 1290 \n", 249 | " \n", 250 | "=================================================================\n", 251 | "Total params: 235,146\n", 252 | "Trainable params: 235,146\n", 253 | "Non-trainable params: 0\n", 254 | "_________________________________________________________________\n" 255 | ] 256 | } 257 | ], 258 | "source": [ 259 | "# fetch the input shape, i.e. the number of features\n", 260 | "input_shape = x_train.shape[1] # input_shape = 784\n", 261 | "\n", 262 | "model = tf.keras.Sequential(\n", 263 | " [\n", 264 | " # input layer\n", 265 | " tf.keras.layers.Input(shape=(input_shape)),\n", 266 | " # hidden layers\n", 267 | " tf.keras.layers.Dense(256, activation=\"relu\"),\n", 268 | " tf.keras.layers.Dropout(0.2),\n", 269 | " tf.keras.layers.Dense(128, activation=\"relu\"),\n", 270 | " tf.keras.layers.Dropout(0.3),\n", 271 | " # output layer\n", 272 | " tf.keras.layers.Dense(10, activation=\"softmax\"),\n", 273 | " ]\n", 274 | ")\n", 275 | "\n", 276 | "model.summary()" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": 8, 282 | "metadata": { 283 | "id": "MmAZIsPVFS12" 284 | }, 285 | "outputs": [], 286 | "source": [ 287 | "loss_fn = tf.losses.CategoricalCrossentropy()\n", 288 | "optimizer = tf.optimizers.Adam(learning_rate=0.001)\n", 289 | "metrics = [tf.keras.metrics.CategoricalAccuracy()]\n", 290 | "\n", 291 | "# Compile the model\n", 292 | "model.compile(optimizer=optimizer, loss=loss_fn, metrics=metrics)" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 9, 298 | "metadata": { 299 | "colab": { 300 | "base_uri": "https://localhost:8080/" 301 | }, 302 | "id": "qM1_OmruFZu2", 303 | "outputId": "5026099a-5668-4304-afd2-5682857e3503" 304 | }, 305 | "outputs": [ 306 | { 307 | "name": "stdout", 308 | "output_type": "stream", 309 | "text": [ 310 | "Epoch 1/10\n", 311 | "1875/1875 [==============================] - 19s 9ms/step - loss: 0.2759 - categorical_accuracy: 0.9169 - val_loss: 0.1205 - val_categorical_accuracy: 0.9630\n", 312 | "Epoch 2/10\n", 313 | "1875/1875 [==============================] - 16s 9ms/step - loss: 0.1293 - categorical_accuracy: 0.9611 - val_loss: 0.0920 - val_categorical_accuracy: 0.9720\n", 314 | "Epoch 3/10\n", 315 | "1875/1875 [==============================] - 12s 6ms/step - loss: 0.0977 - categorical_accuracy: 0.9705 - val_loss: 0.0913 - val_categorical_accuracy: 0.9728\n", 316 | "Epoch 4/10\n", 317 | "1875/1875 [==============================] - 10s 6ms/step - loss: 0.0832 - categorical_accuracy: 0.9743 - val_loss: 0.0771 - val_categorical_accuracy: 0.9766\n", 318 | "Epoch 5/10\n", 319 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0707 - categorical_accuracy: 0.9780 - val_loss: 0.0686 - val_categorical_accuracy: 0.9800\n", 320 | "Epoch 6/10\n", 321 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0647 - categorical_accuracy: 0.9796 - val_loss: 0.0728 - val_categorical_accuracy: 0.9800\n", 322 | "Epoch 7/10\n", 323 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0555 - categorical_accuracy: 0.9818 - val_loss: 0.0718 - val_categorical_accuracy: 0.9804\n", 324 | "Epoch 8/10\n", 325 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0526 - categorical_accuracy: 0.9836 - val_loss: 0.0789 - val_categorical_accuracy: 0.9801\n", 326 | "Epoch 9/10\n", 327 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0492 - categorical_accuracy: 0.9840 - val_loss: 0.0685 - val_categorical_accuracy: 0.9827\n", 328 | "Epoch 10/10\n", 329 | "1875/1875 [==============================] - 9s 5ms/step - loss: 0.0448 - categorical_accuracy: 0.9859 - val_loss: 0.0686 - val_categorical_accuracy: 0.9823\n" 330 | ] 331 | }, 332 | { 333 | "data": { 334 | "text/plain": [ 335 | "" 336 | ] 337 | }, 338 | "execution_count": 9, 339 | "metadata": {}, 340 | "output_type": "execute_result" 341 | } 342 | ], 343 | "source": [ 344 | "model.fit(x_train, y_train, batch_size=32, epochs=10, validation_data=(x_test, y_test))" 345 | ] 346 | }, 347 | { 348 | "cell_type": "markdown", 349 | "metadata": { 350 | "id": "7znRvZ2WJ1fj" 351 | }, 352 | "source": [ 353 | "## Early Stopping \n", 354 | "[Back to Top](#toc)\n", 355 | "\n", 356 | "Early stopping is a regularization technique used to prevent overfitting in deep learning models. The idea behind early stopping is to monitor the performance of the model on a validation set during training, and stop the training process when the performance starts to degrade.\n", 357 | "\n", 358 | "The validation set is a set of data that is used to evaluate the model's performance during training. The performance is typically measured by a metric such as accuracy or loss. The validation set is used to ensure that the model is generalizing well to new data, and not just memorizing the training data.\n", 359 | "\n", 360 | "During training, the performance of the model on the validation set is monitored at regular intervals (for example, after every epoch). If the performance of the model on the validation set stops improving for a certain number of consecutive iterations (for example, after several epochs), the training process is stopped. This helps to prevent the model from overfitting by stopping the training process before the model reaches its maximum number of training iterations.\n", 361 | "\n", 362 | "In TensorFlow, early stopping can be implemented by setting a `tf.keras.callbacks.EarlyStopping` callback during the training process, which monitors the performance of the model on a validation set and stops the training process when the performance stops improving. You can specify the metric to monitor, the number of iterations to wait before stopping the training, and whether to restore the weights of the best performing model.\n", 363 | "\n", 364 | "Here is an example of how to use early stopping in TensorFlow:\n", 365 | "\n", 366 | "```\n", 367 | "es = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5)\n", 368 | "model.fit(x_train, y_train, batch_size=32, epochs=100, validation_data=(x_test, y_test), callbacks=[es])\n", 369 | "```\n", 370 | "\n", 371 | "In this example, we used a `tf.keras.callbacks.EarlyStopping` callback to monitor the validation loss, we set the patience to 5, that means the training will stop if the validation loss stop improving after 5 epochs.\n", 372 | "\n", 373 | "Early stopping is a simple yet effective regularization technique that can be used to prevent overfitting in deep learning models. It helps to ensure that the model is generalizing well to new data and not just memorizing the training data, by stopping the training process when the performance on the validation set starts to degrade.\n", 374 | "\n", 375 | "\n", 376 | "\n" 377 | ] 378 | }, 379 | { 380 | "cell_type": "code", 381 | "execution_count": 10, 382 | "metadata": { 383 | "colab": { 384 | "base_uri": "https://localhost:8080/" 385 | }, 386 | "id": "igUjgjEE6RhO", 387 | "outputId": "fe8b4758-31b3-45d5-c448-49421a7a10dc" 388 | }, 389 | "outputs": [ 390 | { 391 | "name": "stdout", 392 | "output_type": "stream", 393 | "text": [ 394 | "Model: \"sequential_1\"\n", 395 | "_________________________________________________________________\n", 396 | " Layer (type) Output Shape Param # \n", 397 | "=================================================================\n", 398 | " dense_3 (Dense) (None, 256) 200960 \n", 399 | " \n", 400 | " dropout_2 (Dropout) (None, 256) 0 \n", 401 | " \n", 402 | " dense_4 (Dense) (None, 128) 32896 \n", 403 | " \n", 404 | " dropout_3 (Dropout) (None, 128) 0 \n", 405 | " \n", 406 | " dense_5 (Dense) (None, 10) 1290 \n", 407 | " \n", 408 | "=================================================================\n", 409 | "Total params: 235,146\n", 410 | "Trainable params: 235,146\n", 411 | "Non-trainable params: 0\n", 412 | "_________________________________________________________________\n" 413 | ] 414 | } 415 | ], 416 | "source": [ 417 | "# fetch the input shape, i.e. the number of features\n", 418 | "input_shape = x_train.shape[1] # input_shape = 784\n", 419 | "\n", 420 | "model = tf.keras.Sequential(\n", 421 | " [\n", 422 | " # input layer\n", 423 | " tf.keras.layers.Input(shape=(input_shape)),\n", 424 | " # hidden layers\n", 425 | " tf.keras.layers.Dense(256, activation=\"relu\"),\n", 426 | " tf.keras.layers.Dropout(0.2),\n", 427 | " tf.keras.layers.Dense(128, activation=\"relu\"),\n", 428 | " tf.keras.layers.Dropout(0.3),\n", 429 | " # output layer\n", 430 | " tf.keras.layers.Dense(10, activation=\"softmax\"),\n", 431 | " ]\n", 432 | ")\n", 433 | "\n", 434 | "model.summary()" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": 11, 440 | "metadata": { 441 | "id": "rf1duGcn6RhO" 442 | }, 443 | "outputs": [], 444 | "source": [ 445 | "loss_fn = tf.losses.CategoricalCrossentropy()\n", 446 | "optimizer = tf.optimizers.Adam(learning_rate=0.001)\n", 447 | "metrics = [tf.keras.metrics.CategoricalAccuracy()]\n", 448 | "\n", 449 | "# Compile the model\n", 450 | "model.compile(optimizer=optimizer, loss=loss_fn, metrics=metrics)" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 12, 456 | "metadata": { 457 | "colab": { 458 | "base_uri": "https://localhost:8080/" 459 | }, 460 | "id": "Ncnolz876RhO", 461 | "outputId": "32edf916-c99b-403c-e44a-02baed809a7b" 462 | }, 463 | "outputs": [ 464 | { 465 | "name": "stdout", 466 | "output_type": "stream", 467 | "text": [ 468 | "Epoch 1/20\n", 469 | "1875/1875 [==============================] - 9s 4ms/step - loss: 0.2709 - categorical_accuracy: 0.9184 - val_loss: 0.1101 - val_categorical_accuracy: 0.9660\n", 470 | "Epoch 2/20\n", 471 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.1252 - categorical_accuracy: 0.9618 - val_loss: 0.0966 - val_categorical_accuracy: 0.9698\n", 472 | "Epoch 3/20\n", 473 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0989 - categorical_accuracy: 0.9697 - val_loss: 0.0728 - val_categorical_accuracy: 0.9785\n", 474 | "Epoch 4/20\n", 475 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0801 - categorical_accuracy: 0.9748 - val_loss: 0.0776 - val_categorical_accuracy: 0.9776\n", 476 | "Epoch 5/20\n", 477 | "1875/1875 [==============================] - 8s 4ms/step - loss: 0.0698 - categorical_accuracy: 0.9782 - val_loss: 0.0710 - val_categorical_accuracy: 0.9777\n", 478 | "Epoch 6/20\n", 479 | "1875/1875 [==============================] - 9s 5ms/step - loss: 0.0613 - categorical_accuracy: 0.9808 - val_loss: 0.0743 - val_categorical_accuracy: 0.9795\n", 480 | "Epoch 7/20\n", 481 | "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0566 - categorical_accuracy: 0.9826 - val_loss: 0.0713 - val_categorical_accuracy: 0.9787\n", 482 | "Epoch 8/20\n", 483 | "1875/1875 [==============================] - 9s 5ms/step - loss: 0.0541 - categorical_accuracy: 0.9827 - val_loss: 0.0857 - val_categorical_accuracy: 0.9781\n" 484 | ] 485 | }, 486 | { 487 | "data": { 488 | "text/plain": [ 489 | "" 490 | ] 491 | }, 492 | "execution_count": 12, 493 | "metadata": {}, 494 | "output_type": "execute_result" 495 | } 496 | ], 497 | "source": [ 498 | "early_stopping = tf.keras.callbacks.EarlyStopping(\n", 499 | " monitor=\"val_loss\", patience=3, mode=\"min\"\n", 500 | ")\n", 501 | "\n", 502 | "model.fit(\n", 503 | " x_train,\n", 504 | " y_train,\n", 505 | " batch_size=32,\n", 506 | " epochs=20,\n", 507 | " validation_data=(x_test, y_test),\n", 508 | " callbacks=[early_stopping],\n", 509 | ")" 510 | ] 511 | } 512 | ], 513 | "metadata": { 514 | "colab": { 515 | "provenance": [] 516 | }, 517 | "kernelspec": { 518 | "display_name": "Python 3 (ipykernel)", 519 | "language": "python", 520 | "name": "python3" 521 | }, 522 | "language_info": { 523 | "codemirror_mode": { 524 | "name": "ipython", 525 | "version": 3 526 | }, 527 | "file_extension": ".py", 528 | "mimetype": "text/x-python", 529 | "name": "python", 530 | "nbconvert_exporter": "python", 531 | "pygments_lexer": "ipython3", 532 | "version": "3.8.15" 533 | } 534 | }, 535 | "nbformat": 4, 536 | "nbformat_minor": 1 537 | } 538 | -------------------------------------------------------------------------------- /Week 02 - Optimization and Regularization/3. Reproducibility, Callbacks, and Tensorboard.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "collapsed": false 7 | }, 8 | "source": [ 9 | "# Reproducibility, Callbacks, and TensorBoard\n", 10 | "\"Open\n", 11 | "\n", 12 | "We will be diving into the concepts of reproducibility, monitoring and analyzing the training process using TensorBoard.\n", 13 | "The ability to reproduce the results of a model is crucial for scientific research and the development of machine learning models. It allows for the verification of results, the identification of bugs, and the ability to reproduce results for further experimentation.\n", 14 | "\n", 15 | "We will begin by setting random seeds to ensure reproducibility across libraries such as TensorFlow, NumPy, and scikit-learn.\n", 16 | "Then, we will explore how to save and load models using TensorFlow's model persistence API. This will allow us to save and load the weights and architecture of a model for further use.\n", 17 | "\n", 18 | "Finally, we will look at TensorBoard, a visualization tool for monitoring the training process and analyzing the performance of our models. We will learn how to use TensorBoard to track metrics such as accuracy, loss, and gradients, which will help us to understand the performance of our models and identify any issues that may be occurring during training.\n", 19 | "\n", 20 | "By the end of this tutorial, you will have a solid understanding of the importance of reproducibility and model persistence, as well as the use of TensorBoard for monitoring and analyzing the training process." 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": { 26 | "collapsed": false 27 | }, 28 | "source": [ 29 | "## Table of Contents \n", 30 | "* [Reproducibility](#reprod)\n", 31 | "* [Keras Callbacks](#callbacks)\n", 32 | "* [TensorBoard](#tensorboard)\n", 33 | "* [Fashion MNIST](#example)" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 3, 39 | "metadata": { 40 | "collapsed": false 41 | }, 42 | "outputs": [], 43 | "source": [ 44 | "import numpy as np\n", 45 | "import pandas as pd\n", 46 | "import matplotlib.pyplot as plt\n", 47 | "import tensorflow as tf" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": { 53 | "collapsed": false 54 | }, 55 | "source": [ 56 | "## Reproducibility \n", 57 | "[Back to Top](#toc)\n", 58 | "\n", 59 | "Reproducibility is a crucial aspect of scientific research and machine learning development. It ensures that results can be verified, bugs can be identified, and experimentation can be repeated. In order to achieve reproducibility in TensorFlow and machine learning in general, it is important to set random seeds.\n", 60 | "\n", 61 | "In TensorFlow, the random seed can be set using the tf.random.set_seed() function. This function should be called before any other TensorFlow operations are used. For example, to set the seed to 42, we can do:\n", 62 | "\n", 63 | "```\n", 64 | "tf.random.set_seed(42)\n", 65 | "```\n", 66 | "\n", 67 | "In addition to setting the random seed in TensorFlow, it is also important to set the random seed in other libraries such as NumPy and scikit-learn. In NumPy, we can set the seed using `np.random.seed(42)`, and in scikit-learn, we can set the seed using `random_state=42` in the initialization of an estimator or other functions that introduce randomness like `train_test_split` with shuffling enabled.\n", 68 | "\n", 69 | "It's also important to record the versions of all libraries used and the system details. This can be done using `pip freeze > requirements.txt` or `conda list > environment.yml`\n", 70 | "\n", 71 | "By setting the random seed across all libraries and recording the versions of libraries and system details, we can ensure that results can be easily reproduced.\n", 72 | "\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 4, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "seed = 42\n", 84 | "tf.random.set_seed(seed)\n", 85 | "np.random.seed(seed)\n", 86 | "\n", 87 | "from sklearn.ensemble import RandomForestClassifier\n", 88 | "\n", 89 | "model = RandomForestClassifier(random_state=seed)" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": { 95 | "collapsed": false 96 | }, 97 | "source": [ 98 | "### Keras Callbacks \n", 99 | "[Back to Top](#toc)\n", 100 | "\n", 101 | "Keras callbacks are a powerful tool for monitoring and customizing the training process of a model. We briefly touched on the subject when we used EarlyStopping as a regularization technique in earlier lectures, but now we will delve deeper into the capabilities of callbacks and see how they can help us improve the reproducibility and monitoring of our models.\n", 102 | "\n", 103 | "Callbacks in Keras are functions that are applied at certain stages of the training process, such as at the end of each epoch, or when training is about to begin. These functions can be used to perform a variety of tasks, such as saving the model's weights, early stopping, and logging to TensorBoard.\n", 104 | "\n", 105 | "In this companion code, we will explore the different types of callbacks available in Keras and see how they can be used to improve the reproducibility and monitoring of our models. We will use callbacks to save the model's weights, monitor training metrics, and log data to TensorBoard.\n", 106 | "\n", 107 | "It's important to note that callbacks can be used in conjunction with other techniques such as setting random seeds to ensure reproducibility. With the help of callbacks, we can have more control over the training process and be able to better understand and optimize our models." 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": { 113 | "collapsed": false 114 | }, 115 | "source": [ 116 | "#### Types of Callbacks\n", 117 | "\n", 118 | "Some of the most common callbacks built into Keras include the following:\n", 119 | "\n", 120 | "* **ModelCheckpoint**: saves the model after every epoch or at a specified frequency, allowing you to save only the best performing models.\n", 121 | "* **EarlyStopping**: stops training when a certain metric, such as validation loss, stops improving after a specified number of epochs.\n", 122 | "* **TensorBoard**: allows you to visualize metrics and model architecture in TensorFlow's TensorBoard.\n", 123 | "* **CSVLogger**: saves the training metrics to a CSV file, allowing you to easily track performance over time.\n", 124 | "* **LearningRateScheduler**: allows you to schedule the learning rate to change over time, such as starting with a high learning rate and decreasing it over time.\n", 125 | "\n", 126 | "To learn more, check out the full list at Keras Callbacks [docs](https://tensorflow.org/api_docs/python/tf/keras/callbacks/callback), you can also find custom written callbacks posted on GitHub suchs as [Telegram notification callback](https://github.com/qubvel/keras_telegram_callback), you can also write your own by extending the `Callback` class as outlined in this [tutorial](https://www.tensorflow.org/guide/keras/custom_callback)." 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": { 132 | "collapsed": false 133 | }, 134 | "source": [ 135 | "### Introduction to Tensorboard \n", 136 | "[Back to Top](#toc)\n", 137 | "\n", 138 | "Tensorboard is a powerful tool developed by TensorFlow team that allows us to visualize and analyze our model's training process. It provides a dashboard of various visualizations such as scalars, histograms, distributions, and more, that can help us understand how our model is performing during training and identify potential issues.\n", 139 | "\n", 140 | "Some key features of Tensorboard include:\n", 141 | "\n", 142 | "- **Scalars**: Allows us to track the progress of our model's training by plotting metrics such as loss, accuracy, and learning rate over time.\n", 143 | "- **Histograms**: Helps us to understand the distribution of our model's parameters and gradients, which can be useful for identifying issues such as vanishing gradients.\n", 144 | "- **Distributions**: Similar to histograms, it helps to understand the distribution of the parameters and gradients.\n", 145 | "- **Images**: Allows us to view images generated by our model, which can be useful for identifying issues such as overfitting.\n", 146 | "\n", 147 | "Overall, Tensorboard is an essential tool for monitoring and analyzing the performance of our models during training. It can help us identify issues early on and make necessary adjustments to our model to improve its performance.\n", 148 | "\n", 149 | "You can learn more about Tensorboard by visiting its official documentation [here](https://www.tensorflow.org/tensorboard)" 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": { 155 | "collapsed": false 156 | }, 157 | "source": [ 158 | "### Fashion MNIST \n", 159 | "[Back to Top](#toc)\n", 160 | "In this section, we will put everything we have learned about reproducibility, callbacks, and TensorBoard into practice by creating an end-to-end code example. We will use the popular Fashion MNIST dataset, which consists of image of 10 different articles of clothing, and will create a simple feed forward neural networks to classify them.\n", 161 | "\n", 162 | "First, we will set the random seeds for reproducibility and load the dataset. Then, we will define our model architecture and compile it with appropriate loss function and metrics. We will then create an instance of the TensorBoard callback, which will allow us to visualize and monitor the training process. Additionally, we will create an instance of the ModelCheckpoint callback to save the best model weights.\n", 163 | "\n", 164 | "Next, we will train the model using the fit() method and passing in our callbacks as arguments. After training, we will evaluate the model on the test set and save the results. Finally, we will use the TensorBoard callback to launch TensorBoard and visualize the training process." 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 14, 170 | "metadata": { 171 | "collapsed": false 172 | }, 173 | "outputs": [], 174 | "source": [ 175 | "# set the random seed\n", 176 | "seed = 42\n", 177 | "tf.random.set_seed(seed)\n", 178 | "np.random.seed(seed)" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 15, 184 | "metadata": { 185 | "collapsed": true 186 | }, 187 | "outputs": [ 188 | { 189 | "name": "stdout", 190 | "output_type": "stream", 191 | "text": [ 192 | "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n", 193 | "29515/29515 [==============================] - 0s 1us/step\n", 194 | "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n", 195 | "26421880/26421880 [==============================] - 6s 0us/step\n", 196 | "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n", 197 | "5148/5148 [==============================] - 0s 0us/step\n", 198 | "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n", 199 | "4422102/4422102 [==============================] - 1s 0us/step\n" 200 | ] 201 | }, 202 | { 203 | "data": { 204 | "text/plain": [ 205 | "((60000, 28, 28), (60000,), (10000, 28, 28), (60000,))" 206 | ] 207 | }, 208 | "execution_count": 15, 209 | "metadata": {}, 210 | "output_type": "execute_result" 211 | } 212 | ], 213 | "source": [ 214 | "# Load and preprocess the data\n", 215 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()\n", 216 | "\n", 217 | "x_train.shape, y_train.shape, x_test.shape, y_train.shape" 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": 16, 223 | "metadata": { 224 | "collapsed": false 225 | }, 226 | "outputs": [], 227 | "source": [ 228 | "# Pick a random image index\n", 229 | "index = np.random.randint(0, x_train.shape[0])" 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": 17, 235 | "metadata": { 236 | "collapsed": false 237 | }, 238 | "outputs": [ 239 | { 240 | "data": { 241 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGxCAYAAADLfglZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAgyElEQVR4nO3dfXBU5eH28WvJyxJiWBpCkg2EkEFQJJROQXkpyFtJiQMtolPUmRbG1tEKOEyktpS2pGqJQ0fKHxRsHSeCQmXaQaWFEaNAqCJOZLBQCgoaIAoxECAJCWxIcp4/eNhfVxLwPu7mzm6+n5kzw549V86dwwlXDnv2Xo/jOI4AALCgm+0BAAC6LkoIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIXdKLL74oj8ejDz74ICxfz+PxaP78+WH5Wv/7NYuKilznf/3rX2v69Onq27evPB6P5s6dG7axAeFCCQEx6o9//KNqamr0/e9/X4mJibaHA7Qp3vYAAERGfX29unW78nvmSy+9ZHk0QNu4EgLacenSJT3++OP61re+JZ/Pp9TUVI0ZM0avv/56u5k///nPGjx4sLxer2677Ta98sor12xTVVWlhx9+WP369VNiYqJyc3P1u9/9Ts3NzWEd/9UCAjozroSAdgQCAZ09e1aLFi1S37591dTUpLfeekuzZs1SSUmJfvzjH4dsv3nzZu3YsUNPPvmkkpOTtXr1at1///2Kj4/XvffeK+lKAd1xxx3q1q2bfvvb32rgwIF677339PTTT+vYsWMqKSm57pgGDBggSTp27FgkvmWgw1FCQDt8Pl9IKbS0tGjKlCk6d+6cVq5ceU0JnTlzRuXl5crIyJAk3XXXXcrLy9PixYuDJVRUVKRz587p4MGD6t+/vyRpypQpSkpK0qJFi/Tzn/9ct912W7tjio/nRxaxhet14Dr+9re/6Tvf+Y5uuukmxcfHKyEhQS+88IIOHTp0zbZTpkwJFpAkxcXFafbs2Tp69Kg+++wzSdI///lPTZo0SVlZWWpubg4uBQUFkqSysrLrjufo0aM6evRoGL9DwC5KCGjHpk2b9MMf/lB9+/bVyy+/rPfee0/l5eV68MEHdenSpWu2z8zMbHddTU2NJOmLL77QP/7xDyUkJIQsQ4cOlXTlagroSri2B9rx8ssvKzc3Vxs3bpTH4wmuDwQCbW5fVVXV7rrevXtLktLS0vTNb35Tv//979v8GllZWV932EBUoYSAdng8HiUmJoYUUFVVVbt3x7399tv64osvgv8l19LSoo0bN2rgwIHq16+fJGn69OnaunWrBg4cqG984xuR/yaATo4SQpe2ffv2Nu80u+uuuzR9+nRt2rRJjz76qO69915VVlbqqaeekt/v15EjR67JpKWlafLkyfrNb34TvDvu8OHDIbdpP/nkkyotLdXYsWP12GOP6ZZbbtGlS5d07Ngxbd26Vc8991ywsNpy8803S9JXel2orKxMp0+flnSlEI8fP66///3vkqQJEyaoT58+N/waQMQ5QBdUUlLiSGp3qaiocBzHcZ555hlnwIABjtfrdYYMGeI8//zzztKlS50v/+hIcubNm+esXr3aGThwoJOQkODceuutzvr166/Z9+nTp53HHnvMyc3NdRISEpzU1FRnxIgRzpIlS5wLFy6EfM2lS5eGZHNycpycnJyv9D1OmDCh3e9vx44dJocLiBiP4zhOx1cfAADcHQcAsIgSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWd7s2qra2tOnnypFJSUkLeqQ4AiA6O46i+vl5ZWVk3/FyrTldCJ0+eVHZ2tu1hAAC+psrKyuvOACJ1whJKSUmxPQRE0NWJPE08/fTTxpm0tDTjjCQNGjTIVc7UCy+8YJyZOXOmceamm24yzkjSuXPnjDOff/65ceYXv/iFcYaZxqPHV/n3PGIltHr1av3hD3/QqVOnNHToUK1cuVLjx4+/YY7/gottbj5yOikpyTjTo0cP44zUcb8Eeb1e40xycrJxxm0JNTU1GWfcHHM+gjy2fZV/zyNyBmzcuFELFy7UkiVLtG/fPo0fP14FBQU6ceJEJHYHAIhSESmhFStW6Cc/+Yl++tOfasiQIVq5cqWys7O1Zs2aSOwOABClwl5CTU1N2rt3r/Lz80PW5+fna/fu3ddsHwgEVFdXF7IAALqGsJfQmTNn1NLSEvxgr6syMjLa/OTJ4uJi+Xy+4MKdcQDQdUTsVcEvvyDlOE6bL1ItXrxYtbW1waWysjJSQwIAdDJhvzsuLS1NcXFx11z1VFdXX3N1JF25S8jNnUIAgOgX9iuhxMREjRgxQqWlpSHrr36kMQAAV0XkfUKFhYX60Y9+pJEjR2rMmDH6y1/+ohMnTuiRRx6JxO4AAFEqIiU0e/Zs1dTU6Mknn9SpU6eUl5enrVu3KicnJxK7AwBEKY/jOI7tQfyvuro6+Xw+28NAhEydOtU4M2XKFOPM4MGDjTOSuxkG3NxMk5CQYJwZMGCAccbt9EXr1q0zzhw+fNg4U1FRYZz597//bZyBHbW1terZs+d1t2HODACANZQQAMAaSggAYA0lBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYA0lBACwJiKzaAPtOX36tHHm+PHjxhm3M7ZXV1cbZ5KSkowzw4YNM87U1NQYZxoaGowzbvd19uxZ40xtba1xBrGFKyEAgDWUEADAGkoIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIAGANJQQAsIYSAgBYwyza6FDZ2dnGmXHjxhlnLl68aJyRpISEBOPMgAEDjDNxcXHGmfr6euPM5cuXjTOSNGLECONMr169jDPjx483zvz+9783zqDz4koIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIAGANJQQAsIYSAgBYQwkBAKxhAlN0qMzMTOPMzTffbJx54403jDOSlJKSYpxpaWkxzjQ2NnbIfpKTk40zkrvJSHv27Gmc6d27t3EGsYUrIQCANZQQAMAaSggAYA0lBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYA0lBACwhglM0aF69OhhnDl8+LBxxuPxGGfc5pqbmztkPxkZGcYZN2OTpPLycuNM//79jTOffvqpcQaxhSshAIA1lBAAwJqwl1BRUZE8Hk/I4uYzZAAAsS8irwkNHTpUb731VvBxXFxcJHYDAIhyESmh+Ph4rn4AADcUkdeEjhw5oqysLOXm5uq+++677h0wgUBAdXV1IQsAoGsIewmNGjVK69at07Zt2/T888+rqqpKY8eOVU1NTZvbFxcXy+fzBZfs7OxwDwkA0EmFvYQKCgp0zz33aNiwYfrud7+rLVu2SJLWrl3b5vaLFy9WbW1tcKmsrAz3kAAAnVTE36yanJysYcOG6ciRI20+7/V65fV6Iz0MAEAnFPH3CQUCAR06dEh+vz/SuwIARJmwl9CiRYtUVlamiooKvf/++7r33ntVV1enOXPmhHtXAIAoF/b/jvvss890//3368yZM+rTp49Gjx6tPXv2KCcnJ9y7AgBEubCX0CuvvBLuL4kY4uYW/MbGRuPMyZMnjTOSlJWVZZy5fPmycSYxMdE4k5eXZ5x59913jTOStH//fuNMWlqacSYhIcE4g9jC3HEAAGsoIQCANZQQAMAaSggAYA0lBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYE3EP9QO+F8tLS3GGTcThJ4/f944I0kjRowwzvTv398489RTTxlnli9fbpwZMmSIcUaSpk+fbpz5+OOPjTNnzpwxziC2cCUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAa5hFGx2qqanJOFNfX2+cqaqqMs5I7sbnZhbturo640zv3r07ZD+StHv3buNMdXW1cSYtLc04g9jClRAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWMMEpuhQra2txhmPx2Oc+eSTT4wzklRTU2Oc8Xq9xpl3333XOHP58mXjzNmzZ40zkvT5558bZxobG40zbo4dYgtXQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDROYokPV19cbZ7744gvjjNuJMd1MEupmfG6OwzvvvGOcGTdunHFGks6fP2+cSU1NNc4EAgHjDGILV0IAAGsoIQCANcYltGvXLs2YMUNZWVnyeDx67bXXQp53HEdFRUXKyspSUlKSJk6cqIMHD4ZrvACAGGJcQg0NDRo+fLhWrVrV5vPLly/XihUrtGrVKpWXlyszM1NTp0519X/gAIDYZnxjQkFBgQoKCtp8znEcrVy5UkuWLNGsWbMkSWvXrlVGRoY2bNighx9++OuNFgAQU8L6mlBFRYWqqqqUn58fXOf1ejVhwgTt3r27zUwgEFBdXV3IAgDoGsJaQlVVVZKkjIyMkPUZGRnB576suLhYPp8vuGRnZ4dzSACATiwid8d5PJ6Qx47jXLPuqsWLF6u2tja4VFZWRmJIAIBOKKxvVs3MzJR05YrI7/cH11dXV19zdXSV1+t1/cZCAEB0C+uVUG5urjIzM1VaWhpc19TUpLKyMo0dOzacuwIAxADjK6ELFy7o6NGjwccVFRX68MMPlZqaqv79+2vhwoVatmyZBg0apEGDBmnZsmXq0aOHHnjggbAOHAAQ/YxL6IMPPtCkSZOCjwsLCyVJc+bM0YsvvqgnnnhCFy9e1KOPPqpz585p1KhRevPNN5WSkhK+UQMAYoJxCU2cOFGO47T7vMfjUVFRkYqKir7OuBCjampqjDNnz541zjQ3NxtnJCk5Odk4096dn+H2ySefGGdGjx7tal9ubhDq27evcaapqck4g9jC3HEAAGsoIQCANZQQAMAaSggAYA0lBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYA0lBACwJqyfrArcSO/evY0zPXr0MM60tLQYZ6T/+3RgE/v373e1L1P/+c9/jDMXL150tS83s5B362b+O21cXJxxBrGFKyEAgDWUEADAGkoIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIAGANJQQAsIYJTNGhWltbjTMZGRnGGbcTmLqZhPPMmTOu9mXq/fffN87Ex7v7EU9MTDTO1NbWGmcGDhxonEFs4UoIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIAGANJQQAsIYSAgBYQwkBAKxhAlN0KDcThLqZwDQQCBhnJCk3N9c48/HHH7val6nm5mbjTJ8+fVzty+fzGWfc/N26nWAVsYMrIQCANZQQAMAaSggAYA0lBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYA0lBACwhtkD0aHcTKjZvXt340xjY6NxRpIaGhqMM59//rmrfZmqqqoyzly8eNHVvurr640zFy5c6JD9ILZwJQQAsIYSAgBYY1xCu3bt0owZM5SVlSWPx6PXXnst5Pm5c+fK4/GELKNHjw7XeAEAMcS4hBoaGjR8+HCtWrWq3W2mTZumU6dOBZetW7d+rUECAGKT8Y0JBQUFKigouO42Xq9XmZmZrgcFAOgaIvKa0M6dO5Wenq7BgwfroYceUnV1dbvbBgIB1dXVhSwAgK4h7CVUUFCg9evXa/v27Xr22WdVXl6uyZMnKxAItLl9cXGxfD5fcMnOzg73kAAAnVTY3yc0e/bs4J/z8vI0cuRI5eTkaMuWLZo1a9Y12y9evFiFhYXBx3V1dRQRAHQREX+zqt/vV05Ojo4cOdLm816vV16vN9LDAAB0QhF/n1BNTY0qKyvl9/sjvSsAQJQxvhK6cOGCjh49GnxcUVGhDz/8UKmpqUpNTVVRUZHuuece+f1+HTt2TL/61a+Ulpamu+++O6wDBwBEP+MS+uCDDzRp0qTg46uv58yZM0dr1qzRgQMHtG7dOp0/f15+v1+TJk3Sxo0blZKSEr5RAwBignEJTZw4UY7jtPv8tm3bvtaAENtOnjzZIftJSkpylcvJyTHOnD9/3tW+OmI/Ho/H1b4GDRpknHEzgen1/i1B18DccQAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALAm4p+sCvyv06dPG2caGxuNM/Hx7k7tXr16GWc6ahZtNwKBgKuc2+Nn6uzZsx2yH3ReXAkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDVMYIoO5WZCTTcThObm5hpnJCktLc0442ZS1o7SrZu73zNTUlKMM27+njpqolR0XlwJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1zB6IDtW9e3fjzMiRI40zbifuPHz4sHGmqqrK1b46QnJysqvcwIEDjTNuJn9tbW01ziC2cCUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANYwgSk6lJvJPqdNm2accTuBaSAQMM7Ex3feH6OPP/7YVe573/ueceaWW24xzqxevdo4g9jClRAAwBpKCABgjVEJFRcX6/bbb1dKSorS09M1c+ZMffTRRyHbOI6joqIiZWVlKSkpSRMnTtTBgwfDOmgAQGwwKqGysjLNmzdPe/bsUWlpqZqbm5Wfn6+GhobgNsuXL9eKFSu0atUqlZeXKzMzU1OnTlV9fX3YBw8AiG5Gr6i+8cYbIY9LSkqUnp6uvXv36s4775TjOFq5cqWWLFmiWbNmSZLWrl2rjIwMbdiwQQ8//HD4Rg4AiHpf6zWh2tpaSVJqaqokqaKiQlVVVcrPzw9u4/V6NWHCBO3evbvNrxEIBFRXVxeyAAC6Btcl5DiOCgsLNW7cOOXl5Un6v9tvMzIyQrbNyMho99bc4uJi+Xy+4JKdne12SACAKOO6hObPn6/9+/frr3/96zXPeTyekMeO41yz7qrFixertrY2uFRWVrodEgAgyrh6l92CBQu0efNm7dq1S/369Quuz8zMlHTlisjv9wfXV1dXX3N1dJXX65XX63UzDABAlDO6EnIcR/Pnz9emTZu0fft25ebmhjyfm5urzMxMlZaWBtc1NTWprKxMY8eODc+IAQAxw+hKaN68edqwYYNef/11paSkBF/n8fl8SkpKksfj0cKFC7Vs2TINGjRIgwYN0rJly9SjRw898MADEfkGAADRy6iE1qxZI0maOHFiyPqSkhLNnTtXkvTEE0/o4sWLevTRR3Xu3DmNGjVKb775plJSUsIyYABA7DAqIcdxbriNx+NRUVGRioqK3I4JMczNBKaffPKJcebq2wZMuXl98qabbnK1r46QkJDgKjdgwADjzOnTp40zx48fN84gtjB3HADAGkoIAGANJQQAsIYSAgBYQwkBAKyhhAAA1lBCAABrKCEAgDWUEADAGkoIAGANJQQAsIYSAgBYQwkBAKxx9cmqQEdyM7O120/rdZOLj++8P0ZfZeb7tsTFxRln+vbta5xxM/M2YgtXQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgTeedeRH4/5qamowz3bt3d7Uvj8djnOnVq5erfXUEN9+PJLW2thpn3Ex6Wl9fb5xBbOFKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsYQJTdHqNjY3GmR49erjal5tJOP1+v6t9dYSkpCRXuYsXLxpn3ExG6nZ8iB1cCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYA0lBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANUxgik6vpaXFOBMf7+7U7tbN/PeyW2+91dW+TPl8PuNMc3Ozq315PB7jzOXLl40zbseH2MGVEADAGkoIAGCNUQkVFxfr9ttvV0pKitLT0zVz5kx99NFHIdvMnTtXHo8nZBk9enRYBw0AiA1GJVRWVqZ58+Zpz549Ki0tVXNzs/Lz89XQ0BCy3bRp03Tq1KngsnXr1rAOGgAQG4xevX3jjTdCHpeUlCg9PV179+7VnXfeGVzv9XqVmZkZnhECAGLW13pNqLa2VpKUmpoasn7nzp1KT0/X4MGD9dBDD6m6urrdrxEIBFRXVxeyAAC6Btcl5DiOCgsLNW7cOOXl5QXXFxQUaP369dq+fbueffZZlZeXa/LkyQoEAm1+neLiYvl8vuCSnZ3tdkgAgCjj+n1C8+fP1/79+/XOO++ErJ89e3bwz3l5eRo5cqRycnK0ZcsWzZo165qvs3jxYhUWFgYf19XVUUQA0EW4KqEFCxZo8+bN2rVrl/r163fdbf1+v3JycnTkyJE2n/d6vfJ6vW6GAQCIckYl5DiOFixYoFdffVU7d+5Ubm7uDTM1NTWqrKyU3+93PUgAQGwyek1o3rx5evnll7VhwwalpKSoqqpKVVVVunjxoiTpwoULWrRokd577z0dO3ZMO3fu1IwZM5SWlqa77747It8AACB6GV0JrVmzRpI0ceLEkPUlJSWaO3eu4uLidODAAa1bt07nz5+X3+/XpEmTtHHjRqWkpIRt0ACA2GD833HXk5SUpG3btn2tAQEAug5m0Uan16dPH+NMa2urq311797dODNy5EhX+zI1depU44zbm36ampqMM0OGDDHOHDhwwDiD2MIEpgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDROYotNbuXKlcebBBx90ta+9e/caZ1566SVX+zL15ptvGmcOHTrkal+9evUyzrz99tvGmUAgYJxBbOFKCABgDSUEALCGEgIAWEMJAQCsoYQAANZQQgAAayghAIA1lBAAwBpKCABgDSUEALCGEgIAWNPp5o5zHMf2ENDJuJlf7MKFC6721djYaJy5fPmyq32ZcvOz0dDQ4Gpf8fHm/zS43Rdi11c5Zz1OJ/tX/7PPPlN2drbtYQAAvqbKykr169fvutt0uhJqbW3VyZMnlZKSIo/HE/JcXV2dsrOzVVlZqZ49e1oaoX0chys4DldwHK7gOFzRGY6D4ziqr69XVlaWunW7/qs+ne6/47p163bD5uzZs2eXPsmu4jhcwXG4guNwBcfhCtvHwefzfaXtuDEBAGANJQQAsCaqSsjr9Wrp0qXyer22h2IVx+EKjsMVHIcrOA5XRNtx6HQ3JgAAuo6ouhICAMQWSggAYA0lBACwhhICAFhDCQEArImqElq9erVyc3PVvXt3jRgxQv/6179sD6lDFRUVyePxhCyZmZm2hxVxu3bt0owZM5SVlSWPx6PXXnst5HnHcVRUVKSsrCwlJSVp4sSJOnjwoJ3BRtCNjsPcuXOvOT9Gjx5tZ7ARUlxcrNtvv10pKSlKT0/XzJkz9dFHH4Vs0xXOh69yHKLlfIiaEtq4caMWLlyoJUuWaN++fRo/frwKCgp04sQJ20PrUEOHDtWpU6eCy4EDB2wPKeIaGho0fPhwrVq1qs3nly9frhUrVmjVqlUqLy9XZmampk6dqvr6+g4eaWTd6DhI0rRp00LOj61bt3bgCCOvrKxM8+bN0549e1RaWqrm5mbl5+eHzODdFc6Hr3IcpCg5H5wocccddziPPPJIyLpbb73V+eUvf2lpRB1v6dKlzvDhw20PwypJzquvvhp83Nra6mRmZjrPPPNMcN2lS5ccn8/nPPfccxZG2DG+fBwcx3HmzJnj/OAHP7AyHluqq6sdSU5ZWZnjOF33fPjycXCc6DkfouJKqKmpSXv37lV+fn7I+vz8fO3evdvSqOw4cuSIsrKylJubq/vuu0+ffvqp7SFZVVFRoaqqqpBzw+v1asKECV3u3JCknTt3Kj09XYMHD9ZDDz2k6upq20OKqNraWklSamqqpK57Pnz5OFwVDedDVJTQmTNn1NLSooyMjJD1GRkZqqqqsjSqjjdq1CitW7dO27Zt0/PPP6+qqiqNHTtWNTU1todmzdW//65+bkhSQUGB1q9fr+3bt+vZZ59VeXm5Jk+e7OpDAaOB4zgqLCzUuHHjlJeXJ6lrng9tHQcpes6HTvdRDtfz5c8XchznmnWxrKCgIPjnYcOGacyYMRo4cKDWrl2rwsJCiyOzr6ufG5I0e/bs4J/z8vI0cuRI5eTkaMuWLZo1a5bFkUXG/PnztX//fr3zzjvXPNeVzof2jkO0nA9RcSWUlpamuLi4a36Tqa6uvuY3nq4kOTlZw4YN05EjR2wPxZqrdwdyblzL7/crJycnJs+PBQsWaPPmzdqxY0fI5491tfOhvePQls56PkRFCSUmJmrEiBEqLS0NWV9aWqqxY8daGpV9gUBAhw4dkt/vtz0Ua3Jzc5WZmRlybjQ1NamsrKxLnxuSVFNTo8rKypg6PxzH0fz587Vp0yZt375dubm5Ic93lfPhRsehLZ32fLB4U4SRV155xUlISHBeeOEF57///a+zcOFCJzk52Tl27JjtoXWYxx9/3Nm5c6fz6aefOnv27HGmT5/upKSkxPwxqK+vd/bt2+fs27fPkeSsWLHC2bdvn3P8+HHHcRznmWeecXw+n7Np0ybnwIEDzv333+/4/X6nrq7O8sjD63rHob6+3nn88ced3bt3OxUVFc6OHTucMWPGOH379o2p4/Czn/3M8fl8zs6dO51Tp04Fl8bGxuA2XeF8uNFxiKbzIWpKyHEc509/+pOTk5PjJCYmOt/+9rdDbkfsCmbPnu34/X4nISHBycrKcmbNmuUcPHjQ9rAibseOHY6ka5Y5c+Y4jnPlttylS5c6mZmZjtfrde68807nwIEDdgcdAdc7Do2NjU5+fr7Tp08fJyEhwenfv78zZ84c58SJE7aHHVZtff+SnJKSkuA2XeF8uNFxiKbzgc8TAgBYExWvCQEAYhMlBACwhhICAFhDCQEArKGEAADWUEIAAGsoIQCANZQQAMAaSggAYA0lBACwhhICAFjz/wB9F8HQNtyGMAAAAABJRU5ErkJggg==", 242 | "text/plain": [ 243 | "
" 244 | ] 245 | }, 246 | "metadata": {}, 247 | "output_type": "display_data" 248 | } 249 | ], 250 | "source": [ 251 | "# Plot the image\n", 252 | "plt.imshow(x_train[index], cmap=\"gray\")\n", 253 | "plt.title(\"Label: \" + str(y_train[index]))\n", 254 | "plt.show()" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": 18, 260 | "metadata": { 261 | "collapsed": false 262 | }, 263 | "outputs": [ 264 | { 265 | "data": { 266 | "text/plain": [ 267 | "((60000, 784), (10000, 784))" 268 | ] 269 | }, 270 | "execution_count": 18, 271 | "metadata": {}, 272 | "output_type": "execute_result" 273 | } 274 | ], 275 | "source": [ 276 | "# reshape the data to be in the format (batch_size, input_dim) because Dense layers require one dimensional input\n", 277 | "x_train = x_train.reshape(-1, 28 * 28)\n", 278 | "x_test = x_test.reshape(-1, 28 * 28)\n", 279 | "\n", 280 | "x_train.shape, x_test.shape" 281 | ] 282 | }, 283 | { 284 | "cell_type": "code", 285 | "execution_count": 19, 286 | "metadata": { 287 | "collapsed": false 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "# Normalize the data\n", 292 | "x_train = x_train / 255.0\n", 293 | "x_test = x_test / 255.0" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 20, 299 | "metadata": { 300 | "collapsed": false 301 | }, 302 | "outputs": [ 303 | { 304 | "data": { 305 | "text/plain": [ 306 | "(TensorShape([60000, 10]), TensorShape([10000, 10]))" 307 | ] 308 | }, 309 | "execution_count": 20, 310 | "metadata": {}, 311 | "output_type": "execute_result" 312 | } 313 | ], 314 | "source": [ 315 | "# convert the labels from integers to categorical using one hot encoding\n", 316 | "depth = len(set(y_train)) # calculate the number of classes to use as depth\n", 317 | "y_train = tf.one_hot(y_train, depth=depth)\n", 318 | "y_test = tf.one_hot(y_test, depth=depth)\n", 319 | "\n", 320 | "y_train.shape, y_test.shape" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": null, 326 | "metadata": {}, 327 | "outputs": [], 328 | "source": [ 329 | "# Note that TensorFlow uses a stateful random number generator, so you need to set the seed before each model intalization to get reproducible results\n", 330 | "tf.random.set_seed(seed)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": 21, 336 | "metadata": { 337 | "collapsed": false 338 | }, 339 | "outputs": [ 340 | { 341 | "name": "stdout", 342 | "output_type": "stream", 343 | "text": [ 344 | "Model: \"sequential_1\"\n", 345 | "_________________________________________________________________\n", 346 | " Layer (type) Output Shape Param # \n", 347 | "=================================================================\n", 348 | " dense_3 (Dense) (None, 256) 200960 \n", 349 | " \n", 350 | " dropout_2 (Dropout) (None, 256) 0 \n", 351 | " \n", 352 | " dense_4 (Dense) (None, 128) 32896 \n", 353 | " \n", 354 | " dropout_3 (Dropout) (None, 128) 0 \n", 355 | " \n", 356 | " dense_5 (Dense) (None, 10) 1290 \n", 357 | " \n", 358 | "=================================================================\n", 359 | "Total params: 235,146\n", 360 | "Trainable params: 235,146\n", 361 | "Non-trainable params: 0\n", 362 | "_________________________________________________________________\n" 363 | ] 364 | } 365 | ], 366 | "source": [ 367 | "# fetch the input shape, i.e. the number of features\n", 368 | "input_shape = x_train.shape[1] # input_shape = 784\n", 369 | "\n", 370 | "model = tf.keras.Sequential(\n", 371 | " [\n", 372 | " # input layer\n", 373 | " tf.keras.layers.Input(shape=(input_shape)),\n", 374 | " # hidden layers\n", 375 | " tf.keras.layers.Dense(256, activation=\"relu\"),\n", 376 | " tf.keras.layers.Dropout(0.2),\n", 377 | " tf.keras.layers.Dense(128, activation=\"relu\"),\n", 378 | " tf.keras.layers.Dropout(0.3),\n", 379 | " # output layer\n", 380 | " tf.keras.layers.Dense(10, activation=\"softmax\"),\n", 381 | " ]\n", 382 | ")\n", 383 | "\n", 384 | "model.summary()" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": 22, 390 | "metadata": { 391 | "collapsed": false 392 | }, 393 | "outputs": [], 394 | "source": [ 395 | "loss_fn = tf.losses.CategoricalCrossentropy()\n", 396 | "optimizer = tf.optimizers.Adam(learning_rate=0.001)\n", 397 | "metrics = [tf.keras.metrics.CategoricalAccuracy()]\n", 398 | "\n", 399 | "# Compile the model\n", 400 | "model.compile(optimizer=optimizer, loss=loss_fn, metrics=metrics)" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": 23, 406 | "metadata": { 407 | "collapsed": false 408 | }, 409 | "outputs": [], 410 | "source": [ 411 | "# import datetime, a built-in python module for working with dates and times, we will use it to define a unique folder for each experiment\n", 412 | "import datetime\n", 413 | "\n", 414 | "# Get current date and time for unique logdir name\n", 415 | "now = datetime.datetime.now()\n", 416 | "now_formatted = now.strftime(\"%Y_%m_%d_%H_%M_%S\")\n", 417 | "logdir = f\"logs/fit/{now_formatted}\"\n", 418 | "\n", 419 | "# Define Tensorboard callback\n", 420 | "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=logdir)\n", 421 | "\n", 422 | "# Define Early Stopping callback\n", 423 | "early_stopping_callback = tf.keras.callbacks.EarlyStopping(\n", 424 | " monitor=\"val_loss\", patience=5\n", 425 | ")\n", 426 | "\n", 427 | "# Define Model Checkpoint callback\n", 428 | "model_path = f\"models/fit/{now_formatted}/best_model.h5\"\n", 429 | "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n", 430 | " filepath=model_path, monitor=\"val_loss\", save_best_only=True\n", 431 | ")" 432 | ] 433 | }, 434 | { 435 | "cell_type": "code", 436 | "execution_count": 24, 437 | "metadata": { 438 | "collapsed": false 439 | }, 440 | "outputs": [ 441 | { 442 | "name": "stdout", 443 | "output_type": "stream", 444 | "text": [ 445 | "Epoch 1/10\n" 446 | ] 447 | }, 448 | { 449 | "name": "stderr", 450 | "output_type": "stream", 451 | "text": [ 452 | "2023-01-25 16:19:07.863291: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n" 453 | ] 454 | }, 455 | { 456 | "name": "stdout", 457 | "output_type": "stream", 458 | "text": [ 459 | "1875/1875 [==============================] - 5s 3ms/step - loss: 0.5514 - categorical_accuracy: 0.8025 - val_loss: 0.4282 - val_categorical_accuracy: 0.8435\n", 460 | "Epoch 2/10\n", 461 | "1875/1875 [==============================] - 5s 2ms/step - loss: 0.4104 - categorical_accuracy: 0.8515 - val_loss: 0.4268 - val_categorical_accuracy: 0.8484\n", 462 | "Epoch 3/10\n", 463 | "1875/1875 [==============================] - 5s 3ms/step - loss: 0.3783 - categorical_accuracy: 0.8620 - val_loss: 0.3742 - val_categorical_accuracy: 0.8633\n", 464 | "Epoch 4/10\n", 465 | "1875/1875 [==============================] - 5s 2ms/step - loss: 0.3553 - categorical_accuracy: 0.8702 - val_loss: 0.3550 - val_categorical_accuracy: 0.8705\n", 466 | "Epoch 5/10\n", 467 | "1875/1875 [==============================] - 5s 2ms/step - loss: 0.3387 - categorical_accuracy: 0.8765 - val_loss: 0.3577 - val_categorical_accuracy: 0.8695\n", 468 | "Epoch 6/10\n", 469 | "1875/1875 [==============================] - 5s 2ms/step - loss: 0.3287 - categorical_accuracy: 0.8780 - val_loss: 0.3748 - val_categorical_accuracy: 0.8621\n", 470 | "Epoch 7/10\n", 471 | "1875/1875 [==============================] - 5s 2ms/step - loss: 0.3197 - categorical_accuracy: 0.8827 - val_loss: 0.3491 - val_categorical_accuracy: 0.8714\n", 472 | "Epoch 8/10\n", 473 | "1875/1875 [==============================] - 5s 3ms/step - loss: 0.3079 - categorical_accuracy: 0.8856 - val_loss: 0.3479 - val_categorical_accuracy: 0.8783\n", 474 | "Epoch 9/10\n", 475 | "1875/1875 [==============================] - 5s 3ms/step - loss: 0.3003 - categorical_accuracy: 0.8888 - val_loss: 0.3273 - val_categorical_accuracy: 0.8805\n", 476 | "Epoch 10/10\n", 477 | "1875/1875 [==============================] - 5s 2ms/step - loss: 0.2980 - categorical_accuracy: 0.8892 - val_loss: 0.3315 - val_categorical_accuracy: 0.8828\n" 478 | ] 479 | }, 480 | { 481 | "data": { 482 | "text/plain": [ 483 | "" 484 | ] 485 | }, 486 | "execution_count": 24, 487 | "metadata": {}, 488 | "output_type": "execute_result" 489 | } 490 | ], 491 | "source": [ 492 | "model.fit(\n", 493 | " x_train,\n", 494 | " y_train,\n", 495 | " batch_size=32,\n", 496 | " epochs=10,\n", 497 | " validation_data=(x_test, y_test),\n", 498 | " callbacks=[\n", 499 | " tensorboard_callback,\n", 500 | " early_stopping_callback,\n", 501 | " model_checkpoint_callback,\n", 502 | " ],\n", 503 | ")" 504 | ] 505 | } 506 | ], 507 | "metadata": { 508 | "kernelspec": { 509 | "display_name": "Python 3.8.15 ('AI4Climate')", 510 | "language": "python", 511 | "name": "python3" 512 | }, 513 | "language_info": { 514 | "codemirror_mode": { 515 | "name": "ipython", 516 | "version": 2 517 | }, 518 | "file_extension": ".py", 519 | "mimetype": "text/x-python", 520 | "name": "python", 521 | "nbconvert_exporter": "python", 522 | "pygments_lexer": "ipython2", 523 | "version": "3.8.15" 524 | }, 525 | "vscode": { 526 | "interpreter": { 527 | "hash": "34307d50f60b3a1739b0120251282ceb48aab2a74c47593a1ed36c86798eb2e0" 528 | } 529 | } 530 | }, 531 | "nbformat": 4, 532 | "nbformat_minor": 0 533 | } 534 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /Week 01 - Review and Intro to Deep Learning/1. Introduction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [], 7 | "toc_visible": true 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | } 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "source": [ 21 | "# Deep Learning with TensorFlow\n", 22 | "\n", 23 | "\"Open\n", 24 | "\n", 25 | "TensorFlow is an open-source library for deep learning that was developed by Google. It provides a flexible and powerful framework for building, training, and deploying deep learning models. TensorFlow also has a large and active community, which provides a wealth of resources and tutorials for learning and using the library.\n", 26 | "\n", 27 | "In this notebook, we will be learning how to use TensorFlow to build and train deep learning models for climate change prediction. We will also be learning about regularization techniques and how they can be used to prevent overfitting and improve the generalization of deep learning models.\n", 28 | "\n", 29 | "Helpful External Resources:\n", 30 | "* [TensorFlow Guides](https://www.tensorflow.org/guide)\n", 31 | "* [TensorFlow Tutorials](https://www.tensorflow.org/tutorials)" 32 | ], 33 | "metadata": { 34 | "id": "uBVSz-GUz2Us", 35 | "pycharm": { 36 | "name": "#%% md\n" 37 | } 38 | } 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "source": [ 43 | "## Table of Contents \n", 44 | "\n", 45 | "* [Tensors](#tensors)\n", 46 | "* [Layers](#layers)\n", 47 | "* [Activation Functions](#activation-functions)\n", 48 | "* [Models](#models)\n", 49 | "* [Losses](#losses)\n", 50 | "* [Optimizers](#optimizers)\n", 51 | "* [Putting it All Together](#examples)" 52 | ], 53 | "metadata": { 54 | "id": "FsyfJp872iWq", 55 | "pycharm": { 56 | "name": "#%% md\n" 57 | } 58 | } 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 1, 63 | "metadata": { 64 | "id": "nYWU9Ihcz0SO", 65 | "pycharm": { 66 | "name": "#%%\n" 67 | } 68 | }, 69 | "outputs": [], 70 | "source": [ 71 | "import numpy as np\n", 72 | "import pandas as pd\n", 73 | "import matplotlib.pyplot as plt\n", 74 | "import tensorflow as tf" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "source": [ 80 | "## Tensors \n", 81 | "[Back to Top](#toc)\n", 82 | "\n", 83 | "\n", 84 | "Tensors are the fundamental building blocks of TensorFlow. They are multi-dimensional arrays that can represent scalars, vectors, matrices, and higher-dimensional arrays of data. Tensors are similar to numpy arrays and can be used in similar ways, but they also have some unique features that make them well suited for deep learning.\n", 85 | "\n", 86 | "In TensorFlow, tensors are used to represent the inputs and outputs of a model, as well as the model's parameters. They are also used to represent gradients, which are used in the training process. Tensors can be created, manipulated and transformed with a variety of functions provided by TensorFlow.\n", 87 | "\n", 88 | "Let's take a look at how to create and manipulate tensors in TensorFlow" 89 | ], 90 | "metadata": { 91 | "id": "le9lBOC02PmI", 92 | "pycharm": { 93 | "name": "#%% md\n" 94 | } 95 | } 96 | }, 97 | { 98 | "cell_type": "code", 99 | "source": [ 100 | "# create a tensor with shape (2,3) and filled with zeros\n", 101 | "tensor_a = tf.zeros((2, 3))\n", 102 | "print(\"Tensor A:\\n\", tensor_a, \"\\n\")\n", 103 | "\n", 104 | "# create a tensor with shape (2,3) and filled with ones\n", 105 | "tensor_b = tf.ones((2, 3))\n", 106 | "print(\"Tensor B:\\n\", tensor_b, \"\\n\")\n", 107 | "\n", 108 | "# create a tensor with shape (2,3) and filled with a constant value\n", 109 | "tensor_c = tf.fill((3, 2), 4.0)\n", 110 | "print(\"Tensor C:\\n\", tensor_c, \"\\n\")\n", 111 | "\n", 112 | "# create a tensor from a numpy array\n", 113 | "numpy_array = np.array([[1, 2, 3], [4, 5, 6]])\n", 114 | "tensor_d = tf.constant(numpy_array)\n", 115 | "print(\"Tensor D:\\n\", tensor_d)" 116 | ], 117 | "metadata": { 118 | "colab": { 119 | "base_uri": "https://localhost:8080/" 120 | }, 121 | "id": "HQNdLVwd2ml0", 122 | "outputId": "9c4c9b21-b15e-478c-a566-c252f42ae1c5", 123 | "pycharm": { 124 | "name": "#%%\n" 125 | } 126 | }, 127 | "execution_count": 2, 128 | "outputs": [ 129 | { 130 | "output_type": "stream", 131 | "name": "stdout", 132 | "text": [ 133 | "Tensor A:\n", 134 | " tf.Tensor(\n", 135 | "[[0. 0. 0.]\n", 136 | " [0. 0. 0.]], shape=(2, 3), dtype=float32) \n", 137 | "\n", 138 | "Tensor B:\n", 139 | " tf.Tensor(\n", 140 | "[[1. 1. 1.]\n", 141 | " [1. 1. 1.]], shape=(2, 3), dtype=float32) \n", 142 | "\n", 143 | "Tensor C:\n", 144 | " tf.Tensor(\n", 145 | "[[4. 4.]\n", 146 | " [4. 4.]\n", 147 | " [4. 4.]], shape=(3, 2), dtype=float32) \n", 148 | "\n", 149 | "Tensor D:\n", 150 | " tf.Tensor(\n", 151 | "[[1 2 3]\n", 152 | " [4 5 6]], shape=(2, 3), dtype=int64)\n" 153 | ] 154 | } 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "source": [ 160 | "# addition of two tensors\n", 161 | "addition = tf.add(tensor_a, tensor_b)\n", 162 | "print(\"Tensor A + Tensor B:\\n\", addition, \"\\n\")\n", 163 | "\n", 164 | "# matrix multiplication of two tensors\n", 165 | "# note that in matrix multiplication, the number of columns of the first matrix must be equal to the number of rows of the second matrix.\n", 166 | "matmul = tf.matmul(tensor_b, tensor_c)\n", 167 | "print(\"Tensor B . Tensor C:\\n\", matmul, \"\\n\")\n", 168 | "\n", 169 | "# transpose of a tensor\n", 170 | "transpose = tf.transpose(tensor_d)\n", 171 | "print(\"Tensor D Transposed:\\n\", transpose, \"\\n\")" 172 | ], 173 | "metadata": { 174 | "colab": { 175 | "base_uri": "https://localhost:8080/" 176 | }, 177 | "id": "g-wi-6Ul4BVz", 178 | "outputId": "67aa2c73-2c39-4a57-decb-36b0fc088edb", 179 | "pycharm": { 180 | "name": "#%%\n" 181 | } 182 | }, 183 | "execution_count": 3, 184 | "outputs": [ 185 | { 186 | "output_type": "stream", 187 | "name": "stdout", 188 | "text": [ 189 | "Tensor A + Tensor B:\n", 190 | " tf.Tensor(\n", 191 | "[[1. 1. 1.]\n", 192 | " [1. 1. 1.]], shape=(2, 3), dtype=float32) \n", 193 | "\n", 194 | "Tensor B . Tensor C:\n", 195 | " tf.Tensor(\n", 196 | "[[12. 12.]\n", 197 | " [12. 12.]], shape=(2, 2), dtype=float32) \n", 198 | "\n", 199 | "Tensor D Transposed:\n", 200 | " tf.Tensor(\n", 201 | "[[1 4]\n", 202 | " [2 5]\n", 203 | " [3 6]], shape=(3, 2), dtype=int64) \n", 204 | "\n" 205 | ] 206 | } 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "source": [ 212 | "# get shape of a tensor\n", 213 | "shape = tensor_a.shape\n", 214 | "print(\"Tensor A Shape:\", shape, \"\\n\")\n", 215 | "\n", 216 | "# get specific element of a tensor\n", 217 | "element = tensor_d[1, 2]\n", 218 | "print(\"Element at Tensor D index [1,2]:\", element)" 219 | ], 220 | "metadata": { 221 | "colab": { 222 | "base_uri": "https://localhost:8080/" 223 | }, 224 | "id": "Rfrsd0pX4e8m", 225 | "outputId": "be5038bc-ebf9-40df-ddec-408e5fb85424", 226 | "pycharm": { 227 | "name": "#%%\n" 228 | } 229 | }, 230 | "execution_count": 4, 231 | "outputs": [ 232 | { 233 | "output_type": "stream", 234 | "name": "stdout", 235 | "text": [ 236 | "Tensor A Shape: (2, 3) \n", 237 | "\n", 238 | "Element at Tensor D index [1,2]: tf.Tensor(6, shape=(), dtype=int64)\n" 239 | ] 240 | } 241 | ] 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "source": [ 246 | "## Layers \n", 247 | "[Back to Top](#toc)\n", 248 | "\n", 249 | "\n", 250 | "In TensorFlow, the `tf.keras` module provides a high-level API for building and training deep learning models. The `tf.keras.layers` module provides a variety of pre-built layers that can be used to construct neural networks. These layers are the building blocks of a neural network and can be combined to create complex architectures.\n", 251 | "\n", 252 | "`tf.keras.layers.Dense` layer is used to create fully connected layers in a neural network. It takes in the number of units as an argument and creates that many neurons in the layer.\n" 253 | ], 254 | "metadata": { 255 | "id": "4XiW4FSG6gNm", 256 | "pycharm": { 257 | "name": "#%% md\n" 258 | } 259 | } 260 | }, 261 | { 262 | "cell_type": "code", 263 | "source": [ 264 | "# create a dense layer with 4 units\n", 265 | "dense_layer = tf.keras.layers.Dense(4)\n", 266 | "\n", 267 | "# create a tensor with shape (2, 8) as input (2 input samples, each one with 8 features)\n", 268 | "input_tensor = tf.random.normal([2, 8])\n", 269 | "\n", 270 | "# pass the input tensor through the dense layer\n", 271 | "output_tensor = dense_layer(input_tensor)\n", 272 | "\n", 273 | "print(\"Output Tensor:\", output_tensor)" 274 | ], 275 | "metadata": { 276 | "colab": { 277 | "base_uri": "https://localhost:8080/" 278 | }, 279 | "id": "mm-kOO8o6HYK", 280 | "outputId": "41680933-08f0-42d7-8987-6c386f2baaf1", 281 | "pycharm": { 282 | "name": "#%%\n" 283 | } 284 | }, 285 | "execution_count": 5, 286 | "outputs": [ 287 | { 288 | "output_type": "stream", 289 | "name": "stdout", 290 | "text": [ 291 | "Output Tensor: tf.Tensor(\n", 292 | "[[ 0.05513961 -0.5339481 1.4474454 -0.7458242 ]\n", 293 | " [-0.25193113 0.12729234 0.7375244 -0.65112084]], shape=(2, 4), dtype=float32)\n" 294 | ] 295 | } 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "source": [ 301 | "# access the weights of the dense layer\n", 302 | "weights = dense_layer.weights\n", 303 | "\n", 304 | "# print the shape of the weights, these weights are randomly initalized\n", 305 | "print(\"Weights:\", weights) # Weights Shape: [(8, 4), (4,)]" 306 | ], 307 | "metadata": { 308 | "colab": { 309 | "base_uri": "https://localhost:8080/" 310 | }, 311 | "id": "7kQHmKXn7nE2", 312 | "outputId": "46891215-0c16-48b0-df32-f090d873b83b", 313 | "pycharm": { 314 | "name": "#%%\n" 315 | } 316 | }, 317 | "execution_count": 6, 318 | "outputs": [ 319 | { 320 | "output_type": "stream", 321 | "name": "stdout", 322 | "text": [ 323 | "Weights: [, ]\n" 333 | ] 334 | } 335 | ] 336 | }, 337 | { 338 | "cell_type": "markdown", 339 | "source": [ 340 | "## Activation Functions \n", 341 | "[Back to Top](#toc)\n", 342 | "\n", 343 | "Activation functions are an important component of neural networks and are used to introduce non-linearity into the model. They are applied element-wise to the output of a layer and are used to introduce non-linearity into the model. Activation functions are used to introduce non-linearity because a neural network composed of only linear layers would be equivalent to a linear model, which is not expressive enough to model complex patterns in data.\n", 344 | "\n", 345 | "Some common activation functions used in neural networks are:\n", 346 | "\n", 347 | "**ReLU (Rectified Linear Unit)**: ReLU is one of the most widely used activation functions. It is computationally efficient, easy to implement and does not saturate for positive values of x. It is mainly used in the hidden layers of neural networks.\n", 348 | "\n", 349 | "Code: `tf.nn.relu`\n", 350 | "\n", 351 | "Math: $f(x) = max(0, x)$\n", 352 | "\n", 353 | "\n", 354 | "**Sigmoid**: Sigmoid function maps the input to a value between 0 and 1. It is mainly used in the output layer of binary classification problems to get a probability value.\n", 355 | "\n", 356 | "Code: `tf.nn.sigmoid`\n", 357 | "\n", 358 | "Math: $f(x) = \\frac{1}{1 + e^{-x}}$\n", 359 | "\n", 360 | "\n", 361 | "**Tanh (Hyperbolic Tangent)**: The tanh function maps the input to a value between -1 and 1. It is similar to sigmoid function but it is zero-centered, which means it has negative values as well. It is mainly used in the hidden layers of neural networks.\n", 362 | "\n", 363 | "Code: `tf.nn.tanh`\n", 364 | "\n", 365 | "Math: $f(x) = \\frac{1}{1 + e^{-x}}$\n", 366 | "\n", 367 | "\n", 368 | "**Softmax**: The softmax function is mainly used in the output layer of multi-class classification problems to get a probability distribution over the classes.\n", 369 | "\n", 370 | "Code: `tf.nn.softmax`\n", 371 | "\n", 372 | "Math: $f(x_i) = \\frac{e^{x_i}}{\\sum(e^{x_j})}$\n" 373 | ], 374 | "metadata": { 375 | "id": "_DjgShC18y1Z", 376 | "pycharm": { 377 | "name": "#%% md\n" 378 | } 379 | } 380 | }, 381 | { 382 | "cell_type": "code", 383 | "source": [ 384 | "# create a dense layer with 4 units\n", 385 | "dense_layer_w_activation = tf.keras.layers.Dense(4, activation=tf.nn.relu)\n", 386 | "\n", 387 | "# pass the input tensor through the dense layer\n", 388 | "output_tensor = dense_layer_w_activation(input_tensor)\n", 389 | "\n", 390 | "print(\n", 391 | " \"Output Tensor:\", output_tensor\n", 392 | ") # note that some of the outputs are zeros because we used ReLU" 393 | ], 394 | "metadata": { 395 | "colab": { 396 | "base_uri": "https://localhost:8080/" 397 | }, 398 | "id": "HxntfY-1-WDc", 399 | "outputId": "77291c33-2bc2-4773-8c1c-dfa1d8a5488b", 400 | "pycharm": { 401 | "name": "#%%\n" 402 | } 403 | }, 404 | "execution_count": 7, 405 | "outputs": [ 406 | { 407 | "output_type": "stream", 408 | "name": "stdout", 409 | "text": [ 410 | "Output Tensor: tf.Tensor(\n", 411 | "[[0.2585567 0. 0. 0.37567353]\n", 412 | " [0.19427118 2.1795511 1.9128449 0. ]], shape=(2, 4), dtype=float32)\n" 413 | ] 414 | } 415 | ] 416 | }, 417 | { 418 | "cell_type": "markdown", 419 | "source": [ 420 | "## Models \n", 421 | "[Back to Top](#toc)\n", 422 | "\n", 423 | "In TensorFlow with Keras, we can stack multiple layers together to create a sequential model by using the `tf.keras.Sequential` class. This class allows us to create a linear stack of layers, where the output of one layer is used as the input for the next layer.\n", 424 | "\n", 425 | "When building models in TensorFlow with Keras, the way we build the model will depend on the type of output we are trying to predict. Here are some guidelines on how to build a model for different types of outputs:\n", 426 | "\n", 427 | "* Binary Classification: When the output is binary, it means that there are only two possible outcomes, such as true or false, yes or no, etc. In this case, the last layer of the model should be a single sigmoid neuron, which will output a value between 0 and 1, representing the probability of the positive class.\n", 428 | "\n", 429 | "* Multinomial Classification: When the output is multinomial, it means that there are more than two possible outcomes, such as red, green, blue, etc. In this case, the last layer of the model should be a softmax layer, which will output a probability distribution over the classes.\n", 430 | "* Regression: When the output is a single continuous value, such as a price, temperature, etc. In this case, the last layer of the model should be a single neuron with no activation function." 431 | ], 432 | "metadata": { 433 | "id": "qB_hSBU08HGU", 434 | "pycharm": { 435 | "name": "#%% md\n" 436 | } 437 | } 438 | }, 439 | { 440 | "cell_type": "code", 441 | "source": [ 442 | "# create an empty sequential model\n", 443 | "model = tf.keras.Sequential(\n", 444 | " [\n", 445 | " # specify the dimension of the input, in this case it's a vector of 8 feautres\n", 446 | " tf.keras.layers.Input(shape=(8,)),\n", 447 | " # a dense layer with 4 units and ReLU activation\n", 448 | " tf.keras.layers.Dense(4, activation=tf.nn.relu),\n", 449 | " # a dense layer with 1 units and sigmoid activation\n", 450 | " tf.keras.layers.Dense(1, activation=\"sigmoid\"),\n", 451 | " ]\n", 452 | ")\n", 453 | "\n", 454 | "# Print the model summary\n", 455 | "model.summary()" 456 | ], 457 | "metadata": { 458 | "colab": { 459 | "base_uri": "https://localhost:8080/" 460 | }, 461 | "id": "sGeJT_GU7kDn", 462 | "outputId": "9e5855da-c01c-4a2a-d66c-2bbb51abb732", 463 | "pycharm": { 464 | "name": "#%%\n" 465 | } 466 | }, 467 | "execution_count": 8, 468 | "outputs": [ 469 | { 470 | "output_type": "stream", 471 | "name": "stdout", 472 | "text": [ 473 | "Model: \"sequential\"\n", 474 | "_________________________________________________________________\n", 475 | " Layer (type) Output Shape Param # \n", 476 | "=================================================================\n", 477 | " dense_2 (Dense) (None, 4) 36 \n", 478 | " \n", 479 | " dense_3 (Dense) (None, 1) 5 \n", 480 | " \n", 481 | "=================================================================\n", 482 | "Total params: 41\n", 483 | "Trainable params: 41\n", 484 | "Non-trainable params: 0\n", 485 | "_________________________________________________________________\n" 486 | ] 487 | } 488 | ] 489 | }, 490 | { 491 | "cell_type": "code", 492 | "source": [ 493 | "# pass the input tensor through the whole model\n", 494 | "model_output = model.predict(input_tensor)\n", 495 | "\n", 496 | "print(\n", 497 | " \"Model Outputs:\", model_output\n", 498 | ") # notice that the output for each sample is a single number between 0 and 1 because we used Sigmoid" 499 | ], 500 | "metadata": { 501 | "colab": { 502 | "base_uri": "https://localhost:8080/" 503 | }, 504 | "id": "bye2hhtT7YfI", 505 | "outputId": "7d2c86fc-b5ba-48c3-e39b-0ead3fe2450f", 506 | "pycharm": { 507 | "name": "#%%\n" 508 | } 509 | }, 510 | "execution_count": 9, 511 | "outputs": [ 512 | { 513 | "output_type": "stream", 514 | "name": "stdout", 515 | "text": [ 516 | "1/1 [==============================] - 0s 252ms/step\n", 517 | "Model Outputs: [[0.9021145 ]\n", 518 | " [0.74977976]]\n" 519 | ] 520 | } 521 | ] 522 | }, 523 | { 524 | "cell_type": "markdown", 525 | "source": [ 526 | "## Losses \n", 527 | "[Back to Top](#toc)\n", 528 | "\n", 529 | "A loss function is a function that measures the difference between the predicted output and the actual output. The goal of training a neural network is to minimize the loss function. There are several loss functions available in TensorFlow, and the appropriate loss function to use depends on the type of output and use case.\n", 530 | "\n", 531 | "Here are some commonly used loss functions in TensorFlow, along with the appropriate output type and use case:\n", 532 | "\n", 533 | "* **Binary Cross-Entropy**: This loss function is used for binary classification problems, where the output is a probability of one class. It is calculated as the negative log likelihood of the true labels given the predicted labels. In TensorFlow, the `tf.losses.BinaryCrossentropy` class can be used to calculate binary cross-entropy loss.\n", 534 | "\n", 535 | "* **Categorical Cross-Entropy**: This loss function is used for multi-class classification problems, where the output is a probability distribution over multiple classes. It is calculated as the negative log likelihood of the true labels given the predicted labels. In TensorFlow, the `tf.losses.CategoricalCrossentropy` class can be used to calculate categorical cross-entropy loss.\n", 536 | "\n", 537 | "* **Mean Absolute Error (MAE)**: This loss function is also used for regression problems, where the output is a single continuous value. It is calculated as the average of the absolute differences between the predicted values and the true values. In TensorFlow, the `tf.losses.MeanAbsoluteError` class can be used to calculate mean absolute error loss.\n", 538 | "\n", 539 | "* **Huber Loss**: This loss function is also used for regression problems, it is a combination of both MAE and MSE. It's less sensitive to outliers than MSE. In TensorFlow, the `tf.losses.Huber` class can be used to calculate Huber loss.\n" 540 | ], 541 | "metadata": { 542 | "id": "8Lo3sPBLA4mR", 543 | "pycharm": { 544 | "name": "#%% md\n" 545 | } 546 | } 547 | }, 548 | { 549 | "cell_type": "markdown", 550 | "source": [ 551 | "## Optimizers \n", 552 | "[Back to Top](#toc)\n", 553 | "\n", 554 | "An optimizer is an algorithm used to update the model's parameters in order to minimize the loss function. There are several optimizers available in TensorFlow, and the appropriate optimizer to use depends on the specific problem and use case.\n", 555 | "\n", 556 | "Here are some commonly used optimizers in TensorFlow, along with a brief explanation of how they work:\n", 557 | "\n", 558 | "* **Stochastic Gradient Descent (SGD)**: This is a simple optimization algorithm that updates the model's parameters based on the gradient of the loss function with respect to the parameters. It is computationally efficient and easy to implement, but can be sensitive to the learning rate and may get stuck in local minima.\n", 559 | "* **Adagrad**: This optimizer adapts the learning rate for each parameter based on the historical gradient information. The learning rate is increased for parameters that receive small gradients and decreased for parameters that receive large gradients. This can help the optimizer to converge faster and more efficiently, but can also lead to the learning rate becoming very small for some parameters and slowing down the convergence.\n", 560 | "* **Adam (Adaptive Moment Estimation)**: Adam is an optimization algorithm that combines the ideas of momentum and Adagrad. It uses an adaptive learning rate for each parameter, similar to Adagrad, but also takes into account the previous updates, similar to momentum. This allows Adam to converge faster and more efficiently than other optimizers, making it a popular choice for deep learning tasks.\n" 561 | ], 562 | "metadata": { 563 | "id": "65wi6yE8C_8K", 564 | "pycharm": { 565 | "name": "#%% md\n" 566 | } 567 | } 568 | }, 569 | { 570 | "cell_type": "markdown", 571 | "source": [ 572 | "## Putting it all together \n", 573 | "[Back to Top](#toc)\n", 574 | "\n", 575 | "Putting everything together, building a deep learning model with TensorFlow involves several steps. First, we need to prepare the data by loading it and preprocessing it as needed. Then we need to define the model architecture, including the input layer, hidden layers, and output layer, along with the appropriate activation functions for each layer. We also need to define the loss function and optimizer to be used for training.\n", 576 | "\n" 577 | ], 578 | "metadata": { 579 | "id": "fTP0TnacDoVw", 580 | "pycharm": { 581 | "name": "#%% md\n" 582 | } 583 | } 584 | }, 585 | { 586 | "cell_type": "code", 587 | "source": [ 588 | "# Load and preprocess the data\n", 589 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n", 590 | "\n", 591 | "x_train.shape, y_train.shape, x_test.shape, y_train.shape" 592 | ], 593 | "metadata": { 594 | "colab": { 595 | "base_uri": "https://localhost:8080/" 596 | }, 597 | "id": "jW_W_xzxEHJo", 598 | "outputId": "e5a30dc5-b3ae-4897-de2b-8d900e267bcd", 599 | "pycharm": { 600 | "name": "#%%\n" 601 | } 602 | }, 603 | "execution_count": 10, 604 | "outputs": [ 605 | { 606 | "output_type": "execute_result", 607 | "data": { 608 | "text/plain": [ 609 | "((60000, 28, 28), (60000,), (10000, 28, 28), (60000,))" 610 | ] 611 | }, 612 | "metadata": {}, 613 | "execution_count": 10 614 | } 615 | ] 616 | }, 617 | { 618 | "cell_type": "code", 619 | "source": [ 620 | "# Pick a random image index\n", 621 | "index = np.random.randint(0, x_train.shape[0])\n", 622 | "\n", 623 | "# Plot the image\n", 624 | "plt.imshow(x_train[index], cmap=\"gray\")\n", 625 | "plt.title(\"Label: \" + str(y_train[index]))\n", 626 | "plt.show()" 627 | ], 628 | "metadata": { 629 | "colab": { 630 | "base_uri": "https://localhost:8080/", 631 | "height": 281 632 | }, 633 | "id": "Fv4ewQxzESI5", 634 | "outputId": "e433eef4-ac26-48c5-9eda-fd721010ec7f", 635 | "pycharm": { 636 | "name": "#%%\n" 637 | } 638 | }, 639 | "execution_count": 11, 640 | "outputs": [ 641 | { 642 | "output_type": "display_data", 643 | "data": { 644 | "text/plain": [ 645 | "
" 646 | ], 647 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAQsUlEQVR4nO3df6zV9X3H8ecLxFSpQxBHiIK0RKdMo12Y6x+Kbl0LGhEr1mjiZHGKMSrr0i0zwiYqmm6udS4kjbfBCtppG4RJGA6dq1KzrRMM4sWfYEBFBJlVNC6p4nt/nC/tBe/5nMv3e37d+3k9kpt77vd9vuf7vif3db+/zvf7UURgZkPfsE43YGbt4bCbZcJhN8uEw26WCYfdLBMOu1kmHPaMSXpK0tXtntc6w2EfAiRtk/THne6jHtUskrRD0gfFP4rf7XRfuXHYrR2+BVwFnA2MAf4LeKCjHWXIYR/CJI2WtFrSu5J+WTw+/qCnTZb0P5L2SnpU0pg+839V0n9Kel/S85LOLdnKl4BnIuL1iNgHPAhMKflaVpLDPrQNA34EnABMBP4PWHzQc66kttYdD3wK/BOApOOAfwUWUVsb/yXwiKRjD16IpInFP4SJdfp4mNo/lZMkjQDmAP9W8XezQ3RYpxuw1omI/wUe2f+zpDuAnx30tAcioreo/w2wUdIc4ApgTUSsKZ73hKT1wPnA0oOW8wZwdKKVncAzwCvAPuBN4I/K/l5WjtfsQ5ikIyXdK2m7pL3AOuBoScP7PO3NPo+3AyOAsdS2Br5VrLHfl/Q+cBa1LYBD9bfA7wMTgC8AtwL/IenIEq9lJTnsQ9t3gN8B/iAifguYVkxXn+dM6PN4IvAJsIfaP4EHIuLoPl8jI+K7Jfo4A/hJRLwVEZ9GxP3AaLzf3lYO+9AxQtIX+nwdBhxFbT/9/eLA2y39zHeFpCnFWvY2YHmfg2gzJU2XNLx4zXP7OcA3EM9S20oYJ2mYpD+htgWxpdRvaqU47EPHGmrB3v+1EPhH4Ahqa+r/pv+DYg8A9wPvUNvEngcQEW8Cs4CbgXepren/in7+ZooDdB8lDtD9HfA8sBF4H/gLYHZEvH/ov6aVJd+8wiwPXrObZcJhN8uEw26WCYfdLBNt/QSdJB8NNGuxiFB/0yut2SXNkPSKpC2SbqryWmbWWqVPvRUfuXwV+DrwFrUPTlweES8m5vGa3azFWrFmPxPYUly2+CtqVzbNqvB6ZtZCVcJ+HAdeRPFWMe0AkuZKWl9cMWVmHdLyA3QR0QP0gDfjzTqpypp9BwdeMXV8Mc3MulCVsD8LnCjpS5IOBy4DVjWnLTNrttKb8RHxqaQbgLXAcOC+iNjctM7MrKnaetWb99nNWq8lH6oxs8HDYTfLhMNulgmH3SwTDrtZJhx2s0w47GaZcNjNMuGwm2XCYTfLhMNulgmH3SwTDrtZJhx2s0w47GaZcNjNMuGwm2XCYTfLhMNulgmH3SwTDrtZJhx2s0w47GaZcNjNMuGwm2XCYTfLhMNulgmH3SwTDrtZJkoP2Wy/ceWVVybro0aNStYvvvjiZH3FihWV5q/y2lK/A4L+2rZt25L1VatWHWpL1iKVwi5pG/AhsA/4NCKmNqMpM2u+ZqzZ/zAi9jThdcyshbzPbpaJqmEP4HFJGyTN7e8JkuZKWi9pfcVlmVkFVTfjz4qIHZJ+G3hC0ssRsa7vEyKiB+gBkBQVl2dmJVVas0fEjuL7bmAlcGYzmjKz5isddkkjJR21/zHwDaC3WY2ZWXMpotyWtaQvU1ubQ2134J8j4o4G8wzazfhXX321bm3ChAnJeUeMGNHsdtrmjTfeSNYvvPDCZL231///2y0i+v1wROl99oh4HTi9dEdm1lY+9WaWCYfdLBMOu1kmHHazTDjsZpkofeqt1MIG8am3ffv21a218z3sT+oy0507dybnXbRoUbK+ffv2ZP3ll19O1jvp8MMPr1ubOHFiGzs5NFu2bKk0f71Tb16zm2XCYTfLhMNulgmH3SwTDrtZJhx2s0w47GaZ8K2kB4Gnn346WU/dSvqDDz5odjtN0+g21Y0sWLAgWZ88eXLd2hVXXJGct1FvVT9bsXz58rq1yy67rNJr1+M1u1kmHHazTDjsZplw2M0y4bCbZcJhN8uEw26WCZ9nH6Dhw4fXrfX09CTnveqqqyotu+r56FaaNGlSsj5z5sy6tUZDTU+bNi1Z7/R9BFKWLFmSrF977bVt6uQ3vGY3y4TDbpYJh90sEw67WSYcdrNMOOxmmXDYzTLh+8Y3wZFHHpms33XXXcl61XOu11xzTd3a6tWrk/O+++67yfp5552XrN95553J+mmnnZasp1S9pnz9+vV1a/fee29y3nXr1iXrjTS6X//HH39c6fVTSt83XtJ9knZL6u0zbYykJyS9Vnwf3cxmzaz5BrIZfz8w46BpNwFPRsSJwJPFz2bWxRqGPSLWAe8dNHkWsLR4vBS4qMl9mVmTlf1s/LiI2L9T8g4wrt4TJc0F5pZcjpk1SeULYSIiUgfeIqIH6IGhe4DObDAoe+ptl6TxAMX33c1rycxaoWzYVwFzisdzgEeb046ZtUrD8+ySHgLOBcYCu4BbgH8BfgpMBLYDl0bEwQfx+nutLDfjR44cmazffffdyXqV6+E3bNiQrO/ZsydZnzHj4BMxB2rl5zS2bt2arN92223J+tq1a+vWGv3eg1m98+wN99kj4vI6pa9V6sjM2soflzXLhMNulgmH3SwTDrtZJhx2s0z4Etcu0OhSzocffjhZnz17djPbOUDVy0xTl9DecccdyXkXL16crFv/Sl/iamZDg8NulgmH3SwTDrtZJhx2s0w47GaZcNjNMuHz7IPA6aefnqw//vjjdWtjx46ttOxhw9Lrg97e3mT9ggsuqFvbvn17qZ4szefZzTLnsJtlwmE3y4TDbpYJh90sEw67WSYcdrNMVB4Rxlpv2rRpyfoxxxxTt1b1cxSfffZZsn7KKack6ytXrqxbmz9/fnLexx57LFm3Q+M1u1kmHHazTDjsZplw2M0y4bCbZcJhN8uEw26WCV/P3gUa3Zt906ZNyXqjc90pqfu6Q+Peql4vn7JgwYJkfdmyZcn622+/3cx2Bo3S17NLuk/Sbkm9faYtlLRD0sbi6/xmNmtmzTeQzfj7gRn9TL87Is4ovtY0ty0za7aGYY+IdcB7bejFzFqoygG6GyRtKjbzR9d7kqS5ktZLWl9hWWZWUdmw/wCYDJwB7AS+V++JEdETEVMjYmrJZZlZE5QKe0Tsioh9EfEZ8EPgzOa2ZWbNVirsksb3+fGbQPp+wmbWcQ3Ps0t6CDgXGAvsAm4pfj4DCGAbcG1E7Gy4MJ9n79cll1ySrDcanz2l0Xn06dOnJ+uNzrNfffXVyfp1112XrFdZ9po16ZNAM2fOLL3swazeefaGN6+IiMv7mbykckdm1lb+uKxZJhx2s0w47GaZcNjNMuGwm2XCt5LuAmeffXay3ugUVMrevXuT9UaXzzZy4403lq7fc889yXnnzZuXrE+aNClZP+GEE+rWchwu2mt2s0w47GaZcNjNMuGwm2XCYTfLhMNulgmH3SwTPs/eBW6//fZk/frrry/92kcccUSynjoXDa09H93o8uqqw0U/+OCDdWuNPtswFHnNbpYJh90sEw67WSYcdrNMOOxmmXDYzTLhsJtlwkM2d4FG16t/8sknLVv2okWLkvWFCxe2bNknn3xysr558+ZkvdHf7muvvVa3VmWY625XeshmMxsaHHazTDjsZplw2M0y4bCbZcJhN8uEw26WiYbXs0uaACwDxlEborknIu6RNAb4CTCJ2rDNl0bEL1vXqrXCggULkvXe3t5kffny5aWXfeqpp5ae1w7dQNbsnwLfiYgpwFeB6yVNAW4CnoyIE4Eni5/NrEs1DHtE7IyI54rHHwIvAccBs4ClxdOWAhe1qkkzq+6Q9tklTQK+AvwCGBcRO4vSO9Q2882sSw34HnSSvgg8Anw7Ivb2/Tx3RES9z71LmgvMrdqomVUzoDW7pBHUgv7jiFhRTN4laXxRHw/s7m/eiOiJiKkRMbUZDZtZOQ3DrtoqfAnwUkR8v09pFTCneDwHeLT57ZlZszS8xFXSWcDPgReA/ff2vZnafvtPgYnAdmqn3t5r8Fq+xLWEc845J1lfuXJl3dqoUaMqLXvYsPT64NZbb03WZ8+eXbc2ZcqUSstudKvpJUuW1K3NnTt09yzrXeLacJ89Ip4B6l1w/bUqTZlZ+/gTdGaZcNjNMuGwm2XCYTfLhMNulgmH3SwTvpX0EJA6zz5z5sxKr93oNtet/PtptOynnnoqWU/97h9//HGZlgYF30raLHMOu1kmHHazTDjsZplw2M0y4bCbZcJhN8uEz7MPAccee2zd2vTp05Pzzp8/P1k/6aSTkvVW/v1s3bo1WZ83b16yvnbt2ma2M2j4PLtZ5hx2s0w47GaZcNjNMuGwm2XCYTfLhMNulgmfZ7ekG264oWPLXrx4cceWPZj5PLtZ5hx2s0w47GaZcNjNMuGwm2XCYTfLhMNulomBjM8+AVgGjAMC6ImIeyQtBK4B3i2eenNErGnwWj7PbtZi9c6zDyTs44HxEfGcpKOADcBFwKXARxHxDwNtwmE3a716YT9sADPuBHYWjz+U9BJwXHPbM7NWO6R9dkmTgK8Avygm3SBpk6T7JI2uM89cSeslra/UqZlVMuDPxkv6IvA0cEdErJA0DthDbT/+dmqb+lc1eA1vxpu1WOl9dgBJI4DVwNqI+H4/9UnA6og4tcHrOOxmLVb6QhjVhtJcArzUN+jFgbv9vgn0Vm3SzFpnIEfjzwJ+DrwAfFZMvhm4HDiD2mb8NuDa4mBe6rW8ZjdrsUqb8c3isJu1nq9nN8ucw26WCYfdLBMOu1kmHHazTDjsZplw2M0y4bCbZcJhN8uEw26WCYfdLBMOu1kmHHazTDjsZploeMPJJtsDbO/z89hiWjfq1t66tS9wb2U1s7cT6hXaej375xYurY+IqR1rIKFbe+vWvsC9ldWu3rwZb5YJh90sE50Oe0+Hl5/Srb11a1/g3spqS28d3Wc3s/bp9JrdzNrEYTfLREfCLmmGpFckbZF0Uyd6qEfSNkkvSNrY6fHpijH0dkvq7TNtjKQnJL1WfO93jL0O9bZQ0o7ivdso6fwO9TZB0s8kvShps6Q/L6Z39L1L9NWW963t++yShgOvAl8H3gKeBS6PiBfb2kgdkrYBUyOi4x/AkDQN+AhYtn9oLUl/D7wXEd8t/lGOjoi/7pLeFnKIw3i3qLd6w4z/KR1875o5/HkZnViznwlsiYjXI+JXwMPArA700fUiYh3w3kGTZwFLi8dLqf2xtF2d3rpCROyMiOeKxx8C+4cZ7+h7l+irLToR9uOAN/v8/BbdNd57AI9L2iBpbqeb6ce4PsNsvQOM62Qz/Wg4jHc7HTTMeNe8d2WGP6/KB+g+76yI+D3gPOD6YnO1K0VtH6ybzp3+AJhMbQzAncD3OtlMMcz4I8C3I2Jv31on37t++mrL+9aJsO8AJvT5+fhiWleIiB3F993ASmq7Hd1k1/4RdIvvuzvcz69FxK6I2BcRnwE/pIPvXTHM+CPAjyNiRTG54+9df321633rRNifBU6U9CVJhwOXAas60MfnSBpZHDhB0kjgG3TfUNSrgDnF4znAox3s5QDdMox3vWHG6fB71/HhzyOi7V/A+dSOyG8F5neihzp9fRl4vvja3OnegIeobdZ9Qu3Yxp8BxwBPAq8B/w6M6aLeHqA2tPcmasEa36HezqK2ib4J2Fh8nd/p9y7RV1veN39c1iwTPkBnlgmH3SwTDrtZJhx2s0w47GaZcNjNMuGwm2Xi/wHTqVzmtKG2nwAAAABJRU5ErkJggg==\n" 648 | }, 649 | "metadata": { 650 | "needs_background": "light" 651 | } 652 | } 653 | ] 654 | }, 655 | { 656 | "cell_type": "code", 657 | "source": [ 658 | "# reshape the data to be in the format (batch_size, input_dim) because Dense layers require one dimensional input\n", 659 | "x_train = x_train.reshape(-1, 28 * 28)\n", 660 | "x_test = x_test.reshape(-1, 28 * 28)\n", 661 | "\n", 662 | "x_train.shape, x_test.shape" 663 | ], 664 | "metadata": { 665 | "colab": { 666 | "base_uri": "https://localhost:8080/" 667 | }, 668 | "id": "z0iSgwwxELmV", 669 | "outputId": "b6f21904-2248-4b31-b1a9-f105191dc02a", 670 | "pycharm": { 671 | "name": "#%%\n" 672 | } 673 | }, 674 | "execution_count": 12, 675 | "outputs": [ 676 | { 677 | "output_type": "execute_result", 678 | "data": { 679 | "text/plain": [ 680 | "((60000, 784), (10000, 784))" 681 | ] 682 | }, 683 | "metadata": {}, 684 | "execution_count": 12 685 | } 686 | ] 687 | }, 688 | { 689 | "cell_type": "code", 690 | "source": [ 691 | "# Normalize the data\n", 692 | "x_train = x_train / 255.0\n", 693 | "x_test = x_test / 255.0" 694 | ], 695 | "metadata": { 696 | "id": "E8sS-G2aEmUr", 697 | "pycharm": { 698 | "name": "#%%\n" 699 | } 700 | }, 701 | "execution_count": 13, 702 | "outputs": [] 703 | }, 704 | { 705 | "cell_type": "code", 706 | "source": [ 707 | "# convert the labels from integers to categorical using one hot encoding\n", 708 | "depth = len(set(y_train)) # calculate the number of classes to use as depth\n", 709 | "y_train = tf.one_hot(y_train, depth=depth)\n", 710 | "y_test = tf.one_hot(y_test, depth=depth)\n", 711 | "\n", 712 | "y_train.shape, y_test.shape" 713 | ], 714 | "metadata": { 715 | "colab": { 716 | "base_uri": "https://localhost:8080/" 717 | }, 718 | "id": "kONnV_mDEw6x", 719 | "outputId": "1f73f97e-41aa-4742-e383-606b5bf1418a", 720 | "pycharm": { 721 | "name": "#%%\n" 722 | } 723 | }, 724 | "execution_count": 14, 725 | "outputs": [ 726 | { 727 | "output_type": "execute_result", 728 | "data": { 729 | "text/plain": [ 730 | "(TensorShape([60000, 10]), TensorShape([10000, 10]))" 731 | ] 732 | }, 733 | "metadata": {}, 734 | "execution_count": 14 735 | } 736 | ] 737 | }, 738 | { 739 | "cell_type": "code", 740 | "source": [ 741 | "# fetch the input shape, i.e. the number of features\n", 742 | "input_shape = x_train.shape[1] # input_shape = 784\n", 743 | "\n", 744 | "model = tf.keras.Sequential(\n", 745 | " [\n", 746 | " # input layer\n", 747 | " tf.keras.layers.Input(shape=(input_shape)),\n", 748 | " # hidden layers\n", 749 | " tf.keras.layers.Dense(256, activation=\"relu\"),\n", 750 | " tf.keras.layers.Dense(128, activation=\"relu\"),\n", 751 | " # output layer\n", 752 | " tf.keras.layers.Dense(10, activation=\"softmax\"),\n", 753 | " ]\n", 754 | ")\n", 755 | "\n", 756 | "model.summary()" 757 | ], 758 | "metadata": { 759 | "colab": { 760 | "base_uri": "https://localhost:8080/" 761 | }, 762 | "id": "xNBIUnOhFCVs", 763 | "outputId": "37c4dbbe-b9e6-48af-c069-be66bb841d53", 764 | "pycharm": { 765 | "name": "#%%\n" 766 | } 767 | }, 768 | "execution_count": 21, 769 | "outputs": [ 770 | { 771 | "output_type": "stream", 772 | "name": "stdout", 773 | "text": [ 774 | "Model: \"sequential_3\"\n", 775 | "_________________________________________________________________\n", 776 | " Layer (type) Output Shape Param # \n", 777 | "=================================================================\n", 778 | " dense_10 (Dense) (None, 256) 200960 \n", 779 | " \n", 780 | " dense_11 (Dense) (None, 128) 32896 \n", 781 | " \n", 782 | " dense_12 (Dense) (None, 10) 1290 \n", 783 | " \n", 784 | "=================================================================\n", 785 | "Total params: 235,146\n", 786 | "Trainable params: 235,146\n", 787 | "Non-trainable params: 0\n", 788 | "_________________________________________________________________\n" 789 | ] 790 | } 791 | ] 792 | }, 793 | { 794 | "cell_type": "code", 795 | "source": [ 796 | "loss_fn = tf.losses.CategoricalCrossentropy()\n", 797 | "optimizer = tf.optimizers.Adam(learning_rate=0.001)\n", 798 | "metrics = [tf.keras.metrics.CategoricalAccuracy()]\n", 799 | "\n", 800 | "# Compile the model\n", 801 | "model.compile(optimizer=optimizer, loss=loss_fn, metrics=metrics)" 802 | ], 803 | "metadata": { 804 | "id": "MmAZIsPVFS12", 805 | "pycharm": { 806 | "name": "#%%\n" 807 | } 808 | }, 809 | "execution_count": 19, 810 | "outputs": [] 811 | }, 812 | { 813 | "cell_type": "code", 814 | "source": [ 815 | "model.fit(x_train, y_train, batch_size=32, epochs=10, validation_data=(x_test, y_test))" 816 | ], 817 | "metadata": { 818 | "colab": { 819 | "base_uri": "https://localhost:8080/" 820 | }, 821 | "id": "qM1_OmruFZu2", 822 | "outputId": "c6c8769e-175b-4d47-f5e0-eba32728c3bd", 823 | "pycharm": { 824 | "name": "#%%\n" 825 | } 826 | }, 827 | "execution_count": 20, 828 | "outputs": [ 829 | { 830 | "output_type": "stream", 831 | "name": "stdout", 832 | "text": [ 833 | "Epoch 1/10\n", 834 | "1875/1875 [==============================] - 14s 7ms/step - loss: 0.2043 - categorical_accuracy: 0.9399 - val_loss: 0.1031 - val_categorical_accuracy: 0.9667\n", 835 | "Epoch 2/10\n", 836 | "1875/1875 [==============================] - 11s 6ms/step - loss: 0.0857 - categorical_accuracy: 0.9734 - val_loss: 0.0970 - val_categorical_accuracy: 0.9703\n", 837 | "Epoch 3/10\n", 838 | "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0604 - categorical_accuracy: 0.9811 - val_loss: 0.0776 - val_categorical_accuracy: 0.9769\n", 839 | "Epoch 4/10\n", 840 | "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0454 - categorical_accuracy: 0.9858 - val_loss: 0.0729 - val_categorical_accuracy: 0.9786\n", 841 | "Epoch 5/10\n", 842 | "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0350 - categorical_accuracy: 0.9888 - val_loss: 0.0891 - val_categorical_accuracy: 0.9742\n", 843 | "Epoch 6/10\n", 844 | "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0292 - categorical_accuracy: 0.9899 - val_loss: 0.0819 - val_categorical_accuracy: 0.9780\n", 845 | "Epoch 7/10\n", 846 | "1875/1875 [==============================] - 11s 6ms/step - loss: 0.0240 - categorical_accuracy: 0.9923 - val_loss: 0.0742 - val_categorical_accuracy: 0.9815\n", 847 | "Epoch 8/10\n", 848 | "1875/1875 [==============================] - 11s 6ms/step - loss: 0.0207 - categorical_accuracy: 0.9929 - val_loss: 0.0873 - val_categorical_accuracy: 0.9779\n", 849 | "Epoch 9/10\n", 850 | "1875/1875 [==============================] - 11s 6ms/step - loss: 0.0179 - categorical_accuracy: 0.9943 - val_loss: 0.0920 - val_categorical_accuracy: 0.9785\n", 851 | "Epoch 10/10\n", 852 | "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0174 - categorical_accuracy: 0.9943 - val_loss: 0.1026 - val_categorical_accuracy: 0.9787\n" 853 | ] 854 | }, 855 | { 856 | "output_type": "execute_result", 857 | "data": { 858 | "text/plain": [ 859 | "" 860 | ] 861 | }, 862 | "metadata": {}, 863 | "execution_count": 20 864 | } 865 | ] 866 | } 867 | ] 868 | } --------------------------------------------------------------------------------