├── img ├── README.md └── Banner_DL.png ├── LICENSE ├── .gitignore └── README.md /img/README.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /img/Banner_DL.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ElizaLo/Deep-Learning/HEAD/img/Banner_DL.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Yelyzaveta L 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | [![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FElizaLo%2FDeep-Learning&count_bg=%2304749C&title_bg=%23B5B5BC&icon=python.svg&icon_color=%23E7E7E7&title=Repository+Views&edge_flat=false)](https://hits.seeyoufarm.com) 4 | 5 | This repository contains examples of deep learning algorithms implemented in Python with mathematics behind them being explained. 6 | 7 | > - [ ] For **Machine Learning** algorithms please check [Machine Learning](https://github.com/ElizaLo/Machine-Learning) repository. 8 | 9 | > - [ ] For **Natural Language Processing** (NLU = NLP + NLG) please check [Natural Language Processing](https://github.com/ElizaLo/NLP-Natural-Language-Processing) repository. 10 | 11 | > - [ ] For **Computer Vision** please check [Computer Vision](https://github.com/ElizaLo/Computer-Vision) repository. 12 | 13 | 14 | ## 🎓 University Courses 15 | 16 | - [ ] [CS 231N: Convolutional Neural Networks for Visual Recognition, Stanford](https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv) 17 | - [ ] [CS 224N: Natural Language Processing with Deep Learning, Stanford](https://www.youtube.com/playlist?list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z) 18 | - [ ] [Machine Learning Crash Course](https://techdevguide.withgoogle.com/paths/machine-learning/featured/ml-crash-course#) 19 | - [ ] [fast.ai: Practical Deep Learning for Coders](https://course.fast.ai)" 20 | - [ ] [CS 285: Deep Reinforcement Learning, UC Berkeley](http://rail.eecs.berkeley.edu/deeprlcourse/) 21 | - [ ] [CSC 2541: Differentiable Inference and Generative Models](http://www.cs.toronto.edu/~duvenaud/courses/csc2541/index.html) 22 | - [ ] [MIT 6.S191: Introduction to Deep Learning](http://introtodeeplearning.com) 23 | - [MIT 6.S191: Introduction to Deep Learning ](https://www.youtube.com/playlist?list=PLtBw6njQRU-rwp5__7C0oIVt26ZgjG9NI) 24 | - [ ] [Frontiers of Deep Learning (Simons Institute)](https://www.youtube.com/playlist?list=PLgKuh-lKre11ekU7g-Z_qsvjDD8cT-hi9) 25 | - [Course website](https://simons.berkeley.edu/workshops/dl2019-1) 26 | - [ ] [New Deep Learning Techniques](https://www.youtube.com/playlist?list=PLHyI3Fbmv0SdM0zXj31HWjG9t9Q0v2xYN) 27 | - [Course website](http://www.ipam.ucla.edu/programs/workshops/new-deep-learning-techniques/?tab=overview) 28 | - [ ] [Geometry of Deep Learning (Microsoft Research)](https://www.youtube.com/playlist?list=PLD7HFcN7LXRe30qq36It2XCljxc340O_d) 29 | - [Course website](https://www.microsoft.com/en-us/research/event/ai-institute-2019/) 30 | - [ ] [Deep Multi-Task and Meta Learning (Stanford CS330)](https://www.youtube.com/playlist?list=PLoROMvodv4rMC6zfYmnD7UG3LVvwaITY5) 31 | - [Course Website](http://cs330.stanford.edu/) 32 | - [ ] [Advanced Deep Learning & Reinforcement Learning 2020 (DeepMind / UCL)](https://www.youtube.com/playlist?list=PLqYmG7hTraZDNJre23vqCGIVpfZ_K2RZs) 33 | - [ ] [Deep Reinforcment Learning, Decision Making and Control (UC Berkeley CS285)](https://www.youtube.com/playlist?list=PLkFD6_40KJIwhWJpGazJ9VSj9CFMkb79A) 34 | - [ ] [Full Stack Deep Learning 2019](https://www.youtube.com/playlist?list=PL1T8fO7ArWlcf3Hc4VMEVBlH8HZm_NbeB) 35 | - [ ] [Emerging Challenges in Deep Learning](https://www.youtube.com/playlist?list=PLgKuh-lKre10BpafDrv0fg2VNUweWXWVd) 36 | - [ ] [Deep|Bayes 2019 Summer School](https://www.youtube.com/playlist?list=PLe5rNUydzV9QHe8VDStpU0o8Yp63OecdW) 37 | - [ ] [Workshop on Theory of Deep Learning: Where next (Institure for Advanced Study)](https://www.youtube.com/playlist?list=PLdDZb3TwJPZ5dqqg_S-rgJqSFeH4DQqFQ) 38 | - [ ] [Deep Learning: Alchemy or Science? (Institure for Advanced Study)](https://www.youtube.com/playlist?list=PLdDZb3TwJPZ7aAxhIHALBoh8l6-UxmMNP) 39 | 40 | ## 🔹 Coursera Courses 41 | 42 |
43 | List of Coursera Courses 44 | 50 |
51 | 52 | 53 | ## 📚 Books 54 |
55 | List of Books 56 | 63 |
64 | 65 | ## Papers 66 | 67 | | Title | Description, Information | 68 | | :---: | :--- | 69 | |[Deep Learning Papers Reading Roadmap](https://github.com/floodsung/Deep-Learning-Papers-Reading-Roadmap)|Deep Learning papers reading roadmap for anyone who are eager to learn this amazing tech!| 70 | 71 | 72 | ## Videos 73 |
74 | Other useful links 75 | 80 |
81 | 82 | 83 | ## :octocat: GitHub Repositories 84 | 85 | | Title | Description, Information | 86 | | :---: | :--- | 87 | |[NVIDIA Deep Learning Examples for Tensor Cores](https://github.com/NVIDIA/DeepLearningExamples)|Deep Learning Examples| 88 | 89 | ## Contests 90 |
91 | Other useful links 92 | 99 |
100 | 101 | ## 📌 Other 102 |
103 | Other useful links 104 | 109 |
110 | 111 | * [Caffe](https://github.com/weiliu89/caffe) – a fast open framework for deep learning; 112 | * [Deep Learning](http://www.deeplearningbook.org) - Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016); 113 | * [Deep Learning](https://www.udacity.com/course/deep-learning--ud730) от Google — короткий курс для продвинутых. Основное внимание уделяется библиотеке для глубинного обучения TensorFlow; 114 | * [Deep Learning at Oxford](https://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu) (2015) – a YouTube playlist with lectures ([read more](http://www.cs.ox.ac.uk/teaching/courses/2014-2015/ml/)); 115 | * [awesome-deep-vision](https://github.com/kjw0612/awesome-deep-vision) – a curated list of deep learning resources for computer vision; 116 | * [awesome-deep-learning-papers](https://github.com/terryum/awesome-deep-learning-papers) – a curated list of the most cited deep learning papers (since 2010); 117 | * [Deep Learning Tutorials](https://github.com/subokita/DeepLearningTutorials) – notes and code; 118 | * [dl-docker](https://github.com/saiprashanths/dl-docker) – an all-in-one Docker image for deep learning. Contains all the popular DL frameworks (TensorFlow, Theano, Torch, Caffe, etc.); 119 | * [Self-Study Courses for Deep Learning](https://developer.nvidia.com/deep-learning-courses) от NVDIA — self-paced classes for deep learning that feature interactive lectures, hands-on exercises, and recordings of the office hours Q&A with instructors. You’ll learn everything you need to design, train, and integrate neural network-powered artificial intelligence into your applications with widely used open-source frameworks and NVIDIA software. During the hands-on exercises, you will use GPUs and deep learning software in the cloud; 120 | * [deep-rl-tensorflow](https://github.com/carpedm20/deep-rl-tensorflow) - ensorFlow implementation of Deep Reinforcement Learning papers; 121 | * [TensorFlow 101](https://github.com/sjchoi86/Tensorflow-101) – Tensorflow tutorials; 122 | * [Introduction to Deep Learning for Image Recognition](https://github.com/rouseguy/scipyUS2016_dl-image) – this notebook accompanies the Introduction to Deep Learning for Image Recognition workshop to explain the core concepts of deep learning with emphasis on classifying images as the application; 123 | 124 | ## Main skills required by the Deep Learning Engineer / Deep Learning Research Engineer 125 | 126 | > The [research](https://apps.ucu.edu.ua/en/articles-and-research/data-science-job-market-2020-1/data-scientists-skills-2020-1/) made by **Faculty of Applied Sciences at UCU**. Link on main [article](https://apps.ucu.edu.ua/en/articles-and-research/data-science-job-market-2020-1/). 127 | 128 | ### Deep Learning Engineer / Deep Learning Research Engineer 129 | 130 | 1. Python3: numpy, scikit-learn, pandas, scipy. 131 | 2. Statistics (regression, properties of distributions, statistical tests, and proper usage, etc.) and probability theory. 132 | 3. Deep learning frameworks: Tensorflow, PyTorch; MxNet, Caffe, Keras. 133 | 4. Deep learning architectures: VGG, ResNet, Inception, MobileNet. 134 | 5. Deepnets, hyperparameter optimization, visualization, interpretation. 135 | 6. Machine learning models. 136 | 137 | ### Python for Deep Learning and Research 138 | 139 | - Basic algorithms and common tasks 140 | - Classical algorithms 141 | - Computational complexity 142 | - Useful Libraries and Frameworks 143 | - CPU vs GPU parallelization 144 | - Cloud and GPU Integration 145 | - Data Visualization 146 | - Vectors and Vectorization 147 | - Image Processing 148 | - Language Processing 149 | 150 | ### Mathematics for Deep Learning 151 | 152 | - Common Notation and Core Ideas 153 | - Linear Algebra 154 | - N-dim Spaces 155 | - Vectors, Matrices and Operators 156 | - Mathematical and Function Analysis calculus 157 | - Derivative and Partial derivative 158 | - Chain Rule 159 | - Probability theory 160 | - Introduction to Statistics 161 | 162 | ### Linear, Polynomial and Multivariate Regression 163 | 164 | - Price prediction Task 165 | - Linear Regression 166 | - Least square method 167 | - Loss Function 168 | - Optimization Task 169 | - Gradient Descent 170 | - MLE — Maximum Likelihood Estimation 171 | - Data Preprocessing 172 | - Model Visualization 173 | - Data Normalization 174 | - Polynomial Regression 175 | - Multivariate Regression 176 | 177 | ### Introduction Computer Vision 178 | 179 | - Basic idea of Computer Vision 180 | - Classical Computer Vision 181 | - Deep Learning and CV 182 | - Core Idea of Semantic Gap 183 | - Classification Task 184 | - N-dim Spaces and Metrics 185 | - Common datasets 186 | - Mnist and Fashion-Mnist 187 | - Cifar10 and Cifar100 188 | - Cats vs Dogs 189 | - ImageNet and MS COCO 190 | - Euclidean Distance 191 | - Nearest Neighbour 192 | 193 | ### Classification and Computer Vision 194 | 195 | - Image Classification 196 | - Cosine Similarity 197 | - Manhattan distance 198 | - KNN 199 | - Train / Val / Test data split 200 | - Logistic Regression 201 | - Logistic Regression and Maximum Likelihood Estimation 202 | - Loss function and Cross Entropy 203 | - Accuracy and Metrics 204 | - Precision, Recall and F1 205 | 206 | ### Neural Networks 207 | 208 | - Rosenblatt’s Perceptron 209 | - Artificial Neuron 210 | - Warren McCulloch and Walter Pitts Neuron 211 | - Fully Connected (Linear, Dense, Affine) Layer 212 | - Activation Layers 213 | - BackPropagation Algorithm 214 | - Stochastic Gradient Descent 215 | - Biological Neuron and Analogy 216 | 217 | ### Computation graphs and Deep Learning Frameworks 218 | 219 | - Computational graphs 220 | - Differentiable graphs 221 | - Deep Learning Frameworks 222 | - Custom Framework Realization 223 | - Linear operations and Activation Realizations 224 | - Main Blocks Of Deep Learning FrameWorks 225 | - Custom Model and Train 226 | - Optimizator realization 227 | - TensorFlow 228 | - Keras 229 | - PyTorch 230 | 231 | ### Deep Learning 232 | - Neural Networks Problems 233 | - Activation Functions 234 | - Weights Initialization 235 | - Initialization Techniks 236 | - Overfitting and Underfitting 237 | - Regularization Methods 238 | - L1 and L2 Regularization 239 | - Ensemble of Models 240 | - Dropout 241 | - Hyper Parameters Search 242 | - Optimizations behind SGD 243 | - Momentum and Nesterov Momentum 244 | - Adagrad, RMSprop 245 | - Adam, Nadam 246 | - Batch-Normalization 247 | 248 | ### Unsupervised Learning 249 | 250 | - Dimensionality reduction 251 | - Feature Learning 252 | - Vector Representation 253 | - Embeddings 254 | - Kernel Method 255 | - Clusterization 256 | - k-means Clusterization 257 | - Hierarchical Clusterization 258 | - Neural Networks and Unsupervised Learning 259 | - Autoencoders 260 | - Autoencoders architectures 261 | - Tasks for Autoencoders 262 | - Problem of Image Generation 263 | - Image Denoising Task 264 | 265 | ### Introduction to Deep Learning in Computer Vision 266 | 267 | - Problems of Fully Connected Neural Networks 268 | - Towards Convolution Neural Network 269 | - CNN as feature extractor 270 | - Computer Vision tasks 271 | - Transfer Learning 272 | - Transfer Learning in Practice 273 | - What Next (breath: CNN Architectures, Image Detection, Segmentation, GANs) 274 | 275 | ### Introduction to Natural Language Processing 276 | 277 | - Introduction to Natural Language Processing 278 | - Text classification 279 | - Words Preprocessing and Representation 280 | - Part-of-Speech tagging (PoS tagging) 281 | - Tokenization, Lemmatization and Stemming 282 | - Bag of Words 283 | - TF-IDF 284 | - Distributive semantics 285 | - Vector Semantics 286 | - Term-document matrix 287 | - Word context matrix 288 | - Dense Vectors and Embeddings 289 | - Word2Vec 290 | - What Next (breath: RNN, Seq2Seq, Attention, Transformers, Modern Language Models) 291 | --------------------------------------------------------------------------------