├── .gitignore ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | .idea/ 4 | *.py[cod] 5 | *$py.class 6 | 7 | # C extensions 8 | *.so 9 | 10 | # Distribution / packaging 11 | .Python 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | cover/ 54 | 55 | # Translations 56 | *.mo 57 | *.pot 58 | 59 | # Django stuff: 60 | *.log 61 | local_settings.py 62 | db.sqlite3 63 | db.sqlite3-journal 64 | 65 | # Flask stuff: 66 | instance/ 67 | .webassets-cache 68 | 69 | # Scrapy stuff: 70 | .scrapy 71 | 72 | # Sphinx documentation 73 | docs/_build/ 74 | 75 | # PyBuilder 76 | .pybuilder/ 77 | target/ 78 | 79 | # Jupyter Notebook 80 | .ipynb_checkpoints 81 | 82 | # IPython 83 | profile_default/ 84 | ipython_config.py 85 | 86 | # pyenv 87 | # For a library or package, you might want to ignore these files since the code is 88 | # intended to run in multiple environments; otherwise, check them in: 89 | # .python-version 90 | 91 | # pipenv 92 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 93 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 94 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 95 | # install all needed dependencies. 96 | #Pipfile.lock 97 | 98 | # poetry 99 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 100 | # This is especially recommended for binary packages to ensure reproducibility, and is more 101 | # commonly ignored for libraries. 102 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 103 | #poetry.lock 104 | 105 | # pdm 106 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 107 | #pdm.lock 108 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 109 | # in version control. 110 | # https://pdm.fming.dev/#use-with-ide 111 | .pdm.toml 112 | 113 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 114 | __pypackages__/ 115 | 116 | # Celery stuff 117 | celerybeat-schedule 118 | celerybeat.pid 119 | 120 | # SageMath parsed files 121 | *.sage.py 122 | 123 | # Environments 124 | .env 125 | .venv 126 | env/ 127 | venv/ 128 | ENV/ 129 | env.bak/ 130 | venv.bak/ 131 | 132 | # Spyder project settings 133 | .spyderproject 134 | .spyproject 135 | 136 | # Rope project settings 137 | .ropeproject 138 | 139 | # mkdocs documentation 140 | /site 141 | 142 | # mypy 143 | .mypy_cache/ 144 | .dmypy.json 145 | dmypy.json 146 | 147 | # Pyre type checker 148 | .pyre/ 149 | 150 | # pytype static type analyzer 151 | .pytype/ 152 | 153 | # Cython debug symbols 154 | cython_debug/ 155 | 156 | # PyCharm 157 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 158 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 159 | # and can be added to the global gitignore or merged into this file. For a more nuclear 160 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 161 | #.idea/ 162 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 BEAM-Labs 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DLTutorial 2 | 新学生深度学习入门，可以按照顺序和需求自行安排、调节进度。 3 | 4 | # 1. 基础入门功课、tutorial等 5 | ## 1.1. Andrew Ng 的 Deep Learning Specialization及课后作业 6 | 视频地址： https://www.bilibili.com/video/BV1pJ41127Q2 7 | HW: https://github.com/ppx-hub/deep-learning-specialization-all-homework/tree/main/Homework-NoAnswer 8 | HW solution: https://github.com/amanchadha/coursera-deep-learning-specialization 9 | 10 | - S1: Neural Networks and Deep Learning 11 | - video: from **P9 - 1: What is a neural network** to **P50 - 8: What does this have to do with the brain?** 12 | - topics covered: logistic regression, computational graph, activation function, backpropagation and etc 13 | 14 | - S2: Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization 15 | - video: from **P51 - 1: Train / Dev / Test sets** to **P85 - 11: TensorFlow**, 注：现在均使用pytorch 16 | - topics covered: bias variance tradeoff, regularization, dropout, gradient descent(Momentum, RMSprop, Adam), learning rate decay, batch normalization and etc 17 | 18 | - S3: Structuring Machine Learning Projects 19 | - video: from **P86 -1 : Why ML Strategy** to **P107 - 10: Whether to use End-to-end Deep Learning** 20 | - topics covered: transfer learning, multi-task learning and etc 21 | - 这个好像是没有作业的？ 22 | 23 | - S4: Convolutional Neural Networks 24 | - video: from **P108 - 1: Computer Vision** to **P150 - 11: 1D and 3D Generalizations** 25 | - topics covered: CNN basics (padding, pooling and etc), ResNet, Data augmentation, YOLO, U-Net, Siamese Network and etc 26 | 27 | - S5: Sequence Models 28 | - video: from **P151 - 1: Why Sequence Models?** to **P180 - 8: Attention Model** 29 | - video: transformer network部分我在b站没有找到视频，有找到的同学可以补充，youtube视频可以参考这里：https://www.youtube.com/watch?v=S7oA5C43Rbc&t=18037s， 时间大概从5小时左右开始 30 | - topics covered: GRU, LSTM, word2vec, glove, beam search, attention model, transformer and etc 31 | 32 | ## 1.2. 李宏毅机器学习 33 | - B站2023年课程链接：https://www.bilibili.com/video/BV1TD4y137mP 34 | - B站2024年课程链接（主要为生成式AI，如GPT和Diffusion）：https://www.bilibili.com/video/BV1BJ4m1e7g8 35 | - 2023年课程网页：https://speech.ee.ntu.edu.tw/~hylee/ml/2023-spring.php 36 | - 2024年课程网页：https://speech.ee.ntu.edu.tw/~hylee/genai/2024-spring.php 37 | - 最大优点是课程网页上有代码作业，课程上难度偏简单，主要以讲解概念为主。 38 | 39 | ## 1.3. 动手学强化学习 40 | - 课程连接：https://hrl.boyuai.com/chapter/intro/ 41 | - 视频连接：https://www.boyuai.com/elites/course/xVqhU42F5IDky94x 42 | - 风格与《动手学深度学习》类似，但不是李沐团队的课程。上交老师出品，内容还是不错的。 43 | 44 | # 2. 经典入门文章，所有研究方向必读 45 | ## 2.1. transformer 架构细节学习 46 | - transformer比较重要，请首先阅读[原文](https://arxiv.org/abs/1706.03762)，没有看懂的部分记录下来 47 | - 完成 https://www.bilibili.com/video/BV1pu411o7BE/， 看是否能够回答之前记录的问题， 并找出自己阅读论文时候漏掉的关键点 48 | - 阅读 [illustrated transformer](https://jalammar.github.io/illustrated-transformer/) 49 | - 完成 [annotated transformer](http://nlp.seas.harvard.edu/annotated-transformer/)，熟悉代码细节 50 | 51 | 52 | ## 2.2. Pre-trained Language Models (PLM) 53 | limu paper reading repo: https://github.com/mli/paper-reading 54 | - 2.2.1. 阅读下列预训练语言模型文章: GPT, BERT, GPT2, 请**先自己阅读**，之后和[这里](https://github.com/mli/paper-reading)进行对比，比较下自己是否漏读了重要内容 55 | - **Improving Language Understanding by Generative Pre-Training**. *Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever*. Preprint. [[pdf](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)] [[project](https://openai.com/blog/language-unsupervised/)] (**GPT**) 56 | - **BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding**. *Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova*. NAACL 2019. [[pdf](https://arxiv.org/pdf/1810.04805.pdf)] [[code & model](https://github.com/google-research/bert)] 57 | - **Language Models are Unsupervised Multitask Learners**. *Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever*. Preprint. [[pdf](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)] [[code](https://github.com/openai/gpt-2)] (**GPT-2**) 58 | - **RoBERTa: A Robustly Optimized BERT Pretraining Approach**. *Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov*. Preprint. [[pdf](https://arxiv.org/pdf/1907.11692.pdf)] [[code & model](https://github.com/pytorch/fairseq)] (optional) 59 | 60 | - 2.2.2. 阅读seq2seq语言模型相关文章 61 | - **Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer**. *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*. Preprint. [[pdf](https://arxiv.org/pdf/1910.10683.pdf)] [[code & model](https://github.com/google-research/text-to-text-transfer-transformer)] (**T5**) 62 | - **mT5: A massively multilingual pre-trained text-to-text transformer**. *Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel* Preprint. [[pdf](https://arxiv.org/abs/2010.11934)](**mT5**) 63 | 64 | - 2.2.3. 建议完成语言模型的pre-train (demo)和fine-tuning 65 | - 请**不要**直接调用huggingface中[run_glue.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py)里的trainer进行fine-tune， 但可以使用其下载和load data 66 | - 也可以利用[script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)自行下载[数据](https://gluebenchmark.com/tasks)， 67 | - fine-tune language models (BERT, GPT, RoBERTA, T5 and etc) on GLUE benchmark (MRPC和RTE数据集较小，可以优先只考虑这两个数据集)，注意调整超参数用来获得更好的结果 68 | - 在[wikitext](https://huggingface.co/datasets/wikitext)上pre-train 自己的语言模型，在MRPC和RTE数据集上fine-tune, 并和BERT的结果进行比较 69 | 70 | 71 | ## 2.3. Multi-modal, diffusion, LLM and etc (TODO) 72 | 73 | 74 | # 3. 根据研究方向的细分文章 (TODO) 75 | ## 3.1. Bio Language Models 76 | ## 3.2. Structure Prediction 77 | ## 3.3. Proteomics 78 | ## 3.4. Design and et al. 79 | 80 | 81 | # 4. Miscellaneous 82 | ## 4.1. Good informative talks 83 | - [Stanford CS25: V4 I Jason Wei & Hyung Won Chung of OpenAI](https://www.youtube.com/watch?v=3gb-ZkVRemQ)：讲叙一些基本原理和思考，强烈建议 84 | - [Yannic Kilcher](https://www.youtube.com/@YannicKilcher), 他的文章讲解方式是非常好的，值得学习。例如 [transformer](https://www.youtube.com/watch?v=iDulhoQ2pro), [Sparse Expert Models](https://www.youtube.com/watch?v=ccBMRryxGog&list=LL&index=21), [switch transformer](https://www.youtube.com/watch?v=ccBMRryxGog&list=LL&index=21) and etc. 85 | --------------------------------------------------------------------------------