├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2024 Yuxuan Tong and others 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Awesome LLM Research 2 | 3 | > Curation of resources for **LLM research**, **screened** by @tongyx361 to ensure **high quality** and accompanied with **elaborately-written concise descriptions** to help readers get the gist as quickly as possible. 4 | 5 | [![Awesome](https://awesome.re/badge.svg)](https://github.com/tongyx361/Awesome-LLM-Research) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) 6 | 7 | 🐱 [GitHub](https://github.com/tongyx361/Awesome-LLM-Research) | 📝 [Notion (Interactable)](https://tongyx361.notion.site/Awesome-LLM-Research-7b999071d476409cb1fbfdd081f87086) | 🐦 [X(Twitter)](https://twitter.com/tongyx361/status/1780956572384145515) | 🐶 [Zhihu(知乎)](https://zhuanlan.zhihu.com/p/708331040) 8 | 9 | ✨ Featured by: 10 | 11 | - Theory & practice **comprehensive introductory** materials. 12 | - **Classic/high-quality** information sources. 13 | - **Latest hot-spot** information sources. 14 | 15 | 📊 There is also [an **interactable (i.e. sort / filter / search)** version of the following table](https://tongyx361.notion.site/6958f3f8753a4458813991a709894699?v=af2e57fc6c274a74a1404452c9014bb4). 16 | 17 | 📥 You can **subscribe to our updates** in the following ways: 18 | 19 | - **Follow** the [**X(Twitter) account** @tongyx361](https://x.com/tongyx361), 20 | - **Follow** the [**Zhihu(知乎) account** @天欲雪](https://www.zhihu.com/people/bai-li-tian-he-84), 21 | - **Watch releases in this GitHub repository**: upper right corner→Watch->Custom->Releases. 22 | 23 | 📢 If you have any **suggestions**, please don't hesitate to 24 | 25 | - **comment** in the [**Notion** page](https://www.notion.so/tongyx361/Awesome-LLM-Research-7b999071d476409cb1fbfdd081f87086), 26 | - **reply** to the [**X(Twitter)** thread](https://twitter.com/tongyx361/status/1780956572384145515), 27 | - post an **issue** in the [**GitHub** repository](https://github.com/tongyx361/Awesome-LLM-Research), 28 | - or [**E-mail** *Yuxuan Tong*](tongyuxuan361@gmail.com). 29 | 30 | | Link | Abstract | Description | Language | Modality | Update Cycle | Type | 31 | | ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ------------ | ----------- | 32 | | [国立台湾大学: 李宏毅机器学习 - CS自学指南](https://csdiy.wiki/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/LHY/) | **Basic theory and fundamental works** of Deep Learning | Lectures from different years have different focuses, e.g. 2023 focuses on LLM. | EN(Text) ZH(Speech) | Speech Text Code | Year | Basic | 33 | | [Introduction - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/chapter1/1) | Basic NLP **practice** (based on HuggingFace ecosystem) | *HuggingFace* is so accessible that its success is a given (but this also comes with some hidden price for developers). | EN ZH … | Text Code | Dynamic | Basic | 34 | | [Yao Fu’s Blog](https://yaofu.notion.site/Yao-Fu-s-Blog-b536c3d6912149a395931f1e871370db) | Fundamental research topics **walkthrough** | Such as emergent abilities, reasoning, long-context modeling. | EN | Text | Months | Fundamental | 35 | | [Transformer Math 101 \| EleutherAI Blog](https://blog.eleuther.ai/transformer-math/) | *Transformer*-related math estimation - Basic | Basic arithmetic about *Transformer*-based models. | EN | Text | None | Basic | 36 | | [分析transformer模型的参数量、计算量、中间激活、KV cache - 知乎](https://zhuanlan.zhihu.com/p/624740065) | *Transformer*-related math estimation - Mediate | Detailed analysis of calculations in *Transformer*-based model. | ZH | Text | None | Basic | 37 | | [*紫气东来* - 知乎](https://www.zhihu.com/people/zi-qi-dong-lai-1/posts) | **Specific** engineering details | Such as inference and training frameworks. | ZH | Text | Weeks | Practical | 38 | | [GitHub - liguodongiot/llm-action](https://github.com/liguodongiot/llm-action?tab=readme-ov-file) | Engineering detail **summaries** | Summarizing AI engineering techniques, such as inference, parallel computing, etc. | ZH | Text | Days | Practical | 39 | | 微信公众号：*大猿搬砖简记* | **Illustrated** **source code (e.g. vLLM, CUDA)** and algorithms (e.g. FlashAttention) | | ZH | Text | Weeks | Practical | 40 | | [游凯超 - 知乎](https://www.zhihu.com/people/youkaichao) | **Infrastructure-level** engineering details | Such as *CUDA*, *NCCL*, `torch.compile` and other side infrastructures like *Docker*, etc. | ZH | Text | Days | Practical | 41 | | [Alignment Guidebook - Notion](https://efficient-unicorn-451.notion.site/Alignment-Guidebook-e5c64df77c0a4b528b7951e87337fa78) | Introduction to LLM **Alignment (SFT + RL)** | | EN | Text | Dynamic | Basic | 42 | | [Spinning Up in Deep RL! — Spinning Up documentation](https://spinningup.openai.com/en/latest/) | Basic **Deep RL** | | EN | Text Code
| None | Basic | 43 | | [科学空间\|Scientific Spaces](https://kexue.fm/) | Blogs combining **graceful theories** and solid experiments | Blogs by *Jianlin Su (苏剑林)*, the author of *RoPE* (de facto standard of positional encoding now), versed in math and ML theory while not unfamiliar with experiments and practice. | ZH | Text | Weeks | Fundamental | 44 | | [Research](https://openai.com/research) | ***OpenAI*** research blogs | “We keep re-discovering what *OpenAI* discovered five years ago.” | EN | Text | Months | Fundamental | 45 | | [Research \\ Anthropic](https://www.anthropic.com/research) | ***Anthropic*** research blogs | | EN | Text | Months | Fundamental | 46 | | [Transformer Circuits Thread](https://transformer-circuits.pub/) | Amazingly insightful and **open** ***Anthropic*** **interpretability** team research blogs | | EN | Text | Month | Fundamental | 47 | | E.g. [\[2312.11805\] Gemini: A Family of Highly Capable Multimodal Models](https://arxiv.org/abs/2312.11805) | LLM **technical reports** | Such technical reports, while usually not very detailed, often do reveal some important details of SotA LLMs. | EN | Text | Months | Fundamental | 48 | | [Hazy Research](https://hazyresearch.stanford.edu/blog) | Blogs of **pioneer visions** | Blogs from *Hazy Research* led by *Christopher Ré* @ *Stanford* (one of the best NLP&AI research groups around the world). | EN | Text | Months | Fundamental | 49 | | [Ilya 30u30](https://arc.net/folder/D0472A20-9C20-4D3F-B145-D2865C0A9FEE) | Short reading list to understand **the fundamentals of the AI today**, said to be **from *Ilya***. | Not the most frontier and not the most suitable for research starters, but really fundamental for essential understanding. | EN | Text | None | Fundamental | 50 | | [FAI-Seminar](https://www.fai-seminar.ac.cn/) | High-quality talks (largely contributed by **Yao class alumna**) | | ZH | Speech Text | Week | Trending | 51 | | [Cool Papers - Immersive Paper Discovery](https://papers.cool/) | **Daily *arXiv*** paper & *Kimi* interaction | | EN | Text | Day | Trending | 52 | | [Daily Papers - Hugging Face](https://huggingface.co/papers) | The most popular paper selection on *Twitter*. | | EN | Text | Day | Trending | 53 | | 微信公众号： *SparksofAGI* | Individual paper selection, some of which **common popular paper collections might not notice** | Selected by *Jianbo Dai* (戴建波)* (senior researcher at *Huawei*). | ZH | Text | Weeks | Trending | 54 | | 微信公众号： *AINLP* | **Curations** of other AI 微信公众号：s | | ZH | Text | Day | Trending | 55 | | 中文 AI 媒体四大顶号：*机器之心*、*新智元*、*量子位*、*夕小瑶科技说* | **Popular** paper selection | | ZH | Text | Day | Trending | 56 | | 微信公众号： *arXiv 每日学术速递* | *arXiv* paper from **broader domains** | | ZH | Text | Day | Auxiliary | 57 | | 微信公众号： *AI 前线* | Various AI news **(not limited to research)** | | ZH | Text | Day | Auxiliary | 58 | | Video channel *Song Zhao* ([*YouTube*](https://www.youtube.com/@zhaosong2031) / [*BiliBili*](https://space.bilibili.com/3546587376650961)) | Various **practical academic-relevant affairs** (e.g. paper submission, job choices) | A little “abstract” though … | ZH | Speech Text | Weeks | Auxiliary | 59 | --------------------------------------------------------------------------------