├── .gitignore ├── LICENSE ├── README.md ├── diffusion_planning_slides.pdf ├── diffusion_planning_tutorial.pdf ├── figs ├── ACT.png ├── GAIL_algo.png ├── GAIL_paper.png ├── IL_limit.png ├── classifierfree_algo.png ├── dalle.png ├── defined_guidance.png ├── denoise_image.gif ├── denoise_traj.gif ├── diffuse_teaser.gif ├── diffuse_what.png ├── distribution_match.gif ├── generative_model.png ├── guidance.png ├── guidance_algo.png ├── guidance_function.png ├── image_diffuse.jpg ├── inpaint.gif ├── noisy_image.png ├── offline_limitation.png ├── planner_policy_model.png ├── planning_error.jpeg ├── policy_vs_diffusion.png ├── safe_diffuser.png ├── score_function.png ├── traj.png └── traj_diffuse.png └── slides.md /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Spyder project settings 132 | .spyderproject 133 | .spyproject 134 | 135 | # Rope project settings 136 | .ropeproject 137 | 138 | # mkdocs documentation 139 | /site 140 | 141 | # mypy 142 | .mypy_cache/ 143 | .dmypy.json 144 | dmypy.json 145 | 146 | # Pyre type checker 147 | .pyre/ 148 | 149 | # pytype static type analyzer 150 | .pytype/ 151 | 152 | # Cython debug symbols 153 | cython_debug/ 154 | 155 | # PyCharm 156 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 157 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 158 | # and can be added to the global gitignore or merged into this file. For a more nuclear 159 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 160 | #.idea/ 161 | 162 | .DS_Store 163 | 164 | node_modules/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Diffusion Model for Control and Planning Tutorial 2 | 3 | > Chaoyi Pan 2024.2.26 4 | 5 | Tutorial outline: 6 | 7 | 1. 🔄 Recap: What is a Diffusion Model / What Problem Does It Solve? 8 | 2. 🚀 Motivation: Why Do We Need a Diffuser in Control and Planning? 9 | 3. 🛠️ Practice: How to Use a Diffuser in Control and Planning? 10 | 4. 📚 Literatures: Recent Research in a Diffuser for Control and Planning 11 | 5. 📝 Summary & Limitations: What We Can Do and What We Cannot Do 12 | 13 | ## 🔄 Recap: Diffusion Model 14 | 15 | A diffusion model is a generative model capable of generating samples from a given distribution. It serves as a powerful tool for distribution matching, extensively utilized in image generation, text generation, and other creative tasks. 16 | 17 | ![diffusion examples](figs/dalle.png) 18 | 19 | At the core of the diffusion model is the score function, representing the noise direction used to update samples to match the target distribution. Through learning this score function, the diffusion model can adeptly generate samples from the specified distribution. 20 | 21 | $$ 22 | \boldsymbol{x}_{i+1} \leftarrow \boldsymbol{x}_i+c \nabla \log p\left(\boldsymbol{x}_i\right)+\sqrt{2 c} \boldsymbol{\epsilon}, \quad i=0,1, \ldots, K 23 | $$ 24 | 25 | ![Annealed Langevin dynamics combine a sequence of Langevin chains with gradually decreasing noise scales.](figs/distribution_match.gif) 26 | 27 | The diffusion model have several advantages over other generative models, including but not limited to the following: 28 | 29 | * **Multimodal**: It effectively handles multimodal distributions, a challenge often encountered when directly predicting distributions. 30 | * **Scalable**: This approach scales well with high-dimensional distribution matching problems, making it versatile for various applications. 31 | * **Stable**: Grounded in solid math and a standard multi-stage diffusion training procedure, the model ensures stability during training. 32 | * **Non-autoregressive**: Its capability to predict entire trajectory sequences in a non-autoregressive manner enables efficient handling of non-autoregressive and multimodal distribution matching challenges. 33 | 34 | ## 🚀 Motivation: Why Do We Need a Diffuser in Control and Planning? 35 | 36 | **Generative Models in Control and Planning** 37 | 38 | 39 | Before diffusion models became popular, there were other generative models used in control and planning. Below are a few examples of how other generative models have been applied in imitation learning: 40 | 41 | ![Other generative model overview](figs/generative_model.png) 42 | 43 | | Generative model | Application | Idea | Limitation | 44 | | --- | --- | --- | --- | 45 | | GAN | Generative Adversarial Imitation Learning (GAIL): Learning a discriminator and training a policy to fool the discriminator | ![GAIL algorithm](figs/GAIL_algo.png) | Difficult to handle multimodal distributions and unstable training | 46 | | VAE | Action Chunking with Transformers (ACT) ([ALOHA](https://tonyzhaozh.github.io/aloha/)): Learning a latent space (encoder: expert action sequence + observation -> latent) and using the latent space for planning (decoder: latent + more observation -> action sequence prediction) | ![ACT algorithm](figs/ACT.png) | Challenging to train | 47 | 48 | In these scenarios, the objective is to match the distribution of the dataset, similar to the goal of diffusion models. Compared with other generative models like GANs and VAEs, diffusion models are better at handling multimodal data and offer more stable training processes. 49 | 50 | **What to Learn with the Generative Model?** 51 | 52 | From a planning and reinforcement learning perspective, there are numerous scenarios where matching the dataset's distribution is crucial, such as: 53 | 54 | | Scenario | Challenge | Solution | Illustrations | 55 | | --- | --- | --- | --- | 56 | | Imitation Learning | $\min \|\|p_\theta(\tau) - p_{\text{data}}(\tau)\|\|$ **Match the demonstrations**' trajectory distribution (high-dimensional+multi-modality) with limited data. Common methods like GAIL use adversarial training to match the distribution. | Diffusion models excel at matching the distribution of the expert's actions with high capacity and expressiveness. | ![imitation learning limitation](figs/IL_limit.png)| 57 | | Offline Reinforcement Learning | $\max J_\theta(\tau) \ge J_\text{data}(\tau) \text{s.t.} \|\|p_\theta(\tau) - p_{\text{data}}(\tau)\|\| < \epsilon$ **Perform better than demonstrations** with a large number of demonstrations. Here, it's essential to ensure the policy's action distribution is close to the dataset while improving performance. Common methods like CQL penalize out-of-distribution (OOD) samples, making the method overly conservative. | Diffusion models can match the dataset's action and regularize the policy's action distribution effectively. | ![offline RL limitation](figs/offline_limitation.png)| 58 | | Model-based Reinforcement Learning | **Match the dynamic model** and (sometimes) the policy's action distribution. This involves first learning the model and then using it to plan in an autoregressive manner. This method suffers from compounding errors. | Diffusion models are adept at handling non-autoregressive and multimodal distribution matching by predicting the entire trajectory sequence at once. | ![model-based RL limitation](figs/planning_error.jpeg)| 59 | 60 | Diffusion models can be utilized to learn the policy, the planner, or the model itself. Each of these applications can be viewed as a distribution matching problem. The next section will delve into the choices and implications of using diffusion models in these contexts. 61 | 62 | 63 | ## 🛠️ Practice: How to Use the Diffuser? 64 | 65 | ### What to Diffuse? 66 | 67 | In practice, there are several ways to incorporate the diffusion model into control and planning. The most common method is known as the 'diffuser': 68 | 69 | The diffuser operates by concatenating the state and action, allowing us to diffuse the state-action sequence. This process is akin to diffusing a single-channel image. The training of the model follows a similar approach to image generation. Initially, noise is added to the state-action sequence. Subsequently, the model is trained to predict the score function or noise vector. 70 | 71 | Here a local field method is implemented with a temporal convolutional network (TCN) to impose local consistency on the state-action sequence. 72 | 73 | | Task | Thing's to Diffuse | How to Diffuse | 74 | |------|--------------------|----------------| 75 | | Image Generation | ![Noise Image](figs/denoise_image.gif) | ![Image Diffuse](figs/image_diffuse.jpg) | 76 | | Planning | ![Diffusion Model Diffuse Future State-Action Sequence](figs/denoise_traj.gif) | ![Diffuse Process for Next Trajectory](figs/traj_diffuse.png) | 77 | 78 | 79 | ### How to Impose Constraints/Objectives? 80 | 81 | However, when training a model with given trajectory data, this model can only replicate the same actions as in the data, which is not what we want. We want the model to generalize to new tasks and constraints. To achieve this, we need to make the model conditional on the task and constraints. There are a few ways to do this: 82 | 83 | **Guidance Function** 84 | 85 | The guidance function directly shifts the distribution/cost or learned value, etc., or trains a discriminator (classifier) to obtain the guidance function. There are two common ways to get the guidance function: 86 | 87 | 1. Predefined the guidance function: This approach is easy to implement but might lead to out-of-distribution (OOD) samples, breaking the learned distribution. 88 | 89 | $$ 90 | \tilde{p}_\theta(\boldsymbol{\tau}) \propto p_\theta(\boldsymbol{\tau}) h(\boldsymbol{\tau}) \\ 91 | \boldsymbol{\tau}^{i-1} = \mathcal{N}\left(\mu+\alpha \Sigma \nabla \mathcal{J}(\mu), \Sigma^i\right) 92 | $$ 93 | 94 | ![defined guidance function](figs/defined_guidance.png) 95 | 96 | 1. Learned guidance function: This approach is more common in training image generation models, which involves training a classifier to obtain the guidance function. However, this method requires training a classifier in an adversarial manner, which might lead to unstable training. 97 | 98 | $$ 99 | \begin{aligned} 100 | \nabla \log p\left(\boldsymbol{x}_t \mid y\right) & =\nabla \log \left(\frac{p\left(\boldsymbol{x}_t\right) p\left(y \mid \boldsymbol{x}_t\right)}{p(y)}\right) \\ 101 | & =\nabla \log p\left(\boldsymbol{x}_t\right)+\nabla \log p\left(y \mid \boldsymbol{x}_t\right)-\nabla \log p(y) \\ 102 | & =\underbrace{\nabla \log p\left(\boldsymbol{x}_t\right)}_{\text {unconditional score }}+\underbrace{\nabla \log p\left(y \mid \boldsymbol{x}_t\right)}_{\text {adversarial gradient }} 103 | \end{aligned} 104 | $$ 105 | 106 | **Classifier-Free Method** 107 | 108 | By performing the following transformation, we can obtain the guidance function without training a classifier. The two terms below are known as the unconditional score and the conditional score, respectively. In practice, we can omit the input of the conditional score to achieve the unconditional score. 109 | 110 | $$ 111 | \begin{aligned} 112 | \nabla \log p\left(\boldsymbol{x}_t \mid y\right) & =\nabla \log p\left(\boldsymbol{x}_t\right)+\gamma\left(\nabla \log p\left(\boldsymbol{x}_t \mid y\right)-\nabla \log p\left(\boldsymbol{x}_t\right)\right) \\ 113 | & =\nabla \log p\left(\boldsymbol{x}_t\right)+\gamma \nabla \log p\left(\boldsymbol{x}_t \mid y\right)-\gamma \nabla \log p\left(\boldsymbol{x}_t\right) \\ 114 | & =\underbrace{\gamma \nabla \log p\left(\boldsymbol{x}_t \mid y\right)}_{\text {conditional score }}+\underbrace{(1-\gamma) \nabla \log p\left(\boldsymbol{x}_t\right)}_{\text {unconditional score }} 115 | \end{aligned} 116 | $$ 117 | 118 | | Guidance Function Method | Classifier-Free Method | 119 | | --- | --- | 120 | | ![Guidance Function](figs/guidance_algo.png) | ![Classifier-Free Guidance](figs/classifierfree_algo.png) | 121 | 122 | **Inpainting** 123 | 124 | If the control problem involves specific state constraints (such as the initial/target state or constraints), we can simply fix the state and fill in the missing parts of the distribution. This approach is extremely useful in goal-reaching and navigation tasks. 125 | 126 | ![inpainting](figs/inpaint.gif) 127 | 128 | ![different ways to impose constraints/objectives](figs/guidance_function.png) 129 | 130 | ## 📚 Literatures: Research Progress in Diffusion Models for Control and Planning 131 | 132 | A detailed summary of each method can be found [here](https://panchaoyi.notion.site/14ed102954ce4da79f146a641925afd7?v=a447d0b5d5a949dfbb18e253c3492a8f&pvs=4). 133 | 134 | $$ 135 | \color{red}\underbrace{\nabla_x \log P}_{\text{how to get score function}} 136 | \color{black}( 137 | \color{blue}\underbrace{x}_{\text{what to diffuse}} 138 | \color{black}| 139 | \color{green}\underbrace{y}_{\text{how to impose constraints/objectives}} 140 | \color{black}) 141 | $$ 142 | 143 | The heart of the diffusion model is understanding how to obtain the score function. Based on the methods for obtaining the score function, what to diffuse, and how to impose constraints/objectives, we can categorize the recent research in diffusion models for control and planning into the following three axes: 144 | 145 | 146 | **Axis 1: How to Get the Score Function** 147 | 148 | * Data-driven: Learning from data by manually adding noise 149 | * Hybrid: Learning from another optimization process 150 | * Model-based: Calculating analytically from the model 151 | 152 | ![Different ways to get the score function](figs/score_function.png) 153 | 154 | 155 | The data-driven method is the most common approach to obtaining the score function, which involves adding noise to the data and then training the model to predict the score function. The hybrid method learns the score function from the intermediate results of another optimization process, typically used in specific optimization problems. Finally, if you can calculate the score function analytically, then you can use Langevin dynamics to estimate the final distribution. 156 | 157 | **Axis 2: What to Diffuse** 158 | 159 | * Action: Directly diffuse for the next action 160 | * State: Learn the model 161 | * State-sequence: Diffuse for the next state sequence, or sometimes state-action sequence, or action sequence (for position control) 162 | 163 | ![Different things to diffuse](figs/diffuse_what.png) 164 | 165 | What to diffuse depends on the goals. For instance, to regularize the action distribution to the dataset, diffusing the action is suitable. To learn a dynamically feasible optimal trajectory, diffusing the state-sequence is appropriate. Sometimes, diffusing the action sequence makes execution easier, while other times inverse dynamics are needed to obtain the action sequence. 166 | 167 | **Axis 3: How to Impose Constraints/Objectives** 168 | 169 | * Guidance function: Predefined or learned 170 | * Classifier-free: Use the unconditional score and conditional score 171 | * Inpainting: Fix the state and fill in the missing parts of the distribution 172 | 173 | ![Different ways to impose constraints/objectives](figs/guidance.png) 174 | 175 | The guidance function is the easiest way to impose constraints/objectives, which involves multiplying the guidance function with the model's distribution. The classifier-free method uses the unconditional and conditional scores to impose constraints/objectives, as seen in methods like `decision diffuser`, `adaptive diffuser`, etc. Inpainting involves fixing the state and filling in the missing parts of the distribution, useful in goal-reaching or navigation tasks. This approach is complementary to the guidance function and classifier-free method. 176 | 177 | One interesting work in this line called `safe diffuser` is to solve the quadratic programming (QP) problem at each step to add hard constraints. This method is useful in safety-critical tasks, where they prove that the diffusion model can satisfy the constraints if it converges. 178 | 179 | ![safe diffuser solving QP each step to add hard constraints](figs/safe_diffuser.png) 180 | 181 | 182 | ## 📝 Summary & Limitations: What are the Challenges? 183 | 184 | From the above discussion, we can see that the diffusion model can be used in control and planning to match the distribution of the dataset, which is widely used in imitation learning, offline reinforcement learning, and model-based reinforcement learning. The diffusion model can be used to learn the policy, the planner, or the model, which can also be viewed as a distribution matching problem. 185 | 186 | **Why Does Diffusion Work?** 187 | 188 | Compared with learning the explicit policy directly or learning the energy-based model, the diffusion model can handle multimodal distribution and higher-dimensional distribution matching by iteratively predicting the score function. This greatly smooths the distribution matching process and makes the training more stable and scalable. 189 | 190 | ![Compare distribution with other models](figs/policy_vs_diffusion.png) 191 | 192 | **Limitations** 193 | 194 | However, the diffusion model also has its limitations, which include: 195 | 196 | 1. Computational cost: The diffusion model requires a longer time to train (a few GPU days compared with tens of minutes) and inference (iterative sample steps compared with one forward pass). This makes high-frequency control and planning difficult to use with the diffusion model. 197 | 2. Handling shifting distribution: In online RL, the distribution of the policy will keep changing. Adapting the diffusion model to the new distribution requires a large amount of data and a long time to train. This limits the diffusion model to be trained in a fixed rather than a dynamic dataset. 198 | 3. High variance: Depending on the initial guess and random sampling, the variance of the diffusion model is high, which limits its application in high-precision or safety-critical tasks. 199 | 4. Constraint satisfaction: The diffusion model does not guarantee to satisfy the constraints, especially when tested in a constraint different from the training set. This limits its application in adapting to new constraints and tasks. 200 | 201 | **Resources** 202 | 203 | For more information, please refer to the following resources: 204 | 205 | [Paper list with labels](https://panchaoyi.notion.site/14ed102954ce4da79f146a641925afd7?v=a447d0b5d5a949dfbb18e253c3492a8f&pvs=4): A detailed summary of recent papers organized by me. 206 | 207 | [Diffusion for RL survey paper](https://arxiv.org/abs/2311.01223): A comprehensive survey paper on diffusion models in RL. 208 | 209 | [Diffusion for RL repo](https://github.com/apexrl/Diff4RLSurvey): A comprehensive repo on diffusion models in RL. 210 | 211 | [Awesome Diffusion RL repo](https://github.com/opendilab/awesome-diffusion-model-in-rl): Another comprehensive repo on diffusion models in RL. 212 | -------------------------------------------------------------------------------- /diffusion_planning_slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/diffusion_planning_slides.pdf -------------------------------------------------------------------------------- /diffusion_planning_tutorial.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/diffusion_planning_tutorial.pdf -------------------------------------------------------------------------------- /figs/ACT.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/ACT.png -------------------------------------------------------------------------------- /figs/GAIL_algo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/GAIL_algo.png -------------------------------------------------------------------------------- /figs/GAIL_paper.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/GAIL_paper.png -------------------------------------------------------------------------------- /figs/IL_limit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/IL_limit.png -------------------------------------------------------------------------------- /figs/classifierfree_algo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/classifierfree_algo.png -------------------------------------------------------------------------------- /figs/dalle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/dalle.png -------------------------------------------------------------------------------- /figs/defined_guidance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/defined_guidance.png -------------------------------------------------------------------------------- /figs/denoise_image.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/denoise_image.gif -------------------------------------------------------------------------------- /figs/denoise_traj.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/denoise_traj.gif -------------------------------------------------------------------------------- /figs/diffuse_teaser.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/diffuse_teaser.gif -------------------------------------------------------------------------------- /figs/diffuse_what.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/diffuse_what.png -------------------------------------------------------------------------------- /figs/distribution_match.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/distribution_match.gif -------------------------------------------------------------------------------- /figs/generative_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/generative_model.png -------------------------------------------------------------------------------- /figs/guidance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/guidance.png -------------------------------------------------------------------------------- /figs/guidance_algo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/guidance_algo.png -------------------------------------------------------------------------------- /figs/guidance_function.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/guidance_function.png -------------------------------------------------------------------------------- /figs/image_diffuse.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/image_diffuse.jpg -------------------------------------------------------------------------------- /figs/inpaint.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/inpaint.gif -------------------------------------------------------------------------------- /figs/noisy_image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/noisy_image.png -------------------------------------------------------------------------------- /figs/offline_limitation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/offline_limitation.png -------------------------------------------------------------------------------- /figs/planner_policy_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/planner_policy_model.png -------------------------------------------------------------------------------- /figs/planning_error.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/planning_error.jpeg -------------------------------------------------------------------------------- /figs/policy_vs_diffusion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/policy_vs_diffusion.png -------------------------------------------------------------------------------- /figs/safe_diffuser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/safe_diffuser.png -------------------------------------------------------------------------------- /figs/score_function.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/score_function.png -------------------------------------------------------------------------------- /figs/traj.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/traj.png -------------------------------------------------------------------------------- /figs/traj_diffuse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jc-bao/diffuser-control-tutorial/a61c2c5246c7a0d36b74d680e7f8b8688ccd78e2/figs/traj_diffuse.png -------------------------------------------------------------------------------- /slides.md: -------------------------------------------------------------------------------- 1 | --- 2 | theme: seriph 3 | highlighter: shiki 4 | class: 'text-center' 5 | title: Diffusion Model for Control and Planning Tutorial 6 | background: 'figs/diffuse_teaser.gif' 7 | layout: cover 8 | --- 9 | 10 | # 📚 Tutorial: Diffusion Model for Control and Planning 11 | 12 | 13 | --- 14 | transition: fade 15 | class: text-left 16 | --- 17 | 18 | # 📖 Tutorial Outline 19 | 20 | 21 | 22 | 1. 🔄 Recap: What is a Diffusion Model? 23 | 2. 🚀 Motivation: Why a Generative Model in Control and Planning? 24 | 3. 🛠️ Practice: How to Use the Diffuser? 25 | 4. 📚 Literatures: Recent Research Progress in Diffusion for RL/Control 26 | 5. 📝 Summary & Challenges in Diffusion Models 27 | 28 | 29 | 30 | --- 31 | 32 | # 🔄 Recap: What is a Diffusion Model? 33 | 34 | - Keynote: Generative model for distribution matching. 35 | - Applications: Image and text generation, creative tasks. 36 | ![diffusion examples](figs/dalle.png) 37 | 38 | --- 39 | 40 | # 🔄 Recap: What is a Diffusion Model? 41 | 42 | - Keynote: Generative model for distribution matching. 43 | - Applications: Image and text generation, creative tasks. 44 | - Core: Score function for sample generation and distribution description. 45 | $$ 46 | \boldsymbol{x}_{i+1} \leftarrow \boldsymbol{x}_i+c \nabla \log p\left(\boldsymbol{x}_i\right)+\sqrt{2 c} \boldsymbol{\epsilon}, \quad i=0,1, \ldots, K 47 | $$ 48 | ![Annealed Langevin dynamics combine a sequence of Langevin chains with gradually decreasing noise scales.](figs/distribution_match.gif) 49 | 50 | --- 51 | 52 | # 🔄 Recap: What is a Diffusion Model? 53 | 54 | - Keynote: Generative model for distribution matching. 55 | - Applications: Image and text generation, creative tasks. 56 | - Core: Score function for sample generation and distribution description. 57 | - Advantages: 58 | 59 | 60 | - 🌟 Multimodal: Effective with multimodal distributions. 61 | - 📈 Scalable: Suits high-dimensional problems. 62 | - 🔒 Stable: Grounded in solid mathematics and training. 63 | - 🔄 Non-autoregressive: Predicts entire trajectories efficiently. 64 | 65 | 66 | 67 | --- 68 | 69 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 70 | 71 | ## Generative Models in Control and Planning 72 | 73 | - Generative Models: application in imitation learning to match expert data. 74 | - Examples: GANs, VAEs in imitation learning. 75 | ![Other generative model overview](figs/generative_model.png) 76 | 77 | --- 78 | 79 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 80 | 81 | ## Generative Models in Control and Planning 82 | 83 | - Generative Models: application in imitation learning to match expert data. 84 | - Examples: GANs, VAEs in imitation learning. 85 | - GAN in GAIL: Discriminator learning and policy training. 86 | - Idea: Train a discriminator to distinguish between expert and agent data. 87 | - Limitation: Struggles with multimodal distributions, unstable training. 88 | 89 | GAIL algorithm 90 | 91 | --- 92 | 93 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 94 | 95 | ## Generative Models in Control and Planning 96 | 97 | - Generative Models: Crucial in control and planning. 98 | - Examples: GANs, VAEs in imitation learning. 99 | - VAE in ACT ([ALOHA](https://tonyzhaozh.github.io/aloha/)): Latent space learning for planning. 100 | - Idea: learn a latent space for planning and control. (generate action in chunks) 101 | - Limitation: hard to train. 102 | 103 | ACT algorithm 104 | 105 | --- 106 | layout: iframe 107 | 108 | url: https://tonyzhaozh.github.io/aloha/ 109 | --- 110 | 111 | --- 112 | 113 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 114 | 115 | ## What to Learn with the Generative Model? 116 | 117 | Scenario: Imitation Learning $\min \|p_\theta(\tau) - p_{\text{data}}(\tau)\|$ 118 | 119 | - Challenge: Match high-dimensional, multimodal trajectory distributions. 120 | - Solution: Diffusion models for expressive distribution matching. 121 | - Common Method: GAIL with adversarial training. 122 | - Limitation: Struggles with multimodal distributions, unstable training. 123 | 124 | Imitation Learning Limitation 125 | 126 | --- 127 | 128 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 129 | 130 | ## What to Learn with the Generative Model? 131 | 132 | Scenario: Offline Reinforcement Learning $\max J_\theta(\tau) \ge J_\text{data}(\tau) \text{s.t.} \|p_\theta(\tau) - p_{\text{data}}(\tau)\| < \epsilon$ 133 | 134 | - Challenge: Outperform demonstrations, ensure close action distribution. 135 | - Solution: Diffusion models to match action distribution effectively. 136 | - Common Method: CQL, penalizes out-of-distribution samples. 137 | - Limitation: over-conservative. 138 | 139 | Offline RL Limitation 140 | 141 | --- 142 | 143 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 144 | 145 | ## What to Learn with the Generative Model? 146 | 147 | Scenario: Model-based Reinforcement Learning 148 | 149 | - Challenge: Match dynamic model and policy's action distribution. 150 | - Solution: Diffusion models for non-autoregressive, multimodal matching. 151 | - Common method: planning with learned dynamics. 152 | - Limitation: compounding error in long-horizon planning. 153 | 154 | Model-based RL Limitation 155 | 156 | --- 157 | 158 | # 🚀 Motivation: Why a Generative Model in Control and Planning? 159 | 160 | ## What to Learn with the Generative Model? 161 | 162 | Key: using a powerful model to matching a high-dimensional, multimodal distribution. 163 | 164 | - Action/Value distribution matching: grounded in demostrations -> offline RL. 165 | - Trajectory distribution matching: dynamic feasibility and optimal trajectory distribution -> model-based RL. 166 | - Transition distribution matching: dynamics matching in a non-autoregressive manner -> model-based RL. 167 | 168 | --- 169 | 170 | # 🛠️ Practice: How to Use the Diffuser? 171 | 172 | ## What to Diffuse? 173 | 174 | - Most common: diffuse trajectory (`diffuser`). 175 | - Diffused variable `x`: state, action sequence. $\tau = \{s_0, a_0, s_1, a_1, \ldots, s_T, a_T\}$. 176 | 177 | | Task | Thing's to Diffuse | How to Diffuse | 178 | |------|--------------------|----------------| 179 | | Image Generation | Noise Image | Image Diffuse | 180 | | Planning | Diffusion Model Diffuse Future State-Action Sequence | Diffuse Process for Next Trajectory | 181 | 182 | --- 183 | layout: iframe 184 | 185 | url: https://diffusion-planning.github.io 186 | --- 187 | 188 | --- 189 | 190 | # 🛠️ Practice: How to Use the Diffuser? 191 | 192 | ### How to Impose Constraints/Objectives? 193 | 194 | - Objective: make the trained model can generalize to new constraints and tasks. 195 | - Common case: goal-conditioned, safety, new task etc. 196 | - Possible Methods: 197 | 198 | 199 | - Guidance function (d): shift distribution with extra gradient. 200 | - Classifier-free method: learn a model can both represent conditional and unconditional distribution. 201 | - Inpainting (a): fill in the missing part of the trajectory by fixing certain start and end state. 202 | 203 | 204 | 205 | Guidance Function 206 | 207 | --- 208 | 209 | # 🛠️ Practice: How to Use the Diffuser? 210 | 211 | ### How to Impose Constraints/Objectives? 212 | 213 | - Guidance function: shift distribution with extra gradient. 214 | - Predefined the guidance function: 215 | - Method: shift distribution with a manually defined function 216 | - Limitation: Might lead to OOD samples, which break the learned diffusion process. 217 | 218 | $$ 219 | \tilde{p}_\theta(\boldsymbol{\tau}) \propto p_\theta(\boldsymbol{\tau}) h(\boldsymbol{\tau}) \\ 220 | \boldsymbol{\tau}^{i-1} = \mathcal{N}\left(\mu+\alpha \Sigma \nabla \mathcal{J}(\mu), \Sigma^i\right) 221 | $$ 222 | 223 | Guidance Function 224 | 225 | --- 226 | 227 | # 🛠️ Practice: How to Use the Diffuser? 228 | 229 | ### How to Impose Constraints/Objectives? 230 | 231 | - Guidance function: shift distribution with extra gradient. ▶️ leads to OOD samples 232 | - Predefined the guidance function: 233 | - Method: shift distribution with a manually defined function 234 | - Limitation: Might lead to OOD samples, which break the learned diffusion process. 235 | - Learned classifier: 236 | - Method: learning a classifier to distinguish between different constraints. (similar to GAN) 237 | - Limitation: Hard to tune parameters. 238 | 239 | 240 | $$ 241 | \begin{aligned} 242 | \nabla \log p\left(\boldsymbol{x}_t \mid y\right) & =\nabla \log \left(\frac{p\left(\boldsymbol{x}_t\right) p\left(y \mid \boldsymbol{x}_t\right)}{p(y)}\right) \\ 243 | & =\nabla \log p\left(\boldsymbol{x}_t\right)+\nabla \log p\left(y \mid \boldsymbol{x}_t\right)-\nabla \log p(y) \\ 244 | & =\underbrace{\nabla \log p\left(\boldsymbol{x}_t\right)}_{\text {unconditional score }}+\underbrace{\nabla \log p\left(y \mid \boldsymbol{x}_t\right)}_{\text {adversarial gradient }} 245 | \end{aligned} 246 | $$ 247 | 248 | --- 249 | 250 | # 🛠️ Practice: How to Use the Diffuser? 251 | 252 | ### How to Impose Constraints/Objectives? 253 | 254 | - Guidance function: shift distribution with extra gradient. ▶️ leads to OOD samples 255 | - Classifier-Free Method: learn a model can both represent conditional and unconditional distribution. 256 | - Method: drop out the condition term to learn a model can represent both conditional and unconditional distribution. 257 | 258 | $$ 259 | \begin{aligned} 260 | \nabla \log p\left(\boldsymbol{x}_t \mid y\right) & =\nabla \log p\left(\boldsymbol{x}_t\right)+\gamma\left(\nabla \log p\left(\boldsymbol{x}_t \mid y\right)-\nabla \log p\left(\boldsymbol{x}_t\right)\right) \\ 261 | & =\nabla \log p\left(\boldsymbol{x}_t\right)+\gamma \nabla \log p\left(\boldsymbol{x}_t \mid y\right)-\gamma \nabla \log p\left(\boldsymbol{x}_t\right) \\ 262 | & =\underbrace{\gamma \nabla \log p\left(\boldsymbol{x}_t \mid y\right)}_{\text {conditional score }}+\underbrace{(1-\gamma) \nabla \log p\left(\boldsymbol{x}_t\right)}_{\text {unconditional score }} 263 | \end{aligned} 264 | $$ 265 | 266 | --- 267 | 268 | # 🛠️ Practice: How to Use the Diffuser? 269 | 270 | ### How to Impose Constraints/Objectives? 271 | 272 | - Guidance function: shift distribution with extra gradient. ▶️ leads to OOD samples 273 | - Classifier-Free Method: learn a model can both represent conditional and unconditional distribution. 274 | 275 | | Guidance Function Method | Classifier-Free Method | 276 | | --- | --- | 277 | | Guidance Function | Classifier-Free Guidance | 278 | 279 | --- 280 | 281 | # 🛠️ Practice: How to Use the Diffuser? 282 | 283 | ### How to Impose Constraints/Objectives? 284 | 285 | - Guidance function: shift distribution with extra gradient. ▶️ leads to OOD samples 286 | - Classifier-Free Method: learn a model can both represent conditional and unconditional distribution. 287 | - Inpainting: fill in the missing part of the trajectory by fixing certain start and end state. 288 | - Method: fix the start and end state, and fill in the missing part of the trajectory. 289 | 290 | Inpainting 291 | 292 | --- 293 | 294 | # 🛠️ Practice: How to Use the Diffuser? 295 | 296 | * Common thing to diffuse: trajectory. 297 | * Common way to impose constraints/add objectives: guidance function, classifier-free method, inpainting. 298 | 299 | --- 300 | 301 | # 📚 Literatures: Recent Research Progress in Diffusion for RL/Control 302 | 303 | A detailed summary of each method can be found [here](https://panchaoyi.notion.site/14ed102954ce4da79f146a641925afd7?v=a447d0b5d5a949dfbb18e253c3492a8f&pvs=4). 304 | 305 | The key of diffusion: how to get the score function. 306 | 307 | $$ 308 | \color{red}\underbrace{\nabla_x \log P}_{\text{how to get score function}} 309 | \color{black}( 310 | \color{blue}\underbrace{x}_{\text{what to diffuse}} 311 | \color{black}| 312 | \color{green}\underbrace{y}_{\text{how to impose constraints/objectives}} 313 | \color{black}) 314 | $$ 315 | 316 | 317 | 318 | - How to get score function: data-driven v.s. analytical. 319 | - What to diffuse: sequential v.s. non-sequential. 320 | - How to impose constraints/objectives: hard v.s. soft. 321 | 322 | 323 | 324 | --- 325 | 326 | # 📚 Literatures: Recent Research Progress in Diffusion for RL/Control 327 | 328 | $$ 329 | \color{red}\underbrace{\nabla_x \log P}_{\text{how to get score function}} 330 | \color{black}( 331 | \color{blue}\underbrace{x}_{\text{what to diffuse}} 332 | \color{black}| 333 | \color{green}\underbrace{y}_{\text{how to impose constraints/objectives}} 334 | \color{black}) 335 | $$ 336 | 337 | - How to get score function: data-driven v.s. analytical. 338 | - Data-driven: learn the score function from data. 339 | - Hybrid: learning from optimization intermediate results. 340 | - Analytical: use the analytical score function. 341 | 342 | Different ways to get the score function 343 | 344 | --- 345 | 346 | # 📚 Literatures: Recent Research Progress in Diffusion for RL/Control 347 | 348 | $$ 349 | \color{red}\underbrace{\nabla_x \log P}_{\text{how to get score function}} 350 | \color{black}( 351 | \color{blue}\underbrace{x}_{\text{what to diffuse}} 352 | \color{black}| 353 | \color{green}\underbrace{y}_{\text{how to impose constraints/objectives}} 354 | \color{black}) 355 | $$ 356 | 357 | - How to get score function: data-driven v.s. analytical. 358 | - What to diffuse: sequential v.s. non-sequential. 359 | - Action/Value: learn a model to match action/value distribution, serve as regularizer and policy. 360 | - Transition: learn a model to match transition distribution, serve as a world mode. ▶️ MPC 361 | - Trajectory: learn a model to match trajectory distribution, serve as a TO solver. (planning state v.s. state-action v.s. action) 362 | 363 | Different things to diffuse 364 | 365 | --- 366 | 367 | # 📚 Literatures: Recent Research Progress in Diffusion for RL/Control 368 | 369 | $$ 370 | \color{red}\underbrace{\nabla_x \log P}_{\text{how to get score function}} 371 | \color{black}( 372 | \color{blue}\underbrace{x}_{\text{what to diffuse}} 373 | \color{black}| 374 | \color{green}\underbrace{y}_{\text{how to impose constraints/objectives}} 375 | \color{black}) 376 | $$ 377 | 378 | - How to get score function: data-driven v.s. analytical. 379 | - What to diffuse: sequential v.s. non-sequential. 380 | - How to impose constraints/objectives: hard v.s. soft. 381 | - Guidance function: Predefined or learned 382 | - Classifier-free: Use the unconditional score and conditional score (most common) 383 | - Inpainting: Fix the state and fill in the missing parts of the distribution (complimentary to the other two) 384 | 385 | --- 386 | 387 | # 📚 Literatures: Recent Research Progress in Diffusion for RL/Control 388 | 389 | $$ 390 | \color{red}\underbrace{\nabla_x \log P}_{\text{how to get score function}} 391 | \color{black}( 392 | \color{blue}\underbrace{x}_{\text{what to diffuse}} 393 | \color{black}| 394 | \color{green}\underbrace{y}_{\text{how to impose constraints/objectives}} 395 | \color{black}) 396 | $$ 397 | 398 | - How to get score function: data-driven v.s. analytical. 399 | - What to diffuse: sequential v.s. non-sequential. 400 | - How to impose constraints/objectives: hard v.s. soft. 401 | 402 | safe diffuser solving QP each step to add hard constraints 403 | 404 | --- 405 | 406 | # 📝 Summary & Challenges in Diffusion Models 407 | 408 | 409 | 410 | - Diffusion in robotics: matches demostration distribution from data. 411 | - Use cases: imitation learning, offline RL, model-based RL. 412 | - Role: Learns policy, trajectory, or model as a regularizer/world model/planner. 413 | 414 | 415 | 416 | --- 417 | 418 | # 📝 Summary & Challenges in Diffusion Models 419 | 420 | - Diffusion in robotics: matches dataset distribution in control and planning. 421 | - Use cases: imitation learning, offline RL, model-based RL. 422 | - Role: Learns policy, planner, or model as a distribution matching problem. 423 | - Advantages: high-dimensional matching, stability, scalability. 424 | ![Compare distribution with other models](figs/policy_vs_diffusion.png) 425 | 426 | --- 427 | 428 | # 📝 Summary & Challenges in Diffusion Models 429 | 430 | - Diffusion in robotics: matches dataset distribution in control and planning. 431 | - Use cases: imitation learning, offline RL, model-based RL. 432 | - Role: Learns policy, planner, or model as a distribution matching problem. 433 | - Challenges: 434 | 435 | 436 | - 🕒 Computational cost: longer training and inference time. 437 | - 🔀 Shifting distribution: difficulties in adapting to dynamic datasets. 438 | - 📊 High variance: inconsistent performance in precision tasks. 439 | - ⛔ Constraint satisfaction: limited adaptability to new constraints. 440 | 441 | 442 | 443 | --- 444 | class: text-center 445 | --- 446 | 447 | # 🎉 Thank you! 448 | 449 | ## Useful Resources 450 | 451 | [Paper list with labels](https://panchaoyi.notion.site/14ed102954ce4da79f146a641925afd7?v=a447d0b5d5a949dfbb18e253c3492a8f&pvs=4) 452 | 453 | [Diffusion for RL survey paper](https://arxiv.org/abs/2311.01223) 454 | 455 | [Diffusion for RL repo](https://github.com/apexrl/Diff4RLSurvey) 456 | 457 | [Awesome Diffusion RL repo](https://github.com/opendilab/awesome-diffusion-model-in-rl) --------------------------------------------------------------------------------