├── LICENSE ├── README.md ├── commonsense_reasoning └── requirements.txt ├── dora.png ├── dora_lora_lego.jpeg └── dora_lora_yoda_emoji.jpg /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2024, NVIDIA Corporation. All rights reserved. 2 | 3 | Nvidia Source Code License-NC 4 | 5 | 1. Definitions 6 | 7 | “Licensor” means any person or entity that distributes its Work. 8 | 9 | “Work” means (a) the original work of authorship made available under this license, which may include software, documentation, 10 | or other files, and (b) any additions to or derivative works thereof that are made available under this license. 11 | 12 | The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. 13 | copyright law; provided, however, that for the purposes of this license, derivative works shall not include works that 14 | remain separable from, or merely link (or bind by name) to the interfaces of, the Work. 15 | 16 | Works are “made available” under this license by including in or with the Work either (a) a copyright notice referencing 17 | the applicability of this license to the Work, or (b) a copy of this license. 18 | 19 | 2. License Grant 20 | 21 | 2.1 Copyright Grant. Subject to the terms and conditions of this license, each Licensor grants to you a perpetual, 22 | worldwide, non-exclusive, royalty-free, copyright license to use, reproduce, prepare derivative works of, publicly 23 | display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form. 24 | 25 | 3. Limitations 26 | 27 | 3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this license, (b) you include a 28 | complete copy of this license with your distribution, and (c) you retain without modification any copyright, patent, 29 | trademark, or attribution notices that are present in the Work. 30 | 31 | 3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution 32 | of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 33 | applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. 34 | Notwithstanding Your Terms, this license (including the redistribution requirements in Section 3.1) will continue to apply 35 | to the Work itself. 36 | 37 | 3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use non-commercially. 38 | Notwithstanding the foregoing, NVIDIA Corporation and its affiliates may use the Work and any derivative works commercially. 39 | As used herein, “non-commercially” means for research or evaluation purposes only. 40 | 41 | 3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim 42 | or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under 43 | this license from such Licensor (including the grant in Section 2.1) will terminate immediately. 44 | 45 | 3.5 Trademarks. This license does not grant any rights to use any Licensor’s or its affiliates’ names, logos, or trademarks, 46 | except as necessary to reproduce the notices described in this license. 47 | 48 | 3.6 Termination. If you violate any term of this license, then your rights under this license (including the grant in Section 2.1) 49 | will terminate immediately. 50 | 51 | 4. Disclaimer of Warranty. 52 | 53 | THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES 54 | OR CONDITIONS OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING 55 | ANY ACTIVITIES UNDER THIS LICENSE. 56 | 57 | 5. Limitation of Liability. 58 | 59 | EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, 60 | OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL 61 | DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, 62 | BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER DAMAGES OR LOSSES), EVEN IF THE LICENSOR 63 | HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 64 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 |

DoRA: Weight-Decomposed Low-Rank Adaptation

3 |

4 | 5 |

6 | 7 |

8 | 9 | # This repo is now deprecated, please visit [NVlabs/DoRA](https://github.com/NVlabs/DoRA) instead!! 10 | 11 | ## DoRA: Weight-Decomposed Low-Rank Adaptation 12 | 13 | Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen 14 | 15 | Paper: https://arxiv.org/abs/2402.09353 16 | 17 | Project page: https://nbasyl.github.io/DoRA-project-page/ 18 | 19 | DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing DoRA, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. 20 | 21 | ## Quick Start and some tricks regarding finetuing with DoRA 22 | ### HuggingFace PEFT 23 | DoRA is now supported by the Huggingface PEFT package. You can install the PEFT package using 24 | ```bash 25 | pip install git+https://github.com/huggingface/peft.git -q 26 | ``` 27 | 28 | After PEFT is installed, you can simply set the `use_dora` argument of `LoraConfig()` to `True` for applying DoRA. 29 | 30 | An example could be as follows: 31 | ```bash 32 | from peft import LoraConfig 33 | 34 | # Initialize DoRA configuration 35 | config = ( 36 | use_dora=True, ... 37 | ) 38 | ``` 39 | Please refer to the official [documentation](https://huggingface.co/docs/peft/en/developer_guides/lora#weight-decomposed-low-rank-adaptation-dora) for more details. 40 | 41 | ### DoRA hyperparameters settings 42 | > [!NOTE] 43 | > 💡 While fine-tuning with DoRA, by utilizing the configuration of LoRA can already achieve better results most of the time, achieving optimal performance compared to LoRA still requires adjustments to the hyperparameters. 44 | 45 | > We suggest starting with a slightly lower learning rate than that of LoRA, and users may also experiment with varying lora dropout ratios. 46 | 47 | > User may also start with half of the rank of the LoRA configuration which oftentime can already results in comparable or even superior accuracy compared to that of LoRA. 48 | 49 | ## Contact 50 | Shih-Yang Liu: [shihyangl@nvidia.com](shihyangl@nvidia.com) or [sliuau@connect.ust.hk](sliuau@connect.ust.hk) 51 | 52 | ## Licenses 53 | Copyright © 2024, NVIDIA Corporation. All rights reserved. 54 | 55 | This work is made available under the NVIDIA Source Code License-NC. Click [here](https://github.com/nbasyl/DoRA/blob/main/LICENSE) to view a copy of this license. 56 | 57 | ## Citation 58 | If you find DoRA useful, please cite it by using the following BibTeX entry. 59 | ```bibtex 60 | @article{liu2024dora, 61 | title={{DoRA}: Weight-Decomposed Low-Rank Adaptation}, 62 | author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung}, 63 | booktitle={arXiv:2402.09353}, 64 | url={arxiv.org/abs/2402.09353}, 65 | year={2024} 66 | } 67 | ``` 68 | -------------------------------------------------------------------------------- /commonsense_reasoning/requirements.txt: -------------------------------------------------------------------------------- 1 | accelerate==0.25.0 2 | appdirs==1.4.4 3 | bitsandbytes==0.41.3.post2 4 | black==23.12.0 5 | black[jupyter] 6 | datasets==2.15.0 7 | fire==0.5.0 8 | git+https://github.com/huggingface/transformers.git@v4.36.0 9 | gradio==4.9.0 10 | scipy==1.11.4 11 | sentencepiece==0.1.99 -------------------------------------------------------------------------------- /dora.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nbasyl/DoRA/13391c3ac376a89cf4efdabe64083d618ecb7018/dora.png -------------------------------------------------------------------------------- /dora_lora_lego.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nbasyl/DoRA/13391c3ac376a89cf4efdabe64083d618ecb7018/dora_lora_lego.jpeg -------------------------------------------------------------------------------- /dora_lora_yoda_emoji.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nbasyl/DoRA/13391c3ac376a89cf4efdabe64083d618ecb7018/dora_lora_yoda_emoji.jpg --------------------------------------------------------------------------------