├── 01-colossalai-sft-kaggle.ipynb
├── 02-colossalai-sft-colab.ipynb
├── README.md
├── 千“垂”百炼 - 【医疗&健康】 ChatDoctor（上）.pdf
├── 千“垂”百炼 - 【医疗&健康】 ChatDoctor（下）.pdf
├── 千“垂”百炼 - 【医疗&健康】 ChatDoctor（中）.pdf
├── 千“垂”百炼 - 【医疗&健康】 MING.pdf
├── 千“垂”百炼 - 【医疗&健康】 MedicalGPT-zh.pdf
└── 千“垂”百炼 - 【医疗&健康】 SoulChat.pdf
/01-colossalai-sft-kaggle.ipynb:
--------------------------------------------------------------------------------
1 | {"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**语言模型练习场01：**ColossalAI（SFT部分）\n- https://github.com/hpcaitech/ColossalAI\n\n**注意：**\n- 此notebook**只演示在kaggle notebook下如何跑通ColossalAI的SFT部分**，并不会包含超参数的调整、对结果的分析等\n- 类似的操作放到google colab理论上应该也可以跑通\n- **如果你有自己的机器**，则此notebook对你的帮助可能不大（因为你不需要在notebook上进行训练）\n- 此notebook的受众是**手里没有GPU资源，但是又想熟悉和浅浅尝试ColossalAI的小伙伴**","metadata":{}},{"cell_type":"markdown","source":"**数据的准备：**\n1. 根据[官方文档的提示](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)，在运行前需要准备好数据\n2. 数据可以在[这里下载](https://github.com/XueFuzhao/InstructionWild/tree/main/data)。注意不要下载seed文件（因为seed文件只有instruction，而没有response），要下载README里面提到的json文件，例如instinwild_ch.json\n3.将数据上传到Kaggle的Dataset中（需要创建自己的Dataset，可以为仅自己可见），按照Kaggle的步骤操作即可","metadata":{}},{"cell_type":"markdown","source":"**Kaggle Notebook的准备：**\n1. 在界面右方**添加Dataset**，选择自己创建的Dataset。选择后，在代码中可以通过绝对路径访问。比如刚才创建的数据集名字叫做“instructdata”，其中我们上传了文件“instinwild_ch_small.json”，则在代码中，我们就可以通过这个路径访问数据集：/kaggle/input/instructdata/instinwild_ch_small.json\n2. 最好选择**GPU T4x2**（Kaggle Notebook界面右方的Accelerator中选择）。如果选择P100可能会在安装过程中报错","metadata":{}},{"cell_type":"markdown","source":"**特别要注意的地方：**\n1. 每周有**30个小时**的GPU使用时间\n2. 每一次启动notebook**最长只能运行12个小时**（如果启动了但是不怎么使用，比如没有运行任何cell，也没有什么编辑的动作，可能也会在12个小时以内被强行终止。与google colab不同的时，长时间运行cell是可以的）\n3. 一旦被终止，则**不能再找回输出的数据**！（输出的数据会放到/kaggle/working/路径下，如果需要里面的数据，必须在终止运行之前就下载下来）","metadata":{}},{"cell_type":"markdown","source":"## 安装环境","metadata":{}},{"cell_type":"markdown","source":"大体上是按照官方文档说明安装。**但是**，如果严格按照文档安装会报错。原因是ColossalAI是一个非常活跃的项目，每日都会有不同的代码变化。文档的部分内容可能还不能及时得到更新。所以，我们针对目前的情况，对安装顺序和细节做了一点微调。 **（当前时间为：2023年4月30日）**\n\n相信在不远的将来，ColossalAI团队会把这些小bug修好，并且把文档立刻完善起来。","metadata":{}},{"cell_type":"markdown","source":"### 1 安装ColossalAI\n\n执行完下面的命令后，此时你会发现，下载的文件是放在了/kaggle/working/ColossalAI下","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/hpcaitech/ColossalAI.git","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:57:00.220637Z","iopub.execute_input":"2023-04-30T13:57:00.221204Z","iopub.status.idle":"2023-04-30T13:57:03.206251Z","shell.execute_reply.started":"2023-04-30T13:57:00.221169Z","shell.execute_reply":"2023-04-30T13:57:03.205056Z"},"trusted":true},"execution_count":1,"outputs":[{"name":"stdout","text":"Cloning into 'ColossalAI'...\nremote: Enumerating objects: 24949, done.\u001b[K\nremote: Counting objects: 100% (2362/2362), done.\u001b[K\nremote: Compressing objects: 100% (479/479), done.\u001b[K\nremote: Total 24949 (delta 1987), reused 2083 (delta 1881), pack-reused 22587\u001b[K\nReceiving objects: 100% (24949/24949), 23.09 MiB | 29.19 MiB/s, done.\nResolving deltas: 100% (16582/16582), done.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"**安装ColossalAI**\n\n如果不执行这个安装，可能会出现这个[错误](https://github.com/hpcaitech/ColossalAI/issues/3629)：“ImportError: cannot import name 'ColoInitContext' from 'colossalai.zero'”","metadata":{}},{"cell_type":"code","source":"import os\nos.chdir('./ColossalAI')\n!pip install .","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:57:03.208770Z","iopub.execute_input":"2023-04-30T13:57:03.209133Z","iopub.status.idle":"2023-04-30T13:57:28.240848Z","shell.execute_reply.started":"2023-04-30T13:57:03.209098Z","shell.execute_reply":"2023-04-30T13:57:28.239606Z"},"trusted":true},"execution_count":2,"outputs":[{"name":"stdout","text":"Processing /kaggle/working/ColossalAI\n Preparing metadata (setup.py) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (1.21.6)\nRequirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (4.64.1)\nRequirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (5.9.3)\nRequirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (23.0)\nCollecting pre-commit\n Downloading pre_commit-2.21.0-py2.py3-none-any.whl (201 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m201.9/201.9 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: rich in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (13.2.0)\nRequirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (8.1.3)\nCollecting fabric\n Downloading fabric-3.0.1-py3-none-any.whl (53 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.3/53.3 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hCollecting contexttimer\n Downloading contexttimer-0.3.3.tar.gz (4.9 kB)\n Preparing metadata (setup.py) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: ninja in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (1.11.1)\nRequirement already satisfied: torch>=1.11 in /opt/conda/lib/python3.7/site-packages (from colossalai==0.2.8) (1.13.0)\nCollecting safetensors\n Downloading safetensors-0.3.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m33.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n\u001b[?25hRequirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.11->colossalai==0.2.8) (4.4.0)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from click->colossalai==0.2.8) (4.11.4)\nCollecting paramiko>=2.4\n Downloading paramiko-3.1.0-py3-none-any.whl (211 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.2/211.2 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hCollecting invoke>=2.0\n Downloading invoke-2.1.0-py3-none-any.whl (159 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m159.9/159.9 kB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from pre-commit->colossalai==0.2.8) (6.0)\nCollecting cfgv>=2.0.0\n Downloading cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)\nCollecting identify>=1.0.0\n Downloading identify-2.5.23-py2.py3-none-any.whl (98 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.8/98.8 kB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: virtualenv>=20.10.0 in /opt/conda/lib/python3.7/site-packages (from pre-commit->colossalai==0.2.8) (20.17.1)\nCollecting nodeenv>=0.11.1\n Downloading nodeenv-1.7.0-py2.py3-none-any.whl (21 kB)\nRequirement already satisfied: markdown-it-py<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from rich->colossalai==0.2.8) (2.1.0)\nRequirement already satisfied: pygments<3.0.0,>=2.6.0 in /opt/conda/lib/python3.7/site-packages (from rich->colossalai==0.2.8) (2.14.0)\nRequirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.7/site-packages (from markdown-it-py<3.0.0,>=2.1.0->rich->colossalai==0.2.8) (0.1.2)\nRequirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from nodeenv>=0.11.1->pre-commit->colossalai==0.2.8) (59.8.0)\nCollecting pynacl>=1.5\n Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m856.7/856.7 kB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hCollecting bcrypt>=3.2\n Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m593.7/593.7 kB\u001b[0m \u001b[31m42.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: cryptography>=3.3 in /opt/conda/lib/python3.7/site-packages (from paramiko>=2.4->fabric->colossalai==0.2.8) (38.0.2)\nRequirement already satisfied: filelock<4,>=3.4.1 in /opt/conda/lib/python3.7/site-packages (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8) (3.9.0)\nRequirement already satisfied: platformdirs<3,>=2.4 in /opt/conda/lib/python3.7/site-packages (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8) (2.6.2)\nRequirement already satisfied: distlib<1,>=0.3.6 in /opt/conda/lib/python3.7/site-packages (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8) (0.3.6)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->click->colossalai==0.2.8) (3.11.0)\nRequirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.7/site-packages (from cryptography>=3.3->paramiko>=2.4->fabric->colossalai==0.2.8) (1.15.1)\nRequirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.12->cryptography>=3.3->paramiko>=2.4->fabric->colossalai==0.2.8) (2.21)\nBuilding wheels for collected packages: colossalai, contexttimer\n Building wheel for colossalai (setup.py) ... \u001b[?25ldone\n\u001b[?25h Created wheel for colossalai: filename=colossalai-0.2.8-py3-none-any.whl size=1059097 sha256=50ed72a86bf2ae29440764d5c46f67d000fcafbe8d4a5d8f6a947f5e57e85c70\n Stored in directory: /tmp/pip-ephem-wheel-cache-3th5gmed/wheels/3e/97/46/e40c7da8c6931df2650672912b14531b399ef776670745f133\n Building wheel for contexttimer (setup.py) ... \u001b[?25ldone\n\u001b[?25h Created wheel for contexttimer: filename=contexttimer-0.3.3-py3-none-any.whl size=5818 sha256=15b3da44f55d3cf68ee6b623d09715157fa4a60ff4f805c607bca5acf41c83f4\n Stored in directory: /root/.cache/pip/wheels/f4/67/63/f276d2acab046618878e3eaf13c5a356c9a500baf21403f345\nSuccessfully built colossalai contexttimer\nInstalling collected packages: safetensors, contexttimer, nodeenv, invoke, identify, cfgv, bcrypt, pynacl, pre-commit, paramiko, fabric, colossalai\nSuccessfully installed bcrypt-4.0.1 cfgv-3.3.1 colossalai-0.2.8 contexttimer-0.3.3 fabric-3.0.1 identify-2.5.23 invoke-2.1.0 nodeenv-1.7.0 paramiko-3.1.0 pre-commit-2.21.0 pynacl-1.5.0 safetensors-0.3.1\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n\u001b[0m","output_type":"stream"}]},{"cell_type":"markdown","source":"**安装transformers**\n\n这里我们安装的是hpcaitech下的transformers，如果直接pip install transformers是否可行并没有测试","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/hpcaitech/transformers\nos.chdir('./transformers')\n!pip install .","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:57:28.243773Z","iopub.execute_input":"2023-04-30T13:57:28.244481Z","iopub.status.idle":"2023-04-30T13:58:15.962347Z","shell.execute_reply.started":"2023-04-30T13:57:28.244440Z","shell.execute_reply":"2023-04-30T13:58:15.960975Z"},"trusted":true},"execution_count":3,"outputs":[{"name":"stdout","text":"Cloning into 'transformers'...\nremote: Enumerating objects: 124468, done.\u001b[K\nremote: Total 124468 (delta 0), reused 0 (delta 0), pack-reused 124468\u001b[K\nReceiving objects: 100% (124468/124468), 127.08 MiB | 26.91 MiB/s, done.\nResolving deltas: 100% (93344/93344), done.\nProcessing /kaggle/working/ColossalAI/transformers\n Installing build dependencies ... \u001b[?25ldone\n\u001b[?25h Getting requirements to build wheel ... \u001b[?25ldone\n\u001b[?25h Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (23.0)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (3.9.0)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (1.21.6)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (4.11.4)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (6.0)\nRequirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (0.13.2)\nRequirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (0.13.3)\nRequirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (4.64.1)\nRequirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (2.28.2)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers==4.28.0.dev0) (2021.11.10)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.28.0.dev0) (4.4.0)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers==4.28.0.dev0) (3.11.0)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.28.0.dev0) (2022.12.7)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.28.0.dev0) (2.1.1)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.28.0.dev0) (1.26.14)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->transformers==4.28.0.dev0) (3.4)\nBuilding wheels for collected packages: transformers\n Building wheel for transformers (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25h Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6790611 sha256=4220935232e4fb5bbdd639242eec8975f925c105da87c0d4d0137e013c5479a5\n Stored in directory: /tmp/pip-ephem-wheel-cache-u7rm17k7/wheels/f8/7e/62/d660e4bfe297957f2a56ddb6284d5815eba12ca9dfe5b1cf73\nSuccessfully built transformers\nInstalling collected packages: transformers\n Attempting uninstall: transformers\n Found existing installation: transformers 4.27.4\n Uninstalling transformers-4.27.4:\n Successfully uninstalled transformers-4.27.4\nSuccessfully installed transformers-4.28.0.dev0\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n\u001b[0m","output_type":"stream"}]},{"cell_type":"markdown","source":"**安装Chat部分需要的库**","metadata":{}},{"cell_type":"code","source":"os.chdir('/kaggle/working/ColossalAI/applications/Chat/')\n!pip install .","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:58:15.966733Z","iopub.execute_input":"2023-04-30T13:58:15.967109Z","iopub.status.idle":"2023-04-30T13:58:28.362540Z","shell.execute_reply.started":"2023-04-30T13:58:15.967072Z","shell.execute_reply":"2023-04-30T13:58:28.361342Z"},"trusted":true},"execution_count":4,"outputs":[{"name":"stdout","text":"Processing /kaggle/working/ColossalAI/applications/Chat\n Preparing metadata (setup.py) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: transformers>=4.20.1 in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (4.28.0.dev0)\nRequirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (4.64.1)\nRequirement already satisfied: datasets in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (2.1.0)\nCollecting loralib\n Downloading loralib-0.1.1-py3-none-any.whl (8.8 kB)\nRequirement already satisfied: colossalai>=0.2.4 in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (0.2.8)\nRequirement already satisfied: torch<2.0.0,>=1.12.1 in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (1.13.0)\nCollecting langchain\n Downloading langchain-0.0.27-py3-none-any.whl (124 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.9/124.9 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: tokenizers in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (0.13.2)\nRequirement already satisfied: fastapi in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (0.89.1)\nCollecting sse_starlette\n Downloading sse_starlette-0.10.3-py3-none-any.whl (8.0 kB)\nRequirement already satisfied: wandb in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (0.14.0)\nRequirement already satisfied: sentencepiece in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (0.1.97)\nRequirement already satisfied: gpustat in /opt/conda/lib/python3.7/site-packages (from coati==1.0.0) (1.0.0)\nRequirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (1.21.6)\nRequirement already satisfied: safetensors in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (0.3.1)\nRequirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (8.1.3)\nRequirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (23.0)\nRequirement already satisfied: fabric in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (3.0.1)\nRequirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (5.9.3)\nRequirement already satisfied: pre-commit in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (2.21.0)\nRequirement already satisfied: rich in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (13.2.0)\nRequirement already satisfied: ninja in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (1.11.1)\nRequirement already satisfied: contexttimer in /opt/conda/lib/python3.7/site-packages (from colossalai>=0.2.4->coati==1.0.0) (0.3.3)\nRequirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch<2.0.0,>=1.12.1->coati==1.0.0) (4.4.0)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from transformers>=4.20.1->coati==1.0.0) (4.11.4)\nRequirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from transformers>=4.20.1->coati==1.0.0) (2.28.2)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers>=4.20.1->coati==1.0.0) (3.9.0)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from transformers>=4.20.1->coati==1.0.0) (6.0)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers>=4.20.1->coati==1.0.0) (2021.11.10)\nRequirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /opt/conda/lib/python3.7/site-packages (from transformers>=4.20.1->coati==1.0.0) (0.13.3)\nRequirement already satisfied: multiprocess in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (0.70.14)\nRequirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (3.8.3)\nRequirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (1.3.5)\nRequirement already satisfied: pyarrow>=5.0.0 in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (5.0.0)\nRequirement already satisfied: xxhash in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (3.2.0)\nRequirement already satisfied: responses<0.19 in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (0.18.0)\nRequirement already satisfied: dill in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (0.3.6)\nRequirement already satisfied: fsspec[http]>=2021.05.0 in /opt/conda/lib/python3.7/site-packages (from datasets->coati==1.0.0) (2023.1.0)\nRequirement already satisfied: starlette==0.22.0 in /opt/conda/lib/python3.7/site-packages (from fastapi->coati==1.0.0) (0.22.0)\nRequirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /opt/conda/lib/python3.7/site-packages (from fastapi->coati==1.0.0) (1.10.4)\nRequirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/lib/python3.7/site-packages (from starlette==0.22.0->fastapi->coati==1.0.0) (3.6.2)\nRequirement already satisfied: nvidia-ml-py<=11.495.46,>=11.450.129 in /opt/conda/lib/python3.7/site-packages (from gpustat->coati==1.0.0) (11.495.46)\nRequirement already satisfied: six>=1.7 in /opt/conda/lib/python3.7/site-packages (from gpustat->coati==1.0.0) (1.16.0)\nRequirement already satisfied: blessed>=1.17.1 in /opt/conda/lib/python3.7/site-packages (from gpustat->coati==1.0.0) (1.19.1)\nRequirement already satisfied: sqlalchemy in /opt/conda/lib/python3.7/site-packages (from langchain->coati==1.0.0) (1.4.46)\nRequirement already satisfied: GitPython!=3.1.29,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (3.1.30)\nRequirement already satisfied: pathtools in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (0.1.2)\nRequirement already satisfied: protobuf!=4.21.0,<5,>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (3.20.3)\nRequirement already satisfied: setproctitle in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (1.3.2)\nRequirement already satisfied: sentry-sdk>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (1.18.0)\nRequirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (59.8.0)\nRequirement already satisfied: appdirs>=1.4.3 in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (1.4.4)\nRequirement already satisfied: docker-pycreds>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from wandb->coati==1.0.0) (0.4.0)\nRequirement already satisfied: wcwidth>=0.1.4 in /opt/conda/lib/python3.7/site-packages (from blessed>=1.17.1->gpustat->coati==1.0.0) (0.2.6)\nRequirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (1.3.1)\nRequirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (6.0.4)\nRequirement already satisfied: charset-normalizer<3.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (2.1.1)\nRequirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (4.0.2)\nRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (1.8.2)\nRequirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (22.2.0)\nRequirement already satisfied: asynctest==0.13.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (0.13.0)\nRequirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from aiohttp->datasets->coati==1.0.0) (1.3.3)\nRequirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.7/site-packages (from GitPython!=3.1.29,>=1.0.0->wandb->coati==1.0.0) (4.0.10)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->transformers>=4.20.1->coati==1.0.0) (3.4)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->transformers>=4.20.1->coati==1.0.0) (2022.12.7)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->transformers>=4.20.1->coati==1.0.0) (1.26.14)\nRequirement already satisfied: paramiko>=2.4 in /opt/conda/lib/python3.7/site-packages (from fabric->colossalai>=0.2.4->coati==1.0.0) (3.1.0)\nRequirement already satisfied: invoke>=2.0 in /opt/conda/lib/python3.7/site-packages (from fabric->colossalai>=0.2.4->coati==1.0.0) (2.1.0)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers>=4.20.1->coati==1.0.0) (3.11.0)\nRequirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->datasets->coati==1.0.0) (2.8.2)\nRequirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->datasets->coati==1.0.0) (2023.3)\nRequirement already satisfied: virtualenv>=20.10.0 in /opt/conda/lib/python3.7/site-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (20.17.1)\nRequirement already satisfied: cfgv>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (3.3.1)\nRequirement already satisfied: nodeenv>=0.11.1 in /opt/conda/lib/python3.7/site-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (1.7.0)\nRequirement already satisfied: identify>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (2.5.23)\nRequirement already satisfied: markdown-it-py<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from rich->colossalai>=0.2.4->coati==1.0.0) (2.1.0)\nRequirement already satisfied: pygments<3.0.0,>=2.6.0 in /opt/conda/lib/python3.7/site-packages (from rich->colossalai>=0.2.4->coati==1.0.0) (2.14.0)\nRequirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.7/site-packages (from sqlalchemy->langchain->coati==1.0.0) (2.0.1)\nRequirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.7/site-packages (from anyio<5,>=3.4.0->starlette==0.22.0->fastapi->coati==1.0.0) (1.3.0)\nRequirement already satisfied: smmap<6,>=3.0.1 in /opt/conda/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->coati==1.0.0) (5.0.0)\nRequirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.7/site-packages (from markdown-it-py<3.0.0,>=2.1.0->rich->colossalai>=0.2.4->coati==1.0.0) (0.1.2)\nRequirement already satisfied: cryptography>=3.3 in /opt/conda/lib/python3.7/site-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (38.0.2)\nRequirement already satisfied: pynacl>=1.5 in /opt/conda/lib/python3.7/site-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (1.5.0)\nRequirement already satisfied: bcrypt>=3.2 in /opt/conda/lib/python3.7/site-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (4.0.1)\nRequirement already satisfied: platformdirs<3,>=2.4 in /opt/conda/lib/python3.7/site-packages (from virtualenv>=20.10.0->pre-commit->colossalai>=0.2.4->coati==1.0.0) (2.6.2)\nRequirement already satisfied: distlib<1,>=0.3.6 in /opt/conda/lib/python3.7/site-packages (from virtualenv>=20.10.0->pre-commit->colossalai>=0.2.4->coati==1.0.0) (0.3.6)\nRequirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.7/site-packages (from cryptography>=3.3->paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (1.15.1)\nRequirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.12->cryptography>=3.3->paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (2.21)\nBuilding wheels for collected packages: coati\n Building wheel for coati (setup.py) ... \u001b[?25ldone\n\u001b[?25h Created wheel for coati: filename=coati-1.0.0-py3-none-any.whl size=73195 sha256=114e624d66aa7c22966144210804c68036dda419db454e1fb5a6b657d58b5879\n Stored in directory: /tmp/pip-ephem-wheel-cache-pshxknq0/wheels/19/ab/40/58b2528cfb9dab45fa2cdceeff3538d85c2c72d65872c4de6a\nSuccessfully built coati\nInstalling collected packages: loralib, sse_starlette, langchain, coati\nSuccessfully installed coati-1.0.0 langchain-0.0.27 loralib-0.1.1 sse_starlette-0.10.3\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n\u001b[0m","output_type":"stream"}]},{"cell_type":"markdown","source":"我们还需要将**analyzer**的部分复制到当前操作系统的对应的**python packages**中（否则可能将来会报错：找不到analyzer）\n\n如果你问，我怎么知道packages的路径是哪里呢？其实在上面执行各种pip install的过程中，就可以发现这个路径。","metadata":{}},{"cell_type":"code","source":"!cp -r /kaggle/working/ColossalAI/colossalai/_analyzer/ /opt/conda/lib/python3.7/site-packages/colossalai/","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:58:28.364761Z","iopub.execute_input":"2023-04-30T13:58:28.365247Z","iopub.status.idle":"2023-04-30T13:58:29.347282Z","shell.execute_reply.started":"2023-04-30T13:58:28.365204Z","shell.execute_reply":"2023-04-30T13:58:29.345935Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"markdown","source":"### 2 预训练模型的下载（以bloom为例）","metadata":{}},{"cell_type":"markdown","source":"首先**需要执行这个命令**（如果你是在colab中做尝试，可能也需要）\n\n如果不安装下面的命令会出现什么情况呢？\n- 从huggingface中git clone下来的模型看似下载下来了，但是其实下载下来的并不是实质的模型文件（如果你检查文件的大小，只有几B）\n- 一旦下载下来的文件并不是实质的模型，则在运行SFT代码的时候会报错：**safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge**","metadata":{}},{"cell_type":"code","source":"!sudo apt-get install git-lfs\n!git lfs install","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:58:29.349334Z","iopub.execute_input":"2023-04-30T13:58:29.349759Z","iopub.status.idle":"2023-04-30T13:58:33.723349Z","shell.execute_reply.started":"2023-04-30T13:58:29.349713Z","shell.execute_reply":"2023-04-30T13:58:33.722173Z"},"trusted":true},"execution_count":6,"outputs":[{"name":"stdout","text":"Reading package lists... Done\nBuilding dependency tree \nReading state information... Done\ngit-lfs is already the newest version (2.9.2-1).\n0 upgraded, 0 newly installed, 0 to remove and 76 not upgraded.\nUpdated git hooks.\nGit LFS initialized.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"**下载ColossalAI支持的系列模型**，我们以bloomz-560m为例。在下面，我们将模型放在了/kaggle/working/中，但是这里并不是强制的，可以根据自己的喜欢变换位置。","metadata":{}},{"cell_type":"code","source":"os.chdir('/kaggle/working/')\n!git clone https://huggingface.co/bigscience/bloomz-560m","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:58:33.726131Z","iopub.execute_input":"2023-04-30T13:58:33.726802Z","iopub.status.idle":"2023-04-30T13:59:00.114858Z","shell.execute_reply.started":"2023-04-30T13:58:33.726758Z","shell.execute_reply":"2023-04-30T13:59:00.113640Z"},"trusted":true},"execution_count":7,"outputs":[{"name":"stdout","text":"Cloning into 'bloomz-560m'...\nremote: Enumerating objects: 1332, done.\u001b[K\nremote: Counting objects: 100% (10/10), done.\u001b[K\nremote: Compressing objects: 100% (10/10), done.\u001b[K\nremote: Total 1332 (delta 3), reused 0 (delta 0), pack-reused 1322\u001b[K\nReceiving objects: 100% (1332/1332), 7.18 MiB | 22.55 MiB/s, done.\nResolving deltas: 100% (616/616), done.\nFiltering content: 100% (8/8), 2.11 GiB | 88.79 MiB/s, done.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### 3 运行SFT","metadata":{}},{"cell_type":"code","source":"os.chdir('/kaggle/working/ColossalAI/applications/Chat/examples')","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:59:00.116823Z","iopub.execute_input":"2023-04-30T13:59:00.117188Z","iopub.status.idle":"2023-04-30T13:59:00.124033Z","shell.execute_reply.started":"2023-04-30T13:59:00.117154Z","shell.execute_reply":"2023-04-30T13:59:00.123036Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"markdown","source":"**执行SFT代码**\n\n我们这里是直接运行的py文件，如果你按照[文档的说明](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)，运行sh脚本文件（!bash train_sft.sh）也是可以的。其本质上，都是运行这个train_sft.py文件。\n\n在下面的命令中，为了演示\n- 我们只是用一个非常非常非常小的数据集（--dataset）去跑程序（小到就只有5条数据）\n- model：改为了“bloom”\n- pretrain：改成了我们自己下载的模型地址\n- save_path: 改成了我们想放的目录地址\n\n**需要注意的是：**\n- Kaggle Notebook的GPU是T4x2,所以显存大概有12+12=24G。所以我们在运行训练的时候可以设置**--nproc_per_node=2**（如果是1的话，则有一块GPU会闲置，显存也会砍半）\n- 其他参数可能需要你自己去多多探索：比如lora、gradient checkingpoint等。更多参数的说明见[官方文档](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)","metadata":{}},{"cell_type":"code","source":"!torchrun --standalone --nproc_per_node=2 train_sft.py \\\n --pretrain \"/kaggle/working/bloomz-560m\" \\\n --model 'bloom' \\\n --strategy colossalai_zero2 \\\n --log_interval 50 \\\n --save_path \"/kaggle/working/bloomz-560m-finetuned\" \\\n --dataset \"/kaggle/input/instructdata/instinwild_ch_small.json\" \\\n --batch_size 4 \\\n --accumulation_steps 8 \\\n --lr 2e-5 \\\n --max_datasets_size 512 \\\n --max_epochs 1","metadata":{"execution":{"iopub.status.busy":"2023-04-30T13:59:00.125449Z","iopub.execute_input":"2023-04-30T13:59:00.126251Z","iopub.status.idle":"2023-04-30T14:03:20.357385Z","shell.execute_reply.started":"2023-04-30T13:59:00.126214Z","shell.execute_reply":"2023-04-30T14:03:20.354320Z"},"trusted":true},"execution_count":9,"outputs":[{"name":"stdout","text":"\u001b[2;36m[04/30/23 13:59:19]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/colossalai/c\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35montext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m522\u001b[0m set_device \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: process rank \u001b[1;36m0\u001b[0m is \n\u001b[2;36m \u001b[0m bound to device \u001b[1;36m0\u001b[0m \n\u001b[2;36m[04/30/23 13:59:24]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/colossalai/c\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35montext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m558\u001b[0m set_seed \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: initialized seed on\n\u001b[2;36m \u001b[0m rank \u001b[1;36m0\u001b[0m, numpy: \u001b[1;36m42\u001b[0m, python random: \u001b[1;36m42\u001b[0m, \n\u001b[2;36m \u001b[0m ParallelMode.DATA: \u001b[1;36m42\u001b[0m, ParallelMode.TENSOR: \u001b[1;36m42\u001b[0m,the \n\u001b[2;36m \u001b[0m default parallel seed is ParallelMode.DATA. \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/colossalai/\u001b[0m\u001b[95mi\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[95mnitialize.py\u001b[0m:\u001b[1;36m119\u001b[0m launch \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Distributed \n\u001b[2;36m \u001b[0m environment is initialized, data parallel size: \u001b[1;36m1\u001b[0m, \n\u001b[2;36m \u001b[0m pipeline parallel size: \u001b[1;36m1\u001b[0m, tensor parallel size: \u001b[1;36m1\u001b[0m \n\u001b[2;36m[04/30/23 14:03:09]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/datase\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mt/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m121\u001b[0m __init__ \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loading data\u001b[33m...\u001b[0m \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/datase\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mt/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m123\u001b[0m __init__ \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loaded \u001b[1;36m6\u001b[0m examples. \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/datase\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mt/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m126\u001b[0m __init__ \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Limiting dataset to\n\u001b[2;36m \u001b[0m \u001b[1;36m512\u001b[0m examples. \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/datase\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mt/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m129\u001b[0m __init__ \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Formatting \n\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/datase\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mt/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m137\u001b[0m __init__ \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Tokenizing \n\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m This may take some time\u001b[33m...\u001b[0m \nsteps: 0it [00:00, ?it/s]\u001b[2;36m[04/30/23 14:03:13]\u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/traine\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mr/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m86\u001b[0m fit \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: batch_i\u001b[1;92md:0\u001b[0m, \n\u001b[2;36m \u001b[0m abnormal loss: \u001b[1;36m3.6484375\u001b[0m \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: \n\u001b[2;36m \u001b[0m \u001b[35m/opt/conda/lib/python3.7/site-packages/coati/traine\u001b[0m\n\u001b[2;36m \u001b[0m \u001b[35mr/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m86\u001b[0m fit \n\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: batch_i\u001b[1;92md:1\u001b[0m, \n\u001b[2;36m \u001b[0m abnormal loss: \u001b[1;36m4.1015625\u001b[0m \nsteps: 0it [00:02, ?it/s]\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### 4 下载SFT完成的模型","metadata":{}},{"cell_type":"code","source":"os.chdir('/kaggle/working/')","metadata":{"execution":{"iopub.status.busy":"2023-04-30T14:03:20.361678Z","iopub.execute_input":"2023-04-30T14:03:20.362417Z","iopub.status.idle":"2023-04-30T14:03:20.368009Z","shell.execute_reply.started":"2023-04-30T14:03:20.362368Z","shell.execute_reply":"2023-04-30T14:03:20.367092Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"markdown","source":"理论上，我们是可以通过Kaggle Notebook右边的界面，选择想要下载的文件进行下载。但是由于Kaggle界面做的并不是很好，**经常会出现目录不能显示的问题**。好在，你应该是知道训练好的模型是放在了哪里。我们可以找到另一种办法将文件下载下来。","metadata":{}},{"cell_type":"markdown","source":"首先，我们把**完成的模型进行打包**（这样我们就可以一下字全都下载下来了，而不用一个一个的下载）","metadata":{}},{"cell_type":"code","source":"!tar -czvf bloomz-560m-finetuned.tar.gz bloomz-560m-finetuned","metadata":{"execution":{"iopub.status.busy":"2023-04-30T14:03:20.369504Z","iopub.execute_input":"2023-04-30T14:03:20.369861Z","iopub.status.idle":"2023-04-30T14:04:11.809532Z","shell.execute_reply.started":"2023-04-30T14:03:20.369826Z","shell.execute_reply":"2023-04-30T14:04:11.808272Z"},"trusted":true},"execution_count":11,"outputs":[{"name":"stdout","text":"bloomz-560m-finetuned/\nbloomz-560m-finetuned/config.json\nbloomz-560m-finetuned/pytorch_model.bin\nbloomz-560m-finetuned/tokenizer_config.json\nbloomz-560m-finetuned/generation_config.json\nbloomz-560m-finetuned/special_tokens_map.json\nbloomz-560m-finetuned/tokenizer.json\n","output_type":"stream"}]},{"cell_type":"markdown","source":"完成打包后，我们**想办法获得下载链接**。执行下面的命令后，直接点击链接便可下载。","metadata":{}},{"cell_type":"code","source":"from IPython.display import FileLink\nFileLink(r'bloomz-560m-finetuned.tar.gz')","metadata":{"execution":{"iopub.status.busy":"2023-04-30T14:04:11.811207Z","iopub.execute_input":"2023-04-30T14:04:11.811637Z","iopub.status.idle":"2023-04-30T14:04:11.821318Z","shell.execute_reply.started":"2023-04-30T14:04:11.811584Z","shell.execute_reply":"2023-04-30T14:04:11.820184Z"},"trusted":true},"execution_count":12,"outputs":[{"execution_count":12,"output_type":"execute_result","data":{"text/plain":"/kaggle/working/bloomz-560m-finetuned.tar.gz","text/html":"bloomz-560m-finetuned.tar.gz
"},"metadata":{}}]},{"cell_type":"markdown","source":"### 5 小结\n仅仅跑起来只是一个开始。祝愿每个小伙伴都会最终获得更好的GPU资源、更棒的数据和更出色的属于自己的语言模型！","metadata":{}}]}
2 |
--------------------------------------------------------------------------------
/02-colossalai-sft-colab.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "provenance": [],
7 | "gpuType": "T4",
8 | "collapsed_sections": [
9 | "NY7XsonIG8Ev",
10 | "bYe-_gnDHEEa"
11 | ]
12 | },
13 | "kernelspec": {
14 | "name": "python3",
15 | "display_name": "Python 3"
16 | },
17 | "language_info": {
18 | "name": "python"
19 | },
20 | "accelerator": "GPU",
21 | "gpuClass": "standard"
22 | },
23 | "cells": [
24 | {
25 | "cell_type": "markdown",
26 | "source": [
27 | "# 语言模型练习场02：ColossalAI（SFT部分）Google Colab版本"
28 | ],
29 | "metadata": {
30 | "id": "AamIytcNnxL1"
31 | }
32 | },
33 | {
34 | "cell_type": "markdown",
35 | "source": [
36 | "- [https://github.com/hpcaitech/ColossalAI](https://github.com/hpcaitech/ColossalAI)"
37 | ],
38 | "metadata": {
39 | "id": "yAw_VzT6pVwW"
40 | }
41 | },
42 | {
43 | "cell_type": "markdown",
44 | "source": [
45 | "**注意：**\n",
46 | "- 此notebook**只演示在google colab下如何跑通ColossalAI的SFT部分**，并不会包含超参数的调整、对结果的分析等\n",
47 | "- 如果想在**Kaggle Notebook**中跑，可以看这篇文章的Kaggle版：[穷穷穷孩子如何体验ColossalAI SFT（Kaggle篇）](https://mp.weixin.qq.com/s/Q29uSNxvPMy0rC-QxHiGZA)\n",
48 | "- **如果你有自己的机器**，则此notebook对你的帮助可能不大（因为你不需要在notebook上进行训练）\n",
49 | "- 此notebook的受众是手里没有GPU资源，但是又想熟悉和浅浅尝试ColossalAI的小伙伴\n",
50 | "- Google Colab目前可以获得的免费运算资源是**T4（大概有15GB的显存）**\n",
51 | "- **运行太长时间会自动终止**。所以建议重要的运行结果保存路径要指向自己的google drive目录中，而不那么重要的文件可以直接安装到默认目录下（终止后会被自动删除）"
52 | ],
53 | "metadata": {
54 | "id": "hmoMi8BspejU"
55 | }
56 | },
57 | {
58 | "cell_type": "markdown",
59 | "source": [
60 | "**数据的准备：**\n",
61 | "- 根据[官方文档的提示](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)，在运行前需要准备好数据（https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples）\n",
62 | "- 数据可以在[这里下载](https://github.com/XueFuzhao/InstructionWild/tree/main/data)。注意不要下载seed文件（因为seed文件只有instruction，而没有response），要下载README里面提到的json文件，例如instinwild_ch.json（https://github.com/XueFuzhao/InstructionWild/tree/main/data）"
63 | ],
64 | "metadata": {
65 | "id": "lvubgzEuGch2"
66 | }
67 | },
68 | {
69 | "cell_type": "markdown",
70 | "source": [
71 | "## 安装环境"
72 | ],
73 | "metadata": {
74 | "id": "zN_47P5fG3c_"
75 | }
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "source": [
80 | "### 1 安装ColossalAI"
81 | ],
82 | "metadata": {
83 | "id": "NY7XsonIG8Ev"
84 | }
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": null,
89 | "metadata": {
90 | "colab": {
91 | "base_uri": "https://localhost:8080/"
92 | },
93 | "id": "PhpvqFt06DSm",
94 | "outputId": "62102879-51e9-4014-a248-7e2020d83ded"
95 | },
96 | "outputs": [
97 | {
98 | "output_type": "stream",
99 | "name": "stdout",
100 | "text": [
101 | "Cloning into 'ColossalAI'...\n",
102 | "remote: Enumerating objects: 26116, done.\u001b[K\n",
103 | "remote: Counting objects: 100% (5/5), done.\u001b[K\n",
104 | "remote: Compressing objects: 100% (5/5), done.\u001b[K\n",
105 | "remote: Total 26116 (delta 0), reused 0 (delta 0), pack-reused 26111\u001b[K\n",
106 | "Receiving objects: 100% (26116/26116), 23.36 MiB | 34.12 MiB/s, done.\n",
107 | "Resolving deltas: 100% (17448/17448), done.\n"
108 | ]
109 | }
110 | ],
111 | "source": [
112 | "!git clone https://github.com/hpcaitech/ColossalAI.git"
113 | ]
114 | },
115 | {
116 | "cell_type": "code",
117 | "source": [
118 | "import os\n",
119 | "os.chdir('./ColossalAI')\n",
120 | "!pip install ."
121 | ],
122 | "metadata": {
123 | "id": "kWmQqVbu6Ncc",
124 | "colab": {
125 | "base_uri": "https://localhost:8080/"
126 | },
127 | "outputId": "7554b306-038f-4b06-a5e0-a7734c8e8945"
128 | },
129 | "execution_count": null,
130 | "outputs": [
131 | {
132 | "output_type": "stream",
133 | "name": "stdout",
134 | "text": [
135 | "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
136 | "Processing /content/ColossalAI\n",
137 | " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
138 | "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (1.22.4)\n",
139 | "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (4.65.0)\n",
140 | "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (5.9.5)\n",
141 | "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (23.1)\n",
142 | "Collecting pre-commit (from colossalai==0.2.8)\n",
143 | " Downloading pre_commit-3.3.2-py2.py3-none-any.whl (202 kB)\n",
144 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m202.8/202.8 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
145 | "\u001b[?25hRequirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (13.3.4)\n",
146 | "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (8.1.3)\n",
147 | "Collecting fabric (from colossalai==0.2.8)\n",
148 | " Downloading fabric-3.0.1-py3-none-any.whl (53 kB)\n",
149 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.3/53.3 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
150 | "\u001b[?25hCollecting contexttimer (from colossalai==0.2.8)\n",
151 | " Downloading contexttimer-0.3.3.tar.gz (4.9 kB)\n",
152 | " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
153 | "Collecting ninja (from colossalai==0.2.8)\n",
154 | " Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)\n",
155 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.0/146.0 kB\u001b[0m \u001b[31m21.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
156 | "\u001b[?25hRequirement already satisfied: torch>=1.11 in /usr/local/lib/python3.10/dist-packages (from colossalai==0.2.8) (2.0.1+cu118)\n",
157 | "Collecting safetensors (from colossalai==0.2.8)\n",
158 | " Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
159 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m39.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
160 | "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (3.12.0)\n",
161 | "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (4.5.0)\n",
162 | "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (1.11.1)\n",
163 | "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (3.1)\n",
164 | "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (3.1.2)\n",
165 | "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11->colossalai==0.2.8) (2.0.0)\n",
166 | "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.11->colossalai==0.2.8) (3.25.2)\n",
167 | "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.11->colossalai==0.2.8) (16.0.5)\n",
168 | "Collecting invoke>=2.0 (from fabric->colossalai==0.2.8)\n",
169 | " Downloading invoke-2.1.2-py3-none-any.whl (160 kB)\n",
170 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m160.1/160.1 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
171 | "\u001b[?25hCollecting paramiko>=2.4 (from fabric->colossalai==0.2.8)\n",
172 | " Downloading paramiko-3.1.0-py3-none-any.whl (211 kB)\n",
173 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.2/211.2 kB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
174 | "\u001b[?25hCollecting cfgv>=2.0.0 (from pre-commit->colossalai==0.2.8)\n",
175 | " Downloading cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)\n",
176 | "Collecting identify>=1.0.0 (from pre-commit->colossalai==0.2.8)\n",
177 | " Downloading identify-2.5.24-py2.py3-none-any.whl (98 kB)\n",
178 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.8/98.8 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
179 | "\u001b[?25hCollecting nodeenv>=0.11.1 (from pre-commit->colossalai==0.2.8)\n",
180 | " Downloading nodeenv-1.8.0-py2.py3-none-any.whl (22 kB)\n",
181 | "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai==0.2.8) (6.0)\n",
182 | "Collecting virtualenv>=20.10.0 (from pre-commit->colossalai==0.2.8)\n",
183 | " Downloading virtualenv-20.23.0-py3-none-any.whl (3.3 MB)\n",
184 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m81.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
185 | "\u001b[?25hRequirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai==0.2.8) (2.2.0)\n",
186 | "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai==0.2.8) (2.14.0)\n",
187 | "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->colossalai==0.2.8) (0.1.2)\n",
188 | "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nodeenv>=0.11.1->pre-commit->colossalai==0.2.8) (67.7.2)\n",
189 | "Collecting bcrypt>=3.2 (from paramiko>=2.4->fabric->colossalai==0.2.8)\n",
190 | " Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)\n",
191 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m593.7/593.7 kB\u001b[0m \u001b[31m54.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
192 | "\u001b[?25hRequirement already satisfied: cryptography>=3.3 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai==0.2.8) (40.0.2)\n",
193 | "Collecting pynacl>=1.5 (from paramiko>=2.4->fabric->colossalai==0.2.8)\n",
194 | " Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)\n",
195 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m856.7/856.7 kB\u001b[0m \u001b[31m67.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
196 | "\u001b[?25hCollecting distlib<1,>=0.3.6 (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8)\n",
197 | " Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)\n",
198 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.5/468.5 kB\u001b[0m \u001b[31m50.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
199 | "\u001b[?25hRequirement already satisfied: platformdirs<4,>=3.2 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->colossalai==0.2.8) (3.3.0)\n",
200 | "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11->colossalai==0.2.8) (2.1.2)\n",
201 | "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11->colossalai==0.2.8) (1.3.0)\n",
202 | "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/dist-packages (from cryptography>=3.3->paramiko>=2.4->fabric->colossalai==0.2.8) (1.15.1)\n",
203 | "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.12->cryptography>=3.3->paramiko>=2.4->fabric->colossalai==0.2.8) (2.21)\n",
204 | "Building wheels for collected packages: colossalai, contexttimer\n",
205 | " Building wheel for colossalai (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
206 | " Created wheel for colossalai: filename=colossalai-0.2.8-py3-none-any.whl size=1106778 sha256=c48863e1d61e5a17a6d7d3fbd80022a7990e07e012015591dd40042ddb3b2760\n",
207 | " Stored in directory: /tmp/pip-ephem-wheel-cache-bobgacrn/wheels/b1/fb/16/e46aa3127ee272b8cac710c8f76aa02445d96aaeed9da956ea\n",
208 | " Building wheel for contexttimer (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
209 | " Created wheel for contexttimer: filename=contexttimer-0.3.3-py3-none-any.whl size=5803 sha256=21ac5cf4b74bf53b9a96c4f8705106a7107d03a3c3f118b2515e3f7419170e06\n",
210 | " Stored in directory: /root/.cache/pip/wheels/72/1c/da/cfd97201d88ccce214427fa84a5caeb91fef7c5a1b4c4312b4\n",
211 | "Successfully built colossalai contexttimer\n",
212 | "Installing collected packages: safetensors, ninja, distlib, contexttimer, virtualenv, nodeenv, invoke, identify, cfgv, bcrypt, pynacl, pre-commit, paramiko, fabric, colossalai\n",
213 | "Successfully installed bcrypt-4.0.1 cfgv-3.3.1 colossalai-0.2.8 contexttimer-0.3.3 distlib-0.3.6 fabric-3.0.1 identify-2.5.24 invoke-2.1.2 ninja-1.11.1 nodeenv-1.8.0 paramiko-3.1.0 pre-commit-3.3.2 pynacl-1.5.0 safetensors-0.3.1 virtualenv-20.23.0\n"
214 | ]
215 | }
216 | ]
217 | },
218 | {
219 | "cell_type": "markdown",
220 | "source": [
221 | "### 2 安装transformers\n",
222 | "这里我们安装的是hpcaitech下的transformers，如果直接pip install transformers是否可行并没有测试。"
223 | ],
224 | "metadata": {
225 | "id": "bYe-_gnDHEEa"
226 | }
227 | },
228 | {
229 | "cell_type": "code",
230 | "source": [
231 | "!git clone https://github.com/hpcaitech/transformers\n",
232 | "os.chdir('./transformers')\n",
233 | "!pip install ."
234 | ],
235 | "metadata": {
236 | "colab": {
237 | "base_uri": "https://localhost:8080/"
238 | },
239 | "id": "PJTkDrHUhFpP",
240 | "outputId": "11383ade-42af-4f76-ab3e-f47f955a863d"
241 | },
242 | "execution_count": null,
243 | "outputs": [
244 | {
245 | "output_type": "stream",
246 | "name": "stdout",
247 | "text": [
248 | "Cloning into 'transformers'...\n",
249 | "remote: Enumerating objects: 124468, done.\u001b[K\n",
250 | "remote: Total 124468 (delta 0), reused 0 (delta 0), pack-reused 124468\u001b[K\n",
251 | "Receiving objects: 100% (124468/124468), 127.28 MiB | 27.99 MiB/s, done.\n",
252 | "Resolving deltas: 100% (93320/93320), done.\n",
253 | "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
254 | "Processing /content/ColossalAI/transformers\n",
255 | " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
256 | " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
257 | " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
258 | "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (3.12.0)\n",
259 | "Collecting huggingface-hub<1.0,>=0.11.0 (from transformers==4.28.0.dev0)\n",
260 | " Downloading huggingface_hub-0.14.1-py3-none-any.whl (224 kB)\n",
261 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.5/224.5 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
262 | "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (1.22.4)\n",
263 | "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (23.1)\n",
264 | "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (6.0)\n",
265 | "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (2022.10.31)\n",
266 | "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (2.27.1)\n",
267 | "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.28.0.dev0)\n",
268 | " Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
269 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m64.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
270 | "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (4.65.0)\n",
271 | "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.28.0.dev0) (2023.4.0)\n",
272 | "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.28.0.dev0) (4.5.0)\n",
273 | "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (1.26.15)\n",
274 | "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (2022.12.7)\n",
275 | "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (2.0.12)\n",
276 | "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (3.4)\n",
277 | "Building wheels for collected packages: transformers\n",
278 | " Building wheel for transformers (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
279 | " Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6790611 sha256=92968d77f2b7dc7aa1557f1ab5447bb5ddbf00b2936e257790df7e00d1659f10\n",
280 | " Stored in directory: /tmp/pip-ephem-wheel-cache-pyix5uo5/wheels/87/f3/6f/b220a07b1eb427c5c698eed3338325ec784fe66427d0989fa6\n",
281 | "Successfully built transformers\n",
282 | "Installing collected packages: tokenizers, huggingface-hub, transformers\n",
283 | "Successfully installed huggingface-hub-0.14.1 tokenizers-0.13.3 transformers-4.28.0.dev0\n"
284 | ]
285 | }
286 | ]
287 | },
288 | {
289 | "cell_type": "markdown",
290 | "source": [
291 | "### 3 安装Chat需要的库"
292 | ],
293 | "metadata": {
294 | "id": "Y_KsSNVYHM_I"
295 | }
296 | },
297 | {
298 | "cell_type": "code",
299 | "source": [
300 | "os.chdir('../applications/Chat/')\n",
301 | "!pip install ."
302 | ],
303 | "metadata": {
304 | "colab": {
305 | "base_uri": "https://localhost:8080/"
306 | },
307 | "id": "F73wlOCahZRz",
308 | "outputId": "e6c92639-0202-47d4-d917-637dc8ba064e"
309 | },
310 | "execution_count": null,
311 | "outputs": [
312 | {
313 | "output_type": "stream",
314 | "name": "stdout",
315 | "text": [
316 | "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
317 | "Processing /content/ColossalAI/applications/Chat\n",
318 | " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
319 | "Requirement already satisfied: transformers>=4.20.1 in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (4.28.0.dev0)\n",
320 | "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (4.65.0)\n",
321 | "Collecting datasets (from coati==1.0.0)\n",
322 | " Downloading datasets-2.12.0-py3-none-any.whl (474 kB)\n",
323 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m474.6/474.6 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
324 | "\u001b[?25hCollecting loralib (from coati==1.0.0)\n",
325 | " Downloading loralib-0.1.1-py3-none-any.whl (8.8 kB)\n",
326 | "Requirement already satisfied: colossalai>=0.2.4 in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (0.2.8)\n",
327 | "Collecting torch<2.0.0,>=1.12.1 (from coati==1.0.0)\n",
328 | " Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)\n",
329 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m887.5/887.5 MB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
330 | "\u001b[?25hCollecting langchain (from coati==1.0.0)\n",
331 | " Downloading langchain-0.0.178-py3-none-any.whl (892 kB)\n",
332 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m892.2/892.2 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
333 | "\u001b[?25hRequirement already satisfied: tokenizers in /usr/local/lib/python3.10/dist-packages (from coati==1.0.0) (0.13.3)\n",
334 | "Collecting fastapi (from coati==1.0.0)\n",
335 | " Downloading fastapi-0.95.2-py3-none-any.whl (56 kB)\n",
336 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.0/57.0 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
337 | "\u001b[?25hCollecting sse_starlette (from coati==1.0.0)\n",
338 | " Downloading sse_starlette-1.6.1-py3-none-any.whl (9.6 kB)\n",
339 | "Collecting wandb (from coati==1.0.0)\n",
340 | " Downloading wandb-0.15.3-py3-none-any.whl (2.0 MB)\n",
341 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
342 | "\u001b[?25hCollecting sentencepiece (from coati==1.0.0)\n",
343 | " Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
344 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
345 | "\u001b[?25hCollecting gpustat (from coati==1.0.0)\n",
346 | " Downloading gpustat-1.1.tar.gz (97 kB)\n",
347 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
348 | "\u001b[?25h Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
349 | " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
350 | " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
351 | "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (1.22.4)\n",
352 | "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (5.9.5)\n",
353 | "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (23.1)\n",
354 | "Requirement already satisfied: pre-commit in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (3.3.2)\n",
355 | "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (13.3.4)\n",
356 | "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (8.1.3)\n",
357 | "Requirement already satisfied: fabric in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (3.0.1)\n",
358 | "Requirement already satisfied: contexttimer in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (0.3.3)\n",
359 | "Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (1.11.1)\n",
360 | "Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from colossalai>=0.2.4->coati==1.0.0) (0.3.1)\n",
361 | "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch<2.0.0,>=1.12.1->coati==1.0.0) (4.5.0)\n",
362 | "Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
363 | " Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
364 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
365 | "\u001b[?25hCollecting nvidia-cudnn-cu11==8.5.0.96 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
366 | " Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
367 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
368 | "\u001b[?25hCollecting nvidia-cublas-cu11==11.10.3.66 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
369 | " Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
370 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
371 | "\u001b[?25hCollecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch<2.0.0,>=1.12.1->coati==1.0.0)\n",
372 | " Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
373 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
374 | "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<2.0.0,>=1.12.1->coati==1.0.0) (67.7.2)\n",
375 | "Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<2.0.0,>=1.12.1->coati==1.0.0) (0.40.0)\n",
376 | "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (3.12.0)\n",
377 | "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (0.14.1)\n",
378 | "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (6.0)\n",
379 | "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (2022.10.31)\n",
380 | "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.20.1->coati==1.0.0) (2.27.1)\n",
381 | "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->coati==1.0.0) (9.0.0)\n",
382 | "Collecting dill<0.3.7,>=0.3.0 (from datasets->coati==1.0.0)\n",
383 | " Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n",
384 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
385 | "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->coati==1.0.0) (1.5.3)\n",
386 | "Collecting xxhash (from datasets->coati==1.0.0)\n",
387 | " Downloading xxhash-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
388 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.5/212.5 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
389 | "\u001b[?25hCollecting multiprocess (from datasets->coati==1.0.0)\n",
390 | " Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)\n",
391 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
392 | "\u001b[?25hRequirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets->coati==1.0.0) (2023.4.0)\n",
393 | "Collecting aiohttp (from datasets->coati==1.0.0)\n",
394 | " Downloading aiohttp-3.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)\n",
395 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m28.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
396 | "\u001b[?25hCollecting responses<0.19 (from datasets->coati==1.0.0)\n",
397 | " Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
398 | "Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from fastapi->coati==1.0.0) (1.10.7)\n",
399 | "Collecting starlette<0.28.0,>=0.27.0 (from fastapi->coati==1.0.0)\n",
400 | " Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
401 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
402 | "\u001b[?25hCollecting nvidia-ml-py>=11.450.129 (from gpustat->coati==1.0.0)\n",
403 | " Downloading nvidia_ml_py-11.525.112-py3-none-any.whl (35 kB)\n",
404 | "Collecting blessed>=1.17.1 (from gpustat->coati==1.0.0)\n",
405 | " Downloading blessed-1.20.0-py2.py3-none-any.whl (58 kB)\n",
406 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.4/58.4 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
407 | "\u001b[?25hRequirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain->coati==1.0.0) (2.0.10)\n",
408 | "Collecting async-timeout<5.0.0,>=4.0.0 (from langchain->coati==1.0.0)\n",
409 | " Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
410 | "Collecting dataclasses-json<0.6.0,>=0.5.7 (from langchain->coati==1.0.0)\n",
411 | " Downloading dataclasses_json-0.5.7-py3-none-any.whl (25 kB)\n",
412 | "Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /usr/local/lib/python3.10/dist-packages (from langchain->coati==1.0.0) (2.8.4)\n",
413 | "Collecting openapi-schema-pydantic<2.0,>=1.2 (from langchain->coati==1.0.0)\n",
414 | " Downloading openapi_schema_pydantic-1.2.4-py3-none-any.whl (90 kB)\n",
415 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
416 | "\u001b[?25hRequirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain->coati==1.0.0) (8.2.2)\n",
417 | "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb->coati==1.0.0)\n",
418 | " Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n",
419 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m21.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
420 | "\u001b[?25hCollecting sentry-sdk>=1.0.0 (from wandb->coati==1.0.0)\n",
421 | " Downloading sentry_sdk-1.24.0-py2.py3-none-any.whl (206 kB)\n",
422 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m206.5/206.5 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
423 | "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb->coati==1.0.0)\n",
424 | " Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
425 | "Collecting pathtools (from wandb->coati==1.0.0)\n",
426 | " Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
427 | " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
428 | "Collecting setproctitle (from wandb->coati==1.0.0)\n",
429 | " Downloading setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
430 | "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->coati==1.0.0) (1.4.4)\n",
431 | "Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb->coati==1.0.0) (3.20.3)\n",
432 | "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->coati==1.0.0) (23.1.0)\n",
433 | "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->coati==1.0.0) (2.0.12)\n",
434 | "Collecting multidict<7.0,>=4.5 (from aiohttp->datasets->coati==1.0.0)\n",
435 | " Downloading multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
436 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
437 | "\u001b[?25hCollecting yarl<2.0,>=1.0 (from aiohttp->datasets->coati==1.0.0)\n",
438 | " Downloading yarl-1.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (268 kB)\n",
439 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m26.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
440 | "\u001b[?25hCollecting frozenlist>=1.1.1 (from aiohttp->datasets->coati==1.0.0)\n",
441 | " Downloading frozenlist-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (149 kB)\n",
442 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m149.6/149.6 kB\u001b[0m \u001b[31m19.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
443 | "\u001b[?25hCollecting aiosignal>=1.1.2 (from aiohttp->datasets->coati==1.0.0)\n",
444 | " Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
445 | "Requirement already satisfied: wcwidth>=0.1.4 in /usr/local/lib/python3.10/dist-packages (from blessed>=1.17.1->gpustat->coati==1.0.0) (0.2.6)\n",
446 | "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from blessed>=1.17.1->gpustat->coati==1.0.0) (1.16.0)\n",
447 | "Collecting marshmallow<4.0.0,>=3.3.0 (from dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
448 | " Downloading marshmallow-3.19.0-py3-none-any.whl (49 kB)\n",
449 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.1/49.1 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
450 | "\u001b[?25hCollecting marshmallow-enum<2.0.0,>=1.5.1 (from dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
451 | " Downloading marshmallow_enum-1.5.1-py2.py3-none-any.whl (4.2 kB)\n",
452 | "Collecting typing-inspect>=0.4.0 (from dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
453 | " Downloading typing_inspect-0.8.0-py3-none-any.whl (8.7 kB)\n",
454 | "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb->coati==1.0.0)\n",
455 | " Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
456 | "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
457 | "\u001b[?25hRequirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.20.1->coati==1.0.0) (1.26.15)\n",
458 | "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.20.1->coati==1.0.0) (2022.12.7)\n",
459 | "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.20.1->coati==1.0.0) (3.4)\n",
460 | "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain->coati==1.0.0) (2.0.2)\n",
461 | "Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from starlette<0.28.0,>=0.27.0->fastapi->coati==1.0.0) (3.6.2)\n",
462 | "Requirement already satisfied: invoke>=2.0 in /usr/local/lib/python3.10/dist-packages (from fabric->colossalai>=0.2.4->coati==1.0.0) (2.1.2)\n",
463 | "Requirement already satisfied: paramiko>=2.4 in /usr/local/lib/python3.10/dist-packages (from fabric->colossalai>=0.2.4->coati==1.0.0) (3.1.0)\n",
464 | "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->coati==1.0.0) (2.8.2)\n",
465 | "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->coati==1.0.0) (2022.7.1)\n",
466 | "Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (3.3.1)\n",
467 | "Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (2.5.24)\n",
468 | "Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (1.8.0)\n",
469 | "Requirement already satisfied: virtualenv>=20.10.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->colossalai>=0.2.4->coati==1.0.0) (20.23.0)\n",
470 | "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai>=0.2.4->coati==1.0.0) (2.2.0)\n",
471 | "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->colossalai>=0.2.4->coati==1.0.0) (2.14.0)\n",
472 | "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi->coati==1.0.0) (1.3.0)\n",
473 | "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->coati==1.0.0)\n",
474 | " Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
475 | "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->colossalai>=0.2.4->coati==1.0.0) (0.1.2)\n",
476 | "Requirement already satisfied: bcrypt>=3.2 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (4.0.1)\n",
477 | "Requirement already satisfied: cryptography>=3.3 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (40.0.2)\n",
478 | "Requirement already satisfied: pynacl>=1.5 in /usr/local/lib/python3.10/dist-packages (from paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (1.5.0)\n",
479 | "Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain->coati==1.0.0)\n",
480 | " Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
481 | "Requirement already satisfied: distlib<1,>=0.3.6 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->colossalai>=0.2.4->coati==1.0.0) (0.3.6)\n",
482 | "Requirement already satisfied: platformdirs<4,>=3.2 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->colossalai>=0.2.4->coati==1.0.0) (3.3.0)\n",
483 | "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/dist-packages (from cryptography>=3.3->paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (1.15.1)\n",
484 | "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.12->cryptography>=3.3->paramiko>=2.4->fabric->colossalai>=0.2.4->coati==1.0.0) (2.21)\n",
485 | "Building wheels for collected packages: coati, gpustat, pathtools\n",
486 | " Building wheel for coati (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
487 | " Created wheel for coati: filename=coati-1.0.0-py3-none-any.whl size=75334 sha256=13d5d09bdbb4b2b47045312ca337e68b88dfd2044087fbaf0c05bd593c1b4a35\n",
488 | " Stored in directory: /tmp/pip-ephem-wheel-cache-tez1zypt/wheels/49/ba/eb/98b39707d3bcca1d3ecf646b531cdb25f480bd44ec5c0edafb\n",
489 | " Building wheel for gpustat (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
490 | " Created wheel for gpustat: filename=gpustat-1.1-py3-none-any.whl size=26280 sha256=a748ad4da7967293e3504c25de991a95a408f0d9a9caf2a91b7d4e09b0ee58fd\n",
491 | " Stored in directory: /root/.cache/pip/wheels/ee/d0/2c/1e02440645c2318ba03aea99993a44a9108dc8f74de0bd370b\n",
492 | " Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
493 | " Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=48ee994acbcb9e407204168f303fd9c252c0f7f64f4948b689f062e60891e7e3\n",
494 | " Stored in directory: /root/.cache/pip/wheels/e7/f3/22/152153d6eb222ee7a56ff8617d80ee5207207a8c00a7aab794\n",
495 | "Successfully built coati gpustat pathtools\n",
496 | "Installing collected packages: sentencepiece, pathtools, nvidia-ml-py, xxhash, smmap, setproctitle, sentry-sdk, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, mypy-extensions, multidict, marshmallow, loralib, frozenlist, docker-pycreds, dill, blessed, async-timeout, yarl, typing-inspect, starlette, responses, openapi-schema-pydantic, nvidia-cudnn-cu11, multiprocess, marshmallow-enum, gpustat, gitdb, aiosignal, torch, sse_starlette, GitPython, fastapi, dataclasses-json, aiohttp, wandb, langchain, datasets, coati\n",
497 | " Attempting uninstall: torch\n",
498 | " Found existing installation: torch 2.0.1+cu118\n",
499 | " Uninstalling torch-2.0.1+cu118:\n",
500 | " Successfully uninstalled torch-2.0.1+cu118\n",
501 | "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
502 | "torchaudio 2.0.2+cu118 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\n",
503 | "torchdata 0.6.1 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\n",
504 | "torchtext 0.15.2 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\n",
505 | "torchvision 0.15.2+cu118 requires torch==2.0.1, but you have torch 1.13.1 which is incompatible.\u001b[0m\u001b[31m\n",
506 | "\u001b[0mSuccessfully installed GitPython-3.1.31 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 blessed-1.20.0 coati-1.0.0 dataclasses-json-0.5.7 datasets-2.12.0 dill-0.3.6 docker-pycreds-0.4.0 fastapi-0.95.2 frozenlist-1.3.3 gitdb-4.0.10 gpustat-1.1 langchain-0.0.178 loralib-0.1.1 marshmallow-3.19.0 marshmallow-enum-1.5.1 multidict-6.0.4 multiprocess-0.70.14 mypy-extensions-1.0.0 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-ml-py-11.525.112 openapi-schema-pydantic-1.2.4 pathtools-0.1.2 responses-0.18.0 sentencepiece-0.1.99 sentry-sdk-1.24.0 setproctitle-1.3.2 smmap-5.0.0 sse_starlette-1.6.1 starlette-0.27.0 torch-1.13.1 typing-inspect-0.8.0 wandb-0.15.3 xxhash-3.2.0 yarl-1.9.2\n"
507 | ]
508 | }
509 | ]
510 | },
511 | {
512 | "cell_type": "markdown",
513 | "source": [
514 | "## 预训练模型的下载（以bloom为例）"
515 | ],
516 | "metadata": {
517 | "id": "qbtHtV4pHXQ4"
518 | }
519 | },
520 | {
521 | "cell_type": "code",
522 | "source": [
523 | "os.chdir('/content/ColossalAI/applications/Chat/examples')"
524 | ],
525 | "metadata": {
526 | "id": "WxRdvkAvhlJW"
527 | },
528 | "execution_count": null,
529 | "outputs": []
530 | },
531 | {
532 | "cell_type": "markdown",
533 | "source": [
534 | "如果不安装下面的命令会出现什么情况呢？\n",
535 | "- 从huggingface中git clone下来的模型看似下载下来了，但是其实下载下来的并不是实质的模型文件（如果你检查文件的大小，只有几B）\n",
536 | "- 一旦下载下来的文件并不是实质的模型，则在运行SFT代码的时候会报错：**safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge**"
537 | ],
538 | "metadata": {
539 | "id": "GdlFgQzmHo2t"
540 | }
541 | },
542 | {
543 | "cell_type": "code",
544 | "source": [
545 | "!sudo apt-get install git-lfs\n",
546 | "!git lfs install"
547 | ],
548 | "metadata": {
549 | "colab": {
550 | "base_uri": "https://localhost:8080/"
551 | },
552 | "id": "C4Pu4RmJiEK6",
553 | "outputId": "e49226b3-fd13-4f48-9f41-43f23fa017d8"
554 | },
555 | "execution_count": null,
556 | "outputs": [
557 | {
558 | "output_type": "stream",
559 | "name": "stdout",
560 | "text": [
561 | "Reading package lists... Done\n",
562 | "Building dependency tree \n",
563 | "Reading state information... Done\n",
564 | "git-lfs is already the newest version (2.9.2-1).\n",
565 | "0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.\n",
566 | "Updated git hooks.\n",
567 | "Git LFS initialized.\n"
568 | ]
569 | }
570 | ]
571 | },
572 | {
573 | "cell_type": "markdown",
574 | "source": [
575 | "**下载ColossalAI支持的系列模型**，我们以bloomz-560m为例。在下面，我们将模型放在了当前的目录下（ColossalAI/applications/Chat/examples），但是这里并不是强制的，可以根据自己的喜欢变换位置。"
576 | ],
577 | "metadata": {
578 | "id": "9JeE6R7QH4dJ"
579 | }
580 | },
581 | {
582 | "cell_type": "code",
583 | "source": [
584 | "!git clone https://huggingface.co/bigscience/bloomz-560m"
585 | ],
586 | "metadata": {
587 | "colab": {
588 | "base_uri": "https://localhost:8080/"
589 | },
590 | "id": "3xUK88K7iydi",
591 | "outputId": "90626fc0-dd6c-46d9-d68b-08289ae034f6"
592 | },
593 | "execution_count": null,
594 | "outputs": [
595 | {
596 | "output_type": "stream",
597 | "name": "stdout",
598 | "text": [
599 | "Cloning into 'bloomz-560m'...\n",
600 | "remote: Enumerating objects: 1332, done.\u001b[K\n",
601 | "remote: Counting objects: 100% (10/10), done.\u001b[K\n",
602 | "remote: Compressing objects: 100% (7/7), done.\u001b[K\n",
603 | "remote: Total 1332 (delta 3), reused 10 (delta 3), pack-reused 1322\u001b[K\n",
604 | "Receiving objects: 100% (1332/1332), 7.18 MiB | 22.90 MiB/s, done.\n",
605 | "Resolving deltas: 100% (616/616), done.\n",
606 | "Filtering content: 100% (8/8), 2.11 GiB | 57.58 MiB/s, done.\n"
607 | ]
608 | }
609 | ]
610 | },
611 | {
612 | "cell_type": "markdown",
613 | "source": [
614 | "## 运行SFT"
615 | ],
616 | "metadata": {
617 | "id": "wnZ6xjQfIEpL"
618 | }
619 | },
620 | {
621 | "cell_type": "markdown",
622 | "source": [
623 | "我们这里是直接运行的py文件，如果你按照[文档的说明](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)，运行sh脚本文件（!bash train_sft.sh）也是可以的。其本质上，都是运行这个train_sft.py文件。\n",
624 | "\n",
625 | "**训练完成的模型存放地址（save_path）**这里写的示例路径是临时路径，一旦notebook停止运行，这个路径就失效了。所以要么在notebook停止前先下载训练好的模型，要么这里可以考虑填写到google drive的目录下。\n",
626 | "\n",
627 | "在下面的命令中，为了演示\n",
628 | "- 我们只是用一个非常非常非常小的数据集（--dataset）去跑程序（小到就只有5条数据）\n",
629 | "- model：改为了“bloom”\n",
630 | "- pretrain：改成了我们自己下载的模型地址\n",
631 | "- save_path: 改成了我们想放的目录地址\n",
632 | "- 其他参数可能需要你自己去多多探索：比如lora、gradient checkingpoint等。更多参数的说明见[官方文档](https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat/examples)"
633 | ],
634 | "metadata": {
635 | "id": "9w6QPTmIIr8J"
636 | }
637 | },
638 | {
639 | "cell_type": "code",
640 | "source": [
641 | "!torchrun --standalone --nproc_per_node=1 train_sft.py \\\n",
642 | " --pretrain \"./bloomz-560m\" \\\n",
643 | " --model 'bloom' \\\n",
644 | " --strategy colossalai_zero2 \\\n",
645 | " --log_interval 50 \\\n",
646 | " --save_path \"./bloomz-560m-finetuned\" \\\n",
647 | " --dataset \"./instinwild_ch_small.json\" \\\n",
648 | " --batch_size 4 \\\n",
649 | " --accumulation_steps 8 \\\n",
650 | " --lr 2e-5 \\\n",
651 | " --max_datasets_size 512 \\\n",
652 | " --max_epochs 1"
653 | ],
654 | "metadata": {
655 | "colab": {
656 | "base_uri": "https://localhost:8080/"
657 | },
658 | "id": "WVu629ZLjf6G",
659 | "outputId": "6fd1ef2f-6483-42a4-b43a-65f9181bac2c"
660 | },
661 | "execution_count": null,
662 | "outputs": [
663 | {
664 | "output_type": "stream",
665 | "name": "stdout",
666 | "text": [
667 | "2023-05-24 11:46:01.994320: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
668 | "\u001b[2;36m[05/24/23 11:46:05]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
669 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/colossalai/\u001b[0m\n",
670 | "\u001b[2;36m \u001b[0m \u001b[35mcontext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m522\u001b[0m set_device \n",
671 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: process rank \u001b[1;36m0\u001b[0m is \n",
672 | "\u001b[2;36m \u001b[0m bound to device \u001b[1;36m0\u001b[0m \n",
673 | "\u001b[2;36m[05/24/23 11:46:13]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
674 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/colossalai/\u001b[0m\n",
675 | "\u001b[2;36m \u001b[0m \u001b[35mcontext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m558\u001b[0m set_seed \n",
676 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: initialized seed on\n",
677 | "\u001b[2;36m \u001b[0m rank \u001b[1;36m0\u001b[0m, numpy: \u001b[1;36m42\u001b[0m, python random: \u001b[1;36m42\u001b[0m, \n",
678 | "\u001b[2;36m \u001b[0m ParallelMode.DATA: \u001b[1;36m42\u001b[0m, ParallelMode.TENSOR: \u001b[1;36m42\u001b[0m,the \n",
679 | "\u001b[2;36m \u001b[0m default parallel seed is ParallelMode.DATA. \n",
680 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
681 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/colossalai/\u001b[0m\n",
682 | "\u001b[2;36m \u001b[0m \u001b[95minitialize.py\u001b[0m:\u001b[1;36m115\u001b[0m launch \n",
683 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Distributed \n",
684 | "\u001b[2;36m \u001b[0m environment is initialized, data parallel size: \u001b[1;36m1\u001b[0m, \n",
685 | "\u001b[2;36m \u001b[0m pipeline parallel size: \u001b[1;36m1\u001b[0m, tensor parallel size: \u001b[1;36m1\u001b[0m \n",
686 | "/usr/local/lib/python3.10/dist-packages/colossalai/kernel/op_builder/utils.py:94: UserWarning: [extension] The CUDA version on the system (11.8) does not match with the version (11.7) torch was compiled with. The mismatch is found in the minor version. As the APIs are compatible, we will allow compilation to proceed. If you encounter any issue when using the built kernel, please try to build it again with fully matched CUDA versions\n",
687 | " warnings.warn(\n",
688 | "/usr/local/lib/python3.10/dist-packages/colossalai/kernel/op_builder/utils.py:94: UserWarning: [extension] The CUDA version on the system (11.8) does not match with the version (11.7) torch was compiled with. The mismatch is found in the minor version. As the APIs are compatible, we will allow compilation to proceed. If you encounter any issue when using the built kernel, please try to build it again with fully matched CUDA versions\n",
689 | " warnings.warn(\n",
690 | "\u001b[2;36m[05/24/23 11:52:04]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
691 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
692 | "\u001b[2;36m \u001b[0m \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m121\u001b[0m __init__ \n",
693 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loading data\u001b[33m...\u001b[0m \n",
694 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
695 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
696 | "\u001b[2;36m \u001b[0m \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m123\u001b[0m __init__ \n",
697 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loaded \u001b[1;36m6\u001b[0m examples. \n",
698 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
699 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
700 | "\u001b[2;36m \u001b[0m \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m126\u001b[0m __init__ \n",
701 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Limiting dataset to\n",
702 | "\u001b[2;36m \u001b[0m \u001b[1;36m512\u001b[0m examples. \n",
703 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
704 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
705 | "\u001b[2;36m \u001b[0m \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m129\u001b[0m __init__ \n",
706 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Formatting \n",
707 | "\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m \n",
708 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
709 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/datas\u001b[0m\n",
710 | "\u001b[2;36m \u001b[0m \u001b[35met/\u001b[0m\u001b[95msft_dataset.py\u001b[0m:\u001b[1;36m137\u001b[0m __init__ \n",
711 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Tokenizing \n",
712 | "\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m This may take some time\u001b[33m...\u001b[0m \n",
713 | "steps: 0it [00:00, ?it/s]\u001b[2;36m[05/24/23 11:52:09]\u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: \n",
714 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/train\u001b[0m\n",
715 | "\u001b[2;36m \u001b[0m \u001b[35mer/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m86\u001b[0m fit \n",
716 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: batch_i\u001b[1;92md:0\u001b[0m, \n",
717 | "\u001b[2;36m \u001b[0m abnormal loss: \u001b[1;36m3.6484375\u001b[0m \n",
718 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: \n",
719 | "\u001b[2;36m \u001b[0m \u001b[35m/usr/local/lib/python3.10/dist-packages/coati/train\u001b[0m\n",
720 | "\u001b[2;36m \u001b[0m \u001b[35mer/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m86\u001b[0m fit \n",
721 | "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m colossalai - colossalai - WARNING: batch_i\u001b[1;92md:1\u001b[0m, \n",
722 | "\u001b[2;36m \u001b[0m abnormal loss: \u001b[1;36m4.109375\u001b[0m \n",
723 | "steps: 0it [00:03, ?it/s]\n"
724 | ]
725 | }
726 | ]
727 | },
728 | {
729 | "cell_type": "markdown",
730 | "source": [
731 | "## 小结\n",
732 | "仅仅跑起来只是一个开始。祝愿每个小伙伴最终都会获得更好的GPU资源、更棒的数据和更出色的属于自己的语言模型！"
733 | ],
734 | "metadata": {
735 | "id": "RQKDNZogLUNs"
736 | }
737 | },
738 | {
739 | "cell_type": "code",
740 | "source": [],
741 | "metadata": {
742 | "id": "H7HM6E5VLW2u"
743 | },
744 | "execution_count": null,
745 | "outputs": []
746 | }
747 | ]
748 | }
749 |
--------------------------------------------------------------------------------
/千“垂”百炼 - 【医疗&健康】 ChatDoctor（上）.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/createmomo/Open-Source-Language-Model-Pocket/5e89e7703ab7e6e7d01dd496847e08eb0ad27b83/千“垂”百炼 - 【医疗&健康】 ChatDoctor（上）.pdf
--------------------------------------------------------------------------------
/千“垂”百炼 - 【医疗&健康】 ChatDoctor（下）.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/createmomo/Open-Source-Language-Model-Pocket/5e89e7703ab7e6e7d01dd496847e08eb0ad27b83/千“垂”百炼 - 【医疗&健康】 ChatDoctor（下）.pdf
--------------------------------------------------------------------------------
/千“垂”百炼 - 【医疗&健康】 ChatDoctor（中）.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/createmomo/Open-Source-Language-Model-Pocket/5e89e7703ab7e6e7d01dd496847e08eb0ad27b83/千“垂”百炼 - 【医疗&健康】 ChatDoctor（中）.pdf
--------------------------------------------------------------------------------
/千“垂”百炼 - 【医疗&健康】 MING.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/createmomo/Open-Source-Language-Model-Pocket/5e89e7703ab7e6e7d01dd496847e08eb0ad27b83/千“垂”百炼 - 【医疗&健康】 MING.pdf
--------------------------------------------------------------------------------
/千“垂”百炼 - 【医疗&健康】 MedicalGPT-zh.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/createmomo/Open-Source-Language-Model-Pocket/5e89e7703ab7e6e7d01dd496847e08eb0ad27b83/千“垂”百炼 - 【医疗&健康】 MedicalGPT-zh.pdf
--------------------------------------------------------------------------------
/千“垂”百炼 - 【医疗&健康】 SoulChat.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/createmomo/Open-Source-Language-Model-Pocket/5e89e7703ab7e6e7d01dd496847e08eb0ad27b83/千“垂”百炼 - 【医疗&健康】 SoulChat.pdf
--------------------------------------------------------------------------------