├── inferless-runtime-config.yaml ├── inferless.yaml ├── input_schema.py ├── app.py └── README.md /inferless-runtime-config.yaml: -------------------------------------------------------------------------------- 1 | build: 2 | cuda_version: "12.1.1" 3 | python_packages: 4 | - vllm==0.6.6.post1 5 | - hf-transfer==0.1.9 6 | - huggingface-hub==0.27.1 7 | -------------------------------------------------------------------------------- /inferless.yaml: -------------------------------------------------------------------------------- 1 | # Inferless config file (version: 2.0.0) 2 | version: 2.0.0 3 | 4 | name: deepseek-r1-distill-qwen-32b 5 | import_source: LOCAL 6 | 7 | # you can choose the options between ONNX, TENSORFLOW, PYTORCH, Only applicable for FILE type import. 8 | source_framework_type: PYTORCH 9 | -------------------------------------------------------------------------------- /input_schema.py: -------------------------------------------------------------------------------- 1 | INPUT_SCHEMA = { 2 | "prompt": { 3 | 'datatype': 'STRING', 4 | 'required': True, 5 | 'shape': [1], 6 | 'example': ["What is deep learning?"] 7 | }, 8 | "temperature": { 9 | 'datatype': 'FP32', 10 | 'required': False, 11 | 'shape': [1], 12 | 'example': [0.7] 13 | }, 14 | "top_p": { 15 | 'datatype': 'FP32', 16 | 'required': False, 17 | 'shape': [1], 18 | 'example': [0.1] 19 | }, 20 | "repetition_penalty": { 21 | 'datatype': 'FP32', 22 | 'required': False, 23 | 'shape': [1], 24 | 'example': [1.18] 25 | }, 26 | "max_tokens": { 27 | 'datatype': 'INT16', 28 | 'required': False, 29 | 'shape': [1], 30 | 'example': [256] 31 | }, 32 | "top_k":{ 33 | 'datatype': 'INT8', 34 | 'required': False, 35 | 'shape': [1], 36 | 'example': [40] 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- 1 | from vllm import LLM 2 | from vllm.sampling_params import SamplingParams 3 | from transformers import AutoTokenizer 4 | 5 | class InferlessPythonModel: 6 | def initialize(self): 7 | model_id = "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" 8 | self.llm = LLM(model=model_id,gpu_memory_utilization=0.9,max_model_len=5000,dtype="float16") 9 | self.tokenizer = AutoTokenizer.from_pretrained(model_id) 10 | 11 | def infer(self, inputs): 12 | prompts = inputs["prompt"] 13 | temperature = inputs.get("temperature",0.7) 14 | top_p = inputs.get("top_p",0.1) 15 | repetition_penalty = inputs.get("repetition_penalty",1.18) 16 | top_k = int(inputs.get("top_k",40)) 17 | max_tokens = inputs.get("max_tokens",256) 18 | 19 | sampling_params = SamplingParams(temperature=temperature,top_p=top_p, 20 | repetition_penalty=repetition_penalty, 21 | top_k=top_k,max_tokens=max_tokens 22 | ) 23 | input_text = self.tokenizer.apply_chat_template([{"role": "user", "content": prompts}], tokenize=False) 24 | result = self.llm.generate(input_text, sampling_params) 25 | result_output = [output.outputs[0].text for output in result] 26 | 27 | return {"generated_text":result_output[0]} 28 | 29 | def finalize(self): 30 | self.llm = None 31 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tutorial - Deploy DeepSeek-R1-Distill-Qwen-32B using Inferless 2 | [DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B) is a distilled variant within the DeepSeek-R1 series. The dataset used for training is meticulously curated from the DeepSeek-R1 model, with Qwen2.5-32B serving as the foundational base model. This model has undergone supervised fine-tuning to achieve enhanced performance and efficiency. 3 | 4 | ## TL;DR: 5 | - Deployment of DeepSeek-R1-Distill-Qwen-32B model using [vLLM](https://github.com/vllm-project/vllm). 6 | - You can expect an average tokens/sec of `21.95` and a latency of `5.88` sec for generating a text of `128` tokens. This setup has an average cold start time of `39.95` sec. 7 | - Dependencies defined in `inferless-runtime-config.yaml`. 8 | - GitHub/GitLab template creation with `app.py`, `inferless-runtime-config.yaml` and `inferless.yaml`. 9 | - `InferlessPythonModel` class in `app.py` with `initialize`, `infer`, and `finalize` functions. 10 | - Custom runtime creation with necessary system and Python packages. 11 | - Recommended GPU: NVIDIA A100 for optimal performance. 12 | - Custom runtime selection in advanced configuration. 13 | - Final review and deployment on the Inferless platform. 14 | 15 | ### Fork the Repository 16 | Get started by forking the repository. You can do this by clicking on the fork button in the top right corner of the repository page. 17 | 18 | This will create a copy of the repository in your own GitHub account, allowing you to make changes and customize it according to your needs. 19 | 20 | ### Create a Custom Runtime in Inferless 21 | To access the custom runtime window in Inferless, simply navigate to the sidebar and click on the Create new Runtime button. A pop-up will appear. 22 | 23 | Next, provide a suitable name for your custom runtime and proceed by uploading the **inferless-runtime-config.yaml** file given above. Finally, ensure you save your changes by clicking on the save button. 24 | 25 | ### Import the Model in Inferless 26 | Log in to your inferless account, select the workspace you want the model to be imported into and click the `Add a custom model` button. 27 | 28 | - Select `Github` as the method of upload from the Provider list and then select your Github Repository and the branch. 29 | - Choose the type of machine, and specify the minimum and maximum number of replicas for deploying your model. 30 | - Configure Custom Runtime ( If you have pip or apt packages), choose Volume, Secrets and set Environment variables like Inference Timeout / Container Concurrency / Scale Down Timeout 31 | - Once you click “Continue,” click Deploy to start the model import process. 32 | 33 | Enter all the required details to Import your model. Refer [this link](https://docs.inferless.com/integrations/git-custom-code/git--custom-code) for more information on model import. 34 | 35 | --- 36 | ## Curl Command 37 | Following is an example of the curl command you can use to make inference. You can find the exact curl command in the Model's API page in Inferless. 38 | ```bash 39 | curl --location '' \ 40 | --header 'Content-Type: application/json' \ 41 | --header 'Authorization: Bearer ' \ 42 | --data '{ 43 | "inputs": [ 44 | { 45 | "name": "prompt", 46 | "shape": [ 47 | 1 48 | ], 49 | "data": [ 50 | "Explain Deep Learning." 51 | ], 52 | "datatype": "BYTES" 53 | }, 54 | { 55 | "name": "temperature", 56 | "optional": true, 57 | "shape": [ 58 | 1 59 | ], 60 | "data": [ 61 | 0.7 62 | ], 63 | "datatype": "FP64" 64 | }, 65 | { 66 | "name": "top_p", 67 | "optional": true, 68 | "shape": [ 69 | 1 70 | ], 71 | "data": [ 72 | 0.1 73 | ], 74 | "datatype": "FP64" 75 | }, 76 | { 77 | "name": "repetition_penalty", 78 | "optional": true, 79 | "shape": [ 80 | 1 81 | ], 82 | "data": [ 83 | 1.18 84 | ], 85 | "datatype": "FP64" 86 | }, 87 | { 88 | "name": "top_k", 89 | "optional": true, 90 | "shape": [ 91 | 1 92 | ], 93 | "data": [ 94 | 40 95 | ], 96 | "datatype": "INT32" 97 | }, 98 | { 99 | "name": "max_tokens", 100 | "optional": true, 101 | "shape": [ 102 | 1 103 | ], 104 | "data": [ 105 | 256 106 | ], 107 | "datatype": "INT32" 108 | } 109 | ] 110 | }' 111 | ``` 112 | 113 | --- 114 | ## Customizing the Code 115 | Open the `app.py` file. This contains the main code for inference. The `InferlessPythonModel` has three main functions, initialize, infer and finalize. 116 | 117 | **Initialize** - This function is executed during the cold start and is used to initialize the model. If you have any custom configurations or settings that need to be applied during the initialization, make sure to add them in this function. 118 | 119 | **Infer** - This function is where the inference happens. The infer function leverages both RequestObjects and ResponseObjects to handle inputs and outputs in a structured and maintainable way. 120 | - RequestObjects: Defines the input schema, validating and parsing the input data. 121 | - ResponseObjects: Encapsulates the output data, ensuring consistent and structured API responses. 122 | 123 | ```python 124 | def infer(self, request: RequestObjects) -> ResponseObjects: 125 | sampling_params = SamplingParams(temperature=request.temperature,top_p=request.top_p,repetition_penalty=request.repetition_penalty, 126 | top_k=request.top_k,max_tokens=request.max_tokens) 127 | input_text = self.tokenizer.apply_chat_template([{"role": "user", "content": request.prompt}], tokenize=False) 128 | result = self.llm.generate(input_text, sampling_params) 129 | result_output = [output.outputs[0].text for output in result] 130 | 131 | generateObject = ResponseObjects(generated_text = result_output[0]) 132 | return generateObject 133 | ``` 134 | 135 | **Finalize** - This function is used to perform any cleanup activity for example you can unload the model from the gpu by setting to `None`. 136 | ```python 137 | def finalize(self): 138 | self.llm = None 139 | ``` 140 | 141 | 142 | For more information refer to the [Inferless docs](https://docs.inferless.com/). 143 | --------------------------------------------------------------------------------