/v1 \
21 | --restart always \
22 | ghcr.io/open-webui/open-webui:main
23 | ```
24 |
25 | 1. Open it in the browser:
26 |
27 | On the top of the web page, you can see the model `qwen/Qwen1.5-0.5B-Chat`.
28 |
29 | 
30 |
--------------------------------------------------------------------------------
/docs/deployment/frameworks/triton.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: NVIDIA Triton
3 | ---
4 | [](){ #deployment-triton }
5 |
6 | The [Triton Inference Server](https://github.com/triton-inference-server) hosts a tutorial demonstrating how to quickly deploy a simple [facebook/opt-125m](https://huggingface.co/facebook/opt-125m) model using vLLM. Please see [Deploying a vLLM model in Triton](https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton) for more details.
7 |
--------------------------------------------------------------------------------
/docs/deployment/integrations/kserve.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: KServe
3 | ---
4 | [](){ #deployment-kserve }
5 |
6 | vLLM can be deployed with [KServe](https://github.com/kserve/kserve) on Kubernetes for highly scalable distributed model serving.
7 |
8 | Please see [this guide](https://kserve.github.io/website/latest/modelserving/v1beta1/llm/huggingface/) for more details on using vLLM with KServe.
9 |
--------------------------------------------------------------------------------
/docs/deployment/integrations/kubeai.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: KubeAI
3 | ---
4 | [](){ #deployment-kubeai }
5 |
6 | [KubeAI](https://github.com/substratusai/kubeai) is a Kubernetes operator that enables you to deploy and manage AI models on Kubernetes. It provides a simple and scalable way to deploy vLLM in production. Functionality such as scale-from-zero, load based autoscaling, model caching, and much more is provided out of the box with zero external dependencies.
7 |
8 | Please see the Installation Guides for environment specific instructions:
9 |
10 | - [Any Kubernetes Cluster](https://www.kubeai.org/installation/any/)
11 | - [EKS](https://www.kubeai.org/installation/eks/)
12 | - [GKE](https://www.kubeai.org/installation/gke/)
13 |
14 | Once you have KubeAI installed, you can
15 | [configure text generation models](https://www.kubeai.org/how-to/configure-text-generation-models/)
16 | using vLLM.
17 |
--------------------------------------------------------------------------------
/docs/deployment/integrations/llmaz.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: llmaz
3 | ---
4 | [](){ #deployment-llmaz }
5 |
6 | [llmaz](https://github.com/InftyAI/llmaz) is an easy-to-use and advanced inference platform for large language models on Kubernetes, aimed for production use. It uses vLLM as the default model serving backend.
7 |
8 | Please refer to the [Quick Start](https://github.com/InftyAI/llmaz?tab=readme-ov-file#quick-start) for more details.
9 |
--------------------------------------------------------------------------------
/docs/features/quantization/README.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Quantization
3 | ---
4 | [](){ #quantization-index }
5 |
6 | Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
7 |
8 | Contents:
9 |
10 | - [Supported_Hardware](supported_hardware.md)
11 | - [Auto_Awq](auto_awq.md)
12 | - [Bnb](bnb.md)
13 | - [Bitblas](bitblas.md)
14 | - [Gguf](gguf.md)
15 | - [Gptqmodel](gptqmodel.md)
16 | - [Int4](int4.md)
17 | - [Int8](int8.md)
18 | - [Fp8](fp8.md)
19 | - [Modelopt](modelopt.md)
20 | - [Quark](quark.md)
21 | - [Quantized_Kvcache](quantized_kvcache.md)
22 | - [Torchao](torchao.md)
23 |
--------------------------------------------------------------------------------
/docs/getting_started/installation/.nav.yml:
--------------------------------------------------------------------------------
1 | nav:
2 | - README.md
3 | - gpu.md
4 | - cpu.md
5 | - ai_accelerator.md
--------------------------------------------------------------------------------
/docs/getting_started/installation/README.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Installation
3 | ---
4 | [](){ #installation-index }
5 |
6 | vLLM supports the following hardware platforms:
7 |
8 | - [GPU](gpu.md)
9 | - [NVIDIA CUDA](gpu.md#nvidia-cuda)
10 | - [AMD ROCm](gpu.md#amd-rocm)
11 | - [Intel XPU](gpu.md#intel-xpu)
12 | - [CPU](cpu.md)
13 | - [Intel/AMD x86](cpu.md#intelamd-x86)
14 | - [ARM AArch64](cpu.md#arm-aarch64)
15 | - [Apple silicon](cpu.md#apple-silicon)
16 | - [IBM Z (S390X)](cpu.md#ibm-z-s390x)
17 | - [Other AI accelerators](ai_accelerator.md)
18 | - [Google TPU](ai_accelerator.md#google-tpu)
19 | - [Intel Gaudi](ai_accelerator.md#intel-gaudi)
20 | - [AWS Neuron](ai_accelerator.md#aws-neuron)
21 |
--------------------------------------------------------------------------------
/docs/getting_started/installation/device.template.md:
--------------------------------------------------------------------------------
1 | # Installation
2 |
3 | ## Requirements
4 |
5 | ## Set up using Python
6 |
7 | ### Pre-built wheels
8 |
9 | ### Build wheel from source
10 |
11 | ## Set up using Docker
12 |
13 | ### Pre-built images
14 |
15 | ### Build image from source
16 |
17 | ## Extra information
18 |
--------------------------------------------------------------------------------
/docs/getting_started/installation/python_env_setup.inc.md:
--------------------------------------------------------------------------------
1 | It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:
2 |
3 | ```console
4 | uv venv --python 3.12 --seed
5 | source .venv/bin/activate
6 | ```
7 |
--------------------------------------------------------------------------------
/docs/mkdocs/hooks/remove_announcement.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import os
4 | from typing import Literal
5 |
6 |
7 | def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
8 | # see https://docs.readthedocs.io/en/stable/reference/environment-variables.html # noqa
9 | if os.getenv('READTHEDOCS_VERSION_TYPE') == "tag":
10 | # remove the warning banner if the version is a tagged release
11 | docs_dir = os.path.dirname(__file__)
12 | announcement_path = os.path.join(docs_dir,
13 | "mkdocs/overrides/main.html")
14 | # The file might be removed already if the build is triggered multiple
15 | # times (readthedocs build both HTML and PDF versions separately)
16 | if os.path.exists(announcement_path):
17 | os.remove(announcement_path)
18 |
--------------------------------------------------------------------------------
/docs/mkdocs/javascript/run_llm_widget.js:
--------------------------------------------------------------------------------
1 | // Add RunLLM widget
2 | document.addEventListener("DOMContentLoaded", function () {
3 | var script = document.createElement("script");
4 | script.type = "module";
5 | script.id = "runllm-widget-script"
6 |
7 | script.src = "https://widget.runllm.com";
8 |
9 | script.setAttribute("version", "stable");
10 | script.setAttribute("runllm-keyboard-shortcut", "Mod+j"); // cmd-j or ctrl-j to open the widget.
11 | script.setAttribute("runllm-name", "vLLM");
12 | script.setAttribute("runllm-position", "BOTTOM_RIGHT");
13 | script.setAttribute("runllm-position-y", "120px");
14 | script.setAttribute("runllm-position-x", "20px");
15 | script.setAttribute("runllm-assistant-id", "207");
16 |
17 | script.async = true;
18 | document.head.appendChild(script);
19 | });
20 |
--------------------------------------------------------------------------------
/docs/mkdocs/overrides/main.html:
--------------------------------------------------------------------------------
1 | {% extends "base.html" %}
2 |
3 | {% block announce %}
4 | You are viewing the latest developer preview docs. Click here to view docs for the latest stable release.
5 | {% endblock %}
6 |
--------------------------------------------------------------------------------
/docs/models/extensions/fastsafetensor.md:
--------------------------------------------------------------------------------
1 | Loading Model weights with fastsafetensors
2 | ===================================================================
3 |
4 | Using fastsafetensors library enables loading model weights to GPU memory by leveraging GPU direct storage. See [their GitHub repository](https://github.com/foundation-model-stack/fastsafetensors) for more details.
5 | For enabling this feature, set the environment variable ``USE_FASTSAFETENSOR`` to ``true``
6 |
--------------------------------------------------------------------------------
/docs/serving/integrations/langchain.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: LangChain
3 | ---
4 | [](){ #serving-langchain }
5 |
6 | vLLM is also available via [LangChain](https://github.com/langchain-ai/langchain) .
7 |
8 | To install LangChain, run
9 |
10 | ```console
11 | pip install langchain langchain_community -q
12 | ```
13 |
14 | To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.
15 |
16 | ```python
17 | from langchain_community.llms import VLLM
18 |
19 | llm = VLLM(model="mosaicml/mpt-7b",
20 | trust_remote_code=True, # mandatory for hf models
21 | max_new_tokens=128,
22 | top_k=10,
23 | top_p=0.95,
24 | temperature=0.8,
25 | # tensor_parallel_size=... # for distributed inference
26 | )
27 |
28 | print(llm("What is the capital of France ?"))
29 | ```
30 |
31 | Please refer to this [Tutorial](https://python.langchain.com/docs/integrations/llms/vllm) for more details.
32 |
--------------------------------------------------------------------------------
/docs/serving/integrations/llamaindex.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: LlamaIndex
3 | ---
4 | [](){ #serving-llamaindex }
5 |
6 | vLLM is also available via [LlamaIndex](https://github.com/run-llama/llama_index) .
7 |
8 | To install LlamaIndex, run
9 |
10 | ```console
11 | pip install llama-index-llms-vllm -q
12 | ```
13 |
14 | To run inference on a single or multiple GPUs, use `Vllm` class from `llamaindex`.
15 |
16 | ```python
17 | from llama_index.llms.vllm import Vllm
18 |
19 | llm = Vllm(
20 | model="microsoft/Orca-2-7b",
21 | tensor_parallel_size=4,
22 | max_new_tokens=100,
23 | vllm_kwargs={"swap_space": 1, "gpu_memory_utilization": 0.5},
24 | )
25 | ```
26 |
27 | Please refer to this [Tutorial](https://docs.llamaindex.ai/en/latest/examples/llm/vllm/) for more details.
28 |
--------------------------------------------------------------------------------
/docs/training/rlhf.md:
--------------------------------------------------------------------------------
1 | # Reinforcement Learning from Human Feedback
2 |
3 | Reinforcement Learning from Human Feedback (RLHF) is a technique that fine-tunes language models using human-generated preference data to align model outputs with desired behaviors.
4 |
5 | vLLM can be used to generate the completions for RLHF. The best way to do this is with libraries like [TRL](https://github.com/huggingface/trl), [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) and [verl](https://github.com/volcengine/verl).
6 |
7 | See the following basic examples to get started if you don't want to use an existing library:
8 |
9 | - [Training and inference processes are located on separate GPUs (inspired by OpenRLHF)](../examples/offline_inference/rlhf.md)
10 | - [Training and inference processes are colocated on the same GPUs using Ray](../examples/offline_inference/rlhf_colocate.md)
11 | - [Utilities for performing RLHF with vLLM](../examples/offline_inference/rlhf_utils.md)
12 |
--------------------------------------------------------------------------------
/docs/usage/README.md:
--------------------------------------------------------------------------------
1 | # Using vLLM
2 |
3 | vLLM supports the following usage patterns:
4 |
5 | - [Inference and Serving](../serving/offline_inference.md): Run a single instance of a model.
6 | - [Deployment](../deployment/docker.md): Scale up model instances for production.
7 | - [Training](../training/rlhf.md): Train or fine-tune a model.
8 |
--------------------------------------------------------------------------------
/examples/offline_inference/disaggregated-prefill-v1/README.md:
--------------------------------------------------------------------------------
1 | # Disaggregated Prefill V1
2 |
3 | This example contains scripts that demonstrate disaggregated prefill in the offline setting of vLLM.
4 |
5 | ## Files
6 |
7 | - `run.sh` - A helper script that will run `prefill_example.py` and `decode_example.py` sequentially.
8 | - Make sure you are in the `examples/offline_inference/disaggregated-prefill-v1` directory before running `run.sh`.
9 | - `prefill_example.py` - A script which performs prefill only, saving the KV state to the `local_storage` directory and the prompts to `output.txt`.
10 | - `decode_example.py` - A script which performs decode only, loading the KV state from the `local_storage` directory and the prompts from `output.txt`.
11 |
--------------------------------------------------------------------------------
/examples/offline_inference/disaggregated-prefill-v1/run.sh:
--------------------------------------------------------------------------------
1 | rm -rf local_storage/
2 |
3 | if [ -f "output.txt" ]; then
4 | rm output.txt
5 | fi
6 |
7 | # The directory of current script
8 | SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
9 |
10 | VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 "$SCRIPT_DIR/prefill_example.py"
11 | VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 "$SCRIPT_DIR/decode_example.py"
12 |
--------------------------------------------------------------------------------
/examples/offline_inference/openai_batch/openai_example_batch.jsonl:
--------------------------------------------------------------------------------
1 | {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
2 | {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
3 |
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/.helmignore:
--------------------------------------------------------------------------------
1 | *.png
2 | .git/
3 | ct.yaml
4 | lintconf.yaml
5 | values.schema.json
6 | /workflows
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/Chart.yaml:
--------------------------------------------------------------------------------
1 | apiVersion: v2
2 | name: chart-vllm
3 | description: Chart vllm
4 |
5 | # A chart can be either an 'application' or a 'library' chart.
6 | #
7 | # Application charts are a collection of templates that can be packaged into versioned archives
8 | # to be deployed.
9 | #
10 | # Library charts provide useful utilities or functions for the chart developer. They're included as
11 | # a dependency of application charts to inject those utilities and functions into the rendering
12 | # pipeline. Library charts do not define any templates and therefore cannot be deployed.
13 | type: application
14 |
15 | # This is the chart version. This version number should be incremented each time you make changes
16 | # to the chart and its templates, including the app version.
17 | # Versions are expected to follow Semantic Versioning (https://semver.org/)
18 | version: 0.0.1
19 |
20 | maintainers:
21 | - name: mfournioux
22 |
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/ct.yaml:
--------------------------------------------------------------------------------
1 | chart-dirs:
2 | - charts
3 | validate-maintainers: false
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/templates/configmap.yaml:
--------------------------------------------------------------------------------
1 | {{- if .Values.configs -}}
2 | apiVersion: v1
3 | kind: ConfigMap
4 | metadata:
5 | name: "{{ .Release.Name }}-configs"
6 | namespace: {{ .Release.Namespace }}
7 | data:
8 | {{- with .Values.configs }}
9 | {{- toYaml . | nindent 2 }}
10 | {{- end }}
11 | {{- end -}}
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/templates/custom-objects.yaml:
--------------------------------------------------------------------------------
1 | {{- if .Values.customObjects }}
2 | {{- range .Values.customObjects }}
3 | {{- tpl (. | toYaml) $ }}
4 | ---
5 | {{- end }}
6 | {{- end }}
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/templates/poddisruptionbudget.yaml:
--------------------------------------------------------------------------------
1 | apiVersion: policy/v1
2 | kind: PodDisruptionBudget
3 | metadata:
4 | name: "{{ .Release.Name }}-pdb"
5 | namespace: {{ .Release.Namespace }}
6 | spec:
7 | maxUnavailable: {{ default 1 .Values.maxUnavailablePodDisruptionBudget }}
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/templates/pvc.yaml:
--------------------------------------------------------------------------------
1 | {{- if .Values.extraInit }}
2 | apiVersion: v1
3 | kind: PersistentVolumeClaim
4 | metadata:
5 | name: "{{ .Release.Name }}-storage-claim"
6 | namespace: {{ .Release.Namespace }}
7 | spec:
8 | accessModes:
9 | - ReadWriteOnce
10 | resources:
11 | requests:
12 | storage: {{ .Values.extraInit.pvcStorage }}
13 | {{- end }}
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/templates/secrets.yaml:
--------------------------------------------------------------------------------
1 | apiVersion: v1
2 | kind: Secret
3 | metadata:
4 | name: "{{ .Release.Name }}-secrets"
5 | namespace: {{ .Release.Namespace }}
6 | type: Opaque
7 | data:
8 | {{- range $key, $val := .Values.secrets }}
9 | {{ $key }}: {{ $val | b64enc | quote }}
10 | {{- end }}
--------------------------------------------------------------------------------
/examples/online_serving/chart-helm/templates/service.yaml:
--------------------------------------------------------------------------------
1 | apiVersion: v1
2 | kind: Service
3 | metadata:
4 | name: "{{ .Release.Name }}-service"
5 | namespace: {{ .Release.Namespace }}
6 | spec:
7 | type: ClusterIP
8 | ports:
9 | - name: {{ include "chart.service-port-name" . }}
10 | port: {{ include "chart.service-port" . }}
11 | targetPort: {{ include "chart.container-port-name" . }}
12 | protocol: TCP
13 | selector:
14 | {{- include "chart.labels" . | nindent 4 }}
--------------------------------------------------------------------------------
/examples/online_serving/disaggregated_serving/README.md:
--------------------------------------------------------------------------------
1 | # Disaggregated Serving
2 |
3 | This example contains scripts that demonstrate the disaggregated serving features of vLLM.
4 |
5 | ## Files
6 |
7 | - `disagg_proxy_demo.py` - Demonstrates XpYd (X prefill instances, Y decode instances).
8 | - `kv_events.sh` - Demonstrates KV cache event publishing.
9 |
--------------------------------------------------------------------------------
/examples/online_serving/prometheus_grafana/docker-compose.yaml:
--------------------------------------------------------------------------------
1 | # docker-compose.yaml
2 | version: "3"
3 |
4 | services:
5 | prometheus:
6 | image: prom/prometheus:latest
7 | extra_hosts:
8 | - "host.docker.internal:host-gateway" # allow a direct connection from container to the local machine
9 | ports:
10 | - "9090:9090" # the default port used by Prometheus
11 | volumes:
12 | - ${PWD}/prometheus.yaml:/etc/prometheus/prometheus.yml # mount Prometheus config file
13 |
14 | grafana:
15 | image: grafana/grafana:latest
16 | depends_on:
17 | - prometheus
18 | ports:
19 | - "3000:3000" # the default port used by Grafana
20 |
--------------------------------------------------------------------------------
/examples/online_serving/prometheus_grafana/prometheus.yaml:
--------------------------------------------------------------------------------
1 | # prometheus.yaml
2 | global:
3 | scrape_interval: 5s
4 | evaluation_interval: 30s
5 |
6 | scrape_configs:
7 | - job_name: vllm
8 | static_configs:
9 | - targets:
10 | - 'host.docker.internal:8000'
11 |
--------------------------------------------------------------------------------
/examples/online_serving/utils.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | from openai import APIConnectionError, OpenAI
4 | from openai.pagination import SyncPage
5 | from openai.types.model import Model
6 |
7 |
8 | def get_first_model(client: OpenAI) -> str:
9 | """
10 | Get the first model from the vLLM server.
11 | """
12 | try:
13 | models: SyncPage[Model] = client.models.list()
14 | except APIConnectionError as e:
15 | raise RuntimeError(
16 | "Failed to get the list of models from the vLLM server at "
17 | f"{client.base_url} with API key {client.api_key}. Check\n"
18 | "1. the server is running\n"
19 | "2. the server URL is correct\n"
20 | "3. the API key is correct"
21 | ) from e
22 |
23 | if len(models.data) == 0:
24 | raise RuntimeError(f"No models found on the vLLM server at {client.base_url}")
25 |
26 | return models.data[0].id
27 |
--------------------------------------------------------------------------------
/examples/others/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-decoder-config.yaml:
--------------------------------------------------------------------------------
1 | local_cpu: False
2 | max_local_cpu_size: 0
3 | #local_disk:
4 | max_local_disk_size: 0
5 | remote_serde: NULL
6 |
7 | enable_nixl: True
8 | nixl_role: "receiver"
9 | nixl_peer_host: "localhost"
10 | nixl_peer_port: 55555
11 | nixl_buffer_size: 1073741824 # 1GB
12 | nixl_buffer_device: "cuda"
13 | nixl_enable_gc: True
14 |
--------------------------------------------------------------------------------
/examples/others/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml:
--------------------------------------------------------------------------------
1 | local_cpu: False
2 | max_local_cpu_size: 0
3 | #local_disk:
4 | max_local_disk_size: 0
5 | remote_serde: NULL
6 |
7 | enable_nixl: True
8 | nixl_role: "sender"
9 | nixl_peer_host: "localhost"
10 | nixl_peer_port: 55555
11 | nixl_buffer_size: 1073741824 # 1GB
12 | nixl_buffer_device: "cuda"
13 | nixl_enable_gc: True
14 |
--------------------------------------------------------------------------------
/examples/template_alpaca.jinja:
--------------------------------------------------------------------------------
1 | {{ (messages|selectattr('role', 'equalto', 'system')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'system')|list) else '' }}
2 |
3 | {% for message in messages %}
4 | {% if message['role'] == 'user' %}
5 | ### Instruction:
6 | {{ message['content']|trim -}}
7 | {% if not loop.last %}
8 |
9 |
10 | {% endif %}
11 | {% elif message['role'] == 'assistant' %}
12 | ### Response:
13 | {{ message['content']|trim -}}
14 | {% if not loop.last %}
15 |
16 |
17 | {% endif %}
18 | {% elif message['role'] == 'user_context' %}
19 | ### Input:
20 | {{ message['content']|trim -}}
21 | {% if not loop.last %}
22 |
23 |
24 | {% endif %}
25 | {% endif %}
26 | {% endfor %}
27 | {% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}
28 | ### Response:
29 | {% endif %}
--------------------------------------------------------------------------------
/examples/template_baichuan.jinja:
--------------------------------------------------------------------------------
1 | {{ (messages|selectattr('role', 'equalto', 'system')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'system')|list) else '' }}
2 |
3 | {%- for message in messages -%}
4 | {%- if message['role'] == 'user' -%}
5 | {{- '' + message['content'] -}}
6 | {%- elif message['role'] == 'assistant' -%}
7 | {{- '' + message['content'] -}}
8 | {%- endif -%}
9 | {%- endfor -%}
10 |
11 | {%- if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
12 | {{- '' -}}
13 | {% endif %}
--------------------------------------------------------------------------------
/examples/template_chatglm.jinja:
--------------------------------------------------------------------------------
1 | {%- set counter = namespace(index=0) -%}
2 | {%- for message in messages -%}
3 | {%- if message['role'] == 'user' -%}
4 | {{- '[Round ' + counter.index|string + ']\n问：' + message['content'] -}}
5 | {%- set counter.index = counter.index + 1 -%}
6 | {%- endif -%}
7 | {%- if message['role'] == 'assistant' -%}
8 | {{- '\n答：' + message['content'] -}}
9 | {%- if (loop.last and add_generation_prompt) or not loop.last -%}
10 | {{- '\n' -}}
11 | {%- endif -%}
12 | {%- endif -%}
13 | {%- endfor -%}
14 |
15 |
16 | {%- if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
17 | {{- '\n答：' -}}
18 | {%- endif -%}
--------------------------------------------------------------------------------
/examples/template_chatglm2.jinja:
--------------------------------------------------------------------------------
1 | {%- set counter = namespace(index=1) -%}
2 | {%- for message in messages -%}
3 | {%- if message['role'] == 'user' -%}
4 | {{- '[Round ' + counter.index|string + ']\n\n问：' + message['content'] -}}
5 | {%- set counter.index = counter.index + 1 -%}
6 | {%- endif -%}
7 | {%- if message['role'] == 'assistant' -%}
8 | {{- '\n\n答：' + message['content'] -}}
9 | {%- if (loop.last and add_generation_prompt) or not loop.last -%}
10 | {{- '\n\n' -}}
11 | {%- endif -%}
12 | {%- endif -%}
13 | {%- endfor -%}
14 |
15 |
16 | {%- if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
17 | {{- '\n\n答：' -}}
18 | {%- endif -%}
--------------------------------------------------------------------------------
/examples/template_chatml.jinja:
--------------------------------------------------------------------------------
1 | {% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
2 | {% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
--------------------------------------------------------------------------------
/examples/template_falcon.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages -%}
2 | {%- if message['role'] == 'user' -%}
3 | {{- 'User: ' + message['content'] -}}
4 | {%- elif message['role'] == 'assistant' -%}
5 | {{- 'Assistant: ' + message['content'] -}}
6 | {%- endif -%}
7 | {%- if (loop.last and add_generation_prompt) or not loop.last -%}
8 | {{- '\n' -}}
9 | {%- endif -%}
10 | {%- endfor -%}
11 |
12 |
13 | {%- if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
14 | {{- 'Assistant:' -}}
15 | {% endif %}
--------------------------------------------------------------------------------
/examples/template_falcon_180b.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages -%}
2 | {%- if message['role'] == 'system' -%}
3 | {{- 'System: ' + message['content'] -}}
4 | {%- elif message['role'] == 'user' -%}
5 | {{- 'User: ' + message['content'] -}}
6 | {%- elif message['role'] == 'assistant' -%}
7 | {{- 'Falcon: ' + message['content'] -}}
8 | {%- endif -%}
9 | {%- if (loop.last and add_generation_prompt) or not loop.last -%}
10 | {{- '\n' -}}
11 | {%- endif -%}
12 | {%- endfor -%}
13 |
14 |
15 | {%- if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
16 | {{- 'Falcon:' -}}
17 | {% endif %}
--------------------------------------------------------------------------------
/examples/template_teleflm.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages %}
2 | {%- if message['role'] == 'user' %}
3 | {{- '<_user>' + message['content']|trim }}
4 | {%- elif message['role'] == 'system' %}
5 | {{- '<_system>' + message['content']|trim }}
6 | {%- elif message['role'] == 'assistant' %}
7 | {{- '<_bot>' + message['content'] }}
8 | {%- endif %}
9 | {%- endfor %}
10 | {%- if add_generation_prompt %}
11 | {{- '<_bot>' }}
12 | {%- endif %}
13 |
--------------------------------------------------------------------------------
/examples/template_vlm2vec.jinja:
--------------------------------------------------------------------------------
1 | {%- if messages | length > 1 -%}
2 | {{ raise_exception('Embedding models should only embed one message at a time') }}
3 | {%- endif -%}
4 |
5 | {% set vars = namespace(parts=[], next_image_id=1) %}
6 | {%- for message in messages -%}
7 | {%- for content in message['content'] -%}
8 | {%- if content['type'] == 'text' -%}
9 | {%- set vars.parts = vars.parts + [content['text']] %}
10 | {%- elif content['type'] == 'image' -%}
11 | {%- set vars.parts = vars.parts + ['<|image_{i:d}|>'.format(i=vars.next_image_id)] %}
12 | {%- set vars.next_image_id = vars.next_image_id + 1 %}
13 | {%- endif -%}
14 | {%- endfor -%}
15 | {%- endfor -%}
16 | {{ vars.parts | join(' ') }}
17 |
--------------------------------------------------------------------------------
/format.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | echo "vLLM linting system has been moved from format.sh to pre-commit hooks."
4 | echo "Please run 'pip install -r requirements/lint.txt', followed by"
5 | echo "'pre-commit install' to install the pre-commit hooks."
6 | echo "Then linters will run automatically before each commit."
--------------------------------------------------------------------------------
/requirements/build.txt:
--------------------------------------------------------------------------------
1 | # Should be mirrored in pyproject.toml
2 | cmake>=3.26.1
3 | ninja
4 | packaging>=24.2
5 | setuptools>=77.0.3,<80.0.0
6 | setuptools-scm>=8
7 | torch==2.7.0
8 | wheel
9 | jinja2>=3.1.6
10 | regex
11 |
--------------------------------------------------------------------------------
/requirements/cuda.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | numba == 0.60.0; python_version == '3.9' # v0.61 doesn't support Python 3.9. Required for N-gram speculative decoding
5 | numba == 0.61.2; python_version > '3.9'
6 |
7 | # Dependencies for NVIDIA GPUs
8 | ray[cgraph]>=2.43.0, !=2.44.* # Ray Compiled Graph, required for pipeline parallelism in V1.
9 | torch==2.7.0
10 | torchaudio==2.7.0
11 | # These must be updated alongside torch
12 | torchvision==0.22.0 # Required for phi3v processor. See https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
13 | # https://github.com/facebookresearch/xformers/releases/tag/v0.0.30
14 | xformers==0.0.30; platform_system == 'Linux' and platform_machine == 'x86_64' # Requires PyTorch >= 2.7
15 |
--------------------------------------------------------------------------------
/requirements/dev.txt:
--------------------------------------------------------------------------------
1 | -r lint.txt
2 | -r test.txt
3 |
4 | # Avoid adding requirements directly to this file.
5 | # Instead, modify the two files referenced above.
6 |
--------------------------------------------------------------------------------
/requirements/docs.txt:
--------------------------------------------------------------------------------
1 | mkdocs
2 | mkdocs-api-autonav
3 | mkdocs-material
4 | mkdocstrings-python
5 | mkdocs-gen-files
6 | mkdocs-awesome-nav
7 | python-markdown-math
8 | regex
9 | ruff
10 |
--------------------------------------------------------------------------------
/requirements/hpu.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | # Dependencies for HPU code
5 | ray
6 | triton==3.1.0
7 | pandas
8 | numpy==1.26.4
9 | tabulate
10 | setuptools>=77.0.3,<80.0.0
11 | setuptools-scm>=8
12 | vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@f1f6624
13 |
--------------------------------------------------------------------------------
/requirements/lint.txt:
--------------------------------------------------------------------------------
1 | # formatting
2 | pre-commit==4.0.1
3 |
--------------------------------------------------------------------------------
/requirements/neuron.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | # Dependencies for Neuron devices
5 | packaging>=24.2
6 | setuptools>=77.0.3,<80.0.0
7 | torch-neuronx >= 2.5.0
8 | neuronx-cc>=2.0.0a0
9 | torchvision # Required for Llama3.2 multimodal image preprocessing
10 |
--------------------------------------------------------------------------------
/requirements/rocm-build.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | --extra-index-url https://download.pytorch.org/whl/rocm6.2.4
5 | torch==2.7.0
6 | torchvision==0.22.0
7 | torchaudio==2.7.0
8 |
9 | triton==3.2
10 | cmake>=3.26.1,<4
11 | packaging>=24.2
12 | setuptools>=77.0.3,<80.0.0
13 | setuptools-scm>=8
14 | wheel
15 | jinja2>=3.1.6
16 | amdsmi==6.2.4
17 |
--------------------------------------------------------------------------------
/requirements/rocm-test.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | # entrypoints test
5 | # librosa==0.10.2.post1 # required by audio tests in entrypoints/openai
6 | audioread==3.0.1
7 | cffi==1.17.1
8 | decorator==5.2.1
9 | lazy-loader==0.4
10 | platformdirs==4.3.6
11 | pooch==1.8.2
12 | #pycparse==2.22
13 | soundfile==0.13.1
14 | soxr==0.5.0.post1
15 | librosa==0.10.2.post1
16 |
17 | # entrypoints test
18 | #vllm[video] # required by entrypoints/openai/test_video.py
19 | decord==0.6.0
20 |
21 | # entrypoints test
22 | #sentence-transformers # required by entrypoints/openai/test_score.py
23 | sentence-transformers==3.4.1
24 |
25 | # Basic Models Test
26 | matplotlib==3.10.3
27 |
28 | # Multi-Modal Models Test (Extended) 3
29 | blobfile==3.0.0
30 |
31 |
32 |
--------------------------------------------------------------------------------
/requirements/rocm.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | numba == 0.60.0; python_version == '3.9' # v0.61 doesn't support Python 3.9. Required for N-gram speculative decoding
5 | numba == 0.61.2; python_version > '3.9'
6 |
7 | # Dependencies for AMD GPUs
8 | boto3
9 | botocore
10 | datasets
11 | ray>=2.10.0,<2.45.0
12 | peft
13 | pytest-asyncio
14 | tensorizer>=2.9.0
15 | setuptools-scm>=8
16 | setuptools>=77.0.3,<80.0.0
17 | runai-model-streamer==0.11.0
18 | runai-model-streamer-s3==0.11.0
19 |
--------------------------------------------------------------------------------
/requirements/xpu.txt:
--------------------------------------------------------------------------------
1 | # Common dependencies
2 | -r common.txt
3 |
4 | ray>=2.9
5 | cmake>=3.26.1
6 | packaging>=24.2
7 | setuptools-scm>=8
8 | setuptools>=77.0.3,<80.0.0
9 | wheel
10 | jinja2>=3.1.6
11 | datasets # for benchmark scripts
12 |
13 | torch==2.7.0+xpu
14 | torchaudio
15 | torchvision
16 | pytorch-triton-xpu
17 | --extra-index-url=https://download.pytorch.org/whl/xpu
18 |
19 | # Please refer xpu doc, we need manually install intel-extension-for-pytorch 2.6.10+xpu due to there are some conflict dependencies with torch 2.6.0+xpu
20 | # FIXME: This will be fix in ipex 2.7. just leave this here for awareness.
21 | intel-extension-for-pytorch==2.7.10+xpu
22 | oneccl_bind_pt==2.7.0+xpu
23 | --extra-index-url=https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
24 |
--------------------------------------------------------------------------------
/tests/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/__init__.py
--------------------------------------------------------------------------------
/tests/async_engine/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/async_engine/__init__.py
--------------------------------------------------------------------------------
/tests/async_engine/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | Since this module is V0 only, set VLLM_USE_V1=0 for
10 | all tests in the module.
11 | """
12 | monkeypatch.setenv('VLLM_USE_V1', '0')
13 |
--------------------------------------------------------------------------------
/tests/basic_correctness/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/basic_correctness/__init__.py
--------------------------------------------------------------------------------
/tests/basic_correctness/test_cpu_offload.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from ..utils import compare_two_settings
5 |
6 |
7 | def test_cpu_offload():
8 | compare_two_settings("meta-llama/Llama-3.2-1B-Instruct", [],
9 | ["--cpu-offload-gb", "1"])
10 |
--------------------------------------------------------------------------------
/tests/benchmarks/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/benchmarks/__init__.py
--------------------------------------------------------------------------------
/tests/benchmarks/test_latency_cli.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import subprocess
4 |
5 | import pytest
6 |
7 | MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct"
8 |
9 |
10 | @pytest.mark.benchmark
11 | def test_bench_latency():
12 | command = [
13 | "vllm", "bench", "latency", "--model", MODEL_NAME, "--input-len", "32",
14 | "--output-len", "1", "--enforce-eager", "--load-format", "dummy"
15 | ]
16 | result = subprocess.run(command, capture_output=True, text=True)
17 | print(result.stdout)
18 | print(result.stderr)
19 |
20 | assert result.returncode == 0, f"Benchmark failed: {result.stderr}"
21 |
--------------------------------------------------------------------------------
/tests/benchmarks/test_throughput_cli.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import subprocess
4 |
5 | import pytest
6 |
7 | MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct"
8 |
9 |
10 | @pytest.mark.benchmark
11 | def test_bench_throughput():
12 | command = [
13 | "vllm", "bench", "throughput", "--model", MODEL_NAME, "--input-len",
14 | "32", "--output-len", "1", "--enforce-eager", "--load-format", "dummy"
15 | ]
16 | result = subprocess.run(command, capture_output=True, text=True)
17 | print(result.stdout)
18 | print(result.stderr)
19 |
20 | assert result.returncode == 0, f"Benchmark failed: {result.stderr}"
21 |
--------------------------------------------------------------------------------
/tests/compile/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/compile/__init__.py
--------------------------------------------------------------------------------
/tests/compile/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | # TEST V1: this should be removed. Right now V1 overrides
7 | # all the torch compile logic. We should re-enable this
8 | # as we add torch compile support back to V1.
9 | @pytest.fixture(scope="function", autouse=True)
10 | def use_v0_only(monkeypatch):
11 | """
12 | Since this module is V0 only, set VLLM_USE_V1=0 for
13 | all tests in the module.
14 | """
15 | monkeypatch.setenv('VLLM_USE_V1', '0')
16 |
--------------------------------------------------------------------------------
/tests/compile/piecewise/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/compile/piecewise/__init__.py
--------------------------------------------------------------------------------
/tests/config/test_config.yaml:
--------------------------------------------------------------------------------
1 | port: 12312
2 | served_model_name: mymodel
3 | tensor_parallel_size: 2
4 | trust_remote_code: true
5 | multi_step_stream_outputs: false
6 |
--------------------------------------------------------------------------------
/tests/config/test_config_with_model.yaml:
--------------------------------------------------------------------------------
1 | # Same as test_config.yaml but with model specified
2 | model: config-model
3 | port: 12312
4 | served_model_name: mymodel
5 | tensor_parallel_size: 2
6 | trust_remote_code: true
7 | multi_step_stream_outputs: false
8 |
--------------------------------------------------------------------------------
/tests/core/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/core/__init__.py
--------------------------------------------------------------------------------
/tests/core/block/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/core/block/__init__.py
--------------------------------------------------------------------------------
/tests/core/block/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import pytest
5 |
6 |
7 | @pytest.fixture()
8 | def should_do_global_cleanup_after_test() -> bool:
9 | """Disable the global cleanup fixture for tests in this directory. This
10 | provides a ~10x speedup for unit tests that don't load a model to GPU.
11 |
12 | This requires that tests in this directory clean up after themselves if they
13 | use the GPU.
14 | """
15 | return False
16 |
--------------------------------------------------------------------------------
/tests/core/block/e2e/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/core/block/e2e/__init__.py
--------------------------------------------------------------------------------
/tests/core/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | Since this module is V0 only, set VLLM_USE_V1=0 for
10 | all tests in the module.
11 | """
12 | monkeypatch.setenv('VLLM_USE_V1', '0')
13 |
--------------------------------------------------------------------------------
/tests/detokenizer/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/detokenizer/__init__.py
--------------------------------------------------------------------------------
/tests/detokenizer/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(autouse=True)
7 | def v1(run_with_both_engines):
8 | # Simple autouse wrapper to run both engines for each test
9 | # This can be promoted up to conftest.py to run for every
10 | # test in a package
11 | pass
12 |
--------------------------------------------------------------------------------
/tests/distributed/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/distributed/__init__.py
--------------------------------------------------------------------------------
/tests/distributed/test_distributed_oot.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from ..entrypoints.openai.test_oot_registration import (
5 | run_and_test_dummy_opt_api_server)
6 |
7 |
8 | def test_distributed_oot(dummy_opt_path: str):
9 | run_and_test_dummy_opt_api_server(dummy_opt_path, tp=2)
10 |
--------------------------------------------------------------------------------
/tests/encoder_decoder/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/encoder_decoder/__init__.py
--------------------------------------------------------------------------------
/tests/engine/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/engine/__init__.py
--------------------------------------------------------------------------------
/tests/engine/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | Since this module is V0 only, set VLLM_USE_V1=0 for
10 | all tests in the module.
11 | """
12 | monkeypatch.setenv('VLLM_USE_V1', '0')
13 |
--------------------------------------------------------------------------------
/tests/entrypoints/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/entrypoints/__init__.py
--------------------------------------------------------------------------------
/tests/entrypoints/llm/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/entrypoints/llm/__init__.py
--------------------------------------------------------------------------------
/tests/entrypoints/llm/test_prompt_validation.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import pytest
5 |
6 | from vllm import LLM
7 |
8 |
9 | @pytest.fixture(autouse=True)
10 | def v1(run_with_both_engines):
11 | # Simple autouse wrapper to run both engines for each test
12 | # This can be promoted up to conftest.py to run for every
13 | # test in a package
14 | pass
15 |
16 |
17 | def test_empty_prompt():
18 | llm = LLM(model="openai-community/gpt2", enforce_eager=True)
19 | with pytest.raises(ValueError, match='decoder prompt cannot be empty'):
20 | llm.generate([""])
21 |
22 |
23 | @pytest.mark.skip_v1
24 | def test_out_of_vocab_token():
25 | llm = LLM(model="openai-community/gpt2", enforce_eager=True)
26 | with pytest.raises(ValueError, match='out of vocabulary'):
27 | llm.generate({"prompt_token_ids": [999999]})
28 |
--------------------------------------------------------------------------------
/tests/entrypoints/offline_mode/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/entrypoints/offline_mode/__init__.py
--------------------------------------------------------------------------------
/tests/entrypoints/openai/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/entrypoints/openai/__init__.py
--------------------------------------------------------------------------------
/tests/entrypoints/openai/correctness/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/entrypoints/openai/correctness/__init__.py
--------------------------------------------------------------------------------
/tests/entrypoints/openai/tool_parsers/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/entrypoints/openai/tool_parsers/__init__.py
--------------------------------------------------------------------------------
/tests/fastsafetensors_loader/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/fastsafetensors_loader/__init__.py
--------------------------------------------------------------------------------
/tests/fastsafetensors_loader/test_fastsafetensors_loader.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm import SamplingParams
5 | from vllm.config import LoadFormat
6 |
7 | test_model = "openai-community/gpt2"
8 |
9 | prompts = [
10 | "Hello, my name is",
11 | "The president of the United States is",
12 | "The capital of France is",
13 | "The future of AI is",
14 | ]
15 | # Create a sampling params object.
16 | sampling_params = SamplingParams(temperature=0.8, top_p=0.95, seed=0)
17 |
18 |
19 | def test_model_loader_download_files(vllm_runner):
20 | with vllm_runner(test_model,
21 | load_format=LoadFormat.FASTSAFETENSORS) as llm:
22 | deserialized_outputs = llm.generate(prompts, sampling_params)
23 | assert deserialized_outputs
24 |
--------------------------------------------------------------------------------
/tests/kernels/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/kernels/__init__.py
--------------------------------------------------------------------------------
/tests/kernels/allclose_default.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import torch
5 |
6 | # Reference default values of atol and rtol are from
7 | # https://github.com/pytorch/pytorch/blob/6d96beb6bec24d73ee3f080bac54d2104068f675/test/test_transformers.py#L67
8 | default_atol = {torch.float16: 1e-3, torch.bfloat16: 1e-3, torch.float: 1e-5}
9 | default_rtol = {
10 | torch.float16: 1e-3,
11 | torch.bfloat16: 1.6e-2,
12 | torch.float: 1.3e-6
13 | }
14 |
15 |
16 | def get_default_atol(output) -> float:
17 | return default_atol[output.dtype]
18 |
19 |
20 | def get_default_rtol(output) -> float:
21 | return default_rtol[output.dtype]
22 |
--------------------------------------------------------------------------------
/tests/kernels/attention/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import pytest
5 |
6 | from vllm.utils import (create_kv_caches_with_random,
7 | create_kv_caches_with_random_flash)
8 |
9 |
10 | @pytest.fixture()
11 | def kv_cache_factory():
12 | return create_kv_caches_with_random
13 |
14 |
15 | @pytest.fixture()
16 | def kv_cache_factory_flashinfer():
17 | return create_kv_caches_with_random_flash
18 |
--------------------------------------------------------------------------------
/tests/kernels/core/test_opcheck.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | """
4 | Tests for miscellaneous utilities
5 | """
6 |
7 | import torch
8 |
9 | from tests.kernels.utils import opcheck
10 |
11 |
12 | def test_convert_fp8_opcheck():
13 | data = torch.randn((256, 256), dtype=torch.float32, device="cuda")
14 | result = torch.empty_like(data, dtype=torch.float8_e4m3fn)
15 | opcheck(torch.ops._C_cache_ops.convert_fp8, (result, data, 1.0, "fp8"))
16 |
17 |
18 | # TODO: Add this back, currently fails with
19 | # csrc/cuda_utils_kernels.cu:15 'invalid argument'
20 | # @pytest.mark.skipif(not current_platform.is_cuda(),
21 | # reason="Only supported for CUDA")
22 | # def test_cuda_utils_opcheck():
23 | # opcheck(torch.ops._C_cuda_utils.get_device_attribute, (0, 0))
24 | # opcheck(
25 | # torch.ops._C_cuda_utils.
26 | # get_max_shared_memory_per_block_device_attribute, (0, ))
27 |
--------------------------------------------------------------------------------
/tests/kernels/core/test_permute_cols.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import pytest
5 | import torch
6 |
7 | from tests.kernels.utils import opcheck
8 | from vllm._custom_ops import permute_cols
9 |
10 |
11 | @pytest.mark.parametrize('shape', [(1, 512), (544, 4096), (67, 8192)])
12 | @pytest.mark.parametrize('dtype', [torch.bfloat16, torch.float16])
13 | def test_permute_cols(shape, dtype):
14 | x = torch.randn(shape, dtype=dtype).cuda()
15 | perm = torch.randperm(x.shape[1]).to(torch.int).cuda()
16 | opcheck(torch.ops._C.permute_cols, (x, perm))
17 | y = permute_cols(x, perm)
18 | torch.testing.assert_close(y, x[:, perm])
--------------------------------------------------------------------------------
/tests/kernels/moe/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/kernels/moe/__init__.py
--------------------------------------------------------------------------------
/tests/kv_transfer/test_lookup_buffer.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | RANK=0 python3 test_lookup_buffer.py &
3 | PID0=$!
4 | RANK=1 python3 test_lookup_buffer.py &
5 | PID1=$!
6 |
7 | wait $PID0
8 | wait $PID1
9 |
--------------------------------------------------------------------------------
/tests/kv_transfer/test_send_recv.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | RANK=0 python3 test_send_recv.py &
4 | PID0=$!
5 | RANK=1 python3 test_send_recv.py &
6 | PID1=$!
7 |
8 | wait $PID0
9 | wait $PID1
10 |
--------------------------------------------------------------------------------
/tests/lora/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/lora/__init__.py
--------------------------------------------------------------------------------
/tests/metrics/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/metrics/__init__.py
--------------------------------------------------------------------------------
/tests/mistral_tool_use/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/mistral_tool_use/__init__.py
--------------------------------------------------------------------------------
/tests/model_executor/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/model_executor/__init__.py
--------------------------------------------------------------------------------
/tests/models/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/__init__.py
--------------------------------------------------------------------------------
/tests/models/language/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/language/__init__.py
--------------------------------------------------------------------------------
/tests/models/language/generation/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/language/generation/__init__.py
--------------------------------------------------------------------------------
/tests/models/language/pooling/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/language/pooling/__init__.py
--------------------------------------------------------------------------------
/tests/models/multimodal/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/multimodal/__init__.py
--------------------------------------------------------------------------------
/tests/models/multimodal/generation/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/multimodal/generation/__init__.py
--------------------------------------------------------------------------------
/tests/models/multimodal/generation/vlm_utils/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/multimodal/generation/vlm_utils/__init__.py
--------------------------------------------------------------------------------
/tests/models/multimodal/pooling/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/multimodal/pooling/__init__.py
--------------------------------------------------------------------------------
/tests/models/multimodal/processing/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/multimodal/processing/__init__.py
--------------------------------------------------------------------------------
/tests/models/quantization/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/models/quantization/__init__.py
--------------------------------------------------------------------------------
/tests/mq_llm_engine/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/mq_llm_engine/__init__.py
--------------------------------------------------------------------------------
/tests/mq_llm_engine/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | Since this module is V0 only, set VLLM_USE_V1=0 for
10 | all tests in the module.
11 | """
12 | monkeypatch.setenv('VLLM_USE_V1', '0')
13 |
--------------------------------------------------------------------------------
/tests/multi_step/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/multi_step/__init__.py
--------------------------------------------------------------------------------
/tests/multimodal/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/multimodal/__init__.py
--------------------------------------------------------------------------------
/tests/multimodal/assets/image1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/multimodal/assets/image1.png
--------------------------------------------------------------------------------
/tests/multimodal/assets/image2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/multimodal/assets/image2.png
--------------------------------------------------------------------------------
/tests/multimodal/assets/rgba.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/multimodal/assets/rgba.png
--------------------------------------------------------------------------------
/tests/multimodal/utils.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import numpy as np
5 | from PIL import Image
6 |
7 |
8 | def random_image(rng: np.random.RandomState, min_wh: int, max_wh: int):
9 | w, h = rng.randint(min_wh, max_wh, size=(2, ))
10 | arr = rng.randint(0, 255, size=(w, h, 3), dtype=np.uint8)
11 | return Image.fromarray(arr)
12 |
13 |
14 | def random_video(
15 | rng: np.random.RandomState,
16 | min_frames: int,
17 | max_frames: int,
18 | min_wh: int,
19 | max_wh: int,
20 | ):
21 | num_frames = rng.randint(min_frames, max_frames)
22 | w, h = rng.randint(min_wh, max_wh, size=(2, ))
23 | return rng.randint(0, 255, size=(num_frames, w, h, 3), dtype=np.uint8)
24 |
25 |
26 | def random_audio(
27 | rng: np.random.RandomState,
28 | min_len: int,
29 | max_len: int,
30 | sr: int,
31 | ):
32 | audio_len = rng.randint(min_len, max_len)
33 | return rng.rand(audio_len), sr
34 |
--------------------------------------------------------------------------------
/tests/neuron/1_core/test_neuron_quant.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | from vllm.model_executor.layers.quantization.neuron_quant import (
4 | NeuronQuantConfig)
5 |
6 |
7 | def test_get_supported_act_dtypes():
8 | neuron_quant_config = NeuronQuantConfig()
9 | supported_act_dtypes = neuron_quant_config.get_supported_act_dtypes()
10 | target_list = ["any_dtype1", "any_dtype2"]
11 | for dtype in target_list:
12 | assert dtype in supported_act_dtypes
13 |
--------------------------------------------------------------------------------
/tests/plugins/lora_resolvers/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/plugins/lora_resolvers/__init__.py
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_model/setup.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from setuptools import setup
5 |
6 | setup(name='vllm_add_dummy_model',
7 | version='0.1',
8 | packages=['vllm_add_dummy_model'],
9 | entry_points={
10 | 'vllm.general_plugins':
11 | ["register_dummy_model = vllm_add_dummy_model:register"]
12 | })
13 |
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_model/vllm_add_dummy_model/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm import ModelRegistry
5 |
6 |
7 | def register():
8 | # Test directly passing the model
9 | from .my_opt import MyOPTForCausalLM
10 |
11 | if "MyOPTForCausalLM" not in ModelRegistry.get_supported_archs():
12 | ModelRegistry.register_model("MyOPTForCausalLM", MyOPTForCausalLM)
13 |
14 | # Test passing lazy model
15 | if "MyGemma2Embedding" not in ModelRegistry.get_supported_archs():
16 | ModelRegistry.register_model(
17 | "MyGemma2Embedding",
18 | "vllm_add_dummy_model.my_gemma_embedding:MyGemma2Embedding",
19 | )
20 |
21 | if "MyLlava" not in ModelRegistry.get_supported_archs():
22 | ModelRegistry.register_model("MyLlava",
23 | "vllm_add_dummy_model.my_llava:MyLlava")
24 |
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_model/vllm_add_dummy_model/my_opt.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from typing import Optional
5 |
6 | import torch
7 |
8 | from vllm.model_executor.models.opt import OPTForCausalLM
9 | from vllm.model_executor.sampling_metadata import SamplingMetadata
10 |
11 |
12 | class MyOPTForCausalLM(OPTForCausalLM):
13 |
14 | def compute_logits(
15 | self, hidden_states: torch.Tensor,
16 | sampling_metadata: SamplingMetadata) -> Optional[torch.Tensor]:
17 | # this dummy model always predicts the first token
18 | logits = super().compute_logits(hidden_states, sampling_metadata)
19 | if logits is not None:
20 | logits.zero_()
21 | logits[:, 0] += 1.0
22 | return logits
23 |
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_platform/setup.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from setuptools import setup
5 |
6 | setup(
7 | name='vllm_add_dummy_platform',
8 | version='0.1',
9 | packages=['vllm_add_dummy_platform'],
10 | entry_points={
11 | 'vllm.platform_plugins': [
12 | "dummy_platform_plugin = vllm_add_dummy_platform:dummy_platform_plugin" # noqa
13 | ]
14 | })
15 |
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from typing import Optional
5 |
6 |
7 | def dummy_platform_plugin() -> Optional[str]:
8 | return "vllm_add_dummy_platform.dummy_platform.DummyPlatform"
9 |
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/dummy_attention_backend.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.attention.backends.flash_attn import FlashAttentionBackend
5 |
6 |
7 | class DummyAttentionBackend(FlashAttentionBackend):
8 |
9 | @staticmethod
10 | def get_name() -> str:
11 | return "Dummy_Backend"
12 |
--------------------------------------------------------------------------------
/tests/plugins/vllm_add_dummy_platform/vllm_add_dummy_platform/dummy_platform.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.platforms.cuda import CudaPlatform
5 |
6 |
7 | class DummyPlatform(CudaPlatform):
8 | device_name = "DummyDevice"
9 |
10 | def get_attn_backend_cls(self, backend_name, head_size, dtype,
11 | kv_cache_dtype, block_size, use_v1, use_mla):
12 | return "vllm_add_dummy_platform.dummy_attention_backend.DummyAttentionBackend" # noqa E501
13 |
--------------------------------------------------------------------------------
/tests/plugins_tests/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | Since this module is V0 only, set VLLM_USE_V1=0 for
10 | all tests in the module.
11 | """
12 | monkeypatch.setenv('VLLM_USE_V1', '0')
--------------------------------------------------------------------------------
/tests/prefix_caching/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/prefix_caching/__init__.py
--------------------------------------------------------------------------------
/tests/prompts/example.txt:
--------------------------------------------------------------------------------
1 | vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.
2 | Briefly describe the major milestones in the development of artificial intelligence from 1950 to 2020.
3 | Compare and contrast artificial intelligence with human intelligence in terms of processing information.
4 | Describe the basic components of a neural network and how it can be trained.
5 | Write a short story about a robot that dreams for the first time.
6 | Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.
7 | Explain the cultural significance of the Mona Lisa painting, and how its perception might vary in Western versus Eastern societies.
8 | Translate the following English sentence into Japanese, French, and Swahili: 'The early bird catches the worm.'
9 |
--------------------------------------------------------------------------------
/tests/quantization/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/quantization/__init__.py
--------------------------------------------------------------------------------
/tests/quantization/utils.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.model_executor.layers.quantization import get_quantization_config
5 | from vllm.platforms import current_platform
6 |
7 |
8 | def is_quant_method_supported(quant_method: str) -> bool:
9 | # Currently, all quantization methods require Nvidia or AMD GPUs
10 | if not (current_platform.is_cuda() or current_platform.is_rocm()):
11 | return False
12 |
13 | capability = current_platform.get_device_capability()
14 | assert capability is not None
15 |
16 | min_capability = get_quantization_config(quant_method).get_min_capability()
17 |
18 | return capability.to_int() >= min_capability
19 |
--------------------------------------------------------------------------------
/tests/reasoning/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/reasoning/__init__.py
--------------------------------------------------------------------------------
/tests/runai_model_streamer_test/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/runai_model_streamer_test/__init__.py
--------------------------------------------------------------------------------
/tests/samplers/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/samplers/__init__.py
--------------------------------------------------------------------------------
/tests/spec_decode/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/spec_decode/__init__.py
--------------------------------------------------------------------------------
/tests/spec_decode/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | Since this module is V0 only, set VLLM_USE_V1=0 for
10 | all tests in the module.
11 | """
12 | monkeypatch.setenv('VLLM_USE_V1', '0')
13 |
--------------------------------------------------------------------------------
/tests/spec_decode/e2e/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/spec_decode/e2e/__init__.py
--------------------------------------------------------------------------------
/tests/standalone_tests/python_only_compile.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | # This script tests if the python only compilation works correctly
3 | # for users who do not have any compilers installed on their system
4 |
5 | set -e
6 | set -x
7 |
8 | cd /vllm-workspace/
9 |
10 | # uninstall vllm
11 | pip3 uninstall -y vllm
12 | # restore the original files
13 | mv test_docs/vllm ./vllm
14 |
15 | # remove all compilers
16 | apt remove --purge build-essential -y
17 | apt autoremove -y
18 |
19 | echo 'import os; os.system("touch /tmp/changed.file")' >> vllm/__init__.py
20 |
21 | VLLM_TEST_USE_PRECOMPILED_NIGHTLY_WHEEL=1 VLLM_USE_PRECOMPILED=1 pip3 install -vvv -e .
22 |
23 | # Run the script
24 | python3 -c 'import vllm'
25 |
26 | # Check if the clangd log file was created
27 | if [ ! -f /tmp/changed.file ]; then
28 | echo "changed.file was not created, python only compilation failed"
29 | exit 1
30 | fi
31 |
--------------------------------------------------------------------------------
/tests/tensorizer_loader/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/tensorizer_loader/__init__.py
--------------------------------------------------------------------------------
/tests/tensorizer_loader/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 | from vllm.distributed import cleanup_dist_env_and_memory
6 | from vllm.model_executor.model_loader.tensorizer import TensorizerConfig
7 |
8 |
9 | @pytest.fixture(autouse=True)
10 | def cleanup():
11 | cleanup_dist_env_and_memory(shutdown_ray=True)
12 |
13 |
14 | @pytest.fixture(autouse=True)
15 | def tensorizer_config():
16 | config = TensorizerConfig(tensorizer_uri="vllm")
17 | return config
18 |
--------------------------------------------------------------------------------
/tests/test_embedded_commit.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import vllm
5 |
6 |
7 | def test_embedded_commit_defined():
8 | assert hasattr(vllm, "__version__")
9 | assert hasattr(vllm, "__version_tuple__")
10 | assert vllm.__version__ != "dev"
11 | assert vllm.__version_tuple__ != (0, 0, "dev")
12 |
--------------------------------------------------------------------------------
/tests/test_outputs.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.outputs import RequestOutput
5 |
6 |
7 | def test_request_output_forward_compatible():
8 | output = RequestOutput(request_id="test_request_id",
9 | prompt="test prompt",
10 | prompt_token_ids=[1, 2, 3],
11 | prompt_logprobs=None,
12 | outputs=[],
13 | finished=False,
14 | example_arg_added_in_new_version="some_value")
15 | assert output is not None
16 |
--------------------------------------------------------------------------------
/tests/test_seed_behavior.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import random
4 |
5 | import numpy as np
6 | import torch
7 |
8 | from vllm.platforms.interface import Platform
9 |
10 |
11 | def test_seed_behavior():
12 | # Test with a specific seed
13 | Platform.seed_everything(42)
14 | random_value_1 = random.randint(0, 100)
15 | np_random_value_1 = np.random.randint(0, 100)
16 | torch_random_value_1 = torch.randint(0, 100, (1, )).item()
17 |
18 | Platform.seed_everything(42)
19 | random_value_2 = random.randint(0, 100)
20 | np_random_value_2 = np.random.randint(0, 100)
21 | torch_random_value_2 = torch.randint(0, 100, (1, )).item()
22 |
23 | assert random_value_1 == random_value_2
24 | assert np_random_value_1 == np_random_value_2
25 | assert torch_random_value_1 == torch_random_value_2
26 |
--------------------------------------------------------------------------------
/tests/tokenization/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/tokenization/__init__.py
--------------------------------------------------------------------------------
/tests/tokenization/test_tokenizer.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import pytest
5 | from transformers import PreTrainedTokenizerBase
6 |
7 | from vllm.transformers_utils.tokenizer import get_tokenizer
8 |
9 | TOKENIZER_NAMES = [
10 | "facebook/opt-125m",
11 | "gpt2",
12 | ]
13 |
14 |
15 | @pytest.mark.parametrize("tokenizer_name", TOKENIZER_NAMES)
16 | def test_tokenizer_revision(tokenizer_name: str):
17 | # Assume that "main" branch always exists
18 | tokenizer = get_tokenizer(tokenizer_name, revision="main")
19 | assert isinstance(tokenizer, PreTrainedTokenizerBase)
20 |
21 | # Assume that "never" branch always does not exist
22 | with pytest.raises(OSError, match='not a valid git identifier'):
23 | get_tokenizer(tokenizer_name, revision="never")
24 |
--------------------------------------------------------------------------------
/tests/tool_use/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/tool_use/__init__.py
--------------------------------------------------------------------------------
/tests/tpu/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/tpu/__init__.py
--------------------------------------------------------------------------------
/tests/tpu/lora/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/tpu/lora/__init__.py
--------------------------------------------------------------------------------
/tests/tracing/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/tracing/__init__.py
--------------------------------------------------------------------------------
/tests/v1/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/__init__.py
--------------------------------------------------------------------------------
/tests/v1/e2e/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/e2e/__init__.py
--------------------------------------------------------------------------------
/tests/v1/engine/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/engine/__init__.py
--------------------------------------------------------------------------------
/tests/v1/entrypoints/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/entrypoints/__init__.py
--------------------------------------------------------------------------------
/tests/v1/entrypoints/llm/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/entrypoints/llm/__init__.py
--------------------------------------------------------------------------------
/tests/v1/kv_connector/unit/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/kv_connector/unit/__init__.py
--------------------------------------------------------------------------------
/tests/v1/sample/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/sample/__init__.py
--------------------------------------------------------------------------------
/tests/v1/shutdown/utils.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | """Shutdown test utils"""
4 |
5 | SHUTDOWN_TEST_TIMEOUT_SEC = 120
6 | SHUTDOWN_TEST_THRESHOLD_BYTES = 2 * 2**30
7 |
--------------------------------------------------------------------------------
/tests/v1/structured_output/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/structured_output/__init__.py
--------------------------------------------------------------------------------
/tests/v1/tpu/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/tpu/__init__.py
--------------------------------------------------------------------------------
/tests/v1/tpu/worker/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/tpu/worker/__init__.py
--------------------------------------------------------------------------------
/tests/v1/worker/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/v1/worker/__init__.py
--------------------------------------------------------------------------------
/tests/vllm_test_utils/setup.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from setuptools import setup
5 |
6 | setup(
7 | name='vllm_test_utils',
8 | version='0.1',
9 | packages=['vllm_test_utils'],
10 | )
11 |
--------------------------------------------------------------------------------
/tests/vllm_test_utils/vllm_test_utils/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | """
4 | vllm_utils is a package for vLLM testing utilities.
5 | It does not import any vLLM modules.
6 | """
7 |
8 | from .blame import BlameResult, blame
9 | from .monitor import MonitoredValues, monitor
10 |
11 | __all__ = ["blame", "BlameResult", "monitor", "MonitoredValues"]
12 |
--------------------------------------------------------------------------------
/tests/weight_loading/models-large.txt:
--------------------------------------------------------------------------------
1 | compressed-tensors, nm-testing/Mixtral-8x7B-Instruct-v0.1-W4A16-quantized, main
2 | compressed-tensors, nm-testing/Mixtral-8x7B-Instruct-v0.1-W4A16-channel-quantized, main
3 | compressed-tensors, nm-testing/Mixtral-8x7B-Instruct-v0.1-W8A16-quantized, main
4 | compressed-tensors, nm-testing/test-w4a16-mixtral-actorder-group, main
5 | gptq_marlin, TheBloke/Mixtral-8x7B-v0.1-GPTQ, main
6 | gptq_marlin, TheBloke/Mixtral-8x7B-v0.1-GPTQ, gptq-8bit-128g-actorder_True
7 | awq_marlin, casperhansen/deepseek-coder-v2-instruct-awq, main
8 | compressed-tensors, RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16, main
--------------------------------------------------------------------------------
/tests/worker/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/tests/worker/__init__.py
--------------------------------------------------------------------------------
/tests/worker/conftest.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import pytest
4 |
5 |
6 | @pytest.fixture(scope="function", autouse=True)
7 | def use_v0_only(monkeypatch):
8 | """
9 | This module tests V0 internals, so set VLLM_USE_V1=0.
10 | """
11 | monkeypatch.setenv('VLLM_USE_V1', '0')
--------------------------------------------------------------------------------
/tools/check_repo.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | # Checks whether the repo is clean and whether tags are available (necessary to correctly produce vllm version at build time)
3 |
4 | if ! git diff --quiet; then
5 | echo "Repo is dirty" >&2
6 |
7 | exit 1
8 | fi
9 |
10 | if ! git describe --tags; then
11 | echo "No tags are present. Is this a shallow clone? git fetch --unshallow --tags" >&2
12 |
13 | exit 1
14 | fi
15 |
--------------------------------------------------------------------------------
/tools/ep_kernels/install_system_drivers.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 |
3 | # prepare workspace directory
4 | WORKSPACE=$1
5 | if [ -z "$WORKSPACE" ]; then
6 | export WORKSPACE=$(pwd)/ep_kernels_workspace
7 | fi
8 |
9 | if [ ! -d "$WORKSPACE" ]; then
10 | mkdir -p $WORKSPACE
11 | fi
12 |
13 | # build and install gdrcopy driver
14 | pushd $WORKSPACE
15 | cd gdrcopy_src
16 | ./insmod.sh
17 | # run gdrcopy_copybw to test the installation
18 | $WORKSPACE/gdrcopy_install/bin/gdrcopy_copybw
19 |
20 | # turn on IBGDA
21 | echo 'options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"' | tee -a /etc/modprobe.d/nvidia.conf
22 | update-initramfs -u
23 |
24 | echo "Please reboot the system to apply the changes"
25 |
--------------------------------------------------------------------------------
/tools/ep_kernels/install_system_libraries.sh:
--------------------------------------------------------------------------------
1 | set -ex
2 |
3 | # prepare workspace directory
4 | WORKSPACE=$1
5 | if [ -z "$WORKSPACE" ]; then
6 | export WORKSPACE=$(pwd)/ep_kernels_workspace
7 | fi
8 |
9 | if [ ! -d "$WORKSPACE" ]; then
10 | mkdir -p $WORKSPACE
11 | fi
12 |
13 | # build and install gdrcopy system packages
14 | pushd $WORKSPACE
15 | cd gdrcopy_src/packages
16 | apt install devscripts -y
17 | CUDA=${CUDA_HOME:-/usr/local/cuda} ./build-deb-packages.sh
18 | dpkg -i *.deb
19 |
--------------------------------------------------------------------------------
/tools/mypy.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | CI=${1:-0}
4 | PYTHON_VERSION=${2:-local}
5 |
6 | if [ "$CI" -eq 1 ]; then
7 | set -e
8 | fi
9 |
10 | if [ $PYTHON_VERSION == "local" ]; then
11 | PYTHON_VERSION=$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')
12 | fi
13 |
14 | run_mypy() {
15 | echo "Running mypy on $1"
16 | if [ "$CI" -eq 1 ] && [ -z "$1" ]; then
17 | mypy --python-version "${PYTHON_VERSION}" "$@"
18 | return
19 | fi
20 | mypy --follow-imports skip --python-version "${PYTHON_VERSION}" "$@"
21 | }
22 |
23 | run_mypy # Note that this is less strict than CI
24 | run_mypy tests
25 | run_mypy vllm/attention
26 | run_mypy vllm/compilation
27 | run_mypy vllm/distributed
28 | run_mypy vllm/engine
29 | run_mypy vllm/executor
30 | run_mypy vllm/inputs
31 | run_mypy vllm/lora
32 | run_mypy vllm/model_executor
33 | run_mypy vllm/plugins
34 | run_mypy vllm/prompt_adapter
35 | run_mypy vllm/spec_decode
36 | run_mypy vllm/worker
37 | run_mypy vllm/v1
38 |
--------------------------------------------------------------------------------
/tools/png-lint.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # Ensure that *.excalidraw.png files have the excalidraw metadata
4 | # embedded in them. This ensures they can be loaded back into
5 | # the tool and edited in the future.
6 |
7 | find . -iname '*.excalidraw.png' | while read -r file; do
8 | if git check-ignore -q "$file"; then
9 | continue
10 | fi
11 | if ! grep -q "excalidraw+json" "$file"; then
12 | echo "$file was not exported from excalidraw with 'Embed Scene' enabled."
13 | exit 1
14 | fi
15 | done
16 |
--------------------------------------------------------------------------------
/tools/shellcheck.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | set -e
3 |
4 | scversion="stable"
5 |
6 | if [ -d "shellcheck-${scversion}" ]; then
7 | export PATH="$PATH:$(pwd)/shellcheck-${scversion}"
8 | fi
9 |
10 | if ! [ -x "$(command -v shellcheck)" ]; then
11 | if [ "$(uname -s)" != "Linux" ] || [ "$(uname -m)" != "x86_64" ]; then
12 | echo "Please install shellcheck: https://github.com/koalaman/shellcheck?tab=readme-ov-file#installing"
13 | exit 1
14 | fi
15 |
16 | # automatic local install if linux x86_64
17 | wget -qO- "https://github.com/koalaman/shellcheck/releases/download/${scversion?}/shellcheck-${scversion?}.linux.x86_64.tar.xz" | tar -xJv
18 | export PATH="$PATH:$(pwd)/shellcheck-${scversion}"
19 | fi
20 |
21 | # TODO - fix warnings in .buildkite/scripts/hardware_ci/run-amd-test.sh
22 | find . -name "*.sh" ".git" -prune -not -path "./.buildkite/scripts/hardware_ci/run-amd-test.sh" -print0 | xargs -0 -I {} sh -c 'git check-ignore -q "{}" || shellcheck -s bash "{}"'
23 |
--------------------------------------------------------------------------------
/use_existing_torch.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import glob
5 |
6 | requires_files = glob.glob('requirements/*.txt')
7 | requires_files += ["pyproject.toml"]
8 | for file in requires_files:
9 | print(f">>> cleaning {file}")
10 | with open(file) as f:
11 | lines = f.readlines()
12 | if "torch" in "".join(lines).lower():
13 | print("removed:")
14 | with open(file, 'w') as f:
15 | for line in lines:
16 | if 'torch' not in line.lower():
17 | f.write(line)
18 | else:
19 | print(line.strip())
20 | print(f"<<< done cleaning {file}")
21 | print()
22 |
--------------------------------------------------------------------------------
/vllm/adapter_commons/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/adapter_commons/__init__.py
--------------------------------------------------------------------------------
/vllm/adapter_commons/layers.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from dataclasses import dataclass
5 |
6 |
7 | @dataclass
8 | class AdapterMapping:
9 | # Per every token in input_ids:
10 | index_mapping: tuple[int, ...]
11 | # Per sampled token:
12 | prompt_mapping: tuple[int, ...]
13 |
14 | def __post_init__(self):
15 | self.index_mapping = tuple(self.index_mapping)
16 | self.prompt_mapping = tuple(self.prompt_mapping)
--------------------------------------------------------------------------------
/vllm/adapter_commons/request.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from abc import ABC, abstractmethod
5 |
6 |
7 | class AdapterRequest(ABC):
8 | """
9 | Base class for adapter requests.
10 | """
11 |
12 | @property
13 | @abstractmethod
14 | def adapter_id(self) -> int:
15 | raise NotImplementedError
16 |
17 | def __post_init__(self) -> None:
18 | if self.adapter_id < 1:
19 | raise ValueError(f"id must be > 0, got {self.adapter_id}")
20 |
21 | def __eq__(self, value: object) -> bool:
22 | return isinstance(
23 | value, self.__class__) and self.adapter_id == value.adapter_id
24 |
25 | def __hash__(self) -> int:
26 | return hash(self.adapter_id)
27 |
--------------------------------------------------------------------------------
/vllm/assets/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/assets/__init__.py
--------------------------------------------------------------------------------
/vllm/attention/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.attention.backends.abstract import (AttentionBackend,
5 | AttentionMetadata,
6 | AttentionMetadataBuilder,
7 | AttentionState, AttentionType)
8 | from vllm.attention.layer import Attention
9 | from vllm.attention.selector import get_attn_backend
10 |
11 | __all__ = [
12 | "Attention",
13 | "AttentionBackend",
14 | "AttentionMetadata",
15 | "AttentionType",
16 | "AttentionMetadataBuilder",
17 | "Attention",
18 | "AttentionState",
19 | "get_attn_backend",
20 | ]
21 |
--------------------------------------------------------------------------------
/vllm/attention/backends/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/attention/backends/__init__.py
--------------------------------------------------------------------------------
/vllm/attention/backends/mla/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/attention/backends/mla/__init__.py
--------------------------------------------------------------------------------
/vllm/attention/ops/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/attention/ops/__init__.py
--------------------------------------------------------------------------------
/vllm/attention/ops/blocksparse_attention/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/attention/ops/blocksparse_attention/__init__.py
--------------------------------------------------------------------------------
/vllm/benchmarks/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/benchmarks/__init__.py
--------------------------------------------------------------------------------
/vllm/compilation/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/compilation/__init__.py
--------------------------------------------------------------------------------
/vllm/core/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/core/__init__.py
--------------------------------------------------------------------------------
/vllm/core/block/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/core/block/__init__.py
--------------------------------------------------------------------------------
/vllm/device_allocator/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/device_allocator/__init__.py
--------------------------------------------------------------------------------
/vllm/distributed/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from .communication_op import *
5 | from .parallel_state import *
6 | from .utils import *
7 |
--------------------------------------------------------------------------------
/vllm/distributed/device_communicators/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/distributed/device_communicators/__init__.py
--------------------------------------------------------------------------------
/vllm/distributed/device_communicators/neuron_communicator.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import torch
4 |
5 | from vllm.distributed.device_communicators.base_device_communicator import (
6 | DeviceCommunicatorBase)
7 | from vllm.platforms import current_platform
8 |
9 | if current_platform.is_neuron():
10 | import torch_xla.core.xla_model as xm
11 |
12 |
13 | class NeuronCommunicator(DeviceCommunicatorBase):
14 |
15 | def all_reduce(self, x: torch.Tensor) -> torch.Tensor:
16 | return xm.all_reduce(xm.REDUCE_SUM, x)
17 |
18 | def all_gather(self, x: torch.Tensor, dim: int = -1) -> torch.Tensor:
19 | assert dim == -1, "Neuron only supports dim=-1 for all-gather."
20 | return xm.all_gather(x, dim=dim)
21 |
--------------------------------------------------------------------------------
/vllm/distributed/kv_transfer/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.distributed.kv_transfer.kv_transfer_state import (
5 | KVConnectorBaseType, ensure_kv_transfer_initialized, get_kv_transfer_group,
6 | has_kv_transfer_group, is_v1_kv_transfer_group)
7 |
8 | __all__ = [
9 | "get_kv_transfer_group", "has_kv_transfer_group",
10 | "is_v1_kv_transfer_group", "ensure_kv_transfer_initialized",
11 | "KVConnectorBaseType"
12 | ]
13 |
--------------------------------------------------------------------------------
/vllm/distributed/kv_transfer/disagg_prefill_workflow.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/distributed/kv_transfer/disagg_prefill_workflow.jpg
--------------------------------------------------------------------------------
/vllm/distributed/kv_transfer/kv_connector/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/distributed/kv_transfer/kv_connector/__init__.py
--------------------------------------------------------------------------------
/vllm/distributed/kv_transfer/kv_connector/v1/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | from vllm.distributed.kv_transfer.kv_connector.v1.base import (
4 | KVConnectorBase_V1, KVConnectorRole)
5 |
6 | __all__ = ["KVConnectorRole", "KVConnectorBase_V1"]
7 |
--------------------------------------------------------------------------------
/vllm/distributed/kv_transfer/kv_lookup_buffer/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/distributed/kv_transfer/kv_lookup_buffer/__init__.py
--------------------------------------------------------------------------------
/vllm/distributed/kv_transfer/kv_pipe/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/distributed/kv_transfer/kv_pipe/__init__.py
--------------------------------------------------------------------------------
/vllm/engine/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/engine/__init__.py
--------------------------------------------------------------------------------
/vllm/engine/output_processor/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/engine/output_processor/__init__.py
--------------------------------------------------------------------------------
/vllm/entrypoints/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/entrypoints/__init__.py
--------------------------------------------------------------------------------
/vllm/entrypoints/cli/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/entrypoints/cli/__init__.py
--------------------------------------------------------------------------------
/vllm/entrypoints/cli/benchmark/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/entrypoints/cli/benchmark/__init__.py
--------------------------------------------------------------------------------
/vllm/entrypoints/cli/benchmark/serve.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | import argparse
4 |
5 | from vllm.benchmarks.serve import add_cli_args, main
6 | from vllm.entrypoints.cli.benchmark.base import BenchmarkSubcommandBase
7 | from vllm.entrypoints.cli.types import CLISubcommand
8 |
9 |
10 | class BenchmarkServingSubcommand(BenchmarkSubcommandBase):
11 | """ The `serve` subcommand for vllm bench. """
12 |
13 | def __init__(self):
14 | self.name = "serve"
15 | super().__init__()
16 |
17 | @property
18 | def help(self) -> str:
19 | return "Benchmark the online serving throughput."
20 |
21 | def add_cli_args(self, parser: argparse.ArgumentParser) -> None:
22 | add_cli_args(parser)
23 |
24 | @staticmethod
25 | def cmd(args: argparse.Namespace) -> None:
26 | main(args)
27 |
28 |
29 | def cmd_init() -> list[CLISubcommand]:
30 | return [BenchmarkServingSubcommand()]
31 |
--------------------------------------------------------------------------------
/vllm/entrypoints/cli/types.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import argparse
5 |
6 | from vllm.utils import FlexibleArgumentParser
7 |
8 |
9 | class CLISubcommand:
10 | """Base class for CLI argument handlers."""
11 |
12 | name: str
13 |
14 | @staticmethod
15 | def cmd(args: argparse.Namespace) -> None:
16 | raise NotImplementedError("Subclasses should implement this method")
17 |
18 | def validate(self, args: argparse.Namespace) -> None:
19 | # No validation by default
20 | pass
21 |
22 | def subparser_init(
23 | self,
24 | subparsers: argparse._SubParsersAction) -> FlexibleArgumentParser:
25 | raise NotImplementedError("Subclasses should implement this method")
26 |
--------------------------------------------------------------------------------
/vllm/entrypoints/openai/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/entrypoints/openai/__init__.py
--------------------------------------------------------------------------------
/vllm/executor/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/executor/__init__.py
--------------------------------------------------------------------------------
/vllm/logging_utils/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.logging_utils.formatter import NewLineFormatter
5 |
6 | __all__ = [
7 | "NewLineFormatter",
8 | ]
9 |
--------------------------------------------------------------------------------
/vllm/logging_utils/formatter.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | import logging
5 |
6 |
7 | class NewLineFormatter(logging.Formatter):
8 | """Adds logging prefix to newlines to align multi-line messages."""
9 |
10 | def __init__(self, fmt, datefmt=None, style="%"):
11 | logging.Formatter.__init__(self, fmt, datefmt, style)
12 |
13 | def format(self, record):
14 | msg = logging.Formatter.format(self, record)
15 | if record.message != "":
16 | parts = msg.split(record.message)
17 | msg = msg.replace("\n", "\r\n" + parts[0])
18 | return msg
19 |
--------------------------------------------------------------------------------
/vllm/lora/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/lora/__init__.py
--------------------------------------------------------------------------------
/vllm/lora/ops/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/lora/ops/__init__.py
--------------------------------------------------------------------------------
/vllm/lora/ops/torch_ops/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.lora.ops.torch_ops.lora_ops import bgmv_expand # noqa: F401
5 | from vllm.lora.ops.torch_ops.lora_ops import (bgmv_expand_slice, bgmv_shrink,
6 | sgmv_expand, sgmv_expand_slice,
7 | sgmv_shrink)
8 |
9 | __all__ = [
10 | "bgmv_expand",
11 | "bgmv_expand_slice",
12 | "bgmv_shrink",
13 | "sgmv_expand",
14 | "sgmv_expand_slice",
15 | "sgmv_shrink",
16 | ]
17 |
--------------------------------------------------------------------------------
/vllm/lora/ops/triton_ops/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.lora.ops.triton_ops.lora_expand_op import lora_expand
5 | from vllm.lora.ops.triton_ops.lora_kernel_metadata import LoRAKernelMeta
6 | from vllm.lora.ops.triton_ops.lora_shrink_op import lora_shrink
7 |
8 | __all__ = [
9 | "lora_expand",
10 | "lora_shrink",
11 | "LoRAKernelMeta",
12 | ]
13 |
--------------------------------------------------------------------------------
/vllm/lora/ops/xla_ops/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.lora.ops.xla_ops.lora_ops import (bgmv_expand, bgmv_expand_slice,
5 | bgmv_shrink)
6 |
7 | __all__ = ["bgmv_expand", "bgmv_expand_slice", "bgmv_shrink"]
8 |
--------------------------------------------------------------------------------
/vllm/lora/punica_wrapper/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.lora.punica_wrapper.punica_base import PunicaWrapperBase
5 | from vllm.lora.punica_wrapper.punica_selector import get_punica_wrapper
6 |
7 | __all__ = [
8 | "PunicaWrapperBase",
9 | "get_punica_wrapper",
10 | ]
11 |
--------------------------------------------------------------------------------
/vllm/lora/punica_wrapper/punica_selector.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.logger import init_logger
5 | from vllm.platforms import current_platform
6 | from vllm.utils import resolve_obj_by_qualname
7 |
8 | from .punica_base import PunicaWrapperBase
9 |
10 | logger = init_logger(__name__)
11 |
12 |
13 | def get_punica_wrapper(*args, **kwargs) -> PunicaWrapperBase:
14 | punica_wrapper_qualname = current_platform.get_punica_wrapper()
15 | punica_wrapper_cls = resolve_obj_by_qualname(punica_wrapper_qualname)
16 | punica_wrapper = punica_wrapper_cls(*args, **kwargs)
17 | assert punica_wrapper is not None, \
18 | "the punica_wrapper_qualname(" + punica_wrapper_qualname + ") is wrong."
19 | logger.info_once("Using %s.", punica_wrapper_qualname.rsplit(".", 1)[1])
20 | return punica_wrapper
21 |
--------------------------------------------------------------------------------
/vllm/model_executor/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.model_executor.parameter import (BasevLLMParameter,
5 | PackedvLLMParameter)
6 | from vllm.model_executor.sampling_metadata import (SamplingMetadata,
7 | SamplingMetadataCache)
8 | from vllm.model_executor.utils import set_random_seed
9 |
10 | __all__ = [
11 | "SamplingMetadata",
12 | "SamplingMetadataCache",
13 | "set_random_seed",
14 | "BasevLLMParameter",
15 | "PackedvLLMParameter",
16 | ]
17 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/model_executor/layers/__init__.py
--------------------------------------------------------------------------------
/vllm/model_executor/layers/fused_moe/configs/README:
--------------------------------------------------------------------------------
1 | This directory contains tuned configurations for different settings of the fused_moe kernel.
2 | For different settings of
3 | - E (number of experts)
4 | - N (intermediate size)
5 | - device_name (torch.cuda.get_device_name())
6 | the JSON file contains a mapping from M (batch size) to the chosen configuration.
7 |
8 | The example configurations provided are for the Mixtral model for TP2 on H100
9 | and TP4 on A100. Mixtral has intermediate size N = 14336, i.e. for TP2 we have
10 | N = 7168 and for TP4 we have N = 3584.
11 |
12 | See `benchmark/kernels/benchmark_moe.py` on how to generate these config files.
13 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/mamba/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/model_executor/layers/mamba/__init__.py
--------------------------------------------------------------------------------
/vllm/model_executor/layers/mamba/ops/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/model_executor/layers/mamba/ops/__init__.py
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/compressed_tensors/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/model_executor/layers/quantization/compressed_tensors/__init__.py
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/kernels/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/model_executor/layers/quantization/kernels/__init__.py
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/quark/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/model_executor/layers/quantization/quark/__init__.py
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/quark/schemes/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from .quark_scheme import QuarkScheme
5 | from .quark_w4a4_mxfp4 import QuarkW4A4MXFP4
6 | from .quark_w8a8_fp8 import QuarkW8A8Fp8
7 | from .quark_w8a8_int8 import QuarkW8A8Int8
8 |
9 | __all__ = ["QuarkScheme", "QuarkW8A8Fp8", "QuarkW8A8Int8", "QuarkW4A4MXFP4"]
10 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from .layer_utils import replace_parameter, update_tensor_inplace
5 |
6 | __all__ = ['update_tensor_inplace', 'replace_parameter']
7 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=1536,K=1536,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 32,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 32,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=1536,K=7168,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 64,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=1536,K=7168,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 32,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=2048,K=512,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 64,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=2304,K=7168,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 32,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 16,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=24576,K=7168,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 16,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 64,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=24576,K=7168,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 32,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 32,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 32,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=256,K=7168,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 4
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 8,
24 | "num_stages": 5
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=7168,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=32768,K=512,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=32768,K=512,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=512,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 32,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 64,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=4608,K=7168,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=512,K=7168,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 32,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 16,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 64,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=576,K=7168,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "4": {
3 | "BLOCK_SIZE_M": 16,
4 | "BLOCK_SIZE_N": 32,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "8": {
11 | "BLOCK_SIZE_M": 16,
12 | "BLOCK_SIZE_N": 32,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 32,
15 | "num_warps": 4,
16 | "num_stages": 4
17 | }
18 | }
19 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=576,K=7168,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 64,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 64,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=1024,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 16,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=1152,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 1,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=128,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 16,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 64,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 16,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 64,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=16384,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 64,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=16384,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 16,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 32,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 32,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=18432,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 64,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=18432,device_name=NVIDIA_L20Y,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 64,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 128,
6 | "GROUP_SIZE_M": 16,
7 | "num_warps": 4,
8 | "num_stages": 3
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 64,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 128,
14 | "GROUP_SIZE_M": 32,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 64,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 128,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=2048,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 64,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=2304,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 1,
15 | "num_warps": 4,
16 | "num_stages": 2
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 1,
23 | "num_warps": 4,
24 | "num_stages": 2
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=256,device_name=NVIDIA_L20,dtype=fp8_w8a8,block_shape=[128,128].json:
--------------------------------------------------------------------------------
1 | {
2 | "2048": {
3 | "BLOCK_SIZE_M": 128,
4 | "BLOCK_SIZE_N": 128,
5 | "BLOCK_SIZE_K": 64,
6 | "GROUP_SIZE_M": 64,
7 | "num_warps": 4,
8 | "num_stages": 2
9 | },
10 | "3072": {
11 | "BLOCK_SIZE_M": 128,
12 | "BLOCK_SIZE_N": 128,
13 | "BLOCK_SIZE_K": 64,
14 | "GROUP_SIZE_M": 32,
15 | "num_warps": 4,
16 | "num_stages": 3
17 | },
18 | "4096": {
19 | "BLOCK_SIZE_M": 128,
20 | "BLOCK_SIZE_N": 128,
21 | "BLOCK_SIZE_K": 64,
22 | "GROUP_SIZE_M": 64,
23 | "num_warps": 4,
24 | "num_stages": 3
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/vllm/model_executor/models/phi3.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | # Adapted from llama.py
5 | """Inference-only Phi3 model code inherit from Llama.py"""
6 |
7 | from vllm.model_executor.models.llama import LlamaForCausalLM
8 |
9 |
10 | class Phi3ForCausalLM(LlamaForCausalLM):
11 |
12 | packed_modules_mapping = {
13 | "qkv_proj": [
14 | "qkv_proj",
15 | ],
16 | "gate_up_proj": [
17 | "gate_up_proj",
18 | ],
19 | }
20 |
--------------------------------------------------------------------------------
/vllm/plugins/lora_resolvers/README.md:
--------------------------------------------------------------------------------
1 | # LoRA Resolver Plugins
2 |
3 | This directory contains vLLM general plugins for dynamically discovering and loading LoRA adapters
4 | via the LoRAResolver plugin framework.
5 |
6 | Note that `VLLM_ALLOW_RUNTIME_LORA_UPDATING` must be set to true to allow LoRA resolver plugins
7 | to work, and `VLLM_PLUGINS` must be set to include the desired resolver plugins.
8 |
9 | # lora_filesystem_resolver
10 | This LoRA Resolver is installed with vLLM by default.
11 | To use, set `VLLM_PLUGIN_LORA_CACHE_DIR` to a local directory. When vLLM receives a request
12 | for a LoRA adapter `foobar` it doesn't currently recognize, it will look in that local directory
13 | for a subdirectory `foobar` containing a LoRA adapter. If such an adapter exists, it will
14 | load that adapter, and then service the request as normal. That adapter will then be available
15 | for future requests as normal.
16 |
--------------------------------------------------------------------------------
/vllm/plugins/lora_resolvers/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/plugins/lora_resolvers/__init__.py
--------------------------------------------------------------------------------
/vllm/profiler/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/profiler/__init__.py
--------------------------------------------------------------------------------
/vllm/prompt_adapter/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/prompt_adapter/__init__.py
--------------------------------------------------------------------------------
/vllm/py.typed:
--------------------------------------------------------------------------------
1 | # Marker file for PEP 561.
2 | # The vllm package uses inline types.
3 |
--------------------------------------------------------------------------------
/vllm/reasoning/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from .abs_reasoning_parsers import ReasoningParser, ReasoningParserManager
5 | from .deepseek_r1_reasoning_parser import DeepSeekR1ReasoningParser
6 | from .granite_reasoning_parser import GraniteReasoningParser
7 | from .qwen3_reasoning_parser import Qwen3ReasoningParser
8 |
9 | __all__ = [
10 | "ReasoningParser",
11 | "ReasoningParserManager",
12 | "DeepSeekR1ReasoningParser",
13 | "GraniteReasoningParser",
14 | "Qwen3ReasoningParser",
15 | ]
16 |
--------------------------------------------------------------------------------
/vllm/scripts.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.entrypoints.cli.main import main as vllm_main
5 | from vllm.logger import init_logger
6 |
7 | logger = init_logger(__name__)
8 |
9 |
10 | # Backwards compatibility for the move from vllm.scripts to
11 | # vllm.entrypoints.cli.main
12 | def main():
13 | logger.warning("vllm.scripts.main() is deprecated. Please re-install "
14 | "vllm or use vllm.entrypoints.cli.main.main() instead.")
15 | vllm_main()
16 |
--------------------------------------------------------------------------------
/vllm/spec_decode/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/spec_decode/__init__.py
--------------------------------------------------------------------------------
/vllm/third_party/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/third_party/__init__.py
--------------------------------------------------------------------------------
/vllm/transformers_utils/chat_templates/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | from .registry import get_chat_template_fallback_path
4 |
5 | __all__ = ["get_chat_template_fallback_path"]
6 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/chat_templates/template_basic.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages -%}
2 | {{- message['content'] -}}
3 | {%- endfor -%}
4 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/chat_templates/template_blip2.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages -%}
2 | {%- if message['role'] == 'user' -%}
3 | {{- 'Question: ' + message['content'] + ' ' -}}
4 | {%- elif message['role'] == 'assistant' -%}
5 | {{- 'Answer: ' + message['content'] + ' ' -}}
6 | {%- endif -%}
7 | {%- endfor -%}
8 |
9 | {%- if add_generation_prompt -%}
10 | {{- 'Answer:' -}}
11 | {% endif %}
12 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/chat_templates/template_chatml.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages -%}
2 | {{- '<|im_start|>' + message['role'] + '\n' + message['content'] -}}
3 | {%- if (loop.last and add_generation_prompt) or not loop.last -%}
4 | {{- '<|im_end|>' + '\n' -}}
5 | {%- endif -%}
6 | {%- endfor -%}
7 |
8 | {%- if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
9 | {{- '<|im_start|>assistant\n' -}}
10 | {%- endif -%}
11 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/chat_templates/template_deepseek_vl2.jinja:
--------------------------------------------------------------------------------
1 | {%- if messages[0]['role'] == 'system' -%}
2 | {%- set system_message = messages[0]['content'] -%}
3 | {%- set messages = messages[1:] -%}
4 | {%- else -%}
5 | {% set system_message = '' -%}
6 | {%- endif -%}
7 |
8 | {{ bos_token + system_message }}
9 | {%- for message in messages -%}
10 | {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {%- endif -%}
13 |
14 | {%- if message['role'] == 'user' -%}
15 | {{ '<|User|>: ' + message['content'] + '\n\n' }}
16 | {%- elif message['role'] == 'assistant' -%}
17 | {{ '<|Assistant|>: ' + message['content'] + eos_token + '\n\n' }}
18 | {%- endif -%}
19 | {%- endfor -%}
20 |
21 | {%- if add_generation_prompt -%}
22 | {{ '<|Assistant|>: ' }}
23 | {%- endif -%}
24 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/chat_templates/template_fuyu.jinja:
--------------------------------------------------------------------------------
1 | {%- for message in messages -%}
2 | {{- message['content'] + '\n' -}}
3 | {%- endfor -%}
4 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/configs/h2ovl.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | # Adapted from
5 | # https://huggingface.co/h2oai/h2ovl-mississippi-2b/blob/main/configuration_h2ovl_chat.py
6 | # --------------------------------------------------------
7 | # H2OVL-Mississippi
8 | # Copyright (c) 2024 H2O.AI
9 | # Licensed under Apache 2.0 License [see LICENSE for details]
10 | # --------------------------------------------------------
11 |
12 | from .internvl import InternVLChatConfig
13 |
14 |
15 | class H2OVLChatConfig(InternVLChatConfig):
16 | model_type = "h2ovl_chat"
17 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/configs/nvlm_d.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | # Adapted from
5 | # https://huggingface.co/nvidia/NVLM-D-72B/blob/main/configuration_nvlm_d.py
6 | # --------------------------------------------------------
7 | # NVLM-D
8 | # Copyright (c) 2024 NVIDIA
9 | # Licensed under Apache 2.0 License [see LICENSE for details]
10 | # --------------------------------------------------------
11 | from .internvl import InternVLChatConfig
12 |
13 |
14 | class NVLM_D_Config(InternVLChatConfig):
15 | model_type = 'NVLM_D'
16 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/processors/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.transformers_utils.processors.deepseek_vl2 import (
5 | DeepseekVLV2Processor)
6 | from vllm.transformers_utils.processors.ovis import OvisProcessor
7 |
8 | __all__ = ["DeepseekVLV2Processor", "OvisProcessor"]
9 |
--------------------------------------------------------------------------------
/vllm/transformers_utils/tokenizers/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from .mistral import (MistralTokenizer, maybe_serialize_tool_calls,
5 | truncate_tool_call_ids, validate_request_params)
6 |
7 | __all__ = [
8 | "MistralTokenizer", "maybe_serialize_tool_calls", "truncate_tool_call_ids",
9 | "validate_request_params"
10 | ]
11 |
--------------------------------------------------------------------------------
/vllm/triton_utils/__init__.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 |
4 | from vllm.triton_utils.importing import (HAS_TRITON, TritonLanguagePlaceholder,
5 | TritonPlaceholder)
6 |
7 | if HAS_TRITON:
8 | import triton
9 | import triton.language as tl
10 | else:
11 | triton = TritonPlaceholder()
12 | tl = TritonLanguagePlaceholder()
13 |
14 | __all__ = ["HAS_TRITON", "triton", "tl"]
15 |
--------------------------------------------------------------------------------
/vllm/usage/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/usage/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/attention/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/attention/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/attention/backends/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/attention/backends/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/attention/backends/mla/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/attention/backends/mla/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/core/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/core/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/core/sched/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/core/sched/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/core/sched/utils.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | from vllm.v1.request import Request, RequestStatus
4 |
5 |
6 | def check_stop(request: Request, max_model_len: int) -> bool:
7 | if (request.num_tokens >= max_model_len
8 | or request.num_output_tokens >= request.max_tokens):
9 | request.status = RequestStatus.FINISHED_LENGTH_CAPPED
10 | return True
11 |
12 | sampling_params = request.sampling_params
13 | last_token_id = request.output_token_ids[-1]
14 | if (not sampling_params.ignore_eos
15 | and last_token_id == request.eos_token_id):
16 | request.status = RequestStatus.FINISHED_STOPPED
17 | return True
18 |
19 | if last_token_id in (sampling_params.stop_token_ids or ()):
20 | request.status = RequestStatus.FINISHED_STOPPED
21 | request.stop_reason = last_token_id
22 | return True
23 | return False
24 |
--------------------------------------------------------------------------------
/vllm/v1/engine/exceptions.py:
--------------------------------------------------------------------------------
1 | # SPDX-License-Identifier: Apache-2.0
2 | # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 | class EngineGenerateError(Exception):
4 | """Raised when a AsyncLLM.generate() fails. Recoverable."""
5 | pass
6 |
7 |
8 | class EngineDeadError(Exception):
9 | """Raised when the EngineCore dies. Unrecoverable."""
10 |
11 | def __init__(self, *args, suppress_context: bool = False, **kwargs):
12 | ENGINE_DEAD_MESSAGE = "EngineCore encountered an issue. See stack trace (above) for the root cause." # noqa: E501
13 |
14 | super().__init__(ENGINE_DEAD_MESSAGE, *args, **kwargs)
15 | # Make stack trace clearer when using with LLMEngine by
16 | # silencing irrelevant ZMQError.
17 | self.__suppress_context__ = suppress_context
18 |
--------------------------------------------------------------------------------
/vllm/v1/executor/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/executor/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/metrics/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/metrics/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/sample/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/sample/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/sample/ops/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/sample/ops/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/sample/tpu/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/sample/tpu/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/spec_decode/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/spec_decode/__init__.py
--------------------------------------------------------------------------------
/vllm/v1/worker/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/v1/worker/__init__.py
--------------------------------------------------------------------------------
/vllm/vllm_flash_attn/.gitkeep:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/vllm_flash_attn/.gitkeep
--------------------------------------------------------------------------------
/vllm/worker/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ROCm/vllm/cdfe72b0140fbe2f569b3cbc69c95de1c4d6988a/vllm/worker/__init__.py
--------------------------------------------------------------------------------