How to Serve an LLM with an OpenAI-Compatible API on a GPU Server

Tutorial Contents

Concepts & Requirements

1

Understanding OpenAI-Compatible LLM Serving

An OpenAI-compatible API allows applications to communicate with a self-hosted language model using request formats similar to those provided by the OpenAI API.

Rather than rewriting your application around a model-specific interface, you can use an inference server that exposes familiar endpoints, including:

• /v1/chat/completions – Sends conversational requests to the model.
• /v1/completions – Generates text from a supplied prompt.
• /v1/models – Lists models available through the API.

The inference server receives requests, processes them using the loaded model, and returns generated responses in a compatible format.

For this deployment, we will use vLLM, an open-source inference and serving engine designed for efficient LLM execution. It supports OpenAI-compatible HTTP endpoints and GPU-based inference. One important distinction is that API compatibility does not mean every OpenAI feature is automatically available. Supported endpoints, parameters, and model capabilities depend on the inference engine and the model being deployed.

2

Server Requirements

Before deploying an LLM, prepare a GPU server with sufficient computing resources for your selected model. The required hardware depends on several factors, including model size, numerical precision, context length, concurrent requests, and inference workload.

GPU memory is one of the most important considerations. A larger model generally requires more memory to load and serve. Additional VRAM is also needed for the KV cache, which stores intermediate attention information during inference.

For example, a model using 16-bit weights requires approximately two bytes per parameter for its raw weights alone. Actual memory requirements are higher because of runtime allocations, model architecture, context length, and concurrent requests. For production deployments, select a GPU configuration based on the model's documented requirements and expected workload rather than model parameter count alone.

Preparation & Installation

3

Prepare the GPU Server

We will use Ubuntu Server and an NVIDIA GPU for this example. Start by connecting to your server through SSH:

ssh username@YOUR_SERVER_IP

Update the operating system packages and install basic utilities:

sudo apt update
sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv git curl wget build-essential

Next, verify that the NVIDIA driver is installed correctly:

nvidia-smi

This command displays information about the detected NVIDIA GPUs, driver version, GPU memory, and current utilization. If the command cannot communicate with the GPU, check the NVIDIA driver installation before continuing.

The CUDA environment must also be compatible with the PyTorch and vLLM versions you intend to install. Having an NVIDIA GPU does not automatically mean that every CUDA-dependent application will work without the correct software configuration. For this tutorial, we will use the official vLLM installation approach.

4

Install vLLM

Create a separate Python environment for the inference server. Using an isolated environment helps prevent dependency conflicts with other Python applications running on the server.

First, install the uv Python package manager:

curl -LsSf https://astral.sh/uv/install.sh | sh

Reload your shell configuration if necessary, then create a project directory and Python 3.12 virtual environment:

mkdir ~/llm-server
cd ~/llm-server
uv venv --python 3.12 --seed
source .venv/bin/activate

Install vLLM:

uv pip install vllm --torch-backend=auto

The --torch-backend=auto option allows uv to select an appropriate PyTorch backend based on the detected environment. After installation, verify that the command is available:

vllm --version

5

Download and Configure an LLM

Next, select an open-weight language model that is compatible with vLLM. For this example, we will use Qwen/Qwen2.5-7B-Instruct, which is an instruction-tuned model suitable for demonstrating conversational inference through a chat API.

Set the model identifier as an environment variable, and configure a Hugging Face cache directory to keep downloaded model files organized:

export MODEL_ID="Qwen/Qwen2.5-7B-Instruct"
mkdir -p ~/models
export HF_HOME=~/models

Deployment & Testing

6

Start the OpenAI-Compatible API Server

With vLLM installed and the model selected, you can start the inference server. First, generate an API key for the example deployment (use a unique secret rather than copying this example value):

export VLLM_API_KEY="replace-with-a-long-random-secret"

Now launch vLLM:

vllm serve "$MODEL_ID" \
    --host 127.0.0.1 \
    --port 8000 \
    --dtype auto \
    --max-model-len 4096 \
    --api-key "$VLLM_API_KEY"

We have used 127.0.0.1 so the API listens only on the server's loopback interface. This is a safer starting point because the service is not directly exposed to the public network. Once the model is loaded, the API should be available at http://127.0.0.1:8000/v1.

7

Test the API with a Chat Completion Request

Open another SSH session and request the list of models available through the API:

curl http://127.0.0.1:8000/v1/models \
    -H "Authorization: Bearer $VLLM_API_KEY"

Next, send a chat completion request:

curl http://127.0.0.1:8000/v1/chat/completions \
    -H "Authorization: Bearer $VLLM_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Qwen/Qwen2.5-7B-Instruct",
      "messages": [
        {
          "role": "user",
          "content": "Explain how GPU acceleration improves LLM inference."
        }
      ],
      "temperature": 0.7,
      "max_tokens": 200
    }'

If everything is configured correctly, the server should return a JSON response containing the requested model identifier, the generated message, completion status, and token usage information.

8

Connect to the API Using Python

One of the main advantages of an OpenAI-compatible interface is that existing applications can use familiar API client libraries. For this example, we will use the official OpenAI Python SDK.

Create a separate Python environment or use an existing application environment, then install the SDK: pip install openai

Create a Python file named test_llm.py:

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key=os.environ["VLLM_API_KEY"]
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful technical assistant."
        },
        {
            "role": "user",
            "content": "Explain the benefits of dedicated GPU servers."
        }
    ],
    temperature=0.7,
    max_tokens=200
)

print(response.choices[0].message.content)

Make sure the API key is exported in the terminal running the Python application (export VLLM_API_KEY="your-configured-api-key") and run the script: python test_llm.py

Production & Monitoring

9

Expose the API for Remote Applications

So far, the API has been accessible only from the GPU server itself. To allow applications on other machines to connect, configure a reverse proxy such as Nginx between external clients and the inference server.

• Keep the vLLM service bound to localhost.
• Configure Nginx to forward permitted API requests to the internal vLLM endpoint.
• Enable HTTPS using a valid TLS certificate.
• Implement authentication and access restrictions at the proxy or application layer.
• Configure firewall rules to allow only the required network connections.

10

Monitor GPU Resources and Inference Performance

After deploying the model, monitor the server to understand how it performs under actual workloads. Run this command to refresh GPU information every two seconds:

watch -n 2 nvidia-smi

If GPU memory becomes insufficient, consider reducing the context length, selecting a smaller model, using a supported quantization method, or distributing the model across multiple GPUs where supported.

11

Common Deployment Issues

• GPU not detected: Run nvidia-smi and verify that the installed NVIDIA driver is compatible with the GPU and software environment.
• CUDA or PyTorch compatibility errors: Check the vLLM installation requirements. Using a fresh virtual environment and a supported installation combination can help prevent dependency conflicts.
• Insufficient GPU memory: Check the model's memory requirements, maximum sequence length, and KV cache allocation. Consider a smaller model or an appropriate quantized configuration.
• Model loading takes too long: Large models can require considerable time to download and initialize. Check available storage, network connectivity, and model repository access.
• API requests fail: Verify that the server is running, the requested endpoint is correct, the model identifier matches the deployed model, and the authentication key is valid.