How to Deploy Zhipu AI GLM-4.5-Air with SGLang on a Dedicated GPU Server

Tutorial Contents

Phase 1: Understand the Model & Choose Hardware

1

Understand the GLM-4.5-Air Model

GLM-4.5-Air is a mixture-of-experts (MoE) language model designed for reasoning, coding and agent-based tasks. Its architecture contains approximately 106 billion total parameters, with approximately 12 billion active parameters during inference.

For this deployment, we use the FP8 checkpoint:

zai-org/GLM-4.5-Air-FP8

FP8 is a lower-precision model format that reduces the memory required for model weights compared with higher-precision formats. However, the model still requires substantial GPU and system memory.

SGLang is the inference framework used in this guide. It provides model-serving capabilities and an OpenAI-compatible API.

2

Choose Suitable Dedicated GPU Hardware

Hardware selection depends on the model format, available GPU memory, context length and expected workload.

Z.ai publishes specific NVIDIA GPU configurations for GLM-4.5-Air-FP8. Its documented full-featured inference configuration uses two H100 GPUs. For the full 128K context length, the documented configuration increases to four H100 GPUs.

The following is a suitable reference configuration for a deployment targeting the full context length:

A two-H100 configuration is documented for standard FP8 inference, while four H100 GPUs are specified for the full 128K context. Actual performance and memory requirements depend on the workload and software versions.

Phase 2: Prepare the Server & Install SGLang

3

Prepare Ubuntu and Verify the NVIDIA GPUs

Start by connecting to the dedicated server through SSH:

ssh username@SERVER_IP

Update the operating system:

sudo apt update
sudo apt upgrade -y

Install basic utilities:

sudo apt install -y curl git wget build-essential

Install and verify the NVIDIA driver. A fresh Ubuntu installation does not normally include the proprietary NVIDIA driver required for GPU inference. Check the available driver recommendations:

ubuntu-drivers devices

Install Ubuntu's recommended NVIDIA driver:

sudo ubuntu-drivers install

Reboot if required:

sudo reboot

Reconnect to the server and verify GPU detection:

nvidia-smi

List the installed GPUs:

nvidia-smi -L

The output should show the NVIDIA GPUs installed in the server.

Check available memory and storage:

free -h
df -h

For multi-GPU deployments, also inspect GPU connectivity:

nvidia-smi topo -m

This helps identify the GPU interconnect topology, which is relevant when distributing model computation across multiple GPUs.

4

Create a Python Environment and Install SGLang

Using an isolated Python environment helps separate the inference framework and its dependencies from system Python packages.

Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Load the environment:

source "$HOME/.local/bin/env"

Create a working directory:

mkdir -p ~/glm-deployment
cd ~/glm-deployment

Create a Python 3.12 virtual environment:

uv venv --python 3.12

Activate it:

source .venv/bin/activate

Install SGLang using its documented installation command:

uv pip install --prerelease=allow sglang

The --prerelease=allow option is included because some SGLang dependencies publish prerelease packages. Do not assume that adding the [all] extra is necessary for this standard NVIDIA installation.

Verify the installed package:

python -m pip show sglang

For reproducible deployments, record the installed package versions and retain the environment details used for testing.

5

Configure the Hugging Face Model Cache

The model weights need to be downloaded before inference can begin. Store the downloaded files on storage with sufficient free capacity.

Create the cache directory with the correct ownership:

sudo install -d -o "$USER" -g "$(id -gn)" /data/huggingface

Set the Hugging Face cache location:

echo 'export HF_HOME=/data/huggingface' >> ~/.bashrc
source ~/.bashrc

Verify the variable:

echo "$HF_HOME"

Expected Output: /data/huggingface

This configuration makes the cache location persistent for future Bash sessions. Confirm that the service account used to run SGLang can write to this directory.

The GLM-4.5-Air-FP8 repository is publicly available. If you use a private or gated model repository in another deployment, configure the required Hugging Face authentication separately.

Phase 3: Launch the Model & Test the API

6

Start the GLM-4.5-Air-FP8 Server

With the virtual environment active, launch SGLang:

python -m sglang.launch_server \
    --model-path zai-org/GLM-4.5-Air-FP8 \
    --served-model-name glm-4.5-air-fp8 \
    --host 127.0.0.1 \
    --port 30000

The command specifies the model repository, the model name exposed through the API, and the address and port used by the server.

The first startup downloads the model files into the configured Hugging Face cache. Download duration depends on model size, available storage and network throughput.

Keep the terminal open while the server initializes. Monitor the output for model-loading errors and wait until the server reports that it is ready to receive requests.

Why use 127.0.0.1? Binding to 127.0.0.1 restricts the API listener to the local machine. This prevents the inference endpoint from being directly exposed through the server's public network interface.

The command above intentionally uses the basic model-serving configuration documented for this checkpoint. Advanced features such as speculative decoding should be enabled only after verifying compatibility with the installed SGLang release and the model implementation.

7

Test the OpenAI-Compatible API

Once the server is ready, open another SSH session to the machine. First, check whether the model endpoint responds:

curl http://127.0.0.1:30000/v1/models

Next, send a chat completion request:

curl http://127.0.0.1:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "glm-4.5-air-fp8",
      "messages": [
        {
          "role": "user",
          "content": "Explain how a dedicated GPU server supports large language model inference."
        }
      ],
      "max_tokens": 300
    }'

A successful request returns a JSON response containing the generated assistant message.

The API uses the familiar /v1/chat/completions endpoint. This makes it possible to integrate the deployed model with applications and tools that support the OpenAI-compatible API format.

8

Connect an Application Using Python

Applications can communicate with the local inference endpoint through the OpenAI Python SDK. With the virtual environment active, install the client:

uv pip install openai

Create a Python file named test_glm.py:

from openai import OpenAI

client = OpenAI(
    api_key="local-development",
    base_url="http://127.0.0.1:30000/v1"
)

response = client.chat.completions.create(
    model="glm-4.5-air-fp8",
    messages=[
        {
            "role": "user",
            "content": "Explain the role of GPU memory in LLM inference."
        }
    ],
    max_tokens=300
)

print(response.choices[0].message.content)

Run the script:

python test_glm.py

The client sends the request to the local SGLang server and prints the generated response.

Phase 4: Monitoring, systemd & Security

9

Monitor GPU and System Resources

Monitoring is important when running large language models. GPU memory usage, system RAM, storage and request performance can affect the reliability of the service.

Monitor GPU activity:

watch -n 2 nvidia-smi

Check system memory and available storage:

free -h
df -h

During inference, observe GPU utilization and memory consumption. If the system experiences out-of-memory errors, investigate the model configuration, context length, concurrency and available memory before increasing workload.

For a production deployment, collect application logs and monitor resource usage over time rather than relying only on manual terminal checks.

10

Run SGLang as a systemd Service

Running SGLang in an interactive SSH session is suitable for initial testing, but it is not a convenient long-term process-management approach. A systemd service can start the application automatically and restart it after certain failures.

First, stop the manually launched server using Ctrl+C. Then create a service file:

sudo nano /etc/systemd/system/sglang.service

Add the following configuration, replacing YOUR_LINUX_USER with the actual Linux account used for the deployment:

[Unit]
Description=SGLang GLM-4.5-Air Inference Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=YOUR_LINUX_USER
WorkingDirectory=/home/YOUR_LINUX_USER/glm-deployment
Environment=HF_HOME=/data/huggingface
ExecStart=/home/YOUR_LINUX_USER/glm-deployment/.venv/bin/python -m sglang.launch_server --model-path zai-org/GLM-4.5-Air-FP8 --served-model-name glm-4.5-air-fp8 --host 127.0.0.1 --port 30000
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Save the file, then reload systemd, enable the service and start it:

sudo systemctl daemon-reload
sudo systemctl enable sglang
sudo systemctl start sglang

Check its status:

sudo systemctl status sglang

To inspect application logs:

sudo journalctl -u sglang -f

The service runs under the specified Linux account, so ensure that account has access to the virtual environment and model cache.

11

Secure the Inference Endpoint

The example deployment binds the API to localhost. If an application on another machine needs access, use an authenticated HTTPS reverse proxy or another controlled private-network access method.

For production deployments:

• Do not expose an unauthenticated inference endpoint directly to the public internet.
• Use HTTPS for remote API communication.
• Configure authentication at the reverse proxy or API gateway.
• Restrict access using firewall rules and network policies.
• Keep API credentials outside source code and restrict access to secret files.
• Regularly update the operating system and inference dependencies.

If the server is intended for internal use, keep the inference listener private and allow access only through the required application network.

Phase 5: Troubleshooting Common Problems

12

Fix Common Deployment Issues

If something does not work as expected, start with the following checks:

Avoid changing several inference parameters simultaneously when troubleshooting. Make one adjustment at a time so that the effect can be identified.