GLM-5.2 NVFP4 Inference on Modal 8×B200

Serve zai-org/GLM-5.2 quantized to NVFP4 on a Modal 8×B200 node via vLLM, OpenAI-compatible, with a correctness gate against the BF16 reference and a throughput/latency benchmark. The verifier sends hidden prompts to your /app/submission/launch_server.sh and compares against a well-tuned vLLM NVFP4 baseline.

Model

zai-org/GLM-5.2 — 753B-param Mixture-of-Experts (40B active), 256 routed experts + 1 shared (8 activated/token), 78 transformer layers + 1 MTP layer at index 78, DeepSeek-style MLA attention + DSA sparse-attention indexer (glm_moe_dsa architecture), hidden 6144, vocab 154880.
BF16 checkpoint ~1.37 TB; NVFP4 quantized ~459 GB (~3× smaller).
Quantization: NVFP4 — E2M1 4-bit weights + FP8 E4M3 block scales, 16-element blocks, produced with NVIDIA Model Optimizer (modelopt). Only the MoE expert FFNs (routed + shared) are quantized. Attention (MLA projections q/kv + DSA indexer), the MoE router (mlp.gate), and lm_head are kept in BF16 — SGLang/vLLM's deepseek_v2 MLA path cannot consume NVFP4 attention weights.
KV cache: FP8 (--kv-cache-dtype fp8) to maximize context headroom.
License: MIT (pure open).
Use a community NVFP4 checkpoint: Lorbus/GLM-5.2-NVFP4, Mapika/GLM-5.2-NVFP4, or lukealonso/GLM-5.2-NVFP4. Pick one and pin the revision in launch_server.sh.

Hardware

Modal 8×B200, single node — @app.function(gpu="B200:8"). 8× B200 on one physical machine, 8×192 GB HBM = 1536 GB total. NVFP4 tensor cores are Blackwell-only. This is the entire GPU budget — no second node, no disaggregated prefill/decode pools. One vLLM engine, TP=8, collocated prefill+decode.
Weights ~459 GB → ~57 GB/GPU at TP=8, leaving ~135 GB/GPU for KV cache, CUDA graphs, and runtime buffers. TP=8 is mandatory: TP=4 would split the node into two 4-GPU engines and leave only ~77 GB/GPU for KV — too tight for GLM-5.2's context window, and you cannot run two TP=8 engines on 8 GPUs.
No disaggregation. Disaggregated prefill/decode requires a second 8×B200 node (one TP=8 prefill + one TP=8 decode with NIXL/Mooncake KV transfer). On a single node, use chunked prefill (--enable-chunked-prefill with a tuned --max-num-batched-tokens chunk size) to get the ITL-stability benefit of disaggregation without splitting the GPU pool.

Tools / stack

vLLM v0.23.0+ — required for modelopt / modelopt_fp4 NVFP4 support and the glm_moe_dsa model registration. Pin a known-good version (e.g. vllm==0.23.0 or newer) in the Modal image.
NVIDIA Model Optimizer (tensorrt-model-optimizer) — only needed if you re-quantize; for serving a pre-quantized HF checkpoint you do not need it.
FlashInfer — Blackwell MoE / FP4 kernels. Set VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 (and the equivalent NVFP4 env flag for your vLLM version) in the image env.
huggingface_hub[hf_transfer] — fast shard download (~459 GB).
Modal SDK — modal.Image, modal.Volume for HF/vLLM/flashinfer caches, @modal.web_server to expose the vLLM HTTP server.
CUDA 12.8+ devel image as the base (nvidia/cuda:12.8.1-devel-ubuntu22.04). vLLM ships its own CUDA runtime; the devel base is for any JIT kernel compilation.

Files

/app/submission/launch_server.sh
- Main file you modify. Verifier executes it with PORT and MODEL_PATH env vars. It must start a vLLM OpenAI-compatible server listening on $PORT.
/app/submission/
- Owned submission root. launch_server.sh plus any helper scripts, vLLM patches, or env files must live under this tree.
/app/serve_modal.py
- Modal entrypoint defining the image, volumes, @app.function(gpu="B200:8"), and @modal.web_server wrapper that subprocess.Popens vllm serve.
/app/run_dev_bench.py
- Public dev benchmark: launches your server, sends test requests, reports TTFT / throughput. Hidden verifier uses different workloads.
/app/verify_serving.py
- Sanity check that the server starts and produces coherent outputs.
/app/optimize.py
- Convenience: verify + benchmark in sequence.

Reference serve command

vllm serve "$MODEL_PATH" \
  --served-model-name glm-5.2-nvfp4 \
  --host 0.0.0.0 --port "$PORT" \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --max-model-len 1000000 \
  --moe-runner-backend flashinfer_cutlass \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.92 \
  --async-scheduling

--max-num-batched-tokens is the chunked-prefill chunk size. Tune it: smaller (e.g. 2048–4096) favors stable ITL under concurrent decode; larger (e.g. 16384–32768) favors prefill throughput / TTFT. The hidden benchmark tests both serial and concurrent workloads — pick a value that balances both, or make it configurable and sweep it during dev.

Equivalent Modal skeleton:

import modal

app = modal.App("glm-5.2-nvfp4")

vllm_image = (
    modal.Image.from_registry(
        "nvidia/cuda:12.8.1-devel-ubuntu22.04", add_python="3.12"
    )
    .entrypoint([])
    .uv_pip_install(
        "vllm==0.23.0",
        "huggingface_hub[hf_transfer]==0.36.0",
    )
    .env({
        "HF_HUB_ENABLE_HF_TRANSFER": "1",
        "VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8": "1",
    })
)

hf_cache = modal.Volume.from_name("hf-cache", create_if_missing=True)
vllm_cache = modal.Volume.from_name("vllm-cache", create_if_missing=True)

MINUTES = 60

@app.function(
    image=vllm_image,
    gpu="B200:8",
    scaledown_window=10 * MINUTES,
    timeout=60 * MINUTES,
    volumes={
        "/root/.cache/huggingface": hf_cache,
        "/root/.cache/vllm": vllm_cache,
    },
)
@modal.concurrent(max_inputs=8)
@modal.web_server(port=8000, startup_timeout=45 * MINUTES)
def serve():
    import subprocess, os
    cmd = [
        "vllm", "serve", os.environ["MODEL_PATH"],
        "--served-model-name", "glm-5.2-nvfp4",
        "--host", "0.0.0.0", "--port", "8000",
        "--quantization", "modelopt_fp4",
        "--kv-cache-dtype", "fp8",
        "--tensor-parallel-size", "8",
        "--trust-remote-code",
        "--max-model-len", "1000000",
        "--moe-runner-backend", "flashinfer_cutlass",
        "--enable-chunked-prefill",
        "--max-num-batched-tokens", "8192",
        "--gpu-memory-utilization", "0.92",
        "--async-scheduling",
    ]
    subprocess.Popen(" ".join(cmd), shell=True)

What you can do

Tune chunked prefill — this is the primary single-node ITL lever. --enable-chunked-prefill + --max-num-batched-tokens controls how large a prefill chunk can interrupt an ongoing decode batch. Sweep it (2048 → 32768) against run_dev_bench.py to find the TTFT/ITL knee for the hidden workload mix.
Tune other vLLM scheduler flags: --gpu-memory-utilization, --max-num-seq, --max-num-batched-tokens, --block-size, --enable-prefix-caching, CUDA graphs (--enforce-eager vs --no-enforce-eager), --async-scheduling, --num-scheduler-steps.
Patch vLLM source in site-packages (find with python -c "import vllm; print(vllm.__path__[0])") — keep patches under /app/submission/ and reapply at startup.
Swap the NVFP4 checkpoint (Lorbus vs Mapika vs lukealonso) and pin revisions; they differ in which layers are quantized and calibration data.
Re-quantize from zai-org/GLM-5.2 with ModelOpt if no community checkpoint meets the correctness gate (slow; ~459 GB write).
Write custom Triton / CuTe DSL / FlashInfer kernels for the MoE FP4 GEMM or MLA attention and plug into vLLM.
Use a different serving backend (SGLang v0.5.13.post1+ with --quantization modelopt_fp4 --moe-runner-backend flashinfer_cutlass) if it beats vLLM on the benchmark — but the verifier invokes launch_server.sh, so the OpenAI-compatible HTTP interface must match.

What has to stay correct

The verifier runs your server and the BF16 baseline on the same hidden prompts (temperature=0, greedy) and compares outputs at the token level. The prompt set includes normal text, long-context (>32K), degenerate inputs, and adversarial edge cases. NVFP4 introduces small numeric drift; outputs are expected to match the BF16 reference within the gate's tolerance (typically exact greedy match on the hidden set, but confirm the gate's tolerance in verify_serving.py). If the correctness gate fails, your score is zero regardless of speed.

Watch for:

DSA sparse-attention indexer must stay BF16 — quantizing it breaks correctness.
MLA q/kv projections must stay BF16.
MTP layer (index 78) handling — confirm whether your checkpoint includes it and whether vLLM enables MTP (--enable-mtp or config) consistently with the baseline.

Resource constraints

Analyze GPU memory before changing config (nvidia-smi, server logs). vLLM pre-allocates KV cache, CUDA graph capture, and runtime buffers — these compete with the ~459 GB of weights. OOM kills the server and scores zero. At TP=8 you have ~135 GB/GPU headroom after weights; pick --gpu-memory-utilization and --max-model-len so KV cache fits with margin for CUDA graphs. First cold start downloads ~459 GB from HF — use a persistent modal.Volume for /root/.cache/huggingface so subsequent starts are fast.

Constraints

You CAN:

modify /app/submission/launch_server.sh and serve_modal.py, create helper files under /app/submission
modify vLLM source in site-packages (keep patches under /app/submission)
swap/pin the NVFP4 checkpoint and revision
install additional packages from the pre-built cache (no internet at runtime)

You CANNOT:

access or reference /tests/ or hidden verifier files
disable the timer daemon
access the internet at runtime (model must be cached or in the image/volume)
run on fewer than 8 B200s (weights do not fit)
use a second node or disaggregated prefill/decode (single 8×B200 node only)

How to work

# Quick sanity check
uv run --no-sync python verify_serving.py

# Full benchmark
uv run --no-sync python run_dev_bench.py

# Both in sequence
uv run --no-sync python optimize.py

To iterate fast, start the server manually and benchmark from another shell:

# Terminal 1
PORT=30000 MODEL_PATH=Lorbus/GLM-5.2-NVFP4 bash /app/submission/launch_server.sh

# Terminal 2
uv run --no-sync python run_dev_bench.py --no-server --port 30000

First run: pre-warm the HF cache volume by running a one-off Modal function that downloads the checkpoint once, so the served container doesn't pay the ~459 GB download on every cold start.

Time budget

Wall-clock budget is enforced by Harbor and exposed via a timer daemon:

cat /app/.timer/remaining_secs
cat /app/.timer/elapsed_secs
test -f /app/.timer/alert_30min
test -f /app/.timer/alert_10min
test -f /app/.timer/alert_5min

Keep a working /app/submission/launch_server.sh at all times. Leave time for a final correctness check and benchmark run.

Persisted submission state

Everything needed to run your submission at verification time must be in /app/submission/. Your entrypoint is /app/submission/launch_server.sh. If your final server behavior depends on helper scripts, copied site-packages patches, custom kernels, or config files, keep them under /app/submission/ and have launch_server.sh reapply or import them at startup. Direct edits in site-packages alone may be lost during replay.

Done. Submission prepared in /Users/root1/Downloads/projects/glm-5.2-inference-devin/.

What's there

serve_modal.py                 Modal app: CUDA 12.8 devel + vLLM 0.23.0 image,
                               persistent HF/vLLM volumes, B200:8, web_server,
                               prewarm() helper. Submission baked into image
                               via add_local_dir (modal 1.3 API).
prewarm_volume.py              Standalone checkpoint download helper.
submission/
  launch_server.sh             THE verifier entrypoint. execs vllm OpenAI server.
  patches/README.md            Drop-in vLLM site-packages overrides, reapplied
                               at startup.
README.md                      Tuning rationale + deploy/iterate flow.

Key design decisions

launch_server.sh mirrors the reference serve command exactly (quantization modelopt_fp4, --kv-cache-dtype fp8, TP=8, flashinfer_cutlass MoE runner, chunked prefill, async scheduling, CUDA graphs on) with every knob env-overridable. Defaults chosen to balance the hidden benchmark's serial + concurrent mix.
MAX_NUM_BATCHED_TOKENS=8192 — the primary single-node ITL/TTFT lever; the balanced knee for GLM-5.2's MoE+MLA on B200. Sweep 2048→32768 if dev bench shows a lean.
MTP off — matches the reference baseline; enabling it risks a token-level mismatch vs the BF16 greedy gate.
Checkpoint: Lorbus/GLM-5.2-NVFP4 default, revision pin via MODEL_REVISION (defaults to main; pin a commit hash for reproducible verification). Swap to Mapika/lukealonso via env if the correctness gate fails.
Patches: any vLLM source edits under submission/patches/vllm/... are copied over the installed package at startup, so they survive container replay.
FlashInfer env set both on the image and defensively in the shell script (VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 + the NVFP4 variant, name varies across vLLM patch versions).

Verified

bash -n on launch_server.sh — OK
ast.parse on both Python files — OK
serve_modal.py imports under modal 1.3.0.post1 (App, prewarm, serve, VOLUMES all build) — OK

Caveat

This machine has no /app, no timer daemon, no B200s, and no Modal creds — so I could not run verify_serving.py / run_dev_bench.py or actually prewarm the 459 GB volume. Those steps need to run in the real Harbor/Modal environment. The two things to do first once there:

uv run --no-sync python serve_modal.py prewarm — populate the HF volume once.
uv run --no-sync python verify_serving.py — sanity check the server starts and produces coherent output, then run_dev_bench.py to sweep MAX_NUM_BATCHED_TOKENS.

One thing worth flagging honestly: I could not confirm the exact NVFP4 env-flag name for vLLM 0.23.0 (the spec only documents the mxfp4/mxfp8 one). I set both candidates defensively; if the server logs warn an unknown env var, that's harmless, but if FP4 kernels fall back to a slower path you'll see it in the bench numbers and should grep vLLM 0.23.0 source for the correct flag name.