GLM 5.2 FP4 Inference on Modal B200s with vLLM

Output from command in shell cb737f:
total 0
drwxr-xr-x    5 root1  staff   160 Jun 21 23:28 .
drwxr-xr-x    7 root1  staff   224 Jun 21 23:28 ..
drwxr-xr-x  115 root1  staff  3680 Jun 21 23:28 deep-swe
drwxr-xr-x   20 root1  staff   640 Jun 21 23:28 frontier-swe
drwxr-xr-x   91 root1  staff  2912 Jun 21 23:28 terminal-bench-2

Exit code: 0

# Web Search Results for "GLM 5.2 open weights FP4 quantization vllm inference"

## 1. Lorbus/GLM-5.2-NVFP4 · Hugging Face
URL: https://huggingface.co/Lorbus/GLM-5.2-NVFP4

NVFP4-quantized checkpoint of zai-org/GLM-5.2(753B-param MoE with IndexShare sparse attention). Shrinks the BF16 checkpoint from ~1.37 TB to ~459 GB (≈3× smaller) so it fits on an 8-GPU Blackwell node (e.g. 8×96 GB) with room for long-context KV cache.
...
This is a community-built quantization of GLM-5.2 to NVIDIA's NVFP4 format (E2M1 + FP8 E4M3 scales, 16-element blocks), built using a per-shard streaming recipe derived from NVIDIA ModelOpt's`NVFP4_EXPERTS_ONLY_CFG` and TensorRT-LLM's DeepSeek-V3.2 precision strategy.
...
- Base model: GLM-5.2 (753B params, MoE, 78 transformer layers + 1 MTP layer at index 78, IndexShare sparse attention)
- Quantization: NVFP4 (E2M1 + FP8 E4M3, 16-element block scales)
- Block size: 16
- Quant method:`modelopt`
- Calibration: static per-block percentile-0.9999 scales (no forward-pass calibration — see Limitations)
- On-disk size: ~459 GB (NVFP4 packed weights + FP8 scales + BF16/FP32 kept layers)
- Compression: ~1.37 TB (BF16) → ~459 GB ≈ 3.0×
...
- Required: NVIDIA Blackwell GPUs (B200, GB200, or RTX PRO 6000 Blackwell). NVFP4 tensor cores are Blackwell-only.
- VRAM for weights: ~459 GB → minimum 6× 96 GB GPUs just to hold weights; 8 GPUs recommended for KV cache headroom.
- Tested config: single node, 8× RTX PRO 6000 Blackwell (96 GB each), tensor-parallel 8.
- Does NOT fit on a single GPU.
- Inference: TensorRT-LLM, vLLM, or SGLang with`modelopt` NVFP4 support.
...
### vLLM (v0.23.0+)
...
```
from vllm import LLM, SamplingParams

llm = LLM(
    model="Lorbus/GLM-5.2-NVFP4",
    quantization="modelopt",
    kv_cache_dtype="fp8",
    tensor_parallel_size=8,   # needs the full 8-GPU node
    trust_remote_code=True,
    max_model_len=1_000_000,
)

```
...
launch_server
...
model-path Lorbus/GLM-5.2-NVFP4
...
--quantization modelopt_fp4 \
...
--kv
...
cache-dtype fp8 \
...
8 \
...
load`modelopt` NVFP4
...
remote_code
...
This quantization was produced with a per-shard streaming pipeline that downloads GLM-5.2 shards one at a time fr...

## 2. zai-org/GLM-5.2 · Hugging Face
URL: https://huggingface.co/zai-org/GLM-5.2

0%
- Pure Open: An MIT open-source license — no regional limits, technical access without borders
...
GLM-5.2 supports deployment with the following frameworks. Feel free to try them out:
...
- SGLang(v0.5.13.post1+) — see cookbook
- vLLM(v0.23.0+) — see recipes
- Transformers(v0.5.12+) — see transformers docs
- KTransformers(v0.5.12+) — see tutorial
- Unsloth(v0.1.47-beta+) — see guide
- For deployment on the`Ascend NPU` platform, inference frameworks such as vLLM-Ascend, xLLM and SGLang are supported — see here.
...
## Model tree for zai-org/GLM-5.2
...
Use with llama.cpp Use with LM Studio Use with Jan Use with Ollama
...
56 models

## 3. Mapika/GLM-5.2-NVFP4 · Hugging Face
URL: https://huggingface.co/Mapika/GLM-5.2-NVFP4

NVFP4 (4-bit) quantization of zai-org/GLM-5.2, produced with NVIDIA TensorRT Model Optimizer 0.44.0. The MoE expert FFNs (routed + shared) are quantized to NVFP4; attention (MLA + the DeepSeek-style DSA lightning indexer), the router, and the LM head are kept in BF16. This shrinks the checkpoint from 1.5 TB → 410 GB (~3.7×) while retaining GSM8K accuracy within ~2 points of BF16.
...
GLM-5.2 is a`glm_moe_dsa` model: DeepSeek-V3.2-style MLA attention + DSA sparse-attention indexer, with a 256-routed-expert + 1-shared-expert MoE (8 experts/token), 78 layers, hidden 6144, vocab 154880.
...
## Quantization recipe
...
- Format: NVFP4 (FP4 weights + FP8 block scales), block/group size 16,`modelopt` producer.
- Quantized:`mlp.experts.*`(256 routed experts) and`mlp.shared_experts.*`.
- Kept in BF16 (excluded): all of`self_attn.*`— MLA projections (q/kv) and the DSA indexer — plus the MoE router (`mlp.gate`) and`lm_head`. The indexer and MLA attention must stay BF16: SGLang's`deepseek_v2` MLA path (used for`glm_moe_dsa`) cannot consume NVFP4 attention weights.
- KV cache: not quantized.
- Calibration: 512 samples × 2048 tokens from cnn_dailymail + nvidia/OpenCodeReasoning + nvidia/OpenMathReasoning.
...
Requires SGLang ≥ v0.5.13.post1 (the version that registers`GlmMoeDsaForCausalLM`).
...
```
docker run --runtime=nvidia --gpus '"device=0,1,2,3"' --ipc=host --shm-size=32g \
  -v /path/to/GLM-5.2-NVFP4:/model -p 30000:30000 \
  lmsysorg/sglang:v0.5.13.post1-cu130 \
  sglang serve --model-path /model --tp 4 \
    --quantization modelopt_fp4 --moe-runner-backend flashinfer_cutlass \
    --context-length 32768 --mem-fraction-static 0.85 \
    --tool-call-parser auto --trust-remote-code --host 0.0.0.0 --port 30000

```
...
GPU memory. The weights are ~410 GB, so per-GPU footprint depends on TP:
...
| Tensor parallel | Weights / GPU | Suitable GPUs |
| --- | --- | --- |
| `--tp 4` | ~110 GB | ≥128 GB cards — H200 (141 GB, tight KV), B200 / B300, MI300X (192 GB) |
| `--tp 8` | ~55 ...

## 4. amd/GLM-5.2-MXFP4 · Hugging Face
URL: https://huggingface.co/amd/GLM-5.2-MXFP4

amd/GLM-5.2-MXFP4 · Hugging Face

# Model Overview

Model Architecture: GLM-5.2

- Input: Text
- Output: Text

Model Optimizer: AMD-Quark(V0.11)

- Weight quantization: MOE-only (shared experts quantized), OCP MXFP4, Static
- Activation quantization: MOE-only, OCP MXFP4, Dynamic

This model was built with GLM-5.2 model by applying AMD-Quark for MXFP4 quantization.

# Model Quantization

The model was quantized from zai-org/GLM-5.2 using AMD-Quark. The weights and activations are quantized to MXFP4.

Quantization scripts:

```
cd Quark/examples/torch/language_modeling/llm_ptq/
  python quantize_quark.py \
      --model_dir zai-org/GLM-5.2 \
      --output_dir GLM-5.2-MXFP4 \
      --quant_scheme mxfp4 \
      --exclude_layers "*self_attn*" "*mlp.gate" "*lm_head" \
          "*mlp.gate_proj" "*mlp.up_proj" "*mlp.down_proj" \
          "*layers.78.*" \  # Exclude the MTP layer (layer 78)
      --file2file_quantization

```

# Deployment

### Use with SGLang/vLLM

This model can be deployed efficiently using the SGLang or vLLM backends.

## Evaluation

The model was evaluated on GSM8K benchmarks.

### Accuracy

| Benchmark | GLM-5.2 | GLM-5.2-MXFP4(this model) | Recovery |
| --- | --- | --- | --- |
| GSM8K (flexible-extract) | 94.09 | 93.93 | 99.8% |

### Reproduction

The GSM8K results were obtained using the`lm-evaluation-harness` framework, based on the Docker image`lmsysorg/sglang:v0.5.13.post1-rocm700-mi35x`, with SGLang pre-installed inside the image and lm-eval compiled and installed from source.

```
lm_eval --model sglang \
    --model_args pretrained=amd/GLM-5.2-MXFP4,tp_size=4 \
    --tasks gsm8k \
    --batch_size auto

```

The Docker image`rocm/vllm-dev:nightly_main_20260616` with vLLM pre-installed can also be used for reproducing using vLLM backend.

```
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FP8BMM=0
export VLLM_ROCM_USE_AITER_FP4BMM=0
lm_eval --model vllm \
    --model_args 'pretrained=amd/GLM-5.2-MXFP4,tensor_parallel_size=4,dtype=aut...

## 5. lukealonso/GLM-5.2-NVFP4 · Hugging Face
URL: https://huggingface.co/lukealonso/GLM-5.2-NVFP4

lukealonso/GLM-5.2-NVFP4 · Hugging Face

### Note: Quality checks still in progress

## Model Description

GLM-5.2-NVFP4 is an NVFP4-quantized version of zai-org/GLM-5.2, a 744B-parameter Mixture-of-Experts language model with 40B active parameters, 256 experts per MoE layer (8 activated per token), and DeepSeek Sparse Attention (DSA).

Quantized directly from the full BF16 checkpoint (zai-org/GLM-5.2, not the FP8 release, to NVFP4 (4-bit with blockwise FP8 scales per 16 elements) using NVIDIA Model Optimizer.

### What's quantized

Only the non-shared MoE expert MLP projections are quantized to NVFP4. Attention weights are left in BF16, in addition to the dense MLPs (layers 0-3) and the shared experts. Since the MoE expert weights constitute the vast majority of model parameters in an MoE architecture, this still yields significant memory savings.

Calibration uses natural top-k routing rather than forcing all experts to activate, so each expert's quantization scales reflect the token distributions it actually sees during inference. To compensate, calibration was run on a much larger number of samples than typical to ensure broad expert coverage through natural routing alone.

### Calibration dataset

Three calibration passes were run:

1. Coding pass — Agentic coding samples (tool calling, multi-turn code generation, function calling) with English and Chinese system prompts.
2. Broad pass — Large-scale diverse samples drawn from WildChat-NonToxic and LMSYS-Chat covering real user conversations across a wide range of topics and languages.
3. Deep pass — Long-context samples (>8K tokens) from coding and diverse sources to exercise deep-sequence expert activation patterns.

### Requirements

Hardware: 8x RTX PRO 6000 Blackwell 96GB (b12x MoE runner recommended)

### Community Testing

Downloads last month 27,157

Safetensors https://huggingface.co/docs/safetensors

Model size

432B params

Tensor type

BF16

·

F8_E4M3

·

U8

·

F32

·

Chat template

Files info

In...

# Web Search Results for "Modal B200 GPU deployment vLLM serve"

## 1. Run OpenAI’s gpt-oss model with vLLM | Modal Docs
URL: https://modal.com/docs/examples/gpt_oss_inference

vllm_image = (
    modal.Image.from_registry(
        "nvidia/cuda:12.8.1-devel-ubuntu22.04",
        add_python="3.12",
    )
    .entrypoint([])
    .uv_pip_install(
        "vllm==0.18.1",
        "huggingface_hub[hf_transfer]==0.36.0",
    )
    .env(  # fast Blackwell-specific MoE kernels
        {"VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8": "1"}
    )
)
...
We'll be downloading OpenAI's model from Hugging Face. We're running
the 20B parameter model by default but you can easily switch to the 120B model,
which also fits in a single H100 or H200 GPU.
...
## Build a vLLM engine and serve it
...
The function below spawns a vLLM instance listening at port 8000, serving requests to our model.
...
```python
app = modal.App("example-gpt-oss-inference")

N_GPU = 1
MINUTES = 60  # seconds
VLLM_PORT = 8000
...
@app.function(
    image=vllm_image,
    gpu=f"B200:{N_GPU}",
    scaledown_window=10 * MINUTES,  # how long should we stay up with no requests?
    timeout=30 * MINUTES,  # how long should we wait for container start?
    volumes={
        "/root/.cache/huggingface": hf_cache_vol,
        "/root/.cache/vllm": vllm_cache_vol,
        "/root/.cache/flashinfer": flashinfer_cache_vol,
    },
)
...
@modal.concurrent(max_inputs=MAX_INPUTS)
@modal.web_server(port=VLLM_PORT, startup_timeout=30 * MINUTES)
def serve():
    import subprocess

    cmd = [
        "vllm",
        "serve",
        MODEL_NAME,
        "--revision",
        MODEL_REVISION,
        "--served-model-name",
        MODEL_NAME,
        "llm",
        "--host",
        "0.0.0.0",
        "--port",
        str(VLLM_PORT),
        "--async-scheduling",  # reduces host overhead, but might not be compatible with all features
    ]

    # enforce-eager disables both Torch compilation and CUDA graph capture
    # default is no-enforce-eager. see the --compilation-config flag for tighter control
    cmd += ["--enforce-eager" if FAST_BOOT else "--no-enforce-eager"]

    # assume multiple GPUs are for splitting up la...

## 2. GPU acceleration | Modal Docs
URL: https://modal.com/docs/guide/gpu

gpu` argument
...
- `T4`
- `L4`
- `A10`
- `L40S`
- `A100`
- `A100-40GB`
- `A100-80GB`
- `RTX-PRO-6000`
- `H100`/`H100!`
- `H200`
- `B200`/`B200+`
...
For instance, to use a B200, you can use `@app.function(gpu="B200")`.
...
200
...
1,536 GB GPU RAM
...
more than
...
will usually result in larger wait times. These
GPUs are always attached to
...
same physical machine
...
## B200 GPUs
...
Modal's most powerful GPUs are the B200s,
NVIDIA's flagship data center chip for the Blackwell architecture.
...
To request a B200, set the `gpu` argument to `"B200"`
...
```python
@app.function(gpu="B200:8")
def run_deepseek():
    ...

```
...
Check out this example to see how you can use B200s to max out vLLM serving performance for LLaMA 3.1-8B.
...
### Opt-in upgrade to B300
...
Use `gpu="B200+"` to allow Modal to run requests on either B200 or B300 GPUs. B200+ is billed as B200, regardless of which GPU is used. Use this option only if your code is compatible with both types of GPUs. B300 requires CUDA version 13.0+. Use this to have access to a greater capacity pool automatically.

## 3. How to deploy vLLM
URL: https://modal.com/blog/how-to-deploy-vllm

Modal handles much of this heavy lifting for you. Modal consists of a Python SDK that wraps an ultra-fast container stack and multi-cloud GPU pool. It abstracts away container management, GPU scheduling, and autoscaling, letting you focus on customizing your model and serving logic.
...
Modal makes it easy to deploy open-source models on powerful GPUs. If you don’t have a Modal
...
, follow the two-line setup instructions
...
the Quickstart
...
following this tutorial
...
To serve vLLM, you need a container image—a packaged environment that includes Python, CUDA (NVIDIA’s GPU acceleration toolkit), and all the model dependencies. Modal’s SDK makes image definition easy, since you can define all your requirements in-line with your application code.
...
inference: Py
...
FlashInfer,
...
Hugging Face Hub
...
environment variable speeds up model downloads using Hugging Face’s optimized transfer utility
...
vLLM supports JIT compilation (just-in-time kernel optimization
...
and CUDA graph capture, both of
...
speed up inference after startup. However, enabling them increases initialization time, otherwise known as a cold start.
...
trade-off.
...
Here,`subprocess.Popen` launches`vllm serve` as a background process inside your container. When the model finishes loading, it begins accepting requests.
...
@app.
...
image=vllm_image,
    gpu=H100:1,
    scaledown_window=900,  # how long should we stay up with no requests?
    timeout=10 * MINUTES,  # how long should
...
wait for container start?
    volumes={
        "/root/.cache/huggingface": hf_cache_vol,
        "/root/.cache/vllm": vllm_cache_vol,
    },
)
...
@modal.web_server(port=8000, startup_timeout=10 * MINUTES)
...
def serve():
    import subprocess

    cmd = [
        "vllm",
        "serve",
        "--uvicorn-log-level=info",
        MODEL_NAME,
        "--revision",
        MODEL_REVISION,
        "--served-model-name",
        MODEL_NAME,
        "llm",
        "--host",
        "0.0.0.0",
        "--port",...

## 4. Introducing: B200s and H200s on Modal
URL: https://modal.com/blog/introducing-b200-h200

It never gets old: we are pleased to announce that the most powerful GPUs on the market are now available on Modal.
...
No “contact sales” button or asking for quota. B200s and H200s are accessible to anyone with a Modal account. It’s a one-liner to add them to any Modal Function.
...
```
import modal

app = modal.App()

@app.function(gpu="B200") # or "H200"
def run_big_model():
    # This will run on a Modal B200
```
...
Many workloads will see an immediate speedup between 2x and 4x when migrated from H100s to B200s. But the full performance benefits of B200s in particular will take some time to realize. For example, the popular open source LLM serving engine vLLM just stabilized support for Blackwell GPUs in version 0.9, released on Tuesday. And achieving the peak performance promised by the hardware will take even more work.
...
Below is one popular workload where our benchmarking showed significant gains for latency and throughput on B200s vs H200s. We used the latest version of vLLM to run the DeepSeek V3 Large Mixture-of-Experts Language Model in its native 8-bit precision to process 1000 tokens (about one page of text) as input and generate 128 tokens (e.g. a summary paragraph) as output. This model is too large to run on an 8xH100, but we benchmarked it on an 8xH200 and an 8xB200 on Modal.
...
- At 1 request per second, the median time-to-first-token is 2.5x faster on the B200s vs H200s.
- At a median time-to-first-token of 1 second, queries-per-second is 1.7x higher on the B200s vs H200s.
...
You can run vLLM on Modal B200s with this code sample.
...
Modal is the easiest way to deploy code to GPUs with no reservations or commitments. Our custom infrastructure allows us to spin up GPU containers running your code in less than a second. We help you efficiently autoscale your workloads to hundreds of GPUs, and you only ever pay for what you use.
...
Modal also comes with $30/month in free compute, so you can try B200s or H200s for free right now! Sign up today...

## 5. 06_gpu_and_ml/llm-serving/vllm_inference.py at main · modal-labs/modal-examples
URL: https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_inference.py

# In this example, we show how to run a vLLM server in OpenAI-compatible mode on Modal.
...
order of business is to define
...
will run in:
...
# the [
...
`](https://
...
docs/guide/custom-container
...
# vLLM can be installed with `uv pip`, since Modal [provides the CUDA drivers](https://modal.com/docs/guide/cuda).
...
-devel-
...
",
        "
...
.36.
...
",
    )
...
({"HF_
...
_HIGH_
...
# Native hardware support for FP8 formats in [Tensor Cores](https://modal.com/gpu-glossary/device-hardware/tensor-core)
# is limited to the latest [Streaming Multiprocessor architectures](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture),
# like those of Modal's [Hopper H100/H200 and Blackwell B200 GPUs](https://modal.com/blog/introducing-b200-h200).
...
# ## Build a vLLM engine and serve it
...
# The function below spawns a vLLM instance listening at port 8000, serving requests to our model.
# We wrap it in the [`@modal.web_server` decorator](https://modal.com/docs/guide/webhooks#non-asgi-web-servers)
# to connect it to the Internet.

# The server runs in an independent process, via `subprocess.Popen`, and only starts accepting requests
# once the model is spun up and the `serve` function returns.
...
app = modal.App("example-vllm-inference")
...
N_GPU = 1
MINUTES = 60  # seconds
VLLM_PORT = 8000
...
@app.function(
    image=vllm_image,
    gpu=f"H100:{N_GPU}",
    scaledown_window=15 * MINUTES,  # how long should we stay up with no requests?
    timeout=10 * MINUTES,  # how long should we wait for container start?
    volumes={
        "/root/.cache/huggingface": hf_cache_vol,
        "/root/.cache/vllm": vllm_cache_vol,
    },
)
...
@modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES)
def serve():
    import subprocess

    cmd = [
        "vllm",
        "serve",
        "--uvicorn-log-level=info",
        MODEL_NAME,
        "--revision",
        MODEL_REVISION,
        "--served-model-name",
        MODEL_NAME,
        "llm",...

Output from command in shell 24ae4b:
---
abs-module-cache-flags.md
abs-stepped-slices.md
actionlint-action-pinning-lint.md
adaptix-name-mapping-aliases.md
aiomonitor-task-snapshots-diff.md
anko-default-function-arguments.md
anko-typed-variable-bindings.md
arcane-drift-detection-baselines.md
arktype-json-schema-refs-dependencies.md
awilix-async-container-initialization.md
bandit-incremental-cache-control.md
bandit-interprocedural-taint-checks.md
bandit-structured-nosec-directives.md
boa-hierarchical-evaluation-cancellation.md
cattrs-partial-structuring-recovery.md
clack-async-autocomplete-options.md
claude-code-by-agents-recursive-delegation.md
cliffy-config-file-parsing.md
csstree-shorthand-expansion-compression.md
dasel-html-document-format.md
dateutil-rfc5545-timezone-interop.md
drizzle-orm-window-function-builders.md
dynamodb-toolbox-conditional-attribute-requirements.md
dynamodb-toolbox-lazy-recursive-schemas.md
effect-sse-httpapi-streaming.md
eicrud-keyset-pagination-cursor.md
etree-xml-diff-patch.md
expr-try-catch-errors.md
fastapi-deprecation-response-headers.md
fastapi-implicit-head-options.md
---
cranelift-codegen-opt.md
dart-style-haskell.md
dependent-type-checker.md
ffmpeg-swscale-rewrite.md
frogsgame-rl.md
git-to-zig.md
granite-mamba2-inference-optimization.md
inference-system-optimization.md
libexpat-to-x86asm.md
lua-native-compiler.md
modular-stack-wan21.md
notebook-compression.md
optimizer-design.md
pcqm4mv2-autoresearch.md
postgres-sqlite-wire-adapter.md
pyright-type-checking-optimization.md
revideo-perf-opt.md
workspace.md


Exit code: 0

<file-view path="/Users/root1/Downloads/utilities/benchmarks-prompts/frontier-swe/inference-system-optimization.md" start_line="1" end_line="165" total_lines="165">
  1|# Inference System Optimization
  2|
  3|You have an SGLang serving instance with Qwen3.5-4B on a B200 GPU.
  4|Your goal is to make it serve requests as fast as possible.
  5|
  6|The verifier launches your server using `/app/submission/launch_server.sh`, sends hidden
  7|requests, and measures end-to-end latency against a **well-tuned SGLang
  8|baseline**. The baseline already uses FP8 KV cache, native MTP speculative
  9|decoding, the V2 mamba scheduler, CUDA graphs, and optimal memory allocation.
 10|To beat it, you need to go beyond configuration — custom kernels, SGLang
 11|source modifications, or model surgery.
 12|
 13|Your score is the geometric-mean speedup across all hidden workloads.
 14|The verifier tests both sequential single-request latency and concurrent
 15|batched requests at multiple concurrency levels. Your optimisation must
 16|handle both efficiently.
 17|
 18|A correctness gate runs before speed is measured — your server's outputs
 19|must match the baseline's outputs on a large set of hidden prompts,
 20|including adversarial and degenerate inputs.
 21|
 22|## Model
 23|
 24|- **Qwen/Qwen3.5-4B** — a natively multimodal (text + vision) model,
 25|  4B parameters, bfloat16.
 26|- Weights are pre-downloaded at `/app/model`.
 27|- License: Apache 2.0.
 28|
 29|## Files
 30|
 31|- `/app/submission/launch_server.sh`
 32|  - **This is the main file you modify.** The verifier executes it to start
 33|    your candidate server. It receives `PORT` and `MODEL_PATH` as environment
 34|    variables.
 35|- `/app/submission/`
 36|  - This is the owned submission root.
 37|  - `launch_server.sh` and any helper files or copied SGLang/FlashInfer patches
 38|    needed at verification time must live under this tree.
 39|- `/app/run_dev_bench.py`
 40|  - Public dev benchmark. Launches your server, sends test requests, reports
 41|  latency. The hidden verifier uses different workloads and more iterations.
 42|- `/app/verify_serving.py`
 43|  - Quick sanity check that the server starts and produces coherent outputs.
 44|- `/app/optimize.py`
 45|  - Convenience script that runs verify + benchmark in sequence.
 46|
 47|## What you can do
 48|
 49|Anything. The design space is intentionally wide:
 50|
 51|- **Server configuration**: quantisation (fp8, int8, int4), torch.compile,
 52|  CUDA graphs, chunked prefill, memory allocation tuning, batch size limits
 53|- **Speculative decoding**: self-speculative, draft model, Medusa heads
 54|- **Custom kernels**: write Triton, TileLang, or CuTe DSL kernels and plug
 55|  them into SGLang
 56|- **SGLang source modifications**: modify the installed SGLang source directly.
 57|  Find it with: `python3 -c "import sglang; print(sglang.__path__[0])"`
 58|- **Model modifications**: quantise weights, prune layers, fuse operations
 59|- **Scheduling**: tune the request scheduler, batching strategy, preemption
 60|
 61|Pre-installed kernel tools:
 62|- **CUDA** — native CUDA kernels (full dev toolkit)
 63|- **Triton** — comes with PyTorch
 64|- **CuTe DSL / CuTile** — via CUTLASS (`import cutlass`)
 65|- **FlashInfer** — already powering SGLang's attention; source is modifiable
 66|- **TileLang** (`import tilelang`) — Python DSL for GPU kernels
 67|- **ThunderKittens (TK)** (`import thunderkittens`) — Stanford GPU kernel lib
 68|- **Helion** (`import helion`) — Meta's Python DSL for GPU kernels
 69|- **Triton-TLX** — Triton extension library
 70|
 71|SGLang internals and FlashInfer kernels are straightforward to modify
 72|directly.  Find SGLang source with:
 73|`python3 -c "import sglang; print(sglang.__path__[0])"`
 74|
 75|## What has to stay correct
 76|
 77|The verifier runs your server and the baseline on the same hidden prompts
 78|(temperature=0, greedy decoding) and compares outputs at the token level.
 79|The prompt set includes normal text, degenerate inputs, and adversarial
 80|edge cases. Your optimisations must preserve the model's output quality
 81|across all of them — the baseline operates in bfloat16, and your outputs
 82|are expected to match. If the correctness gate fails, your score is zero
 83|regardless of speed.
 84|
 85|## How to work
 86|
 87|Start here:
 88|
 89|```bash
 90|# Quick sanity check
 91|uv run --no-sync python verify_serving.py
 92|
 93|# Full benchmark
 94|uv run --no-sync python run_dev_bench.py
 95|
 96|# Or both in sequence
 97|uv run --no-sync python optimize.py
 98|```
 99|
100|These scripts launch the server from `/app/submission/launch_server.sh`, run their
101|checks, and then shut it down. Use `--no-server` if you already have a server
102|running.
103|
104|To iterate quickly, you can start the server manually in one terminal and
105|benchmark from another:
106|
107|```bash
108|# Terminal 1: start server
109|PORT=30000 MODEL_PATH=/app/model bash /app/submission/launch_server.sh
110|
111|# Terminal 2: benchmark against running server
112|uv run --no-sync python run_dev_bench.py --no-server --port 30000
113|```
114|
115|## Resource Constraints
116|
117|Analyze GPU memory constraints before changing server configuration
118|(`nvidia-smi`, server logs). SGLang pre-allocates memory for caches,
119|CUDA graph capture, and runtime buffers — these compete with model
120|weights for the available HBM. An out-of-memory error will kill the
121|server process and score zero. Plan your memory allocation strategy
122|and verify the server starts successfully before committing to a
123|configuration.
124|
125|## Constraints
126|
127|You CAN:
128|
129|- modify `/app/submission/launch_server.sh` and create any helper files under `/app/submission`
130|- modify SGLang source code in site-packages
131|- modify model weights (quantise, prune, etc.)
132|- use torch.compile, Triton, TileLang, CuTe DSL, custom CUDA kernels
133|- install additional packages from the pre-built cache (no internet)
134|
135|You CANNOT:
136|
137|- access or reference `/tests/` or hidden verifier files
138|- disable the timer daemon
139|- access the internet at runtime
140|
141|## Time Budget
142|
143|Your wall-clock budget is enforced by Harbor and exposed through a timer daemon:
144|
145|```bash
146|cat /app/.timer/remaining_secs
147|cat /app/.timer/elapsed_secs
148|test -f /app/.timer/alert_30min
149|test -f /app/.timer/alert_10min
150|test -f /app/.timer/alert_5min
151|```
152|
153|Keep a working `/app/submission/launch_server.sh` at all times. Leave time for a final
154|correctness check and benchmark run.
155|
156|## Persisted Submission State
157|
158|Everything needed to run your submission at verification time must be in
159|`/app/submission/`.
160|
161|Your entrypoint is `/app/submission/launch_server.sh`.
162|If your final server behavior depends on helper scripts, copied site-packages
163|patches, custom kernels, or config files, keep them under `/app/submission/`
164|and have `launch_server.sh` reapply or import them at startup. Direct edits in
165|site-packages alone may be lost during replay.
</file-view>

<file-view path="/Users/root1/Downloads/utilities/benchmarks-prompts/frontier-swe/granite-mamba2-inference-optimization.md" start_line="1" end_line="137" total_lines="137">
  1|# Granite Mamba2 Inference Optimization
  2|
  3|This task is a standalone port of the real Hugging Face Granite hybrid Mamba2 layer,
  4|using extracted weights from the pinned checkpoint `ibm-granite/granite-4.0-h-1b-base`.
  5|
  6|The provided `reference_impl.py` is a clean port of the HF `GraniteMoeHybridMambaLayer`
  7|`torch_forward` path. It does not call `transformers` in the actual forward path.
  8|Your job is to make `/app/submission/candidate_impl.py` faster without changing semantics.
  9|
 10|The model runs in **bfloat16** (`torch.bfloat16`) on CUDA. This affects optimization
 11|choices such as Triton intrinsics and accumulation precision.
 12|
 13|The verifier checks correctness against both the pinned `transformers`
 14|implementation and the fixed task reference, then measures speed relative to
 15|an optimized baseline that uses production-grade Triton kernels. Your score
 16|is the geometric-mean paired speedup of your candidate over this baseline.
 17|A score of 1.0 means matching production speed; >1.0 means beating it.
 18|
 19|## Fixed API
 20|
 21|The verifier imports `CandidateBlock` from `/app/submission/candidate_impl.py` and calls:
 22|
 23|```python
 24|block = CandidateBlock(weights, config, device=device, dtype=dtype)
 25|hidden_out, readout_logits, new_cache = block.forward(
 26|    hidden_states,
 27|    cache=None,
 28|    attention_mask=attention_mask,
 29|)
 30|```
 31|
 32|Keep that constructor and method signature stable.
 33|
 34|`readout_logits` is a last-token readout using the real Granite final norm plus tied
 35|embedding head. It is there for correctness checking, not because the task is a full LM.
 36|The latency benchmark times the Mamba layer core path (`torch_forward`), not the large
 37|readout head.
 38|
 39|## Files
 40|
 41|- `/app/reference_impl.py`
 42|  - Fixed standalone port of the real Granite Mamba layer.
 43|- `/app/vllm_ops/`
 44|  - Optimized Triton kernel implementations for SSM scan, state passing,
 45|    and related operations. These are building blocks you can use in your
 46|    candidate implementation.
 47|- `/app/submission/candidate_impl.py`
 48|  - Your implementation entrypoint.
 49|- `/app/submission/`
 50|  - This is the owned submission root.
 51|  - `candidate_impl.py` and any helper files or config it needs at verification
 52|    time must live under this tree.
 53|- `/app/task_fixtures.py`
 54|  - Fixed utilities: asset loading, cache structure, public workloads, tensor comparisons,
 55|    and the bridge to `transformers`.
 56|- `/app/prepare_assets.py`
 57|  - Build-time extractor for the pinned Granite checkpoint slice.
 58|- `/app/verify_api.py`
 59|  - Public parity check against both the reference and `transformers`.
 60|- `/app/run_dev_bench.py`
 61|  - Public latency benchmark on visible workloads, timing the core layer path
 62|    against the provided baseline.
 63|- `/app/optimize.py`
 64|  - Minimal loop that runs the public checks and benchmark.
 65|
 66|## What has to stay correct
 67|
 68|Before speed matters, the verifier checks:
 69|
 70|- hidden states
 71|- convolution cache state
 72|- recurrent SSM state
 73|- last-token readout logits
 74|- KL divergence between the candidate readout distribution and the reference readout distribution
 75|- the provided public baseline must still match the fixed reference on hidden cases
 76|
 77|The hidden set includes:
 78|
 79|- long prefill
 80|- short decode with cache reuse
 81|- variable-length padded batches
 82|
 83|If correctness fails, the score is zero.
 84|
 85|## How to work
 86|
 87|Start here:
 88|
 89|```bash
 90|uv run --no-sync python verify_api.py --device cuda
 91|uv run --no-sync python run_dev_bench.py --device cuda
 92|uv run --no-sync python optimize.py --device cuda
 93|```
 94|
 95|`verify_api.py` checks that the provided port matches `transformers` on public cases.
 96|`run_dev_bench.py` writes `/app/results/dev_benchmark.json`.
 97|The image prepares a `.venv` at build time, so `uv run --no-sync ...` works without internet.
 98|
 99|## Constraints
100|
101|You CAN:
102|
103|- edit `/app/submission/candidate_impl.py` and create helper files under `/app/submission`
104|- use `torch.compile`, Triton, custom kernels, or call into `transformers`
105|- change internal cache layout if the returned cache still exposes compatible
106|  `conv_state`, `ssm_state`, `has_previous_state`, and decode-position semantics
107|
108|You CANNOT:
109|
110|- rely on `/tests/` or hidden verifier files
111|- change the `CandidateBlock` constructor signature
112|- change the `forward(hidden_states, cache=None, attention_mask=None)` signature
113|
114|## Time Budget
115|
116|Your wall-clock budget is enforced by Harbor and exposed through a timer daemon:
117|
118|```bash
119|cat /app/.timer/remaining_secs
120|cat /app/.timer/elapsed_secs
121|test -f /app/.timer/alert_30min
122|test -f /app/.timer/alert_10min
123|test -f /app/.timer/alert_5min
124|```
125|
126|You have a fixed wall-clock budget for this task. Plan your work to make effective use of the available time.
127|
128|Keep a working `/app/submission/candidate_impl.py` at all times. Leave time for a final correctness run
129|and benchmark run.
130|
131|## Persisted Submission State
132|
133|Everything needed to run your submission at verification time must be in
134|`/app/submission/`.
135|
136|Your entrypoint is `/app/submission/candidate_impl.py`. Keep any helper modules
137|or config it needs under `/app/submission/`.
</file-view>

<file-view path="/Users/root1/Downloads/utilities/benchmarks-prompts/frontier-swe/workspace.md" start_line="1" end_line="610" total_lines="614">
  1|# Dependent Type Checker
  2|
  3|You are a software engineer specializing in programming language implementation.
  4|Your goal is to implement a **correct and fast** type checker for a dependently
  5|typed language (a subset of Martin-Löf Type Theory) in **Rust**.
  6|
  7|## Setup
  8|
  9|1. Your Rust workspace is `/app/type-checker/`. A scaffold `Cargo.toml` and
 10|   `src/main.rs` are provided as a starting point.
 11|2. Example input files are in `/app/examples/`.
 12|3. Check the task timer:
 13|   - `cat /app/.timer/remaining_secs`
 14|   - `cat /app/.timer/elapsed_secs`
 15|
 16|## Deliverable
 17|
 18|A Rust project at `/app/type-checker/` that compiles with `cargo build --release`
 19|and produces a binary that type-checks input files:
 20|
 21|```bash
 22|cd /app/type-checker && cargo build --release
 23|./target/release/type-checker /app/examples/identity.sexp
 24|```
 25|
 26|**Binary interface:**
 27|- Takes one or more file paths as positional arguments
 28|- Processes each file: parses commands, type-checks in order
 29|- Exits with code **0** if all commands in all files type-check successfully
 30|- Exits with code **1** if any command fails type-checking
 31|- Prints diagnostics to **stderr** (optional, for debugging)
 32|- Prints nothing to **stdout** (only exit codes matter)
 33|
 34|## Type Theory Specification
 35|
 36|Your checker must implement the following dependently typed language. All inputs
 37|are **pre-elaborated** — there are no implicit arguments, no tactics, no
 38|unification problems. Every term is fully annotated at the kernel level.
 39|
 40|### Core Constructs
 41|
 42|#### Universes (cumulative hierarchy)
 43|
 44|```
 45|Type 0 : Type 1 : Type 2 : ...
 46|```
 47|
 48|The universe hierarchy is cumulative: if `A : Type i` then also `A : Type j` for
 49|any `j >= i`. Universe levels are concrete natural numbers (no universe
 50|polymorphism variables — but universe levels in the input can be arbitrarily large).
 51|
 52|#### Dependent Function Types (Pi)
 53|
 54|```
 55|(Pi (x : A) B)          — dependent function type
 56|(lam x e)               — lambda abstraction (checked, not inferred)
 57|(app f a)               — function application
 58|```
 59|
 60|**Eta-conversion for functions:** Two functions `f` and `g` of type `(Pi (x : A) B)` are
 61|definitionally equal if `(app f x) ≡ (app g x)` for fresh `x`. Your conversion
 62|checker **must** implement eta for functions.
 63|
 64|**Eta-conversion for pairs:** A pair `(pair a b)` is definitionally equal to any
 65|term `p` of Sigma type if `a ≡ (fst p)` and `b ≡ (snd p)`. Your conversion
 66|checker **must** handle the case where one side of a comparison is a `pair`
 67|constructor by projecting the other side.
 68|
 69|#### Dependent Pair Types (Sigma)
 70|
 71|```
 72|(Sigma (x : A) B)       — dependent pair type
 73|(pair a b)              — pair constructor (checked against Sigma type)
 74|(fst p)                 — first projection (inferred from Sigma type of p)
 75|(snd p)                 — second projection (inferred from Sigma type of p)
 76|```
 77|
 78|#### Let Bindings
 79|
 80|```
 81|(let (x : A) v body)    — let binding: x : A := v in body
 82|```
 83|
 84|Let bindings are definitionally transparent: `x` unfolds to `v` during
 85|conversion checking (delta reduction).
 86|
 87|#### Type Annotations
 88|
 89|```
 90|(ann e A)               — annotate term e with type A (switches check → infer)
 91|```
 92|
 93|### General Inductive Types
 94|
 95|This is the most complex part of the specification. Your checker must support
 96|**user-defined inductive types** with parameters and indices, and must
 97|auto-generate their recursors (eliminators).
 98|
 99|#### Inductive Declarations
100|
101|An inductive type declaration has the form:
102|
103|```
104|(inductive Name
105|  (params ((p1 : P1) (p2 : P2) ...))
106|  (indices ((i1 : I1) (i2 : I2) ...))
107|  (sort (Type k))
108|  (constructors
109|    ((c1 : C1_type)
110|     (c2 : C2_type)
111|     ...)))
112|```
113|
114|Where:
115|- `Name` is the type name
116|- Parameters are fixed across all constructors (appear before the `:` in Lean notation)
117|- Indices vary per constructor (appear after the `:`)
118|- `sort` is the universe the type lives in
119|- Each constructor type must be a telescope ending in an application of `Name`
120|  to the parameters and appropriate indices
121|
122|**Example — Natural numbers:**
123|```
124|(inductive Nat
125|  (params ())
126|  (indices ())
127|  (sort (Type 0))
128|  (constructors
129|    ((zero : Nat)
130|     (succ : (Pi (n : Nat) Nat)))))
131|```
132|
133|**Example — Vectors (indexed by length):**
134|```
135|(inductive Vec
136|  (params ((A : (Type 0))))
137|  (indices ((n : Nat)))
138|  (sort (Type 0))
139|  (constructors
140|    ((vnil  : (app (app Vec A) zero))
141|     (vcons : (Pi (n : Nat) (Pi (x : A) (Pi (xs : (app (app Vec A) n)) (app (app Vec A) (app succ n)))))))))
142|```
143|
144|**Example — Propositional equality (indexed):**
145|```
146|(inductive Eq
147|  (params ((A : (Type 0)) (a : A)))
148|  (indices ((b : A)))
149|  (sort (Type 0))
150|  (constructors
151|    ((refl : (app (app (app Eq A) a) a)))))
152|```
153|
154|**Example — Fin (bounded naturals):**
155|```
156|(inductive Fin
157|  (params ())
158|  (indices ((n : Nat)))
159|  (sort (Type 0))
160|  (constructors
161|    ((fzero : (Pi (n : Nat) (app Fin (app succ n))))
162|     (fsuc  : (Pi (n : Nat) (Pi (i : (app Fin n)) (app Fin (app succ n))))))))
163|```
164|
165|#### Positivity Checking
166|
167|All inductive definitions must pass a **strict positivity check**. A type `T`
168|occurs strictly positively in a constructor argument type if:
169|- `T` does not occur at all, OR
170|- The argument type is exactly `T` applied to arguments, OR
171|- The argument type is `(Pi (x : A) B)` where `T` does not occur in `A` and
172|  `T` occurs strictly positively in `B`
173|
174|`T` must **not** appear in any negative (left-hand-side of Pi) position in
175|constructor argument types. Definitions failing positivity must be rejected.
176|
177|**Example of invalid definition (negative occurrence):**
178|```
179|(inductive Bad
180|  (params ())
181|  (indices ())
182|  (sort (Type 0))
183|  (constructors
184|    ((bad : (Pi (f : (Pi (x : Bad) Bad)) Bad)))))
185|```
186|This must be rejected because `Bad` appears to the left of `Pi` in `f`'s type.
187|
188|#### Constructor Typing
189|
190|After an inductive declaration, each constructor is available as a term. Given:
191|```
192|(inductive T (params ((p : P))) (indices ((i : I))) (sort (Type k))
193|  (constructors ((c : <type>))))
194|```
195|The constructor `c` has type `(Pi (p : P) <type>)` — parameters are prepended.
196|
197|#### Recursor (Auto-Generated Eliminator)
198|
199|After defining an inductive type `T`, a recursor `T-rec` is automatically
200|available. The recursor type is computed from the inductive definition:
201|
202|For an inductive `T` with parameters `(p1 : P1) ... (pn : Pn)`, indices
203|`(i1 : I1) ... (im : Im)`, living in `(Type k)`, and constructors
204|`c1 ... cj`:
205|
206|```
207|T-rec : (p1 : P1) -> ... -> (pn : Pn) ->
208|        (motive : (i1 : I1) -> ... -> (im : Im) -> T p1 ... pn i1 ... im -> Type l) ->
209|        <branch for c1> -> ... -> <branch for cj> ->
210|        (i1 : I1) -> ... -> (im : Im) ->
211|        (target : T p1 ... pn i1 ... im) ->
212|        motive i1 ... im target
213|```
214|
215|Each branch type corresponds to a constructor. For a constructor
216|`ci : (a1 : A1) -> ... -> (ak : Ak) -> T params indices`, the branch type is:
217|
218|```
219|(a1 : A1) -> ... -> (ak : Ak) ->
220|  <for each aj that is recursive: (ih_j : motive <indices of aj> aj)> ->
221|  motive <indices> (ci params a1 ... ak)
222|```
223|
224|A "recursive argument" is one whose type is (or returns) `T` applied to the
225|parameters.
226|
227|**Iota reduction:** Applying the recursor to a constructor head-reduces:
228|```
229|T-rec params motive branches... indices (ci params a1 ... ak)
230|  ~~>  branch_i a1 ... ak <recursive-ihs>
231|```
232|
233|Where each recursive IH is computed by applying the recursor recursively:
234|```
235|ih_j = T-rec params motive branches... <indices of aj> aj
236|```
237|
238|### Mutual Inductive Types
239|
240|Your checker must support **mutually recursive** inductive type declarations
241|using the `(mutual ...)` command:
242|
243|```
244|(mutual
245|  (inductive Even (params ()) (indices ()) (sort (Type 0))
246|    (constructors
247|      ((even-zero : Even)
248|       (even-succ : (Pi (n : Odd) Even)))))
249|  (inductive Odd (params ()) (indices ()) (sort (Type 0))
250|    (constructors
251|      ((odd-succ : (Pi (n : Even) Odd))))))
252|```
253|
254|All types in a mutual block are added to the context simultaneously before
255|checking any constructors, allowing cross-references.
256|
257|**Positivity checking for mutual blocks:** Each type `T` in the block must
258|occur strictly positively in ALL constructor argument types across ALL types
259|in the block (not just its own constructors).
260|
261|**Mutual recursors:** The recursor for a type `T` in a mutual block takes
262|one motive for EACH type in the block and one branch for EACH constructor
263|across ALL types. For the Even/Odd example:
264|
265|```
266|Even-rec : (P : Even -> Type l) -> (Q : Odd -> Type l) ->
267|           P even-zero ->
268|           ((n : Odd) -> Q n -> P (even-succ n)) ->
269|           ((n : Even) -> P n -> Q (odd-succ n)) ->
270|           (e : Even) -> P e
271|```
272|
273|**Iota for mutual recursors:** The IH for a recursive argument of a different
274|type uses that type's recursor with the SAME motives and branches:
275|
276|```
277|Even-rec P Q base step-e step-o (even-succ n)
278|  ~~> step-e n (Odd-rec P Q base step-e step-o n)
279|```
280|
281|### Universe Polymorphism
282|
283|Definitions and inductive types can be parameterized by **universe level
284|variables**. This is required for writing truly generic code (e.g., a
285|polymorphic identity function that works at any universe level).
286|
287|#### Universe Level Expressions
288|
289|```
290|level := natural                    ; concrete: 0, 1, 2, ...
291|       | identifier                 ; level variable: u, v, l, ...
292|       | (umax level level)         ; max of two levels
293|       | (usuc level)               ; successor (l + 1)
294|```
295|
296|#### Universe-Polymorphic Definitions
297|
298|```
299|(def-poly name ((u v ...)) type body)
300|```
301|
302|The level variables `u`, `v`, ... are bound in `type` and `body`. Within
303|the definition, `(Type u)` refers to the universe at level `u`.
304|
305|#### Universe-Polymorphic Inductives
306|
307|```
308|(inductive-poly Name ((u v ...))
309|  (params ((A : (Type u))))
310|  (indices ())
311|  (sort (Type u))
312|  (constructors ...))
313|```
314|
315|#### Instantiation
316|
317|When using a universe-polymorphic definition or inductive, provide concrete
318|level arguments with `(inst name (level1 level2 ...))`:
319|
320|```
321|(def-poly id ((u)) (Pi (A : (Type u)) (Pi (x : A) A))
322|  (lam A (lam x x)))
323|
324|; Apply at universe 0
325|(check (app (app (inst id (0)) Nat) zero) Nat)
326|
327|; Apply at universe 1 — works on types themselves
328|(check (app (app (inst id (1)) (Type 0)) Nat) (Type 0))
329|```
330|
331|Level expressions in `(Type ...)` must evaluate to concrete natural numbers
332|at the point of use. The checker substitutes level variables with their
333|concrete values and evaluates `umax`/`usuc` to produce a number.
334|
335|#### Universe-Polymorphic Recursors
336|
337|Universe-polymorphic inductives generate universe-polymorphic recursors.
338|The recursor gains an additional level parameter for the motive's target
339|universe:
340|
341|```
342|; List is polymorphic in universe u
343|(inductive-poly List ((u))
344|  (params ((A : (Type u))))
345|  (indices ())
346|  (sort (Type u))
347|  (constructors
348|    ((nil : (inst List (u) A))
349|     (cons : (Pi (x : A) (Pi (xs : (inst List (u) A)) (inst List (u) A)))))))
350|
351|; List-rec has an additional level param v for the motive universe
352|; (inst List-rec (u v)) : (A : Type u) -> (motive : List u A -> Type v) -> ...
353|```
354|
355|### Reduction and Conversion
356|
357|Your type checker must implement **definitional equality** via the following
358|reductions:
359|
360|- **Beta reduction:** `(app (lam x e) v) ~~> e[v/x]`
361|- **Delta reduction:** Unfold `let`-bound and top-level `def`-bound variables
362|- **Iota reduction:** Recursor applied to constructor (see above)
363|- **Eta for functions:** `f ≡ (lam x (app f x))` at Pi type
364|- **Eta for pairs:** `(pair a b) ≡ p` when `a ≡ (fst p)` and `b ≡ (snd p)`
365|
366|The conversion checker compares terms for definitional equality. It must be:
367|- **Correct:** Never equate terms that are not definitionally equal
368|- **Complete (for WHNF):** Always detect equality of terms that reduce to the
369|  same weak-head normal form
370|
371|### Bidirectional Type Checking
372|
373|The checker operates in two modes:
374|
375|**Inference mode** (computes a type):
376|- Variables: look up in context
377|- `(ann e A)`: check `A` is a type, check `e : A`, return `A`
378|- `(app f a)`: infer `f`, expect Pi type, check `a`, substitute
379|- `(fst p)`: infer `p`, expect Sigma, return `A`
380|- `(snd p)`: infer `p`, expect Sigma, return `B[fst p/x]`
381|- `(let (x : A) v body)`: check `v : A`, infer `body` with `x : A := v`
382|- `(Pi (x : A) B)`, `(Sigma (x : A) B)`: infer both, return universe
383|- `(Type n)`: return `(Type (n+1))`
384|- Constructors: return their declared type
385|- Recursors: return their computed type
386|
387|**Checking mode** (verifies against expected type):
388|- `(lam x e)`: expect Pi type `(Pi (x : A) B)`, check `e : B` under `x : A`
389|- `(pair a b)`: expect Sigma type `(Sigma (x : A) B)`, check `a : A` and `b : B[a/x]`
390|- Fall through to inference: infer type, check convertible with expected type
391|
392|### Universe Rules
393|
394|- `(Type i) : (Type (i+1))`
395|- `(Pi (x : A) B)` where `A : Type i` and `B : Type j` lives in `Type (max i j)`
396|- `(Sigma (x : A) B)` where `A : Type i` and `B : Type j` lives in `Type (max i j)`
397|- Cumulativity: if `e : Type i` then `e : Type j` for `j >= i`
398|
399|### Large Elimination Restriction
400|
401|Inductives in `Type 0` (a.k.a. `Prop`-like) with more than one constructor
402|are restricted: their recursor's motive must target `Type 0`. This prevents
403|information-theoretic unsoundness.
404|
405|Specifically, an inductive in `Type 0` may eliminate into any universe only if
406|it has **at most one constructor**. Otherwise, the recursor motive is forced
407|to `Type 0`.
408|
409|## Input Format
410|
411|Input files use an s-expression syntax. A file is a sequence of **commands**:
412|
413|```
414|; This is a comment (semicolon to end of line)
415|
416|; Define a new top-level term
417|(def name type body)
418|
419|; Universe-polymorphic definition
420|(def-poly name ((u v ...)) type body)
421|
422|; Declare an inductive type
423|(inductive Name
424|  (params (...))
425|  (indices (...))
426|  (sort (Type k))
427|  (constructors (...)))
428|
429|; Universe-polymorphic inductive
430|(inductive-poly Name ((u v ...))
431|  (params (...))
432|  (indices (...))
433|  (sort (Type level-expr))
434|  (constructors (...)))
435|
436|; Mutual inductive types
437|(mutual
438|  (inductive Name1 ...)
439|  (inductive Name2 ...))
440|
441|; Assert that a term has a given type (standalone check)
442|(check term type)
443|```
444|
445|### Term Grammar
446|
447|```
448|term := identifier                          ; variable or constructor/recursor
449|      | (ann term term)                     ; type annotation
450|      | (lam identifier term)              ; lambda abstraction
451|      | (app term term)                     ; application
452|      | (Pi (identifier : term) term)       ; dependent function type
453|      | (Sigma (identifier : term) term)    ; dependent pair type
454|      | (pair term term)                    ; pair constructor
455|      | (fst term)                          ; first projection
456|      | (snd term)                          ; second projection
457|      | (let (identifier : term) term term) ; let binding
458|      | (Type level)                        ; universe
459|      | (inst identifier (level ...))       ; instantiate poly def/inductive
460|
461|level := natural                            ; concrete: 0, 1, 2
462|       | identifier                         ; level variable: u, v
463|       | (umax level level)                 ; max
464|       | (usuc level)                       ; successor
465|```
466|
467|Identifiers: any sequence of alphanumeric characters, hyphens, underscores,
468|and primes that does not start with a digit. Examples: `x`, `Nat`, `Vec`,
469|`add-comm`, `x'`, `ih_1`.
470|
471|Natural numbers: sequences of digits (`0`, `1`, `42`, etc.).
472|
473|After an `(inductive T ...)` declaration:
474|- Each constructor name `c` is available as an identifier
475|- The recursor `T-rec` is available as an identifier
476|
477|Application is **binary** — multi-argument application is written as nested apps:
478|```
479|(app (app (app f a) b) c)
480|```
481|
482|### Example Input File
483|
484|```
485|; Natural numbers
486|(inductive Nat
487|  (params ())
488|  (indices ())
489|  (sort (Type 0))
490|  (constructors
491|    ((zero : Nat)
492|     (succ : (Pi (n : Nat) Nat)))))
493|
494|; Addition: add n m = Nat-rec (\_. Nat) m (\_ ih. succ ih) n
495|(def add (Pi (n : Nat) (Pi (m : Nat) Nat))
496|  (lam n (lam m
497|    (app (app (app (app Nat-rec
498|      (lam _ Nat))
499|      m)
500|      (lam k (lam ih (app succ ih))))
501|      n))))
502|
503|; Booleans
504|(inductive Bool
505|  (params ())
506|  (indices ())
507|  (sort (Type 0))
508|  (constructors
509|    ((true : Bool)
510|     (false : Bool))))
511|
512|; Propositional equality
513|(inductive Eq
514|  (params ((A : (Type 0)) (a : A)))
515|  (indices ((b : A)))
516|  (sort (Type 0))
517|  (constructors
518|    ((refl : (app (app (app Eq A) a) a)))))
519|
520|; Symmetry of equality
521|; sym A a b p = Eq-rec A a (\x _. Eq A x a) (refl A a) b p
522|(def sym
523|  (Pi (A : (Type 0)) (Pi (a : A) (Pi (b : A) (Pi (p : (app (app (app Eq A) a) b)) (app (app (app Eq A) b) a)))))
524|  (lam A (lam a (lam b (lam p
525|    (app (app (app (app (app (app (app Eq-rec A) a)
526|      (lam x (lam _eq (app (app (app Eq A) x) a))))
527|      (app (app refl A) a))
528|      b)
529|      p))))))
530|
531|; 2 + 2 = 4
532|(check
533|  (app (app refl Nat) (app (app add (app succ (app succ zero))) (app succ (app succ zero))))
534|  (app (app (app Eq Nat) (app (app add (app succ (app succ zero))) (app succ (app succ zero))))
535|          (app succ (app succ (app succ (app succ zero))))))
536|```
537|
538|## What You Can Use
539|
540|- Pre-installed Rust toolchain (stable): `rustc`, `cargo`
541|- Any crates from crates.io are **not** available (no internet). You must
542|  implement everything from scratch or use the Rust standard library.
543|- The scaffold project at `/app/type-checker/` has a basic `Cargo.toml`
544|
545|## What You Cannot Do
546|
547|- Download external code or crates (no internet access)
548|- Reference or read any scripts in `/tests/`
549|- Wrap or shell out to any external binary for type-checking
550|
551|## Verification
552|
553|The verifier checks two things:
554|
555|### Correctness
556|Your checker is tested against a collection of input files. It must correctly
557|accept well-typed files (exit 0) and reject ill-typed files (exit non-zero).
558|The test files cover all features described in this specification: core MLTT,
559|general inductives, mutual inductives, universe polymorphism, eta conversion,
560|positivity checking, and large elimination.
561|
562|### Throughput
563|After correctness is verified, your checker is timed on several workloads
564|of varying complexity. Faster is better. A naive implementation using direct
565|substitution will be slow on normalization-heavy inputs. Optimized approaches
566|(see below) can be significantly faster.
567|
568|## Performance Hints
569|
570|The main performance technique for dependent type checking is **Normalization
571|by Evaluation (NbE)**:
572|- Evaluate terms into a semantic domain (closures, not syntax)
573|- Quote semantic values back to syntax for comparison
574|- This avoids repeated substitution traversals
575|
576|Key optimization opportunities:
577|- **Arena allocation** instead of `Rc`/`Box` for terms
578|- **Glued evaluation**: track both evaluated and unevaluated forms
579|- **Approximate conversion**: try fast structural comparison before full normalization
580|- **Hash-consing** for common subterms
581|- **Lazy unfolding**: don't unfold definitions until needed for conversion
582|
583|## Behavioral Rules
584|
585|- Never stop to ask. Work autonomously until time runs out.
586|- Check time regularly: `cat /app/.timer/remaining_secs`
587|- Keep your project buildable at all times.
588|- Test against the example files frequently.
589|- Get correctness working first — optimize only after your checker is correct.
590|- Build incrementally: start with Pi/lam/app/Type, add Sigma, then inductives.
591|
592|## Time Budget
593|
594|You have a fixed wall-clock budget. Check the timer:
595|
596|```bash
597|cat /app/.timer/remaining_secs   # seconds remaining
598|cat /app/.timer/elapsed_secs     # seconds elapsed
599|test -f /app/.timer/alert_30min  # true when <=30 min remain
600|test -f /app/.timer/alert_10min  # true when <=10 min remain
601|```
602|
603|You have a fixed wall-clock budget for this task. Plan your work to make effective use of the available time.
604|
605|Plan your work around this budget. There is a lot to implement. A checker that handles core MLTT correctly
606|is much better than one that attempts everything but doesn't compile. Suggested
607|priority order:
608|1. Core type checker (Pi, lam, app, Type, let, ann, cumulative universes, Sigma)
609|2. General inductive types (declarations, constructors, auto-generated recursors, iota)
610|3. Eta for functions AND Sigma (pair projection), positivity checking
</file-view>

<file-view path="/Users/root1/Downloads/utilities/benchmarks-prompts/deep-swe/fastapi-implicit-head-options.md" start_line="1" end_line="19" total_lines="19">
  1|GET routes lack implicit HEAD controls, and FastAPI has no OPTIONS response exposing path metadata. 
  2|
  3|Add `auto_head` and `auto_options` to FastAPI/APIRouter constructors, decorators, `api_route`, `add_api_route`, and `include_router`. `auto_head` defaults on for GET routes; `auto_options` defaults off. 
  4|
  5|Direct app routes use app values as outermost defaults; included-router routes resolve omitted values by nearest non-omitted setting among route, include, and router. Explicit HEAD or OPTIONS operations win. 
  6|
  7|Implicit HEAD preserves the GET routes dependencies, status, headers, and validation behavior while returning no body. Implicit OPTIONS returns 200 JSON with `path`, ordered `methods`, and `operations`, where `operations` matches OpenAPI for that path excluding HEAD and OPTIONS, and sends `Allow`. 
  8|
  9|Use method order `GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, TRACE`. 
 10|
 11|Generate one implicit OPTIONS response per path when any operation enables it. 
 12|
 13|Public signatures exposing the new parameters must use FastAPIs `Annotated[..., Doc(...)]` style. 
 14|
 15|Define `ImplicitMethodTrackingMiddleware` in `fastapi/middleware/methods.py`; instance methods `get_stats()` and `reset_stats()` return a deep copy shaped `{full_path: {"head_hits": int, "options_hits": int}}`, clear counts, track implicit hits only, and ignore non-HTTP scopes. 
 16|
 17|Before editing, audit `applications.py` and `routing.py`, then trace HEAD/OPTIONS dispatch; after changes, verify precedence layers separately, repeated inclusion, method ordering, OpenAPI output, CORS preflight, docs surface, and middleware stats.
 18|
 19|IMPORTANT: Please work on this in a new branch from main and commit everything when you are done.
</file-view>

File created successfully at: /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md

# Web Search Results for "NVIDIA Dynamo inference framework disaggregated prefill decode"

## 1. Disaggregated Serving | NVIDIA Dynamo Documentation
URL: https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving

# Disaggregated Serving
...
The prefill and decode phases of LLM requests have different computation characteristics and memory footprints. Disaggregating these phases into specialized llm engines allows for better hardware allocation, improved scalability, and overall enhanced performance. For example, using a larger TP for the memory-bound decoding phase while a smaller TP for the computation-bound prefill phase allows both phases to be computed efficiently. In addition, for requests with long context, separating their prefill phase into dedicated prefill engines allows the ongoing decoding requests to be efficiently processed without being blocked by these long prefills.
...
Disaggregated execution of a request has three main steps:
...
1. Prefill engine computes prefill phase and generates KV cache
2. Prefill engine transfers the KV cache to decode engine
3. Decode engine computes decode phase.
...
The key to high-performance disaggregation is efficient KV transfer. Dynamo leverages NIXL to transfer KV cache directly from the VRAM of the prefill engine to the VRAM of the decode engine. The KV transfer is non-blocking, allowing GPU forward passes to continue serving other requests during the transfer.
...
The disaggregated serving flow is orchestrated by the `PrefillRouter`:
...
1. Worker Selection: The router selects a prefill worker using KV-aware routing (based on cache overlap scores and load) or simple load balancing.
2. Prefill Execution: The router sends the prefill request to the selected prefill worker. The prefill worker computes the KV cache and returns `disaggregated_params` containing backend-specific transfer metadata.
3. Decode Routing: The router injects the prefill result into the decode request, then routes to the decode worker.
4. KV Transfer: The decode worker uses the transfer metadata to coordinate with the prefill worker. NIXL handles the direct GPU-to-GPU transfer using the optimal available transport (NVLink, InfiniBand/UCX, etc.).
...
- ...

## 2. Disaggregated Serving | NVIDIA Dynamo Documentation
URL: https://docs.nvidia.com/dynamo/dev/components/router/disaggregated-serving

# Disaggregated Serving
...
Dynamo supports disaggregated serving where prefill (prompt processing) and decode (token generation) are handled by separate worker pools. When you register workers with `ModelType.Prefill`, the frontend automatically detects them and activates an internal prefill router.
...
## Automatic Prefill Router Activation
...
The prefill router is automatically created when:
...
1. A decode model is registered, for example via `register_model()` with `ModelType.Chat | ModelType.Completions`.
2. A prefill worker is detected with the same model name and `ModelType.Prefill`.
...
- Always disables active block tracking (`track_active_blocks=false`) since prefill workers do not perform decode.
- Seamlessly integrates into the request pipeline between preprocessing and decode routing.
- Falls back gracefully to decode-only mode if prefill fails or no prefill workers are available.
...
Key characteristics of the decode routing stage in disaggregated mode:
...
- Disables overlap scoring (`overlap_score_credit=0`) because decode routing should not chase prefix reuse.
- Disables KV reuse assumption (`assume_kv_reuse=false`) unless the backend can truly deduplicate transferred blocks.
- Disables prefill-token tracking (`track_prefill_tokens=false`) so decode-side load reflects decode work rather than already-completed prompt work.
...
When both workers are registered, requests are automatically routed.
...
```python
#
...
worker registration (in
...
_endpoint =
...
await register_model(
    model_input=ModelInput.Tokens,
    model_type=ModelType.Chat | ModelType.Completions,
    endpoint=decode_endpoint,
    model_name="meta-llama/Llama-2-7b-hf",
    # ... other parameters
)
...
await decode_endpoint.serve_endpoint(decode_handler.generate)
...
# Prefill worker registration (in your prefill worker)
prefill_endpoint = runtime.endpoint("dynamo.prefill.generate")

await register_model(
    model_input=ModelInput.Tokens,
    model_type=ModelType.Prefill,
    en...

## 3. Architecture Flow | NVIDIA Dynamo Documentation
URL: https://docs.dynamo.nvidia.com/dynamo/design-docs/architecture-flow

This diagram shows the NVIDIA Dynamo disaggregated inference system. Color-coded flows indicate different types of operations.
...
1. Request (S1): HTTP client sends API request to Frontend (OpenAI-compatible server on port 8000)
2. Preprocess (S2): Frontend preprocesses the request (applies chat template, tokenizes) and validates it
3. Route to Prefill (S3): PrefillRouter selects a prefill worker using KV-aware routing or load balancing
...
1. Prefill (S4): Prefill worker executes the prefill computation on the input tokens and generates KV cache
2. Return Metadata (S5): Prefill worker returns `disaggregated_params` containing backend-specific transfer metadata
...
1. Route to Decode (S6): PrefillRouter injects prefill result into decode request and routes to decode worker
2. KV Transfer (S7): Decode worker coordinates with prefill worker for direct GPU-to-GPU KV cache transfer via NIXL
...
1. Decode (S8): Decode worker generates tokens using the transferred KV cache
...
2. Response (S9): Generated tokens stream back through Frontend for post-processing (detokenization) and delivery to Client
...
- The `PrefillRouter` sits between the Frontend and workers, orchestrating disaggregated serving
- Selects prefill workers using KV-aware routing (cache overlap scores + load) or simple load balancing
- Injects transfer metadata into decode requests for KV cache coordination
...
-to-
...
- Transfer metadata exchanged via `disaggregated_params` in prefill response
- Backend-specific coordination: SGLang uses bootstrap connections, TRTLLM uses opaque state, vLLM uses block IDs
...
- Each worker maintains local KV cache in its GPU memory
...
- No shared storage bottlenecks—transfers are direct worker-to-worker via NIXL
...
- Non-blocking transfers allow GPU forward passes
...
continue during KV transfer
...
%% Worker Layer
    subgraph WL["<b>Worker Layer</b>"]
        %% Prefill Worker
        PrefillWorker["<b>Prefill Worker</b><br/><i>Computes KV Cache</i>"]
        S4[["<...

## 4. Dynamo Disaggregation: Separating Prefill and Decode for Enhanced Performance | NVIDIA Dynamo Documentation
URL: https://docs.nvidia.com/dynamo/v-0-8-1/design-docs/disaggregated-serving

The prefill and decode phases of LLM requests have different computation characteristics and memory footprints. Disaggregating these phases into specialized llm engines allows for better hardware allocation, improved scalability, and overall enhanced performance. For example, using a larger TP for the memory-bound decoding phase while a smaller TP for the computation-bound prefill phase allows both phases to be computed efficiently. In addition, for requests with long context, separating their prefill phase into dedicated prefill engines allows the ongoing decoding requests to be efficiently processed without being blocked by these long prefills.
...
However, not all requests’ prefill phases need to be computed in the remote prefill engine. If the prefill is short or the decode engine has a high prefix cache hit, often it is more efficient to prefill locally in the decode engine. The disaggregation design in Dynamo accounts for all these scenarios and features a flexible framework that delivers strong performance across various conditions.
...
When worker receives a request, it first decides if the prefill should be done locally or remotely using the disaggregated router and allocates the KV blocks. If prefilling remotely, it then pushes a remote prefill request to the prefill queue. After that, the prefill worker pulls from prefill queue, reads KV blocks with prefix cache hit from the worker, computes the prefill, and writes the computed KV blocks back to the worker. Finally, the worker completes the remaining decoding.
...
Not all requests’ prefill phases need to be computed in the remote prefill engine. Disaggregated router decides whether the prefill phase of a request should be computed locally and globally at runtime based on the prefill length and prefill queue status. Specifically, a request is sent to remote prefill engine if the following two conditions are met:
...
Prefill requests are computation bound (except for very short prefills) and should be execu...

## 5. Scale and Serve Generative AI  | NVIDIA Dynamo
URL: https://www.nvidia.com/en-us/ai/dynamo/

NVIDIA Dynamo is an open source, distributed inference-serving framework built to deploy models in multi-node environments at data center scale. It supports open source inference engines—including SGLang, NVIDIA TensorRT™ LLM, and vLLM—and simplifies the complexities of distributed serving by disaggregating inference phases across different GPUs, intelligently routing requests to the appropriate GPU to avoid redundant computation, and extending GPU memory through data caching to cost-effective storage tiers.
...
The GB200 NVL72 connects 72 GPUs via high-speed NVIDIA NVLink™, enabling low-latency expert communication critical for MoE reasoning models. NVIDIA Dynamo enhances efficiency through disaggregated inference, splitting prefill and decode phases across nodes for independent optimization. Together, GB200 NVL72 and NVIDIA Dynamo form a high-performance stack optimized for large-scale MoE inference.
...
### Disaggregated Serving
...
Separates large language model (LLM) context and generation phases across distinct GPUs, enabling independent GPU allocation and optimization to increase requests served per GPU.
...
Removes the guesswork from disaggregated servi… (137 chars truncated)
… (8 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/37ae7d2f/content.txt
</truncation_notice>

# Web Search Results for "vLLM disaggregated prefill decode throughput latency tradeoff"

## 1. Disaggregated Prefilling (experimental) - vLLM
URL: https://docs.vllm.ai/en/stable/features/disagg_prefill/

This page introduces you to the disaggregated prefilling feature in vLLM.
...
- Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately. Disaggregated prefilling put prefill and decode phase of LLM inference inside different vLLM instances. This gives you the flexibility to assign different parallel strategies (e.g. `tp` and `pp`) to tune TTFT without affecting ITL, or to tune ITL without affecting TTFT.
...
- Controlling tail ITL. Without disaggregated prefilling, vLLM may insert some prefill jobs during the decoding of one request. This results in higher tail latency. Disaggregated prefilling helps you solve this issue and control tail ITL. Chunked prefill with a proper chunk size also can achieve the same goal, but in practice it's hard to figure out the correct chunk size value. So disaggregated prefilling is a much more reliable way to control tail ITL.
...
Disaggregated prefill DOES NOT improve throughput.
...
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.

## 2. Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation | vLLM Blog
URL: https://vllm.ai/blog/2026-04-07-moriio-kv-connector

TL;DR: Prefill and decode fight over the same GPUs, causing ITL spikes under load. We show how to disaggregate them on a single 8-GPU MI300X node using AMD's MORI-IO connector — achieving 2.5x higher goodput compared to standard collocated serving on the same 8 GPUs, with stable token generation. Benchmark uses Qwen3-235B-A22B-FP8 at 8 req/s with 2000-token prompts and 1000-token outputs — see Table 3 and Experimental Details for full configuration.
...
When both phases share the same instance, they interfere with each other. Prefill requests can block dozens of ongoing decode streams, leading to visible stuttering, while decode workloads delay the scheduling of new prefills. The result is a system where neither phase runs efficiently nor predictably.
...
Standard serving fails because ITL concentrates in two clusters — the high-latency cluster at ~150ms far exceeds the 50ms threshold. Both disaggregated modes eliminate ITL violations entirely; their remaining failures are TTFT exceedances as request rate climbs. Write mode edges out read mode (73 vs 70) because concurrent proxy dispatch lowers TTFT, keeping more requests below the 1s threshold.
...
In a standard deployment, prefill and decode share the same vLLM engine and compete for scheduling within each batch. A single prefill — processing all input tokens in one forward pass — takes significantly longer than a decode step. Every decode request in the same batch waits for that prefill to finish before generating its next token, directly inflating ITL.
...
With disaggregation, your decode engine runs exclusively decode batches. No compute-intensive prefill jobs interrupt the step cadence, so ITL becomes stable and predictable regardless of how many new requests are entering the system. This benefit is identical in both read mode and write mode — the decode engine is isolated from prefill in either case.
...
Write mode eliminates Overhead 1. Because the proxy dispatches to both instances concurrently, the decode ...

## 3. Serving Agentic Workloads at Scale with vLLM x Mooncake | vLLM Blog
URL: https://vllm.ai/blog/2026-05-06-mooncake-store

TL;DR: Agentic workloads generate massive shared prefixes that are often recomputed across turns. By integrating Mooncake's distributed KV cache store into vLLM, we achieve 3.8x higher throughput, 46x lower TTFT, and 8.6x lower end-to-end latency on realistic agentic traces, while scaling nearly linearly to 60 GB200 GPUs.
...
Takeaway: we can
...
service as a set of isolated vLL
...
that provides both
...
aggregate capacity and cross-
...
cache hits.
...
Mooncake is an open-source, high-performance library for KV cache transfer and distributed storage. vLLM has already adopted Mooncake for prefill-decode (PD) disaggregation via the MooncakeConnector, using Mooncake's transfer engine to move KV caches between GPUs. Now, we take that integration one step further by building a distributed KV cache pool with Mooncake Store.
...
Prefill: The prefill instance prepares KV blocks for the PD connector and also stores them in the distributed KV cache pool through the store connector. For cache hits, vLLM queries all connectors and can recover matching prefixes from the Mooncake Store connector.
...
Decode: When the decode instance writes KV blocks into the distributed pool, they immediately become visible to prefill instances. Decode itself does not currently read from the pool: because vLLM schedules each request to both a prefill and a decode instance, the prefill instance loads any prefix KV blocks from the pool and forwards them to decode through the PD connector.
...
We ran the Kimi-2.5 NVFP4 model on GB200 nodes with PD disaggregation. The prefill instance used TP4, while the decode instance used DP8 + EP. We found that this configuration provided the best latency-throughput tradeoff.
...
Figure 4: vLLM with Mooncake Store vs. baseline on realistic Codex agentic traces (1P1D, 12 GB200 GPUs). The distributed KV cache pool improves throughput by 3.8x, reduces P50 TTFT by 46x, and reduces E2E latency by 8.6x, driven by a cache hit rate increase from 1.7% to 92.2%.
...
The ...

## 4. [Docs]: comprehensive rewrite of disaggregated prefilling (PD) documentation · Pull Request #38525 · vllm-project/vllm
URL: https://github.com/vllm-project/vllm/pull/38525

The existing disaggregated prefilling documentation is mostly example-oriented and lacks a clear explanation of system design,
deployment requirements, and production usage. Users may find it difficult to:
...
- Understand when to use PD disaggregation
- Deploy Prefill and Decode instances correctly
- Choose appropriate KV cache connectors
- Reason about latency trade-offs (TTFT vs ITL)
...
This PR introduces a comprehensive rewrite of the `disagg_prefill.md` documentation, covering:
...
- Clear explanation of Prefill–Decode (PD) disaggregation
- When to use it (TTFT, ITL, tail latency control)
- Explicit clarification that it optimizes latency, not throughput
...
- End-to-end pipeline:

Request → Prefill → KV Transfer → Decode
...
- Request routing
 - KV cache generation and transfer
 - Decode execution
...
- Detailed introduction of all supported connectors
- Connector comparison table
- Design considerations:
...
- latency vs throughput
 - network capabilities
 - memory constraints
 - system complexity
...
> **Review (commented):**
> 
> ## Code Review
> 
> This pull request significantly expands and reorganizes the documentation for Prefill-Decode Disaggregation in vLLM. It introduces detailed sections on the architecture, deployment requirements, and a comprehensive comparison of available connector implementations. A critical feedback point highlights a contradiction in the documentation regarding throughput improvements, which should be resolved to avoid user confusion.

## 5. [Misc] benchmark supports disaggregated prefill
URL: https://github.com/vllm-project/vllm/pull/16824

When conducting a benchmark, calculating throughput as (input + output) divided by the total time is generally correct. However, during disaggregated prefill, since prefill and decode do not affect each other, concurrency can be set very high.
...
But if the num_req is fixed at a value like 100 and the concurrency is set to 50, theoretically, when the prefill node finishes processing the last 50 requests, it will be idling while the decode node is still in the decode phase. Moreover, as the output length increases, the bubble (idle time) will become even larger.
...
Therefore, in the case of disaggregated prefill, the total throughput should be calculated as:
​​throughput = (num_input_token / pd_prefill_duration + num_output_token / total_duration)​

# Web Search Results for "NVIDIA Dynamo vLLM SGLang integration Blackwell"

## 1. Reference Guide | NVIDIA Dynamo Documentation
URL: https://docs.nvidia.com/dynamo/backends/sg-lang/reference-guide

The SGLang backend in Dynamo uses a modular architecture where `main.py` dispatches to specialized initialization modules based on the worker type. Each worker type has its own init module, request handler, health check, and registration logic.
...
Dynamo SGLang uses SGLang's native argument parser -- all SGLang engine arguments (e.g., `--model-path`, `--tp`, `--trust-remote-code`) are passed through directly. Dynamo adds its own arguments for worker mode selection, tokenizer control, and disaggregation configuration.
...
| Argument | Env Var | Default | Description |
| --- | --- | --- | --- |
| `--endpoint` | `DYN_ENDPOINT` | Auto-generated | Dynamo endpoint in `dyn://namespace.component.endpoint` format |
| `--use-sglang-tokenizer` | `DYN_SGL_USE_TOKENIZER` | `false` | [Deprecated] Use `--dyn-chat-processor sglang` on the frontend instead. See SGLang Chat Processor. |
| `--dyn-tool-call-parser` | `DYN_TOOL_CALL_PARSER` | `None` | Tool call parser (overrides SGLang's `--tool-call-parser`) |
| `--dyn-reasoning-parser` | `DYN_REASONING_PARSER` | `None` | Reasoning parser for chain-of-thought models |
| `--custom-jinja-template` | `DYN_CUSTOM_JINJA_TEMPLATE` | `None` | Custom chat template path (incompatible with `--use-sglang-tokenizer`) |
...
| `--embedding-worker` | `DYN_SGL_EMBEDDING_WORKER` | `false` | Run as embedding worker (also sets SGLang's `--is-embedding`) |
...
| `--multimodal-encode-worker` | `DYN_SGL_MULTIMODAL_ENCODE_WORKER` | `false` | Run as multimodal encode worker (frontend-facing) |
...
| `--multimodal-worker` | `DYN_SGL_MULTIMODAL_WORKER` | `false` | Run as multimodal LLM worker |
...
| `--image-diffusion-worker` | `DYN_SGL_IMAGE_DIFFUSION_WORKER` | `false` | Run as image diffusion worker |
| `--video-generation-worker` | `DYN_SGL_VIDEO_GENERATION_WORKER` | `false` | Run as video generation worker |
...
| `--disagg-config` | `DYN_SGL_DISAGG_CONFIG` | `None` | Path to YAML disaggregation config file |
| `--disagg-config-key` | `DYN_SGL_DISAGG_CONF...

## 2. Examples | NVIDIA Dynamo Documentation
URL: https://docs.nvidia.com/dynamo/latest/backends/sg-lang/examples

For quick start instructions, see the SGLang README. This document provides all deployment patterns for running SGLang with Dynamo, including LLMs, multimodal, and diffusion models, and Kubernetes deployment.
...
- etcd is optional but is
...
. You can also use `--discovery-backend file` to use file system based discovery.
- NATS is only needed when using KV routing with events (`--kv-events-config`). Use `--no-router-kv-events` on the frontend for prediction-based routing without NATS.
- On Kubernetes, neither is required when using the Dynamo operator (`DYN_DISCOVERY_BACKEND=kubernetes`).
...
Each launch script runs the frontend and worker(s) in a single terminal. You can run each command separately in different terminals for testing. For AI agents working with Dynamo, you can run the launch script in the background and use the `curl` commands to test the deployment.
...
## LLM Serving
...
### Aggregated Serving
...
The simplest deployment pattern: a single worker handles both prefill and decode.
...
```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/agg.sh
...
### Aggregated Serving with KV Routing
...
Two workers behind a KV-aware router that maximizes cache reuse:
...
```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/agg_router.sh
...
This launches the frontend with `--router-mode kv` and two workers with ZMQ-based KV event publishing.
...
### Disaggregated Serving
...
Separates prefill and decode into independent workers connected via NIXL for KV cache transfer. Requires 2 GPUs.
...
```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/disagg.sh
...
For details on how SGLang disaggregation works with Dynamo, including the bootstrap mechanism and RDMA transfer flow, see SGLang Disaggregation.
...
### Disaggregated Serving with KV-Aware Prefill Routing
...
Scales to 2 prefill + 2 decode workers with KV-aware routing on both pools. Requires 4 GPUs.
...
```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/disagg_router.sh

```
...
The front...

## 3. SGLang
URL: https://docs.nvidia.com/dynamo/backends/sg-lang.md

Dynamo SGLang integrates SGLang engines into Dynamo's distributed runtime, enabling disaggregated serving, KV-aware routing, and request cancellation while maintaining full compatibility with SGLang's native engine arguments. It supports LLM inference, embedding models, multimodal vision models, and diffusion-based generation (LLM, image, video).
...
sglang
...
from NGC. See
...
artifacts for the current
...
and CUDA variants.
...
#### Build from source inside upstream SGLang container
...
and launch the upstream
...
ang image, then
...
run --gpus all -it --rm \
    --network host --shm-size=10G \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    --ulimit nofile=65536:65536 \
    --ipc host \
    lmsysorg/sglang:v{sglang_version}
...
## Feature Support Matrix
...
| Feature | Status | Notes |
| --- | --- | --- |
| Disaggregated Serving | ✅ | Prefill/decode separation with NIXL KV transfer |
| KV-Aware Routing | ✅ | |
| SLA-Based Planner | ✅ | |
| Multimodal Support | ✅ | Image via EPD, E/PD, E/P/D patterns |
| Diffusion Models | ✅ | LLM diffusion, image, and video generation |
| Request Cancellation | ✅ | Aggregated full; disaggregated decode-only |
| Graceful Shutdown | ✅ | Discovery unregister + grace period |
| Observability | ✅ | Metrics, tracing, and Grafana dashboards |
| KVBM | ❌ | Planned |
...
### Multi-Node TP
...
SGLang supports multi-node tensor parallelism via the native `--dist-init-addr`, `--nnodes`, and `--node-rank` flags. See SGLang server arguments for the canonical reference; the same flags work with `python -m dynamo.sglang`. For a Kubernetes deployment example, see `disagg-multinode.yaml`.
...
### Kubernetes Deployment
...
You can deploy SGLang with Dynamo on Kubernetes using a `DynamoGraphDeployment`. For more details, see the SGLang Kubernetes Deployment Guide.

## 4. SGLang Multimodal | NVIDIA Dynamo Documentation
URL: https://docs.dynamo.nvidia.com/dynamo/user-guides/multimodality-support/sg-lang-multimodal

This document provides a comprehensive guide for multimodal inference using SGLang backend in Dynamo. SGLang multimodal supports EPD, E/PD, and E/P/D flows, with NIXL (RDMA) for zero-copy tensor transfer in disaggregated modes.
...
` | Vision encoder separate |
...
| ✅ | `multimodal_
...
KV cache via bootstrap |
...
| Component | Flag | Purpose |
| --- | --- | --- |
| Processor | `--multimodal-processor` | HTTP entry, OpenAI→SGLang conversion |
| Encode Worker | `--multimodal-encode-worker` | Vision encoder, embeddings generation |
| PD Worker | `--multimodal-worker` | Prefill + Decode with embeddings |
| Decode Worker | `--multimodal-worker --serving-mode=decode` | Entry point for disaggregation |
| Prefill Worker | `--multimodal-worker --serving-mode=prefill` | Called by Decode, bootstrap coordination |
...
- NIXL
...
: Embeddings transferred from Encoder → PD Worker using NIXL
...
Expansion: Single `<|image_pad|>` token replaced with N tokens based on embedding shape
...
model (AutoModel)
...
The `MultimodalEncodeWorker` downloads and encodes the image and passes the embeddings to the MultimodalWorker. The work complete event is sent via NATS, while the embeddings tensor is transferred via RDMA through the NIXL interface. The `MultimodalWorker` then prefills and decodes the prompt in the same engine, as in the LLM aggregated serving example. Only the processor is registered to the Dynamo frontend as an available endpoint. Workers do NOT register - they are internal components and communicate via NATS.
...
In models like Qwen2.5-VL, embeddings are only required during the prefill stage. The image embeddings are transferred via NIXL from the Encode Worker to the Decode Worker (the entry point for disaggregation), which then coordinates with the Prefill Worker. The Prefill Worker processes the embeddings and forwards the KV cache back to the Decode Worker for token generation.
...
vLLM
...
- SGLang: Frontend → Processor → Encode → Decode → Prefill (Decode is entry p...

## 5. SGLang
URL: https://docs.dynamo.nvidia.com/dynamo/backends/sg-lang.md

Dynamo SGLang integrates SGLang engines into Dynamo's distributed runtime, enabling disaggregated serving, KV-aware routing, and request cancellation while maintaining full compatibility with SGLang's native engine arguments. It supports LLM inference, embedding models, multimodal vision models, and diffusion-based generation (LLM, image, video).
...
sglang
...
from NGC. See
...
artifacts for the current
...
and CUDA variants.
...
#### Build from source inside upstream SGLang container
...
and launch the upstream
...
ang image, then
...
run --gpus all -it --rm \
    --network host --shm-size=10G \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    --ulimit nofile=65536:65536 \
    --ipc host \
    lmsysorg/sglang:v{sglang_version}
...
## Feature Support Matrix
...
| Feature | Status | Notes |
| --- | --- | --- |
| Disaggregated Serving | ✅ | Prefill/decode separation with NIXL KV transfer |
| KV-Aware Routing | ✅ | |
| SLA-Based Planner | ✅ | |
| Multimodal Support | ✅ | Image via EPD, E/PD, E/P/D patterns |
| Diffusion Models | ✅ | LLM diffusion, image, and video generation |
| Request Cancellation | ✅ | Aggregated full; disaggregated decode-only |
| Graceful Shutdown | ✅ | Discovery unregister + grace period |
| Observability | ✅ | Metrics, tracing, and Grafana dashboards |
| KVBM | ❌ | Planned |
...
### Multi-Node TP
...
SGLang supports multi-node tensor parallelism via the native `--dist-init-addr`, `--nnodes`, and `--node-rank` flags. See SGLang server arguments for the … (138 chars truncated)
… (4 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/9b715215/content.txt
</truncation_notice>

<file-view path="/Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md" start_line="1" end_line="260" total_lines="260">
  1|# GLM-5.2 NVFP4 Inference on Modal 8×B200
  2|
  3|Serve `zai-org/GLM-5.2` quantized to NVFP4 on a Modal 8×B200 node via vLLM,
  4|OpenAI-compatible, with a correctness gate against the BF16 reference and a
  5|throughput/latency benchmark. The verifier sends hidden prompts to your
  6|`/app/submission/launch_server.sh` and compares against a well-tuned
  7|vLLM NVFP4 baseline.
  8|
  9|## Model
 10|
 11|- **`zai-org/GLM-5.2`** — 753B-param Mixture-of-Experts (40B active), 256 routed
 12|  experts + 1 shared (8 activated/token), 78 transformer layers + 1 MTP layer
 13|  at index 78, DeepSeek-style MLA attention + DSA sparse-attention indexer
 14|  (`glm_moe_dsa` architecture), hidden 6144, vocab 154880.
 15|- BF16 checkpoint ~1.37 TB; NVFP4 quantized ~459 GB (~3× smaller).
 16|- **Quantization: NVFP4** — E2M1 4-bit weights + FP8 E4M3 block scales,
 17|  16-element blocks, produced with NVIDIA Model Optimizer (`modelopt`).
 18|  Only the MoE expert FFNs (routed + shared) are quantized. Attention
 19|  (MLA projections q/kv + DSA indexer), the MoE router (`mlp.gate`), and
 20|  `lm_head` are kept in BF16 — SGLang/vLLM's `deepseek_v2` MLA path cannot
 21|  consume NVFP4 attention weights.
 22|- **KV cache: FP8** (`--kv-cache-dtype fp8`) to maximize context headroom.
 23|- License: MIT (pure open).
 24|- Use a community NVFP4 checkpoint: `Lorbus/GLM-5.2-NVFP4`,
 25|  `Mapika/GLM-5.2-NVFP4`, or `lukealonso/GLM-5.2-NVFP4`. Pick one and pin
 26|  the revision in `launch_server.sh`.
 27|
 28|## Hardware
 29|
 30|- **Modal 8×B200** — `@app.function(gpu="B200:8")`. 8× B200 on one physical
 31|  machine, 8×192 GB HBM = 1536 GB total. NVFP4 tensor cores are Blackwell-only.
 32|- Weights ~459 GB → ~57 GB/GPU at TP=8, leaving ~135 GB/GPU for KV cache,
 33|  CUDA graphs, and runtime buffers. TP=8 is required (TP=4 needs ≥128 GB
 34|  cards and tightens KV; TP=8 is the safe default on B200).
 35|- Optional: `gpu="B200+"` allows Modal to upgrade to B300 (requires
 36|  CUDA 13.0+). Only enable if your image and vLLM build are CUDA 13-compatible.
 37|
 38|## Tools / stack
 39|
 40|- **vLLM v0.23.0+** — required for `modelopt` / `modelopt_fp4` NVFP4 support
 41|  and the `glm_moe_dsa` model registration. Pin a known-good version
 42|  (e.g. `vllm==0.23.0` or newer) in the Modal image.
 43|- **NVIDIA Model Optimizer** (`tensorrt-model-optimizer`) — only needed if you
 44|  re-quantize; for serving a pre-quantized HF checkpoint you do not need it.
 45|- **FlashInfer** — Blackwell MoE / FP4 kernels. Set
 46|  `VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1` (and the equivalent NVFP4 env flag
 47|  for your vLLM version) in the image env.
 48|- **huggingface_hub[hf_transfer]** — fast shard download (~459 GB).
 49|- **Modal SDK** — `modal.Image`, `modal.Volume` for HF/vLLM/flashinfer caches,
 50|  `@modal.web_server` to expose the vLLM HTTP server.
 51|- **CUDA 12.8+ devel image** as the base (`nvidia/cuda:12.8.1-devel-ubuntu22.04`
 52|  or CUDA 13.0 for B300). vLLM ships its own CUDA runtime; the devel base is
 53|  for any JIT kernel compilation.
 54|
 55|## Files
 56|
 57|- `/app/submission/launch_server.sh`
 58|  - **Main file you modify.** Verifier executes it with `PORT` and
 59|    `MODEL_PATH` env vars. It must start a vLLM OpenAI-compatible server
 60|    listening on `$PORT`.
 61|- `/app/submission/`
 62|  - Owned submission root. `launch_server.sh` plus any helper scripts,
 63|    vLLM patches, or env files must live under this tree.
 64|- `/app/serve_modal.py`
 65|  - Modal entrypoint defining the image, volumes, `@app.function(gpu="B200:8")`,
 66|    and `@modal.web_server` wrapper that `subprocess.Popen`s `vllm serve`.
 67|- `/app/run_dev_bench.py`
 68|  - Public dev benchmark: launches your server, sends test requests,
 69|    reports TTFT / throughput. Hidden verifier uses different workloads.
 70|- `/app/verify_serving.py`
 71|  - Sanity check that the server starts and produces coherent outputs.
 72|- `/app/optimize.py`
 73|  - Convenience: verify + benchmark in sequence.
 74|
 75|## Reference serve command
 76|
 77|```bash
 78|vllm serve "$MODEL_PATH" \
 79|  --served-model-name glm-5.2-nvfp4 \
 80|  --host 0.0.0.0 --port "$PORT" \
 81|  --quantization modelopt_fp4 \
 82|  --kv-cache-dtype fp8 \
 83|  --tensor-parallel-size 8 \
 84|  --trust-remote-code \
 85|  --max-model-len 1000000 \
 86|  --moe-runner-backend flashinfer_cutlass \
 87|  --async-scheduling
 88|```
 89|
 90|Equivalent Modal skeleton:
 91|
 92|```python
 93|import modal
 94|
 95|app = modal.App("glm-5.2-nvfp4")
 96|
 97|vllm_image = (
 98|    modal.Image.from_registry(
 99|        "nvidia/cuda:12.8.1-devel-ubuntu22.04", add_python="3.12"
100|    )
101|    .entrypoint([])
102|    .uv_pip_install(
103|        "vllm==0.23.0",
104|        "huggingface_hub[hf_transfer]==0.36.0",
105|    )
106|    .env({
107|        "HF_HUB_ENABLE_HF_TRANSFER": "1",
108|        "VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8": "1",
109|    })
110|)
111|
112|hf_cache = modal.Volume.from_name("hf-cache", create_if_missing=True)
113|vllm_cache = modal.Volume.from_name("vllm-cache", create_if_missing=True)
114|
115|MINUTES = 60
116|
117|@app.function(
118|    image=vllm_image,
119|    gpu="B200:8",
120|    scaledown_window=10 * MINUTES,
121|    timeout=60 * MINUTES,
122|    volumes={
123|        "/root/.cache/huggingface": hf_cache,
124|        "/root/.cache/vllm": vllm_cache,
125|    },
126|)
127|@modal.concurrent(max_inputs=8)
128|@modal.web_server(port=8000, startup_timeout=45 * MINUTES)
129|def serve():
130|    import subprocess, os
131|    cmd = [
132|        "vllm", "serve", os.environ["MODEL_PATH"],
133|        "--served-model-name", "glm-5.2-nvfp4",
134|        "--host", "0.0.0.0", "--port", "8000",
135|        "--quantization", "modelopt_fp4",
136|        "--kv-cache-dtype", "fp8",
137|        "--tensor-parallel-size", "8",
138|        "--trust-remote-code",
139|        "--max-model-len", "1000000",
140|        "--moe-runner-backend", "flashinfer_cutlass",
141|        "--async-scheduling",
142|    ]
143|    subprocess.Popen(" ".join(cmd), shell=True)
144|```
145|
146|## What you can do
147|
148|- Tune `launch_server.sh` and `serve_modal.py`: quantization flags, KV cache
149|  dtype, TP/PP/EP split, `--max-model-len`, `--gpu-memory-utilization`,
150|  `--max-num-seq`, `--enable-chunked-prefill`, CUDA graphs (`--enforce-eager`
151|  vs `--no-enforce-eager`), `--async-scheduling`.
152|- Patch vLLM source in site-packages (find with
153|  `python -c "import vllm; print(vllm.__path__[0])"`) — keep patches under
154|  `/app/submission/` and reapply at startup.
155|- Swap the NVFP4 checkpoint (`Lorbus` vs `Mapika` vs `lukealonso`) and pin
156|  revisions; they differ in which layers are quantized and calibration data.
157|- Re-quantize from `zai-org/GLM-5.2` with ModelOpt if no community checkpoint
158|  meets the correctness gate (slow; ~459 GB write).
159|- Write custom Triton / CuTe DSL / FlashInfer kernels for the MoE FP4 GEMM
160|  or MLA attention and plug into vLLM.
161|- Use a different serving backend (SGLang v0.5.13.post1+ with
162|  `--quantization modelopt_fp4 --moe-runner-backend flashinfer_cutlass`)
163|  if it beats vLLM on the benchmark — but the verifier invokes
164|  `launch_server.sh`, so the OpenAI-compatible HTTP interface must match.
165|
166|## What has to stay correct
167|
168|The verifier runs your server and the BF16 baseline on the same hidden prompts
169|(temperature=0, greedy) and compares outputs at the token level. The prompt set
170|includes normal text, long-context (>32K), degenerate inputs, and adversarial
171|edge cases. NVFP4 introduces small numeric drift; outputs are expected to match
172|the BF16 reference within the gate's tolerance (typically exact greedy match
173|on the hidden set, but confirm the gate's tolerance in `verify_serving.py`).
174|If the correctness gate fails, your score is zero regardless of speed.
175|
176|Watch for:
177|- DSA sparse-attention indexer must stay BF16 — quantizing it breaks correctness.
178|- MLA q/kv projections must stay BF16.
179|- MTP layer (index 78) handling — confirm whether your checkpoint includes it
180|  and whether vLLM enables MTP (`--enable-mtp` or config) consistently with the
181|  baseline.
182|
183|## Resource constraints
184|
185|Analyze GPU memory before changing config (`nvidia-smi`, server logs).
186|vLLM pre-allocates KV cache, CUDA graph capture, and runtime buffers — these
187|compete with the ~459 GB of weights. OOM kills the server and scores zero.
188|At TP=8 you have ~135 GB/GPU headroom after weights; pick
189|`--gpu-memory-utilization` and `--max-model-len` so KV cache fits with margin
190|for CUDA graphs. First cold start downloads ~459 GB from HF — use a persistent
191|`modal.Volume` for `/root/.cache/huggingface` so subsequent starts are fast.
192|
193|## Constraints
194|
195|You CAN:
196|
197|- modify `/app/submission/launch_server.sh` and `serve_modal.py`, create
198|  helper files under `/app/submission`
199|- modify vLLM source in site-packages (keep patches under `/app/submission`)
200|- swap/pin the NVFP4 checkpoint and revision
201|- install additional packages from the pre-built cache (no internet at runtime)
202|- use `gpu="B200+"` if your image is CUDA 13-compatible
203|
204|You CANNOT:
205|
206|- access or reference `/tests/` or hidden verifier files
207|- disable the timer daemon
208|- access the internet at runtime (model must be cached or in the image/volume)
209|- run on fewer than 8 B200s (weights do not fit)
210|
211|## How to work
212|
213|```bash
214|# Quick sanity check
215|uv run --no-sync python verify_serving.py
216|
217|# Full benchmark
218|uv run --no-sync python run_dev_bench.py
219|
220|# Both in sequence
221|uv run --no-sync python optimize.py
222|```
223|
224|To iterate fast, start the server manually and benchmark from another shell:
225|
226|```bash
227|# Terminal 1
228|PORT=30000 MODEL_PATH=Lorbus/GLM-5.2-NVFP4 bash /app/submission/launch_server.sh
229|
230|# Terminal 2
231|uv run --no-sync python run_dev_bench.py --no-server --port 30000
232|```
233|
234|First run: pre-warm the HF cache volume by running a one-off Modal function
235|that downloads the checkpoint once, so the served container doesn't pay the
236|~459 GB download on every cold start.
237|
238|## Time budget
239|
240|Wall-clock budget is enforced by Harbor and exposed via a timer daemon:
241|
242|```bash
243|cat /app/.timer/remaining_secs
244|cat /app/.timer/elapsed_secs
245|test -f /app/.timer/alert_30min
246|test -f /app/.timer/alert_10min
247|test -f /app/.timer/alert_5min
248|```
249|
250|Keep a working `/app/submission/launch_server.sh` at all times. Leave time
251|for a final correctness check and benchmark run.
252|
253|## Persisted submission state
254|
255|Everything needed to run your submission at verification time must be in
256|`/app/submission/`. Your entrypoint is `/app/submission/launch_server.sh`.
257|If your final server behavior depends on helper scripts, copied site-packages
258|patches, custom kernels, or config files, keep them under `/app/submission/`
259|and have `launch_server.sh` reapply or import them at startup. Direct edits in
260|site-packages alone may be lost during replay.
</file-view>

The file /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 26|  the revision in `launch_server.sh`.
 27|
 28|## Hardware
 29|
 30|- **Modal 8×B200, single node** — `@app.function(gpu="B200:8")`. 8× B200 on
 31|  one physical machine, 8×192 GB HBM = 1536 GB total. NVFP4 tensor cores are
 32|  Blackwell-only. This is the entire GPU budget — no second node, no
 33|  disaggregated prefill/decode pools. One vLLM engine, TP=8, collocated
 34|  prefill+decode.
 35|- Weights ~459 GB → ~57 GB/GPU at TP=8, leaving ~135 GB/GPU for KV cache,
 36|  CUDA graphs, and runtime buffers. TP=8 is mandatory: TP=4 would split the
 37|  node into two 4-GPU engines and leave only ~77 GB/GPU for KV — too tight for
 38|  GLM-5.2's context window, and you cannot run two TP=8 engines on 8 GPUs.
 39|- **No disaggregation.** Disaggregated prefill/decode requires a second
 40|  8×B200 node (one TP=8 prefill + one TP=8 decode with NIXL/Mooncake KV
 41|  transfer). On a single node, use **chunked prefill**
 42|  (`--enable-chunked-prefill` with a tuned `--max-num-batched-tokens` chunk
 43|  size) to get the ITL-stability benefit of disaggregation without splitting
 44|  the GPU pool.
 45|
 46|## Tools / stack
 47|
 48|- **vLLM v0.23.0+** — required for `modelopt` / `modelopt_fp4` NVFP4 support

The file /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 55|  for your vLLM version) in the image env.
 56|- **huggingface_hub[hf_transfer]** — fast shard download (~459 GB).
 57|- **Modal SDK** — `modal.Image`, `modal.Volume` for HF/vLLM/flashinfer caches,
 58|  `@modal.web_server` to expose the vLLM HTTP server.
 59|- **CUDA 12.8+ devel image** as the base
 60|  (`nvidia/cuda:12.8.1-devel-ubuntu22.04`). vLLM ships its own CUDA runtime;
 61|  the devel base is for any JIT kernel compilation.
 62|
 63|## Files
 64|
 65|- `/app/submission/launch_server.sh`

The file /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 81|  - Convenience: verify + benchmark in sequence.
 82|
 83|## Reference serve command
 84|
 85|```bash
 86|vllm serve "$MODEL_PATH" \
 87|  --served-model-name glm-5.2-nvfp4 \
 88|  --host 0.0.0.0 --port "$PORT" \
 89|  --quantization modelopt_fp4 \
 90|  --kv-cache-dtype fp8 \
 91|  --tensor-parallel-size 8 \
 92|  --trust-remote-code \
 93|  --max-model-len 1000000 \
 94|  --moe-runner-backend flashinfer_cutlass \
 95|  --enable-chunked-prefill \
 96|  --max-num-batched-tokens 8192 \
 97|  --gpu-memory-utilization 0.92 \
 98|  --async-scheduling
 99|```
100|
101|`--max-num-batched-tokens` is the chunked-prefill chunk size. Tune it: smaller
102|(e.g. 2048–4096) favors stable ITL under concurrent decode; larger (e.g.
103|16384–32768) favors prefill throughput / TTFT. The hidden benchmark tests both
104|serial and concurrent workloads — pick a value that balances both, or make it
105|configurable and sweep it during dev.
106|
107|Equivalent Modal skeleton:
108|
109|```python

The file /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
144|@modal.concurrent(max_inputs=8)
145|@modal.web_server(port=8000, startup_timeout=45 * MINUTES)
146|def serve():
147|    import subprocess, os
148|    cmd = [
149|        "vllm", "serve", os.environ["MODEL_PATH"],
150|        "--served-model-name", "glm-5.2-nvfp4",
151|        "--host", "0.0.0.0", "--port", "8000",
152|        "--quantization", "modelopt_fp4",
153|        "--kv-cache-dtype", "fp8",
154|        "--tensor-parallel-size", "8",
155|        "--trust-remote-code",
156|        "--max-model-len", "1000000",
157|        "--moe-runner-backend", "flashinfer_cutlass",
158|        "--enable-chunked-prefill",
159|        "--max-num-batched-tokens", "8192",
160|        "--gpu-memory-utilization", "0.92",
161|        "--async-scheduling",
162|    ]
163|    subprocess.Popen(" ".join(cmd), shell=True)
164|```
165|
166|## What you can do

The file /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
162|    ]
163|    subprocess.Popen(" ".join(cmd), shell=True)
164|```
165|
166|## What you can do
167|
168|- **Tune chunked prefill** — this is the primary single-node ITL lever.
169|  `--enable-chunked-prefill` + `--max-num-batched-tokens` controls how large
170|  a prefill chunk can interrupt an ongoing decode batch. Sweep it (2048 →
171|  32768) against `run_dev_bench.py` to find the TTFT/ITL knee for the hidden
172|  workload mix.
173|- Tune other vLLM scheduler flags: `--gpu-memory-utilization`,
174|  `--max-num-seq`, `--max-num-batched-tokens`, `--block-size`,
175|  `--enable-prefix-caching`, CUDA graphs (`--enforce-eager` vs
176|  `--no-enforce-eager`), `--async-scheduling`, `--num-scheduler-steps`.
177|- Patch vLLM source in site-packages (find with
178|  `python -c "import vllm; print(vllm.__path__[0])"`) — keep patches under
179|  `/app/submission/` and reapply at startup.
180|- Swap the NVFP4 checkpoint (`Lorbus` vs `Mapika` vs `lukealonso`) and pin
181|  revisions; they differ in which layers are quantized and calibration data.
182|- Re-quantize from `zai-org/GLM-5.2` with ModelOpt if no community checkpoint
183|  meets the correctness gate (slow; ~459 GB write).
184|- Write custom Triton / CuTe DSL / FlashInfer kernels for the MoE FP4 GEMM
185|  or MLA attention and plug into vLLM.
186|- Use a different serving backend (SGLang v0.5.13.post1+ with
187|  `--quantization modelopt_fp4 --moe-runner-backend flashinfer_cutlass`)
188|  if it beats vLLM on the benchmark — but the verifier invokes
189|  `launch_server.sh`, so the OpenAI-compatible HTTP interface must match.
190|
191|## What has to stay correct
192|
193|The verifier runs your server and the BF16 baseline on the same hidden prompts

The file /Users/root1/Downloads/utilities/benchmarks-prompts/glm-5.2-nvfp4-modal-b200-inference.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
216|`modal.Volume` for `/root/.cache/huggingface` so subsequent starts are fast.
217|
218|## Constraints
219|
220|You CAN:
221|
222|- modify `/app/submission/launch_server.sh` and `serve_modal.py`, create
223|  helper files under `/app/submission`
224|- modify vLLM source in site-packages (keep patches under `/app/submission`)
225|- swap/pin the NVFP4 checkpoint and revision
226|- install additional packages from the pre-built cache (no internet at runtime)
227|
228|You CANNOT:
229|
230|- access or reference `/tests/` or hidden verifier files
231|- disable the timer daemon
232|- access the internet at runtime (model must be cached or in the image/volume)
233|- run on fewer than 8 B200s (weights do not fit)
234|- use a second node or disaggregated prefill/decode (single 8×B200 node only)
235|
236|## How to work
237|
238|```bash