# Abstract

We benchmark LLM inference on one Google TPU v6e-4 host (four chips, one VM) with four chips in one VM. We use vLLM 0.20.0 with the tpu-inference backend and an fp8 KV cache.

We test three Qwen3 models:

We measure three parts of inference: prefill, decode, and end-to-end online serving.

The MoE model is fastest in all three tests. It reaches 26,063 tok/s in prefill, compared with 19,714 tok/s for the dense 32B model. At batch size 1, its decode latency is 7.0 ms per token, compared with 17.9 ms for the dense 32B model. In online serving, it reaches 1.27 req/s and 1,303 output tok/s, compared with 0.88 req/s and 901 output tok/s for the dense 32B model.

The 4B model has good latency at low load, but it does not handle high concurrency well on one chip. Its stable serving capacity is about 0.45 req/s.

This is not a TPU-versus-GPU comparison. We did not run a GPU baseline. The goal is to report clear TPU v6e numbers using the same vLLM tools that users would run in practice.

# 1. Motivation

Public TPU data for LLM inference is limited. The only TPU submission Google has made to MLPerf Inference is SDXL image generation, not an LLM. The TPU LLM numbers is measured on Google’s JAX/JetStream stack rather than the vLLM path most practitioners deploy.

We measured the performance ourselves on one TPU v6e-4 host. The goal is to give simple, reproducible numbers for three common model types:

1. a small dense model,
2. a large MoE model,
3. a large dense model.

All comparisons are between these three models on the same TPU host. We do not compare against GPUs, prices, or total cost.

# 2. Experimental Setup

All measurements use one TPU v6e-4 host with four chips in one VM. The 4B model uses one chip. The 30B-A3B and 32B models use all four chips. For per-chip numbers, we divide the four-chip throughput by four.

We use vLLM 0.20.0, JAX 0.10.0, libtpu 0.0.40, Python 3.11 with the tpu-inference backend. The KV cache uses fp8_e5m2 in all runs. We report exact versions because TPU performance can change quickly across releases.

The key comparison is between Qwen3-30B-A3B and Qwen3-32B. They have similar total size, but the MoE model only uses 3B parameters per token, while the dense 32B model uses all 32B.

We run three tests.

For prefill, we use concurrency 1 and output length 1. We vary context length over 512, 1024, 2048, 4096, and 8192 tokens. This measures how fast the model processes the input prompt.

For decode, we fix the output length at 128 tokens and vary batch size over 1, 4, 16, and 64. We mainly report context length 1024, with context length 4096 as a longer-KV case. This measures the cost of generating tokens after prefill.

For online serving, we use vllm bench serve against the OpenAI-compatible endpoint. Input and output lengths are both 1024 tokens. We increase the request rate until the server saturates.

Metrics:

• TTFT: time to first token. This mainly measures prefill.
• TPOT: time per output token. This mainly measures decode.
• E2E latency: full request time.
• Prefill throughput: context length divided by p50 TTFT.
• Decode throughput: batch size × 1000 divided by p50 TPOT.
• Output throughput: generated tokens per second.
• Request throughput: completed requests per second.

We report p50 unless stated otherwise.

# 3. Results

## 3.1 Prefill

The MoE model has the highest prefill throughput.

The 4B result is limited by its single-chip setup and by max-num-batched-tokens=2048. For contexts above 2048, prefill is split into chunks, so the curve flattens.

## 3.2 Decode

At batch size 1, the MoE model is 2.5× faster than the dense 32B model: 7.0 ms versus 17.9 ms at context 1024. This follows active parameter count. Decode is memory-bandwidth-bound. The MoE model reads 3B active parameters per token; the dense 32B model reads all 32B.

The p99/p50 ratio reveals scheduling pressure at high batch sizes. The 30B-A3B at BS64 ctx 1024 has p99 TPOT of 152.7 ms against p50 of 48.1 ms (3.2×). The 4B at BS16 ctx 1024 has p99 TPOT of 647.3 ms against p50 of 41.4 ms (15.6×) — at this operating point the single-chip KV cache is under pressure and requests queue unevenly.

At context 4096, longer KV sequences reduce throughput across all models. The 30B-A3B at BS64 drops from 1,331 tok/s (ctx 1024) to 661 tok/s (ctx 4096). The dense 32B drops less in relative terms (1,096 to 775 tok/s) because its compute is already the bottleneck at ctx 1024. The 4B model saturates at BS16 for ctx 4096: p50 TPOT at BS64 equals BS16 (85.0 ms both), meaning additional concurrency gives no throughput gain on one chip.

## 3.3 Online serving

In online serving, higher request rate gives higher throughput, but also higher latency. Each model reaches a point where adding more requests no longer helps.

The MoE model (30B-A3B) has a stable working zone up to about 0.6 req/s, where p50 TPOT stays below 12 ms and p99 TTFT stays below 1.4 seconds. At 0.8 req/s the server is already queuing: p99 TTFT jumps to 6.3 seconds while p50 TPOT jumps from 11 ms to 19 ms. Saturation at ∞ rate gives tight p99/p50 TPOT (22.8 ms versus 22.3 ms), meaning the server is consistently loaded with no idle cycles.

The dense 32B model’s TPOT p50 is nearly flat from 0.15 to 0.6 req/s (18–21 ms), then doubles at saturation (43.2 ms). Its TTFT p99 is noisy across the sweep, suggesting occasional prefill bursts even at low load.

The 4B model holds stable from 0.2 to 0.45 req/s. At 0.45 req/s it sustains 0.40 req/s and 407 output tok/s, with TPOT p50 of 13.8 ms and E2E p50 of 14.6 seconds. Under unlimited load it collapses entirely: output throughput drops from 407 tok/s to 143 tok/s, p50 TPOT rises from 13.8 ms to 133.8 ms, and p50 E2E latency reaches 206.8 seconds. The p99 TTFT at unlimited load is 219,855 ms — 220 seconds. This is not a degraded state; it is a broken one. We treat the unlimited-load point as a single-chip concurrency limit, not useful serving capacity.

Across all three tests, the MoE model beats the dense 32B model at similar total parameter count:

The MoE model gives about 2.5× lower single-stream decode latency, about 1.4× higher serving capacity, and about 1.3× higher prefill throughput.

# 4. Discussion

Three points stand out.

First, active parameter count matters a lot for decode. The 30B-A3B MoE and the 32B dense model have similar total size, but very different active size. The MoE model uses 3B parameters per token. The dense model uses 32B. This explains the large decode gap: 7.0 ms versus 17.9 ms at batch size 1.

Second, the MoE model also wins in prefill. Its peak prefill throughput is 26,063 tok/s, or about 6.5k tok/s per chip. The dense 32B model reaches 19,714 tok/s, or about 4.9k tok/s per chip. On this host, the MoE model is faster in both prefill and decode.

Third, the small 4B model is limited by single-chip concurrency. It has good low-load latency, but one chip cannot absorb unlimited requests. Its practical serving capacity is about 0.45 req/s. To serve more traffic, the better path is to add chips or replicas, rather than pushing more concurrency onto one chip.

# 5. Conclusion

On one TPU v6e-4 host, using vLLM 0.20.0, tpu-inference, and an fp8_e5m2 KV cache, the large MoE model is the fastest of the three models we tested.

Compared with the dense 32B model, Qwen3-30B-A3B has:

• higher prefill throughput: 26,063 vs 19,714 tok/s,
• lower batch-size-1 decode latency: 7.0 vs 17.9 ms,
• higher serving capacity: 1.27 vs 0.88 req/s,
• higher output throughput at saturation: 1,303 vs 901 tok/s.

The 4B dense model has good latency at low load, but on one chip it is concurrency-limited. Its stable serving capacity is about 0.45 req/s.

# Limitations

These results are for one host only: four TPU v6e chips in one VM. We did not test multi-host scaling.

We did not run a GPU baseline, so this study makes no TPU-versus-GPU claim.

The online serving test uses one input/output shape: 1024 input tokens and 1024 output tokens. Other workloads may shift the balance between prefill and decode.

The results are version-specific. TPU support in vLLM changes quickly, so the numbers here apply to vLLM 0.20.0.

All runs use an fp8_e5m2 KV cache. We did not test other KV cache formats.

# Reproducibility

The main numbers are ./data.json

Figures are generated by ./figures/plot.py

Per-model results are at ./rawdata.zip