Sign inStart now ›
Inference /Benchmarks

Inference benchmarks

One vs two RTX 5070 Ti GPUs on the same model: 2× the throughput, up to 12× faster first word.

Coding assistant: Qwen2.5-Coder-7B

Qwen2.5-Coder-7B (AWQ, 4-bit) · vLLM · RTX 5070 Ti 16 GB · a ~3,200-token code prompt in, a 256-token completion out.

2.04×more completions86 → 177 a minute
6×faster first word16.3 s → 2.6 s at 2 requests/s
0dropped requestsAt 4 requests/s; one GPU dropped 1 in 4
Completions per minute
1 GPU2 GPUs
Completions per minute, by concurrent users. 1 GPU: 1 → 22/min, 2 → 36/min, 4 → 53/min, 8 → 71/min, 16 → 82/min, 32 → 83/min. 2 GPUs: 1 → 20/min, 2 → 40/min, 4 → 68/min, 8 → 100/min, 16 → 124/min, 32 → 146/min.040801201601248163283/min146/min

concurrent users

2 GPUs1 GPU

Wait for the first word · 2 requests/s · shorter is better

2 GPUs
2.6 s
1 GPU
16.3 s

Requests served · 4 requests/s

2 GPUs
100%
1 GPU
74%

One GPU turned away 59 of 229 requests (503); two served all 224.

Twice the developers at the same latency. With the first token under ~5 seconds, one GPU serves 8 concurrent users and two serve 16.

Concurrent users1 GPU · completions/min2 GPUs · completions/min1 GPU · first word2 GPUs · first word
122201.4 s1.7 s
236401.9 s1.5 s
453683.0 s2.0 s
8711005.1 s3.1 s
16821249.5 s5.2 s
328314614.1 s9.6 s

Document extraction: Qwen2.5-VL-7B

Qwen2.5-VL-7B (AWQ) · vLLM · RTX 5070 Ti 16 GB · one document page per request (~3,100 input tokens), JSON out.

2.01×more pages per hour4,200 → 8,460 pages/h
12×faster first word31.5 s → 2.5 s at 2 pages/s
$0.18per 1,000 pagesThe same on one and two GPUs
Pages per hour
1 GPU2 GPUs
Pages per hour, by concurrent users. 1 GPU: 1 → 1,140/h, 2 → 1,970/h, 4 → 2,990/h, 8 → 3,946/h, 16 → 4,186/h, 32 → not measured. 2 GPUs: 1 → 1,020/h, 2 → 1,984/h, 4 → 3,635/h, 8 → 5,287/h, 16 → 6,808/h, 32 → 7,234/h.02K4K6K8K124816324,186/h7,234/h

concurrent users

2 GPUs1 GPU

Wait for the first word · 2 pages/s · shorter is better

2 GPUs
2.5 s
1 GPU
31.5 s

At 2 pages a second one GPU falls behind and its queue grows; two keep up.

Users served with the first word under 6 s

2 GPUs
16 users
1 GPU
8 users

The answers don’t change. Extraction accuracy is the same on one and two GPUs; the second GPU only adds capacity. On invoices and insurance policy cards every field came back right on all 25 test documents of each kind (VIN aside, at 80%).

Concurrent users1 GPU · pages/h2 GPUs · pages/h1 GPU · first word2 GPUs · first word
11,1401,0201.7 s2.0 s
21,9701,9842.1 s2.0 s
42,9903,6353.3 s2.4 s
83,9465,2875.7 s3.4 s
164,1866,80811.0 s5.7 s
32not measured7,234not measured11.4 s

What this means for you

  • Add a GPU, double the throughput. Both models served about twice as much on two GPUs, with the same model, code and settings.
  • Same latency budget, twice the users. Under a ~5–6 s first-word target, one GPU serves 8 users and two serve 16.
  • The cost per answer stays flat. At full load a page costs the same on one GPU or two. The second GPU buys capacity at the same price per answer.
  • A single request is not faster on two GPUs. One request uses one copy of the model, so a second GPU adds capacity, not speed for one caller. On the coding model an answer starts in about 1.5 seconds either way.

How we measured

  • vLLM 0.18 through Ray Serve LLM on RTX 5070 Ti 16 GB cards, one card per replica. “2 GPUs” is a second replica; nothing else changed.
  • Closed loop: N users, each sending the next request once the last answer arrives (the charts and tables). Open loop: requests arrive at a fixed rate whether or not the server keeps up. That saturation test produced the “first word at 2 requests/s” and “requests served” figures.
  • Answers are fixed at 256 tokens so every request does the same work. Latencies are client-side medians and include about 0.8 s of network to the endpoint.
  • Dollar figures for document extraction use $0.75 per GPU-hour.
  • We measured one and two GPUs only, so we don’t claim the same doubling beyond two.
PreviousCalling your modelNext How it works