What it is
Folder: Qwen2.5-VL-32B-AWQ_vision_tensor_parallel_min_36gb_vram. A vision-language model for hard documents: degraded scans, handwriting, dense tables — the inputs where smaller vision models start guessing. The workhorse use case is extraction: send a page image with a prompt, get structured JSON back. It is a general tool; invoices, lab reports, and field notes are all the same problem to it.
The model is Qwen2.5-VL-32B-Instruct in 4-bit AWQ (~19 GB of weights, Apache-2.0), served in plain mode: one worker, one GPU, tensor_parallel_size: 1. Clients use the standard OpenAI vision protocol, so any OpenAI client that can send image content parts works unmodified.
The config
limit_mm_per_prompt caps images per request at 6. With the 32k context and fp8 KV there is headroom for multi-page batches; raise it if your documents run longer.Submit and call
Images travel as OpenAI content parts: a data URL for local files, or a plain URL the runtime can fetch.
Hardware
| VRAM min → rec | CPU | RAM | Free disk |
|---|---|---|---|
| 36 → 48 GB | 8 cores | 96 GB | ~46 GB (image + ~19 GB checkpoint × download headroom) |
In plain mode this wants one big card: an L40S or A40 (48 GB) runs it with room to spare, and an A100-80G fits even the fp16 checkpoint. At 32B parameters, AWQ is the right trade: the larger model more than compensates the 4-bit loss that hurts smaller models on fine print. Tier guidance: Hardware requirements.