DocsGPT itself is light: it runs on 1 vCPU / 2 GiB for one person and 4 vCPU / 8 GiB for a department of 25 to 50 concurrent users with a cloud model. The GPU is sized by the model you serve with Ollama, vLLM, llama.cpp, TGI, SGLang or LMDeploy: an 8B model on a 12 to 16 GB card for 1 to 5 users, a 24 GB card with vLLM for 5 to 25, a 48 to 80 GB card for 25 to 100. A large bank runs a classification pipeline built by the DocsGPT team fully air-gapped on its existing NVIDIA T4 GPUs.
docsgpt 1 vCPU / 2 GiB to 16 vCPU / 16 GiB · gpu by model, 12 GB to 8 × 80 GB vram
Last updated:
Inside the network boundary: the inference engine (Ollama, vLLM, llama.cpp, TGI, SGLang or LMDeploy) serves the LLM on the GPU or CPU; DocsGPT calls it over its API, computes embeddings on CPU by default (FastEmbed on ONNX Runtime, granite-311m), stores vectors in the vector store and files in storage. Nothing crosses the boundary.
What the platform needs before any model, with the language model on a cloud endpoint. Query embedding, retrieval and streaming run here; only generation runs on the GPU.
1 vCPU, 2 GiB RAM, 10 GB disk is enough for one person; 1 GiB RAM fails (the stack boots but the first ingest is killed out of memory and retried forever; no document is ever indexed). 2 vCPU, 4 GiB is the comfortable size.
| CPU | RAM | Disk | Measured | |
|---|---|---|---|---|
| Solo / evaluation1 concurrent user | 1 vCPU | 2 GiB | 10 GB | Boots in 17 s; 0.4 s to first token; a 30-page PDF ingests in 158 s. 1 GiB is not enough. |
| Small teamup to 10 concurrent users | 2 vCPU | 4 GiB | 10 GB | 10 users at 0.5 s to first token, 25 at 1.6 s; ingest twice as fast as on 1 vCPU. |
| Department25 to 50 concurrent users | 4 vCPU | 8 GiB | 10 GB | 25 users at 0.4 s, 50 at 1.7 s (1.0 s with 2 API workers); a 250-stream burst completes with no errors. Tesseract OCR fits on the same box. 2 API workers for 50 users |
| Company100 concurrent users | 16 vCPU | 16 GiB | 10 GB | 100 users at 0.2 to 0.7 s to first token, zero errors. 4 API workers; raise Postgres max_connections |
| Company250 concurrent users · estimated | 16 vCPU (32 vCPU for sub-second) | 32 GiB | 10 GB | 747 of 750 requests ok at 2.1 s median to first token on 16 vCPU. more API workers, a dedicated embeddings worker |
| Company500 concurrent users · estimated | 2 to 3 nodes × 16 vCPU, or 48 vCPU | 48 GiB | 10 GB | A 500-stream burst completes with no errors on 16 vCPU at 12 s median to first token; steady 500 users needs about 3× the CPU of the 250 tier. |
A concurrent user asks, streams a ~5 s answer, waits 2 s and asks again — more aggressive than real use, so user counts are conservative. Docker Compose standalone stack (backend, worker, Redis, PostgreSQL) on x86-64 Linux. Disk is for the slim stack; add 3.5 to 4.5 GB for the docling parser and 2 to 7 GB per local model. Measured 2026-09-16 on DocsGPT 0.21.0.
CPU rule of thumb
1 vCPU
per chat request per second · ~7 active or 25–50 named users
Stack boot
17 s
1 vCPU / 2 GiB · fresh compose up
Image download
~1 GB
slim stack · 2 to 3.2 GB on disk
Per worker process
1.5 GB
RAM after its first ingest (embedding session)
Ingest scales linearly with cores and competes with chat for the same CPU: a 30-page text PDF in 158 / 75 / 40 s on 1 / 2 / 4 vCPU. One document uses one worker process, so above 4 vCPU extra cores buy parallel ingests, not faster ones. Tesseract OCR costs about 4 s per scanned page per core, including chunking and embedding, and no extra RAM beyond the ingest peak. For heavy scan volumes give the parsing worker its own cores rather than a bigger single box, or use a DeepSeek-OCR endpoint on a card with ≥ 12 GB VRAM (7.3 GB fp16 weights; 1.8 GB at INT4) (about 4.5 s per page on an H100, parallel documents at no extra wall time).
Fit by VRAM for chat over documents, by concurrent users. Weights at Q4_K_M plus about 1 GB of KV cache per concurrent 8k-context request for an 8B model, roughly double for 14B and quadruple for 32B.
llama3.1:8b (Q4_K_M) via Ollama on an NVIDIA H100 80 GB generates 143 tokens/s in 13.7 GB with 8 parallel slots at 8k context; through DocsGPT one user gets the first token in 1.8 s, 10 users wait 6.5 s and 25 users 19 s. The queue is Ollama's slot count, not the GPU: the H100 sat mostly idle while users waited. Ollama is fine up to about its OLLAMA_NUM_PARALLEL in concurrent users; beyond that use vLLM (continuous batching), not a bigger card.
| Model class | GPU | Serving stack | |
|---|---|---|---|
| 1 to 5 users | 8B (5 to 6 GB) | one 12 to 16 GB card (RTX 4070/4080; T4 slow) or Apple Silicon 16 GB+ | Ollama, OLLAMA_NUM_PARALLEL=4, OLLAMA_CONTEXT_LENGTH=8192 |
| 5 to 25 users | 8B to 14B (6 to 10 GB) | one 24 GB card (L4, A10G, RTX 4090, RTX 5090) | vLLM; Ollama measured at 6.5 s to first token for 10 users and 19 s for 25, even on an H100 |
| 25 to 100 users | 8B to 32B (6 to 20 GB + KV cache) | one 48 to 80 GB card (L40S, A100, H100) | vLLM with continuous batching and FP8 KV cache |
| 100 to 500 users | 8B to 70B | 2 to 8 × 80 GB (H100, H200), tensor or data parallel | vLLM behind a load balancer; DocsGPT scaled as in the cloud-LLM tiers |
Fit by VRAM. A 70B model at Q4 needs about 42 GB. Whatever the model, DocsGPT itself still needs the CPU and RAM from the footprint table.
Qwen3.8-Flash-Next
Ollama · http://ollama:11434
GLM-5.3-Flash
vLLM · http://gpu-01:8000/v1
GPT-5.6 Sol
DocsGPT · api key · via Arc53
+ Add model bring your own · openai-compatible
Pick the class that fits your memory
Weights plus KV cache for your context and concurrency; the selector below does the arithmetic. Prefer the largest class that fits with headroom for concurrent sessions.
8B → 12–16 GB · 14B → 24 GB · 32B → 48 GB · 70B → 80 GB
Serve it with the engine for your user count
Ollama up to about its parallel-slot count (roughly 8 users); vLLM with continuous batching beyond that. Register the endpoint in Settings › Models and switch models without a restart.
ollama · vllm · OLLAMA_NUM_PARALLEL · OLLAMA_CONTEXT_LENGTH
Measure on your own hardware
Run a benchmark suite of your questions against two or three candidate models with docsgpt-cli bench: assertions on sources and content, latency limits, A/B. The bank case chose its model this way from five-plus candidates. Part of the free on-prem review.
docsgpt-cli bench ./suite · junit output
Pick your card or host to see which model classes fit, at what quantisation and context.
| GPU / host | Memory | Model class | Quantisation | Engine | Context | Fit |
|---|---|---|---|---|---|---|
| NVIDIA T4 | 16 GB | 7–8B | Q4 / INT4 | Ollama, llama.cpp, vLLM | 8k | fitsWhat the bank runs air-gapped on its T4s (fine-tuned model, classification); slow for interactive chat |
| NVIDIA T4 | 16 GB | 12–14B | Q4 | llama.cpp | 4k | tightShort context; latency rises with concurrency |
| NVIDIA T4 | 16 GB | 30B+ | any | — | — | does not fitDoes not fit in 16 GB |
| NVIDIA RTX 4070 / 4080 | 12–16 GB | 7–8B | Q4 | Ollama, llama.cpp | 8k | fits1 to 5 users with Ollama |
| NVIDIA RTX 4070 / 4080 | 12–16 GB | 12–14B | Q4 | llama.cpp | 4k | tightSingle user |
| NVIDIA L4 | 24 GB | 7–8B | FP8 / Q4 | vLLM, Ollama | 16k | fits5 to 25 users with vLLM |
| NVIDIA L4 | 24 GB | 12–14B | Q4 | vLLM, llama.cpp | 8k | fitsLow concurrency |
| NVIDIA L4 | 24 GB | 30–34B | Q4 | llama.cpp | 4k | tightSingle session |
| NVIDIA A10 / A10G | 24 GB | 7–8B | FP16 / Q4 | vLLM, TGI | 16k | fits5 to 25 users with vLLM |
| NVIDIA A10 / A10G | 24 GB | 12–14B | Q4 | vLLM, llama.cpp | 8k | fitsLow concurrency |
| NVIDIA A10 / A10G | 24 GB | 30–34B | Q4 | llama.cpp | 4k | tight |
| NVIDIA RTX 4090 / 5090 | 24–32 GB | 7–8B | FP8 / Q4 | vLLM, Ollama | 16k | fits5 to 25 users with vLLM |
| NVIDIA RTX 4090 / 5090 | 24–32 GB | 12–14B | Q4 | vLLM, llama.cpp | 8k | fitsLow concurrency |
| NVIDIA RTX 4090 / 5090 | 24–32 GB | 30–34B | Q4 | llama.cpp | 4k | tightSingle session |
| NVIDIA A100 | 40 GB | 7–14B | FP16 | vLLM, TGI, SGLang | 32k | fits |
| NVIDIA A100 | 40 GB | 30–34B | Q4 / INT8 | vLLM | 8k | fits |
| NVIDIA A100 | 40 GB | 70B | Q4 | llama.cpp | 4k | tightConsider two GPUs with tensor parallelism |
| NVIDIA L40S | 48 GB | 7–34B | FP16 / FP8 | vLLM, TGI, SGLang | 32k | fits25 to 100 users with vLLM |
| NVIDIA L40S | 48 GB | 70B | Q4 | vLLM, llama.cpp | 8k | tightLow concurrency |
| NVIDIA A100 | 80 GB | 7–34B | FP16 | vLLM, TGI, SGLang, LMDeploy | 32k | fits25 to 100 users with vLLM |
| NVIDIA A100 | 80 GB | 70B | INT8 / Q4 | vLLM, LMDeploy | 8k | fitsLow concurrency at INT8 |
| NVIDIA H100 | 80 GB | 7–34B | FP16 / FP8 | vLLM, TGI, SGLang, LMDeploy | 32k+ | fits8B Q4_K_M via Ollama: 143 tokens/s; 1.8 s to first token for 1 user, 6.5 s for 10 — Ollama's slot queue, so vLLM for teams |
| NVIDIA H100 | 80 GB | 70B | FP8 / INT8 | vLLM, SGLang | 16k | fits |
| Apple M-series | 16 GB unified | 7–8B | Q4 (GGUF) | Ollama, llama.cpp | 8k | fitsSingle user; a small office at most |
| Apple M-series | 16 GB unified | 12B+ | any | — | — | does not fitNo room beside the OS |
| Apple M-series | 32 GB unified | 7–8B | Q4 (GGUF) | Ollama, llama.cpp | 8k | fitsSingle user; evaluation and small teams |
| Apple M-series | 32 GB unified | 12–14B | Q4 | Ollama, llama.cpp | 4k | tight |
| Apple M-series | 64 GB unified | 7–14B | Q4–Q8 | Ollama, llama.cpp | 8k | fitsSingle user |
| Apple M-series | 64 GB unified | 30–34B | Q4 | llama.cpp | 4k | tight |
| Apple M-series | 128 GB unified | 7–34B | Q4–Q8 | Ollama, llama.cpp | 8k | fitsSingle user |
| Apple M-series | 128 GB unified | 70B | Q4 | llama.cpp | 4k | tightSlow generation; single user |
| CPU-only | 32 GB RAM | 1–4B | Q4 (GGUF) | llama.cpp, Ollama | 4k | tight3B via Ollama on 4 vCPU: 4.3 tokens/s, 42 s to first token — a demo, not a deployment |
| CPU-only | 32 GB RAM | 7–8B | Q4 | llama.cpp | 4k | does not fitFits in RAM; 8 to 15 tokens/s even on ~16 fast cores, unusable below |
| CPU-only | 64 GB RAM | 1–4B | Q4 (GGUF) | llama.cpp, Ollama | 8k | tightDemo only |
| CPU-only | 64 GB RAM | 7–8B | Q4 | llama.cpp | 4k | does not fitFits in RAM; too slow for interactive use |
| CPU-only | 64 GB RAM | 12B+ | any | — | — | does not fitLatency unsuitable for interactive use |
Fit by memory: parameters × bytes per weight, plus KV cache for the context and concurrency shown. Latency figures, where given, are measured 2026-09-16.
Enter your concurrent users and where the model runs for a first instance shape. An engineer confirms it in the free on-prem review.
DocsGPT itself needs no GPU: its local embeddings (FastEmbed on ONNX Runtime, granite-311m on new installs) run on CPU in every posture, and the footprint table above is entirely CPU. Only generation wants one. llama3.2:3b (Q4) via Ollama on 4 vCPU / 8 GiB, alongside the stack, generates 4.3 tokens/s and evaluates prompts at 29 tokens/s. Through DocsGPT one user waits 42 s for the first token, because the retrieved context is prompt-evaluated first, and 48 s for the answer; three users wait up to 127 s with Ollama pegging all four cores.
A demo, not a deployment: CPU-only local models are not viable for real use on anything under about 16 fast cores, and even then an 8B model lands around 8 to 15 tokens/s. Use CPU-only inference to evaluate the product, then put the model on a card from the table above, or on a cloud endpoint under your own key while documents stay local.
llama.cpp · Ollama · GGUF Q4 · granite-311m on CPU
| Measured | |
|---|---|
| Model | llama3.2:3b (Q4) via Ollama |
| Host | 4 vCPU / 8 GiB, beside the DocsGPT stack |
| Generation | 4.3 tokens/s |
| Prompt evaluation | 29 tokens/s |
| First token, 1 user | 42 s |
| Full answer, 1 user | 48 s |
| Full answer, 3 users | up to 127 s |
Measured 2026-09-16 on DocsGPT 0.21.0. RAM: 3.9 GB for the model, 4.9 GB once the KV cache fills.
Engines such as vLLM, TGI, SGLang and LMDeploy support tensor parallelism across GPUs on one host, and some support pipeline parallelism across hosts. This is the engine's capability, not DocsGPT's: DocsGPT talks to the engine over its API and does not schedule GPUs. Before adding a GPU, check the serving stack: at 25 users on an H100 the card sat mostly idle while Ollama serialised prompt evaluation past its parallel slots. Continuous batching (vLLM) is the fix from about 8 concurrent users; a second card is the fix for a model that does not fit.
Legacy or existing hardware is a sizing question rather than a blocker — a large bank runs model sizes and pipelines tuned to fit its existing T4 GPUs, chosen from several open-source models with custom benchmarks before deployment, and the DWP assistant runs on government infrastructure with local inference.
tensor parallelism · pipeline parallelism · T4 16 GB × n · vLLM continuous batching
24 h → 16 min
A large Central European commercial bank processes 10,000+ inbound documents a day; time-to-action fell from 24 hours to 16 minutes, with over 99.5 % classification accuracy, running fully air-gapped on the bank's existing NVIDIA T4 GPUs.
1–2 days → ~5 min
Transfer-document review that took 1–2 working days (P95) now completes in about 5 minutes, with 12 deterministic and agent checks and human approval for uncertain cases.
Government infrastructure, local inference
UK Department for Work and Pensions: an internal knowledge assistant on government infrastructure, with local inference.
A 30-minute demo on your use case, or a free on-premises review with an engineer.