How the estimates are made

This page was written and translated automatically by AI and checked by software against its sources, with no human editing. Specifications come from the sources listed; memory requirements and speeds are estimates calculated with published formulas. How we use AI

Every number on this site is either a specification that two independent open sources agree on, or the result of the deterministic code described below. The same code runs when the pages are built and in your browser.

Values updated Oct 19, 2026

What is modelled

Modelled: llama.cpp loading a GGUF file, text only, all layers on the GPU, Flash Attention on, micro-batch 512, one user. For several cards: identical cards in one machine, with llama.cpp’s default layer split. Multimodal models are counted without their vision projector and image processing.

Ollama and LM Studio use llama.cpp but may use different defaults. vLLM, MLX and Transformers allocate memory differently and are not modelled. A “Fits” result does not mean that a given driver or back end supports the model’s architecture or quantization.

Formula, term by term

For each card d, the estimate Q_d is the same quantity llama.cpp prints per device as model + KV + RS + compute buffers. Layers go to cards with llama.cpp’s default split; the input embedding stays in system memory.

Q_d = W_d + KV_d + S_d + C_d
W_d = Σ_{il ∈ d} layer_bytes[il] + [d = N] × out_bytes
KV_d = Σ_{il ∈ d} (k_il + v_il) × b_kv × cells_il;   b_kv: F16 = 2, Q8_0 = 34/32, Q4_0 = 18/32
cells_full = ⌈ctx / 256⌉ × 256
cells_window = ⌈min(ctx, window × n_seq + 512) / 256⌉ × 256
S_d = Σ_{il ∈ d} state_il × 4 B × n_seq
M = (cells_full + cells_window) × 512 × 4 B
L = max(0.25 GiB, n_vocab × 512 × 4 B)
P = M + n_embd × 512 × 4 B
C_d = L + M                        (N = 1)
C_d = L + 3 × P                    (N > 1, d = N)
C_d = 0.25 GiB + 4 × P             (N > 1, d < N)

X = max_d (Q⁺_d ÷ 0.95 + 1 GiB)
Fits: ∀ d: Q⁺_d ÷ 0.95 + 1 GiB ≤ C_low
Doesn't fit: Q⁻ ÷ 1.10 > C_high (N = 1);   Σ_d Q⁻_d ÷ 1.10 > N × C_high (N > 1)
Tight: otherwise
C_low = G − D_card − U;   C_high = G
tok/s ≤ B ÷ (R + R_KV)

W_d: Weights on card d: the bytes of the layers placed on it, read from the GGUF tensor table, plus the output layer on the last card.
KV_d: KV cache on card d: K and V elements per token of each attention layer on the card, times bytes per element, times cached cells. Multi-head latent attention stores only K.
S_d: Recurrent state: fixed-size state of recurrent and linear-attention layers (32-bit), one per parallel sequence.
C_d, L, M, P: Compute buffer: L is the logits buffer, M the attention mask, P the per-copy inputs that pipeline parallelism keeps on each card when several cards are used.
Q⁺: Q⁺: the per-card estimate used for “Fits” — 4 parallel sequences (llama-server’s default -np auto), layers exactly on a card boundary counted on both cards.
Q⁻: Q⁻: the estimate used for “Doesn’t fit” — 1 sequence, each layer counted once, as the source code assigns it.
X: X: the available memory per card needed for “Fits”. Pages show it as “Available memory needed for Fits”.
G, N: G: the nominal memory of one card (the manufacturer’s “24 GB” counted as 24 GiB); N: the number of identical cards.
C_low: C_low: memory counted as available for “Fits” — nominal memory minus the card’s nominal-to-reported gap (D_card) and the largest start-up and runtime use in the profile’s public logs (U), rounded down to a multiple of 64 MiB.
C_high: C_high: the full nominal memory, used for “Doesn’t fit”.
B, R, R_KV: B: memory bandwidth (bytes per second); R: weight bytes read per generated token (non-expert tensors plus the share of experts in use); R_KV: KV cache bytes read per token.

Constants

Constant	Value	Meaning
`ubatch`	512	llama.cpp default micro-batch size (-ub).
`ctxPad`	256	KV cache cells are padded to a multiple of this number.
`computeFloorBytes`	0.25 GiB	Smallest compute buffer per card.
`maskBytesPerCell`	4 B	Bytes per cell and micro-batch token of the attention mask (32-bit).
`fitReservePerDeviceBytes`	1.00 GiB	Reserve llama.cpp’s default --fit keeps free on each card (-fitt 1024 MiB).
`underestimateTolerance`	5%	Largest underestimate allowed by the gold-standard tests; “Fits” divides the estimate by one minus this.
`overestimateTolerance`	10%	Largest overestimate allowed by the gold-standard tests; “Doesn’t fit” divides the estimate by one plus this.
`stateBytesPerElem`	4 B	Bytes per element of recurrent state.
`defaultCtx`	8,192	Default context for tables: this many tokens, or the model’s maximum if shorter.
`defaultSeqs`	4	Parallel sequences assumed for “Fits” (llama-server -np auto, unified KV cache).
`goldCoveredCtx`	262,144	Longest context (cells) on one card covered by the gold-standard logs; the longest-context search stops here.
`maxCards`	8	Most identical cards considered.
`minUsefulCapacityBytes`	2.00 GiB	Smallest memory counted as available on one card for a verdict; below this the result is “Unknown” (except “Does not fit even on 8 cards”).
`pipelineCopies`	4	Copies of layer inputs per card with pipeline parallelism on several cards (GGML_SCHED_MAX_COPIES).
`goldCoveredCtxMulti`	131,072	Longest context (cells) on several cards covered by the gold-standard logs.
`cLowStepBytes`	64 MiB	C_low of a discrete GPU (nominal − D_card − U) is rounded down to a multiple of this.

Weights are not estimated from parameter counts: every quantization’s GGUF files are read (header and tensor table) at a pinned revision, and the bytes of each layer are summed. Tensors llama.cpp keeps in system memory (input embeddings) are left out.

Fits, borderline or doesn’t fit

“Fits” means only this: llama.cpp’s default --fit load check would put all layers on the GPU. llama.cpp makes that check when it loads a model (estimated buffers plus a 1 GiB reserve per card must be no more than the free memory at start-up); this site recomputes it with the same buffer rules and leaves room for estimate errors. It does not promise that peak memory during inference (memory pools, long prompts and other runtime allocations) fits.

The gold-standard tests allow the estimate to be up to 5% too low and up to 10% too high, so each side of the decision uses its own bound:

Fits when, on every card, Q⁺ ÷ 0.95 + 1 GiB is no more than C_low. Q⁺ assumes 4 parallel sequences.
Doesn't fit when Q⁻ ÷ 1.10 is more than the full memory C_high (with several cards: the sum over cards against N × C_high, because the default --fit adjusts the split by itself).
Tight in between: it may or may not fit. Shorten the context or use a lower quantization.
Unknown when no run profile covers the device, for workstation and data-center GPUs (their memory configuration is not established by public logs), or when the memory counted as available is below 2 GiB: only X is shown, plus “Doesn’t fit” if the model is too big even for the full memory.

X = the largest per-card Q⁺ ÷ 0.95 + 1 GiB is the available memory needed for “Fits”. Pages write it as “Available memory needed for Fits”, not “at least”, because a borderline result needs the same amount.

Tables check quantizations from highest to lowest at the default context. A row shows either the highest quantization that fits (with X, the longest context and the speed ceiling), or — if none fits — the highest borderline one (with X), or — if all are too big — the lowest quantization with its X.

The longest context is searched between the default context and the smaller of the model’s maximum and the longest context covered by measured logs (262,144 cells on one card, 131,072 on several). If it still fits there, pages show “≥” that value: longer contexts were not checked.

Several identical cards are considered only when one card does not fit, only for desktop, workstation and data-center GPUs with a run profile, and each count from 2 to 8 is checked on its own (adding a card does not always lower the heaviest card’s load, and pipeline parallelism adds 4 copies of layer inputs). The page names the smallest count that fits with the default even split; it is not a minimum, because the default --fit can adjust the split and may manage with fewer cards. Workstation and data-center GPUs, and GPUs with less than 2 GiB counted as available, only get “Does not fit even on 8 cards” (or no count).

Run profiles

How much memory counts as available comes only from public llama.cpp logs: the total each device reports, the free memory when llama.cpp’s --fit check runs and, where printed, memory unaccounted for after a run. D_card = nominal − reported total of that GPU model (largest if several logs); U = the largest of total − free at the --fit check and positive unaccounted memory after a run, over all valid observations of the profile. When a log only has a device-list reading, the largest list-to-check gap seen in the profile is added. Observations are excluded when a display is attached, other programs already used the GPU, free exceeds total, or a shared server adds unrelated use. A profile without enough observations gives no verdicts.

Profile	U (MiB)	D for desktop GPUs without own data (MiB)	Observations	Verdicts
CUDA (NVIDIA)	2,112	1,054	17	Given
Vulkan (Intel)	832	0	1	Not given: not enough observations
ROCm on Linux (AMD)	1,216	144	11	Given

Consumer GPUs without a measurement of their own use the largest gap between nominal and reported total measured on consumer and workstation cards, in the default configuration. Workstation and data-center GPUs get neither “Fits” nor “Tight”, measured or not, because public logs do not show their memory configuration (for example whether ECC is on): they are “Unknown” unless the model is too big even for the full memory, and for several cards only “Does not fit even on 8 cards” is given. Laptop GPUs, Apple silicon and other unified-memory computers have no profile in this version.

CUDA (NVIDIA)

llama.cpp, CUDA back end (NVIDIA discrete GPU), all layers on the GPU, Flash Attention on, micro-batch (-ub) 512, F16 KV cache, context (-c) as shown; “Fits” is judged with llama-server’s default 4 parallel sequences (-np auto, unified KV cache) and “Doesn’t fit” with 1 sequence; idle GPU with no other programs using its memory.

U = 2,112 MiB; D for desktop GPUs without their own measurement = 1,054 MiB.

Memory unaccounted for after a run in these logs: 236–2,050 MiB (4 observations). Runtime allocations are not part of “Fits”; U includes them only as extra room, not as an upper bound.

D_card: nominal memory minus the total llama.cpp reported for that GPU model (MiB).
GPU	D_card (MiB)
NVIDIA GeForce RTX 3060 12GB	251
NVIDIA GeForce RTX 3090	453
NVIDIA GeForce RTX 4060 Ti 16GB	5
NVIDIA GeForce RTX 4090	496
NVIDIA GeForce RTX 5060	484
NVIDIA GeForce RTX 5090	659
NVIDIA L4	2,098
NVIDIA RTX A4000	410
NVIDIA RTX PRO 6000 Blackwell Workstation Edition	1,054

Observations from public llama.cpp logs, with the quoted log lines.
Device	Back end	Build	Nominal	Reported total (MiB)	Free at start-up (MiB)	Used before loading (MiB)	Unaccounted after run (MiB)	Source
NVIDIA GeForce RTX 3090 (x2, CUDA0)	CUDA	`b11415`	24 GB	24,575	23,054	1,522	—	llama.cpp issues/29811 Log lines `0.00.573.399 I common_memory_breakdown_print: \| - CUDA0 (RTX 3090) \| 24575 = 23054 + (21234 = 17721 + 2964 + 549) + -19713 \|` `0.00.573.400 I common_memory_breakdown_print: \| - CUDA1 (RTX 3090) \| 24575 = 23206 + (20693 = 18689 + 1453 + 549) + -19323 \|` `0.00.969.460 I common_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 3090): 24575 total, 21234 used, 1819 free vs. target of 1024`
NVIDIA GeForce RTX 4090	CUDA	`b10567 (d7fa69b7d)`	24 GB	24,563	23,040	1,575	—	llama.cpp issues/27680 Log lines `common_param: - CUDA0 : NVIDIA GeForce RTX 4090 (24563 MiB, 23040 MiB free)` `llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:08:00.0) - 23040 MiB free` `load_tensors: offloaded 44/44 layers to GPU`
NVIDIA GeForce RTX 4090 (x2, CUDA0)	CUDA	`9293 (1acee6bf8)`	24 GB	24,080	23,672	460	—	llama.cpp issues/23679 Log lines `0.00.205.036 I - CUDA0 : NVIDIA GeForce RTX 4090 (24080 MiB, 23672 MiB free)` `0.00.283.679 I - CUDA1 : NVIDIA GeForce RTX 4090 (24080 MiB, 23672 MiB free)` `0.00.691.813 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:01:00.0) - 23672 MiB free`
NVIDIA GeForce RTX 3090 (x8)	CUDA	`build 1 (0f7fada) ≈ b9557`	24 GB	24,123	23,858	317	—	llama.cpp issues/24608 Log lines `0.01.258.121 I - CUDA0 : NVIDIA GeForce RTX 3090 (24123 MiB, 23858 MiB free)`
NVIDIA GeForce RTX 3090	CUDA	`9274 (52fb93a)`	24 GB	24,126	23,403	775	—	llama.cpp issues/23472 Log lines `0.00.797.657 I common_memory_breakdown_print: \| - CUDA0 (RTX 3090) \| 24126 = 23403 + (25610 = 15621 +`
NVIDIA GeForce RTX 3060	CUDA	`b7342`	12 GB	12,037	—	270	270	llama.cpp issues/17931 Log lines `llama_memory_breakdown_print: \| - CUDA0 (RTX 3060) \| 12037 = 4510 + ( 7256 = 1224 + 4662 + 1369) + 270 \|`
NVIDIA GeForce RTX 4090	CUDA	`7474 (f9ec8858e)`	24 GB	24,082	23,687	447	—	llama.cpp issues/18181 Log lines `llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 4090) : 24082 total,` `llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:82:00.0) - 23687 MiB free`
NVIDIA GeForce RTX 5090	CUDA	`8241 (62b8143ad)`	32 GB	32,606	30,927	2,050	2,050	llama.cpp issues/20308 Log lines `0.18.828.015 I llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5090) (0000:08:00.0) - 30927 MiB free` `0.03.911.366 I llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 5090): 32606 total,` `0.22.996.049 I load_tensors: offloaded 61/61 layers to GPU`
NVIDIA GeForce RTX 4090	CUDA	`8241 (62b8143ad)`	24 GB	24,563	22,988	1,910	1,910	llama.cpp issues/20308 Log lines `0.18.828.027 I llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) (0000:01:00.0) - 22988 MiB free` `0.03.911.366 I llama_params_fit_impl: - CUDA1 (NVIDIA GeForce RTX 4090): 24563 total,`
NVIDIA GeForce RTX 5090	CUDA	`9692 (f3e182816)`	32 GB	32,109	31,585	576	—	llama.cpp issues/24758 Log lines `0.01.070.150 I - CUDA0 : NVIDIA GeForce RTX 5090 (32109 MiB, 31585 MiB free)` `llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 5090) (0000:01:00.0) - 31583 MiB free`
NVIDIA GeForce RTX 5060	CUDA	`build 1 (fae3a28)`	8 GB	7,708	7,573	187	—	llama.cpp issues/21984 Log lines `CUDA0: NVIDIA GeForce RTX 5060 (7708 MiB, 7573 MiB free)` `Device 0: NVIDIA GeForce RTX 5060, compute capability 12.0, VMM: yes, VRAM: 7708 MiB` `\| 0 N/A N/A 1448 G /usr/lib/xorg/Xorg 4MiB \|`
NVIDIA GeForce RTX 4060 Ti	CUDA	`0.3.0-dev build 10672 (511f9c1)`	16 GB	16,379	15,233	1,198	—	llama.cpp issues/27892 Log lines `0.00.590.294 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 4060 Ti (16379 MiB, 15233 MiB free)` `0.01.250.688 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4060 Ti) (0000:01:00.0) - 15233 MiB free` `0.01.820.045 I load_tensors: offloaded 35/35 layers to GPU`
NVIDIA GeForce RTX 3080 (20 GB variant)	CUDA	`build 1 (234a6eb) ≈ B10733`	20 GB	20,479	19,271	1,260	—	llama.cpp issues/28167 Log lines `0.00.239.888 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 3080 (20479 MiB, 19271 MiB free)` `using device CUDA0 (NVIDIA GeForce RTX 3080) (0000:09:00.0) - 19271 MiB free` `0.01.318.328 I load_tensors: offloaded 37/37 layers to GPU`
NVIDIA RTX PRO 6000 Blackwell Workstation Edition	CUDA	`7474 (f9ec8858e)`	96 GB	97,250	96,688	614	—	llama.cpp issues/18181 Log lines `llama_params_fit_impl: - CUDA3 (NVIDIA RTX PRO 6000 Blackwell Workstation Edition): 97250 total,` `llama_model_load_from_file_impl: using device CUDA3 (NVIDIA RTX PRO 6000 Blackwell Workstation Edition)` `(0000:01:00.0) - 96688 MiB free`
NVIDIA RTX A4000	CUDA	`9890 (74976e1)`	16 GB	15,974	15,791	235	—	llama.cpp issues/25594 Log lines `0.00.322.214 I cmn common_param: - CUDA0 : NVIDIA RTX A4000 (15974 MiB, 15791 MiB free)` `0.01.255.214 I llama_prepare_model_devices: using device CUDA0 (NVIDIA RTX A4000) (0000:02:00.0) - 15791 MiB free`
NVIDIA A100-SXM4-40GB	CUDA	`8387 (45172df)`	40 GB	40,444	39,968	528	—	llama.cpp issues/20761 Log lines `ggml_cuda_init: found 1 CUDA devices (Total VRAM: 40444 MiB):` `Device 0: NVIDIA A100-SXM4-40GB, compute capability 8.0, VMM: yes, VRAM: 40444 MiB` `llama_model_load_from_file_impl: using device CUDA0 (NVIDIA A100-SXM4-40GB) (0000:41:00.0) - 39968 MiB free`
NVIDIA L4	CUDA	`b9036-b863d41c7`	24 GB	22,478	22,286	244	236	llama.cpp pull/22546 Log lines `Device 0: NVIDIA L4, compute capability 8.9, VMM: yes, VRAM: 22478 MiB` `llama_prepare_model_devices: using device CUDA0 (NVIDIA L4) (0000:00:03.0) - 22286 MiB free` `common_memory_breakdown_print: \| - CUDA0 (L4) \| 22478 = 22234 + (17095 = 8045 + 8242 + 808) + -16851 \|`

Vulkan (Intel)

llama.cpp, Vulkan back end (Intel discrete GPU), all layers on the GPU, Flash Attention on, micro-batch (-ub) 512, F16 KV cache, context (-c) as shown; “Fits” is judged with 4 parallel sequences and “Doesn’t fit” with 1 sequence; idle GPU. Not enough public measurements yet, so no verdicts use this profile.

U = 832 MiB; D for desktop GPUs without their own measurement = 0 MiB.

D_card: nominal memory minus the total llama.cpp reported for that GPU model (MiB).
GPU	D_card (MiB)
Intel Arc Pro B70	138

Observations from public llama.cpp logs, with the quoted log lines.
Device	Back end	Build	Nominal	Reported total (MiB)	Free at start-up (MiB)	Used before loading (MiB)	Unaccounted after run (MiB)	Source
Intel Arc Pro B70	Vulkan	`build_info: b8871-7fc1c4ef7`	32 GB	32,630	31,861	769	—	llama.cpp issues/22275 Log lines `0.01.712.235 I llama_model_load_from_file_impl: using device Vulkan0 (Intel(R) Arc(TM) Pro B70 Graphics) (unknown id)` `- 31861 MiB free` `0.10.080.962 I load_tensors: offloaded 31/31 layers to GPU`

ROCm on Linux (AMD)

llama.cpp, ROCm back end on Linux (AMD discrete GPU), all layers on the GPU, Flash Attention on, micro-batch (-ub) 512, F16 KV cache, context (-c) as shown; “Fits” is judged with llama-server’s default 4 parallel sequences (-np auto, unified KV cache) and “Doesn’t fit” with 1 sequence; idle GPU with no other programs using its memory.

U = 1,216 MiB; D for desktop GPUs without their own measurement = 144 MiB.

Memory unaccounted for after a run in this log: 279 MiB (1 observation). Runtime allocations are not part of “Fits”; U includes them only as extra room, not as an upper bound.

D_card: nominal memory minus the total llama.cpp reported for that GPU model (MiB).
GPU	D_card (MiB)
AMD Radeon Instinct MI50 (32GB)	16
AMD Radeon AI PRO R9700	144
AMD Radeon RX 6800	16
AMD Radeon RX 6900 XT	16
AMD Radeon RX 7800 XT	16
AMD Radeon RX 7900 XTX	16

Observations from public llama.cpp logs, with the quoted log lines.
Device	Back end	Build	Nominal	Reported total (MiB)	Free at start-up (MiB)	Used before loading (MiB)	Unaccounted after run (MiB)	Source
AMD Radeon RX 7900 XTX	ROCm	`version: 9744 (063d9c1)`	24 GB	24,560	24,494	266	—	llama.cpp issues/24858 Log lines `[44157] 0.00.063.214 I - ROCm0 : Radeon RX 7900 XTX (24560 MiB, 24494 MiB free)` `[44157] 0.00.668.217 I llama_prepare_model_devices: using device ROCm0 (Radeon RX 7900 XTX) (0000:0a:00.0) - 24346 MiB free` `[44157] 0.02.978.818 I load_tensors: offloaded 33/33 layers to GPU`
AMD Radeon RX 7800 XT (x2, ROCm0)	ROCm	`version: b8929-9d34231bb`	16 GB	16,368	16,036	467	—	llama.cpp issues/22351 Log lines `Device 0: AMD Radeon RX 7800 XT, gfx1101 (0x1101), VMM: no, Wave Size: 32, VRAM: 16368 MiB` `llama_model_load_from_file_impl: using device ROCm0 (AMD Radeon RX 7800 XT) (0000:01:00.0) - 16036 MiB free` `common_params_fit_impl: - ROCm0 (AMD Radeon RX 7800 XT): 16368 total, 10610 used, 5291 free vs. target of 1024`
AMD Radeon RX 7900 XTX	ROCm	`version: 9307 (549b9d8)`	24 GB	24,560	24,510	1,135	—	llama.cpp issues/23934 Log lines `0.00.042.225 I - ROCm0 : AMD Radeon RX 7900 XTX (24560 MiB, 24510 MiB free)` `OS: Ubuntu 22.04.5 LTS` `Kernel: 6.8.0-117-generic`
AMD Radeon RX 7900 XTX (x4)	ROCm	`unknown`	24 GB	24,560	24,510	1,135	—	llama.cpp issues/27021 Log lines `- ROCm0 : Radeon RX 7900 XTX (24560 MiB, 24510 MiB free)` `- ROCm1 : Radeon RX 7900 XTX (24560 MiB, 24518 MiB free)` `- ROCm2 : Radeon RX 7900 XTX (24560 MiB, 24518 MiB free)`
AMD Radeon AI PRO R9700 (x8)	ROCm	`build 43 (5854625)`	32 GB	32,624	32,534	1,175	—	llama.cpp issues/26208 Log lines `0.00.089.438 I cmn common_param: - ROCm0 : AMD Radeon AI PRO R9700 (32624 MiB, 32534 MiB free)` `0.08.861.240 I load_tensors: offloaded 66/66 layers to GPU` `Host is a Debian 13 install with backports set to priority 500.`
AMD Radeon AI PRO R9700	ROCm	`build 10616 (45bfc6f91)`	32 GB	32,624	32,538	1,171	—	llama.cpp issues/26208 Log lines `[44399] 0.00.093.902 I cmn common_param: - ROCm0 : AMD Radeon Graphics (32624 MiB, 32538 MiB free)` `[44399] 0.00.093.904 I cmn common_param: - ROCm1 : AMD Radeon Graphics (32624 MiB, 32558 MiB free)` `[44399] 0.00.562.846 I common_memory_breakdown_print: \| - ROCm0 (Graphics) \| 32624 = 31454 + (18994 = 7216 + 9095 + 2682) + -17824 \|`
AMD Instinct MI50 32GB	ROCm	`build 9943 (c15c5c77a)`	32 GB	32,752	32,722	1,115	—	llama.cpp issues/25487 Log lines `0.00.046.167 I cmn common_param: - ROCm0 : AMD Instinct MI60 / MI50 (32752 MiB, 32722 MiB free)` `llama_prepare_model_devices: using device ROCm0 (AMD Instinct MI60 / MI50) (0000:43:00.0) - 32570 MiB free` `0.13.494.533 I load_tensors: offloaded 49/52 layers to GPU`
AMD Radeon RX 6800	ROCm	`version: 746 (da296d6)`	16 GB	16,368	—	660	—	llama.cpp issues/26038 Log lines `\| - ROCm0 (RX 6800) \| 16368 = 15708 + (13826 = 13244 + 544 + 37) + -13166 \|` `\| - ROCm0 (RX 6800) \| 16368 = 16010 + (13527 = 13244 + 245 + 37) + -13169 \|` `GPU: AMD Radeon RX 6800`
AMD Radeon Pro VII (x3)	ROCm	`build: aa374175c (6969)`	16 GB	16,368	—	279	279	llama.cpp issues/17058 Log lines `llama_memory_breakdown_print: \| memory breakdown [MiB] \| total free self model context compute unaccounted \|` `llama_memory_breakdown_print: \| - ROCm0 ((TM) Pro VII) \| 16368 = 10268 + (5820 = 5613 + 56 + 150) + 279 \|` `llama_memory_breakdown_print: \| - ROCm1 ((TM) Pro VII) \| 16368 = 10250 + (5852 = 5336 + 54 + 461) + 265 \|`
AMD Radeon RX 6900 XT	ROCm	`version: 9890 (74976e1)`	16 GB	16,368	16,342	1,111	—	llama.cpp issues/25594 Log lines `0.00.322.247 I cmn common_param: - ROCm0 : AMD Radeon RX 6900 XT (16368 MiB, 16342 MiB free)` `0.01.255.231 I llama_prepare_model_devices: using device ROCm0 (AMD Radeon RX 6900 XT) (0000:03:00.0) - 16342 MiB free` `0.01.790.609 I load_tensors: offloaded 66/66 layers to GPU`
AMD Radeon RX 7900 XTX	ROCm	`version: 8707 (3bd9aa1)`	24 GB	24,560	24,376	219	—	llama.cpp issues/22052 Log lines `Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB` `llama_model_load_from_file_impl: using device ROCm0 (AMD Radeon RX 7900 XTX) (0000:c4:00.0) - 24376 MiB free` `llama_params_fit_impl: - ROCm0 (AMD Radeon RX 7900 XTX): 24560 total, 9473 used, 14868 free vs. target of 1024`

Gold-standard tests

Each case is a public llama.cpp log that prints per-device buffers. The GGUF files of that log are read, the buffers are recomputed per card with the log’s own context, sequences, micro-batch and KV type, and compared with model + KV + RS + compute for the same card. 38 cases (50 cards) run as automated tests on every build and must stay within −5% to +10%; the largest error is +8.85%.

Case	GGUF	Back end	Cards	Device	Context	Sequences	KV	Measured	Our estimate	Error
E01	unsloth/GLM-4.7-Flash-GGUF	Metal	1	MTL0	65,536	4	F16	34,053.1 MiB	34,173.1 MiB	+0.35%
E02	unsloth/GLM-4.7-Flash-GGUF	CUDA	1	CUDA0	380,416	2	F16	76,784.3 MiB	77,208.0 MiB	+0.55%
E03	bartowski/gemma-2-2b-it-GGUF	CUDA	1	CUDA0	4,096	1	F16	3,570.3 MiB	3,581.7 MiB	+0.32%
E05	ggml-org/gemma-3-4b-it-qat-GGUF	Vulkan	1	Vulkan0	131,072	4	F16	6,001.9 MiB	6,261.9 MiB	+4.33%
E06	unsloth/gemma-3-27b-it-GGUF	CUDA	1	CUDA0	8,192	1	Q8_0	14,257.9 MiB	14,266.4 MiB	+0.06%
E07	unsloth/gemma-4-26B-A4B-it-GGUF	ROCm	1	ROCm0	262,144	4	F16	54,994.9 MiB	55,203.4 MiB	+0.38%
E08	bartowski/google_gemma-4-E2B-it-GGUF	Vulkan	1	Vulkan0	12,032	4	F16	2,065.2 MiB	2,090.7 MiB	+1.23%
E10	lmstudio-community/gpt-oss-20b-GGUF	Metal	1	Metal	4,096	1	F16	12,048.6 MiB	12,052.4 MiB	+0.03%
E11	unsloth/gpt-oss-120b-GGUF	Vulkan	1	Vulkan0	131,072	1	F16	65,418.0 MiB	65,654.7 MiB	+0.36%
E12	unsloth/gpt-oss-120b-GGUF	ROCm	3	ROCm0	131,072	1	F16	24,085.8 MiB	24,118.2 MiB	+0.13%
E12	unsloth/gpt-oss-120b-GGUF	ROCm	3	ROCm1	131,072	1	F16	21,826.8 MiB	22,488.5 MiB	+3.03%
E12	unsloth/gpt-oss-120b-GGUF	ROCm	3	ROCm2	131,072	1	F16	21,101.6 MiB	21,325.3 MiB	+1.06%
E13	lmstudio-community/gpt-oss-120b-GGUF	ROCm	3	ROCm0	50,176	4	F16	22,570.5 MiB	22,681.7 MiB	+0.49%
E13	lmstudio-community/gpt-oss-120b-GGUF	ROCm	3	ROCm1	50,176	4	F16	20,664.3 MiB	21,033.4 MiB	+1.79%
E13	lmstudio-community/gpt-oss-120b-GGUF	ROCm	3	ROCm2	50,176	4	F16	19,914.5 MiB	20,003.2 MiB	+0.45%
E15	bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF	Vulkan	1	Vulkan0	32,768	1	F16	27,959.0 MiB	27,988.9 MiB	+0.11%
E17	LiquidAI/LFM2.5-1.2B-Instruct-GGUF	CUDA	1	CUDA0	128,000	4	Q8_0	2,370.8 MiB	2,489.7 MiB	+5.02%
E18	unsloth/Qwen3.5-27B-GGUF	CUDA	1	CUDA0	10,240	1	F16	17,406.9 MiB	17,406.9 MiB	+0.00%
E19	unsloth/Qwen3.5-35B-A3B-GGUF	Vulkan	1	Vulkan0	16,384	1	F16	21,566.6 MiB	21,585.6 MiB	+0.09%
E20	unsloth/Qwen3.5-9B-GGUF	CUDA	1	CUDA0	256,000	4	Q8_0	20,381.1 MiB	20,576.0 MiB	+0.96%
E23	unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF	CUDA	2	CUDA0	131,072	1	Q8_0	21,254.9 MiB	21,294.9 MiB	+0.19%
E23	unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF	CUDA	2	CUDA1	131,072	1	Q8_0	4,032.3 MiB	4,280.2 MiB	+6.15%
E24	bartowski/Mistral-Large-Instruct-2411-GGUF	CUDA	4	CUDA0	32,768	1	F16	32,775.5 MiB	32,783.4 MiB	+0.02%
E24	bartowski/Mistral-Large-Instruct-2411-GGUF	CUDA	4	CUDA1	32,768	1	F16	15,712.7 MiB	15,856.7 MiB	+0.92%
E24	bartowski/Mistral-Large-Instruct-2411-GGUF	CUDA	4	CUDA2	32,768	1	F16	18,503.9 MiB	18,647.9 MiB	+0.78%
E24	bartowski/Mistral-Large-Instruct-2411-GGUF	CUDA	4	CUDA3	32,768	1	F16	15,874.9 MiB	15,930.9 MiB	+0.35%
E25	unsloth/Qwen3.5-27B-GGUF	CUDA	2	CUDA0	10,240	4	F16	13,661.2 MiB	13,738.2 MiB	+0.56%
E25	unsloth/Qwen3.5-27B-GGUF	CUDA	2	CUDA1	10,240	4	F16	14,767.9 MiB	14,423.6 MiB	-2.33%
E26	unsloth/Qwen3.5-27B-GGUF	Vulkan	2	Vulkan0	16,128	4	F16	6,805.8 MiB	6,908.0 MiB	+1.50%
E26	unsloth/Qwen3.5-27B-GGUF	Vulkan	2	Vulkan1	16,128	4	F16	11,830.8 MiB	11,842.3 MiB	+0.10%
E27	lmstudio-community/Qwen3-32B-GGUF	CUDA	1	CUDA0	32,768	1	Q4_0	21,034.4 MiB	21,088.4 MiB	+0.26%
E28	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	CUDA	1	CUDA0	132,096	4	F16	45,560.0 MiB	45,655.0 MiB	+0.21%
E29	lmstudio-community/granite-4.0-micro-GGUF	CUDA	1	CUDA0	2,050	1	F16	2,381.0 MiB	2,439.3 MiB	+2.45%
E30	unsloth/Llama-3.2-3B-Instruct-GGUF	CUDA	1	CUDA0	74,000	1	F16	10,294.9 MiB	10,439.4 MiB	+1.40%
E31	mradermacher/Sugoi-14B-Ultra-HF-i1-GGUF	CUDA	1	CUDA0	512	1	F16	6,674.4 MiB	6,665.4 MiB	-0.13%
E33	unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF	ROCm	1	ROCm0	4,096	20	F16	34,286.3 MiB	34,717.0 MiB	+1.26%
E35	mradermacher/Strawberrylemonade-70B-v1.2-i1-GGUF	ROCm	2	ROCm0	32,768	2	F16	23,499.4 MiB	23,651.3 MiB	+0.65%
E35	mradermacher/Strawberrylemonade-70B-v1.2-i1-GGUF	ROCm	2	ROCm1	32,768	2	F16	23,230.2 MiB	23,267.7 MiB	+0.16%
E37	Qwen/Qwen3-VL-30B-A3B-Instruct-GGUF	ROCm	1	ROCm0	20,224	4	F16	19,721.2 MiB	19,756.7 MiB	+0.18%
E38	unsloth/Llama-3.2-3B-Instruct-GGUF	ROCm	1	ROCm0	10,240	4	F16	3,300.9 MiB	3,314.4 MiB	+0.41%
E39	unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF	Vulkan	1	Vulkan0	100,864	4	Q8_0	22,043.4 MiB	22,215.6 MiB	+0.78%
E40	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	Vulkan	1	Vulkan0	8,448	1	F16	5,718.0 MiB	5,732.0 MiB	+0.24%
E41	unsloth/GLM-4.5-Air-GGUF	Vulkan	1	Vulkan0	24,576	4	Q8_0	46,141.9 MiB	46,179.1 MiB	+0.08%
E43	unsloth/Qwen3-VL-8B-Thinking-GGUF	Vulkan	2	Vulkan0	24,576	4	Q8_0	4,416.4 MiB	4,560.3 MiB	+3.26%
E43	unsloth/Qwen3-VL-8B-Thinking-GGUF	Vulkan	2	Vulkan1	24,576	4	Q8_0	2,734.4 MiB	2,766.3 MiB	+1.17%
E44	ggml-org/Qwen2.5-Coder-1.5B-Instruct-Q8_0-GGUF	Vulkan	1	Vulkan0	4,096	1	F16	1,976.4 MiB	1,981.4 MiB	+0.25%
E45	unsloth/Qwen3-30B-A3B-GGUF	Metal	1	Metal	4,096	1	F16	9,307.4 MiB	9,307.4 MiB	-0.00%
E46	bartowski/mistralai_Voxtral-Mini-3B-2507-GGUF	Metal	1	MTL0	16,384	4	F16	6,256.0 MiB	6,276.0 MiB	+0.32%
E47	unsloth/Qwen3-0.6B-GGUF	Metal	1	Metal	34,304	4	F16	4,425.4 MiB	4,488.4 MiB	+1.42%
E48	sweepai/sweep-next-edit-1.5B	Metal	1	Metal	32,768	4	F16	2,462.6 MiB	2,680.6 MiB	+8.85%

Logs left out, and why

E04: Flash Attention was off; this site models Flash Attention on.
E09: Run with --swa-full, a non-default sliding-window cache.
E16: Several sequences without a unified KV cache; that cache layout is not modelled.
E32: Flash Attention was off; this site models Flash Attention on.
E34: Flash Attention was off; this site models Flash Attention on.
E42: Flash Attention was off; this site models Flash Attention on.

llama.cpp rules by architecture

How each architecture lays out its KV cache and state is transcribed from the llama.cpp source code at commit 5aeac37a35.

Architecture	KV cache layout	Source
`llama`	full attention in every layer	src/models/llama.cpp
`qwen2`	full attention in every layer	src/models/qwen2.cpp
`qwen2moe`	full attention in every layer	src/models/qwen2moe.cpp
`qwen2vl`	full attention in every layer	src/models/qwen2vl.cpp
`qwen3`	full attention in every layer	src/models/qwen3.cpp
`qwen3moe`	full attention in every layer	src/models/qwen3moe.cpp
`qwen3vl`	full attention in every layer	src/models/qwen3vl.cpp
`qwen3vlmoe`	full attention in every layer	src/models/qwen3vlmoe.cpp
`mistral3`	full attention in every layer	src/models/mistral3.cpp
`mistral4`	full attention in every layer	src/models/mistral4.cpp
`phi2`	full attention in every layer	src/models/phi2.cpp
`phi3`	full attention in every layer	src/models/phi3.cpp
`phimoe`	full attention in every layer	src/models/phimoe.cpp
`gemma`	full attention in every layer	src/models/gemma.cpp
`glm4`	full attention in every layer	src/models/glm4.cpp
`glm4moe`	full attention in every layer	src/models/glm4-moe.cpp
`granite`	full attention in every layer	src/models/granite.cpp
`granitemoe`	full attention in every layer	src/models/granite-moe.cpp
`command-r`	full attention in every layer	src/models/command-r.cpp
`olmo`	full attention in every layer	src/models/olmo.cpp
`olmoe`	full attention in every layer	src/models/olmoe.cpp
`starcoder2`	full attention in every layer	src/models/starcoder2.cpp
`internlm2`	full attention in every layer	src/models/internlm2.cpp
`minimax-m2`	full attention in every layer	src/models/minimax-m2.cpp
`hunyuan-moe`	full attention in every layer	src/models/hunyuan-moe.cpp
`hunyuan-dense`	full attention in every layer	src/models/hunyuan-dense.cpp
`hy_v3`	full attention in every layer	src/models/hy-v3.cpp
`ernie4_5`	full attention in every layer	src/models/ernie4-5.cpp
`ernie4_5-moe`	full attention in every layer	src/models/ernie4-5-moe.cpp
`seed_oss`	full attention in every layer	src/models/seed-oss.cpp
`smollm3`	full attention in every layer	src/models/smollm3.cpp
`apertus`	full attention in every layer	src/models/apertus.cpp
`exaone`	full attention in every layer	src/models/exaone.cpp
`nemotron`	full attention in every layer	src/models/nemotron.cpp
`bailingmoe`	full attention in every layer	src/models/bailingmoe.cpp
`bailingmoe2`	full attention in every layer	src/models/bailingmoe2.cpp
`dots1`	full attention in every layer	src/models/dots1.cpp
`arcee`	full attention in every layer	src/models/arcee.cpp
`grovemoe`	full attention in every layer	src/models/grovemoe.cpp
`nanbeige`	full attention in every layer	src/models/nanbeige.cpp
`k2-horizon`	full attention in every layer	src/models/k2-horizon.cpp
`deci`	cache layout not modelled yet — models with this architecture are not listed	src/models/deci.cpp
`deepseek2`	full attention in every layer	src/models/deepseek2.cpp
`deepseek`	full attention in every layer	src/models/deepseek.cpp
`minicpm3`	full attention in every layer	src/models/minicpm3.cpp
`deepseek32`	sparse attention with indexer keys (counted as extra K)	src/models/deepseek32.cpp
`glm-dsa`	sparse attention with indexer keys (counted as extra K)	src/models/glm-dsa.cpp
`gemma2`	sliding window: 1 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/gemma2.cpp
`gemma3`	sliding window: 5 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/gemma3.cpp
`gemma3n`	sliding window: 4 sliding layers per full layer, unless the GGUF file gives a per-layer pattern; cache layout not modelled yet — models with this architecture are not listed	src/models/gemma3n.cpp
`gemma4`	sliding window, per-layer pattern read from the GGUF file	src/models/gemma4.cpp
`cohere2`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/cohere2.cpp
`cohere2moe`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/cohere2moe.cpp
`gpt-oss`	sliding window: 1 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/openai-moe.cpp
`exaone4`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/exaone4.cpp
`exaone-moe`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/exaone-moe.cpp
`afmoe`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/afmoe.cpp
`llama4`	chunked attention: 3 chunked layers per full layer	src/models/llama4.cpp
`muse-glimmer`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/muse-glimmer.cpp
`muse-spark`	sliding window, per-layer pattern read from the GGUF file	src/models/muse-spark.cpp
`mellum`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/mellum.cpp
`smallthinker`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/smallthinker.cpp
`laguna`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/laguna.cpp
`granite_swa`	sliding window, per-layer pattern read from the GGUF file	src/models/granite-swa.cpp
`maple`	sliding window, per-layer pattern read from the GGUF file	src/models/maple.cpp
`spark2_5`	sliding window, per-layer pattern read from the GGUF file	src/models/spark2-5.cpp
`step35`	sliding window, per-layer pattern read from the GGUF file	src/models/step35.cpp
`mimo2`	sliding window, per-layer pattern read from the GGUF file	src/models/mimo2.cpp
`kolibri1`	sliding window, per-layer pattern read from the GGUF file	src/models/kolibri1.cpp
`beam`	sliding window, per-layer pattern read from the GGUF file	src/models/beam.cpp
`maion-coder`	sliding window, per-layer pattern read from the GGUF file	src/models/maion-coder.cpp
`olmo2`	sliding window: 3 sliding layers per full layer, unless the GGUF file gives a per-layer pattern	src/models/olmo2.cpp
`plamo3`	cache layout not modelled yet — models with this architecture are not listed	src/models/plamo3.cpp
`qwen3next`	recurrent / linear-attention layers with a fixed-size state	src/models/qwen3next.cpp
`qwen35`	recurrent / linear-attention layers with a fixed-size state	src/models/qwen35.cpp
`qwen35moe`	recurrent / linear-attention layers with a fixed-size state	src/models/qwen35moe.cpp
`qwen4exp`	recurrent / linear-attention layers with a fixed-size state; cache layout not modelled yet — models with this architecture are not listed	src/models/qwen4exp.cpp
`nemotron_h`	recurrent / linear-attention layers with a fixed-size state	src/models/nemotron-h.cpp
`nemotron_h_moe`	recurrent / linear-attention layers with a fixed-size state	src/models/nemotron-h.cpp
`granitehybrid`	recurrent / linear-attention layers with a fixed-size state	src/models/granite-hybrid.cpp
`jamba`	recurrent / linear-attention layers with a fixed-size state	src/models/jamba.cpp
`lfm2`	recurrent / linear-attention layers with a fixed-size state	src/models/lfm2.cpp
`lfm2moe`	recurrent / linear-attention layers with a fixed-size state	src/models/lfm2moe.cpp
`falcon-h1`	cache layout not modelled yet — models with this architecture are not listed	src/models/falcon-h1.cpp
`kimi-linear`	recurrent / linear-attention layers with a fixed-size state	src/models/kimi-linear.cpp
`kimi-k3`	recurrent / linear-attention layers with a fixed-size state	src/models/kimi-k3.cpp
`glm5-next`	recurrent / linear-attention layers with a fixed-size state; sparse attention with indexer keys (counted as extra K)	src/models/glm5-next.cpp
`minimax-01`	cache layout not modelled yet — models with this architecture are not listed	src/models/minimax-01.cpp
`plamo2`	cache layout not modelled yet — models with this architecture are not listed	src/models/plamo2.cpp
`deepseek4`	cache layout not modelled yet — models with this architecture are not listed	src/models/deepseek4.cpp
`dots3note`	cache layout not modelled yet — models with this architecture are not listed	src/models/dots3note.cpp
`hy_v4`	cache layout not modelled yet — models with this architecture are not listed	src/models/hy-v4.cpp
`minimax-m3`	cache layout not modelled yet — models with this architecture are not listed	src/models/minimax-m3.cpp

Sources and how specifications are checked

Manufacturer websites are never fetched, because their terms forbid automated access. The manufacturer’s specification comes from open datasets that link a manufacturer datasheet row by row (GetDeploying and Epoch AI, when the datasheet is on a manufacturer’s domain or on a document host that the manufacturer itself uses for its datasheets) and from AMD’s ROCm documentation. Those datasheet links are shown as “manufacturer reference”, marked as not fetched by this site. Datasheet links count as the manufacturer’s when they are on one of these domains (including document hosts the manufacturers use): amd.com, apple.com, dam-cdn.nvd.orangelogic.com, geforce.com, intel.com, nvdam.widen.net, nvidia.com
Wikipedia is used as an independent cross-check. Its footnotes are not used as the source of a value, because column and product-name footnotes do not show where a specific field came from.
The manufacturer-specification vote and the cross-check vote must come from different publishers with no reference between them: if the cross-checking Wikipedia page (or dataset row) cites the other publisher, it counts as derived, not as a second vote. A source that copies another without saying so cannot be detected from public evidence.
A value is shown (two_source) when a manufacturer-specification vote and an independent cross-check agree and no healthy source disagrees. Otherwise it is “conflict” (sources disagree), “single” (no manufacturer-specification vote or no independent cross-check) or “missing”.
Rows that an open dataset attributes to TechPowerUp (Epoch AI rows whose datasheet link points to techpowerup.com) are not read at all, because TechPowerUp’s terms forbid derived datasets: they neither vote nor count as a disagreement.
Agreement means: memory size exactly equal, bandwidth within 1%, other numbers within 2%, text equal after normalization.
The value shown is the one most agreeing sources give exactly; every source’s own figure and link is listed next to it. Wikipedia links point to the exact revision used.
Apple silicon, DGX Spark and Strix Halo computers are not published in this version: no open source links a manufacturer datasheet for them row by row.
Model data comes from Hugging Face at pinned revisions. Each model uses one GGUF repository for all its quantizations, chosen by a fixed publisher order; the architecture is read from the GGUF header and, where the original config.json is readable, cross-checked against it.
Original weights, shown for reference only, count the repository’s safetensors files at the top level and in model* folders: with a model.safetensors.index.json, the files it names plus complete sets in other model* folders; without one, every complete set, with a merged file and an identical sharded copy counted once. Other folders (for example separately stored decoders), original/, consolidated* files, LoRA adapters and .bin / .pt files are not counted. When the files do not form one unambiguous set, no size is given.

Parameter counts

Parameters are counted from the GGUF tensor table of the chosen GGUF file: every tensor llama.cpp loads, including the input embedding table. Multi-token-prediction (MTP / NextN) layers, which llama.cpp does not load by default, are not counted; model pages that have them say how many. Active parameters per token count the non-expert tensors plus the experts in use (experts used ÷ experts per layer), but not the input embedding lookup, because only one row of it is read per token. When a model uses one matrix as both its input embedding and its output layer (tied embeddings), every token is multiplied by all of it, so it is counted once, as the output layer. Model cards often also count the input embedding table and MTP layers, so their figures can be noticeably higher, especially for small mixture-of-experts models with large vocabularies. The original repository’s own count is shown separately.

Active parameters are given only for mixture-of-experts models, that is, models whose GGUF header lists more than one expert per layer. A model with a single expert per layer (expert count 1) is dense: every parameter is used for every token, so no separate active figure is shown.

The original repository’s own count is the total Hugging Face reports for its safetensors weights; some repositories state it wrongly in their metadata. It is shown, and included in the downloads, only when it is confirmed: it must be more than 100M parameters, within 1% of the per-type parameter counts that Hugging Face reads from the repository’s weight files, and, for models with a GGUF file, between 0.5 and 2.5 times the GGUF count (the original can also hold a vision or audio encoder or multi-token-prediction layers; the GGUF can be larger when it stores a tied embedding a second time as its output layer). Models without a GGUF file need the first two checks. Otherwise the count is left out.

What each status shows

Status	On the page
`two_source`	Shown: a manufacturer specification linked by an open source agrees with an independent cross-check.
`conflict`	Open sources disagree, so this value is not shown; estimates that depend on it (such as speed) are not given either.
`single`	Not shown: open sources do not give two independent, agreeing values with a manufacturer reference.
`missing`	Not shown: open sources do not give two independent, agreeing values with a manufacturer reference.
`withheld`	The entry was published before, but its sources now disagree, a source changed or it no longer passes the listing rules. Its page stays (not indexed) with the source links; values, estimates and verdicts are hidden. After 30 days with healthy sources it is retired; when two sources agree again it returns.
`retired`	Withheld for 30 days with healthy sources. The page stays as a tombstone (not indexed) and links to a replacement when there is one.

Speed ceiling

Generating a token reads the active weights and the KV cache once, so tokens per second cannot exceed bandwidth divided by the bytes read per token. For mixture-of-experts models only the share of experts in use is counted, from the GGUF tensor table. It is a theoretical upper bound; no speed is given when the bandwidth is not confirmed by two sources.

Public benchmarks

Measured llama.cpp speeds published by the llama.cpp project in its repository (llama-bench, at the pinned llama.cpp commit). The machines in these notes (DGX Spark, Apple M2 Ultra) do not get verdicts on this site in this version; the numbers show how measured speed compares with the bandwidth upper bound, and how it falls as the context grows: llama.cpp benches/ (5aeac37a)

llama-bench results as published (tokens per second; the source file is linked on each number). Prompt processing = reading the prompt; generation = producing new tokens.
Machine	Model (GGUF repository)	Backend	Test	Tokens/s
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	CUDA	prompt processing, 2048 tokens	4,505.8
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	CUDA	generation, 32 tokens	83.4
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	2,689.4
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	CUDA	generation, 32 tokens · after 32,768 tokens of context	61.7
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	CUDA	prompt processing, 2048 tokens	2,443.9
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	CUDA	generation, 32 tokens	58.7
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	1,567.1
NVIDIA DGX Spark (GB10)	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	CUDA	generation, 32 tokens · after 32,768 tokens of context	42.8
NVIDIA DGX Spark (GB10)	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	CUDA	prompt processing, 2048 tokens	2,987.0
NVIDIA DGX Spark (GB10)	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	CUDA	generation, 32 tokens	61.1
NVIDIA DGX Spark (GB10)	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	1,348.2
NVIDIA DGX Spark (GB10)	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	CUDA	generation, 32 tokens · after 32,768 tokens of context	30.2
NVIDIA DGX Spark (GB10)	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	CUDA	prompt processing, 2048 tokens	2,250.3
NVIDIA DGX Spark (GB10)	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	CUDA	generation, 32 tokens	29.4
NVIDIA DGX Spark (GB10)	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	1,471.2
NVIDIA DGX Spark (GB10)	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	CUDA	generation, 32 tokens · after 32,768 tokens of context	22.5
NVIDIA DGX Spark (GB10)	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	CUDA	prompt processing, 2048 tokens	5,948.7
NVIDIA DGX Spark (GB10)	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	CUDA	generation, 32 tokens	81.1
NVIDIA DGX Spark (GB10)	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	5,023.3
NVIDIA DGX Spark (GB10)	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	CUDA	generation, 32 tokens · after 32,768 tokens of context	62.3
NVIDIA DGX Spark (GB10)	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	CUDA	prompt processing, 2048 tokens	2,364.2
NVIDIA DGX Spark (GB10)	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	CUDA	generation, 32 tokens	48.7
NVIDIA DGX Spark (GB10)	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	567.2
NVIDIA DGX Spark (GB10)	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	CUDA	generation, 32 tokens · after 32,768 tokens of context	32.7
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Super-120B-GGUF (nemotron 120B.A12B Q4_K)	CUDA	prompt processing, 2048 tokens	768.8
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Super-120B-GGUF (nemotron 120B.A12B Q4_K)	CUDA	generation, 32 tokens	19.9
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Super-120B-GGUF (nemotron 120B.A12B Q4_K)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	724.4
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Super-120B-GGUF (nemotron 120B.A12B Q4_K)	CUDA	generation, 32 tokens · after 32,768 tokens of context	19.5
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Nano-4B-GGUF (nemotron 4B Q8_0)	CUDA	prompt processing, 2048 tokens	2,761.9
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Nano-4B-GGUF (nemotron 4B Q8_0)	CUDA	generation, 32 tokens	52.9
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Nano-4B-GGUF (nemotron 4B Q8_0)	CUDA	prompt processing, 2048 tokens · after 32,768 tokens of context	2,041.5
NVIDIA DGX Spark (GB10)	ggml-org/Nemotron-3-Nano-4B-GGUF (nemotron 4B Q8_0)	CUDA	generation, 32 tokens · after 32,768 tokens of context	46.7
Apple M2 Ultra	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	MTL,BLAS	prompt processing, 2048 tokens	2,713.4
Apple M2 Ultra	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	MTL,BLAS	generation, 32 tokens	130.0
Apple M2 Ultra	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	MTL,BLAS	prompt processing, 2048 tokens · after 32,768 tokens of context	1,122.6
Apple M2 Ultra	ggml-org/gpt-oss-20b-GGUF (gpt-oss 20B MXFP4 MoE)	MTL,BLAS	generation, 32 tokens · after 32,768 tokens of context	98.3
Apple M2 Ultra	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	MTL,BLAS	prompt processing, 2048 tokens	1,648.7
Apple M2 Ultra	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	MTL,BLAS	generation, 32 tokens	85.6
Apple M2 Ultra	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	MTL,BLAS	prompt processing, 2048 tokens · after 32,768 tokens of context	721.1
Apple M2 Ultra	ggml-org/gpt-oss-120b-GGUF (gpt-oss 120B MXFP4 MoE)	MTL,BLAS	generation, 32 tokens · after 32,768 tokens of context	65.5
Apple M2 Ultra	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	MTL,BLAS	prompt processing, 2048 tokens	2,453.1
Apple M2 Ultra	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	MTL,BLAS	generation, 32 tokens	79.0
Apple M2 Ultra	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	MTL,BLAS	prompt processing, 2048 tokens · after 32,768 tokens of context	431.3
Apple M2 Ultra	ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF (qwen3moe 30B.A3B Q8_0)	MTL,BLAS	generation, 32 tokens · after 32,768 tokens of context	45.3
Apple M2 Ultra	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	MTL,BLAS	prompt processing, 2048 tokens	1,565.9
Apple M2 Ultra	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	MTL,BLAS	generation, 32 tokens	79.7
Apple M2 Ultra	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	MTL,BLAS	prompt processing, 2048 tokens · after 32,768 tokens of context	612.2
Apple M2 Ultra	ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF (qwen2 7B Q8_0)	MTL,BLAS	generation, 32 tokens · after 32,768 tokens of context	56.8
Apple M2 Ultra	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	MTL,BLAS	prompt processing, 2048 tokens	2,923.6
Apple M2 Ultra	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	MTL,BLAS	generation, 32 tokens	134.3
Apple M2 Ultra	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	MTL,BLAS	prompt processing, 2048 tokens · after 32,768 tokens of context	2,129.2
Apple M2 Ultra	ggml-org/gemma-3-4b-it-qat-GGUF (gemma3 4B Q4_0)	MTL,BLAS	generation, 32 tokens · after 32,768 tokens of context	113.1
Apple M2 Ultra	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	MTL,BLAS	prompt processing, 2048 tokens	1,629.3
Apple M2 Ultra	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	MTL,BLAS	generation, 32 tokens	59.6
Apple M2 Ultra	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	MTL,BLAS	prompt processing, 2048 tokens · after 32,768 tokens of context	151.4
Apple M2 Ultra	ggml-org/GLM-4.7-Flash-GGUF (deepseek2 30B.A3B Q8_0)	MTL,BLAS	generation, 32 tokens · after 32,768 tokens of context	21.8

Macs and other unified-memory computers

These computers let the GPU use only part of their memory, and the open sources for them do not cite references per value. They are not published in this version; the run profiles for them are kept for when the sources allow it.

Units

Memory amounts calculated here are in GiB (1,073,741,824 bytes). GPU makers' "24 GB" of video memory is 24 GiB. Hugging Face shows file sizes in decimal GB, so a 5.03 GB file appears here as 4.68 GiB. Bandwidth is in GB/s (10⁹ bytes per second).

Updates

Hugging Face is scanned every 6 hours for new models and new GGUF files; GPU specifications are re-checked daily.