granite-4.0-350m: how much VRAM does it need?

This page was written and translated automatically by AI and checked by software against its sources, with no human editing. Specifications come from the sources listed; memory requirements and speeds are estimates calculated with published formulas. How we use AI

granite-4.0-350m has 352M parameters. At Q4_K_M with a context of 8,192 tokens, a single GPU needs 1.7 GiB of available memory for the result to be “Fits”.

Values updated Oct 19, 2026

Key facts

GGUF files and memory needed

All quantizations come from one GGUF repository at one pinned revision: ibm-granite/granite-4.0-350m-GGUF@b8208a8

Smallest GPU for each quantization

GPUs that can run granite-4.0-350m

“Fits” means llama.cpp's default --fit load check would place all layers on the GPU: our estimate of llama.cpp's buffers, plus its error margin, plus llama.cpp's 1 GiB per-card reserve, is within the memory counted as available. It does not promise that peak memory while generating fits. How results are decided

GPUs without a verdict

• CUDA (NVIDIA): llama.cpp, CUDA back end (NVIDIA discrete GPU), all layers on the GPU, Flash Attention on, micro-batch (-ub) 512, F16 KV cache, context (-c) as shown; “Fits” is judged with llama-server’s default 4 parallel sequences (-np auto, unified KV cache) and “Doesn’t fit” with 1 sequence; idle GPU with no other programs using its memory.
• ROCm on Linux (AMD): llama.cpp, ROCm back end on Linux (AMD discrete GPU), all layers on the GPU, Flash Attention on, micro-batch (-ub) 512, F16 KV cache, context (-c) as shown; “Fits” is judged with llama-server’s default 4 parallel sequences (-np auto, unified KV cache) and “Doesn’t fit” with 1 sequence; idle GPU with no other programs using its memory.

The speed ceiling is memory bandwidth divided by the bytes read per generated token. It is a theoretical upper bound; public llama.cpp benchmarks typically reach about half to four-fifths of it, and small models and mixture-of-experts models often less. Public benchmarks

Original weights

The model’s original weight files (safetensors, counted as described in the methodology) total 0.7 GiB. They are listed for reference only; every estimate on this page is for the GGUF files above, run with llama.cpp. ibm-granite/granite-4.0-350m · Methodology

Sources

• Hugging Face: ibm-granite/granite-4.0-350m@bd8a149
• Hugging Face (GGUF): ibm-granite/granite-4.0-350m-GGUF@b8208a8
• Methodology (llama.cpp 5aeac37a)