Skip to content

TechnologyReading time 8 min.

Estimating GPU memory for language models: weights, KV cache and overhead

Whether a language model fits on a graphics card can be calculated before ordering. The requirement consists of three items, only one of which is fixed with the model.

Managed by the Platform department

A passively cooled data centre graphics card lies flat on a light background, the gold-plated connector facing the camera; next to it, a single memory chip on grey foam.

At a glance

  • The weights occupy the number of parameters times bytes per parameter: two bytes in BF16 or FP16, one byte in FP8 or INT8 and half a byte at four bits.
  • The KV cache grows with context length and the number of concurrent requests, and can occupy more GPU memory than the weights for long contexts.
  • For the runtime environment and activations, an overhead of a tenth to a fifth is a rule of thumb, not a measured value.

Three items in GPU memory

A language model occupies three things in GPU memory: its weights, the KV cache of running requests and a remainder for the runtime environment, intermediate results and fragmentation. The weights are fixed with the model and data type. The KV cache depends on how long the contexts are and how many requests are running concurrently; the software determines the rest.

If you only calculate the weights, you will order too small. A model whose weights just fit into a card has no room left for a single long context.

Calculations in this article are decimal, with 1 GB equal to one billion bytes. Tools in the guest system often display GiB; one GiB is around 7% larger than one GB.

Weights: number of parameters times bytes per parameter

Each parameter occupies as many bytes as its data type dictates. BF16 and FP16 store a value in two bytes, FP8 and INT8 in one byte, a four-bit format in half a byte. With billions of parameters, the product directly yields gigabytes.

The number of parameters is in the model description; the size of the weight files serves as a cross-check, which in BF16 comprise roughly twice as many gigabytes as the model has billions of parameters. For models consisting of multiple experts, all experts count: only a portion of them compute per token, but all of them reside in the GPU memory.

Weights in GB = parameters in billions × bytes per parameter

GPU memory of weights by data type
Data typeBytes per parameter8 billion parameters70 billion parameters
BF16 or FP16216 GB140 GB
FP8 or INT818 GB70 GB
4 bits0.54 GB35 GB

At four bits, the actual value is slightly higher: scaling factors per group of weights are stored alongside, and individual layers often remain in higher precision. Whether a smaller data type maintains the response quality for your task can only be shown by a test with your own data.

This calculation is not sufficient for training. Gradients and optimiser states are added to the weights, and the requirement per parameter increases to a multiple.

KV cache: memory per token and request

So that a model does not recalculate the entire context for each new token, it keeps keys and values of each layer in memory for every previous token. This results in the formula for the KV cache; the two at the beginning stands for key and value.

KV cache in bytes = 2 × layers × KV heads × head dimension × bytes per value × context length × concurrent requests

The number of layers, number of KV heads and head dimension are in the model's configuration file. Many newer models share a KV head between several attention heads; then the smaller number of KV heads counts, not the number of attention heads. Older architectures without this grouping maintain as many KV heads as attention heads, and their KV cache is a multiple for the same size. For models that store keys and values in a compressed form or only keep a window of the context in view, the result is an upper limit.

With the values from Example A below, the formula yields 131,072 bytes per token, around 0.13 MB. A single context with 8,192 tokens thus occupies 1.07 GB, and each additional concurrent request of the same length occupies the same amount again.

The context length is linear, as is the number of concurrent requests. If you halve the permitted context length, you halve the KV cache; if you keep it in FP8 instead of BF16, you do the same.

What the formula does not know

Some runtime environments reserve a fixed portion of the GPU memory at startup and distribute it as a KV cache. The card's utilisation then shows this portion, not the calculated requirement.

Two calculation methods

The overhead for the runtime environment, activations and fragmentation is added to the weights and KV cache. We calculate with 10 to 20%; this is a rule of thumb, not a measured value, and the actual size depends on the software, batch size and context length. For small models, the fixed portion of the runtime environment is more significant; there you are better off calculating with the upper limit.

Requirement = (weights + KV cache) × 1.1 to 1.2

Both examples assume an architecture that is common for models of this size; for a specific model, you insert its values.

Example A: 8 billion parameters in BF16
Weights: 8 billion × 2 bytes
16.0 GB
KV per token: 2 × 32 layers × 8 KV heads × 128 × 2 bytes
131,072 bytes
KV cache: 8,192 tokens × 16 concurrent requests
17.18 GB
Weights and KV cache
33.18 GB
Requirement with overhead
36.5 to 39.8 GB

The KV cache is larger than the weights here. With 4 concurrent requests instead of 16, it drops to 4.29 GB. If weights in FP8 are added, the requirement is between 13.5 and 14.8 GB, and the same model fits on a card with 24 GB.

Example B: 70 billion parameters in FP8
Weights: 70 billion × 1 bytes
70.0 GB
KV per token: 2 × 80 layers × 8 KV heads × 128 × 2 bytes
327,680 bytes
KV cache: 8,192 tokens × 8 concurrent requests
21.47 GB
Weights and KV cache
91.47 GB
Requirement with overhead
100.6 to 109.8 GB

This does not fit into any single-card configuration, the largest of which has 96 GB; in BF16, the weights alone would require 140 GB. With weights in four bits, however, Example B drops to 56.47 GB before overhead.

Insert the load you expect, not the smallest conceivable one: runtime environments limit context length and the number of concurrent requests via settings, and exactly these two values belong in the formula. If the GPU memory is not enough, there are four levers.

  • Fewer concurrent requests reduce the KV cache and lower throughput.
  • Shorter permitted contexts reduce the KV cache and exclude long inputs.
  • A KV cache in FP8 instead of BF16 takes up half the space.
  • A smaller data type for the weights saves a lot as long as the weights are the largest item, but can change the response quality.

A load test with real requests will show which of these is acceptable for your application.

Which configuration fits

The table places the range of both examples next to the GPU memory of each configuration. “Fits” means that the upper limit also fits, “Tight” means that only the lower limit fits; in tight cases, a load test decides.

For Example A, NG-L40S with 48 GB is the smallest configuration that fits. Example B requires NG-H100×2 with 160 GB of GPU memory in FP8; with weights in four bits, however, it fits on a single card.

GPU configurations against the two calculation methods
ConfigurationCardsTotal GPU memoryExample AExample BNet monthly price
NG-RTX40001 × NVIDIA RTX PRO 4000 Blackwell SFF, 24 GB each24 GBNot sufficientNot sufficient€214.00$248.24
NG-L41 × NVIDIA L4, 24 GB each24 GBNot sufficientNot sufficient€540.00$626.40
NG-L40S1 × NVIDIA L40S, 48 GB each48 GBSufficientNot sufficient€1,008.00$1,169.28
NG-H1001 × NVIDIA H100 SXM, 80 GB each80 GBSufficientNot sufficient€1,940.00$2,250.40
NG-RTX60001 × NVIDIA RTX PRO 6000 Blackwell Max-Q, 96 GB each96 GBSufficientNot sufficient€1,199.00$1,390.84
NG-H100×22 × NVIDIA H100 SXM, 80 GB each160 GBSufficientSufficient€3,880.00$4,500.80
NG-H100×44 × NVIDIA H100 SXM, 80 GB each320 GBSufficientSufficient€7,770.00$9,013.20
NG-H200×88 × NVIDIA H200 SXM, 141 GB each1,128 GBSufficientSufficient€30,660.00$35,565.60

The monthly prices are net catalogue prices for Frankfurt am Main. The RTX PRO 6000 is listed in the catalogue in several expansion stages that differ in memory and drives, not in the card; the table lists the cheapest. Each card belongs entirely to your node, without vGPU sharing, so the full GPU memory is available.

The catalogue lists multi-card configurations in the H100 and H200 class, with NVLink between the cards and up to 8 cards per node. A model is then distributed across the cards: each card holds its share of the weights and the KV cache, and each needs its own overhead. The total GPU memory is therefore an upper limit, not a guarantee.

Whether a model fits is determined by the size of the GPU memory; how fast it responds depends primarily on the memory bandwidth when generating the response, and the catalogue shows the memory type per card for this.

A few hours are enough for such a load test. Billing is the same as for all hourly billed lines: recorded to the second, billed per hour, capped monthly; without a term, you pay for the time used. A running card runs until you stop it; there are no interruptible instances at spot prices. Which cards start immediately and which are provided on request based on capacity can be found under the models on the GPU instances page.

The finder on the same page names the smallest card for the weights of your model and calculates the KV cache as a flat-rate overhead; you add long contexts and many concurrent requests using the formula above. If you only want to call one of the ready-made models of the AI platform, you do not need your own card: the AI platform bills per million tokens, with no base fee.

Published on 8 November 2026. Prices and product values in this post are taken from the price list and reflect their current status.

Further reading

Read more

  • A power meter with two current clamps on a light background; one clamp encloses a thick black power cable running across the image.

    What the PUE value measures – and what it does not say

    TechnologyReading time 6 min.

    The PUE value compares the energy demand of a data centre to what the IT actually consumes. It thus answers a narrow question precisely – and many obvious questions not at all.

    Operations department

  • A light grey patch panel from the front; fibre optic connectors are plugged into four dual ports, the orange cables run out of the picture in a flat curve.

    Billing traffic: included volume, overage and one rate for every destination

    TechnologyReading time 8 min.

    Outbound traffic is included up to the product's included volume, inbound traffic costs nothing. Anything above this is billed at a single rate per terabyte – regardless of the location where the system is running and where the data is going.

    Network department

  • Three light grey servers in three sizes side by side on a light background: a small chassis, a flat 1U server and a 2U server with twelve drive bays.

    VPS, cloud instance or dedicated server: a decision guide

    PracticeReading time 6 min.

    The three architectures differ in who owns the processor core, how they are billed and how quickly a system is provisioned. This article categorises them based on the questions that matter before placing an order.

    Platform department

Questions for the department

Technical questions about a post are answered by the person responsible for the department — in a conversation, not a presentation. For quotes and framework agreements, sales will lead the initial conversation.