Reading config.json to Size a Model You Have Never Run

TL;DR: Nine fields carry the sizing. num_hidden_layers, hidden_size, num_attention_heads and num_key_value_heads give the dense shape. n_routed_experts, num_experts_per_tok, n_shared_experts and moe_intermediate_size give the sparse shape and therefore the active parameter count. kv_lora_rank and qk_rope_head_dim, when present, mean the model uses compressed latent attention and the KV cache is those two summed rather than heads times head dimension. From those you compute three things: weight bytes as total parameters times bytes per parameter from quantization_config, KV bytes per token, and the minimum GPU count as weights plus KV plus roughly 15 percent overhead, divided by memory per GPU, rounded up to a parallel degree that divides the head and expert counts. The whole derivation is five minutes and it is the difference between an engineer who can onboard a model and one who runs a command and reports that it crashed.

The fields, and what each one is for

The derivation, worked on a real config

Using GLM-5.3's published config.json as of September 2026: 78 layers, hidden 6,144, 64 heads, head_dim 192, kv_lora_rank 512, qk_rope_head_dim 64, 256 routed experts, 8 per token, 1 shared, moe_intermediate_size 2,048, first_k_dense_replace 3, intermediate_size 12,288, vocab 154,880, FP8 e4m3 with 128x128 block scaling, 753B total parameters.

step 1: weight bytes
  FP8 is 1 byte per parameter, plus a small block-scale overhead
  753e9 x 1 B = 753 GB
  add roughly 1% for scales and metadata: about 760 GB

step 2: KV per token
  kv_lora_rank + qk_rope_head_dim is present, so this is latent attention:
  per layer per token = (512 + 64) x 2 B (bf16 cache) = 1,152 B
  x 78 layers = 89,856 B = 87.75 KB per token

step 3: KV budget for a target concurrency
  target: 256 concurrent sequences at 32,768 tokens average
  KV bytes = 256 x 32,768 x 89,856 = 753 GB
sanity: the KV budget equals the weight footprint at this concurrency, which is the regime
        these models are designed for and is why capacity planning cannot ignore either half

step 4: minimum GPU count
  weights + KV + about 15% for activations, fragmentation and the CUDA context
  (760 + 753) x 1.15 = 1,740 GB
  on B300 at 288 GB:  1,740 / 288 = 6.0  -> 8 GPUs (next valid degree)
  on B200 at 180 GB:  1,740 / 180 = 9.7  -> 16 GPUs
  on H100 at 80 GB:   1,740 / 80  = 21.8 -> 24 or 32 GPUs
  weights alone on H100: 760 / 80 = 9.5 -> 16 GPUs before any KV at all

step 5: which parallel degrees divide cleanly
  64 attention heads divide by 1, 2, 4, 8, 16, 32, 64
  256 routed experts divide by any power of two up to 256
  so TP of 8 and EP of 8, 16, 32 or 64 all partition evenly
  a degree that does not divide the head count forces padding and wastes memory
sanity: the head count is usually the binding constraint on TP, and the expert count is
        usually generous, which is why large MoE deployments use modest TP and large EP

Active parameters, which the card often does not state

active parameters per token, from the MoE fields
  per sparse layer, experts active = num_experts_per_tok + n_shared_experts = 8 + 1 = 9
  one expert's parameters = 3 matrices (gate, up, down) x hidden x moe_intermediate
    = 3 x 6,144 x 2,048 = 37.7M
  per sparse layer = 9 x 37.7M = 340M
  sparse layers = 78 - first_k_dense_replace = 75
  MoE contribution = 75 x 340M = 25.5B

  dense layers = 3, each 3 x hidden x intermediate = 3 x 6,144 x 12,288 = 226M
  dense contribution = 3 x 226M = 0.68B

  attention, all 78 layers, latent projections summed from the config's rank fields
    approximately 165M per layer = 12.9B

  embeddings and output head = 2 x 154,880 x 6,144 = 1.9B

  total active ≈ 25.5 + 0.68 + 12.9 + 1.9 + indexer ≈ 43B
sanity: 43B of 753B is 5.7% active, which matches the sparsity that published summaries of
        this model report, so the derivation reproduces the number rather than assuming it

The traps

• Assuming KV from heads and head dimension when latent fields are present. For GLM-5.3 that overestimates by more than forty times, which turns a workable plan into an absurd one.
• Forgetting that memory is sized by total parameters while speed is bounded by active ones. They are different numbers on every model in this class.
• Ignoring quantization_config. A model released in FP8 sized as bf16 doubles the GPU count for no reason.
• Picking a parallel degree that does not divide the head count, which pads and wastes memory silently.
• Sizing for weights only. At realistic concurrency the KV cache is comparable to the weights, as the worked example shows.

What interviewers are listening for

The derivation done out loud, in that order, with the arithmetic visible. This is one of the few serving questions with a single correct method, so the signal is whether the candidate knows it. The strongest additional move is checking whether the parallel degree divides the head and expert counts, because that catches a class of failure that only appears at launch time. Interviewers also listen for the KV budget being computed at a stated concurrency rather than for one sequence, since a single-sequence number makes any model look free.

Key takeaways

• Nine fields carry sizing: layers, hidden, heads, KV heads, latent ranks, expert counts, expert width, dense-layer count and the quantization config.
• Latent attention means KV per token per layer is kv_lora_rank + qk_rope_head_dim, not heads times head dimension.
• GLM-5.3 works out to about 760 GB of FP8 weights and 87.75 KB of KV per token, so 256 sequences at 32k tokens adds another 753 GB.
• Minimum GPUs is (weights plus KV) times about 1.15, divided by per-GPU memory, rounded up to a degree that divides the head and expert counts.
• Memory follows total parameters and decode speed follows active parameters, and on these models the two differ by twenty times or more.