hardNewGroqCerebras

Why are Groq and Cerebras so fast at batch 1, and what does that speed cost at scale?

Put the weights in on-chip SRAM and the HBM wall disappears: tens of terabytes per second per chip, and a batch-1 step limited by the pipeline rather than the memory. The price is capacity: a 70B needs hundreds of chips per replica, and the cost per token depends on keeping every one of them busy.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Cerebras, Groq and Dataflow AcceleratorsCerebras and Groq attack the same wall from the same direction: keep the working set in on-chip SRAM, which is tens of times faster than HBM, so that memory-bound decode stops being memory-bound. Cerebras does it with a wafer-scale engine holding 44 GB of SRAM; Groq with a deterministic, compiler-scheduled dataflow chip holding a few hundred megabytes per part and many parts per model. Both win single-stream decode by large factors and pay in capacity, cost per token at high batch, and flexibility. Knowing where each fits is the interview question.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Cost per Million TokensThe unit every serving decision cashes out in. It is one formula: the fleet's dollars per second divided by the tokens per second it sustains, scaled to a million, with utilization in the denominator because idle replicas still cost money. This page derives it from a GPU price and a throughput estimate, works it at three batch sizes to show why batching is the main lever, separates prefill from decode pricing, and shows how the same fleet's cost per token moves by 5x between a quiet hour and a busy one.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Advanced

🕸️ Distributed Training🔒 Premium

Ring vs Tree All-ReduceA ring all-reduce moves the minimum possible bytes per rank but takes 2(N-1) steps, so its latency grows with the number of GPUs; a tree finishes in a logarithmic number of steps but is harder to keep bandwidth-optimal. NCCL keeps both, chooses per message size and rank count, and reading its choice is how you diagnose a collective that is slower than the fabric allows.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackMIG versus MPS: what isolation does each give you when sharing a GPU, and which would you pick for a multi-tenant inference node?Next in this trackYou are moving a model to fp4 inference on Blackwell. What breaks first, and how would you measure whether the result is acceptable?Next in this trackDesign a non-blocking network fabric for 512 H100s. How many switches, how are they wired, and where does the NVLink domain end?

FEDITOR'S NOTE

Scored on deriving the batch-1 advantage from bandwidth and the scale disadvantage from capacity, using the same roofline the candidate would use for a GPU, and stating the batch regime where each design wins.

DISCUSSION · 0

No comments yet — be the first to share your approach.