AI Infra Interviews logo
← CUDA, Triton & Kernel Engineering / 11
mediumNewOpenAITogether AI

When do you write a kernel in Triton, and when do you have to drop down to CUDA?

Triton hands you a block of data and writes the thread-level code for you, which covers most of what a serving or training stack actually needs. The four things it does not give you, the performance you give up in each case with numbers, the reversal as Triton gains Hopper features, and how to decide in the room.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Triton hands you a block of data and writes the thread-level code for you, which covers most of what a serving or training stack actually needs. The four things it does not give you, the performance you give up in each case with numbers, the reversal as Triton gains Hopper features, and how to decide in the room.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational
⚡ Kernels & Compilers
Kernel FusionAn elementwise or reduction kernel does a few FLOPs per byte and runs at HBM speed, so a chain of five of them costs five trips through HBM for work that needs one. Fusion collapses the chain into a single kernel that keeps intermediates in registers. It is the first lever for anything memory-bound, and knowing what it cannot fix (weight reads in decode, the GEMMs themselves) is what the interview is really testing.
Core
⚡ Kernels & CompilersSign in
Triton Programming ModelTriton replaces CUDA's thread with a program that owns a whole block of data, and replaces manual shared-memory staging and coalescing with a compiler that derives them from block shapes. Pointer arithmetic on vectors, masks for the tail, and program-id swizzling for L2 reuse are the three idioms every Triton kernel is built from, and the live exercise at Anthropic, OpenAI and the serving startups is usually one of a fused softmax, a LayerNorm or a matmul in exactly this style.
Advanced
⚡ Kernels & Compilers🔒 Premium
Tiled Matrix MultiplicationA matrix multiply has enough reuse to be compute-bound, but only if the kernel captures that reuse in shared memory and registers instead of re-reading HBM. Tiling is how: a block owns an output tile, streams K-slices of A and B through shared memory, and each thread accumulates a small register tile. It is the live-coding exercise that separates people who know the roofline from people who have climbed it.
Foundational
⚡ Kernels & Compilers
CUDA Programming ModelCUDA splits a program into a host that allocates, copies and enqueues work, and a device that runs thousands of identical threads organized as a grid of blocks. Getting the split right, and knowing that a launch returns before the kernel runs, decides whether your first live-coding kernel produces a correct number or a silent zero.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackHere is an Nsight Compute report for a slow kernel. Read it, name the bottleneck, and tell me what you would change.Next in this trackSum 100 million floats on the GPU as fast as the hardware allows. Write the kernel and justify each step.Next in this trackTranspose a large matrix at close to copy bandwidth. Why is the naive version slow, and what does each fix buy?
FEDITOR'S NOTE

Scored on knowing that Triton programs are block-level rather than thread-level, on naming specific cases where hand CUDA still wins and roughly by how much, and on treating the boundary as something that moves.

DISCUSSION · 0

No comments yet — be the first to share your approach.