Fireworks AI CUDA, Triton & Kernel Engineering interview questions

A warp issues one load instruction and the memory system turns it into some number of 32-byte sector requests; that number is the whole story. The arithmetic for contiguous, stride-2 and stride-32 access, the row-major matrix where a loop order change gives 8x, and the two Nsight Compute counters that show the waste.

A chain of five elementwise operations reads and writes the tensor five times when once would do. The byte arithmetic for an unfused chain against a fused one, the launch overhead that matters at small sizes, the three fusion shapes, and the operations where fusion changes nothing because the matmul was at the roof.

One program per row, the row in registers, max then exp then sum then divide, one read and one write of HBM. The runnable kernel with its launch, the numerics (subtract the max, accumulate in fp32), the block-size rule and the wide-row limit, and the arithmetic that says the kernel is done at 90% of copy speed.

The naive kernel reads two bytes per multiply-add; a 128x128 tile with register blocking reads a few hundredths of that. Each step (shared-memory tiles, register blocking, vectorized loads, double buffering, tensor cores) with its intensity arithmetic and speedup, and where a hand kernel stops and cuBLAS begins.

Standard attention writes an N by N score matrix to HBM and reads it back twice. At 8k context that is 134 MB per head. The traffic arithmetic before and after tiling, the online-softmax identity that makes one pass possible, where the kernel lands on the roofline, and why the memory saving matters more than the speed.

Each version fixed a different bottleneck: version 1 fixed memory traffic, version 2 fixed non-matmul work and parallelism, version 3 fixed the fact that softmax and matmul were waiting for each other on Hopper. The arithmetic that shows why a softmax costing one percent of the FLOPs can cost sixty percent of the time.

Four sections, read in a fixed order, and one number that usually names the bottleneck outright. The report of a kernel at 12 percent of DRAM bandwidth while its memory pipeline reads 82 percent busy, what that gap means, the fix it implies, and the numbers the fixed kernel reports back.

At batch 1 a decode step is a few milliseconds of GPU work behind several hundred kernel launches, and the CPU cannot issue them fast enough. The arithmetic showing the GPU starve, what replay changes, and the four constraints capture imposes on memory.

A library call gives you a tuned matmul and nothing around it. Four things a template library gives instead: an epilogue fused into the output stage, one launch for a list of ragged problems, a matmul whose inputs are different types, and a tile schedule that fills the SMs on awkward shapes.

The matmul runs on tiny formats and the accumulator does not, so every low-precision GEMM is really a scaled integer or float product with a correction applied on the way out. Where the scales come from, the three granularities and what each costs, and why one of them needs a periodic promotion into fp32.

Routing hands each expert a different number of tokens, so the layer is a list of ragged matmuls rather than one. Three separate costs a loop pays, the measured share of compute that skewed routing wastes on tile padding, and what a single grouped launch fixes and what it cannot.

Six things decide whether a kernel measurement means anything, and a script that ignores them can report a number that is wrong by a factor of two in either direction. What to control, which statistic to report, and the six ways a large speedup turns out to be a measurement artifact.

A bug that scales with the number of tiles is invisible in a unit test that fits in one tile. The three candidates that produce exactly this signature, the measurement that separates them, and a test design that would have caught all three before the kernel shipped.

The kernel is four lines; the interview is about the other forty. How a global index comes out of block and thread ids, why the bounds check exists, what the launch configuration means for a 100M-element array, and the arithmetic that shows the host-to-device copies cost 50 times more than the add.

Thirty-two addresses per warp instruction, thousands of instructions: say what the pattern is and how many sectors it cost. The address-delta test for contiguous and strided, the sector count that measures the damage, the code that does both, and the edge cases (misalignment, inactive lanes, mixed widths).

Shared memory has 32 banks, each 4 bytes wide, and a warp's access is as slow as the most-loaded bank. The bank of an address, why a column walk down a 32-wide tile puts all 32 lanes in one bank, the padding by one column that spreads them across all 32, and the profiler counter that confirms the fix.

Occupancy is how many warps an SM holds against its maximum, and it is a means, not an end. The arithmetic from registers per thread to resident warps, the launch bound that caps registers and the spills that follow, the profiler's occupancy view, and the kernel where halving occupancy doubled speed.

Triton hands you a block of data and writes the thread-level code for you, which covers most of what a serving or training stack actually needs. The four things it does not give you, the performance you give up in each case with numbers, the reversal as Triton gains Hopper features, and how to decide in the room.

A sum reads every byte once and does one add per element, so the only question is whether you reach the bandwidth ceiling. The ceiling in milliseconds, four versions from an atomic per element to warp shuffles with sixteen loads in flight, and the bytes-in-flight math behind the last jump.