TL;DR: Two differences from a textbook LRU decide the design. Entries are not comparable in size: a 7B model is 14 GB and a 70B is 140 GB, so a cache holding "the last four models" holds anywhere from 56 GB to 560 GB and cannot be reasoned about. Track a byte budget instead, and evict in a loop until the new entry fits rather than evicting exactly one. And an entry can be in use: evicting the weights a replica is currently serving from is not a slow path, it is a crash, so entries are pinnable and pinned entries are never selected as victims. Two failures must be distinguished and both must raise rather than silently succeeding. An entry larger than the whole cache can never be admitted, which is a configuration error. An entry that would fit but cannot because too much is pinned is a transient condition with a different fix, and returning a cache miss for either one hides a real problem.
How to approach it
Name the two differences before writing, since they are the reason the question exists. Implement with an ordered map for recency and an explicit byte counter. Show the eviction loop and the pin check. Then the two error cases, and be explicit that both raise. Close with what the cache is actually protecting, which is the cold-start time.
A strong answer
A typical situation: a serverless GPU platform caches weights on node NVMe with a four-entry LRU. It works during testing with 7B models. In production a request for a 70B model evicts three small models, and a subsequent request for a 7B model that was resident a moment ago pays a full cold start, so the cache's hit rate collapses precisely when a large model is in rotation.
The implementation:
from collections import OrderedDict
class WeightCache:
"""Size-aware LRU with pinning. capacity_bytes is a budget, not a count."""
def __init__(self, capacity_bytes):
self.capacity = capacity_bytes
self._items = OrderedDict() # key -> (size, value); order is recency
self._pinned = set()
self._used = 0
self.evictions = 0
def get(self, key):
if key not in self._items:
return None
self._items.move_to_end(key) # most recently used
return self._items[key][1]
def pin(self, key):
self._pinned.add(key) # in use by a replica: never evict
def unpin(self, key):
self._pinned.discard(key)
def put(self, key, value, size):
if size > self.capacity:
raise ValueError(f"{key!r} is {size} B, larger than the {self.capacity} B cache")
if key in self._items: # replacing: release the old size first
self._used -= self._items[key][0]
del self._items[key]
while self._used + size > self.capacity:
victim = self._evict_candidate()
if victim is None:
raise MemoryError(f"cannot fit {key!r}: {self._pinned} pinned")
self._used -= self._items[victim][0]
del self._items[victim]
self.evictions += 1
self._items[key] = (size, value)
self._used += size
def _evict_candidate(self):
for k in self._items: # OrderedDict iterates oldest first
if k not in self._pinned:
return k
return None # everything resident is pinned
Running it with a 300 GB budget:
put llama-70b (140 GB), mixtral (90 GB) -> used 230 GB of 300 GB
get llama-70b -> touched, so mixtral is now oldest
put qwen-72b (145 GB) -> evicted mixtral
resident: ['llama-70b', 'qwen-72b'], used 285 GB, evictions 1
pin llama-70b, then put a 200 GB model -> MemoryError
put a 400 GB model -> ValueError: larger than the 300 GB cache
The two differences, stated as the design:
1. size, not count
a count-based LRU holding 4 entries holds 56 GB of 7B models or 560 GB of 70B models
the budget is what the node actually has, so the cache must account in bytes
consequence: eviction is a loop, not a single removal, because admitting one large entry
may require evicting several small ones
consequence: an entry larger than the whole budget is impossible rather than expensive
2. pinning
a replica serving requests holds its weights mapped; evicting them under it is not a slow
path, it is a use-after-free or a corrupt read
so the cache must know which entries are in use, and the caller must pin and unpin around
the lifetime of a replica rather than around a request
the eviction candidate scan skips pinned entries, which means the LRU order is advisory:
the oldest entry may be pinned and the second oldest evicted instead
Containers, Images and GPU Cold Starts covers what a miss costs, which is the reason this cache exists, and Inference Autoscaling and Cold Starts covers the scaling loop that is waiting on it.
The two error cases, and why both raise:
ValueError: the entry is larger than the whole cache
a permanent condition. No sequence of evictions can admit it
raising surfaces a configuration error: either the budget is too small for the models this
node is expected to serve, or the node should not be assigned this model at all
returning a miss instead means every request for that model pays a full load and the cache
silently never helps, which can persist for months
MemoryError: it would fit, but too much is pinned
a transient condition with a different fix: wait for a replica to drain, or refuse to place
another replica on this node
raising lets the caller distinguish "try again shortly" from "never going to work"
sanity: both cases return no value, and a naive implementation returns a cache miss for both.
A miss is a lie: it says the entry is not present when the truth is that it cannot be
admitted, and the caller then retries forever
What the cache is protecting:
a miss on a 140 GB model
from node NVMe at 6 GB/s = 23 s
from object storage at 1 GB/s = 140 s
a hit costs the memory map, effectively free
so the cache's value is the difference, and its hit rate translates directly into cold-start
latency, which for a serverless platform is the product
sizing: budget = the working set of models this node serves, plus headroom for one more so an
arrival does not immediately evict something in use
a node serving three models of 140, 90 and 45 GB needs 275 GB plus headroom, so 400 GB
rather than the 300 GB in the example, which is why the example evicts at all
The reversal condition: this cache is worth having only when models are reused on the node. A platform that routes each request to any node with no affinity has a hit rate near zero regardless of the cache's quality, because a node rarely sees the same model twice before eviction. The fix there is in the router rather than the cache: route by model so requests for a given model prefer nodes that already hold it, which converts a useless cache into a useful one. Building a sophisticated cache under a random router is optimizing the wrong component.
What interviewers probe next
- "Why not evict the largest entry instead of the oldest?" That optimizes for fitting the new entry and against the hit rate, since a large model may be the most frequently requested. Size-aware policies exist and need request-frequency data to beat plain recency.
- "What if two replicas share one model?" Reference-count the pin rather than using a set, so the entry is unpinned only when the last replica releases it.
- "How do you handle a partially written entry?" Write to a temporary name and rename atomically, so a crash mid-write leaves no half-entry that a later get would return.
- "Is an OrderedDict the right structure?" It gives O(1) get, put and move-to-end, which is what an LRU needs. The eviction scan is O(pinned) in the worst case, which is small.
Common mistakes
- A count-based cache, so the budget varies by a factor of ten depending on which models are resident.
- No pinning, so the cache can evict weights a replica is serving from.
- Returning a miss when an entry cannot be admitted, which hides both a configuration error and a transient condition.
- Evicting exactly one entry, which fails when the new entry needs the space of several.
Key takeaways
- Account in bytes, not entries: four entries is 56 GB or 560 GB depending on the models.
- Evict in a loop until the entry fits, and skip pinned entries when choosing a victim.
- Two distinct failures, both raising: larger than the cache is permanent, blocked by pins is transient.
- A miss costs 23 s from local NVMe or 140 s from object storage, which is what the hit rate is worth.