What is Interview Forge?

Interview Forge helps you practice mock interviews, follow study plans, track career goals, review learning analytics, and prepare across 36 pages.

Design a Distributed Cache for Machine Learning Engineer

The problem for a Machine Learning Engineer

Functional: get/set/delete by key, TTL expiry, bounded memory per node.
Non-functional: sub-millisecond latency, high throughput, graceful degradation on node loss.
Scale: terabytes of cache, millions of keys, horizontal scaling.
Consistency: a cache is a fast, lossy view of the DB — eventual consistency is fine; never treat it as the source of truth.

Out of scope (covered in detail below)

Persistence, cross-region replication, multi-tier caching, and observability — deferred and discussed in the dedicated section.

As a Machine Learning Engineer, the system design round will probe training, serving, and ml systems. Be prepared to go deep on the areas a Machine Learning Engineer owns and to articulate trade-offs clearly.

Capacity estimation

Resource	Estimate	Notes
Cache size	terabytes	Hot working set
Keys	millions	Small values typically
Latency	<1 ms	In-memory
Hit rate target	90%+	For read-heavy workloads

Architecture overview

Consistent hash ring

            +--------+   client
             | Client |---+
             +--------+   |  hash(key) -> ring position
                         v
              (consistent hash ring)
         NodeA *-------* NodeB
               |       |
         NodeD *       * NodeC
               |
               v
         +----------------+
         | Cache Node (LRU+TTL) |
         +----------------+

A client routes a key to a node via consistent hashing; virtual nodes balance load.

Step-by-step request flow

The client hashes the key and routes to the node responsible for that ring position (with virtual nodes for balance).
On a hit, return the value; on a miss, read the DB and write back (read-through / cache-aside).
Writes invalidate or update the cache (write-through) or write to cache first (write-back) depending on the durability need.
TTL and eviction (LRU/LFU) bound memory; expired/evicted keys are reclaimed.

Unlock the full guide

Premium members get every DSA, system design, company and role interview guide — plus AI practice, mock interviews, and analytics.

Unlock with Premium