What is Interview Forge?

Interview Forge helps you practice mock interviews, follow study plans, track career goals, review learning analytics, and prepare across 36 pages.

Design a Distributed Rate Limiter at NVIDIA — System Design Interview Guide

The problem at NVIDIA

Functional: limit requests per user/API key to N per second (or minute); return 429 with Retry-After when exceeded.
Non-functional: low latency (<1ms added), accurate under concurrency, works across multiple nodes.
Scale: millions of keys, each with its own limit; limits must hold globally, not just per node.
Consistency: a slight over-limit under a partition is acceptable; a large over-limit is not.

Out of scope (covered in detail below)

Per-tier quotas, billing integration, adaptive/ML-based limiting, and multi-tenant isolation — deferred and discussed in the dedicated section.

At NVIDIA, this problem is framed around systems, performance, and parallel thinking. Expect NVIDIA-specific scale numbers and a focus on production readiness and the company's engineering bar.

Capacity estimation

Resource	Estimate	Notes
Requests/sec	millions	Per-key counters in memory
Keys	millions	Each a small counter state
Added latency	<1 ms	Must be in-memory or local
Memory	~tens of GB	Counter state per key

Architecture overview

Edge + shared store

+--------+   +-------------+   coarse    +-----------+   fine     +---------+
| Client |-->| API Gateway  |----------->| Rate Limit|----------->| Redis   |
|        |   | (edge limit) |            | Service   |            | (INCR)  |
+--------+   +-------------+             +-----------+            +---------+
                                                     |
                                                     v
                                              +-----------+
                                              |  Service  |
                                              +-----------+

Coarse limits at the edge, fine-grained limits backed by a shared counter store.

Algorithms

Pick the algorithm by what you need: bursts, smoothness, or downstream protection.

Token bucket: refill tokens at a rate; allow a request if a token is available. Allows bursts up to the bucket size — the default for APIs.
Fixed window: count requests in the current time window; simple but has edge bursts at window boundaries.
Sliding window / sliding window log: smooths the boundary burst; more memory but more accurate.
Leaky bucket: shapes traffic to a steady outflow rate — good for protecting a downstream service.

Sliding window via counters

A memory-efficient sliding window approximates by keeping a count for the current and previous fixed window and interpolating: count = cur + prev * (1 - elapsed/window). O(1) memory, no per-request log.

Unlock the full guide

Premium members get every DSA, system design, company and role interview guide — plus AI practice, mock interviews, and analytics.

Unlock with Premium