Designing a Rate Limiter (and the Ways I Got It Wrong First)

The one-line fix that wasn't

I've shipped a few side projects now that wrap an LLM API behind a route handler. Every one of them has the same shape: user hits an endpoint, endpoint calls a model, model call costs real money. And every one of them, at some point, got hit hard enough by a script or a bug in my own frontend retry logic that I opened a billing dashboard and didn't like what I saw.

The fix sounds trivial. "Just rate limit it." Ten requests a minute per user, done. I wrote that as an if statement once, felt very productive, and it lasted about a day before I realized it was wrong in a way that only shows up under real traffic.

This is the version of that problem I wish someone had walked me through first.

Attempt one: the counter that lies at the edge

The instinct is a counter and a timestamp.

// naive-limiter.ts
const counts = new Map<string, { count: number; windowStart: number }>();
 
function isAllowed(userId: string, limit = 10, windowMs = 60_000): boolean {
  const now = Date.now();
  const entry = counts.get(userId);
 
  if (!entry || now - entry.windowStart > windowMs) {
    counts.set(userId, { count: 1, windowStart: now });
    return true;
  }
 
  if (entry.count >= limit) return false;
 
  entry.count++;
  return true;
}

This is a fixed window counter, and it's not wrong exactly. It's wrong at the edges. Picture a limit of 10 requests per minute. A user sends 10 requests at 0:59, right as the window is about to reset. The window flips at 1:00, and the counter resets to zero. The same user now sends another 10 requests at 1:01.

Twenty requests in two seconds. The limit was "10 per minute." Nothing about that call was ever violated, because each burst landed in a different window. The rule was honest. The protection wasn't.

I found this the hard way when a retrying frontend happened to line up its retries right on a minute boundary during a spike, and my "rate limited" API let through what looked suspiciously like a burst twice the size of the limit. Fixed windows have a boundary problem baked into the math, not a bug I introduced.

Attempt two: sliding window, and the memory bill

The honest fix is to stop resetting at a clock boundary and instead ask "how many requests happened in the last 60 seconds, starting from right now." That's a sliding window, and the simplest version just keeps a log of timestamps.

// sliding-window-log.ts
const logs = new Map<string, number[]>();
 
function isAllowed(userId: string, limit = 10, windowMs = 60_000): boolean {
  const now = Date.now();
  const timestamps = logs.get(userId) ?? [];
 
  // drop anything outside the window before counting
  const recent = timestamps.filter((t) => now - t < windowMs);
 
  if (recent.length >= limit) {
    logs.set(userId, recent);
    return false;
  }
 
  recent.push(now);
  logs.set(userId, recent);
  return true;
}

This one is actually correct. No boundary bug, it always looks at a true trailing 60 seconds, so the double-burst trick doesn't work anymore.

It's also the version I'd never run at scale without changing something. Every allowed user is now an array that grows with every request and gets filtered on every check. A hundred users making occasional requests is nothing. A user hammering an endpoint right at their limit is now doing array filtering on every single call, and if this map lives in memory on one server, a hundred thousand active users is a hundred thousand growing arrays that nothing ever fully cleans up until they go idle.

Correct and cheap turned out to be two different properties, and I only had one of them.

Attempt three: the token bucket, which is what I'd actually ship

The version that shows up in almost every real rate limiter I've since read the source of is the token bucket. The idea: each user has a bucket that holds up to N tokens. Every request costs one token. Tokens refill at a steady rate, not all at once at a boundary.

// token-bucket.ts
type Bucket = { tokens: number; lastRefill: number };
 
const buckets = new Map<string, Bucket>();
 
function isAllowed(
  userId: string,
  capacity = 10,
  refillPerSecond = 10 / 60 // 10 tokens per 60s window
): boolean {
  const now = Date.now();
  const bucket = buckets.get(userId) ?? { tokens: capacity, lastRefill: now };
 
  const elapsedSeconds = (now - bucket.lastRefill) / 1000;
  const refill = elapsedSeconds * refillPerSecond;
 
  bucket.tokens = Math.min(capacity, bucket.tokens + refill);
  bucket.lastRefill = now;
 
  if (bucket.tokens < 1) {
    buckets.set(userId, bucket);
    return false;
  }
 
  bucket.tokens -= 1;
  buckets.set(userId, bucket);
  return true;
}

No boundary bug, because tokens trickle back continuously instead of resetting all at once. No unbounded array, because each user is one small object with two numbers, not a growing log. And it has a property the sliding window log doesn't: it naturally allows a short burst up to the bucket's capacity, then forces a steady trickle after that. For an API in front of a paid model call, that's usually exactly the shape I want, let someone send a quick handful of requests without being punished for normal usage, but never let them sustain a high rate.

The part that actually matters: where does the map live

Here's the thing none of the three snippets above actually solve, and it's the part that mattered most once I deployed anything for real. buckets is a Map sitting in one Node process's memory.

That's fine until you have more than one server instance, or until your platform restarts the process on deploy, or until you're on a serverless platform that might run your route handler on a fresh instance per request. In any of those cases, the counter resets constantly or splits across instances that don't know about each other, and a user with a limit of 10 can quietly get 10 per instance instead of 10 total.

I hit this specifically on a Vercel-deployed route that occasionally got cold-started under load. My in-memory limiter was, functionally, not limiting anything, because "in memory" kept meaning a different memory.

The fix is to move the counter somewhere shared. Redis is the standard answer, and the reason isn't just "it's fast," it's that the read-modify-write of checking and decrementing a token has to be atomic across every server hitting it at once, or you're back to the same race condition the naive fixed-window counter had, just distributed now instead of local.

// redis-token-bucket.ts (pseudocode shape, using ioredis + a Lua script)
const script = `
  local tokens_key = KEYS[1]
  local timestamp_key = KEYS[2]
  local capacity = tonumber(ARGV[1])
  local refill_rate = tonumber(ARGV[2])
  local now = tonumber(ARGV[3])
 
  local last_tokens = tonumber(redis.call("get", tokens_key)) or capacity
  local last_refreshed = tonumber(redis.call("get", timestamp_key)) or now
 
  local elapsed = math.max(0, now - last_refreshed)
  local filled = math.min(capacity, last_tokens + (elapsed * refill_rate))
 
  local allowed = filled >= 1
  local new_tokens = allowed and (filled - 1) or filled
 
  redis.call("set", tokens_key, new_tokens)
  redis.call("set", timestamp_key, now)
 
  return allowed and 1 or 0
`;
 
const allowed = await redis.eval(script, 2, `tokens:${userId}`, `ts:${userId}`, 10, 10 / 60, Date.now());

The Lua script is the important part, not the Redis part. Running this logic as a script means the whole read-check-write happens as one atomic operation on the Redis server, instead of three separate round trips from my app where another request could sneak in between the read and the write. Two servers can hit this at the exact same millisecond for the same user and still only one token gets spent, because Redis executes the script as a single indivisible step. That's the guarantee an in-memory Map can give you for free on one process, and can't give you at all across two.

What I'd actually reach for now

If it's a single server, low stakes, and I want something today: the token bucket in memory, the third snippet above. It's correct for one process and it's twenty lines.

If it's anything deployed with more than one instance, or anything serverless, or anything protecting a paid API call where being wrong costs actual money: Redis with the Lua script, or honestly, a library that's already done this correctly, because I've now written the atomic version by hand exactly once and I don't plan on debugging my own Lua at 1 AM a second time.

The lesson underneath all three attempts is the same one I keep relearning on every backend problem that looks simple from a distance: the hard part was never "count requests and reject after N." That's the easy 80%. The hard part is deciding what "count" and "now" actually mean when two requests might be evaluating them at the exact same instant, possibly on two different machines that don't share memory. Get that part wrong and the limiter isn't broken, it just quietly isn't doing the one thing you built it for.