Caching strategies

Cache-aside, read-through, write-through, write-behind and write-around, plus the failure modes: stampede, penetration, avalanche and hot keys.

The problem

Adding a cache is the first answer to a slow read path, and the follow-ups are always the same: how is it filled, how is it kept fresh, and what happens when it fails or a hot key expires. Have one default and a list of failure modes ready.

How it works

  • Cache-aside (lazy loading) is the default: read cache, on a miss read the DB and fill the cache; on write update the DB then delete the key. Only requested data is cached and a cache outage degrades to DB reads.
  • Read-through moves the load logic into the cache layer. Write-through writes cache and DB synchronously (fresh reads, slower writes). Write-behind writes the cache and flushes to the DB asynchronously (fast, but data loss if the cache dies). Write-around writes straight to the DB and lets reads fill the cache.
  • Invalidation: prefer delete-on-write over update-on-write, since concurrent updates can write stale values last. Even so a race exists (reader loads old row, writer deletes, reader fills old value), so keep a TTL as a safety net; delayed double-delete or CDC-driven invalidation narrows it.
  • Stampede (hot key expires, thousands of misses hit the DB): single-flight or a per-key lock so one caller reloads; jittered TTLs; early refresh or serving stale while revalidating.
  • Penetration (requests for keys that do not exist bypass the cache): cache a short-lived null result, validate input, or put a Bloom filter of existing keys in front.
  • Avalanche (many keys expire together or the cache dies): randomise TTLs, keep replicas or a multi-level cache, rate-limit and degrade to protect the DB.
  • Hot keys: replicate the key under several suffixes, add a small in-process cache with a short TTL, or push to the CDN. Big values and big keys need splitting.
  • Where to cache: client/browser, CDN for static and cacheable responses, application (in-process or Redis/Memcached), and the DB's own buffer pool. Nearer to the user is faster but harder to invalidate.
  • Eviction when memory is full: LRU (recency) is the default, LFU (frequency) resists scans; both are approximate in Redis. Always state the consistency window you accept.

Go sketch

Go
import (
    "math/rand/v2"
    "sync"
    "time"
)

type entry struct {
    val     string
    expires time.Time
}

type call struct {
    wg  sync.WaitGroup
    val string
    err error
}

// Cache is cache-aside with per-key single-flight loading.
type Cache struct {
    mu       sync.Mutex
    items    map[string]entry
    inflight map[string]*call
    ttl      time.Duration
    load     func(key string) (string, error)
}

func NewCache(ttl time.Duration, load func(string) (string, error)) *Cache {
    return &Cache{
        items:    map[string]entry{},
        inflight: map[string]*call{},
        ttl:      ttl,
        load:     load,
    }
}

// Get returns a fresh entry, or lets exactly one caller reload it.
func (c *Cache) Get(key string) (string, error) {
    c.mu.Lock()
    if e, ok := c.items[key]; ok && time.Now().Before(e.expires) {
        c.mu.Unlock()
        return e.val, nil
    }
    if cl, ok := c.inflight[key]; ok {
        c.mu.Unlock()
        cl.wg.Wait()
        return cl.val, cl.err
    }
    cl := &call{}
    cl.wg.Add(1)
    c.inflight[key] = cl
    c.mu.Unlock()

    cl.val, cl.err = c.load(key)

    c.mu.Lock()
    delete(c.inflight, key)
    if cl.err == nil {
        ttl := c.ttl + rand.N(c.ttl/10+1)
        c.items[key] = entry{val: cl.val, expires: time.Now().Add(ttl)}
    }
    c.mu.Unlock()
    cl.wg.Done()
    return cl.val, cl.err
}

// Invalidate is called after the database write succeeds.
func (c *Cache) Invalidate(key string) {
    c.mu.Lock()
    defer c.mu.Unlock()
    delete(c.items, key)
}

In practice

  • Facebook Memcache — The "Scaling Memcache at Facebook" paper uses cache-aside with delete-on-write and introduces leases to curb stale sets and thundering herds.
  • DynamoDB DAX — A write-through, read-through cache in front of DynamoDB that is API-compatible, so applications need few code changes.
  • CDNs — Cache at the edge keyed by URL with TTL and explicit purge; many support stale-while-revalidate to hide origin latency.
  • Redis — Typically used as the cache-aside store with `maxmemory` and an LRU/LFU policy; TTLs with jitter are the standard defence against synchronized expiry.

Trade-offs

Pros

  • Cuts read latency and shields the database from repeated reads
  • Cache-aside is simple and degrades gracefully when the cache is down
  • Single-flight and jitter remove most stampede and avalanche risk cheaply

Cons

  • Stale reads are inherent; the window depends on invalidation and TTL
  • Write-behind can lose acknowledged writes
  • Adds an invalidation problem, a cold-start problem and an extra system to run
  • Hot keys and big values can overload one cache node

See also (other cheatsheets)

← All system design entries