TL;DR - Modulo hashing reshuffles almost every key when the cluster size changes. Consistent hashing puts nodes and keys on a ring so a membership change moves about 1/N of the keys. Add virtual nodes for even load, walk the ring for replicas, and remember it balances keys; traffic needs its own fix.

30-second elevator pitch: “I map keys and servers onto the same hash ring. A key belongs to the first server clockwise from it. When a server joins or dies, one slice of the ring changes owner. Each server gets many virtual positions, so load stays even and a failure spreads evenly across all survivors.”

Why Modulo Hashing Reshuffles on Resize

You run a cache tier with 4 servers and route with server = hash(key) mod 4. It is fast, stateless, and even. Then you add a fifth server.

Under hash(key) mod 5, a key stays put exactly when its hash leaves the same remainder under both divisors. For uniform hashes that happens with probability 1/(N+1), so going from 4 to 5 servers keeps 20 percent of keys and moves 80 percent. In general, adding one node to N moves N/(N+1) of all keys.

For a cache, that is a miss storm that dumps reads onto the database. For a sharded database, it means copying most of the dataset to add capacity.

What Is Consistent Hashing?

Consistent hashing is a placement scheme where adding or removing one node relocates just the keys that node gains or loses, roughly 1/N of the total.

The Ring

Take a hash function with a large output range, such as 0 to 2^32 - 1, and wrap it into a circle. Hash each server’s identifier to get its position. Hash each key with the same function to get the key’s position.

Lookup: Walk Clockwise

A key belongs to the first server clockwise from its position, so each server owns the arc between its predecessor and itself. In practice this is a sorted array of server positions and a binary search for the first position at or after the key’s hash, wrapping to the start.

What Moves on a Join or Leave

When server D joins between A and B, it takes the keys on the arc from A to D, which used to belong to B. Every other arc keeps its owner. If D later fails, that arc falls back to B.

With N evenly loaded nodes, a join moves about 1/(N+1) of the keys and a removal about 1/N. Going from 4 to 5 servers now moves roughly 20 percent, down from 80.

Virtual Nodes

Why Each Server Needs More Than One Point

Random positions split a circle unevenly. With N random points, the expected largest arc is (1 + 1/2 + ... + 1/N) / N of the ring. For N = 10 that sum is about 2.93, so the unluckiest server owns close to 3x its fair share. Failure makes it worse: a dead node’s whole arc lands on one successor, which can double that server’s load at once.

How Virtual Nodes Fix It

A virtual node is one of many positions a physical server occupies on the ring. In place of hashing “server-A” once, you hash “server-A#0” through “server-A#199”. The physical server owns the sum of its small arcs.

The Math to Quote

With N servers and V virtual nodes each, the expected share per server is still 1/N; V changes just the spread, and the average stays put. A server’s share is the sum of V small, roughly independent arcs, so its deviation from 1/N shrinks roughly as 1/sqrt(V). At V = 100 a typical server is within about 10 percent of average; at V = 1,000, about 3 percent.

Failures spread too. A dead server’s virtual nodes are scattered, so its keys flow to many successors, and each of the N - 1 survivors grows from 1/N to about 1/(N-1) of the keys.

The cost is small. Lookup is a binary search over N x V positions: 50 servers with 200 virtual nodes each is 10,000 entries, about 14 comparisons.

Weighting Uneven Hardware

Give a server with twice the memory twice the virtual nodes and it gets about twice the keys. Drain a node gradually by removing its virtual nodes in batches.

Replication on the Ring

To store each key on R servers, find the owner and keep walking clockwise until you have R distinct physical servers. Skip positions belonging to a server already chosen, since adjacent virtual nodes often map to the same machine. A zone-aware walk also skips zones already used, so one zone outage costs at most one copy.

This ordered candidate list is the preference list. Quorum rules, such as write to 2 of 3 and read from 2 of 3, sit on top of it.

Operating It

Who Holds the Ring?

Every router must agree on membership, or two clients will send one key to different servers. A coordination service such as ZooKeeper or etcd can hold the list, nodes can gossip it, or a proxy layer can own routing.

Moving the Data

The ring says who owns a key; getting the data there is a separate job. A cache can let moved keys miss and refill. A database must stream the arc to the new owner first, or read from the previous owner until the copy completes.

Hot Keys Still Hurt

Consistent hashing balances key counts; request rates need a separate fix. One celebrity key still maps to one owner. Cache it in-process at the app tier, or split it into suffixed copies (key#1 to key#8) spread around the ring and read a random one.

Alternatives Worth Naming

• Fixed hash slots - Hash keys into a fixed slot count, such as 16,384 in Redis Cluster, and assign slots to nodes. Pros: explicit, observable rebalancing. Cons: slot count caps how finely you can split.

• Rendezvous hashing - Score every node with hash(key, node) and pick the highest. Pros: minimal movement, just a node list to maintain. Cons: O(N) work per lookup.

• Jump consistent hash - Computes a bucket number purely from the key and the bucket count. Pros: even balance, zero memory. Cons: buckets are numbered, so you add or remove at the end.

• Bounded loads - Cap each node at (1 + epsilon) times the average and push overflow clockwise. Pros: limits skew. Cons: extra bookkeeping.

When to Use in Interviews

Reach for it in a distributed cache, a key-value store, sticky load balancing, or a chat service that pins each room to one server. Skip it when the node count is small and fixed, or a managed database already partitions for you.

Summary

• Ring over modulo - About 1/N of keys move per change, compared with N/(N+1).

• Virtual nodes - Share stays 1/N, spread shrinks as 1/sqrt(V).

• Limits - Hot keys need their own fix; membership must be agreed on.

Practice consistent hashing by designing a distributed cache or key-value store and tracing exactly which keys move when a node joins and when one fails.

Interview Q&A

• Why the ring over hash mod N? - Changing N remaps N/(N+1) of keys, emptying a cache or forcing a near-full reshuffle. The ring moves about 1/N.

• How many virtual nodes? - A few hundred per server. Spread scales as 1/sqrt(V), so 100 to 200 gives roughly 7 to 10 percent deviation at trivial lookup cost.

• What happens when a node fails? - Its arcs pass clockwise, spread across survivors by virtual nodes; replicas already hold the data.

• Does it solve hot keys? - It balances keys, so hot keys need their own fix: an in-process cache or split copies.

• How do clients learn about changes? - From a coordination service, gossip, or a routing proxy. Version the ring so stale clients can detect it.

• Ring or fixed slots? - Slots make rebalancing explicit; a ring with virtual nodes lets you skip choosing a slot count upfront. Either works if you state the trade-off.

Read More Articles

TL;DR - GitGuardian counted about 29 million new secrets on public GitHub in 2025, up 34%, with AI-service credentials up 81% and AI-assisted commits leaking at roughly double the baseline. Answer it with design: fewer long-lived secrets, scanning at every boundary, and secrets swapped out before code reaches a model.

30-second version: “AI tools raise the volume of code and the number of places secrets travel: commits, prompts, MCP configs, logs. Treat every one of those as an egress path, give each one a control, and make credentials short-lived so any slip expires on its own.”

The Numbers

From GitGuardian’s State of Secrets Sprawl 2026 (published March 17, 2026, covering 2025):

• About 29 million newly leaked secrets on public GitHub, up 34% year over year, the largest single-year jump the report has recorded.

• 1,275,105 AI-service secrets leaked, up 81%.

• About 3.2% of commits made with Claude Code’s help leaked a secret, roughly twice the baseline. The report itself cautions that this may reflect human mistakes as much as AI behavior.

• 24,008 unique secrets sat in the MCP configuration files it studied.

• 64% of secrets that were valid in 2022 were still valid in 2026.

• Internal repositories were about 6x more likely than public ones to contain hardcoded secrets.

The last two matter most for design. Leaks stay live until someone revokes them, and private repos need the same scanning as public ones.

How AI Tools Reshape Secret Handling

Before coding agents, a secret leaked in roughly one way: someone committed it. Now there are four or more paths.

• Commits - more code, written faster, with lighter review.

• Prompts - an agent reads .env, a config or shell output and sends it to a model provider.

• Tool configs - MCP servers configured with long-lived tokens in plain JSON files.

• Logs - gateways and observability tools that store request and response bodies (see the LiteLLM 1.107.0 compromise on October 21, 2026).

Each path needs its own control. One pre-commit hook covers the first path.

A Layered Design

Layer 1: Fewer secrets to leak

• Short-lived credentials. Use workload identity and OIDC for CI and cloud access, and session tokens for agents. A token that expires in an hour turns a leak into a short incident report.

• References over values. Configs point to a secret manager entry. MCP configs read from the environment or a keychain.

Layer 2: Scan at every boundary

• Pre-commit and pre-push hooks on developer machines.

• Server-side push protection on the repository host, because hooks can be skipped.

• Log pipelines scanned before persistence.

Layer 3: Keep secrets out of prompts

This is the newest boundary and the one with the most room to grow. Options, in increasing strength:

• Ignore files that keep the agent out of .env and key directories. They are easy; pair them with a scanner for secrets inside source files and command output.

• A local egress proxy that inspects each outbound prompt and swaps detected secrets for stand-in tokens before the request leaves the machine. The provider and every log after it see the stand-in.

The design rule for that proxy: code identifiers can be reversed back into the answer byte-exact, but secret stand-ins must stay one-way. Reversing a secret token in model output would let a prompt injection exfiltrate it through a crafted link.

Layer 4: Plan for a slip and contain it

• Alert on first use of a credential from a new network or ASN.

• Keep a rotation runbook that works in minutes, and rehearse it.

How to Use This in an Interview

When a design includes an LLM or an agent, add one box and one sentence: “Prompts pass through a local egress proxy that tokenizes secrets; the gateway logs metadata by default; agent credentials are session-scoped.” It is cheap to say and shows you treat the model call as egress.

Interview Q&A

• Why is a 2x leak rate a design question more than a training question? - Volume is rising fast enough that a steady human error rate produces more leaks every year. Controls have to scale with commits, independent of headcount.

• Why scan private repos? - In the same report they were about 6x more likely to contain hardcoded secrets, and anyone with read access, or a stolen token, can read them.

• What is the weak point of pre-commit scanning? - It is optional on the client. Pair it with server-side push protection.

• Why one-way for secrets but reversible for identifiers? - Identifiers are needed to read the answer. Secrets stay as stand-ins in model output, because reversing them opens an exfiltration path.

Sources

• GitGuardian, State of Secrets Sprawl 2026: https://www.gitguardian.com/state-of-secrets-sprawl-report-2026

• GitGuardian press release (March 17, 2026): https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026-pr/

• Cloud Security Alliance, LiteLLM 1.107.0 research note: https://labs.cloudsecurityalliance.org/research/csa-research-note-litellm-1107-gateway-log-token-compromise/

Disclosure: I build Pretense, the local egress proxy described in layer 3. It swaps API keys, tokens and code identifiers for stand-ins before a prompt leaves the laptop, restores identifiers byte-exact in the answer, and keeps secrets as stand-ins. By default it flags emails and phone numbers for review.

Read More Articles

TL;DR - On October 21, 2026, attackers published a poisoned LiteLLM 1.107.0 using GitHub tokens that LiteLLM believes leaked through its own AI gateway logs. The lesson is architectural: any box that records prompts and responses records whatever secrets ride inside them. Swap secrets before they leave the laptop, log metadata by default, and keep agent tokens short-lived.

30-second version: “An AI gateway sees every prompt and every answer. If it logs bodies, its log store holds every credential an engineer or agent ever pasted, and it needs the same protection as a vault. Better: keep real secrets on the laptop so the gateway sees stand-ins every time.”

What Happened

LiteLLM is a popular open-source gateway that sits between applications and model providers. According to the Cloud Security Alliance research note published October 23:

• The release: malicious litellm==1.107.0 builds reached PyPI, Docker Hub and GHCR on October 21. The PyPI window was roughly 11:39 to 13:23 UTC. Wiz counted about 49,000 downloads.

• The access: the attacker used stolen GitHub CLI OAuth tokens belonging to three LiteLLM engineers. One token was used to override branch protection, one to approve the release, and one to delete workflow records afterward.

• The suspected leak path: an internet-exposed internal LiteLLM deployment had been logging request and response bodies from coding-agent sessions. LiteLLM reports cases where GitHub CLI tokens ended up inside those bodies, and so inside the logs. This is LiteLLM’s working explanation; the root-cause review is still open.

• The payload: files that run on every Python start and collect cloud, Kubernetes, SSH, registry and AI-provider credentials, then send them to a lookalike domain.

The Design Flaw: Logging Bodies at a Chokepoint

What is an AI gateway? A proxy in front of model providers that handles routing, keys, quotas, cost tracking and, often, logging. It is the AI version of an API gateway, and it inherits the same habit: log everything for debugging.

That habit is where the exposure starts. A coding agent’s traffic is richer than ordinary API traffic. It carries:

• File contents, including .env files and config the agent opened.

• Shell output, including tokens a command printed.

• Tool results, including credentials a tool fetched.

Log those bodies and the log store becomes the richest credential collection in the company: searchable, replicated to backups, and it deserves the same guard as the secret manager.

The CSA note’s own recommendation states it plainly: treat gateway logs like a secret store and make content logging a reviewed exception.

Four Fixes, Cheapest First

1. Log metadata by default

Record model, token counts, latency, user and cost. Turn body logging off by default. When someone needs bodies to debug, turn it on for one user for one hour, and then turn it off again.

2. Rotate what may already be there

If any gateway, proxy or observability tool stored agent request or response bodies, search those stores and their backups for token prefixes (ghp_, gho_, sk-, AKIA) and revoke every hit. A 2026 GitGuardian study found 64% of secrets that were valid in 2022 were still valid in 2026, so plan to revoke each one yourself.

3. Shrink token lifetime

An OAuth token that lives for months turns one log line into months of access. Give agent sessions short-lived, narrowly scoped tokens, and move releases to trusted publishing so the CI pipeline is the one path that ships a package.

4. Swap secrets before they leave the machine

The strongest control moves the fix upstream of every log: the gateway stores exactly what it receives, so send it stand-ins. Run a local proxy on the developer laptop that replaces detected secrets with stand-in tokens before the request leaves. The gateway, the provider and every log downstream then hold the stand-in.

Two details keep everything working:

• Reversible where it is safe. Code identifiers can be swapped back byte-exact in the answer, so the developer reads their own code.

• One-way for secrets. Secret stand-ins stay as stand-ins in model output. If a prompt injection gets the model to echo a stand-in into a link, reversing it would hand the real credential to the attacker.

Where This Shows Up in a System Design Interview

Any design with an LLM in it now has an egress question. Draw the gateway, then say three things:

• Logs hold metadata by default.

• Secrets are tokenized client-side, before the gateway.

• Agent credentials are short-lived and scoped per session.

That is one sentence each and it signals you have read the 2026 incident reports.

Interview Q&A

• Why swap on the client, ahead of the gateway? - Scrubbing inside the gateway happens after it has received the secrets, which can still surface in memory dumps, crash reports or a stray log path. Swapping on the client means stand-ins are all that arrive.

• Is logging prompts ever justified? - Yes, for evaluation and abuse review. Store them encrypted, with short retention and access logging, and scan for credentials before persisting.

• What is the blast radius of one leaked GitHub OAuth token? - Everything that token’s user can do, which in this incident included approving a production release.

• How would you detect this kind of compromise? - Alert on releases outside the normal pipeline, on branch-protection overrides, and on deleted workflow runs.

Sources

• Cloud Security Alliance, research note on the LiteLLM 1.107.0 compromise (October 23, 2026): https://labs.cloudsecurityalliance.org/research/csa-research-note-litellm-1107-gateway-log-token-compromise/

• GitGuardian, State of Secrets Sprawl 2026: https://www.gitguardian.com/state-of-secrets-sprawl-report-2026

Disclosure: I build Pretense, a local proxy that does fix 4: it swaps API keys, tokens (including GitHub gho_/ghp_ tokens) and code identifiers for stand-ins before a request leaves the laptop. Identifiers come back byte-exact; secrets stay as stand-ins in the reply. It does its job when it runs in front of the gateway, and it works alongside fixes 1 to 3.

Read More Articles

30-second elevator pitch: "Real-time updates require efficient push channels. I choose based on latency needs: polling for 2-5 second updates, long polling or SSE for near real-time, WebSockets when I need bidirectional low latency. The second hop is Pub/Sub or consistent hashing to fan out updates from the source to servers."

The Problem

Consider a colla…

Read more

30-second elevator pitch: "Multi-step processes fail halfway - payment charged but inventory not reserved. I use workflow engines like Temporal or AWS Step Functions so each step is durable, failures trigger retries or compensation, and the system picks up exactly where it left off after a crash."

What You Will Learn

Approaches

- Single-server orchestratio…

Read more

30-second elevator pitch: "Contention happens when two users act on the same data at once, like two people buying the last concert ticket. I fix it by making operations atomic, using locks when conflicts are common, or optimistic concurrency when they are rare. For distributed systems, I use sagas or two-phase commit. I always exhaust single-database so…

Read more

30-second elevator pitch: "Caching stores frequently accessed data in memory so reads skip the database. I start with cache-aside using Redis as the default, add CDN for global static assets, and choose invalidation strategy based on freshness requirements. Cache-aside handles most scenarios, and I layer additional strategies only when the problem deman…

Read more

30-second elevator pitch: "For any operation that takes more than a few seconds, I return a job ID immediately and process asynchronously. A message queue decouples submission from processing. Workers pull jobs, execute them, and update status. The client polls for completion or gets a webhook callback. I add exponential backoff for retries and a dead l…

Read more

30-second elevator pitch: "Large files bypass my application server entirely. Clients get a presigned URL and upload directly to S3 in chunks. An S3 event triggers async processing like thumbnails or transcoding. Downloads go through a CDN. My server only handles metadata and authorization."

What You Will Learn

Upload Patterns

- Presigned URL upload (bypas…

Read more

30-second elevator pitch: "Write scaling is harder than read scaling because every write must persist to disk. I exhaust vertical scaling and database tuning first. Then I shard by partition key to spread load linearly. For bursty traffic I add queues and load shedding. For extreme volume I batch writes and use hierarchical aggregation. Each tier builds…

Read more

30-second elevator pitch: "Read traffic grows exponentially faster than writes. I handle this in three tiers: first optimize the existing database with indexes and denormalization, then scale horizontally with read replicas, and finally add caching layers. Each tier builds on the previous one, and I always start with the simplest solution."

The Problem

Co…

Read more