Words are a funny thing. On the one hand we must name things in code (circuit breaker, ingress, throughput), so that it is easier to communicate ideas. On the other hand, once some people use some terms, these same terms become shortcuts for figuring out how competent your conversational partner (=interlocutor) is.

Naming a thing is a shortcut for “I have built the thing”.

It does work the other way too: if you don’t know the name of the thing, it’s surprisingly hard to talk about it, even when you’ve built it a dozen times.

So here are the names, for things you’ve probably already built or fought with, so you can name them and collect your points.

In this post

Load

Load is how much work hits the system. Sometimes we have exact numbers, but most often we have just rough estimates, because past performance does not predict future returns.

Some mental math that helps:

  • an hour has \(3600\ \text{sec}\), a day has \(86{,}400\ \text{sec}\), but we can round it to \(100{,}000\ \text{sec}\)
  • \(1\ \text{million/day}\) ≈ \(12\text{/second}\), or we can round it to \(10\text{/second}\)
  • peak traffic is usually a few times the average
Word Meaning
Requests per second The basic load number: number of inbound requests.
RPS or Q(ueries)PS for queries.
Throughput How much work the system actually gets done per second. Load is what arrives, throughput is what you can handle. Not the same as latency. Latency is how long one request takes, throughput is how many get done per second. Batching can raise throughput and make latency worse at the same time.
Bandwidth How many bytes per second a connection or machine can move. Requests per second counts requests, bandwidth counts their size: \(1{,}000\ \text{requests/second}\) of 1 MB each is 8 Gbit/s.
Ingress / egress Ingress is traffic coming into a system, egress is traffic going out. In system design, they mostly come up because of cost. Not the same as inbound requests. Ingress counts every byte coming in, including the responses to your own outgoing calls: a worker calling 3rd-party APIs has lots of ingress and no inbound requests. And in Kubernetes, an Ingress is something else again: the component that routes outside HTTP requests into the cluster. Traffic between zones and between regions is billed too, not only traffic to the internet.
Latency How long one request takes, usually given as percentiles: p50 is a typical request, p99 is the slowest 1%. Averages. An average of 100 ms can hide a p99 of 5 seconds, and the slowest requests often belong to your biggest customers.
Concurrency How many requests are in flight at the same moment: requests per second × seconds each one takes (Little’s law). 100 requests per second that take 0.5 s each means 50 at any moment, and that’s how many workers or connections you need. Slow requests eat capacity. If the same requests take 5 s instead of 0.5 s, you suddenly need 500 workers, with no extra traffic at all. More in Deep-dive #1: Why a Slow API Is Worse Than a Dead One
Utilization How busy a resource is. Latency doesn’t grow evenly with it: in the simplest queueing model, at 50% busy a request waits in line about as long as it takes to serve, at 90% busy about 9 times as long.
CPU-bound vs I/O-bound Whether work mostly waits for the CPU (parsing, rendering, math) or for I/O (the network, the disk, another API). It decides how many workers a machine can run: CPU-bound work, about one per core. I/O-bound work, hundreds at once, because they’re mostly just waiting (async, green threads). Threads vs processes: threads share one process’s memory, so they’re cheap and pass data around easily, but when the process dies (out of memory, a hard crash), all its threads die with it. Processes are fully separate: heavier, but isolated. Whether threads use several cores depends on the language: in Java they do, in Node and standard Python your code runs on one core per process by default, so you scale with more processes. Many runtimes also have lightweight threads, thousands of them on a few real ones: greenlets (Python) take turns on one core, goroutines (Go) and virtual threads (Java) spread over all of them.
Blocking vs non-blocking (I/O) (async) Blocking: the worker waits, doing nothing, until the answer comes back. Non-blocking (async, an event loop): while one request waits for the network, the same worker serves others. That’s how one process can hold thousands of slow connections. One blocking call inside async code (a sync database driver, a CPU-heavy loop) freezes every request on that worker at once, health checks included. More in Deep-dive #1: Why a Slow API Is Worse Than a Dead One
Read-heavy / write-heavy Which of the two happens more. Dashboards are read-heavy, metric collection is write-heavy.
It decides where caches go and what the database has to be good at.
Back-of-the-envelope An estimate to get the order of magnitude: \(\text{users} \times \text{actions} \times \text{size}\). It decides whether you need anything fancy at all: 35 requests per second is one small server, 1 TB a day is a different story. The goal is to be within 10× of the truth, not within 10%.
DAU, MAU Daily and monthly active users. The usual starting point for an estimate: \(\text{DAU} \times \text{actions / user / day}\) = \(\text{requests / day}\). Not the same as concurrent users. 1 million daily users who each spend 30 minutes a day in the app are about 20,000 online at any moment (\(1\text{M} \times 0.5\ \text{h} / 24\ \text{h}\)).

Building blocks

These are boxes on architecture diagrams.

Word Meaning
CDN Servers around the world, close to your users. The classic job is serving copies of static files (images, JS, CSS). Today they also cache pages and API responses, speed up requests that can’t be cached, block attacks before they reach you, and run small bits of code at the edge. Changing a file behind the same URL. Put a hash of the content in the file name instead (app.3f9a2c.js).
Load balancer Spreads incoming requests across several identical servers. One you run yourself is a new single point of failure. Managed ones can span several zones, but not all of them do by default, and it only helps if your servers are spread across zones too.
Health check The load balancer asking every server “are you OK?” every few seconds, and only sending traffic to the ones that answer. Liveness asks “is the process stuck? Then restart it.” Readiness asks “should it get traffic right now?” Mixing them up restarts servers that didn’t need a restart. Check only the server itself! If every server checks the shared database, one database hiccup takes every server out at once.
Reverse proxy A server that sits in front of your servers and takes requests on their behalf: it ends HTTPS, serves static files, and forwards the rest to the right app server. nginx is the classic one, and load balancers and API gateways are reverse proxies with a specific job. Not the same as a forward proxy. A reverse proxy is set up by the side being called, and hides its servers. A forward proxy is set up by the caller, and hides the caller: a corporate proxy, a VPN, or a service that calls APIs on your behalf.
API gateway A single front door for all your APIs. It checks who’s calling (API keys, tokens), applies rate limits, and routes each request to the right service, so every service doesn’t have to do that itself. Not the same as a load balancer. A load balancer spreads requests over identical copies of one service. A gateway decides which service a request goes to, and whether it’s allowed in at all. In practice, one product often does both.
OAuth A standard way to let an app act on your behalf in another service, without giving it your password. You click “Connect GitHub”, GitHub asks you to approve, and the app gets a token that can do only what you allowed, and that you can take back. OAuth is about access (what the app may do), not login (who you are). “Sign in with Google” uses OpenID Connect, a layer on top of OAuth. Tokens expire and get revoked. Code that calls APIs with OAuth tokens needs a refresh step, and a plan for when the user takes access back.
Stateless server Keeps nothing in memory between requests, so you can run 3 or 30 copies and kill any of them. Moving state out of the servers (into a database or Redis) is what makes them easy to scale. Hidden state: sessions kept in memory, uploads saved to the server’s own disk, a cache that lives inside the process.
Container Your app packaged together with everything it needs to run (the right Python version, the libraries, the config), so it runs the same on your laptop and in production. Docker builds them. Kubernetes runs lots of them across many machines: it starts them, restarts the ones that die, and adds more when the load goes up. Not a virtual machine. A VM brings its own whole operating system, a container shares the machine’s, which is why it starts in seconds instead of minutes.
Queue, message broker A waiting line between whatever creates work and whatever does it (RabbitMQ, SQS, the broker behind Celery). It absorbs spikes, lets you add workers, and holds the work while workers are down. Similar, but not the same as a log (Kafka). A queue is for jobs: one worker does each, then it’s gone. A log is for events: many readers, and they can go back and read them again. Alert on the age of the oldest message, not on the length.
Worker A process that takes jobs off a queue and does them. Celery workers are exactly this.
Scheduler Starts work on a timer, like “every minute, find the metrics that are due”. Cron is the classic one, Celery beat and Kubernetes CronJobs are the same idea. Not the Kubernetes scheduler, which decides which machine a container runs on. Same word, different job. Run exactly one. Celery beat on every server means every job runs once per server.
Log, event stream Like a queue, but reading doesn’t delete anything. Every consumer keeps its own position (the offset), so several consumers can read the same events at their own pace, and go back and read them again (Kafka, Kinesis). Old messages are deleted after a while (7 days by default in Kafka). If a consumer is too slow, then this is how it will miss messages.
Fan-out One thing going to many places: an event to every consumer, a post to every follower’s feed.
Polling vs webhooks Two ways to find out that something changed in another system. Polling: you ask again and again (“anything new?”), and mostly hear “no”. Webhooks: the other side calls you when something happens. Webhooks are cheaper and faster, polling works with every API. Webhooks get lost: your endpoint was down, or they simply never arrived. Most systems that rely on webhooks still poll now and then to catch what they missed.
Cache A fast copy of data that is expensive to get again, usually in Redis, with a TTL (time to live) after which it expires. If we aren’t careful, the cache becomes load-bearing: the database only copes because the cache catches most of the reads. Will the DB survive if the cache is cleared?
Index A sorted lookup next to a table, like the index at the back of a book, so the database can jump straight to the matching rows instead of reading all of them. It turns “find customer 42’s orders” from reading millions of rows into reading a few. A partial index covers only the rows you actually query (WHERE status = 'active'), so it stays small. Every index makes every write a bit slower and takes disk space. And the order of the columns matters: an index on (customer_id, created_at) helps “customer 42, last week”, but not “everything from last week”.
N+1 queries Loading a list with 1 query, then running 1 more query for every item on it: 200 items, 201 queries. The fix: load everything in one go, or batch the lookups. Not the same as N+1 redundancy (spare capacity). It hides in ORMs, because they often do SQL calls in the background. The code might be one innocent loop, but the database sees 201 queries. This usually only becomes noticeable in production and in peak traffic.
Connection pool A few database connections that many requests take turns using, instead of each request opening its own. Few, because connections are expensive: opening one takes several round trips, Postgres runs a whole process for each, and past a point more connections only means more waiting. PgBouncer does this for Postgres. Every new server brings its own pool. 20 servers with 20 connections each is 400 connections.
Primary, standby The primary database takes all the writes. A standby is a copy in another zone that takes over when the primary dies. (The old names were master and slave.) Not the same as a read replica. A standby waits to take over and usually serves no reads, a read replica is there to take reads off the primary. Failover still means a minute or two of errors, and until a new standby is built, there’s no safety net.
Read replica A read-only copy of the database that takes reads off the main one. It can be a few seconds behind. Read-your-writes: a user saves something, the page reloads from a replica that’s a few milliseconds behind, and the change looks lost. It doesn’t take much lag, the next read comes right away.
Time-series database Storage built for (series, time, value) data, with compression and rollups built in: TimescaleDB, InfluxDB, ClickHouse.
Columnar database Stores data column by column instead of row by row. Reading 3 columns out of 30 is fast and similar values compress well, so it’s the tool for analytics over billions of rows (ClickHouse, BigQuery, Snowflake). Updates and deletes are slow, including the ones GDPR requests force on you, and it wants inserts in big batches, not one row at a time.
Search index Elasticsearch. For text search and flexible filtering. It’s a copy that can drift. Never make it the source of truth, keep it rebuildable from the database.
Object storage S3 and friends. Cheap storage for files.
Availability zone One data center (or a few close together) inside a cloud region, with its own power and network. Zones are kilometers apart, so a fire or a power cut takes out one zone, not the whole region.
Region A group of zones in one area, like Frankfurt or Virginia. Between regions, a round trip takes tens to hundreds of milliseconds, inside one it’s a millisecond or two.

Growing

When we need more of something: a bigger machine, more machines, data split into pieces.

Word Meaning
Vertical scaling A bigger machine. Simple, but there’s a ceiling. It’s still one machine, and resizing it usually means a restart.
Horizontal scaling More machines. Needs stateless servers or data that’s split up.
Autoscaling The platform adds servers when the load goes up and removes them when it goes down, based on a number like CPU or requests in flight. It reacts to the spike you had a minute or two ago, because new servers take time to start. Keep some headroom, and scale on the load itself (requests in flight), not only on CPU.
Graceful shutdown When the platform wants a server gone (a deploy, scaling in), it first asks it to stop (SIGTERM). A graceful server stops taking new work, finishes what’s in flight, and then exits. If it takes too long, the platform kills it outright (SIGKILL). Long-running work doesn’t fit into the grace period (Kubernetes gives 30 seconds by default). A 10-minute job has to be safe to kill and run again, or be split into smaller pieces.
Vertical partitioning A database design technique that splits a wide table into smaller tables by dividing its columns. The same idea works one level up: splitting one database into several by what the data is (the app’s tables in one, the events in another). That’s usually the first split, and much cheaper than sharding. Not the same as vertical scaling, which just means a bigger machine.
Partitioning, sharding Splitting one big table or database into pieces by a key, like by customer or by day. “Partitioning” usually means splitting inside one database, “sharding” means splitting across several machines. People use both words for both.
Partition key What you split by. Pick it based on how the data gets read. Queries that don’t include the key have to ask every partition.
Consistent hashing A way to spread keys over servers so that adding a server moves only about \(1/n\) of the keys, instead of almost all of them like \(\text{key} \bmod n\) does. More in my post about it.
Hot partition One partition that happens to get far more traffic than the rest, usually your biggest customer.
Hot key One single key (one customer, one celebrity, one popular chart) that gets a big share of all traffic. It needs caching, copies, or splitting the key. A better partition key doesn’t help, it’s still one key.
Precomputing Calculating the answer ahead of time and storing it, so reads are cheap. You pay for it on write. A materialized view is one way to do it. Not the same as caching. A cache keeps an answer after someone asked for it, precomputing stores it before anyone asks. It only answers the questions you knew in advance.
Rollups Keeping detailed data for a short time and summaries (like hourly averages) for longer. Also called downsampling.
Denormalization Copying data into more than one place, so reads can skip the joins. Every copy has to be updated when the data changes, and one day one of them won’t be.
Retention How long you keep data. Deleting old data is part of the design, not an afterthought.

When things fail

We want to prevent it, expect it and contain it.

Word Meaning
Timeout How long you wait before giving up, on anything that can hang: an HTTP call to another service, a database query (Postgres has statement_timeout), waiting for a lock or for a free connection from the pool. For HTTP, the two to set first are the connect timeout (getting a connection at all) and the read timeout (waiting for the answer). Without timeouts, one slow dependency ties up all your workers. Many HTTP libraries have no timeout by default, Python’s requests waits forever. More on that. And \(timeout × retries\) has to fit inside the caller’s own time limit: 3 tries of 8s fit under a 30s request limit, 3 tries of 15s don’t.
Retry with backoff Try again after 1 s, then 2 s, then 4 s, plus a bit of randomness (jitter), so thousands of clients don’t all retry at the same moment. Retries multiply. 3 layers that each make 3 attempts turn 1 failing call into 27.
Retry storm The dependency is back, but the pile of retries from the outage keeps hammering it, and it goes down again. Backoff alone doesn’t stop it. Expire old retries, cap how many retries run at once, and let a circuit breaker hold them back.
Thundering herd Lots of things waking up at the same moment: every job at :00, or every client reconnecting after an outage.
Cache stampede A popular cached value expires and hundreds of requests rebuild it at the same moment, all hitting the database. The fix: let one request rebuild it while the others wait or get the old value. A special case of a thundering herd: many requests waking up at once for the same missing value.
Idempotent Doing it twice has the same effect as doing it once. “Set the balance to 100” is idempotent, “add 10 to the balance” is not. It matters because a request can succeed while its answer gets lost: the caller retries, and the work happens twice. Idempotent operations make those retries safe. Most operations aren’t idempotent on their own. The usual fix is an ID per operation (an idempotency key): the receiver remembers which IDs it has already handled and skips repeats. Stripe’s API works exactly like this.
At-least-once delivery The queue promises that every message arrives, possibly twice. That’s the normal deal, which is why idempotency matters. Exactly-once is mostly a myth. The opposite is at-most-once. The difference is when the worker confirms (acks) a message: ack before doing the work, and a crash loses it (at-most-once). Ack after the work succeeded, and a crash means it runs again (at-least-once).
Dead-letter queue Where a message goes after failing too many times, so it stops blocking everything and someone can look at it. Sending messages back to be processed again is called a redrive. Alert on anything that lands in it. And don’t redrive automatically: a message that crashes the worker, replayed automatically, is an endless loop.
Consumer lag How far behind a consumer is. Best measured in time (“4 minutes behind”), not in messages. Not the same as replication lag. Consumer lag is a queue or log consumer behind its messages, replication lag is a database replica behind its primary.
Circuit breaker Watch for failures in dependencies. Once they happen, you want to fail fast and re-call the dependency only carefully again later. It has three states: closed (calls go through), open (calls fail right away), and half-open (a trial call checks whether the dependency is back). Failing fast only helps if the caller knows what to do instead: show old data, use a default, or give a clear error. More in Pattern #3: Circuit Breaker design pattern
Bulkhead Separate pools of workers or connections per dependency, like the watertight compartments of a ship: when one floods, the others stay dry. A slow GitHub fills up only the GitHub workers, and Slack notifications keep going. One shared pool for everything means the slowest dependency decides how fast everything else goes. More in Deep-dive #1: Why a Slow API Is Worse Than a Dead One
Noisy neighbor Many customers share a system and one customer’s load hurts everyone else.
Rate limiting A cap on how often something may happen. You apply it to your users, and 3rd-party APIs apply it to you (HTTP 429). Common ways to count: fixed window (simple, but lets bursts through at the edges of each window), sliding window, and token bucket (allows short bursts, then a steady rate). Limits usually count per API key or per IP, so many workers sharing one key also share one limit. And when an API answers 429, wait as long as its Retry-After header says. More in Pattern #1: Caching Rate Limits (in Redis)
Backpressure When the system is full, it slows down or refuses new work, instead of drowning. It applies pressure back to the caller.
Load shedding When overloaded, refuse some requests right away (a fast 503), so the rest still finish in time, instead of every request getting slow. The same works for queues: work that can’t be done in time anymore gets expired instead of done late, like a per-minute job that expires before the next one is scheduled. Not the same as backpressure. Backpressure tells the sender to slow down, load shedding drops the extra work right away.
Graceful degradation When one part is broken, serve something reduced (slightly old data, a banner) instead of an error page.
Single point of failure One box whose death takes everything down. Usually the ones we forgot to draw on the diagram: DNS, an expiring certificate, the one person who knows how to deploy.
N+1 (redundancy) N is how many you need, +1 is the spare. Enough capacity that losing one of anything is fine. If 4 servers handle your peak, you run 5, so any one of them can die and nobody notices. Not the same as N+1 queries. Same name, completely different problem.
Failover Switching to a standby when the primary dies. It’s never instant: often a minute or two for a database, and every client has to notice and reconnect.
Backfill Going back and filling in the data you missed.

Correctness

Word Meaning
Source of truth The one place whose data wins when copies disagree.
Transaction, ACID A transaction groups several changes into one: either all of them happen, or none do. ACID is what the database promises about it: Atomic (all or nothing), Consistent (the data follows its rules, like unique keys), Isolated (transactions running at the same time don’t see each other’s unfinished changes), Durable (once committed, it survives a crash). Isolation is weaker than it sounds. By default, most databases (Postgres included) only promise you won’t see uncommitted changes, not that two transactions can’t trip over each other. More in What to do when a DB transaction fails? Retry it ♻️
Race condition The result depends on which of two things happens first. The classic one is the lost update: two requests both read a counter at 10, both add 1, both save 11, and one of the updates is gone. Locks, transactions and atomic updates (UPDATE ... SET count = count + 1) are all ways to stop it. It never shows up on your laptop, because there’s only one of you clicking. It shows up under load, rarely, and it’s very hard to reproduce.
Optimistic vs pessimistic locking Two ways to stop two people from overwriting each other’s changes. Pessimistic: lock the row while you work on it (SELECT ... FOR UPDATE), and everyone else waits. Optimistic: don’t lock, keep a version number on the row, and only save if it hasn’t changed since you read it (UPDATE ... WHERE version = 7). If it has, someone else got there first: reload and try again. Optimistic is great when conflicts are rare, and turns into endless retries when they’re not.
Deadlock Two transactions each wait for each other. The reason is the order in which locks were made. Both want to lock things A and B, but one started with A, the other with B. Many databases notice this and kill one lock. Your code has to expect that: every transaction should expect to be retried, the must be keep short, and locks must be madein in the same order everywhere. More in What to do when a DB transaction fails? Retry it ♻️
Distributed lock Making sure only one process in the whole cluster does something at a time, for example with Postgres advisory locks or Redis locks. A lock that expires on its own, so a crashed holder can’t block everyone forever, is called a lease. A lock can expire while its holder is still working (a long pause, a slow network), and then two processes both think they have it. That’s fine for a lock that only avoids duplicate work. For a lock that protects correctness (money, data), the storage has to reject the old holder too.
Strong vs eventual consistency Strong means everyone sees a write immediately. Eventual means copies catch up after a moment. Caches and replicas are eventually consistent. “Eventually” has no upper limit. Ask how long, and measure it.
Replication lag How far a replica is behind the primary. Usually under a second, sometimes minutes.
CAP theorem When the network between copies of your data breaks, you have to choose: keep answering and risk giving out old data, or stop answering until the copies agree again. Available or consistent, not both. The C in CAP (every copy shows the latest write) is not the C in ACID (the data follows its rules, like unique keys).
Cache invalidation Deciding when a cached copy is wrong and has to go. Always keep a TTL, even on keys you delete yourself. A missed delete then means stale for minutes, not forever.
Dual write, outbox Writing to two systems (the database and Kafka, say) is never atomic: a crash in between leaves them disagreeing. The usual fix is to write the message into an outbox table in the same database transaction, and let a separate process publish it.
CDC Change data capture: keeping other systems (a search index, a warehouse, a cache) in sync with the database without the app writing to both. Something reads the database’s own change log and publishes every committed change as an event. Debezium is the usual tool.

The bigger picture

Word Meaning
SLO A target you commit to, like “99.9% of metric points land within 5 minutes”. Not the same as an SLA. The SLA is the contract with the customer, usually with money back when it’s missed. The SLO is your own target, set stricter than the SLA. Measuring it where it’s easy (on the server) instead of where users feel it (end to end).
Error budget The failures an SLO allows. 99.95% a month is about 22 minutes of downtime. Budget left: ship faster. Budget spent: work on reliability.
Monitoring Watching the numbers you decided on in advance (error rate, latency, queue age) and alerting when one of them crosses a line. It tells you that something is wrong. Alert on what users feel (errors, slowness), not on every possible cause (CPU at 80%). Too many alerts, and people start ignoring all of them.
Observability How well you can figure out why something is wrong from the data your system already records. Every system has some, the question is whether you can connect the dots. Monitoring tells you the error rate went up. Good observability lets you find out it’s only self-hosted GitLab, since the 14:02 deploy. Plain metrics add everything up and throw away details like the customer ID, to stay cheap. You find out they’re missing in the middle of an incident.
Correlation ID, distributed tracing One ID given to a request when it arrives, and passed along to everything it causes: other services, background jobs, log lines. Search for the ID, and you see everything that request caused.More in Build me a global user_id / request_id / tenant_id
Blast radius How much breaks when one thing breaks. You want it small. A bug in the CSV export breaks the CSV export. A migration that locks the biggest table can take down the whole database, and every feature with it. So before a change, it’s worth asking: if this goes wrong, what’s the worst it can take down?
Multi-tenancy, isolation Many customers on shared infrastructure, without seeing each other’s data or slowing each other down. Pooled means everyone shares the same database and servers, siloed means each customer gets their own. Most SaaS is pooled, sometimes with the biggest customers siloed. One query or one cache key without the tenant ID is all it takes for customers to see each other’s data.
Data residency A legal or contractual rule that data has to stay in a certain place, like “EU customers’ data stays in the EU”.
Migration path How you get from today’s system to the new one without downtime, usually piece by piece with old and new side by side (the “strangler” pattern).
Rollout Shipping behind a feature flag, to a few customers first, or in shadow mode, where the new version runs next to the old one without affecting users. Code gets rolled out carefully, but a config change or a feature flag switched on for everyone takes effect everywhere at once. If it’s wrong, it’s wrong for 100% of users immediately. Roll those out gradually too. (The CrowdStrike outage in July 2024 was exactly this: a configuration update pushed to every machine at once.)
Conway’s law Systems end up shaped like the teams that build them.
Build vs buy Running something yourself or paying for a managed service. The price of “build” is mostly people: someone has to know how to fix it at night.

Next chapter

⏭️ To be continued...