CAP Theorem in Practice: What You Actually Get to Choose

Every distributed systems course teaches the CAP theorem: in the presence of a network partition, a distributed system must choose between consistency and availability. The catchy three-letter framing — pick two of C, A, P — is correct but misleading. P is not a property you choose; it is something the network does to you. And "consistency" in CAP is not what most developers mean by consistency when they think about their applications.

This post is the version of CAP that actually helps when you are picking a database. It uses Daniel Abadi's PACELC extension, which captures the part of the tradeoff CAP leaves out.

CAP, Restated Carefully

The original CAP statement: when a network partition happens, a distributed system can be consistent or available, but not both.

• Consistency in CAP means linearizability — every read sees the most recent write. There is one timeline. This is stronger than the consistency most application code needs.
• Availability in CAP means every non-failing node responds successfully to every request. No timeouts, no errors.
• Partition tolerance is the system's ability to keep operating when nodes cannot reach each other.

The most important thing to understand: P is not optional. Networks partition. Switches fail, kernels panic, virtualization gets confused. If you build a distributed system that cannot survive partitions, you have not built a distributed system. So the real choice during a partition is between C and A.

The systems people call "CA" are not actually CA — they are single-machine systems that have no partitions to worry about because there is no second node.

PACELC: The Other Half of the Tradeoff

CAP only describes behavior during a partition. PACELC adds: when there is no partition (E for "else"), the system still has a choice between latency (L) and consistency (C).

• PA / EL — During a partition, choose availability. During normal operation, optimize for latency. Cassandra, DynamoDB, Riak.
• PC / EC — During a partition, choose consistency. During normal operation, choose consistency. Spanner, FoundationDB, traditional ACID databases with synchronous replication.
• PA / EC — Available during partitions, consistent otherwise. Many "eventual consistency" defaults that can be tightened with a consistency level option.
• PC / EL — Consistent during partitions (refuse writes), low-latency otherwise. Rare in practice — most strongly-consistent systems also have latency costs during normal operation.

This framing is more useful than CAP because it forces you to think about both partition and non-partition behavior. Most production traffic happens in non-partition state — and what your system does during normal operation matters a lot.

What "Choosing C or A" Looks Like in Practice

Real database engines do not give you a binary choice. They give you knobs.

DynamoDB. Default is eventually consistent reads. You can request strongly consistent reads on a per-call basis at higher cost. Writes can be at one node (fast, AZ-local) or quorum (slower, durable across AZs). Tunable per request.

Cassandra. Replication factor (how many copies of each row) and consistency level per operation. ONE is fastest and weakest; LOCAL_QUORUM is the practical default; ALL is strongest and slowest.

INSERT INTO users (id, email) VALUES (1, 'a@example.com')
USING CONSISTENCY LOCAL_QUORUM;

SELECT email FROM users WHERE id = 1
USING CONSISTENCY LOCAL_ONE;

PostgreSQL with synchronous replication. Asynchronous by default (low latency, possible data loss on primary failure). Synchronous to one replica (waits for one acknowledgment). Quorum synchronous (waits for a majority). Stronger guarantees at the cost of write latency.

Spanner / CockroachDB. Strong consistency by default via Paxos or Raft consensus. Write latency includes a network round trip to a majority of replicas.

The lesson: "this database is CP" or "this database is AP" is too coarse. You can tune the same database to behave differently on different operations.

Consistency Levels Application Code Actually Cares About

CAP's "linearizability" is one consistency level. Real applications use a spectrum:

• Strong consistency / linearizability. Read-your-own-writes globally. The state of the world is unambiguous.
• Sequential consistency. All operations appear in some serial order, but readers may see different points in that order at different times.
• Causal consistency. Operations that depend on each other are seen in order; independent operations may be reordered.
• Read-your-own-writes consistency. A user sees the effects of their own writes immediately, but may see other users' writes out of order.
• Eventual consistency. All replicas eventually converge, but at any given moment they may disagree.

A surprising amount of application code is fine with read-your-own-writes — much weaker than linearizability — and the latency benefits of weaker consistency are substantial. Routing a single user's reads to the replica that received their writes is enough.

The Practical Algorithm

When picking a database or a consistency level, ask:

1. What is the latency budget? If the answer is single-digit milliseconds for writes, you cannot afford cross-region consensus. You are in EL territory.
2. What is the consequence of stale reads? Loyalty points, recommendations, view counts — fine to be stale. Bank balances, inventory at the moment of order, two-factor authentication state — not fine.
3. What is the consequence of accepted-then-lost writes during a partition? A payment that the user is told succeeded but disappears is worse than a payment that times out and they retry.
4. How often will partitions actually happen? Within a single region with good operations, partitions are rare. Across regions, they happen.

The honest answer for many applications: most operations want low latency and weak consistency, a handful of operations want strong consistency, and the database needs to support both. Strong consistency for decrement_inventory and record_payment; eventual consistency for update_recommendation_score.

Common Misunderstandings

• "My single-node database has chosen CA." No — it has just not had to choose. CAP only applies to distributed systems.
• "Strong consistency means correct." Strong consistency means linearizable. An application can be correct with weaker consistency and incorrect with strong consistency; consistency is a tool, not a goal.
• "Eventual consistency means it will be wrong forever." No — it means it will be wrong briefly and correct eventually. The brief window matters; designing UI for it is part of the work.
• "CAP requires me to write my own consensus algorithm." No — pick a database that implements the consistency level you need. Writing your own consensus is one of the most expensive engineering projects you can undertake.

Why It Matters

CAP and PACELC are not just academic vocabulary. They are the framework you use to evaluate database choices honestly. "We will use Cassandra because it scales" is not an answer if your application logic assumes strong consistency. "We will use Postgres because it is consistent" is not an answer if your latency budget cannot tolerate the write costs of multi-AZ synchronous replication.

Pick the storage to match the consistency your application actually needs — operation by operation, not in aggregate.

Picking a database for a workload where both consistency and latency matter? We help teams reason through the tradeoffs without retreating to academic vocabulary. scopeforged.com