Distributed locking coordinates access to shared resources across multiple processes or machines. In a single-process application, language-level locks handle synchronization. In distributed systems, processes don't share memory, so coordination requires external mechanisms. Getting distributed locking right is surprisingly difficult, and getting it wrong causes data corruption, lost updates, and race conditions.
The fundamental challenge is that distributed systems lack a global clock and messages can be delayed, lost, or reordered. A process that thinks it holds a lock might actually have lost it. A process waiting for a lock might receive a stale grant. These edge cases turn simple-sounding operations into complex distributed systems problems.
Why You Need Distributed Locks
Distributed locks prevent concurrent operations that would conflict. Classic examples include preventing double-processing of messages, serializing access to external APIs with rate limits, coordinating leader election, and ensuring only one process runs a scheduled task.
Consider a payment webhook handler. Stripe sends a webhook for a successful payment. Due to network issues, Stripe retries the webhook. Without locking, two processes might handle the "same" payment simultaneously, potentially double-crediting the user's account.
class PaymentWebhookHandler
{
public function handle(WebhookPayload $payload): void
{
$paymentId = $payload->paymentId;
// Without locking - race condition possible
if ($this->alreadyProcessed($paymentId)) {
return;
}
// Another process might be here simultaneously
$this->processPayment($payload);
$this->markProcessed($paymentId);
}
}
A distributed lock ensures only one process handles each payment:
class PaymentWebhookHandler
{
public function handle(WebhookPayload $payload): void
{
$paymentId = $payload->paymentId;
$lockKey = "payment:process:{$paymentId}";
$lock = $this->lockManager->acquire($lockKey, ttl: 30);
if (!$lock) {
// Another process is handling this payment
throw new PaymentProcessingInProgressException();
}
try {
if ($this->alreadyProcessed($paymentId)) {
return;
}
$this->processPayment($payload);
$this->markProcessed($paymentId);
} finally {
$lock->release();
}
}
}
Redis-Based Locking
Redis is popular for distributed locking due to its speed and atomic operations. A basic lock uses SET with NX (only set if not exists) and PX (expiration in milliseconds).
class RedisLock
{
private Redis $redis;
public function acquire(string $key, int $ttlMs): ?Lock
{
$token = bin2hex(random_bytes(16));
$acquired = $this->redis->set(
$key,
$token,
['NX', 'PX' => $ttlMs]
);
if (!$acquired) {
return null;
}
return new Lock($key, $token, $this);
}
public function release(string $key, string $token): bool
{
// Atomic check-and-delete to prevent releasing someone else's lock
$script = <<<LUA
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
LUA;
return (bool) $this->redis->eval($script, [$key, $token], 1);
}
}
The token ensures a process only releases its own lock. Without this, a slow process might release a lock that a different process now holds. The Lua script makes the check-and-delete atomic.
Lock Expiration and Safety
Lock TTLs (time-to-live) prevent deadlocks when processes crash. If a process acquires a lock and dies without releasing it, the TTL ensures the lock eventually expires. But TTLs create their own problems.
If the lock expires while work is still in progress, another process can acquire it. Now two processes operate concurrently; exactly what the lock was supposed to prevent. This is called "lock expiration during processing" and it's a fundamental challenge in distributed locking.
Fencing tokens help detect stale locks. Each lock acquisition gets a monotonically increasing token. The protected resource checks that incoming operations have valid tokens and rejects operations with older tokens.
class FencedLockManager
{
public function acquire(string $key, int $ttlMs): ?FencedLock
{
// Atomically increment and set
$fencingToken = $this->redis->incr("lock:fence:{$key}");
$acquired = $this->redis->set(
$key,
json_encode(['token' => $fencingToken, 'owner' => $this->processId]),
['NX', 'PX' => $ttlMs]
);
if (!$acquired) {
return null;
}
return new FencedLock($key, $fencingToken, $this);
}
}
// Resource that uses fencing tokens
class ProtectedResource
{
private int $lastSeenToken = 0;
public function update(int $fencingToken, array $data): void
{
if ($fencingToken <= $this->lastSeenToken) {
throw new StaleLockException(
"Fencing token {$fencingToken} is not newer than {$this->lastSeenToken}"
);
}
$this->lastSeenToken = $fencingToken;
$this->doUpdate($data);
}
}
Lock Renewal
Long-running operations need lock renewal. Instead of setting a long TTL (which delays recovery after crashes), acquire a short TTL and periodically renew it while work continues.
class RenewableLock
{
private string $key;
private string $token;
private int $ttlMs;
private bool $released = false;
private ?TimerInterface $renewalTimer = null;
public function __construct(string $key, string $token, int $ttlMs, RedisLock $manager)
{
$this->key = $key;
$this->token = $token;
$this->ttlMs = $ttlMs;
$this->manager = $manager;
// Renew at half the TTL interval
$this->startRenewal($ttlMs / 2);
}
private function startRenewal(int $intervalMs): void
{
$this->renewalTimer = Loop::addPeriodicTimer(
$intervalMs / 1000,
function () {
if ($this->released) {
return;
}
$renewed = $this->manager->renew($this->key, $this->token, $this->ttlMs);
if (!$renewed) {
// Lost the lock - signal to stop processing
$this->onLockLost();
}
}
);
}
public function release(): void
{
$this->released = true;
if ($this->renewalTimer) {
Loop::cancelTimer($this->renewalTimer);
}
$this->manager->release($this->key, $this->token);
}
}
Redlock Algorithm
Single-Redis locking fails if Redis restarts or fails over. The Redlock algorithm addresses this by acquiring locks across multiple independent Redis instances. A lock is considered held only when acquired from a majority of instances.
class Redlock
{
private array $instances;
private int $quorum;
private int $retryCount = 3;
private int $retryDelayMs = 200;
public function __construct(array $redisInstances)
{
$this->instances = $redisInstances;
$this->quorum = floor(count($instances) / 2) + 1;
}
public function acquire(string $key, int $ttlMs): ?Lock
{
$token = bin2hex(random_bytes(16));
for ($attempt = 0; $attempt < $this->retryCount; $attempt++) {
$startTime = microtime(true) * 1000;
$acquired = 0;
$acquiredInstances = [];
foreach ($this->instances as $instance) {
if ($this->tryAcquire($instance, $key, $token, $ttlMs)) {
$acquired++;
$acquiredInstances[] = $instance;
}
}
// Calculate time elapsed during acquisition
$elapsedMs = (microtime(true) * 1000) - $startTime;
$remainingTtl = $ttlMs - $elapsedMs;
// Check if we have quorum and enough time remaining
if ($acquired >= $this->quorum && $remainingTtl > 0) {
return new Lock($key, $token, $acquiredInstances, $remainingTtl);
}
// Failed to get quorum - release any acquired locks
foreach ($acquiredInstances as $instance) {
$this->release($instance, $key, $token);
}
// Random delay before retry
usleep(rand(0, $this->retryDelayMs) * 1000);
}
return null;
}
}
Redlock is controversial. Martin Kleppmann's analysis showed it can fail under certain timing assumptions. For many applications, a single Redis instance with proper monitoring is sufficient. Use Redlock when you need stronger guarantees and understand its limitations.
Database-Based Locking
Databases can implement distributed locks using row-level locking or advisory locks. This works well when you already have a database and don't want additional infrastructure.
// PostgreSQL advisory locks
class PostgresAdvisoryLock
{
public function acquire(string $key): bool
{
$lockId = crc32($key);
// pg_try_advisory_lock returns true if lock acquired
$result = DB::select(
"SELECT pg_try_advisory_lock(?) as acquired",
[$lockId]
);
return $result[0]->acquired;
}
public function release(string $key): void
{
$lockId = crc32($key);
DB::select("SELECT pg_advisory_unlock(?)", [$lockId]);
}
}
// Table-based locking
class TableBasedLock
{
public function acquire(string $key, int $ttlSeconds): bool
{
$expiresAt = now()->addSeconds($ttlSeconds);
$token = Str::uuid()->toString();
try {
// Atomic insert with conflict handling
DB::table('distributed_locks')->insert([
'key' => $key,
'token' => $token,
'expires_at' => $expiresAt,
]);
return true;
} catch (UniqueConstraintViolationException $e) {
// Lock exists - check if expired
$existing = DB::table('distributed_locks')
->where('key', $key)
->first();
if ($existing && $existing->expires_at < now()) {
// Expired - try to take over
$updated = DB::table('distributed_locks')
->where('key', $key)
->where('token', $existing->token)
->update([
'token' => $token,
'expires_at' => $expiresAt,
]);
return $updated > 0;
}
return false;
}
}
}
When to Avoid Distributed Locks
Distributed locks add complexity and potential failure modes. Before reaching for a lock, consider alternatives.
Idempotent operations don't need locks. If processing the same item twice produces the same result, concurrent processing is safe. Design operations to be idempotent when possible.
Optimistic concurrency uses version numbers instead of locks. Read the current version, do work, write with a version check. If the version changed, retry. This works well for low-contention scenarios.
Message partitioning ensures only one consumer processes each partition. Messages for the same entity go to the same partition, serializing access without explicit locks.
Conclusion
Distributed locking coordinates access to shared resources across processes. Redis provides fast, simple locking for most use cases. Fencing tokens and lock renewal handle edge cases around expiration. Database-based locking works when you want to avoid additional infrastructure.
Distributed locks are difficult to get completely right. Edge cases around timing, expiration, and failures are subtle. Prefer designs that avoid locks when possible; idempotent operations, optimistic concurrency, and partitioning often work better. When you do need locks, understand their failure modes and plan accordingly.