Skip to content

Designing A Distributed Locking System (Redis • Zookeeper • Redlock • DB-Based Locks)

1) Problem Clarification / Làm rõ bài toán

EN

Distributed locks ensure only one process performs a critical action at a time in a distributed system.

Use cases:

  • prevent double payment
  • cron job leader election
  • avoid oversell
  • ensure only one worker processes a task
  • create/update exclusive resources

VI

Distributed lock đảm bảo chỉ 1 tiến trình được chạy critical section.

Dùng cho:

  • chống double payment
  • chọn leader
  • chống oversell
  • đảm bảo mỗi task chỉ được xử lý 1 lần
  • ghi dữ liệu độc quyền

2) Requirements of a Good Distributed Lock

EN

✔ correctness / mutual exclusion
✔ avoid deadlock
✔ avoid starvation
✔ lease expiration
✔ fault tolerance
✔ fencing token to prevent stale lock owners

VI

✔ mutual exclusion (độc quyền)
✔ tránh deadlock
✔ tránh starvation
✔ lease theo thời gian
✔ chịu lỗi
✔ fencing token chống owner cũ

PART A — REDIS-BASED LOCKING

3) Simple Redis Lock (NOT SAFE)

EN

SET key value NX PX 5000

Problems:

  • client crash → lock stays until TTL
  • no fencing token
  • two clients may think they hold lock (edge cases)

VI

Khóa đơn giản trong Redis:

SET key value NX PX 5000

Vấn đề:

  • client crash → lock vẫn giữ
  • không có fencing token
  • edge case có thể 2 client cùng nghĩ mình có lock

4) Redlock Algorithm (Redis Labs)

EN

Redlock uses multiple Redis nodes (usually 5):

Client must acquire lock in majority nodes (3/5).

Process:

  1. Write lock to N/2+1 nodes
  2. Ensure request time < lock TTL
  3. If majority acquired → lock is valid
  4. To release → delete keys

VI

Redlock khóa trên nhiều node:

  • phải acquire trên đa số node (3/5)

Quy trình:

  1. Set lock ở đa số node
  2. Thời gian request < TTL
  3. Đạt đa số → có lock
  4. Release: delete

5) Redlock Controversy

EN

Martin Kleppmann argued Redlock does NOT guarantee correctness under certain failure scenarios (e.g., partitions, clock drift).

Key missing feature:
➡ No fencing token

VI

Martin Kleppmann chỉ ra Redlock không an toàn tuyệt đối trong network partition.

Thiếu quan trọng:
➡ Không có fencing token

PART B — ZOOKEEPER LOCKING (MOST CORRECT)

6) Zookeeper Ephemeral Sequential Nodes

EN

Process:

  1. Client creates ephemeral sequential node
  2. ZK returns a sequence number
  3. Client checks if it has lowest number
  4. The lowest = lock owner
  5. When client disconnects → node auto-deleted
  6. Next lowest takes lock

VI

Zookeeper:

  1. Client tạo node ephemeral sequential
  2. Nhận ID tuần tự
  3. Node có ID nhỏ nhất giữ lock
  4. Disconnect → node tự xoá
  5. Node tiếp theo được lock

This is the safest distributed lock.

PART C — DB-BASED LOCKS

7) Using Database Row Locking

EN

SELECT * FROM lock_table WHERE key='A' FOR UPDATE;

Issues:

  • DB becomes bottleneck
  • lock lost if connection drops
  • not ideal for highly distributed systems

VI

Khoá DB:

SELECT … FOR UPDATE

Nhược:

  • DB thành bottleneck
  • mất khoá nếu connection drop
  • không phù hợp hệ thống lớn

8) PostgreSQL Advisory Locks

EN

Logical lock, not tied to a table.

SELECT pg_try_advisory_lock(12345);

Better than table lock, but:

  • still centralized
  • no fencing token

VI

Advisory lock:

pg_try_advisory_lock

Nhưng:

  • vẫn tập trung
  • không có fencing token

PART D — LEASES & FENCING TOKENS

9) Lease-based Locking (Recommended)

EN

A lock has a lease time.
If holder does not renew → lock expires.

Used by:

  • etcd
  • Zookeeper
  • Consul

VI

Lease: khóa có thời hạn.
Nếu không renew → tự hết hạn.

10) Fencing Tokens (CRITICAL!)

EN

Every time a lock is acquired, generate a monotonically increasing token.

Downstream systems validate token:

token must be > last_token_seen

Prevents old lock holders from writing after losing the lock.

VI

Token fencing:

  • mỗi lần lock → sinh token tăng dần
  • downstream chỉ chấp nhận token lớn hơn

Ngăn trường hợp owner cũ vẫn ghi vào dữ liệu.

PART E — BEST PRACTICES & COMPARISON

11) Comparison Table

EN

FeatureRedis LockRedlockZookeeperDB Lock
Safetylowmediumhighmedium
Fencing tokennonoyesno
Partition toleranceweakmediumstrongweak
Auto-expireyesyesyesno
Complexityeasymediumhardeasy
Use casecache opsdistributed but lightcritical operationssmall cluster

VI

Tiêu chíRedisRedlockZookeeperDB
An toànthấpTBcaoTB
Fencing tokenkhôngkhôngkhông
PartitionyếuTBmạnhyếu
Auto-expirekhông
Độ phức tạpthấpTBcaothấp
Use casecachedịch vụ vừanghiệp vụ criticalcluster nhỏ

12) Use Case Recommendations

EN

Use Zookeeper / etcd for:

  • leader election
  • distributed critical systems
  • message queue coordination

Use Redis Redlock for:

  • short-lived locks
  • non-critical locking
  • quick and easy distributed lock

Use DB locks for:

  • monolithic app
  • small clusters

VI

Dùng Zookeeper/etcd cho:

  • leader election
  • hệ thống critical
  • coordination

Dùng Redlock cho:

  • lock ngắn
  • business không quá critical

Dùng DB cho:

  • hệ thống nhỏ
  • monolith

13) Observability

EN

Log & measure:

  • lock acquisition time
  • lock contention rate
  • renewal failures
  • expired locks
  • fencing token mismatches

VI

Theo dõi:

  • thời gian acquire
  • mức độ tranh chấp
  • lỗi renew
  • lock hết hạn
  • fencing mismatch
Published inAllSystem Design

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *