🦁 The Jungle of Microservices

It all started when our system grew too big for its boots.

We had microservices. We had containers. We had cloud-native dreams. But we also had chaos. Jobs were running twice, data was being written by two different services at the same time, and no one knew who was supposed to be in charge.

"We need a leader," said my teammate.

Not a manager. A process leader. Someone—or something—that could ensure only one node performed critical tasks. We needed coordination.

That’s when we met ZooKeeper.

🧠 What Is ZooKeeper, Really?

ZooKeeper isn’t just a fancy name. It’s a highly consistent, distributed coordination service.

Imagine a shared whiteboard that every service can write on and read from. But this whiteboard is:

  • Highly available
  • Consistent across all nodes
  • Able to detect failures
  • Capable of orchestrating who does what

It’s like a referee in a distributed football match—without it, everyone just kicks the ball.

🗂 ZooKeeper Basics: znodes, Watches, and More

ZooKeeper stores all of its data in a hierarchical file system of znodes:

/
├── app
│   ├── config
│   ├── leader
│   └── locks

Each znode can be:

  • Ephemeral: disappears when a client disconnects
  • Sequential: gets a unique number appended

And clients can watch znodes to get notified when they change.

These simple primitives are incredibly powerful. They enable distributed locks, barriers, configuration stores, and leader election.

🏗️ Under the Hood: ZooKeeper Architecture

ZooKeeper uses a replicated ensemble of servers to maintain consistency and availability. Here's how it works:

  • Leader: Handles all write requests and replicates them to followers.
  • Followers: Serve read requests and vote during elections.
  • Observers: Non-voting nodes that scale reads without affecting quorum.

Core Protocol: Zab (ZooKeeper Atomic Broadcast)

Zab ensures that:

  • All writes are ordered and durable.
  • A quorum (majority) agrees before any write is committed.
  • In case of leader failure, a new leader is elected automatically.

Each change is logged and committed only after syncing to the majority, which makes ZooKeeper highly consistent (CP in CAP).

The architecture is optimized for coordination, not data throughput. This makes it perfect for distributed system primitives—but not high-volume writes.

👑 The Quest for a Leader

We wanted to elect a leader among several microservices. Here’s how ZooKeeper did it:

LeaderSelector selector = new LeaderSelector(client, "/election", new LeaderSelectorListenerAdapter() {
    public void takeLeadership(CuratorFramework client) throws Exception {
        System.out.println("👑 I am the leader now!");
        Thread.sleep(10000); // simulate leadership duties
    }
});
selector.autoRequeue();
selector.start();

Each participant creates a sequential ephemeral node under /election. ZooKeeper assigns a number (e.g., node-0000000003). The node with the lowest number wins and becomes the leader.

If the leader crashes or disconnects, its znode disappears automatically (because it's ephemeral). The next node in line becomes the new leader. This ensures automatic failover with no human intervention.

It's democratic, fair, and incredibly reliable.

🔒 Locks Without the Pain

We used ZooKeeper to implement distributed locking too:

InterProcessMutex lock = new InterProcessMutex(client, "/locks/mylock");
if (lock.acquire(5, TimeUnit.SECONDS)) {
    try {
        System.out.println("🔐 Lock acquired, doing work...");
    } finally {
        lock.release();
    }
}

When multiple services attempt to acquire the same lock, only one succeeds at a time. ZooKeeper queues the others behind ephemeral sequential znodes. As soon as the current lock holder finishes and releases it, the next in line is notified and proceeds.

ZooKeeper also handles crash recovery. If a service crashes while holding a lock, its ephemeral node disappears, allowing others to take over. This makes ZooKeeper a safe and self-healing locking system.

⚠️ The Scaling Wall

We tried using ZooKeeper for sequential ID generation:

String nodePath = client.create()
    .creatingParentsIfNeeded()
    .withMode(CreateMode.PERSISTENT_SEQUENTIAL)
    .forPath("/ids/order-");
System.out.println("Generated ID node: " + nodePath);

It worked flawlessly—for a while. Each request generated a new znode with a globally unique, incrementing number. Perfect for things like order IDs, invoice numbers, or logs.

But then came scale.

When traffic spiked, ZooKeeper became the bottleneck. Every new ID meant a write, and every write meant replicating that change across the ensemble. ZooKeeper simply wasn't built for millions of writes per second.

So we rethought the architecture. Instead of generating every ID through ZooKeeper, we used it to allocate ID blocks to nodes:

// ZooKeeper allocates a block: 100000-100999 to this instance
int nextId = atomicCounter.incrementAndGet(); // within allocated range

This way, each service could safely generate thousands of IDs in memory, without stressing ZooKeeper. It became a coordinator, not a generator.

Lesson learned: ZooKeeper is excellent at leadership and locks—not volume.

🔄 Alternatives & Use Cases

Use ZooKeeper when you need:

  • Leader election 🗳
  • Distributed locks 🔐
  • Shared configuration 📂
  • Service discovery 📡

Don’t use it for:

  • High-throughput writes
  • Large data storage
  • Stream processing

🧘 Lessons From the Keeper

ZooKeeper taught us:

  • Simplicity scales
  • Coordination doesn’t have to be complicated
  • Failover should be automatic
  • A strong leader makes a strong system

Sometimes the best engineer isn’t the fastest, but the most reliable. And in our jungle of microservices, ZooKeeper was the quiet kingmaker that made sure the right service ruled at the right time.

And that made all the difference.

🔗 Resources