Our previous blog posts about service discovery withConsul andetcd discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. In such systems it’s common to assign network locations to service instances dynamically, because their locations can change over time due to autoscaling, failures, or upgrades. To track the current network location of a service instance, clients need a sophisticated service discovery mechanism.
In this blog, we’ll explain how to dynamically add or remove load-balanced servers that are registered with Apache ZooKeeper TM , a tool used for service discovery, in combination with NGINX Plus’ on-the-fly reconfiguration API . With this solution you can automate service discovery and change the set of load-balanced servers without having to reload the NGINX Plus configuration.
Apache ZooKeeper is a software project of the Apache Software Foundation that provides an open source distributed configuration service, synchronization service, and naming registry for large distributed systems. ZooKeeper nodes – or Znodes , as they are popularly called – store their data in a hierarchical name space, much like a file system or a tree data structure. All clients can read from and write to the nodes, making ZooKeeper a shared configuration service.
To make it easier to combine the on-the-fly reconfiguration API with ZooKeeper, we’ve created a sample zookeeper-demo , with step-by-step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker , Docker Compose , and Homebrew , you can spin up a Docker-based environment in which NGINX Plus load balances HTTP traffic to a couple of hello-world applications, with all components running in separate Docker containers.
How the Demo Works
First we spin up a separate Docker container for each of the following apps:
- ZooKeeper – Performs service discovery.
- Registrator – Registers services with ZooKeeper. Registrator monitors the starting and stopping of containers and updates ZooKeeper about the state changes.
- hello – Simulates a backend server. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page containing the web server’s hostname, IP address, and port.
- A second instance of the hello app – Simulates another backend server.
- NGINX Plus – Load balances the above services.
The NGINX Plus container listens on the public port 80, and the built-in NGINX Plus dashboard on port 8080. The ZooKeeper container listens on ports 2181, 2888, and 3888.
Registrator monitors Docker for new containers that are launched with exposed ports, and registers the associated services with ZooKeeper. By setting environment variables within the containers, we can be more explicit about how to register the services with ZooKeeper. For each hello-world container, we set the
environment variable to
to identify the container as an upstream server for NGINX Plus to load balance. When a container quits or is removed, Registrator removes its corresponding Znode entry from ZooKeeper automatically.
Finally, with a tool written in Ruby and included in the sample demo,
, we use ZooKeeper watches
to trigger an external handler (
) every time there is a change in the list of registered service containers. This
script gets the list of all current NGINX Plus upstream servers, uses
to loop through all the containers registered with ZooKeeper that are tagged
, and uses the on-the-fly reconfiguration (
) API to add them to the NGINX Plus upstream group if they’re not listed already. It also then removes from the NGINX Plus upstream group any
-tagged containers that are not registered with ZooKeeper.
Using a script like the one in our demo to dynamically reconfigure upstream groups based on the services registered with ZooKeeper automates upstream reconfiguration in NGINX Plus. This automation also frees you from having to figure out how to issue the API calls correctly and reduces the amount of time between the state change of a service in ZooKeeper and its addition or removal within the upstream group in NGINX Plus.
Check out our other blogs about for server discovery for NGINX Plus:
And try out automated reconfiguration of NGINX Plus upstream groups using ZooKeeper for yourself:
- Download zookeeper-demo from our GitHub repository