In the previous post we booted (or lifted in Crane parlance) our dockerized Puppet infrastructure. All containers are now happily running their services.

With a Vagrant setup I would assign static IP addresses to the virtual machines and create entries in my /etc/hosts file. This way I can use friendly hostnames to communicate with my virtual machines. As far as I can tell there are no easy ways to configure static ip addresses for containers (I'd love to receive feedback in the comments if I'm wrong about this).

When Docker starts a container it assigns an IP address to the container. The IP address is selected from remaining free addresses in a private network range. When you restart a container, Docker assigns a new IP address to the container which might not be the same as the previous one. Clearly we need a way to dynamically track the services and network locations of running containers. This is exactly the domain of service discovery.

Service discovery

Several solutions have appeared that aim to solve service discovery. Most of them provide a central datastore that is highly-available based on some form of clustering. When services start or stop they (un)register their services in the central datastore. When a particular service is required a request can be issued against the datastore on where to find the required service. Usually some sort of API is provided by the service discovery tool. A REST API and DNS are two prominent ways of requesting this information. In my search for service discovery tools I came across etcd, zookeeper and consul amongst others. Each solution varies in complexity and features. I encourage you to read about each one and see how you can benefit from them.

In my research on a simple service discovery tool for my Puppet setup I discovered that all existing solutions were pretty complex to setup and most of the time plain overkill for my needs. The only thing that I needed was a simple DNS based solution that would track container locations and integrate with my host system to be able to connect with my containers in a straight-forward manner. I tried several solutions, including skydock/skydns, Consul and some other tools. Jeff Lindsay has written useful blog posts about this subject. I would recommend to read some of his stuff and check out the tools that he has created.

With all those solutions I ran into issues that they would not allow me to create the seamless integrated experience with docker containers that I was looking for in my development environment. So I decided to create my own little tool for this. I only had one goal:

It would have to be easy to setup and allow full flexibility in choosing the local domain name


Docker-spy is the result of my efforts to create a DNS based tool that tracks container (re)creation and allow seamless integration with my host system. It does two things: run a DNS server and monitor the Docker event stream for container lifecycle events. When it detects that a container has moved to another network location (IP) it updates its internal in-memory database. By binding Docker-spy to the Docker bridge you can ensure that containers are able to find each other by hostname and that you can find the containers by hostname from your host system (by adding docker-spy as one of my namespace servers). The Docker-spy repository has all the details on how to run and customize Docker-spy but generally it works as follows:

Example (Linux based)

Suppose the Docker bridge IP is (this is fairly common)

1. Add the Docker bridge IP as a nameserver to /etc/resolv.conf


2. Start docker-spy

docker run --name docker_spy -p 53:53/udp -p 53:53 -v /var/run/docker.sock:/var/run/docker.sock iverberk/docker-spy  

Add the -d parameter to run in the container in detached mode.

3. Run two containers:

docker run --dns -h agent1.localdomain --rm -it centos:centos6 bash  
docker run --dns -h agent2.localdomain --rm -it centos:centos6 bash  

The DNS parameter ensures that the containers use Docker-spy as their nameserver. If you want to make sure that all containers use this DNS setting you can add it to the Docker daemon options.

4. You should now be able to ping from one container to another based on hostnames and you should also be able to ping from your host system. When the containers are started/stopped/destroy Docker-spy will automatically update or remove the IP addresses from its database and serve the appropriate DNS records.

Now that we can easily communicate with our Puppet stack containers it is time for the final blog post in this series: Provisioning Docker containers with Puppet and The Foreman