Networking Basics #5: Docker Networking — Why Containers Talk to Each Other by Name

Overview

In this installment of the BreadLab Networking Basics series, we'll dive into Docker networking. Specifically, we’ll explore why our web services running in containers can talk to each other by name and how this setup provides a layer of security that hardcoding IP addresses simply cannot match.

flowchart TD
  H[Docker host] --> BR[user defined bridge network]
  BR --> C1[nginx]
  BR --> C2[amp-proxy]
  BR --> C3[blog-backend]

Background

At BreadLab, all our web services run inside Docker containers attached to a single user-defined bridge network named unified-network. This network allows the containers to communicate with each other using container names instead of IP addresses. The internal DNS server provided by Docker handles these name resolutions, making it incredibly convenient and secure.

How It Works

Bridge Network

A bridge network is a virtual switch inside the host machine that provides private IPs (typically in the 172.x range) to containers on the network. These IPs are invisible from the LAN, ensuring that only specific ports are exposed to the outside world.

User-Defined Networks

User-defined networks offer several advantages over Docker's default bridge network:

• Name-based DNS: Containers can resolve each other by name thanks to the internal DNS server.
• Isolation and Security: By default, containers on a user-defined network cannot be reached from the host or the LAN unless specific ports are published.

Published Ports

When you define ports: in your Docker Compose file, you're telling Docker which container ports should be exposed to the host. Only the front door (in our case, nginx and cloudflared) needs this exposure; backends stay unreachable from outside the network. This setup provides free security: our blog admin literally cannot be reached except through the proxy.

Default Bridge vs User-Defined Networks

The default bridge network does not support name-based DNS resolution. Therefore, always create a user-defined network if you need containers to resolve each other by name.

Host Communication

To reach services on the host from inside a container, Docker provides host.docker.internal. In our setup, the amp-proxy service uses an extra_hosts entry mapping it to the host gateway to reach AMP running on another machine's IP. Additionally, we mount /proc for host stats.

A Weirdness: "localhost" Inside a Container

A classic beginner trap is thinking that "localhost" inside a container refers to the host. In reality, it refers to the container itself. This can lead to confusion when troubleshooting network issues.

Results

Our setup has proven robust and secure. However, we encountered an interesting issue during testing: running two cloudflared processes (one native on the host and one in Docker) caused intermittent 502 errors for users. Both processes were listening in different network namespaces but claiming the same tunnel connection. This led to a conflict.

To diagnose this, we initially used ps aux, which only showed the process from inside the container. However, this did not reveal the native host process. The correct approach was to use both docker ps and ps aux simultaneously to get a full picture of what was running in both worlds.

Lessons Learned

1. Use User-Defined Networks: Always create user-defined networks for containers that need name-based DNS resolution.
2. Secure Backends: Expose only the front door (nginx/cloudflared) and keep backends unreachable from outside the network.
3. Be Mindful of "localhost": Remember that "localhost" inside a container refers to the container itself, not the host.
4. Use Both Worlds: When troubleshooting, don't just look at docker ps; check both the host and the container using tools like ps aux.

By following these guidelines, we've ensured our BreadLab environment is both secure and reliable.