Overview
In this installment of our networking basics series, we'll dive into monitoring and health checks. The goal is to ensure that every service in our homelab can be proactively checked for availability, allowing us to detect issues before they impact our customers.
Background
At Breadtoasting.com, maintaining a reliable and responsive infrastructure is crucial. Our services range from game servers to web applications, each requiring robust monitoring mechanisms. In this post, we'll explore how we expose health endpoints and integrate them with external checkers to ensure that any downtime is caught early.
How It Works
Exposing Health Endpoints
Breadlab currently exposes several health-related endpoints:
/api/status: Host uptime and load averages read from /proc.
/api/servers: Game server list with per instance state pulled from AMP.
/api/servers/health: Evaluates whether AMP itself is reachable.
The /api/status endpoint provides a quick overview of the host's health, while /api/servers gives detailed information about game servers. The companion /api/servers/health endpoint ensures that an external checker can alert on issues with AMP.
Caching Results
To prevent hammering the backend services, we cache results for 30 seconds:
Error semantics matter: Our endpoints return distinguishable errors (upstream_unavailable vs not_configured vs not_found) so you can tell "gooten is down" from "we never set the key".
This caching mechanism ensures that frequent polling does not overwhelm our services.
Liveness and Readiness
Understanding liveness and readiness is essential for effective monitoring:
- Liveness: Checks if a process is running.
- Readiness: Ensures the process can serve requests. A hung server passes liveness but fails readiness.
Black Box vs White Box Monitoring
Black box monitoring checks from outside (e.g., whether users can access a service), while white box monitoring examines internal metrics like queue depth and error rates:
You need one of each; outside first, it's what users experience.
Both types are necessary for comprehensive health management.
The Status Page
For a more user-friendly interface, we plan to implement Uptime Kuma as our status page. This self-hosted solution will provide a public dashboard that tells the truth about service availability, including during incidents:
Rule from the business plan: The status page tells the truth, including during incidents.
Transparency builds trust.
Alerting Hygiene
Effective alerting requires careful consideration of what to monitor and when to act:
- Symptom-Based Alerts: Alert on symptoms rather than causes. For example, alert on high error rates but not on configuration issues that don't affect service availability.
- Human-Actionable Alerts: Only alert on things a human should act on at 2 AM.
An alert you ignore twice is deleted or fixed.
Results
Docker Healthchecks
Docker Compose supports healthcheck blocks, which can be used to ensure that containers are healthy:
version: '3.8'
services:
myservice:
image: myimage
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost/api/status"]
interval: 30s
timeout: 10s
retries: 5
By integrating these healthchecks, we can achieve self-healing for crash loops:
Restart: unless-stopped plus a healthcheck gets you self healing for crash loops.
Lessons Learned
- Proactive Monitoring: Expose clear health endpoints and use caching to prevent overloading the backend services.
- Comprehensive Monitoring: Use both black box and white box monitoring techniques to ensure comprehensive coverage.
- Transparent Communication: Implement a status page that tells the truth, even during incidents.
- Effective Alerting: Focus on symptom-based alerts and ensure they are actionable.
By following these principles, we can maintain a reliable infrastructure that minimizes downtime and ensures high availability for our services.