Skip to main content

​
Overview

Pinpoint is deployed on six standalone servers (no Docker Swarm), each running its own Pinpoint stack:
  • n01.pinpoint
  • n02.pinpoint
  • n03.pinpoint
  • n04.pinpoint
  • n05.pinpoint
  • internal-tools-pinpoint (internal-only, same architecture but dedicated to internal services)
External traffic is routed through a Hetzner load balancer, and:
  • Route53 points to the Hetzner load balancer
  • The Hetzner load balancer distributes traffic to the public Pinpoint nodes (n01–n05)

​
Servers & environments

Pinpoint production runs on five public nodes behind the Hetzner load balancer:
  • n01.pinpoint.devops.arabiaweather.com
  • n02.pinpoint.devops.arabiaweather.com
  • n03.pinpoint.devops.arabiaweather.com
  • n04.pinpoint.devops.arabiaweather.com
  • n05.pinpoint.devops.arabiaweather.com
There is also an internal-only node:
  • internal-tools-pinpoint.devops.arabiaweather.com
External traffic flows: Route53 → Hetzner Load Balancer → Pinpoint nodes.

​
How we deploy

Pinpoint is managed with Docker Compose, not Docker Swarm services. On each node, deployments are typically performed as: 
cd /root
docker compose pull
docker compose up -d
The stack includes an autoheal container, which automatically restarts unhealthy containers (including Pinpoint replicas) based on health checks.

​
Node-level deployment (per server)

Each node (for example n04.pinpoint) is managed as a standalone host with a small stack defined in docker-compose.yml under /root:
  • haproxy: front proxy for Pinpoint, using /root/haproxy.cfg
  • pinpoint: main API service (multiple replicas on the node)
  • redis-replica: local Redis instance for overrides (replica of the central Redis cluster)
  • node-exporter and cadvisor: node and container metrics for Grafana
  • A separate promtail stack under /root/promtail for log shipping
  • autoheal: watches container health and automatically restarts unhealthy containers
To inspect a node, SSH into it and check the relevant config files. For example, on the internal node: 
ssh root@internal-tools-pinpoint.devops.arabiaweather.com
Typical locations you will find:
  • /root/docker-compose.yml – services on the node (Pinpoint, HAProxy, Redis replica, exporters)
  • /root/haproxy.cfg – HAProxy frontend/backends pointing at Pinpoint containers
  • /root/promtail/docker-compose.yml and /root/promtail/promtail-config.yml – Promtail setup for shipping logs to Grafana / Loki

​
HAProxy configuration (per node)

Each node runs an HAProxy container listening on:
  • :80 – main HTTP entry point, routing to the local pinpoint containers on port 3000
  • :8404 – Prometheus metrics endpoint for HAProxy
HAProxy:
  • Uses Docker DNS (resolvers docker) to discover Pinpoint containers by the pinpoint service name
  • Performs HTTP health checks on /api/health (expects status 200)
  • Exposes structured JSON logs to stdout (picked up by Promtail)

​
Pinpoint service (per node)

On each node:
  • The pinpoint service is run from a Docker image such as registry.docker.devops.arabiaweather.com/pinpoint:v2025.12.08-hotfix-cache-yazan
  • Multiple replicas (e.g. replicas: 14) are started on the same host
  • Health checks call http://localhost:3000/api/health from inside the container
  • A local volume location_cache provides /code/data (elevation file, location cache)
Pinpoint connects to:
  • Central Redis (e.g. node03.cluster.devops.arabiaweather.com:6379) for cache read/write
  • Local redis-replica service for overrides
  • ModMS at modms.devops.arabiaweather.com
  • Logging stack (via Promtail → Loki → Grafana) for application logs and dashboards

​
Staging Environment

There is a staging Pinpoint instance available via Portainer that still uses the legacy deployment pattern. Use staging to:
  • Validate new releases before production
  • Test configuration changes
  • Perform smoke tests for critical endpoints
The staging environment:
  • Still runs on the older stack (Docker Swarm-style deployment with Nginx in the stack)
  • Does not fully match the new six-node HAProxy + Docker Compose architecture
  • Should be treated as a legacy environment while production runs on the new pattern
For details on how Pinpoint itself uses Redis caching and ModMS, see:

​
Observability

Pinpoint is integrated with Grafana dashboards for:
  • Metrics (request rates, latency, errors, per-node health)
  • Logs (via Promtail sending container and HAProxy logs into Loki, visualized in Grafana)
Use these dashboards to:
  • Verify deployments
  • Monitor traffic and performance across all six nodes
  • Investigate incidents and errors quickly