> ## Documentation Index
> Fetch the complete documentation index at: https://docs-v2.ard.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment

> How Pinpoint is deployed and where it runs

## Overview

Pinpoint is deployed on **six standalone servers** (no Docker Swarm), each running its own Pinpoint stack:

* `n01.pinpoint`
* `n02.pinpoint`
* `n03.pinpoint`
* `n04.pinpoint`
* `n05.pinpoint`
* `internal-tools-pinpoint` (internal-only, same architecture but dedicated to internal services)

External traffic is routed through a **Hetzner load balancer**, and:

* **Route53** points to the Hetzner load balancer
* The Hetzner load balancer distributes traffic to the public Pinpoint nodes (`n01`–`n05`)

## Servers & environments

<Tabs>
  <Tab title="Production">
    Pinpoint production runs on five public nodes behind the Hetzner load balancer:

    * `n01.pinpoint.devops.arabiaweather.com`
    * `n02.pinpoint.devops.arabiaweather.com`
    * `n03.pinpoint.devops.arabiaweather.com`
    * `n04.pinpoint.devops.arabiaweather.com`
    * `n05.pinpoint.devops.arabiaweather.com`

    There is also an internal-only node:

    * `internal-tools-pinpoint.devops.arabiaweather.com`

    <Note>
      External traffic flows: **Route53 → Hetzner Load Balancer → Pinpoint nodes**.
    </Note>
  </Tab>

  <Tab title="Staging (legacy)">
    Staging runs on a legacy stack managed via Portainer:

    * Docker Swarm-style deployment
    * Nginx inside the stack as the main HTTP entry point

    <Warning>
      Staging does not fully match the six-node production architecture. Use it primarily for basic smoke tests, not for detailed performance comparisons.
    </Warning>
  </Tab>
</Tabs>

## How we deploy

Pinpoint is managed with **Docker Compose**, not Docker Swarm services.

On each node, deployments are typically performed as:

```bash theme={null}
cd /root
docker compose pull
docker compose up -d
```

The stack includes an `autoheal` container, which automatically restarts unhealthy containers (including Pinpoint replicas) based on health checks.

## Node-level deployment (per server)

Each node (for example `n04.pinpoint`) is managed as a **standalone host** with a small stack defined in `docker-compose.yml` under `/root`:

* `haproxy`: front proxy for Pinpoint, using `/root/haproxy.cfg`
* `pinpoint`: main API service (multiple replicas on the node)
* `redis-replica`: local Redis instance for **overrides** (replica of the central Redis cluster)
* `node-exporter` and `cadvisor`: node and container metrics for Grafana
* A separate `promtail` stack under `/root/promtail` for log shipping
* `autoheal`: watches container health and automatically restarts unhealthy containers

To inspect a node, SSH into it and check the relevant config files. For example, on the internal node:

```bash theme={null}
ssh root@internal-tools-pinpoint.devops.arabiaweather.com
```

Typical locations you will find:

* `/root/docker-compose.yml` – services on the node (Pinpoint, HAProxy, Redis replica, exporters)
* `/root/haproxy.cfg` – HAProxy frontend/backends pointing at Pinpoint containers
* `/root/promtail/docker-compose.yml` and `/root/promtail/promtail-config.yml` – Promtail setup for shipping logs to Grafana / Loki

### HAProxy configuration (per node)

Each node runs an HAProxy container listening on:

* `:80` – main HTTP entry point, routing to the local `pinpoint` containers on port `3000`
* `:8404` – Prometheus metrics endpoint for HAProxy

HAProxy:

* Uses Docker DNS (`resolvers docker`) to discover Pinpoint containers by the `pinpoint` service name
* Performs HTTP health checks on `/api/health` (expects status `200`)
* Exposes structured JSON logs to stdout (picked up by Promtail)

### Pinpoint service (per node)

On each node:

* The `pinpoint` service is run from a Docker image such as `registry.docker.devops.arabiaweather.com/pinpoint:v2025.12.08-hotfix-cache-yazan`
* Multiple replicas (e.g. `replicas: 14`) are started on the same host
* Health checks call `http://localhost:3000/api/health` from inside the container
* A local volume `location_cache` provides `/code/data` (elevation file, location cache)

Pinpoint connects to:

* **Central Redis** (e.g. `node03.cluster.devops.arabiaweather.com:6379`) for **cache read/write**
* **Local `redis-replica`** service for **overrides**
* **ModMS** at `modms.devops.arabiaweather.com`
* **Logging stack** (via Promtail → Loki → Grafana) for application logs and dashboards

## Staging Environment

There is a **staging Pinpoint instance** available via Portainer that still uses the **legacy deployment pattern**.

Use staging to:

* Validate new releases before production
* Test configuration changes
* Perform smoke tests for critical endpoints

The staging environment:

* Still runs on the **older stack** (Docker Swarm-style deployment with Nginx in the stack)
* Does **not** fully match the new six-node HAProxy + Docker Compose architecture
* Should be treated as a **legacy** environment while production runs on the new pattern

For details on how Pinpoint itself uses Redis caching and ModMS, see:

* [Pinpoint Overview](/data-ai/pinpoint)
* [Architecture](/data-ai/pinpoint/architecture)

## Observability

Pinpoint is integrated with **Grafana dashboards** for:

* **Metrics** (request rates, latency, errors, per-node health)
* **Logs** (via Promtail sending container and HAProxy logs into Loki, visualized in Grafana)

Use these dashboards to:

* Verify deployments
* Monitor traffic and performance across all six nodes
* Investigate incidents and errors quickly