> ## Documentation Index
> Fetch the complete documentation index at: https://docs-v2.ard.co/llms.txt
> Use this file to discover all available pages before exploring further.

# ModMS & Pinpoint Operations

> Complete guide for verifying, troubleshooting, and recovering ModMS and Pinpoint services

This document describes how to verify, troubleshoot, and recover **ModMS** and **Pinpoint** services when issues arise.

## General Troubleshooting

This page is a high-level **operations runbook**. For detailed architecture and deployment docs, see:

* [Pinpoint deployment](/data-ai/pinpoint/deployment)
* [ModMS architecture](/data-ai/modms/architecture)

If **Pinpoint** or **ModMS** returns `5xx` errors, start with the **Failures & Downtime** sections in:

* [Pinpoint FAQ](/data-ai/pinpoint/faq)
* [ModMS FAQ](/data-ai/modms/faq)

Once you have identified whether the issue is primarily with **Pinpoint** or **ModMS**, follow the detailed steps in those FAQs and architecture pages before escalating further.

## When Pinpoint is down

If the issue is primarily user-facing errors from Pinpoint:

* Start with the **Failures & Downtime** section in the [Pinpoint FAQ](/data-ai/pinpoint/faq).
* Use the recommended Grafana dashboards and API checks there to determine if the problem is:
  * Load balancer / infrastructure
  * Pinpoint containers or per-node issues
  * Upstream ModMS, Redis, or Geos
* If ModMS is suspected, continue with the **ModMS** sections below.

## When ModMS is down

If ModMS itself is unhealthy (UI down, stale models, or Pinpoint 5xx traced to ModMS):

* Follow the **Failures & Downtime** section in the [ModMS FAQ](/data-ai/modms/faq).
* For storage and ingestion specifics, see these ModMS pages:
  * [GlusterFS & mountpoint `/data`](/data-ai/modms/faq/mountpoint-not-working)
  * [GlusterFS commands and volume examples](/data-ai/modms/faq/glusterfs-commands)
  * [Server startup time and performance](/data-ai/modms/faq/server-startup-time)

## When both are degraded

Sometimes both Pinpoint and ModMS show issues (for example, high error rates, timeouts, and stale data):

1. Use Pinpoint’s **Failures & Downtime** flow to confirm whether upstreams are the root cause.
2. If ModMS is the bottleneck, prioritize restoring:
   * GlusterFS and `/data` mounts
   * ModMS Server, Indexer, and critical Aggregators
3. Once ModMS is stable and models are fresh, validate Pinpoint again from its FAQ checklist.

## High-level debugging flow

```mermaid theme={null}
flowchart TD
  userIssue["User sees errors/timeouts"] --> checkLB["Check Hetzner LB & Grafana"]
  checkLB --> checkPinpoint["Hit Pinpoint health/forecast API"]
  checkPinpoint --> checkModms["Check ModMS /web + q query"]
  checkModms --> checkRedis["Check Redis cluster (node03)"]
  checkModms --> checkGeos["Check Geos servers & locations"]
  checkModms --> checkGluster["For ModMS issues, check GlusterFS/Nginx & /data mount"]
```

## Servers & access

For an up-to-date view of servers and environments, refer to:

* [Pinpoint deployment – Servers & environments](/data-ai/pinpoint/deployment#servers--environments)
* [ModMS architecture – Servers & environments](/data-ai/modms/architecture#servers--environments)

Use your local SSH configuration (for example entries in `~/.ssh/config`) to reach these nodes, following team security guidelines.