Skip to main content
Common questions and solutions for working with ModMS. If you don’t find your answer here, check the Architecture documentation.

​
Storage & Infrastructure

See the detailed answer: Why is GlusterFS used?
See the troubleshooting guide: Mountpoint /data not working

​
Adding New Models

Follow the step-by-step guide: How to add a new model

​
Performance & Operations

Learn about the startup process: Server startup time

​
GlusterFS Commands

See the command reference: GlusterFS Commands
See example output: Volume Status Example
See example output: Volume Info Example

​
Failures & Downtime

Use this section when ModMS is suspected to be causing Pinpoint 5xx errors, missing data, or slow responses.

​
Common symptoms

  • Pinpoint returns 500 or 502 errors for many locations
  • http://modms.devops.arabiaweather.com/web/ is slow or not loading
  • Models appear stale or missing in the ModMS UI
  • GlusterFS mount at /data is not available on one or more nodes

​
Quick health checks

  • ModMS UI
    • Open http://modms.devops.arabiaweather.com/web/ and confirm it loads and shows recent models.
  • Models freshness API
    • Check the latest last_updated values:
curl -i "http://modms.devops.arabiaweather.com/q?fields=model%20|%20last_updated&from=models"
  • GlusterFS mount
    • On ModMS servers, verify /data is mounted and responsive:
mount | grep /data
df -h /data

​
Typical root causes

  • GlusterFS issues
    • Peers not connected, volume problems, or /data not mounted
  • Nginx / front-end proxy issues
    • Nginx not running or misconfigured in front of the ModMS server
  • Indexer / Aggregator problems
    • Indexer stuck, Aggregator not downloading new runs, or failed jobs

​
Step-by-step debugging

  1. Check ModMS UI and freshness
    • Visit http://modms.devops.arabiaweather.com/web/
    • Use the q endpoint above to confirm models are updating.
  2. Validate GlusterFS health
    • On each ModMS node:
gluster peer status
  • All peers should be in Peer in Cluster (Connected) state.
  • If not, restart GlusterFS:
systemctl restart glusterd
  1. Verify /data mount
    • Ensure /data is mounted and accessible on all nodes:
mount | grep /data
df -h /data
  1. Check Nginx and ModMS containers/services
    • Confirm Nginx and ModMS containers are running and healthy (per your deployment stack).
    • Restart only after validating GlusterFS and /data mounts.
  2. Look at Aggregator/Indexer
    • On the Indexer server, use indexerctl and the steps in the operations/architecture docs to identify stuck or failed jobs.
    • If required, restart the relevant Aggregator or Indexer components following your standard procedures.