> ## Documentation Index
> Fetch the complete documentation index at: https://docs-v2.ard.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Sensu Alerts & Operational Issues

> Common operational issues, troubleshooting steps, and recovery procedures for ModMS and Pinpoint services

This document tracks common operational issues, their resolutions, and troubleshooting procedures for ModMS and Pinpoint services.

## Operational Issues Log

| Component                                | Example                                                                                                                             | Server                                                                  | Issue                                              | Details                                                                                             | Action Taken                                                               | Command                                                                                                                                                                                                        |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Sams Server Raw Data Storage**         |                                                                                                                                     | `sams.devops.arabiaweather.com`                                         | Disk filling with raw data                         | Satellite data accumulation                                                                         | Regularly delete old raw data                                              | `cd /data/raw-data`<br />`# remove old data`                                                                                                                                                                   |
| **Sams Log Files**                       | [Sensu Alert](https://sensu.devops.arabiaweather.com/#/client/ArabiaWeather%20Sensu/sams.devops.arabiaweather.com?check=disk_usage) | `sams.devops.arabiaweather.com`                                         | Excessive log file growth                          | Log files grew too large on the Sams server                                                         | Truncated log files                                                        | `df -h`<br />`find / -xdev -type f -size +500M 2>/dev/null`<br />`truncate -s 0 /var/lib/docker/containers/<container-id>/*.log`                                                                               |
| **Historical Server pengine Cronjob**    |                                                                                                                                     | `htz-historical-01` (144.76.56.17)                                      | High memory usage                                  | `pengine` cronjob consuming excessive memory                                                        | Killed all `pengine` processes                                             | `killall -9 pengine`                                                                                                                                                                                           |
| **Redis Sentinel Alert**                 |                                                                                                                                     | `cluster-n03`, `cluster-n04`                                            | Unnecessary monitoring alert                       | Only two Redis nodes; Sentinel not required                                                         | Disabled Redis Sentinel alert                                              | *No command specified*                                                                                                                                                                                         |
| **Redis Ping Failure (node03 → node04)** |                                                                                                                                     | `cluster-n03`, `cluster-n04`                                            | Slave-to-master ping failure                       | node03 slave couldn't ping node04 master on port 3680 through HAProxy                               | Restarted HAProxy on node03                                                | `redis-cli info replication`<br />`systemctl reload haproxy`                                                                                                                                                   |
| **Bader 1**                              |                                                                                                                                     | `bader-deploy` (85.10.197.28)                                           | Containers unhealthy                               | Docker containers showing unhealthy status                                                          | Restart unhealthy containers                                               | `docker container ls -a`<br />`# restart unhealthy container`                                                                                                                                                  |
| **Bader 2**                              |                                                                                                                                     | `bader.arabiaweather.com`                                               | Restart order causing lost events                  | Removed `last_state` and cleared processed map keys; aggregator republished before engine was ready | Restart engine first, then aggregator                                      | `cd data/aggregator-downloads/`<br />`rm last_state`<br />`redis-cli keys 'maps:processed_files:*' \| xargs redis-cli del`<br />`systemctl restart engine.service`<br />`systemctl restart aggregator.service` |
| **Charts Archiver Jobs**                 | [Sensu Alert](https://sensu.devops.arabiaweather.com/#/client/ArabiaWeather%20Sensu/AW-Sensu-Client?check=charts_archiver_jobs)     | `backend-charts-1` (3.249.150.105), `backend-charts-2` (34.246.218.122) | Disk space issues, deleted processes holding space | Deleted processes consuming disk space, preventing proper operation                                 | Check disk space, identify and clean up deleted processes, restart unicorn | `df -h`<br />`sudo lsof \| grep deleted`<br />`# delete identified processes`<br />`cd /path/to/aw_charts_backend`<br />`/etc/init.d/unicorn start`                                                            |

## Known Instability Issues

* [Pinpoint Temperature Deviation](https://linear.app/ardlabs/issue/ARD-3070/temperatures-deviation-pinpoint)
* [Pinpoint Overrider Not Responding](https://linear.app/ardlabs/issue/ARD-3068/overrider-not-responding)
