Skip to main content
MAMS uses a registry.json file to track where model data is stored across multiple servers and their date ranges.

​
Registry Structure

The registry.json file is a JSON array with one object per server: 
[
  {
    "name": "mams.devops.arabiaweather.com",
    "connection": {
      "host": "mams.devops.arabiaweather.com",
      "port": "8082"
    },
    "archive": {
      "GFS": [["2025-09-01T12:00:00", ""]],
      "UK10": [["2025-01-01T00:00:00", ""]]
    },
    "static": ["ELE", "LSM", "GFSA14", "WRFMEA14"]
  },
  {
    "name": "archive.mams.devops.arabiaweather.com.138.201.49.117",
    "connection": {
      "host": "138.201.49.117",
      "port": "8082"
    },
    "archive": {
      "GFS": [["2024-07-01T00:00:00", "2025-08-31T18:00:00"]],
      "UK10": [["2024-07-01T00:00:00", "2024-12-31T12:00:00"]]
    }
  }
]
Fields:
  • name: Server identifier
  • connection: Connector service host and port (8082)
  • archive: Model names mapped to date ranges [start_date, end_date]
  • static (optional): Static data identifiers (main server only)
Date ranges:
  • Empty end date ("") = ongoing/current data
  • Date format: ISO format (e.g., "2024-07-01T00:00:00")
Location: /data/archives/registry.json on main server

​
Archive Process

When the main server gets full, archive data to an archive server:
1

Copy Data

Use rsync to copy NetCDF files to archive server:
rsync -avz /modms/nc/<model>/ <archive_server>:/data/archives/<model>/
2

Verify Data

SSH to archive server and verify data was copied correctly.
3

Delete from Main Server

Remove data from main server after verification:
rm -rf /modms/nc/<model>/<date_range>/
4

Update Registry

Update registry.json to reflect new data locations and date ranges. See Syncing Registry below.
Always verify data integrity before deleting from main server.

​
Running MAMS Archive

MAMS runs via cron every 3 hours: 
0 */3 * * * /root/archive_modms.sh
The script runs a Docker container that archives specified models from /modms/nc to /data/archives.

​
Syncing Registry

After moving data between servers, update the registry to match actual data distribution.

​
Process

1

Get Date Ranges

For each server, determine date ranges for all models:
  • Find first and last .meta.json files in each model directory
  • Extract runTime from both files to get the date range
2

Compare Servers

Identify:
  • Fully moved: All data on archive server (remove from main entry)
  • Partially moved: Data split between servers (update ranges on both)
  • Only on main: Keep in main entry only
3

Backup Registry

cp /data/archives/registry.json /data/archives/registry.json.backup.$(date +%Y%m%d_%H%M%S)
4

Update Entries

  • Archive server: Add/update entry with all models and their date ranges
  • Main server: Remove fully moved models, update ranges for split models
5

Verify

Check that entries match actual data on each server and date ranges are correct.

​
Update Rules

  • Fully moved: Remove from main server entry
  • Partially moved:
    • Archive server: Date range of moved data
    • Main server: Date range starting after archive end date (use "" for ongoing)
  • Archive entry: Must list all models present with complete date ranges
Always backup registry.json before changes. Incorrect entries cause query routing failures.