Getting started

Repository Setup

Clone the ModMS repository:

git clone https://git.wthr.ws/weathertech/modms/
cd modms

ModMS uses multiple branches for different components. The repository structure uses git worktrees to manage separate branches in parallel directories.

Git Worktree Setup

ModMS has different branches for each component:

aggregator branch: Main branch for the aggregator component
indexer branch: Indexer component code
server branch: Server and receiver components
mams branch: MAMS (Model Archive Management System) - uses same code as server

Setting Up Worktrees

The repository uses git worktrees to allow working with multiple branches simultaneously. Here’s how to set them up:

# Ensure you're on the aggregator branch (or main branch)
git checkout aggregator

# Add worktree for indexer branch
git worktree add indexer-worktree indexer

# Add worktree for server branch
git worktree add server-worktree server

# Add worktree for mams branch (uses server code)
git worktree add mams-worktree mams

If worktrees already exist, you can remove them with git worktree remove <path> and recreate them.

Worktree Structure

After setup, your directory structure will look like:

modms/
├── Docker/              # Aggregator Dockerfile
├── source/             # Aggregator source code
├── static_data/        # Static data files
├── indexer-worktree/   # Indexer branch worktree
│   ├── Docker/
│   │   └── Dockerfile
│   └── source/
└── server-worktree/    # Server branch worktree
    ├── Docker/
    │   ├── Dockerfile
    │   ├── Dockerfile.nginx
    │   └── build.sh
    └── server/

Worktrees allow you to work on multiple branches simultaneously without switching between them. This is useful when building Docker images for different components.

Building Docker Images

Each ModMS component has its own Dockerfile and build process.

Aggregator Image

Build the aggregator Docker image from the root directory:

cd modms
docker build -f Docker/Dockerfile -t modms-aggregator:latest .

Dependencies:

wgrib2 (GRIB file utilities)
eccodes (ECMWF codes library)
LDC compiler (D compiler)
System libraries: libssl, curl, cdo, etc.

Indexer Image

Build the indexer Docker image from the indexer worktree:

cd modms/indexer-worktree
docker build -f Docker/Dockerfile -t modms-indexer:latest .

Dependencies:

eccodes (ECMWF codes library)
LDC compiler (D compiler)
NetCDF libraries
GDAL libraries

Server Image

Build the server Docker image using the build script:

cd modms/server-worktree/Docker
./build.sh v1.0.0

This script builds two images:

registry.docker.devops.arabiaweather.com/modms:v1.0.0
registry.docker.devops.arabiaweather.com/modms-nginx:v1.0.0

Replace v1.0.0 with your desired tag. The build script requires a tag parameter.

Alternatively, build manually:

cd modms/server-worktree
docker build -f Docker/Dockerfile -t modms:latest .
docker build -f Docker/Dockerfile.nginx -t modms-nginx:latest .

MAMS Image

MAMS uses the same code as the server. See the MAMS documentation for build instructions.

Local Testing

To test ModMS components locally, you’ll need to set up directories and configure environment variables.

Required Directories

Create the following directory structure on your host machine:

mkdir -p /data/downloads    # Downloaded GRIB files
mkdir -p /data/state        # State files for tracking runs
mkdir -p /data/models       # Processed NetCDF files
mkdir -p /data/manifests    # Manifest files from aggregator
mkdir -p /data/temp         # Temporary files

You can use custom paths instead of /data/*. Just ensure the paths are consistent across your setup.

Environment Variables

ModMS components use several environment variables. Create a .env file or export them:

# Container paths (used inside containers)
export DOWNLOAD_DIR=/data/downloads
export LAST_STATE=/data/state/<model_name>
export TEMP_DIR=/data/temp
export MODELS_DIR=/data/models

# Additional model-specific variables may be required
# Check your model configuration for specific requirements

Testing Aggregator - Check Run Ready

Check if a model run is ready for processing:

docker run --rm \
  -v /data/downloads:/data/downloads \
  -v /data/state:/data/state \
  -v /data/temp:/data/temp \
  -v /path/to/modms/static_data:/static_data \
  -e DOWNLOAD_DIR=/data/downloads \
  -e LAST_STATE=/data/state/<model_name> \
  -e TEMP_DIR=/data/temp \
  -e MODELS_DIR=/data/models \
  --env-file .env \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --check-run-ready \
  --model <model_name> \
  --runtime <runtime>

Example:

docker run --rm \
  -v /data/downloads:/data/downloads \
  -v /data/state:/data/state \
  -v /data/temp:/data/temp \
  -v /path/to/modms/static_data:/static_data \
  -e DOWNLOAD_DIR=/data/downloads \
  -e LAST_STATE=/data/state/uk10 \
  -e TEMP_DIR=/data/temp \
  -e MODELS_DIR=/data/models \
  --env-file .env \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --check-run-ready \
  --model uk10 \
  --runtime 20251222T120000

Volumes:

/data/downloads: Downloaded GRIB files (read-write)
/data/state: State tracking files (read-write)
/data/temp: Temporary processing files (read-write)
/static_data: Static data files (read-only)

Testing Aggregator - Download and Process

Run the aggregator to download and process a model:

docker run --rm \
  -v /data/downloads:/data/downloads \
  -v /data/manifests:/output \
  -v /data/state:/data/state \
  -v /data/temp:/data/temp \
  -v /path/to/modms/static_data:/static_data \
  -e DOWNLOAD_DIR=/data/downloads \
  -e LAST_STATE=/data/state/<model_name> \
  -e TEMP_DIR=/data/temp \
  -e MODELS_DIR=/data/models \
  --env-file .env \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --models <model_name> \
  --runtime <runtime> \
  --output-dir /output

Volumes:

/data/downloads: Downloaded GRIB files (read-write)
/data/manifests: Output manifest files (read-write, mounted as /output)
/data/state: State tracking files (read-write)
/data/temp: Temporary processing files (read-write)
/static_data: Static data files (read-only)

Output: The aggregator creates a manifest file in /data/manifests with the format: <MODEL>_<RUNTIME>.json

Testing Indexer

Run the indexer to convert GRIB files to NetCDF:

docker run --rm \
  -v /data/manifests:/manifests \
  -v /data/models:/data/models \
  -v /data/downloads:/data/downloads \
  -v /data/temp:/tmp/indexer \
  -e TEMP_DIR=/tmp/indexer \
  -e MODELS_DIR=/data/models \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-indexer:latest \
  ./modms-indexer-bin \
  --manifest /manifests/<manifest_file>.json

Example:

docker run --rm \
  -v /data/manifests:/manifests \
  -v /data/models:/data/models \
  -v /data/downloads:/data/downloads \
  -v /data/temp:/tmp/indexer \
  -e TEMP_DIR=/tmp/indexer \
  -e MODELS_DIR=/data/models \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-indexer:latest \
  ./modms-indexer-bin \
  --manifest /manifests/ARDJO3_20251206T060000.json

Volumes:

/data/manifests: Manifest files (read-only, mounted as /manifests)
/data/models: Output NetCDF files (read-write)
/data/downloads: Source GRIB files (read-only)
/data/temp: Temporary indexer files (read-write, mounted as /tmp/indexer)

Cleanup Operations

Aggregator Cleanup

Clean up old downloaded files:

docker run --rm \
  -v /data/downloads:/data/downloads \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --cleanup

Indexer Cleanup

Clean up old manifest and temporary files:

docker run --rm \
  -v /data/manifests:/manifests \
  -v /data/models:/data/models \
  -v /data/downloads:/data/downloads \
  -v /data/temp:/tmp/indexer \
  -e TEMP_DIR=/tmp/indexer \
  -e MODELS_DIR=/data/models \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-indexer:latest \
  ./modms-indexer-bin \
  --cleanup

MAMS (Model Archive Management System)

MAMS is an archiver system for ModMS NetCDF data. For detailed MAMS documentation, see the MAMS documentation.

MAMS uses the same codebase as the ModMS server and provides identical query capabilities. It archives model data to distributed storage servers using a registry system.

Next Steps

Review the Architecture documentation for system design details
Check the FAQ for common issues and troubleshooting
Explore the API Reference for query capabilities

ModMS

MAMS

Pinpoint

Legacy Met Tools

Geos

Geoterra

Repository Setup

Git Worktree Setup

Setting Up Worktrees

Worktree Structure

Building Docker Images

Aggregator Image

Indexer Image

Server Image

MAMS Image

Local Testing

Required Directories

Environment Variables

Testing Aggregator - Check Run Ready

Testing Aggregator - Download and Process

Testing Indexer

Cleanup Operations

Aggregator Cleanup

Indexer Cleanup

MAMS (Model Archive Management System)

Next Steps

​Repository Setup

​Git Worktree Setup

​Setting Up Worktrees

​Worktree Structure

​Building Docker Images

​Aggregator Image

​Indexer Image

​Server Image

​MAMS Image

​Local Testing

​Required Directories

​Environment Variables

​Testing Aggregator - Check Run Ready

​Testing Aggregator - Download and Process

​Testing Indexer

​Cleanup Operations

​Aggregator Cleanup

​Indexer Cleanup

​MAMS (Model Archive Management System)

​Next Steps

Repository Setup

Git Worktree Setup

Setting Up Worktrees

Worktree Structure

Building Docker Images

Aggregator Image

Indexer Image

Server Image

MAMS Image

Local Testing

Required Directories

Environment Variables

Testing Aggregator - Check Run Ready

Testing Aggregator - Download and Process

Testing Indexer

Cleanup Operations

Aggregator Cleanup

Indexer Cleanup

MAMS (Model Archive Management System)

Next Steps