Skip to main content

​
Repository Setup

Clone the ModMS repository: 
git clone https://git.wthr.ws/weathertech/modms/
cd modms
ModMS uses multiple branches for different components. The repository structure uses git worktrees to manage separate branches in parallel directories.

​
Git Worktree Setup

ModMS has different branches for each component:
  • aggregator branch: Main branch for the aggregator component
  • indexer branch: Indexer component code
  • server branch: Server and receiver components
  • mams branch: MAMS (Model Archive Management System) - uses same code as server

​
Setting Up Worktrees

The repository uses git worktrees to allow working with multiple branches simultaneously. Here’s how to set them up: 
# Ensure you're on the aggregator branch (or main branch)
git checkout aggregator

# Add worktree for indexer branch
git worktree add indexer-worktree indexer

# Add worktree for server branch
git worktree add server-worktree server

# Add worktree for mams branch (uses server code)
git worktree add mams-worktree mams
If worktrees already exist, you can remove them with git worktree remove <path> and recreate them.

​
Worktree Structure

After setup, your directory structure will look like: 
modms/
├── Docker/              # Aggregator Dockerfile
├── source/             # Aggregator source code
├── static_data/        # Static data files
├── indexer-worktree/   # Indexer branch worktree
│   ├── Docker/
│   │   └── Dockerfile
│   └── source/
└── server-worktree/    # Server branch worktree
    ├── Docker/
    │   ├── Dockerfile
    │   ├── Dockerfile.nginx
    │   └── build.sh
    └── server/
Worktrees allow you to work on multiple branches simultaneously without switching between them. This is useful when building Docker images for different components.

​
Building Docker Images

Each ModMS component has its own Dockerfile and build process.

​
Aggregator Image

Build the aggregator Docker image from the root directory: 
cd modms
docker build -f Docker/Dockerfile -t modms-aggregator:latest .
Dependencies:
  • wgrib2 (GRIB file utilities)
  • eccodes (ECMWF codes library)
  • LDC compiler (D compiler)
  • System libraries: libssl, curl, cdo, etc.

​
Indexer Image

Build the indexer Docker image from the indexer worktree: 
cd modms/indexer-worktree
docker build -f Docker/Dockerfile -t modms-indexer:latest .
Dependencies:
  • eccodes (ECMWF codes library)
  • LDC compiler (D compiler)
  • NetCDF libraries
  • GDAL libraries

​
Server Image

Build the server Docker image using the build script: 
cd modms/server-worktree/Docker
./build.sh v1.0.0
This script builds two images:
  • registry.docker.devops.arabiaweather.com/modms:v1.0.0
  • registry.docker.devops.arabiaweather.com/modms-nginx:v1.0.0
Replace v1.0.0 with your desired tag. The build script requires a tag parameter.
Alternatively, build manually: 
cd modms/server-worktree
docker build -f Docker/Dockerfile -t modms:latest .
docker build -f Docker/Dockerfile.nginx -t modms-nginx:latest .

​
MAMS Image

MAMS uses the same code as the server. See the MAMS documentation for build instructions.

​
Local Testing

To test ModMS components locally, you’ll need to set up directories and configure environment variables.

​
Required Directories

Create the following directory structure on your host machine: 
mkdir -p /data/downloads    # Downloaded GRIB files
mkdir -p /data/state        # State files for tracking runs
mkdir -p /data/models       # Processed NetCDF files
mkdir -p /data/manifests    # Manifest files from aggregator
mkdir -p /data/temp         # Temporary files
You can use custom paths instead of /data/*. Just ensure the paths are consistent across your setup.

​
Environment Variables

ModMS components use several environment variables. Create a .env file or export them: 
# Container paths (used inside containers)
export DOWNLOAD_DIR=/data/downloads
export LAST_STATE=/data/state/<model_name>
export TEMP_DIR=/data/temp
export MODELS_DIR=/data/models

# Additional model-specific variables may be required
# Check your model configuration for specific requirements

​
Testing Aggregator - Check Run Ready

Check if a model run is ready for processing: 
docker run --rm \
  -v /data/downloads:/data/downloads \
  -v /data/state:/data/state \
  -v /data/temp:/data/temp \
  -v /path/to/modms/static_data:/static_data \
  -e DOWNLOAD_DIR=/data/downloads \
  -e LAST_STATE=/data/state/<model_name> \
  -e TEMP_DIR=/data/temp \
  -e MODELS_DIR=/data/models \
  --env-file .env \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --check-run-ready \
  --model <model_name> \
  --runtime <runtime>
Example: 
docker run --rm \
  -v /data/downloads:/data/downloads \
  -v /data/state:/data/state \
  -v /data/temp:/data/temp \
  -v /path/to/modms/static_data:/static_data \
  -e DOWNLOAD_DIR=/data/downloads \
  -e LAST_STATE=/data/state/uk10 \
  -e TEMP_DIR=/data/temp \
  -e MODELS_DIR=/data/models \
  --env-file .env \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --check-run-ready \
  --model uk10 \
  --runtime 20251222T120000
Volumes:
  • /data/downloads: Downloaded GRIB files (read-write)
  • /data/state: State tracking files (read-write)
  • /data/temp: Temporary processing files (read-write)
  • /static_data: Static data files (read-only)

​
Testing Aggregator - Download and Process

Run the aggregator to download and process a model: 
docker run --rm \
  -v /data/downloads:/data/downloads \
  -v /data/manifests:/output \
  -v /data/state:/data/state \
  -v /data/temp:/data/temp \
  -v /path/to/modms/static_data:/static_data \
  -e DOWNLOAD_DIR=/data/downloads \
  -e LAST_STATE=/data/state/<model_name> \
  -e TEMP_DIR=/data/temp \
  -e MODELS_DIR=/data/models \
  --env-file .env \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --models <model_name> \
  --runtime <runtime> \
  --output-dir /output
Volumes:
  • /data/downloads: Downloaded GRIB files (read-write)
  • /data/manifests: Output manifest files (read-write, mounted as /output)
  • /data/state: State tracking files (read-write)
  • /data/temp: Temporary processing files (read-write)
  • /static_data: Static data files (read-only)
Output: The aggregator creates a manifest file in /data/manifests with the format: <MODEL>_<RUNTIME>.json

​
Testing Indexer

Run the indexer to convert GRIB files to NetCDF: 
docker run --rm \
  -v /data/manifests:/manifests \
  -v /data/models:/data/models \
  -v /data/downloads:/data/downloads \
  -v /data/temp:/tmp/indexer \
  -e TEMP_DIR=/tmp/indexer \
  -e MODELS_DIR=/data/models \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-indexer:latest \
  ./modms-indexer-bin \
  --manifest /manifests/<manifest_file>.json
Example: 
docker run --rm \
  -v /data/manifests:/manifests \
  -v /data/models:/data/models \
  -v /data/downloads:/data/downloads \
  -v /data/temp:/tmp/indexer \
  -e TEMP_DIR=/tmp/indexer \
  -e MODELS_DIR=/data/models \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-indexer:latest \
  ./modms-indexer-bin \
  --manifest /manifests/ARDJO3_20251206T060000.json
Volumes:
  • /data/manifests: Manifest files (read-only, mounted as /manifests)
  • /data/models: Output NetCDF files (read-write)
  • /data/downloads: Source GRIB files (read-only)
  • /data/temp: Temporary indexer files (read-write, mounted as /tmp/indexer)

​
Cleanup Operations

​
Aggregator Cleanup

Clean up old downloaded files: 
docker run --rm \
  -v /data/downloads:/data/downloads \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-aggregator:latest \
  ./modms-aggregator-bin \
  --cleanup

​
Indexer Cleanup

Clean up old manifest and temporary files: 
docker run --rm \
  -v /data/manifests:/manifests \
  -v /data/models:/data/models \
  -v /data/downloads:/data/downloads \
  -v /data/temp:/tmp/indexer \
  -e TEMP_DIR=/tmp/indexer \
  -e MODELS_DIR=/data/models \
  -e DOWNLOAD_DIR=/data/downloads \
  modms-indexer:latest \
  ./modms-indexer-bin \
  --cleanup

​
MAMS (Model Archive Management System)

MAMS is an archiver system for ModMS NetCDF data. For detailed MAMS documentation, see the MAMS documentation.
MAMS uses the same codebase as the ModMS server and provides identical query capabilities. It archives model data to distributed storage servers using a registry system.

​
Next Steps

  • Review the Architecture documentation for system design details
  • Check the FAQ for common issues and troubleshooting
  • Explore the API Reference for query capabilities