Quick start

Build the images, point the pipeline at a configuration, and process a day.

Prerequisites

  • Docker, with enough memory allocated. MRVA is the constraint: it is launched with a 72 GB limit and has been observed peaking around 65 GB.
  • A MATLAB licence reachable from the build host, including the MATLAB Compiler. This is needed only to build the images — not to run them.
  • Python 3.11+ with uv for the orchestrator.
  • NASA Earthdata credentials in a .netrc for the L2P download stage.

The MATLAB build stage resolves its licence inside the container. If your site uses a network licence server, provide a network.lic naming that server before building — it is deliberately not checked into the repository, so the images can be built anywhere.

1. Build the images

The shared MATLAB base is built first; the module images inherit from it and will fail to build if it is missing.

./build_module.sh all        # base image, then all four modules

./build_module.sh mrva       # or one module at a time
./build_module.sh mrva --debug --no-cache

Only a few MATLAB Compiler licences usually exist at a site. If a module build fails with a licence error while the base image succeeded, the compiler licence is most likely in use elsewhere — the base image needs only a basic MATLAB licence, the module images need the compiler.

2. Configure

Copy config.example.json and set the input, output and static-resource paths for your host. Each stage has its own block, and the sensor table lives under the mrva and l2p sections. The full key-by-key reference is in the configuration reference; the static inputs you must stage yourself are covered in Static input data.

3. Run

uv run run_mur_pipeline.py --config config.json

Useful flags:

FlagEffect
--date YYYY-MM-DDTreat this as the run day, for reprocessing a past date
--process-daysAnalysis day(s) as offsets from the run day
--all-stagesFull nine-day window (reanalysis plus near-real-time)
--preprocess-onlyLand/ice, iQUAM and L2P only; skip the MRVA analysis
--execute STAGERun only named stages; repeatable
--sensors A,BRestrict to specific sensors
--force-nrtForce near-real-time mode for every date
--keep-containersLeave containers in place for debugging
--debugVerbose logging

4. Long runs

A full day can take well over an hour, so detach it and write to a log file:

nohup uv run run_mur_pipeline.py --config config.json \
  > log_$(date +%F_%H%M).txt 2>&1 < /dev/null &

Redirect to a file rather than piping through tee. If the session drops, tee's write to the dead terminal fails and takes the pipeline down with it. Use tail -f on the log instead.

Reading the log

Each stage reports a tick or a cross in the per-day summary. Two things are worth scanning for even on an apparently clean run:

  • WARNING: no <hem> ice available for the requested day — the land/ice stage substituted an older day's sea ice. It still succeeds; the analysis is just using staler ice than you might assume.
  • No L2P input files for <SENSOR> — that sensor contributed nothing to the day.