Architecture

Four MATLAB containers on a shared base image, driven by a Python orchestrator that resolves every input path before any container starts.

Four containers, one base image

Each processing stage is a separate Docker image. All four compile MATLAB code against a shared base, mur-matlab-base:r2024b, which is built once and supplies the common toolchain — so the base must exist before any module will build.

Why containers

The analysis is MATLAB and Fortran with a long dependency tail. Each image compiles its MATLAB sources during the build — which needs a MATLAB Compiler licence — and ships only the compiled binaries plus the MATLAB Runtime on a slim Debian base. Running the pipeline therefore needs no MATLAB licence at all; only rebuilding does.

What each container does

1. Land/Ice Mask Generation (landice/)

Purpose: Create land and sea ice masks for the analysis domain

Input:

• OSI-SAF sea ice concentration data
• Static land masks

Output:

• .gds files: Land/ice masks with bitwise encoding
• .bip files: Ice SST point data

Technology: Containerized MATLAB (R2024b Runtime)

Processing: Generates masks for both p01 (0.01°) and p011 (0.011°) resolutions

See land/ice encoding for the mask bit layout.

2. L2P Satellite Data Processing (l2p/)

Purpose: Convert GHRSST L2P satellite data into MRVA-compatible format

Supported Sensors:

• AMSR2R - Microwave radiometer (all-weather, ~25km resolution)
• MODISA - MODIS Aqua infrared (~1km resolution)
• MODIST - MODIS Terra infrared (~1km resolution)
• AVMTAG - AVHRR MetOp-A infrared (~1-4km resolution)
• AVMTBG - AVHRR MetOp-B infrared (~1-4km resolution)

Input: GHRSST L2P NetCDF files from PO.DAAC

Output: .bic.gz files (Binary Input with Confidence)

Technology: Containerized MATLAB with sensor-specific configurations

Key Features:

• Quality filtering by confidence thresholds
• Nighttime-only observations (avoids diurnal warming effects)
• Sensor-specific scale parameters (La/Lb)
• Stability latency handling (2-3 day data maturity)
• Bias correction and error estimation

Note: MUR uses nighttime observations from infrared sensors to measure foundation temperature and avoid solar-induced diurnal warming. See future enhancements for the daytime-data discussion.

See adding a sensor for the integration workflow.

3. iQUAM Buoy Data Processing (iquam/)

Purpose: Process in-situ buoy/ship observations for MRVA

Input: NOAA iQUAM monthly NetCDF files

Output: .bii files (Binary IQUAM Instrument format)

Technology: Containerized MATLAB orchestrator

Key Features:

• Quality control filtering (quality_level >= 5)
• Platform type handling (ships, buoys, drifters, etc.)
• Monthly file caching for performance
• Temporal aggregation (±3 days)

4. Pipeline Orchestrator (run_mur_pipeline.py)

Purpose: Coordinate all preprocessing steps for a given analysis date

Functionality:

• NRT/REA mode detection
• Temporal window calculation
• Component sequencing
• Container execution management
• L2P download purge (on-demand cleanup of old DOY directories)

Status: Coordinates all four stages (landice, iquam, l2p, mrva)

The container pattern

Multi-stage build

All preprocessing containers follow a common pattern:

Multi-Stage Build:

# Stage 1: Builder (compile MATLAB code)
    FROM mathworks/matlab:r2024b AS builder
    RUN matlab -batch "mcc -m wrapper.m ..."

    # Stage 2: Runtime (minimal execution environment)
    FROM mathworks/matlab-runtime:r2024b
    COPY --from=builder /build/wrapper /app/

Execution:

docker run --rm \
      --platform linux/amd64 \
      --shm-size=512M \
      -v /input:/input:ro \
      -v /output:/output \
      ghcr.io/podaac/mur/<module>:latest \
      <args...>

Key Requirements:

• --shm-size=512M for MATLAB Runtime shared memory
• Volume mounts for input/output data
• Explicitly named flags for every input — never a scanned directory

The explicit-input contract

No container discovers its own inputs. Nothing scans a directory, builds a path from a root, or infers a sensor or date by parsing a filename. Every input arrives as an explicit named flag whose value the Python orchestrator resolved first.

Inputs take exactly two shapes:

• A direct value flag — --seasonal-file <path-or-href> — for anything that resolves to exactly one file.
• A manifest flag — --sensor-inputs-manifest <path-or-href> — for a variable, unbounded count of files, listed in a small JSON file.

Every value may be a local filesystem path or an s3:// href. A shared helper sourced by each entrypoint fetches s3:// values to local scratch before MATLAB starts; local paths pass through untouched. That is what lets the same image run unchanged locally and in a cloud environment — only how Python discovers the values differs.

Required versus optional

An absent optional input is a meaningful state, not an error. MRVA's previous-day coefficient file is the clearest case: when it is missing the analysis builds its reference field from scratch instead of failing. The entrypoints enforce only the genuinely required flags and check that every supplied path actually exists before MATLAB starts, so a misconfigured path fails immediately and by name rather than surfacing as an opaque read error deep inside a long run.

Orchestration

Two orchestrators implement the same contract against different execution backends:

• run_mur_pipeline.py — local Docker. Resolves host paths, bind-mounts them, and invokes docker run per stage.
• run_mur_maap.py — MAAP. Resolves s3:// hrefs and submits jobs. See Running on MAAP for what is and is not wired up yet.

File naming conventions

Regions

Analysis regions are identified by short codes:

• G10 - Global 1km (10km in legacy notation)
• S01 - Southern Ocean 1km
• Additional regions as configured

Date formats

• YYYY - 4-digit year (e.g., 2025)
• DDD - Day of year (001-366)
• YYYYDDD - Combined year and day (e.g., 2025042)

File extensions

• .gds - Grid Data Static (land/ice masks)
• .bip - Binary Ice Points (ice SST observations)
• .bic.gz - Binary Input with Confidence (L2P satellite, compressed)
• .bii - Binary IQUAM Instrument (buoy data)
• .nc - NetCDF (input and final output)

Example: G10_MODISA_2025_042.bic.gz

• Region: G10 (Global 1km)
• Sensor: MODISA (MODIS Aqua)
• Date: 2025, day 42 (February 11)

Where files live

Every path is resolved by the orchestrator from config.json; nothing is hardcoded. The operational layout the pipeline was modelled on looked like this, and is a useful shape to copy:

Input data

/nas2/source/osi-saf/ice/YYYY/    - Ice concentration data
    /nas2/source/podaac/SENSOR/YYYY/  - L2P granules (per sensor)
    /nas2/source/iquam/YYYY/          - iQUAM monthly files

Preprocessed data

/nas2/gds/YYYY/                   - Land/ice masks
    /nas2/bip/YYYY/                   - Ice SST points
    /nas2/bic/SENSOR/YYYY/            - L2P processed (per sensor)
    /nas2/bii/YYYY/                   - iQUAM processed

Output products

/nas2/output/YYYY/                - Final MUR NetCDF files
    /nas2/coef/YYYY/                  - MRVA coefficient files

See configuration for the key that sets each one.

Technology stack

Core dependencies

• Python 3.11+ - Pipeline orchestration
• Docker - Container runtime
• MATLAB R2024b Runtime - Compiled application execution
• uv - Python package and environment management

Data access tools

• podaac-data-subscriber - PO.DAAC L2P data downloads
• wget/curl - OSI-SAF ice data retrieval
• netCDF4 - NetCDF file manipulation

Development tools

• MATLAB R2024b - Code development and compilation
• MATLAB Compiler - Standalone executable generation

Performance characteristics

Processing time (single day, NRT mode)

• Land/Ice: ~2-5 minutes
• L2P MODISA: ~10-20 minutes (200-300 granules)
• L2P MODIST: ~10-20 minutes (200-300 granules)
• L2P AMSR2R: ~5-10 minutes (30-50 granules)
• L2P AVMTBG: ~5-10 minutes (40-80 granules)
• iQUAM: ~5-15 minutes (cached monthly files)
• Total Preprocessing: ~40-80 minutes

MRVA Analysis: ~30-90 minutes (NRT, L0=6); longer for REA (coarser starting scale, L0=2)

Resource requirements

• Memory: 8–16 GB for the preprocessing stages. MRVA is the real constraint — the orchestrator launches it with a 72 GB limit, and it has been observed peaking around 65 GB.
• Disk I/O: Fast storage recommended (SSD preferred)
• Network: Stable connection for PO.DAAC downloads
• Docker: --shm-size=512M minimum for containers

Parallelization

Components can run in parallel:

• Each L2P sensor processes independently
• Land/Ice and iQUAM can run concurrently
• Pipeline orchestrator coordinates sequencing

Migration status

Current state

• ✅ Land/Ice mask generation
• ✅ L2P satellite processing (5 sensors: AMSR2R, MODISA, MODIST, AVMTAG, AVMTBG)
• ✅ iQUAM buoy processing
• ✅ MRVA multi-scale analysis
• ✅ NetCDF4 output generation (full MUR + MUR25 sibling product)
• ✅ Pipeline orchestrator (run_mur_pipeline.py)

All four containers take their inputs as explicit named flags (local path or s3:// href) rather than bind-mounted directories — see the input contract.

Legacy system

The original nrtMRVA.py system used:

• Python wrappers calling MATLAB via subprocess
• Direct MATLAB license dependency
• Monolithic processing script

What containerizing bought

• Licensing: No MATLAB license required in production
• Modularity: Independent container updates
• Testing: Isolated component testing
• Deployment: Simplified production deployment
• Consistency: Identical behavior across environments