iQUAM container

Turns NOAA iQUAM in-situ buoy and ship observations into the daily binary file MRVA anchors against.

Overview

The IQUAM (in-situ SST Quality Monitor) module processes in-situ sea surface temperature observations from buoys, ships, and floats for assimilation into the Multi-scale Ultra-high Resolution (MUR) SST analysis system.

Containerized Deployment: This application is containerized using Docker with a multi-stage build process. See Container setup for deployment instructions, volume mount specifications, and operational guidance.

What This Module Produces

Primary Output: Daily binary files (.bii format) containing quality-controlled SST observations

Processing Steps:

1. Download monthly IQUAM NetCDF files from NOAA (~200-500 MB/month) to an ephemeral working directory — there is no persistent cache; each run downloads fresh (see makedailyiquam.m's own note on this)
2. Filter observations by quality level (keep only quality_level ≥ 5 - highest quality)
3. Convert units (Kelvin → Celsius) and time format (hour + minute → decimal hours)
4. Extract daily subsets from monthly files
5. Reformat to compact binary format (int16, scaled ×100 for 0.01° precision)

Output Characteristics:

• Format: Fortran-compatible binary (.bii)
• Size: ~1-10 MB per day (varies by observation density)
• Content: SST, lat/lon, time, platform type for ~400k-800k observations/day (post-QC)
• Precision: 0.01°C temperature, 0.01° coordinates, 0.01 hour time
• Coverage: Global ocean, all platform types

Purpose: These binary files provide the "ground truth" in-situ measurements that:

• Anchor satellite-derived SST to physical observations
• Fill gaps in regions with poor satellite coverage
• Enable bias detection and correction for satellite sensors
• Constrain the variational analysis solution

In-situ observations provide critical validation and constraints for satellite-derived SST products. These observations serve as "ground truth" measurements that:

• Anchor the analysis to physical observations with known accuracy
• Provide coverage in regions with poor satellite visibility (high cloud cover, high latitudes)
• Help detect and correct systematic biases in satellite sensors
• Contribute to uncertainty quantification and quality assurance

Data Source

Primary Provider: NOAA Center for Satellite Applications and Research (STAR) Product: iQUAM (in-situ SST Quality Monitor) Version 2.10 Data Portal: https://www.star.nesdis.noaa.gov/pub/socd/sst/iquam/v2.10/ Format: NetCDF (GHRSST Level 2i specification) Temporal Coverage: Near real-time + historical archive Update Frequency: Monthly files updated daily

What is iQUAM?

The iQUAM system is NOAA's operational quality monitoring system for in-situ SST measurements. It:

• Aggregates observations from multiple global networks (buoys, ships, floats)
• Applies comprehensive quality control checks (duplicate detection, track validation, spike detection, buddy checks)
• Flags erroneous, noisy, or suspect observations
• Provides quality metadata to enable user-defined filtering
• Maintains consistency across heterogeneous observation platforms

Platform Types

The iQUAM dataset includes observations from multiple platform types, each with different characteristics and error properties:

\*Note: Example counts are from a single day (October 12, 2020 - day 286) documented in platforms.txt. Actual daily counts vary significantly by season, year, network operations, and data transmission success. Current observation counts may differ from these 2020 values.

Code Structure

Active Processing Pipeline (3 files)

1. buoyDataProcessing.m - Per-day processor, compiled entry point
   • Takes one explicit analysis day (year/doy), mode (nrt/rea), and referenceToday from the caller — it does not compute "today" or a processing window itself; the calling orchestrator (run_mur_pipeline.py/run_mur_maap.py) already owns that loop and passes each day explicitly
   • All parameters are individual positional arguments (not a struct)
   • Contains stubs for future REA temporal aggregation
2. makedailyiquam.m - Daily processor
   • Downloads and processes monthly NetCDF files
   • Extracts daily observations
   • Calls writeiquambii() for output
3. writeiquambii.m - Binary file writer
   • Writes .bii format files
   • Accepts configurable output directory

Utility Functions (available but not called in pipeline)

1. readiquambii.m - Read .bii files for inspection/validation
2. refbii2biq.m - REA temporal aggregation (needs refactoring, not yet integrated)

Archived Files

Legacy and obsolete files have been moved to the archive/ directory, which carries its own notes on what each one was for.

Deployment

This application is deployed as a containerized service. See Container setup for complete deployment instructions.

Processing Pipeline

Architecture

The IQUAM processing module operates as a three-stage pipeline:

    flowchart TD
        Start([Start Processing]) --> Stage1

        subgraph Stage1["STAGE 1: PER-DAY PROCESSING (buoyDataProcessing)"]
            direction LR
            S1A[Receive explicit year/doy/mode/<br/>referenceToday from caller]
            S1B[Determine +/-buoyDayRange<br/>rewrite/stability per offset day]
            S1C[Coordinate temporary<br/>working directory]

            S1A --> S1B --> S1C
        end

        subgraph Stage2["STAGE 2: ACQUISITION & QC (makedailyiquam)"]
            direction LR
            S2A[Download monthly<br/>iQUAM NetCDF files]
            S2B[Read and parse<br/>observation data]
            S2C[Apply quality control<br/>filters qual >= 5]
            S2D[Convert units<br/>Kelvin → Celsius]
            S2E[Extract daily subsets<br/>from monthly files]

            S2A --> S2B --> S2C --> S2D --> S2E
        end

        subgraph Stage3["STAGE 3: FORMAT CONVERSION (writeiquambii)"]
            direction LR
            S3A[Scale values to<br/>integer representation]
            S3B[Write Fortran-compatible<br/>binary format .bii]
            S3C["[REA STUB]<br/>Aggregate temporal windows<br/>for analysis ±3 days"]
            S3D["[REA STUB]<br/>Apply platform-specific<br/>error weighting"]
            S3E["[REA STUB]<br/>Generate analysis-ready<br/>observation files .biq"]

            S3A --> S3B
            S3B -.-> S3C -.-> S3D -.-> S3E
        end

        Stage1 --> Stage2 --> Stage2 --> |Calls writeiquambii| Stage3
        Stage3 --> Output[(Current Output:<br/>Global_IQUAM0_YYYY_DDD.bii)]
        Stage3 -.-> |Future REA| Output2[("Future REA Output:<br/>Global_IQUAM0_YYYY_DDD.biq")]

        style Stage1 fill:#e1f5ff
        style Stage2 fill:#fff4e1
        style Stage3 fill:#e8f5e9
        style Output fill:#c8e6c9
        style Output2 fill:#f3e5f5,stroke-dasharray: 5 5
    

Note: Dashed lines indicate REA mode functionality (Stage 3C-E) which is stubbed but not yet implemented. Currently, only NRT mode (.bii file generation) is operational.

Processing Modes

The system operates in two distinct modes based on data maturity:

Near Real-Time (NRT) Mode [CURRENTLY IMPLEMENTED]

• Latency: 1 day behind current date
• Purpose: Provide rapid SST analysis for operational users
• Characteristics:
  • Uses most recent available data (may be incomplete)
  • No reprocessing of previous days (mode='nrt', i.e. realtime true)
  • Faster execution (avoids re-downloads)
  • Outputs: Individual daily .bii files
  • May include provisional/uncorrected observations

Reanalysis (REA) Mode [STUBBED - NOT YET IMPLEMENTED]

• Latency: 4 days behind current date
• Purpose: Generate high-quality, stable SST records with temporal aggregation
• Characteristics:
  • Waits for data consolidation and delayed transmission
  • Applies 2-day stability window before reprocessing
  • Allows corrections from data providers to propagate
  • [Future] Aggregates ±3 day temporal windows (refbii2biq.m)
  • [Future] Applies platform-specific error weighting
  • [Future] Outputs: Aggregated .biq files for analysis
  • Higher data completeness and quality

Who decides the window and mode: the 1-day NRT latency, 4-day REA latency, and 9-day scan window above are the calling orchestrator's decisions (run_mur_pipeline.py/run_mur_maap.py), not buoyDataProcessing.m's. The orchestrator iterates that 9-day window itself and invokes the container once per day, passing that day's year/doy/mode/referenceToday explicitly — buoyDataProcessing.m only ever processes the one day it's told about (plus its own ±buoyDayRange sub-window around that day). REA temporal aggregation functionality is designed but not yet active (see buoyDataProcessing.m:223-247 for stub).

Quality Control Strategy

Multi-Layer Quality Assessment

The iQUAM data undergoes quality control at multiple stages:

1. Source-Level QC (NOAA STAR)

Before data reaches this module, NOAA applies:

• Duplicate Detection: Identifies identical observations from multiple reporting paths
• Track Check: Validates platform position consistency over time
• Geolocation Check: Ensures coordinates are physically plausible
• SST Spike Check: Detects unrealistic temperature jumps
• Buddy Check: Compares observation against 6+ nearby measurements
• Gross Error Probability: Statistical outlier detection

2. Module-Level Filtering (This Code)

Current implementation applies:

Primary Filter: quality_level >= 5

This removes:

• Erroneous observations (quality_level = 1)
• Noisy observations (quality_level = 2)
• Observations without QC metadata (quality_level = 3)
• Low-confidence observations (quality_level = 4)

3. Analysis-Level Weighting (Downstream)

During variational analysis, observations receive platform-specific error weights:

• Ships: σ = 2.0°C (higher uncertainty due to measurement methodology)
• Buoys: σ = 0.5°C (typical for well-calibrated platforms)
• Floats: σ = 0.2°C (high-precision instruments)

These weights influence each observation's contribution to the final SST field.

Quality Flag Encoding

The 16-bit quality flag provides detailed diagnostic information:

Bits 0-1: Overall Quality

• 00 (0) = Normal
• 01 (1) = Erroneous
• 10 (2) = Noisy
• 11 (3) = QC Unavailable

Bits 2-3: Duplicate Status

• 00 = No duplicate found
• 01 = Duplicate kept (this record)
• 10 = Duplicate removed

Bit 4: Track/Geolocation Check

• 0 = Passed (position consistent with platform track)
• 1 = Failed (suspicious position)

Bit 5: SST Spike Check

• 0 = Passed (temperature change within expected range)
• 1 = Failed (unrealistic temperature jump)

Bit 6: Platform ID Validity

• 0 = Valid platform identifier
• 1 = Invalid or unknown identifier

Bit 7: Buddy Check Density

• 0 = 6 or more nearby observations available for validation
• 1 = Fewer than 6 buddies (less confident validation)

Bits 8-15: Probability of Gross Error

• Value = 0-255 (scaled from 0.0 to 1.0)
• Higher values indicate greater likelihood of measurement error

Important: Quality control is only applied to SST measurements. Other variables (wind speed, air temperature, pressure) pass through unfiltered and should be used with caution.

Data Flow and Transformations

Input: Monthly NetCDF Files

Source File: YYYYMM-STAR-L2i_GHRSST-SST-iQuam-GLOBALOCEAN-v02.0-fv01.0.nc
    Size: ~500 MB (monthly, global)
    Variables:
      - time: observation timestamp (seconds since reference)
      - lat: latitude (-90 to 90°)
      - lon: longitude (-180 to 180°)
      - sea_surface_temperature: SST in Kelvin
      - quality_level: composite quality flag (1-5)
      - platform_type: observation platform (1-8)
      - [additional variables not currently used]

Transformation Steps

1. Monthly to Daily Extraction

• Why: MUR analysis runs on a daily cycle
• How: Filter by calendar day from monthly file
• Temporal Window: Extract ±3 days from analysis day (captures temporal continuity)

2. Unit Conversion

• Temperature: Kelvin → Celsius (subtract 273.15)
• Rationale: Analysis algorithms work in Celsius; ice point = 0°C
• Validation: Check SST > -2°C (below freezing point) and < 40°C (maximum realistic value)

3. Time Normalization

• Original: Separate hour, minute fields
• Converted: Decimal hours (hour + minute/60)
• Purpose: Simplifies temporal weighting in analysis (distance from 09:00 UTC analysis time)

4. Spatial Coordinate Handling

• Longitude Convention: -180° to +180° (consistent with MUR grid)
• Scaling: All coordinates scaled by 100 and stored as int16 (precision: 0.01°)
• Reason: Reduces file size by 75% while maintaining adequate precision

Output: Daily Binary Files

Output File: /nas2/iquam/YYYY/Global_IQUAM0_YYYY_DDD.bii
    Size: ~1-10 MB (daily, global)
    Format: Fortran unformatted binary (int16 precision)
    Structure:
      Header:
        - N: number of observations (int32)
        - year: year (int16)
        - day: day of year (int16)
      Data Arrays (N elements each):
        - sst: temperature in Celsius × 100 (int16)
        - lon: longitude × 100 (int16)
        - lat: latitude × 100 (int16)
        - hour: decimal hours × 100 (int16)
        - platform_type: platform code (int8)

File Naming Convention:

• Global: spatial domain (entire Earth)
• IQUAM0: dataset identifier (in-situ observations, version 0)
• YYYY: four-digit year
• DDD: three-digit day of year (001-366)
• .bii: Binary IQUAM Instrument format

Aggregation for Analysis [FUTURE REA MODE]

Current Implementation: Daily .bii files are processed independently with no temporal aggregation.

Planned Implementation: For each analysis day, observations from a 7-day window (±3 days) would be aggregated:

Why a temporal window?

1. Temporal Persistence: SST changes slowly (typical: 0.1-0.5°C/day)
2. Spatial Coverage: Many ocean regions have sparse daily observations
3. Synoptic Analysis: Capture mesoscale features that evolve over days
4. Error Reduction: More observations → better constrained solution

Temporal Weighting (when implemented): Observations farther from analysis time would receive lower weight based on:

• Time difference (decay function)
• Expected SST persistence (varies by location, season)
• Platform measurement uncertainty

Implementation Status: The aggregation functionality (refbii2biq.m) exists but needs refactoring to accept configurable paths. See stub at buoyDataProcessing.m:238-263.

Integration with MUR Analysis

Role in Multi-Scale Variational Analysis (MRVA)

In-situ observations serve multiple purposes in the MUR system:

1. Reference Field Generation

• Purpose: Create initial SST estimate before satellite assimilation
• Why In-Situ First: Independent of satellite biases; direct measurement
• Method: Spatial interpolation of buoy network using coarse-resolution analysis (L=2-7, ~4-250 km)
• Coverage: Sparse but globally distributed; strong constraints near coast/islands

2. Bias Detection and Correction

• Comparison: Satellite observations vs. collocated in-situ measurements
• Detection: Systematic differences indicate sensor calibration drift
• Application: Compute and apply bias corrections to satellite data
• Sensors: Primarily applied to infrared sensors (MODIS, AVHRR) which are more prone to atmospheric contamination

3. Data Assimilation

• Framework: Observations used as constraints in variational analysis cost function
• Weight: Based on platform-specific error characteristics
• Contribution: Greatest impact in data-sparse regions and near coasts

4. Validation and Quality Assurance

• Independent Check: Compare final MUR SST against withheld in-situ observations
• Metrics: RMS error, bias, correlation
• Monitoring: Track analysis performance over time

Temporal Integration

The MRVA algorithm uses a temporal decay function to weight observations:

Weight(t) = W₀ × exp(-Δt² / τ²)

    Where:
      W₀ = base weight (from platform error estimate)
      Δt = time difference from analysis hour (09:00 UTC)
      τ = decay time constant (varies by scale: 12-48 hours)

Scale-Dependent Decay:

• Coarse scales (L=2-4, ~4-16 km): τ = 48 hours (slow SST evolution)
• Medium scales (L=5-8, ~32-256 km): τ = 36-24 hours
• Fine scales (L=9-11, ~512-2048 km): τ = 12-18 hours (fast evolution)

This approach allows the system to:

• Use older observations for large-scale features (stable over days)
• Rely on recent observations for small-scale features (evolve rapidly)
• Gracefully handle gaps in satellite coverage

Configuration and Tuning

Key Parameters

Date Range Control (buoyDataProcessing.m)

The function uses MATLAB's modern arguments block for parameter validation. year, doy, mode, and referenceToday are required (no default) — everything else has a default. Parameters are passed as individual positional arguments (not as a struct), and there is no cacheDir parameter — this module has never had a persistent cache directory.

Parameter List (in order):

buoyDataProcessing(year, doy, mode, referenceToday, ...
                       workDir, logDir, outputDir, ...
                       buoyDayRange, buoyStabilityLatency, ...
                       sourceUrl, enableREA, testing, ...
                       reaAggregationWindow, reaOutputDir)

Note: When calling from command-line or Docker, all parameters must be passed as strings. The function automatically converts numeric parameters from strings.

What this function does not do: compute "today," decide NRT vs. REA, or loop over a multi-day processing window — all of that is the calling orchestrator's job (run_mur_pipeline.py/run_mur_maap.py), which already iterates the full window and calls this function once per day. This function's only date-related work is the ±buoyDayRange sub-window around the one day it's given.

Tuning Guidance:

• Adjust buoyDayRange based on data density and SST variability
• Set buoyStabilityLatency to balance data quality vs. reprocessing frequency
• Set enableREA = 'true' when REA aggregation is implemented

Usage Examples:

% Process day 220 of 2026, NRT mode, using 2026-08-09 as "today"
    buoyDataProcessing('2026', '220', 'nrt', '2026-08-09')

    % Override the working/log/output directories too (still positional —
    % mode/referenceToday can't be skipped since they precede these)
    buoyDataProcessing('2026', '220', 'nrt', '2026-08-09', ...
                       './tmp/work', './logs', '/data/output/iquam')

    % From the compiled executable (command-line) — the container's
    % entrypoint.sh translates named flags into exactly this positional order
    % ./IquamProcessor 2026 220 nrt 2026-08-09 ./tmp/work ./logs /data/output/iquam 3 2

Docker Usage:

The container entrypoint (iquam/bin/entrypoint.sh) is named-args-only — every input is an explicit flag, and there is no positional or zero-argument form:

docker run --rm \
      -v /host/output:/data/output/iquam \
      -v /host/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

--source-url is not exposed as a flag — it stays at buoyDataProcessing.m's own default unless you're modifying the source.

Important Notes:

• All nine flags above are required — there is no "use all defaults" invocation for the container, unlike calling the MATLAB function directly
• run_mur_pipeline.py's run_iquam() and run_mur_maap.py's run_day() already construct these values for you (year/doy from the day being processed, mode from NRT/REA window logic, reference_date from get_reference_today(), buoy_day_range/stability_latency from config.json's iquam.buoy_dayrange/iquam.stable_latency) — you only need to build the command by hand for manual testing

Quality Thresholds (makedailyiquam.m:75)

qual >= 5    % Current: accept only highest quality

Alternative Filters:

% Option 1: Accept Normal + Noisy (more observations, possibly noisier)
    qual >= 3

    % Option 2: iQUAM recommended filter (Normal quality only)
    mod(qual, 4) == 0

    % Option 3: Combine overall quality + buddy check
    (mod(qual, 4) == 0) & (bitand(qual, 128) == 0)

Platform Error Estimates (refbii2biq.m)

Platform Type    Default RMS    Recommended Range
    ---------------------------------------------------
    Ships            2.0°C          1.5 - 3.0°C
    Buoys/Floats     0.5°C          0.3 - 0.7°C
    High-Precision   0.2°C          0.1 - 0.4°C

Tuning Guidance:

• Decrease error for well-maintained networks (gives them more weight)
• Increase error for suspect platforms or high-variability regions
• Validate against independent SST analysis or satellite matchups

Output Directories

Default Configuration (container-friendly relative paths):

./output/iquam/
    └── YYYY/
        ├── Global_IQUAM0_YYYY_001.bii    # Daily binary files
        ├── Global_IQUAM0_YYYY_002.bii
        ├── ...
        └── Global_IQUAM0_YYYY_366.bii

    ./tmp/makebic/
    └── [temporary working files, auto-cleaned at startup — no persistent cache]

    ./logs/
    └── buoy.log                           # Processing log (appended)

There is no cache directory — workDir, outputDir, and logDir are the only directory parameters buoyDataProcessing.m takes.

Deployment: The application runs in a containerized environment. See Container setup for volume mount specifications and configuration options.

Data Quality and Validation

Expected Data Characteristics

Typical Daily Counts:

• Total observations: 500,000 - 900,000 globally
• After QC filtering: 400,000 - 800,000 (70-90% pass rate)
• Spatial distribution: Heavily weighted to Northern Hemisphere, shipping routes, coastal regions

Temporal Variability:

• Higher counts in Northern Hemisphere summer (better transmission conditions)
• Lower counts during major storms (buoy damage/loss)
• Gradual trends due to network expansion/contraction

Known Gaps:

• Southern Ocean (sparse coverage)
• Remote tropical regions (few platforms)
• Polar regions (ice coverage, no platforms)

Quality Assurance Checks

1. Count Monitoring

% Check daily observation count
    if N < 200000
        warning('Low observation count - possible data availability issue')
    end

2. Spatial Coverage

% Check for data in major ocean basins
    basins = {'North Atlantic', 'North Pacific', 'Tropical Pacific', ...};
    for basin in basins
        if count(basin) < threshold
            warning(['Sparse coverage in ' basin])
        end
    end

3. Temperature Range Validation

% Physical validity checks
    assert(all(sst >= -2.0), 'SST below freezing point')
    assert(all(sst <= 40.0), 'SST exceeds maximum realistic value')
    assert(std(sst) > 0.5 & std(sst) < 10, 'Unexpected SST variance')

4. Platform Distribution

% Ensure multiple platform types present
    platform_counts = histcounts(platform_type, 1:9);
    assert(sum(platform_counts > 0) >= 4, 'Insufficient platform diversity')

Operational Considerations

Execution Environment

Software Dependencies:

• MATLAB R2024b (NetCDF reading, binary I/O, orchestration)
• wget (data download)
• Network access to NOAA STAR servers

Computational Resources:

• Memory: ~2 GB per daily process (monthly NetCDF in memory)
• Disk: ~500 MB temporary per day, ~5 MB permanent output per day
• CPU: Minimal (I/O bound, not compute intensive)
• Network: ~50-200 MB download per month (NetCDF files — no cache, downloaded fresh every run that needs them)

Performance Optimization

No Persistent Cache

There is no cache directory or cached .mat file — every invocation downloads the monthly NetCDF file(s) it needs to an ephemeral working directory and discards them afterward. This is a deliberate design choice (see makedailyiquam.m's own comment) to match production behavior and avoid cache-staleness bugs, at the cost of re-downloading on every run.

Conditional Reprocessing

The system avoids unnecessary work by:

• Checking output .bii file existence before processing
• Comparing the file's age against buoyStabilityLatency
• Only reprocessing when rewrite=1 or the existing file is unstable

Parallel Opportunities

Currently sequential, but could parallelize:

• Multiple days can be processed independently
• Different months can be downloaded simultaneously
• Daily extraction and QC could be threaded

Typical Runtime:

• Per day: ~2-5 minutes (download + process; no cache to shortcut a repeat run)
• A full 9-day scan window, one container invocation per day: ~20-45 minutes total, driven by the orchestrator's per-day loop, not by buoyDataProcessing.m itself

Error Handling and Recovery

Common Failure Modes

1. Network Interruption

• Symptom: wget fails, incomplete download
• Recovery: Script retries on next execution (wget resumes partial downloads)
• Prevention: Increase timeout, check network stability

2. Missing Monthly File

• Symptom: NetCDF file not found at source URL
• Cause: NOAA processing delay, server maintenance
• Recovery: Wait 24 hours, data typically appears with delay
• Mitigation: NRT mode can skip and continue; REA mode should retry

3. Corrupted NetCDF

• Symptom: MATLAB netcdf.open() error
• Recovery: Since there's no cache to clear, just re-run with rewrite=1 for that day — the corrupt download will be discarded and re-fetched from sourceUrl
• Command: makedailyiquam(year, doy, 1, outputDir, sourceUrl) with rewrite=1

4. Disk Space Exhaustion

• Symptom: Write errors, incomplete output files
• Prevention: Monitor outputDir and workDir usage (there is no cache directory to also monitor)
• Recovery: Increase quota or adjust mount sizes

5. Zero Observations After QC

• Symptom: Output file has N=0 header
• Cause: All observations failed quality filter
• Recovery: Lower quality threshold temporarily, investigate data quality issue
• Note: Rare but possible during network outages or data provider issues

Logging and Diagnostics

All operations logged to <logDir>/buoy.log (default: ./logs/buoy.log). Each container invocation processes exactly one analysis day, given explicitly by the caller — the log no longer shows a multi-day window being scanned, just that one day's ±buoyDayRange sub-window:

---------- Reference today = 2026-08-09 ----------
      ======== Interim run for Year 2026 Day 218 ========
    Processing year=2026 day=215 rewrite=0
    keeping old ./output/iquam/2026/Global_IQUAM0_2026_215.bii
    Processing year=2026 day=216 rewrite=0
    Processing year=2026 day=217 rewrite=1
    [wget output showing YYYYMM-STAR-L2i_GHRSST-SST-iQuam-GLOBALOCEAN-v02.0-fv01.0.nc download]
    Processing year=2026 day=218 rewrite=1
    Processing year=2026 day=219 rewrite=1
    Processing year=2026 day=220 rewrite=1
    Processing year=2026 day=221 rewrite=1

(This example is one call with --buoy-day-range 3, so days 215-221 around the target day 218 are each evaluated; a day beyond referenceToday would be silently skipped rather than logged as "Processing" or "keeping old.")

Key Log Patterns:

• Interim run - NRT mode (mode=nrt, recent/unstable data, never rewritten)
• Final run - REA mode (mode=rea, older/stable data)
• keeping old - File exists, not reprocessing
• rewrite=1 - Reprocessing this offset day (file missing, or younger than buoyStabilityLatency)
• [REA STUB] - REA aggregation point (not yet implemented)

Key Metrics to Monitor:

• Download success rate
• Observation count trends (check file sizes)
• Processing time per day
• Disk usage growth in outputDir

Future Improvements and Considerations

Potential Enhancements

1. Adaptive Quality Filtering

• Current: Fixed threshold (qual >= 5)
• Proposed: Region-dependent, season-dependent thresholds
• Benefit: Optimize data retention vs. quality tradeoff

2. Platform-Specific Processing

• Current: Uniform processing for all platforms
• Proposed: Specialized handling for high-precision platforms (Argo, IMOS)
• Benefit: Better utilize accuracy of premium observations

3. Real-Time Monitoring Dashboard

• Proposed: Web interface showing:
  • Daily observation counts by platform
  • Spatial coverage maps
  • QC statistics and trends
  • Processing status and errors
• Benefit: Faster detection and response to data issues

4. Alternative Data Sources

• Current: NOAA iQUAM only
• Proposed: Direct feeds from:
  • NDBC (National Data Buoy Center) - lower latency
  • IMOS (Integrated Marine Observing System) - high precision
  • Argo GDAC (Global Data Assembly Center) - float profiles
• Benefit: Increased data volume, reduced latency, enhanced quality

5. Machine Learning QC

• Proposed: Train ML models to detect subtle quality issues:
  • Calibration drift
  • Sensor fouling
  • Position errors
• Benefit: Catch errors missed by traditional QC algorithms

Known Limitations

1. Month Boundary Handling

• Issue: Daily extraction from monthly files at month boundaries (e.g., Jan 1 needs Dec data)
• Current: May miss observations near midnight UTC
• Impact: Minimal (few observations exactly at boundary)
• Solution: Download and process adjacent months at boundaries

2. Data Latency Variability

• Issue: Different platforms have different reporting latencies:
  • Drifting buoys: 1-3 hours
  • Ships: 3-12 hours
  • Delayed-mode Argo: 1-6 months
• Current: Fixed 2-day stability window for all platforms
• Impact: May reprocess unnecessarily or miss delayed corrections
• Solution: Platform-specific latency configuration

3. Error Correlation

• Assumption: Observation errors are independent
• Reality: Systematic biases can affect platform groups (e.g., ship network, regional buoy network)
• Impact: Analysis may underestimate uncertainty in regions dominated by one platform type
• Solution: Implement spatial error correlation in MRVA

4. No Vertical Context

• Current: Surface SST only (buoy/ship hull depth: 0.5-5 meters)
• Missing: Subsurface temperature structure
• Impact: Cannot detect shallow stratification or near-surface gradients
• Solution: Integrate Argo temperature profiles (available but not currently used)

References and Documentation

External Resources

NOAA iQUAM Documentation:

• Product Guide: https://www.star.nesdis.noaa.gov/sod/sst/iquam/
• Quality Flag Specification: https://www.star.nesdis.noaa.gov/sod/sst/iquam/iquam_qc.html
• Version History: https://www.star.nesdis.noaa.gov/sod/sst/iquam/versions.html

GHRSST Specifications:

• Data Specification v2.0: https://www.ghrsst.org/ghrsst-data-services/products/
• In-Situ SST Best Practices: https://www.ghrsst.org/ghrsst-science/sst-definitions/

Platform Networks:

• Global Drifter Program: https://www.aoml.noaa.gov/phod/gdp/
• Tropical Atmosphere Ocean (TAO): https://www.pmel.noaa.gov/tao/
• Argo Float Program: https://argo.ucsd.edu/
• NDBC Buoy Network: https://www.ndbc.noaa.gov/

Citation

If using MUR SST data or this processing system in publications, please cite:

For iQUAM data specifically:

Container setup

Overview

The IQUAM processing module has been containerized using a multi-stage Docker build approach. This deployment strategy compiles the MATLAB application during the build stage (requiring a valid MATLAB license) and creates a lightweight runtime container that only requires the MATLAB Runtime (MCR).

Key Benefits:

• License-free runtime: Only the build process requires a MATLAB license
• Reproducible builds: Consistent execution environment across systems
• Simplified deployment: No MATLAB installation needed on production systems
• Explicit inputs: Every invocation is told exactly which day, mode, and reference date to process — the container never infers "today" or a processing window itself

Architecture

Multi-Stage Build Process

The container build uses two stages:

Stage 1: Builder (requires MATLAB license)

• Base image: mathworks/matlab:r2024b
• Compiles MATLAB code using MATLAB Compiler (mcc)
• Requires network access to JPL license servers
• Produces standalone executable (IquamProcessor)

Stage 2: Runtime (no license required)

• Base image: containers.mathworks.com/matlab-runtime:r2024b
• Contains only the compiled application and MATLAB Runtime
• Significantly smaller than full MATLAB installation
• Can run anywhere without license dependencies

Directory Structure

The container uses a fixed directory structure optimized for volume mounting. There is no cache directory — the module has never persisted downloaded NetCDF data between runs (see makedailyiquam.m's own comment: "no persistent cache needed... matches production behavior and avoids cache staleness issues").

/data/                          # Container working directory
    ├── output/iquam/              # Output .bii files (REQUIRED mount)
    │   └── YYYY/
    │       └── Global_IQUAM0_YYYY_DDD.bii
    ├── logs/                      # Processing logs (REQUIRED mount)
    │   └── buoy.log
    └── /tmp/makebic/              # Temporary workspace (ephemeral, not mounted)

Volume Mounts

Required Mounts

These directories must be mounted from the host system for the container to function correctly:

/tmp/makebic (the working directory) does not need a host mount — it's ephemeral scratch space inside the container, cleaned at the start of each run.

Volume Persistence

Storage Requirements:

• Output: ~5 MB per day, ~1.8 GB per year
• Logs: Grows indefinitely (rotate externally)
• Temporary: ~500 MB peak (ephemeral, can use tmpfs)

Building the Container

Prerequisites

• Docker or compatible container runtime
• Network access to JPL MATLAB license servers during build
• Valid MATLAB and MATLAB Compiler licenses
• network.lic in the mur/ directory (cp network.lic.example network.lic, then edit)
• ~10 GB disk space for build process

Build Command

Use build_module.sh from the mur/ directory:

cd mur
    ./build_module.sh iquam

That builds the mur-matlab-base:r2024b image first if it's missing, verifies network.lic, builds --platform linux/amd64 with the parent mur/ directory as context, and tags the image mur-iquam:latest — the name the pipeline config expects.

Manual Build (Advanced)

Only needed when build_module.sh doesn't fit: a custom tag, an external CI system, or debugging the Dockerfile itself.

IMPORTANT: The Dockerfile references shared utilities from the ../common folder, so the build context must include the parent mur directory. Build from within the iquam directory and set the context to the parent (..). The base image must already exist — build it with ./build_matlab_base.sh first.

# Navigate to the iquam directory
    cd mur/iquam

    # Build for AMD64 (most common Linux servers)
    docker build --platform linux/amd64 -f Dockerfile -t mur-iquam:latest ..

    # Or using Apple's container tools on macOS
    container build --arch amd64 -f Dockerfile -t mur-iquam:latest ..

Note: The .. at the end sets the build context to the parent mur directory, which allows the Dockerfile to access both iquam/ and common/ folders.

Build Process Timeline

• Downloading base images: 2-5 minutes
• Installing dependencies: 1-2 minutes
• MATLAB compilation: 3-8 minutes
• Creating runtime image: 1-2 minutes
• Total: 7-17 minutes (varies by network speed and system)

Build Troubleshooting

Build hangs at compilation step:

Common causes:

• License servers unreachable (firewall/network issues)
• No available licenses (all seats in use)
• VPN required for license access
• Incorrect license server configuration

Compilation timeout (>10 minutes):

The Dockerfile includes a 10-minute timeout for the mcc compilation. If your build consistently times out:

1. Verify license server is responding
2. Check that MATLAB Compiler license is available (not just base MATLAB)
3. Increase timeout in Dockerfile: timeout 600s → timeout 1200s

Running the Container

Basic Usage

The container is named-args-only — it does not infer the date, mode, or window itself. All nine flags below are required; there is no "run with defaults" invocation:

docker run --rm \
      --shm-size=512M \
      -v /local/path/output:/data/output/iquam \
      -v /local/path/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

What One Invocation Does

Given the flags above, the container:

1. Downloads monthly IQUAM NetCDF files from NOAA STAR for the ±buoy-day-range window around the target day (no cache — always freshly downloaded)
2. Extracts and processes daily observations for each offset day in that window
3. Filters observations by quality level (≥5)
4. Writes binary .bii files to /data/output/iquam/YYYY/ for each offset day that needs (re)processing
5. Logs all operations to /data/logs/buoy.log

Who decides what to process: the calling orchestrator (run_mur_pipeline.py/run_mur_maap.py) decides the target day, the NRT/REA mode, and the reference date — normally by iterating a 9-day scan window (1-day NRT latency, 4-day REA latency) and invoking this container once per day in that window. The container itself only ever processes the one day (plus its own ±buoy-day-range sub-window) it's explicitly told about via flags:

• --buoy-day-range: temporal window processed per invocation (±days around the target day)
• --stability-latency: files older than this many days (relative to --reference-date) are not reprocessed unless missing

Scheduled Execution

For operational NRT processing, run the container on a daily schedule. The example below uses date to compute the flag values for "today" — in practice, prefer driving this from run_mur_pipeline.py (which already computes the NRT/REA window and mode) rather than reimplementing that logic in a shell script.

Using cron:

# Add to crontab (runs daily at 12:00 UTC)
    0 12 * * * YEAR=$(date -u -d yesterday +\%Y) DOY=$(date -u -d yesterday +\%j) REF=$(date -u +\%Y-\%m-\%d) && \
      /usr/bin/docker run --rm \
      --shm-size=512M \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year "$YEAR" --doy "$DOY" --mode nrt --reference-date "$REF" \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2 >> /var/log/iquam_cron.log 2>&1

Using systemd timer:

# /etc/systemd/system/iquam-processor.service
    [Unit]
    Description=IQUAM SST Processing
    After=docker.service
    Requires=docker.service

    [Service]
    Type=oneshot
    Environment=YEAR=%Y DOY=%j
    ExecStart=/bin/sh -c '/usr/bin/docker run --rm \
      --shm-size=512M \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year $(date -u +%%Y) --doy $(date -u +%%j) --mode nrt --reference-date $(date -u +%%Y-%%m-%%d) \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2'

    [Install]
    WantedBy=multi-user.target

# /etc/systemd/system/iquam-processor.timer
    [Unit]
    Description=Run IQUAM Processing Daily
    Requires=iquam-processor.service

    [Timer]
    OnCalendar=daily
    Persistent=true
    Unit=iquam-processor.service

    [Install]
    WantedBy=timers.target

Enable and start:

systemctl enable iquam-processor.timer
    systemctl start iquam-processor.timer

Performance Optimization

Required: Shared Memory

MATLAB Runtime requires adequate shared memory. Always include --shm-size=512M:

docker run --rm \
      --shm-size=512M \
      [other options] \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Without this flag, the container may:

• Fail with cryptic MCR errors
• Hang indefinitely
• Crash with segmentation faults

Memory Allocation

For processing large monthly files:

docker run --rm \
      --shm-size=512M \
      --memory="4g" \
      --memory-reservation="2g" \
      --memory-swap="6g" \
      --cpus="2.0" \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Resource Guidelines:

• Minimum RAM: 2 GB
• Recommended RAM: 4 GB
• CPUs: 1-2 (processing is I/O bound, not CPU intensive)

MCR Cache Optimization

This is the MATLAB Runtime's own startup cache (unrelated to IQUAM data — there is no IQUAM data cache). Optimize it the same way as the other containers in this pipeline:

docker run --rm \
      --shm-size=512M \
      -e MCR_CACHE_ROOT=/tmp/mcr_cache \
      -e MCR_CACHE_SIZE=1024M \
      -e MCR_CACHE_VERBOSE=true \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Network Optimization

The container downloads data from NOAA STAR servers on every run (there is no cache to avoid re-downloading). If multiple containers run concurrently or network bandwidth is limited:

1. Stagger scheduled runs rather than launching many containers at once
2. Set download timeouts if using unreliable networks (requires modifying makedailyiquam.m)

Data Management

Log Management

The buoy.log file grows indefinitely. Implement log rotation:

Using logrotate:

# /etc/logrotate.d/iquam
    /data/iquam/logs/buoy.log {
        daily
        rotate 30
        compress
        missingok
        notifempty
        create 0644 root root
    }

Output File Retention

Output .bii files are organized by year:

/data/output/iquam/
    ├── 2023/
    │   ├── Global_IQUAM0_2023_001.bii
    │   ├── Global_IQUAM0_2023_002.bii
    │   └── ...
    └── 2024/
    │   ├── Global_IQUAM0_2024_001.bii
    │   └── ...

Retention recommendations:

• NRT processing: Keep current year + 1 previous year
• Reanalysis: Archive all years (required for consistency)
• Backup: Consider backing up to object storage (S3, etc.)

Monitoring and Health Checks

Container Health Check

Add a health check to ensure the container completed successfully:

# Check exit code of last run
    docker run --rm \
      --shm-size=512M \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

    EXIT_CODE=$?
    if [ $EXIT_CODE -ne 0 ]; then
        echo "IQUAM processing failed with exit code $EXIT_CODE"
        # Send alert (email, Slack, PagerDuty, etc.)
    fi

Log Monitoring

Monitor the log file for errors or unexpected patterns:

# Check for common error patterns
    grep -i "error\|failed\|cannot" /data/iquam/logs/buoy.log | tail -20

    # Check today's processing
    TODAY=$(date +"%Y-%m-%d")
    grep "$TODAY" /data/iquam/logs/buoy.log

    # Check observation counts (should be 400k-800k daily)
    grep "observations" /data/iquam/logs/buoy.log | tail -10

Output Validation

Verify output files were created:

# Check if today's file exists
    YEAR=$(date +"%Y")
    DOY=$(date +"%j")
    OUTPUT_FILE="/data/iquam/output/${YEAR}/Global_IQUAM0_${YEAR}_${DOY}.bii"

    if [ -f "$OUTPUT_FILE" ]; then
        FILE_SIZE=$(stat -f%z "$OUTPUT_FILE" 2>/dev/null || stat -c%s "$OUTPUT_FILE")
        if [ $FILE_SIZE -lt 100000 ]; then
            echo "WARNING: Output file suspiciously small ($FILE_SIZE bytes)"
        else
            echo "Output file created successfully ($FILE_SIZE bytes)"
        fi
    else
        echo "ERROR: Expected output file not found: $OUTPUT_FILE"
    fi

Kubernetes Deployment

CronJob Example

For Kubernetes environments, deploy as a CronJob:

apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: iquam-processor
      namespace: mur-sst
    spec:
      schedule: "0 12 * * *"  # Daily at 12:00 UTC
      successfulJobsHistoryLimit: 3
      failedJobsHistoryLimit: 3
      concurrencyPolicy: Forbid  # Don't run concurrent jobs
      jobTemplate:
        spec:
          template:
            metadata:
              labels:
                app: iquam-processor
            spec:
              restartPolicy: OnFailure
              containers:
              - name: iquam
                image: mur-iquam:latest
                args:
                  - "--year"
                  - "2026"
                  - "--doy"
                  - "220"
                  - "--mode"
                  - "nrt"
                  - "--reference-date"
                  - "2026-08-09"
                  - "--work-dir"
                  - "/tmp/makebic"
                  - "--log-dir"
                  - "/data/logs"
                  - "--output-dir"
                  - "/data/output/iquam"
                  - "--buoy-day-range"
                  - "3"
                  - "--stability-latency"
                  - "2"
                resources:
                  requests:
                    memory: "2Gi"
                    cpu: "1"
                  limits:
                    memory: "4Gi"
                    cpu: "2"
                volumeMounts:
                - name: output
                  mountPath: /data/output/iquam
                - name: logs
                  mountPath: /data/logs
                - name: shm
                  mountPath: /dev/shm
              volumes:
              - name: output
                persistentVolumeClaim:
                  claimName: iquam-output-pvc
              - name: logs
                persistentVolumeClaim:
                  claimName: iquam-logs-pvc
              - name: shm
                emptyDir:
                  medium: Memory
                  sizeLimit: 512Mi

(In practice, the --year/--doy/--mode/--reference-date values need to be computed fresh per run rather than hardcoded as shown here — e.g. by templating this manifest from run_mur_pipeline.py's own window/mode calculation, or by wrapping the CronJob's command in a small script that computes them at container start.)

Persistent Volume Claims

apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: iquam-output-pvc
      namespace: mur-sst
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: iquam-logs-pvc
      namespace: mur-sst
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Docker Compose Deployment

For single-server deployments, use Docker Compose:

# docker-compose.yml
    version: '3.8'

    services:
      iquam-processor:
        image: mur-iquam:latest
        shm_size: 512m
        mem_limit: 4g
        mem_reservation: 2g
        cpus: 2.0
        volumes:
          - iquam-output:/data/output/iquam
          - iquam-logs:/data/logs
        environment:
          - MCR_CACHE_ROOT=/tmp/mcr_cache
          - MCR_CACHE_SIZE=1024M
        command:
          - "--year"
          - "${YEAR}"
          - "--doy"
          - "${DOY}"
          - "--mode"
          - "${MODE}"
          - "--reference-date"
          - "${REFERENCE_DATE}"
          - "--work-dir"
          - "/tmp/makebic"
          - "--log-dir"
          - "/data/logs"
          - "--output-dir"
          - "/data/output/iquam"
          - "--buoy-day-range"
          - "3"
          - "--stability-latency"
          - "2"
        restart: "no"  # Run once, don't restart automatically

    volumes:
      iquam-output:
        driver: local
        driver_opts:
          type: none
          o: bind
          device: /data/iquam/output
      iquam-logs:
        driver: local
        driver_opts:
          type: none
          o: bind
          device: /data/iquam/logs

Run manually (with YEAR/DOY/MODE/REFERENCE_DATE set in the environment or a .env file):

YEAR=2026 DOY=220 MODE=nrt REFERENCE_DATE=2026-08-09 docker-compose up

Security Considerations

Running as Non-Root

To improve security, run the container as a non-root user:

docker run --rm \
      --shm-size=512M \
      --user $(id -u):$(id -g) \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Important: Ensure mounted directories have appropriate permissions for the specified UID/GID.

Read-Only Root Filesystem

For additional security, run with a read-only root filesystem:

docker run --rm \
      --shm-size=512M \
      --read-only \
      --tmpfs /tmp:size=2g \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Network Isolation

If running in a restricted environment, allow outbound connections only to NOAA STAR:

# Using Docker network with egress filtering
    docker network create --driver bridge iquam-net
    docker run --rm \
      --network iquam-net \
      --shm-size=512M \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Troubleshooting

Container Exits Immediately

Symptoms: Container starts and exits with code 0 or 1

Check:

# Run with interactive terminal to see errors
    docker run -it --rm \
      --shm-size=512M \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      --entrypoint /bin/bash \
      mur-iquam:latest

    # Manually run entrypoint to see errors (all 9 flags required)
    /opt/iquam/bin/entrypoint.sh --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Common causes:

• Missing or inaccessible volume mounts
• Insufficient permissions on mounted directories
• MCR initialization failure (check shm-size)
• A required flag omitted, or an unrecognized flag passed — the entrypoint prints a usage message and exits non-zero (positional arguments are not accepted at all)

No Output Files Created

Symptoms: Container completes but no .bii files appear

Check:

# Examine logs
    tail -100 /data/iquam/logs/buoy.log

    # Look for specific error patterns
    grep -i "error\|cannot\|failed" /data/iquam/logs/buoy.log

    # Verify output directory is writable
    docker run --rm \
      -v /data/iquam/output:/data/output/iquam \
      --entrypoint /bin/sh \
      mur-iquam:latest \
      -c "touch /data/output/iquam/test.txt"

Common causes:

• Output directory not writable
• Data download failures (check network/firewall)
• NOAA server unavailable
• --reference-date is in the past relative to the day being processed, so every offset day in the ±buoy-day-range window was skipped as a future date

Memory Errors

Symptoms: Container killed by OOM killer or crashes with memory errors

Solution:

# Increase memory limits
    docker run --rm \
      --shm-size=512M \
      --memory="8g" \
      --memory-swap="16g" \
      -v /data/iquam/output:/data/output/iquam \
      -v /data/iquam/logs:/data/logs \
      mur-iquam:latest \
      --year 2026 --doy 220 --mode nrt --reference-date 2026-08-09 \
      --work-dir /tmp/makebic --log-dir /data/logs --output-dir /data/output/iquam \
      --buoy-day-range 3 --stability-latency 2

Network Download Failures

Symptoms: wget errors in logs

Check:

# Test connectivity to NOAA STAR
    curl -I https://www.star.nesdis.noaa.gov/pub/socd/sst/iquam/v2.10/

    # Check if firewall/proxy is blocking
    docker run --rm --entrypoint wget mur-iquam:latest \
      --spider https://www.star.nesdis.noaa.gov/pub/socd/sst/iquam/v2.10/

Solutions:

• Configure proxy settings if behind corporate firewall
• Add retry logic for transient failures
• Use alternative download method (ftp vs https)

Future Enhancements

REA Mode Support

Currently, REA aggregation (refbii2biq.m) is stubbed, not implemented — passing --mode rea runs the container in REA latency mode (the stability-window/rewrite behavior changes) but does not produce aggregated .biq output yet. Future enhancements will add:

• Temporal aggregation (±3 day windows)
• Platform-specific error weighting
• Output of .biq files for analysis

Enabling REA aggregation (when implemented) will be via buoyDataProcessing.m's enableREA parameter, which is not exposed as a container flag today.

Multi-Architecture Builds

Current build targets linux/amd64 (build_module.sh hard-codes it). Multi-arch support for ARM64 (AWS Graviton, Apple Silicon) would need a manual buildx invocation — note the parent build context, same as every other manual build:

cd mur/iquam
    docker buildx build \
      --platform linux/amd64,linux/arm64 \
      -f Dockerfile \
      -t mur-iquam:latest \
      --push \
      ..

This also requires an ARM64 mur-matlab-base image, which MathWorks does not currently publish.

Configuration Flexibility

Every input this container needs is already an explicit named flag (--year, --doy, --mode, --reference-date, --work-dir, --log-dir, --output-dir, --buoy-day-range, --stability-latency) — there's no remaining case for environment-variable overrides of these values. --source-url remains a MATLAB-side default (buoyDataProcessing.m), not exposed as a flag; that would be a natural next addition if a use case for overriding it arises.

Additional Resources

• IQUAM README - Detailed information about data flow and processing
• NOAA IQUAM Documentation - Upstream data source
• MATLAB Runtime Documentation - MCR details
• Docker Best Practices - Container optimization

Support

For issues related to:

• Container build/deployment: Contact MUR development team
• MATLAB licensing: Contact JPL CAE license administrators
• IQUAM data availability: Check NOAA STAR service status
• MUR SST processing: See main MUR documentation