Daily execution

What a scheduled run actually does, and why the same day gets processed more than once.

Two modes, one pipeline

Every analysis day is processed in one of two modes, chosen by how old it is relative to the run day:

NRT (“Interim”)REA (“Final”)
Applies toRecent days, inside the latency windowOlder days, once inputs have settled
Starting scaleL0=6, from the previous day's coefficientsL0=2, cold start
Existing productAlways rewrittenNever regenerated
Optimises forLatencyCompleteness

Starting near-real-time runs from the previous day's L=6 coefficients is what makes them fast: the basin- and gyre-scale structure is already solved and only the finer scales need refitting. A reanalysis run cannot assume that background is correct, so it starts cold.

What each mode does

NRT Mode (Near Real-Time / "Interim")

  • When: For days within the most recent nrtLatency period (default: 1 day from today)
  • Purpose: Quick preliminary analysis with the most current available data
  • Behavior:
    • Processes recent days that may have incomplete or unstable source data
    • Does NOT overwrite existing MUR products
    • Does NOT re-download stable source files
    • Logged as "Interim run"
    • realtime = 1 flag passed to MATLAB

REA Mode (Reanalysis / "Final")

  • When: For days older than reaLatency (default: 4 days from today)
  • Purpose: Final, authoritative analysis with stable, complete data
  • Behavior:
    • Processes days when all source data should be stable and complete
    • WILL overwrite existing MUR products if they exist
    • MAY re-download source files if within their stablat window
    • Logged as "Final run"
    • realtime = 0 flag passed to MATLAB

The processing window

Today (T)
    │
    ├─ T-1 day  ──→ nrtLatency (end of NRT window)
    │
    ├─ T-4 days ──→ reaLatency (end of REA window / start of NRT window)
    │
    └─ T-9 days ──→ scanLatency (start of processing window)

Processing range: Days from (T - scanLatency) to (T - nrtLatency)

Mode determination:

  • Days from (T-9) to (T-4): REA mode (Final run)
  • Days from (T-3) to (T-1): NRT mode (Interim run)

The run

    sequenceDiagram
    participant Cron as Scheduled run
    participant Orch as run_mur_pipeline.py
    participant Cont as Containers
    participant Store as Storage

    Cron->>Orch: Execute with --config

    Orch->>Orch: Resolve run day and<br/>NRT / REA boundary

    loop For each analysis day in the window

        Orch->>Orch: Mode for this day<br/>NRT if newer than boundary, else REA

        rect rgb(200, 220, 255)
            Note over Orch,Cont: 1. LAND / ICE MASK
            Orch->>Cont: mur-landice, explicit file flags
            Cont->>Cont: Fetch OSI-SAF ice<br/>(walk back if unavailable)
            Cont->>Store: Masks and ice points<br/>at both grid resolutions
        end

        rect rgb(220, 255, 200)
            Note over Orch,Store: 2. BUOYS (+/- 3 day window)
            Orch->>Cont: mur-iquam, year / doy / mode
            Cont->>Store: Binary buoy observations
        end

        rect rgb(255, 240, 200)
            Note over Orch,Store: 3. L2P SENSORS (+/- 2 day window)
            loop For each sensor, for each day in window
                alt Output exists and past stability latency
                    Orch->>Store: Keep existing file
                else Still in flux
                    Orch->>Cont: mur-l2p with granule manifest
                    Cont->>Store: Binary observation bundle
                end
            end
        end

        rect rgb(255, 220, 220)
            Note over Orch,Cont: 4. MRVA ANALYSIS
            alt Product exists and REA mode
                Orch->>Store: Keep existing product
            else Missing, or NRT mode
                Orch->>Orch: Build sensor-inputs manifest<br/>resolve prior coefficient
                Orch->>Cont: mur-mrva, explicit flags + manifest
                Cont->>Store: Coefficients and NetCDF product
            end
        end
    end

    rect rgb(235, 235, 235)
        Note over Orch,Store: 5. PURGE
        Orch->>Store: Drop granules outside the rolling window
    end

    Orch->>Cron: Exit with per-stage summary
    
A scheduled run walks a window of days, regenerating only the inputs that are still in flux, then runs the analysis for each day.

Stage order

Within a day the stages run in dependency order. Land/ice must exist before MRVA; the L2P and iQUAM conversions are independent of each other but both must finish before the analysis starts.

  1. Land/ice mask — fetch sea ice, build the day's masks at both grid resolutions
  2. iQUAM — convert the buoy window
  3. L2P — per sensor, per day in the window, convert granules to a binary bundle
  4. MRVA — fit the analysis, write coefficients and the NetCDF product
  5. Purge — drop downloaded granules that have aged out of the rolling window

A stage reporting success is not always the same as a stage using the data you expect. The land/ice stage will substitute an older day's sea ice if the requested day is unavailable upstream, and only warns about it — see the note on reading logs in the quick start.

Mode behaviour side by side

AspectREA Mode (realtime=0)NRT Mode (realtime=1)
Age of dataOlder than 4 days1-3 days old
Data stabilityComplete and stableMay be incomplete
MUR overwriteNo — an existing final product is keptYes — always reprocessed
Source file rewriteYes — if still within stablatNo — never rewritten
PurposeFinal authoritative productPreliminary quick analysis
Log label"Final run""Interim run"

Source data stability windows

Each data source has a stablat (stability latency) parameter that defines when data becomes stable:

  • Buoy data (IQUAM): 2 days
  • AMSR2R: 2 days
  • MODISA: 2 days
  • MODIST: 3 days
  • AVMTAG/AVMTBG: 2 days

Rewrite logic

For REA mode:

if (today - data_day) < stablat:
        rewrite = 1  # Data still unstable, re-download
    else:
        rewrite = 0  # Data stable, keep existing file

For NRT mode:

rewrite = 0  # Never re-download in NRT mode

A worked example

Today: 2025-12-04 (day 338, year 2025)

Calculated windows

  • day0 (scanLatency): 2025-11-25 (day 329) - start of REA
  • day1 (reaLatency): 2025-11-30 (day 334) - end of REA / start of NRT
  • day2 (nrtLatency): 2025-12-03 (day 337) - end of NRT

Processing loop

DateModeReasonMUR Behavior
2025-11-25 (day 329)REAday ≤ 334Overwrite if exists
2025-11-26 (day 330)REAday ≤ 334Overwrite if exists
2025-11-27 (day 331)REAday ≤ 334Overwrite if exists
2025-11-28 (day 332)REAday ≤ 334Overwrite if exists
2025-11-29 (day 333)REAday ≤ 334Overwrite if exists
2025-11-30 (day 334)REAday ≤ 334Overwrite if exists
2025-12-01 (day 335)NRTday > 334Always process
2025-12-02 (day 336)NRTday > 334Always process
2025-12-03 (day 337)NRTday > 334Always process

Each day processes:

  • Ice/land masks for that specific day
  • Buoy data for ±3 day window (7 days total)
  • L2P sensor data for ±2 day window (5 days total)
  • MRVA analysis for that specific day

Why the window is nine days

The 9-day scan window serves multiple purposes:

  1. Data arrives gradually: Some sensors have 2-3 day latency
  2. Stability verification: Files within stablat may be updated
  3. Gap filling: Missing data from previous runs can be filled
  4. Final consolidation: REA mode ensures complete, stable products

The reference implementation

The containerized pipeline reproduces the scheduling behaviour of nrtMRVA.py, the operational script it replaces. That script's own flow and the exact decisions it makes are recorded here, because they are the specification the container is checked against — the paths and the MATLAB invocation are historical, the logic is not.

Daily flow

    sequenceDiagram
        participant Cron as Daily Cron Job
        participant Script as nrtMRVA.py
        participant MATLAB as MATLAB Runtime
        participant Storage as File System

        Note over Cron,Storage: Script runs daily at scheduled time

        Cron->>Script: Execute nrtMRVA.py

        Script->>Script: Calculate date windows<br/>(scanLatency to nrtLatency)

        Note over Script: Prepare working directories<br/>Clean /nas6/nrtMRVA<br/>Copy code from /home/tmchin/cyc4<br/>Compile MRVA

        loop For each year in range
            loop For each day in year range

                Script->>Script: Determine mode<br/>if day > (T-reaLatency): NRT<br/>else: REA

                Note over Script: Mode determined:<br/>NRT (realtime=1) or REA (realtime=0)

                rect rgb(200, 220, 255)
                    Note over Script,MATLAB: 1. ICE & LANDMASK FILES
                    Script->>MATLAB: Generate makeice*.m script
                    MATLAB->>Storage: Create ice/land files<br/>(p01 and p011 resolutions)
                    MATLAB-->>Script: Complete
                end

                rect rgb(220, 255, 200)
                    Note over Script,Storage: 2. BUOY FILES (±3 day window)
                    loop For each day offset (-3 to +3)
                        Script->>Script: Check file stability<br/>(buoystablat = 2 days)

                        alt File exists AND (stable OR NRT mode)
                            Script->>Storage: Keep existing file
                        else Need to download
                            Script->>MATLAB: Generate makeiquam*.m script
                            MATLAB->>Storage: Download/create IQUAM buoy file
                            MATLAB-->>Script: Complete
                        end
                    end
                end

                rect rgb(255, 240, 200)
                    Note over Script,Storage: 3. L2P SENSOR FILES (±2 day window)
                    loop For each sensor (AMSR2R, MODISA, MODIST, etc.)
                        loop For each day offset (-2 to +2)
                            Script->>Script: Check file stability<br/>(sensor-specific stablat)

                            alt File exists AND stable
                                Script->>Storage: Keep existing .bic.gz file
                            else Need to download
                                Script->>MATLAB: Generate makebiccmd*.m script
                                MATLAB->>Storage: Download/process L2P to BIC
                                MATLAB-->>Script: Complete
                            end
                        end
                    end
                end

                rect rgb(255, 220, 220)
                    Note over Script,MATLAB: 4. MRVA ANALYSIS

                    alt MUR file exists AND REA mode
                        Script->>Storage: KEEP existing MUR product<br/>(Final run - don't reprocess)
                    else MUR missing OR NRT mode
                        Script->>MATLAB: Create MRVAcmd.m<br/>realtime flag set
                        MATLAB->>MATLAB: Run mrva4com.m<br/>(full MRVA processing)
                        MATLAB->>Storage: Write MUR product<br/>(.nc.bz2 file)
                        MATLAB-->>Script: Complete (log to MRVA_YYYY_DDD.log)
                    end
                end

            end
        end

        Script->>Cron: Exit (keep working dirs for debugging)
    

The three decisions

1. Product overwrite

if os.path.exists(MURfile) and (realtime==0):
        # REA mode: keep existing final product
        f.write('KEEPING OLD %s\n'%MURfile)
    else:
        # NRT mode OR missing file: run MRVA
        # Generate MRVAcmd.m and execute

Important: In REA mode, if the MUR product already exists, it is kept (not overwritten). This prevents reprocessing of finalized data.

2. Source file rewrite

# Determine stability of source file(s):
    if datetime.date.today().toordinal() - ordday(d,y) < stablat:
        rewrite = 1  # Still unstable
    else:
        rewrite = 0  # Now stable

    if realtime == 1:
        rewrite = 0  # Override: never rewrite in NRT mode

3. File existence check

if os.path.exists(bicfile) and (rewrite==0):
        f.write('keeping old %s\n'%bicfile)
    else:
        # Download/generate the file

Logging

The operational script logged every action to nrtMRVA.log:

  • Daily execution header with date ranges
  • Mode for each day ("Interim run" vs "Final run")
  • File operations (kept vs created)
  • MATLAB command executions

Individual MRVA runs were logged separately, as MRVA_YYYY_DDD.log.