Daily execution
What a scheduled run actually does, and why the same day gets processed more than once.
Two modes, one pipeline
Every analysis day is processed in one of two modes, chosen by how old it is relative to the run day:
| NRT (“Interim”) | REA (“Final”) | |
|---|---|---|
| Applies to | Recent days, inside the latency window | Older days, once inputs have settled |
| Starting scale | L0=6, from the previous day's coefficients | L0=2, cold start |
| Existing product | Always rewritten | Never regenerated |
| Optimises for | Latency | Completeness |
Starting near-real-time runs from the previous day's L=6 coefficients is what makes them fast: the basin- and gyre-scale structure is already solved and only the finer scales need refitting. A reanalysis run cannot assume that background is correct, so it starts cold.
What each mode does
NRT Mode (Near Real-Time / "Interim")
- When: For days within the most recent
nrtLatencyperiod (default: 1 day from today) - Purpose: Quick preliminary analysis with the most current available data
- Behavior:
- Processes recent days that may have incomplete or unstable source data
- Does NOT overwrite existing MUR products
- Does NOT re-download stable source files
- Logged as "Interim run"
realtime = 1flag passed to MATLAB
REA Mode (Reanalysis / "Final")
- When: For days older than
reaLatency(default: 4 days from today) - Purpose: Final, authoritative analysis with stable, complete data
- Behavior:
- Processes days when all source data should be stable and complete
- WILL overwrite existing MUR products if they exist
- MAY re-download source files if within their
stablatwindow - Logged as "Final run"
realtime = 0flag passed to MATLAB
The processing window
Today (T)
│
├─ T-1 day ──→ nrtLatency (end of NRT window)
│
├─ T-4 days ──→ reaLatency (end of REA window / start of NRT window)
│
└─ T-9 days ──→ scanLatency (start of processing window)
Processing range: Days from (T - scanLatency) to (T - nrtLatency)
Mode determination:
- Days from (T-9) to (T-4): REA mode (Final run)
- Days from (T-3) to (T-1): NRT mode (Interim run)
The run
sequenceDiagram
participant Cron as Scheduled run
participant Orch as run_mur_pipeline.py
participant Cont as Containers
participant Store as Storage
Cron->>Orch: Execute with --config
Orch->>Orch: Resolve run day and<br/>NRT / REA boundary
loop For each analysis day in the window
Orch->>Orch: Mode for this day<br/>NRT if newer than boundary, else REA
rect rgb(200, 220, 255)
Note over Orch,Cont: 1. LAND / ICE MASK
Orch->>Cont: mur-landice, explicit file flags
Cont->>Cont: Fetch OSI-SAF ice<br/>(walk back if unavailable)
Cont->>Store: Masks and ice points<br/>at both grid resolutions
end
rect rgb(220, 255, 200)
Note over Orch,Store: 2. BUOYS (+/- 3 day window)
Orch->>Cont: mur-iquam, year / doy / mode
Cont->>Store: Binary buoy observations
end
rect rgb(255, 240, 200)
Note over Orch,Store: 3. L2P SENSORS (+/- 2 day window)
loop For each sensor, for each day in window
alt Output exists and past stability latency
Orch->>Store: Keep existing file
else Still in flux
Orch->>Cont: mur-l2p with granule manifest
Cont->>Store: Binary observation bundle
end
end
end
rect rgb(255, 220, 220)
Note over Orch,Cont: 4. MRVA ANALYSIS
alt Product exists and REA mode
Orch->>Store: Keep existing product
else Missing, or NRT mode
Orch->>Orch: Build sensor-inputs manifest<br/>resolve prior coefficient
Orch->>Cont: mur-mrva, explicit flags + manifest
Cont->>Store: Coefficients and NetCDF product
end
end
end
rect rgb(235, 235, 235)
Note over Orch,Store: 5. PURGE
Orch->>Store: Drop granules outside the rolling window
end
Orch->>Cron: Exit with per-stage summary
Stage order
Within a day the stages run in dependency order. Land/ice must exist before MRVA; the L2P and iQUAM conversions are independent of each other but both must finish before the analysis starts.
- Land/ice mask — fetch sea ice, build the day's masks at both grid resolutions
- iQUAM — convert the buoy window
- L2P — per sensor, per day in the window, convert granules to a binary bundle
- MRVA — fit the analysis, write coefficients and the NetCDF product
- Purge — drop downloaded granules that have aged out of the rolling window
A stage reporting success is not always the same as a stage using the data you expect. The land/ice stage will substitute an older day's sea ice if the requested day is unavailable upstream, and only warns about it — see the note on reading logs in the quick start.
Mode behaviour side by side
| Aspect | REA Mode (realtime=0) | NRT Mode (realtime=1) |
|---|---|---|
| Age of data | Older than 4 days | 1-3 days old |
| Data stability | Complete and stable | May be incomplete |
| MUR overwrite | No — an existing final product is kept | Yes — always reprocessed |
| Source file rewrite | Yes — if still within stablat | No — never rewritten |
| Purpose | Final authoritative product | Preliminary quick analysis |
| Log label | "Final run" | "Interim run" |
Source data stability windows
Each data source has a stablat (stability latency) parameter that defines when data becomes stable:
- Buoy data (IQUAM): 2 days
- AMSR2R: 2 days
- MODISA: 2 days
- MODIST: 3 days
- AVMTAG/AVMTBG: 2 days
Rewrite logic
For REA mode:
if (today - data_day) < stablat:
rewrite = 1 # Data still unstable, re-download
else:
rewrite = 0 # Data stable, keep existing file
For NRT mode:
rewrite = 0 # Never re-download in NRT mode
A worked example
Today: 2025-12-04 (day 338, year 2025)
Calculated windows
day0(scanLatency): 2025-11-25 (day 329) - start of REAday1(reaLatency): 2025-11-30 (day 334) - end of REA / start of NRTday2(nrtLatency): 2025-12-03 (day 337) - end of NRT
Processing loop
| Date | Mode | Reason | MUR Behavior |
|---|---|---|---|
| 2025-11-25 (day 329) | REA | day ≤ 334 | Overwrite if exists |
| 2025-11-26 (day 330) | REA | day ≤ 334 | Overwrite if exists |
| 2025-11-27 (day 331) | REA | day ≤ 334 | Overwrite if exists |
| 2025-11-28 (day 332) | REA | day ≤ 334 | Overwrite if exists |
| 2025-11-29 (day 333) | REA | day ≤ 334 | Overwrite if exists |
| 2025-11-30 (day 334) | REA | day ≤ 334 | Overwrite if exists |
| 2025-12-01 (day 335) | NRT | day > 334 | Always process |
| 2025-12-02 (day 336) | NRT | day > 334 | Always process |
| 2025-12-03 (day 337) | NRT | day > 334 | Always process |
Each day processes:
- Ice/land masks for that specific day
- Buoy data for ±3 day window (7 days total)
- L2P sensor data for ±2 day window (5 days total)
- MRVA analysis for that specific day
Why the window is nine days
The 9-day scan window serves multiple purposes:
- Data arrives gradually: Some sensors have 2-3 day latency
- Stability verification: Files within
stablatmay be updated - Gap filling: Missing data from previous runs can be filled
- Final consolidation: REA mode ensures complete, stable products
The reference implementation
The containerized pipeline reproduces the scheduling behaviour of
nrtMRVA.py, the operational script it replaces. That script's own
flow and the exact decisions it makes are recorded here, because they are the
specification the container is checked against — the paths and the
MATLAB invocation are historical, the logic is not.
Daily flow
sequenceDiagram
participant Cron as Daily Cron Job
participant Script as nrtMRVA.py
participant MATLAB as MATLAB Runtime
participant Storage as File System
Note over Cron,Storage: Script runs daily at scheduled time
Cron->>Script: Execute nrtMRVA.py
Script->>Script: Calculate date windows<br/>(scanLatency to nrtLatency)
Note over Script: Prepare working directories<br/>Clean /nas6/nrtMRVA<br/>Copy code from /home/tmchin/cyc4<br/>Compile MRVA
loop For each year in range
loop For each day in year range
Script->>Script: Determine mode<br/>if day > (T-reaLatency): NRT<br/>else: REA
Note over Script: Mode determined:<br/>NRT (realtime=1) or REA (realtime=0)
rect rgb(200, 220, 255)
Note over Script,MATLAB: 1. ICE & LANDMASK FILES
Script->>MATLAB: Generate makeice*.m script
MATLAB->>Storage: Create ice/land files<br/>(p01 and p011 resolutions)
MATLAB-->>Script: Complete
end
rect rgb(220, 255, 200)
Note over Script,Storage: 2. BUOY FILES (±3 day window)
loop For each day offset (-3 to +3)
Script->>Script: Check file stability<br/>(buoystablat = 2 days)
alt File exists AND (stable OR NRT mode)
Script->>Storage: Keep existing file
else Need to download
Script->>MATLAB: Generate makeiquam*.m script
MATLAB->>Storage: Download/create IQUAM buoy file
MATLAB-->>Script: Complete
end
end
end
rect rgb(255, 240, 200)
Note over Script,Storage: 3. L2P SENSOR FILES (±2 day window)
loop For each sensor (AMSR2R, MODISA, MODIST, etc.)
loop For each day offset (-2 to +2)
Script->>Script: Check file stability<br/>(sensor-specific stablat)
alt File exists AND stable
Script->>Storage: Keep existing .bic.gz file
else Need to download
Script->>MATLAB: Generate makebiccmd*.m script
MATLAB->>Storage: Download/process L2P to BIC
MATLAB-->>Script: Complete
end
end
end
end
rect rgb(255, 220, 220)
Note over Script,MATLAB: 4. MRVA ANALYSIS
alt MUR file exists AND REA mode
Script->>Storage: KEEP existing MUR product<br/>(Final run - don't reprocess)
else MUR missing OR NRT mode
Script->>MATLAB: Create MRVAcmd.m<br/>realtime flag set
MATLAB->>MATLAB: Run mrva4com.m<br/>(full MRVA processing)
MATLAB->>Storage: Write MUR product<br/>(.nc.bz2 file)
MATLAB-->>Script: Complete (log to MRVA_YYYY_DDD.log)
end
end
end
end
Script->>Cron: Exit (keep working dirs for debugging)
The three decisions
1. Product overwrite
if os.path.exists(MURfile) and (realtime==0):
# REA mode: keep existing final product
f.write('KEEPING OLD %s\n'%MURfile)
else:
# NRT mode OR missing file: run MRVA
# Generate MRVAcmd.m and execute
Important: In REA mode, if the MUR product already exists, it is kept (not overwritten). This prevents reprocessing of finalized data.
2. Source file rewrite
# Determine stability of source file(s):
if datetime.date.today().toordinal() - ordday(d,y) < stablat:
rewrite = 1 # Still unstable
else:
rewrite = 0 # Now stable
if realtime == 1:
rewrite = 0 # Override: never rewrite in NRT mode
3. File existence check
if os.path.exists(bicfile) and (rewrite==0):
f.write('keeping old %s\n'%bicfile)
else:
# Download/generate the file
Logging
The operational script logged every action to nrtMRVA.log:
- Daily execution header with date ranges
- Mode for each day ("Interim run" vs "Final run")
- File operations (kept vs created)
- MATLAB command executions
Individual MRVA runs were logged separately, as MRVA_YYYY_DDD.log.