From the PO.DAAC Cookbook, to access the GitHub version of the notebook, follow this link.

Ocean Satellite and In-situ Comparison in the Cloud

Author: Celia Ou, PO.DAAC

Summary

Here, we compare salinity from the SMAP satellite and Saildrone in-situ measurements. Both datasets are located within the cloud.

Follow along with the Data in Action story:

By the end of this notebook, you will have recreated a similar plot to the one featured in this Data-in-Action story:
https://podaac.jpl.nasa.gov/DataAction-2021-10-05-Monitoring-Changes-in-the-Arctic-Using-Saildrone-SMAP-Satellite-and-Ocean-Models-Data

Shortnames of datasets used here:

SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5: https://podaac.jpl.nasa.gov/dataset/SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5
SAILDRONE_ARCTIC: https://podaac.jpl.nasa.gov/dataset/SAILDRONE_ARCTIC

Requirements

1. Compute environment

This tutorial can only be run in the following environments: - AWS instance running in us-west-2: NASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.

2. Earthdata Login

An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.

3. netrc File

You will need a .netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A .netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata tutorial.

Import Libraries

# To access dataset using Earthaccess
import earthaccess

# To access dataset without Earthaccess
import os
import s3fs
import requests
import glob

# To open dataset
import xarray as xr

# For plotting
import matplotlib.pyplot as plt
import cartopy
import cartopy.crs as ccrs
import cartopy.feature as cfeature
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER

SMAP dataset

Search for and open this dataset as an example of using Earthaccess

auth = earthaccess.login(strategy="netrc")
You're now authenticated with NASA Earthdata Login
Using token with expiration date: 06/18/2023
Using .netrc file for EDL
short_name="SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5"

results = earthaccess.search_data(
    short_name=short_name,
    cloud_hosted=True,
    temporal=("2019-05-01T00:00:00", "2019-10-01T00:00:00"),
    bounding_box=(-170,65,-160,71) # (west, south, east, north)
)
Granules found: 122
ds_sss = xr.open_mfdataset(earthaccess.open(results))
 Opening 122 granules, approx size: 0.0 GB
plot_west = -170
plot_east = -160
plot_south = 60
plot_north = 75

lat_bnds, lon_bnds = [plot_south, plot_north], [plot_west+360, plot_east+360] # Turn the longitudes in (-180,0) to (0,360)
ds_sss_subset_0 = ds_sss.sel(lat=slice(*lat_bnds), lon=slice(*lon_bnds))
ds_sss_subset_0['latitude'] = ds_sss_subset_0.lat
ds_sss_subset_0['longitude'] = ds_sss_subset_0.lon-360
ds_sss_subset = ds_sss_subset_0.swap_dims({'lat':'latitude', 'lon':'longitude'})
ds_sss_subset
<xarray.Dataset>
Dimensions:                 (longitude: 40, latitude: 60, time: 122,
                             uncertainty_components: 9, iceflag_components: 3)
Coordinates:
    lon                     (longitude) float32 190.1 190.4 ... 199.6 199.9
    lat                     (latitude) float32 60.12 60.38 60.62 ... 74.62 74.88
  * time                    (time) datetime64[ns] 2019-04-27T12:00:00 ... 201...
  * latitude                (latitude) float32 60.12 60.38 60.62 ... 74.62 74.88
  * longitude               (longitude) float32 -169.9 -169.6 ... -160.4 -160.1
Dimensions without coordinates: uncertainty_components, iceflag_components
Data variables: (12/19)
    nobs                    (time, latitude, longitude) float64 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    nobs_RF                 (time, latitude, longitude) float64 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    nobs_40km               (time, latitude, longitude) float64 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    sss_smap                (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    sss_smap_RF             (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    sss_smap_unc            (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    ...                      ...
    fland                   (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    gice_est                (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    surtep                  (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    winspd                  (time, latitude, longitude) float32 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    sea_ice_zones           (time, latitude, longitude) int8 dask.array<chunksize=(1, 60, 40), meta=np.ndarray>
    anc_sea_ice_flag        (time, latitude, longitude, iceflag_components) int8 dask.array<chunksize=(1, 60, 40, 3), meta=np.ndarray>
Attributes: (12/65)
    Conventions:                                            CF-1.7, ACDD-1.3
    title:                                                  SMAP ocean surfac...
    version:                                                V5.0 Validated Re...
    summary:                                                The dataset conta...
    acknowledgement:                                        Funded under Subc...
    processing_level:                                       L3
    ...                                                     ...
    Source_of_SMAP_SSS_retrievals:                          T. Meissner, F. W...
    Source_of_ancillary_SST:                                Canada Meteorolog...
    Source_of_ancillary_CCMP_wind_speed:                    Mears, C. et al.,...
    Source_of_ancillary_AMSR2_sea_ice_flag_and_correction:  Meissner, T. and ...
    Source_of_ancillary_land_mask:                          1 km land/water m...
    Source_of_ancillary_reference_SSS_from_HYCOM:           Hybrid Coordinate...
    • time
      PandasIndex
      PandasIndex(DatetimeIndex(['2019-04-27 12:00:00', '2019-04-28 12:00:00',
               '2019-04-29 12:00:00', '2019-04-30 12:00:00',
               '2019-05-01 12:00:00', '2019-05-02 12:00:00',
               '2019-05-03 12:00:00', '2019-05-04 12:00:00',
               '2019-05-05 12:00:00', '2019-05-06 12:00:00',
               ...
               '2019-09-25 12:00:00', '2019-09-26 12:00:00',
               '2019-09-27 12:00:00', '2019-09-28 12:00:00',
               '2019-09-29 12:00:00', '2019-09-30 12:00:00',
               '2019-10-01 12:00:00', '2019-10-02 12:00:00',
               '2019-10-03 12:00:00', '2019-10-04 12:00:00'],
              dtype='datetime64[ns]', name='time', length=122, freq=None))
    • latitude
      PandasIndex
      PandasIndex(Index([60.125, 60.375, 60.625, 60.875, 61.125, 61.375, 61.625, 61.875, 62.125,
       62.375, 62.625, 62.875, 63.125, 63.375, 63.625, 63.875, 64.125, 64.375,
       64.625, 64.875, 65.125, 65.375, 65.625, 65.875, 66.125, 66.375, 66.625,
       66.875, 67.125, 67.375, 67.625, 67.875, 68.125, 68.375, 68.625, 68.875,
       69.125, 69.375, 69.625, 69.875, 70.125, 70.375, 70.625, 70.875, 71.125,
       71.375, 71.625, 71.875, 72.125, 72.375, 72.625, 72.875, 73.125, 73.375,
       73.625, 73.875, 74.125, 74.375, 74.625, 74.875],
      dtype='float32', name='latitude'))
    • longitude
      PandasIndex
      PandasIndex(Index([-169.875, -169.625, -169.375, -169.125, -168.875, -168.625, -168.375,
       -168.125, -167.875, -167.625, -167.375, -167.125, -166.875, -166.625,
       -166.375, -166.125, -165.875, -165.625, -165.375, -165.125, -164.875,
       -164.625, -164.375, -164.125, -163.875, -163.625, -163.375, -163.125,
       -162.875, -162.625, -162.375, -162.125, -161.875, -161.625, -161.375,
       -161.125, -160.875, -160.625, -160.375, -160.125],
      dtype='float32', name='longitude'))
  • Conventions :
    CF-1.7, ACDD-1.3
    title :
    SMAP ocean surface salinity
    version :
    V5.0 Validated Release
    summary :
    The dataset contains the Level 3 8-day running averages of the NASA/RSS Version 5.0 SMAP Salinity Retrieval Algorithm. It includes all necessary ancillary data and the results of all intermediate steps. The data are gridded on a regular 0.25 deg Earth grid. For details see the Release Notes at https://www.remss.com/missions/smap/salinity/.
    acknowledgement :
    Funded under Subcontract No.1664013 between JPL and RSS: Production System for NASA Ocean Salinity Science Team (OSST).
    processing_level :
    L3
    resolution :
    Spatial resolution: approx 70km
    history :
    created by T. Meissner
    date_created :
    2022-03-29 T12:02:30-0700
    date_modified :
    2022-03-29 T12:02:30-0700
    date_issued :
    2022-03-29 T12:02:30-0700
    date_metadata_modified :
    2022-03-29 T12:02:30-0700
    institution :
    Remote Sensing Systems, Santa Rosa, CA, USA
    source :
    RSS SMAP-SSS v5.0 algorithm
    platform :
    SMAP
    instrument :
    SMAP radiometer
    project :
    Production System for NASA Ocean Salinity Science Team (OSST)
    keywords :
    SURFACE SALINITY, SALINITY, SMAP, NASA, RSS
    keywords_vocabulary :
    NASA Global Change Master Directory (GCMD) Science Keywords
    standard_name_vocabulary :
    CF Standard Name Table v78
    license :
    None
    creator_name :
    Thomas Meissner, Remote Sensing Systems
    creator_email :
    meissner@remss.com
    creator_url :
    http://www.remss.com/missions/smap
    publisher_name :
    Thomas Meissner, Frank Wentz, Andrew Manaster, Richard Lindsley, Marty Brewer, Michael Densberger, Remote Sensing Systems
    publisher_email :
    meissner@remss.com
    publisher_url :
    http://www.remss.com/missions/smap
    id :
    10.5067/SMP50-3SPCS
    naming_authority :
    gov.nasa.earthdata
    dataset_citation_authors :
    T. Meissner, F. Wentz, A. Manaster, R. Lindsley, M. Brewer, M. Densberger
    dataset_citation_year :
    2022
    dataset_citation_product :
    Remote Sensing Systems SMAP Level 3 Sea Surface Salinity Standard Mapped Image 8day running
    dataset_citation_version :
    V5.0 Validated Release
    dataset_citation_institution :
    Remote Sensing Systems, Santa Rosa, CA, USA
    dataset_citation_url :
    Available online at www.remss.com/missions/smap
    netCDF_version_id :
    4
    comment :
    Major changes in V5.0: 1. sea-ice flag: based on AMSR-2 surface emissivties and discriminant analysis. 2. sea-ice correction included. 3. formal uncertainty estimates added.
    references :
    1. V5.0 Release Notes at https://www.remss.com/missions/smap/salinity/ 2. Meissner, T.; Wentz, F.J.; Le Vine, D.M. The Salinity Retrieval Algorithms for the NASA Aquarius Version 5 and SMAP Version 3 Releases. Remote Sens. 2018, 10, 1121. https://doi.org/10.3390/rs10071121 3. Meissner, T.; Manaster, A. SMAP Salinity Retrievals near the Sea-Ice Edge Using Multi-Channel AMSR2 Brightness Temperatures. Remote Sens. 2021, 13, 5120. https://doi.org/10.3390/rs13245120
    year_of_observation :
    2019
    center_day_of_observation :
    117
    first_orbit :
    22566
    last_orbit :
    22682
    time_coverage_start :
    2019-04-23T12:00:00Z
    time_coverage_end :
    2019-05-01T12:00:00Z
    time_coverage_duration :
    P8D
    time_coverage_resolution :
    P8D
    cdm_data_type :
    grid
    geospatial_bounds :
    2D
    geospatial_lat_min :
    -90.0
    geospatial_lat_max :
    90.0
    geospatial_lat_resolution :
    0.25
    geospatial_lat_units :
    degrees_north
    geospatial_lon_min :
    0.0
    geospatial_lon_max :
    360.0
    geospatial_lon_resolution :
    0.25
    geospatial_lon_units :
    degrees_east
    geospatial_bounds_vertical_crs :
    EPSG:5831
    geospatial_vertical_min :
    0
    geospatial_vertical_max :
    0
    Source_of_SMAP_SSS_retrievals :
    T. Meissner, F. Wentz, A. Manaster, R. Lindsley, M. Brewer, M. Densberger, Remote Sensing Systems SMAP L2C Sea Surface Salinity, Version 5.0 Validated Release, Remote Sensing Systems, Santa Rosa, CA, USA doi: 10.5067/SMP50-2SOCS www.remss.com/missions/smap.
    Source_of_ancillary_SST :
    Canada Meteorological Center. 2016.GHRSST Level 4 CMC0.1deg Global Foundation Sea Surface Temperature Analysis (GDS version 2). Ver.3.3.doi: 10.5067/GHCMC-4FM03 http://dx.doi.org/10.5067/GHCMC-4FM03.
    Source_of_ancillary_CCMP_wind_speed :
    Mears, C. et al., 2018.Remote Sensing Systems CCMP NRT V2.0 wind speed and direction. Remote Sensing Systems, Santa Rosa, CA.
    Source_of_ancillary_AMSR2_sea_ice_flag_and_correction :
    Meissner, T. and A. Manaster, 2021. SMAP Salinity Retrievals near the Sea-Ice Edge Using Multi-Channel AMSR2 Brightness Temperatures. Remote Sens. 2021, 13, 5120. https://doi.org/10.3390/rs13245120.
    Source_of_ancillary_land_mask :
    1 km land/water mask from OCEAN DISCIPLINE PROCESSING SYSTEM (ODPS) based on World Vector Shoreline (WVS)database and World Data Bank. courtesy of Fred Patt, Goddard Space Flight Center, frederick.s.patt@nasa.gov.
    Source_of_ancillary_reference_SSS_from_HYCOM :
    Hybrid Coordinate Ocean Model, GLBa0.08/expt_90.9, Top layer salinity. Available at www.hycom.org.
    • subset_mean_values = ds_sss_subset.sss_smap.mean(dim = 'time', skipna = True)
subset_mean_values
    <xarray.DataArray 'sss_smap' (latitude: 60, longitude: 40)>
dask.array<mean_agg-aggregate, shape=(60, 40), dtype=float32, chunksize=(60, 40), chunktype=numpy.ndarray>
Coordinates:
    lon        (longitude) float32 190.1 190.4 190.6 190.9 ... 199.4 199.6 199.9
    lat        (latitude) float32 60.12 60.38 60.62 60.88 ... 74.38 74.62 74.88
  * latitude   (latitude) float32 60.12 60.38 60.62 60.88 ... 74.38 74.62 74.88
  * longitude  (longitude) float32 -169.9 -169.6 -169.4 ... -160.6 -160.4 -160.1

    Saildrone dataset

    Accessing this dataset as an example of using s3fs

    s3_cred_endpoint = 'https://archive.podaac.earthdata.nasa.gov/s3credentials'
    def get_temp_creds():
    temp_creds_url = s3_cred_endpoint
    return requests.get(temp_creds_url).json()
    temp_creds_req = get_temp_creds()
#temp_creds_req                      # !!! BEWARE, removing the # on this line will print your temporary S3 credentials.
    fs_s3 = s3fs.S3FileSystem(anon=False, 
                          key=temp_creds_req['accessKeyId'], 
                          secret=temp_creds_req['secretAccessKey'], 
                          token=temp_creds_req['sessionToken'],
                          client_kwargs={'region_name':'us-west-2'})
    bucket = os.path.join('podaac-ops-cumulus-protected/','SAILDRONE_ARCTIC','saildrone-*-1_minutes-*.nc')
sd_files = fs_s3.glob(bucket)
saildrone_files= [fs_s3.open(file) for file in sorted(sd_files)]
len(saildrone_files)
    2
    sd6 = xr.open_dataset(saildrone_files[0])
sd7 = xr.open_dataset(saildrone_files[1])
sd7
    <xarray.Dataset>
Dimensions:                         (trajectory: 1, obs: 215731)
Coordinates:
    latitude                        (trajectory, obs) float64 ...
    longitude                       (trajectory, obs) float64 ...
    time                            (trajectory, obs) datetime64[ns] ...
  * trajectory                      (trajectory) float32 1.037e+03
Dimensions without coordinates: obs
Data variables: (12/79)
    SOG                             (trajectory, obs) float64 ...
    SOG_FILTERED_MEAN               (trajectory, obs) float64 ...
    SOG_FILTERED_STDDEV             (trajectory, obs) float64 ...
    SOG_FILTERED_MAX                (trajectory, obs) float64 ...
    SOG_FILTERED_MIN                (trajectory, obs) float64 ...
    COG                             (trajectory, obs) float64 ...
    ...                              ...
    TEMP_O2_RBR_MEAN                (trajectory, obs) float64 ...
    TEMP_O2_RBR_STDDEV              (trajectory, obs) float64 ...
    CHLOR_WETLABS_MEAN              (trajectory, obs) float64 ...
    CHLOR_WETLABS_STDDEV            (trajectory, obs) float64 ...
    CHLOR_RBR_MEAN                  (trajectory, obs) float64 ...
    CHLOR_RBR_STDDEV                (trajectory, obs) float64 ...
Attributes: (12/45)
    title:                     Arctic NASA MISST 2019 Mission
    summary:                   Saildrone surface observational data for the N...
    ncei_template_version:     NCEI_NetCDF_Trajectory_Template_v2.0
    Conventions:               CF-1.6, ACDD-1.3
    netcdf_version:            4.6.3
    featureType:               trajectory
    ...                        ...
    keywords_vocabulary:       NASA/GCMD
    publisher_name:            Saildrone
    publisher_url:             www.saildrone.com
    publisher_email:           support@saildrone.com
    acknowledgment:            Saildrone. 2019. Saildrone Arctic field campai...
    processing_level:          Level 2

    Plot salinity side-by-side from SMAP and from Saildrone vehicle 1036

    fig = plt.figure(figsize= (16,10))

rows = 1
columns = 2
ax = fig.add_subplot(rows, columns, 1, projection=ccrs.PlateCarree())
ax.add_feature(cartopy.feature.RIVERS)
ax.coastlines()
ax.set_extent([plot_west, plot_east, plot_south, plot_north])
s = subset_mean_values.plot.pcolormesh(add_colorbar = False)
gl = ax.gridlines(color='gray',alpha=0.6,draw_labels=True) 
plt.title('SMAP, May-Oct 2019')
gl.top_labels = False
gl.left_labels = True
gl.right_labels = False
gl.xformatter = LONGITUDE_FORMATTER
gl.yformatter = LATITUDE_FORMATTER
cb = plt.colorbar(s, ax = ax)
cb.set_label('Salinity (PSU)')

ax = fig.add_subplot(rows, columns, 2, projection=ccrs.PlateCarree())
ax.add_feature(cartopy.feature.RIVERS)
ax.coastlines()
ax.set_extent([plot_west, plot_east, plot_south, plot_north])
s6 = plt.scatter(sd6.longitude, sd6.latitude, s = 0.5, c = sd6.SAL_SBE37_MEAN, vmin = 25, vmax = 35, cmap = 'rainbow', transform = ccrs.PlateCarree())
gl = ax.gridlines(color='gray',alpha=0.6,draw_labels=True)
plt.title('Saildrone vehicle 1036, May-Oct 2019')
gl.top_labels = False
gl.left_labels = True
gl.right_labels = False
gl.xformatter = LONGITUDE_FORMATTER
gl.yformatter = LATITUDE_FORMATTER
cb = plt.colorbar(s, ax = ax)
cb.set_label('Salinity (PSU)')
plt.savefig('salinity_comparison_sd1036.png')

    Plot salinity side-by-side from SMAP and from Saildrone vehicle 1037

    fig = plt.figure(figsize= (16,10))

rows = 1
columns = 2
ax = fig.add_subplot(rows, columns, 1, projection=ccrs.PlateCarree())
ax.add_feature(cartopy.feature.RIVERS)
ax.coastlines()
ax.set_extent([plot_west, plot_east, plot_south, plot_north])
s = subset_mean_values.plot.pcolormesh(add_colorbar=False)
gl = ax.gridlines(color='gray',alpha=0.6,draw_labels=True) 
plt.title('SMAP, May-Oct 2019')
gl.top_labels = False
gl.left_labels = True
gl.right_labels = False
gl.xformatter = LONGITUDE_FORMATTER
gl.yformatter = LATITUDE_FORMATTER
cb = plt.colorbar(s, ax = ax)
cb.set_label('Salinity (PSU)')

ax = fig.add_subplot(rows, columns, 2, projection=ccrs.PlateCarree())
ax.add_feature(cartopy.feature.RIVERS)
ax.coastlines()
ax.set_extent([plot_west, plot_east, plot_south, plot_north])
s6 = plt.scatter(sd7.longitude, sd7.latitude, s = 0.5, c = sd7.SAL_SBE37_MEAN, vmin = 25, vmax = 35, cmap = 'rainbow', transform = ccrs.PlateCarree())
gl = ax.gridlines(color='gray',alpha=0.6,draw_labels=True)
plt.title('Saildrone vehicle 1037, May-Oct 2019')
gl.top_labels = False
gl.left_labels = True
gl.right_labels = False
gl.xformatter = LONGITUDE_FORMATTER
gl.yformatter = LATITUDE_FORMATTER
cb = plt.colorbar(s, ax = ax)
cb.set_label('Salinity (PSU)')
plt.savefig('salinity_comparison_sd1037.png')