Zarr in Practice

This notebook demonstrates how to create, explore and modify a Zarr store.

These concepts are explored in more detail in the official Zarr User Guide.

It also shows the use of public Zarr stores for geospatial data.

Environment

The packages needed for this notebook can be installed with conda or mamba. Using the environment.yml from this folder run:

conda env create -f environment.yml

or

mamba env create -f environment.yml

Finally, you may activate and select the kernel in the notebook (running in Jupyter)

conda activate coguide-zarr

The notebook has been tested to work with the listed Conda environment.

How to create a Zarr store

import numpy as np
import xarray as xr
import zarr

First let’s clean up in case we are running this notebook multiple times.

!rm -rf test.zarr
!rm -rf example.zarr

Here we will create a simple local Zarr store.

store = zarr.storage.LocalStore("test.zarr")

The Zarr store does not write anything to disk until there is data in it.

!tree test.zarr

test.zarr  [error opening dir]

0 directories, 0 files

Now let’s create an array. Just calling zarr.create_array writes the array to disk.

arr = zarr.create_array(store=store, data=np.arange(10))

!tree test.zarr

test.zarr

├── c

│   └── 0

└── zarr.json



1 directory, 2 files

We can open the metadata about this Zarr array, which gives us some interesting information. The dataset has a shape of 10 and chunk_shape of 10, so we know all the data was stored in 1 chunk. From the codecs we can see tell that the data was compressed with the zstd compressor.

Notice that the zarr_format (this is the Zarr format version) and node_type are also specified in the metadata.

!cat test.zarr/zarr.json

{
  "shape": [
    10
  ],
  "data_type": "int64",
  "chunk_grid": {
    "name": "regular",
    "configuration": {
      "chunk_shape": [
        10
      ]
    }
  },
  "chunk_key_encoding": {
    "name": "default",
    "configuration": {
      "separator": "/"
    }
  },
  "fill_value": 0,
  "codecs": [
    {
      "name": "bytes",
      "configuration": {
        "endian": "little"
      }
    },
    {
      "name": "zstd",
      "configuration": {
        "level": 0,
        "checksum": false
      }
    }
  ],
  "attributes": {},
  "zarr_format": 3,
  "node_type": "array",
  "storage_transformers": []
}

This was a pretty basic example. Let’s explore the other things we might want to do when creating Zarr.

How to create a group

In this example we’ll use a MemoryStore. This is an in-memory representation of a Zarr store that has no physical representation on-disk. Similar to how numpy is an in-memory representation of an array.

store = zarr.storage.MemoryStore()
root = zarr.create_group(store)
group1 = root.create_group('group1')
group2 = root.create_group('group2')
z1 = group1.create_array(name='array_in_group', shape=(100,100), chunks=(10,10), dtype='i4')
z2 = group2.create_array(name='array_in_group', shape=(1000,1000), chunks=(10,10), dtype='i4')
root.tree()

/
├── group1
│   └── array_in_group (100, 100) int32
└── group2
    └── array_in_group (1000, 1000) int32

How to Examine and Modify the Chunk Shape

If your data is sufficiently large, Zarr will chose a chunksize for you.

store = zarr.storage.MemoryStore()
zarr_no_chunks = zarr.create_array(store, name="no_chunks", data=np.arange(100))
zarr_no_chunks.chunks, zarr_no_chunks.shape

((100,), (100,))

zarr_with_chunks = zarr.create_array(store, name="with_chunks", data=np.arange(10000000))
zarr_with_chunks.chunks, zarr_with_chunks.shape

((156250,), (10000000,))

For zarr_with_chunks we see the chunks are smaller than the shape, so we know the data has been chunked. Other ways to examine the chunk structure are zarr.info_complete() and zarr.cdata_shape.

?zarr_no_chunks.cdata_shape

Type:        property

String form: <property object at 0x71251d4a9670>

Docstring:   The shape of the chunk grid for this array.

zarr_no_chunks.cdata_shape, zarr_with_chunks.cdata_shape

((1,), (64,))

The zarr store with chunks has 64 chunks. The number of chunks multiplied by the chunk size equals the length of the whole array.

zarr_with_chunks.cdata_shape[0] * zarr_with_chunks.chunks[0] == zarr_with_chunks.shape[0]

True

What’s the storage size of these chunks?

First we need to know how to access a particular chunk via its prefix. Think of a prefix as the relative path for every item in the MemoryStore.

[p async for p in store.list()][:10]

['no_chunks/zarr.json',
 'zarr.json',
 'no_chunks/c/0',
 'with_chunks/zarr.json',
 'with_chunks/c/1',
 'with_chunks/c/6',
 'with_chunks/c/7',
 'with_chunks/c/0',
 'with_chunks/c/9',
 'with_chunks/c/2']

Now we can use these prefixes to get the size of the first chunk in bytes. Note that this is the compressed size. We will talk more about compression in the next section.

await store.getsize_prefix("with_chunks/c/0")

195984

If we want the chunksize to be bigger, then we can specify the chunks rather than using the default.

zarr_with_big_chunks = zarr.create_array(store, name="with_big_chunks", data=np.arange(10000000), chunks=(500000))

zarr_with_big_chunks.chunks, zarr_with_big_chunks.shape, zarr_with_big_chunks.cdata_shape

((500000,), (10000000,), (20,))

This Zarr store has 10 million values, stored in 20 chunks of 500,000 data values.

await store.getsize_prefix("with_big_chunks/c/0")

544926

In the real world, you will likely want to deal in Zarr chunks of 1MB uncompressed or greater, and when dealing with remote storage options where data is read over a network and the number of requests should be minimized.

Exploring and Modifying Data Compression

Continuing with data from the example above, we can use zarr.info_complete() or zarr.compressors to learn more about how Zarr has compressed the data for us.

zarr_with_chunks.info_complete()

Type               : Array
Zarr format        : 3
Data type          : Int64(endianness='little')
Fill value         : 0
Shape              : (10000000,)
Chunk shape        : (156250,)
Order              : C
Read-only          : False
Store type         : MemoryStore
Filters            : ()
Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
Compressors        : (ZstdCodec(level=0, checksum=False),)
No. bytes          : 80000000 (76.3M)
No. bytes stored   : 10192187 (9.7M)
Storage ratio      : 7.8
Chunks Initialized : 64

From the information above, we can see that the default compression, zstd, is a good choice for n-dimensional data.

The Blosc compressor is a meta compressor, which means it implements multiple different internal compressors. Let’s try it with lz4 compression (if available – this was the default compression for zarr-python<3).

zarr_blosc_compressed = zarr.create_array(
    store,
    name="blosc_compressed",
    data=np.arange(10000000),
    compressors=zarr.codecs.BloscCodec(cname='lz4', clevel=5, shuffle="shuffle", blocksize=0),
)
zarr_blosc_compressed.info_complete()

Type               : Array
Zarr format        : 3
Data type          : Int64(endianness='little')
Fill value         : 0
Shape              : (10000000,)
Chunk shape        : (156250,)
Order              : C
Read-only          : False
Store type         : MemoryStore
Filters            : ()
Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
Compressors        : (BloscCodec(typesize=8, cname=<BloscCname.lz4: 'lz4'>, clevel=5, shuffle=<BloscShuffle.shuffle: 'shuffle'>, blocksize=0),)
No. bytes          : 80000000 (76.3M)
No. bytes stored   : 514573 (502.5K)
Storage ratio      : 155.5
Chunks Initialized : 64

We can see, from the storage ratio above, that compression has made our data 155 times smaller 😱 .

You can set compressors=None when creating a Zarr array to turn off all compression, but I’m not sure why you would do that.

zarr_uncompressed = zarr.create_array(
    store,
    name="not_compressed",
    data=np.arange(10000000),
    compressors=None,
)
zarr_uncompressed.info_complete()

Type               : Array
Zarr format        : 3
Data type          : Int64(endianness='little')
Fill value         : 0
Shape              : (10000000,)
Chunk shape        : (156250,)
Order              : C
Read-only          : False
Store type         : MemoryStore
Filters            : ()
Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
Compressors        : ()
No. bytes          : 80000000 (76.3M)
No. bytes stored   : 80000511 (76.3M)
Storage ratio      : 1.0
Chunks Initialized : 64

Let’s see what happens when we use a different compression method. We can checkout a full list of registerd Zarr codecs here: https://zarr.readthedocs.io/en/stable/api/zarr/codecs/index.html.

zarr_gzip_compressed = zarr.create_array(
    store,
    name="gzip_compressed",
    data=np.arange(10000000),
    compressors=zarr.codecs.GzipCodec()
)
zarr_gzip_compressed.info_complete()

Type               : Array
Zarr format        : 3
Data type          : Int64(endianness='little')
Fill value         : 0
Shape              : (10000000,)
Chunk shape        : (156250,)
Order              : C
Read-only          : False
Store type         : MemoryStore
Filters            : ()
Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
Compressors        : (GzipCodec(level=5),)
No. bytes          : 80000000 (76.3M)
No. bytes stored   : 15141856 (14.4M)
Storage ratio      : 5.3
Chunks Initialized : 64

In this case, the storage ratio is 5.3 - so not as good! How to chose a compression algorithm is a topic for future investigation. Read more in the Compressors section of the Zarr User Guide.

Consolidating metadata

It’s important to consolidate metadata to minimize requests. Each group and array has a metadata file and when opening the Zarr store every metadata file has be to read. This is the case even if you open the file lazily (for instance using dask within xarray). In order to limit requests to read the whole tree of metadata files, Zarr provides the ability to consolidate metadata at the root of the store.

So far we have only been dealing in single array Zarr data stores. In this next example, we will create a Zarr store with multiple arrays and then consolidate metadata. The speed up with local storage is insignificant, but becomes significant when dealing in remote storage options, which we will see in the following example on accessing cloud storage.

store = zarr.storage.LocalStore("example.zarr")
root = zarr.create_group(store, path="root")

# Let's create many groups and many arrays
num_groups, num_arrays_per_group = 10, 10
for i in range(num_groups):
    group = root.create_group(f'group-{i}')
    for j in range(num_arrays_per_group):
        group.create_array(f'array-{j}', shape=(1000,1000), dtype='i4')

We don’t expect the consolidated metadata to be populated.

!cat {store.root}/zarr.json

{
  "attributes": {},
  "zarr_format": 3,
  "consolidated_metadata": null,
  "node_type": "group"
}

zarr.consolidate_metadata(store)

/home/jsignell/zarr-python/src/zarr/api/asynchronous.py:227: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(

<Group file://example.zarr>

There is a warning about consolidated not being an official part of the specification, but don’t worry about that. It’s normal that sometimes implementation outpaces specification.

Now if we check the root metadata file again we see that the consolidated metadata is populated.

!head -n100 {store.root}/zarr.json

{
  "attributes": {},
  "zarr_format": 3,
  "consolidated_metadata": {
    "kind": "inline",
    "must_understand": false,
    "metadata": {
      "root": {
        "attributes": {},
        "zarr_format": 3,
        "consolidated_metadata": {
          "kind": "inline",
          "must_understand": false,
          "metadata": {}
        },
        "node_type": "group"
      },
      "root/group-1": {
        "attributes": {},
        "zarr_format": 3,
        "consolidated_metadata": {
          "kind": "inline",
          "must_understand": false,
          "metadata": {}
        },
        "node_type": "group"
      },
      "root/group-1/array-4": {
        "shape": [
          1000,
          1000
        ],
        "data_type": "int32",
        "chunk_grid": {
          "name": "regular",
          "configuration": {
            "chunk_shape": [
              250,
              500
            ]
          }
        },
        "chunk_key_encoding": {
          "name": "default",
          "configuration": {
            "separator": "/"
          }
        },
        "fill_value": 0,
        "codecs": [
          {
            "name": "bytes",
            "configuration": {
              "endian": "little"
            }
          },
          {
            "name": "zstd",
            "configuration": {
              "level": 0,
              "checksum": false
            }
          }
        ],
        "attributes": {},
        "zarr_format": 3,
        "node_type": "array",
        "storage_transformers": []
      },
      "root/group-1/array-1": {
        "shape": [
          1000,
          1000
        ],
        "data_type": "int32",
        "chunk_grid": {
          "name": "regular",
          "configuration": {
            "chunk_shape": [
              250,
              500
            ]
          }
        },
        "chunk_key_encoding": {
          "name": "default",
          "configuration": {
            "separator": "/"
          }
        },
        "fill_value": 0,
        "codecs": [
          {
            "name": "bytes",
            "configuration": {
              "endian": "little"
            }
          },
          {
            "name": "zstd",

When opening Zarr stores that have consolidated metadata, use use_consolidated=True.

zarr.open(store, use_consolidated=True)

<Group file://example.zarr>

Example of Cloud-Optimized Access for this Format

Fortunately, there are many publicly accessible cloud archives of Zarr data.

Zarr provides storage backends for many different cloud providers as well as : Zarr Storage guide.

Here are a few we are aware of:

• Zarr data in Microsoft’s Planetary Computer
• Zarr data from Google
• Amazon Sustainability Data Initiative available from Registry of Open Data on AWS - Enter “Zarr” in the Search input box.

Once you have identified a remote Zarr it should be straightforward to open it using xarray.

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'

Tip: You might have seen chunks={} instead of chunks="auto". The difference is chunks={} means use exactly the same chunks as the Zarr store whereas chunks="auto" allows chunks to be combined into meta-chunks if that makes the tasks a better size for dask.

ds = xr.open_dataset(store, engine='zarr', chunks="auto", consolidated=True, zarr_format=2)
ds

<xarray.Dataset> Size: 2GB
Dimensions:      (latitude: 180, nv: 2, longitude: 360, time: 9226)
Coordinates:
    lat_bounds   (latitude, nv) float32 1kB dask.array<chunksize=(180, 2), meta=np.ndarray>
  * latitude     (latitude) float32 720B -90.0 -89.0 -88.0 ... 87.0 88.0 89.0
    lon_bounds   (longitude, nv) float32 3kB dask.array<chunksize=(360, 2), meta=np.ndarray>
  * longitude    (longitude) float32 1kB 0.0 1.0 2.0 3.0 ... 357.0 358.0 359.0
  * time         (time) datetime64[ns] 74kB 1996-10-01 1996-10-02 ... 2021-12-31
    time_bounds  (time, nv) datetime64[ns] 148kB dask.array<chunksize=(9226, 2), meta=np.ndarray>
Dimensions without coordinates: nv
Data variables:
    precip       (time, latitude, longitude) float32 2GB dask.array<chunksize=(400, 180, 360), meta=np.ndarray>
Attributes: (12/45)
    Conventions:                CF-1.6, ACDD 1.3
    Metadata_Conventions:       CF-1.6, Unidata Dataset Discovery v1.0, NOAA ...
    acknowledgment:             This project was supported in part by a grant...
    cdm_data_type:              Grid
    cdr_program:                NOAA Climate Data Record Program for satellit...
    cdr_variable:               precipitation
    ...                         ...
    standard_name_vocabulary:   CF Standard Name Table (v41, 22 February 2017)
    summary:                    Global Precipitation Climatology Project (GPC...
    time_coverage_duration:     P1D
    time_coverage_end:          1996-10-01T23:59:59Z
    time_coverage_start:        1996-10-01T00:00:00Z
    title:                      Global Precipitation Climatatology Project (G...

xarray.Dataset

• Dimensions:
  • latitude: 180
  • nv: 2
  • longitude: 360
  • time: 9226
• Coordinates: (6)
  • lat_bounds
    (latitude, nv)
    float32
    dask.array<chunksize=(180, 2), meta=np.ndarray>

    comment :

    latitude values at the north and south bounds of each pixel.

    Array Chunk Bytes 1.41 kiB 1.41 kiB Shape (180, 2) (180, 2) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • latitude
    (latitude)
    float32
    -90.0 -89.0 -88.0 ... 88.0 89.0

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    units :

    degrees_north

    valid_range :

    [-90.0, 90.0]

    array([-90., -89., -88., -87., -86., -85., -84., -83., -82., -81., -80., -79.,
           -78., -77., -76., -75., -74., -73., -72., -71., -70., -69., -68., -67.,
           -66., -65., -64., -63., -62., -61., -60., -59., -58., -57., -56., -55.,
           -54., -53., -52., -51., -50., -49., -48., -47., -46., -45., -44., -43.,
           -42., -41., -40., -39., -38., -37., -36., -35., -34., -33., -32., -31.,
           -30., -29., -28., -27., -26., -25., -24., -23., -22., -21., -20., -19.,
           -18., -17., -16., -15., -14., -13., -12., -11., -10.,  -9.,  -8.,  -7.,
            -6.,  -5.,  -4.,  -3.,  -2.,  -1.,   0.,   1.,   2.,   3.,   4.,   5.,
             6.,   7.,   8.,   9.,  10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,
            18.,  19.,  20.,  21.,  22.,  23.,  24.,  25.,  26.,  27.,  28.,  29.,
            30.,  31.,  32.,  33.,  34.,  35.,  36.,  37.,  38.,  39.,  40.,  41.,
            42.,  43.,  44.,  45.,  46.,  47.,  48.,  49.,  50.,  51.,  52.,  53.,
            54.,  55.,  56.,  57.,  58.,  59.,  60.,  61.,  62.,  63.,  64.,  65.,
            66.,  67.,  68.,  69.,  70.,  71.,  72.,  73.,  74.,  75.,  76.,  77.,
            78.,  79.,  80.,  81.,  82.,  83.,  84.,  85.,  86.,  87.,  88.,  89.],
          dtype=float32)

  • lon_bounds
    (longitude, nv)
    float32
    dask.array<chunksize=(360, 2), meta=np.ndarray>

    comment :

    longitude values at the west and east bounds of each pixel.

    Array Chunk Bytes 2.81 kiB 2.81 kiB Shape (360, 2) (360, 2) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • longitude
    (longitude)
    float32
    0.0 1.0 2.0 ... 357.0 358.0 359.0

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    units :

    degrees_east

    valid_range :

    [0.0, 360.0]

    array([  0.,   1.,   2., ..., 357., 358., 359.], shape=(360,), dtype=float32)

  • time
    (time)
    datetime64[ns]
    1996-10-01 ... 2021-12-31

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['1996-10-01T00:00:00.000000000', '1996-10-02T00:00:00.000000000',
           '1996-10-03T00:00:00.000000000', ..., '2021-12-29T00:00:00.000000000',
           '2021-12-30T00:00:00.000000000', '2021-12-31T00:00:00.000000000'],
          shape=(9226,), dtype='datetime64[ns]')

  • time_bounds
    (time, nv)
    datetime64[ns]
    dask.array<chunksize=(9226, 2), meta=np.ndarray>

    comment :

    time bounds for each time value

    Array Chunk Bytes 144.16 kiB 144.16 kiB Shape (9226, 2) (9226, 2) Dask graph 1 chunks in 2 graph layers Data type datetime64[ns] numpy.ndarray

• Data variables: (1)
  • precip
    (time, latitude, longitude)
    float32
    dask.array<chunksize=(400, 180, 360), meta=np.ndarray>

    cell_methods :

    area: mean time: mean

    long_name :

    NOAA Climate Data Record (CDR) of Daily GPCP Satellite-Gauge Combined Precipitation

    standard_name :

    lwe_precipitation_rate

    units :

    mm/day

    valid_range :

    [0.0, 100.0]

    Array Chunk Bytes 2.23 GiB 105.30 MiB Shape (9226, 180, 360) (426, 180, 360) Dask graph 23 chunks in 2 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • latitude
    PandasIndex

    PandasIndex(Index([-90.0, -89.0, -88.0, -87.0, -86.0, -85.0, -84.0, -83.0, -82.0, -81.0,
           ...
            80.0,  81.0,  82.0,  83.0,  84.0,  85.0,  86.0,  87.0,  88.0,  89.0],
          dtype='float32', name='latitude', length=180))

  • longitude
    PandasIndex

    PandasIndex(Index([  0.0,   1.0,   2.0,   3.0,   4.0,   5.0,   6.0,   7.0,   8.0,   9.0,
           ...
           350.0, 351.0, 352.0, 353.0, 354.0, 355.0, 356.0, 357.0, 358.0, 359.0],
          dtype='float32', name='longitude', length=360))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1996-10-01', '1996-10-02', '1996-10-03', '1996-10-04',
                   '1996-10-05', '1996-10-06', '1996-10-07', '1996-10-08',
                   '1996-10-09', '1996-10-10',
                   ...
                   '2021-12-22', '2021-12-23', '2021-12-24', '2021-12-25',
                   '2021-12-26', '2021-12-27', '2021-12-28', '2021-12-29',
                   '2021-12-30', '2021-12-31'],
                  dtype='datetime64[ns]', name='time', length=9226, freq=None))

• Attributes: (45)

  Conventions :

  CF-1.6, ACDD 1.3

  Metadata_Conventions :

  CF-1.6, Unidata Dataset Discovery v1.0, NOAA CDR v1.0, GDS v2.0

  acknowledgment :

  This project was supported in part by a grant from the NOAA Climate Data Record (CDR) Program for satellites.

  cdm_data_type :

  Grid

  cdr_program :

  NOAA Climate Data Record Program for satellites, FY 2011.

  cdr_variable :

  precipitation

  comment :

  Processing computer: eagle2.umd.edu

  contributor_name :

  Robert Adler, Jian-Jian Wang

  contributor_role :

  principalInvestigator, processor and custodian

  creator_email :

  jjwang@umd.edu

  creator_name :

  Dr. Jian-Jian Wang

  date_created :

  2017-05-30T16:52:42Z

  geospatial_lat_max :

  90.0

  geospatial_lat_min :

  -90.0

  geospatial_lat_resolution :

  1 degree

  geospatial_lat_units :

  degrees_north

  geospatial_lon_max :

  360.0

  geospatial_lon_min :

  0.0

  geospatial_lon_resolution :

  1 degree

  geospatial_lon_units :

  degrees_east

  history :

  1) 2017-05-30T16:52:42Z, Dr. Jian-Jian Wang, U of Maryland, Created beta (B1) file

  id :

  199610/gpcp_v01r03_daily_d19961001_c20170530.nc

  institution :

  ACADEMIC > UMD/ESSIC > Earth System Science Interdisciplinary Center, University of Maryland

  keywords :

  EARTH SCIENCE > ATMOSPHERE > PRECIPITATION > PRECIPITATION AMOUNT

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Earth Science Keywords, Version 7.0

  license :

  No constraints on data access or use.

  metadata_link :

  gov.noaa.ncdc:XXXXX

  naming_authority :

  gov.noaa.ncdc

  platform :

  GOES (Geostationary Operational Environmental Satellite), GMS (Japan Geostationary Meteorological Satellite), METEOSAT, TIROS > Television Infrared Observation Satellite, DMSP (Defense Meteorological Satellite Program)

  processing_level :

  NASA Level 3

  product_version :

  v01r03

  project :

  GPCP > Global Precipitation Climatology Project

  publisher_email :

  jjwang@umd.edu

  publisher_name :

  NOAA National Centers for Environmental Information (NCEI)

  publisher_url :

  https://www.ncei.noaa.gov

  references :

  Huffman et al. 1997, http://dx.doi.org/10.1175/1520-0477(1997)078<0005:TGPCPG>2.0.CO;2; Adler et al. 2003, http://dx.doi.org/10.1175/1525-7541(2003)004<1147:TVGPCP>2.0.CO;2; Huffman et al. 2009, http://dx.doi.org/10.1029/2009GL040000; Adler et al. 2017, Global Precipitation Climatology Project (GPCP) Daily Analysis: Climate Algorithm Theoretical Basis Document (C-ATBD)

  sensor :

  Imager, TOVS > TIROS Operational Vertical Sounder, SSMI > Special Sensor Microwave/Imager

  source :

  /data1/GPCP_CDR/GPCP_Output/1DD//bin/199610/stfsg3.19961001.s

  spatial_resolution :

  1 degree

  standard_name_vocabulary :

  CF Standard Name Table (v41, 22 February 2017)

  summary :

  Global Precipitation Climatology Project (GPCP) Daily Version 1.3 gridded, merged satellite/gauge precipitation Climate data Record (CDR) from 1996 to present.

  time_coverage_duration :

  P1D

  time_coverage_end :

  1996-10-01T23:59:59Z

  time_coverage_start :

  1996-10-01T00:00:00Z

  title :

  Global Precipitation Climatatology Project (GPCP) Climate Data Record (CDR), Daily V1.3

Microsoft’s Planetary Computer goes above and beyond, providing tutorials alongside each dataset. We recommend exploring these on your own to get an idea of what you can do with Zarr and Xarray. The following example is based off the Zarr tutorial.

This example demonstrates how to access the Daymet Puerto Rico Dataset on MS Planetary Computer:

import pystac
import planetary_computer

url = "https://planetarycomputer.microsoft.com/api/stac/v1/collections/daymet-daily-hi"
collection = pystac.read_file(url)

asset = collection.assets["zarr-abfs"]

# planetary computer requires a special token to be added to the url
planetary_computer.sign_inplace(asset)

ds = xr.open_zarr(
    asset.href,
    **asset.extra_fields["xarray:open_kwargs"],
    storage_options=asset.extra_fields["xarray:storage_options"],
    zarr_format=2,
)
ds

<xarray.Dataset> Size: 69GB
Dimensions:                  (time: 14965, y: 584, x: 284, nv: 2)
Coordinates:
    lat                      (y, x) float32 663kB dask.array<chunksize=(584, 284), meta=np.ndarray>
    lon                      (y, x) float32 663kB dask.array<chunksize=(584, 284), meta=np.ndarray>
  * time                     (time) datetime64[ns] 120kB 1980-01-01T12:00:00 ...
  * x                        (x) float32 1kB -5.802e+06 ... -5.519e+06
  * y                        (y) float32 2kB -3.9e+04 -4e+04 ... -6.22e+05
Dimensions without coordinates: nv
Data variables:
    dayl                     (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    lambert_conformal_conic  int16 2B ...
    prcp                     (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    srad                     (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    swe                      (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    time_bnds                (time, nv) datetime64[ns] 239kB dask.array<chunksize=(365, 2), meta=np.ndarray>
    tmax                     (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    tmin                     (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    vp                       (time, y, x) float32 10GB dask.array<chunksize=(365, 584, 284), meta=np.ndarray>
    yearday                  (time) int16 30kB dask.array<chunksize=(365,), meta=np.ndarray>
Attributes:
    Conventions:       CF-1.6
    Version_data:      Daymet Data Version 4.0
    Version_software:  Daymet Software Version 4.0
    citation:          Please see http://daymet.ornl.gov/ for current Daymet ...
    references:        Please see http://daymet.ornl.gov/ for current informa...
    source:            Daymet Software Version 4.0
    start_year:        1980

xarray.Dataset

• Dimensions:
  • time: 14965
  • y: 584
  • x: 284
  • nv: 2
• Coordinates: (5)
  • lat
    (y, x)
    float32
    dask.array<chunksize=(584, 284), meta=np.ndarray>

    long_name :

    latitude coordinate

    standard_name :

    latitude

    units :

    degrees_north

    Array Chunk Bytes 647.88 kiB 647.88 kiB Shape (584, 284) (584, 284) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • lon
    (y, x)
    float32
    dask.array<chunksize=(584, 284), meta=np.ndarray>

    long_name :

    longitude coordinate

    standard_name :

    longitude

    units :

    degrees_east

    Array Chunk Bytes 647.88 kiB 647.88 kiB Shape (584, 284) (584, 284) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

  • time
    (time)
    datetime64[ns]
    1980-01-01T12:00:00 ... 2020-12-...

    bounds :

    time_bnds

    long_name :

    24-hour day based on local time

    standard_name :

    time

    array(['1980-01-01T12:00:00.000000000', '1980-01-02T12:00:00.000000000',
           '1980-01-03T12:00:00.000000000', ..., '2020-12-28T12:00:00.000000000',
           '2020-12-29T12:00:00.000000000', '2020-12-30T12:00:00.000000000'],
          shape=(14965,), dtype='datetime64[ns]')

  • x
    (x)
    float32
    -5.802e+06 ... -5.519e+06

    long_name :

    x coordinate of projection

    standard_name :

    projection_x_coordinate

    units :

    m

    array([-5.80225e+06, -5.80125e+06, -5.80025e+06, ..., -5.52125e+06,
           -5.52025e+06, -5.51925e+06], shape=(284,), dtype=float32)

  • y
    (y)
    float32
    -3.9e+04 -4e+04 ... -6.22e+05

    long_name :

    y coordinate of projection

    standard_name :

    projection_y_coordinate

    units :

    m

    array([ -39000.,  -40000.,  -41000., ..., -620000., -621000., -622000.],
          shape=(584,), dtype=float32)

• Data variables: (10)
  • dayl
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean

    grid_mapping :

    lambert_conformal_conic

    long_name :

    daylength

    units :

    s

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • lambert_conformal_conic
    ()
    int16
    ...

    false_easting :

    0.0

    false_northing :

    0.0

    grid_mapping_name :

    lambert_conformal_conic

    inverse_flattening :

    298.257223563

    latitude_of_projection_origin :

    42.5

    longitude_of_central_meridian :

    -100.0

    semi_major_axis :

    6378137.0

    standard_parallel :

    [25.0, 60.0]

    [1 values with dtype=int16]

  • prcp
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean time: sum

    grid_mapping :

    lambert_conformal_conic

    long_name :

    daily total precipitation

    units :

    mm/day

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • srad
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean time: mean

    grid_mapping :

    lambert_conformal_conic

    long_name :

    daylight average incident shortwave radiation

    units :

    W/m2

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • swe
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean time: mean

    grid_mapping :

    lambert_conformal_conic

    long_name :

    snow water equivalent

    units :

    kg/m2

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • time_bnds
    (time, nv)
    datetime64[ns]
    dask.array<chunksize=(365, 2), meta=np.ndarray>

    Array Chunk Bytes 233.83 kiB 5.70 kiB Shape (14965, 2) (365, 2) Dask graph 41 chunks in 2 graph layers Data type datetime64[ns] numpy.ndarray

  • tmax
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean time: maximum

    grid_mapping :

    lambert_conformal_conic

    long_name :

    daily maximum temperature

    units :

    degrees C

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • tmin
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean time: minimum

    grid_mapping :

    lambert_conformal_conic

    long_name :

    daily minimum temperature

    units :

    degrees C

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • vp
    (time, y, x)
    float32
    dask.array<chunksize=(365, 584, 284), meta=np.ndarray>

    cell_methods :

    area: mean time: mean

    grid_mapping :

    lambert_conformal_conic

    long_name :

    daily average vapor pressure

    units :

    Pa

    Array Chunk Bytes 9.25 GiB 230.93 MiB Shape (14965, 584, 284) (365, 584, 284) Dask graph 41 chunks in 2 graph layers Data type float32 numpy.ndarray

  • yearday
    (time)
    int16
    dask.array<chunksize=(365,), meta=np.ndarray>

    long_name :

    day of year (DOY) starting with day 1 on Januaray 1st

    Array Chunk Bytes 29.23 kiB 730 B Shape (14965,) (365,) Dask graph 41 chunks in 2 graph layers Data type int16 numpy.ndarray

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1980-01-01 12:00:00', '1980-01-02 12:00:00',
                   '1980-01-03 12:00:00', '1980-01-04 12:00:00',
                   '1980-01-05 12:00:00', '1980-01-06 12:00:00',
                   '1980-01-07 12:00:00', '1980-01-08 12:00:00',
                   '1980-01-09 12:00:00', '1980-01-10 12:00:00',
                   ...
                   '2020-12-21 12:00:00', '2020-12-22 12:00:00',
                   '2020-12-23 12:00:00', '2020-12-24 12:00:00',
                   '2020-12-25 12:00:00', '2020-12-26 12:00:00',
                   '2020-12-27 12:00:00', '2020-12-28 12:00:00',
                   '2020-12-29 12:00:00', '2020-12-30 12:00:00'],
                  dtype='datetime64[ns]', name='time', length=14965, freq=None))

  • x
    PandasIndex

    PandasIndex(Index([-5802250.0, -5801250.0, -5800250.0, -5799250.0, -5798250.0, -5797250.0,
           -5796250.0, -5795250.0, -5794250.0, -5793250.0,
           ...
           -5528250.0, -5527250.0, -5526250.0, -5525250.0, -5524250.0, -5523250.0,
           -5522250.0, -5521250.0, -5520250.0, -5519250.0],
          dtype='float32', name='x', length=284))

  • y
    PandasIndex

    PandasIndex(Index([ -39000.0,  -40000.0,  -41000.0,  -42000.0,  -43000.0,  -44000.0,
            -45000.0,  -46000.0,  -47000.0,  -48000.0,
           ...
           -613000.0, -614000.0, -615000.0, -616000.0, -617000.0, -618000.0,
           -619000.0, -620000.0, -621000.0, -622000.0],
          dtype='float32', name='y', length=584))

• Attributes: (7)

  Conventions :

  CF-1.6

  Version_data :

  Daymet Data Version 4.0

  Version_software :

  Daymet Software Version 4.0

  citation :

  Please see http://daymet.ornl.gov/ for current Daymet data citation information

  references :

  Please see http://daymet.ornl.gov/ for current information on Daymet references

  source :

  Daymet Software Version 4.0

  start_year :

  1980

Additional Resources

• Jupyter Notebook for a high level overview of Zarr on Google Cloud by Tyson Swetnam: