├── binder ├── apt.txt ├── Dockerfile ├── postBuild └── environment.yml ├── images ├── esm-demo.gif └── pangeo-catalog.png ├── .pre-commit-config.yaml ├── LICENSE ├── .gitignore └── README.md /binder/apt.txt: -------------------------------------------------------------------------------- 1 | git 2 | -------------------------------------------------------------------------------- /binder/Dockerfile: -------------------------------------------------------------------------------- 1 | # Note that there must be a tag 2 | FROM pangeo/base-image:2020.06.10 3 | -------------------------------------------------------------------------------- /images/esm-demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/andersy005/intake-pangeo-catalog-EarthCube-2020/master/images/esm-demo.gif -------------------------------------------------------------------------------- /images/pangeo-catalog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/andersy005/intake-pangeo-catalog-EarthCube-2020/master/images/pangeo-catalog.png -------------------------------------------------------------------------------- /binder/postBuild: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | jupyter labextension install --clean \ 4 | @jupyter-widgets/jupyterlab-manager \ 5 | dask-labextension 6 | 7 | #EOF 8 | -------------------------------------------------------------------------------- /binder/environment.yml: -------------------------------------------------------------------------------- 1 | name: earthcube-notebook 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - cartopy 6 | - cftime 7 | - fsspec=0.7.3 8 | - gcsfs 9 | - intake-esm 10 | - matplotlib 11 | - nc-time-axis 12 | - netcdf4 13 | - pangeo-notebook=2020.06.10 14 | - pip 15 | - watermark 16 | - zarr 17 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | 3 | - repo: https://github.com/pre-commit/pre-commit-hooks 4 | rev: v2.5.0 5 | hooks: 6 | - id: trailing-whitespace 7 | - id: end-of-file-fixer 8 | - id: check-docstring-first 9 | - id: check-yaml 10 | - id: double-quote-string-fixer 11 | 12 | - repo: https://github.com/ambv/black 13 | rev: 19.10b0 14 | hooks: 15 | - id: black 16 | args: ["--skip-string-normalization"] 17 | 18 | 19 | - repo: https://gitlab.com/pycqa/flake8 20 | rev: 3.8.0a2 21 | hooks: 22 | - id: flake8 23 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Anderson Banihirwe 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Binder](https://img.shields.io/static/v1.svg?logo=Jupyter&label=Pangeo+Binder&message=GCE+us-central1&color=blue&style=for-the-badge)](https://binder.pangeo.io/v2/gh/andersy005/intake-pangeo-catalog-EarthCube-2020/master) 2 | 3 | # Intake / Pangeo Catalog: Making It Easier To Consume Earth’s Climate and Weather Data 4 | - [Intake / Pangeo Catalog: Making It Easier To Consume Earth’s Climate and Weather Data](#intake--pangeo-catalog-making-it-easier-to-consume-earths-climate-and-weather-data) 5 | - [Authors](#authors) 6 | - [Abstract](#abstract) 7 | - [Try this notebook on Pangeo Binder](#try-this-notebook-on-pangeo-binder) 8 | 9 | ## Authors 10 | 11 | - [Anderson Banihirwe](https://github.com/andersy005) (National Center for Atmospheric Research) 12 | - [Charles Blackmon-Luca](https://github.com/charlesbluca) (Columbia University / Lamont Doherty Earth Observatory) 13 | - [Ryan Abernathey](https://github.com/rabernat) (Columbia University / Lamont Doherty Earth Observatory) 14 | - [Joe Hamman](https://github.com/jhamman) (National Center for Atmospheric Research) 15 | 16 | ## Abstract 17 | 18 | Computer simulations of the Earth’s climate and weather generate huge amounts of data. These data are often persisted on HPC systems or in the cloud across multiple data assets of a variety of formats (netCDF, zarr, etc...). Finding, investigating, loading these data assets into compute-ready data containers costs time and effort. The data user needs to know what data sets are available, the attributes describing each data set, before loading a specific data set and analyzing it. 19 | 20 | In this notebook, we demonstrate the integration of data discovery tools such as [intake](https://intake.readthedocs.io/en/latest/) and [intake-esm](https://intake-esm.readthedocs.io/en/latest/) (an intake plugin) with data stored in cloud optimized formats (zarr). We highlight (1) how these tools provide transparent access to local and remote catalogs and data, (2) the API for exploring arbitrary metadata associated with data, loading data sets into data array containers. 21 | 22 | We also showcase the [Pangeo catalog](https://catalog.pangeo.io/), an open source project to enumerate and organize cloud optimized climate data stored across a variety of providers, and a place where several intake-esm collections are now publicly available. We use one of these public collections as an example to show how an end user would explore and interact with the data, and conclude with a short overview of the catalog's online presence. 23 | 24 | ## Try this notebook on Pangeo Binder 25 | 26 | To try this notebook in your web browser, just click on the binder logo/image: 27 | 28 | [![Binder](https://img.shields.io/static/v1.svg?logo=Jupyter&label=Pangeo+Binder&message=GCE+us-central1&color=blue&style=for-the-badge)](https://binder.pangeo.io/v2/gh/andersy005/intake-pangeo-catalog-EarthCube-2020/master) 29 | --------------------------------------------------------------------------------