├── 01_Introduction_Jupyter_Notebooks.ipynb ├── 02_Geospatial_Data_Access.ipynb ├── 03_Introduction_Jupyter_widgets.ipynb ├── 04_Jupyter_Environment.ipynb ├── README.md ├── era5_test.nc ├── img ├── 2019_esowc_banner.png ├── aws_logo.png ├── binder_logo.png ├── cds_logo.jpg ├── data_services.png ├── gee_logo.jpg ├── gee_logo_2.png ├── jupyterhub.png └── jupyterlab.png └── index.ipynb /01_Introduction_Jupyter_Notebooks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Short course **\"Interactive analysis of Big Earth Data with Jupyter Notebooks\"** | EGU 2019" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# 1 Introducing Jupyter Notebooks - Data analysis made simple" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "
\n", 22 | "<< Index           2 - Geospatial Data Access >>" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "## 1.1. Why Jupyter Notebooks" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "
\n", 37 | "\"Project Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages.\"\n", 38 | "
" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "- #### Started with Python support, now supports over 40 programming languages, including Python, R, Julia, ...\n", 46 | "- #### Notebooks can be easily shared with colleagues via GitHub, NBViewer, etc.\n", 47 | "- #### Code, data and visualizations are combined in one place\n", 48 | "- #### Jupyter widgets allow for interactive outputs in various formats, e.g. images, HTML, videos, etc.\n", 49 | "- #### A great tool for teaching\n", 50 | "- #### JupyterHub allows you to access an environment ready to code " 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "## 1.2. Installation" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "### Installing Jupyter using Anaconda\n", 65 | "\n", 66 | "Anaconda comes with the Jupyter Notebook installed. You just have to download Anaconda and following the installation instructions. Once installed, the jupyter notebook can be started with:" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "jupyter notebook" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "### Installing Jupyter with pip" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "Experienced Python users may want to install Jupyter using Python's package manager \"pip\".\n", 90 | "\n", 91 | "With Python3 you do:" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "python3 -m pip install --upgrade pip\n", 101 | "python3 -m pip install jupyter" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "With Python2 installed, you do: (*however I strongly recommend to switch to Python3 soon*)" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": {}, 115 | "outputs": [], 116 | "source": [ 117 | "python -m pip install --upgrade pip\n", 118 | "python -m pip install jupyter" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "In order to run the notebook, you run the following command at the Terminal:" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "jupyter notebook" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "## 1.3 Jupyter Notebook UI" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "* Notebook dashboard\n", 149 | " * Create new notebook\n", 150 | "* Notebook editor (UI)\n", 151 | " * Menu\n", 152 | " * Toolbar\n", 153 | " * Notebook area and cells\n", 154 | "* Cell types\n", 155 | " * Code\n", 156 | " * Markdown\n", 157 | "* Edit vs. Command mode" 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "metadata": {}, 163 | "source": [ 164 | "## 1.4 Shortcuts" 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": {}, 170 | "source": [ 171 | "* Get an overview of the shortcuts by hitting 'H' or go to Help/Keyboard shortcuts" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "### 1.4.1 Most useful shortcuts" 179 | ] 180 | }, 181 | { 182 | "cell_type": "markdown", 183 | "metadata": {}, 184 | "source": [ 185 | "* Esc - switch to command mode\n", 186 | " * B - insert below\n", 187 | " * A - insert above\n", 188 | " * M - Change current cell to Markdown\n", 189 | " * Y - Change current cell to code\n", 190 | " * DD - Delete cell\n", 191 | "* Enter - go back to edit mode\n", 192 | "* Esc + F - Find and replace on your code\n", 193 | "* Shift + Down / Upwards - Select multiple cells\n", 194 | "* Shift + M - Merge multiple cells" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": {}, 200 | "source": [ 201 | "## 1.5 Cell magic" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "Magic commands can make your life a lot easier, as you only have one command instead of an entire function or multiple lines of code.\n", 209 | "\n", 210 | "- [Extensive overview of magic commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html)\n", 211 | "\n", 212 | "### 1.5.1 Some of the handy ones" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "#### Overview of available magic commands" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "%lsmagic" 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "#### Easily see and set environment variables" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "%env" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "#### Install and list libraries" 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "metadata": {}, 258 | "outputs": [], 259 | "source": [ 260 | "!pip install numpy\n", 261 | "!pip list | grep pandas" 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "metadata": {}, 267 | "source": [ 268 | "#### Write cell content to a file or load a python file" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": {}, 275 | "outputs": [], 276 | "source": [ 277 | "%%writefile hello_world.py\n", 278 | "\n", 279 | "print('Hello World')" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "metadata": {}, 286 | "outputs": [], 287 | "source": [ 288 | "# Let's you check the content of a Python file (code highlighted)\n", 289 | "%pycat hello_world.py" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "#### Time the execution of a cell" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": null, 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "%%time\n", 306 | "\n", 307 | "tmpList = []\n", 308 | "for i in range(100):\n", 309 | " tmpList.append(i+i)\n", 310 | "\n", 311 | "print(tmpList)" 312 | ] 313 | }, 314 | { 315 | "cell_type": "markdown", 316 | "metadata": {}, 317 | "source": [ 318 | "#### Run code from a different kernel in a notebook" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "metadata": {}, 325 | "outputs": [], 326 | "source": [ 327 | "%%bash\n", 328 | "for i in {1..5}\n", 329 | "do\n", 330 | " echo \"i is $i\"\n", 331 | "done" 332 | ] 333 | }, 334 | { 335 | "cell_type": "markdown", 336 | "metadata": {}, 337 | "source": [ 338 | "#### Show matplotlib plots inline" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": {}, 345 | "outputs": [], 346 | "source": [ 347 | "%matplotlib inline" 348 | ] 349 | }, 350 | { 351 | "cell_type": "markdown", 352 | "metadata": {}, 353 | "source": [ 354 | "## 1.6 Widgets" 355 | ] 356 | }, 357 | { 358 | "cell_type": "markdown", 359 | "metadata": {}, 360 | "source": [ 361 | "With widgets your Jupyter Notebooks can turn into an interactive GUI. Widgets are eventful python objects that have a control, such as a slider, textbox, etc." 362 | ] 363 | }, 364 | { 365 | "cell_type": "markdown", 366 | "metadata": {}, 367 | "source": [ 368 | "In order to use widgets, ipywidgets have to be installed and imported" 369 | ] 370 | }, 371 | { 372 | "cell_type": "code", 373 | "execution_count": null, 374 | "metadata": {}, 375 | "outputs": [], 376 | "source": [ 377 | "import ipywidgets as widgets" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "metadata": {}, 383 | "source": [ 384 | "Get a full overview of available widgets [here](https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20List.html)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "### 1.6.1 Basic widget structure" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": null, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "from IPython.display import display\n", 401 | "# Slider with Integers, same is possible with Floats\n", 402 | "w = widgets.IntSlider()\n", 403 | "display(w)" 404 | ] 405 | }, 406 | { 407 | "cell_type": "markdown", 408 | "metadata": {}, 409 | "source": [ 410 | "#### How to populate a widget object with values" 411 | ] 412 | }, 413 | { 414 | "cell_type": "code", 415 | "execution_count": null, 416 | "metadata": {}, 417 | "outputs": [], 418 | "source": [ 419 | "w = widgets.IntSlider(min=100,max=500, value=150, step=10)\n", 420 | "display(w)" 421 | ] 422 | }, 423 | { 424 | "cell_type": "markdown", 425 | "metadata": {}, 426 | "source": [ 427 | "#### Text widgets" 428 | ] 429 | }, 430 | { 431 | "cell_type": "code", 432 | "execution_count": null, 433 | "metadata": {}, 434 | "outputs": [], 435 | "source": [ 436 | "text = widgets.Text(value=\"Hi, Jupyter is awesome!\", disabled=False)\n", 437 | "text" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": {}, 443 | "source": [ 444 | "#### Widgets can be combined" 445 | ] 446 | }, 447 | { 448 | "cell_type": "code", 449 | "execution_count": null, 450 | "metadata": {}, 451 | "outputs": [], 452 | "source": [ 453 | "a = widgets.FloatText()\n", 454 | "b = widgets.FloatSlider()\n", 455 | "display(a,b)\n", 456 | "myLink = widgets.jslink((a,'value'),(b,'value'))" 457 | ] 458 | }, 459 | { 460 | "cell_type": "markdown", 461 | "metadata": {}, 462 | "source": [ 463 | "#### Radio buttons" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": null, 469 | "metadata": {}, 470 | "outputs": [], 471 | "source": [ 472 | "widgets.RadioButtons(\n", 473 | " options=['Monday', 'Tuesday', 'Wednesday'],\n", 474 | " description='Week day:',\n", 475 | " disabled=False\n", 476 | ")" 477 | ] 478 | }, 479 | { 480 | "cell_type": "markdown", 481 | "metadata": {}, 482 | "source": [ 483 | "
\n", 484 | "© 2019 | Julia Wagemann\n", 485 | "\"Creative" 486 | ] 487 | } 488 | ], 489 | "metadata": { 490 | "kernelspec": { 491 | "display_name": "Python 3", 492 | "language": "python", 493 | "name": "python3" 494 | }, 495 | "language_info": { 496 | "codemirror_mode": { 497 | "name": "ipython", 498 | "version": 3 499 | }, 500 | "file_extension": ".py", 501 | "mimetype": "text/x-python", 502 | "name": "python", 503 | "nbconvert_exporter": "python", 504 | "pygments_lexer": "ipython3", 505 | "version": "3.7.0" 506 | } 507 | }, 508 | "nbformat": 4, 509 | "nbformat_minor": 2 510 | } 511 | -------------------------------------------------------------------------------- /02_Geospatial_Data_Access.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Short course **\"Interactive analysis of Big Earth Data with Jupyter Notebooks\"** | EGU 2019" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# 2 Big Earth Data Access\n", 15 | "### from different data repositories, e.g. Climate Data Store, Google Earth Engine or Earth on AWS" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "
\n", 23 | " << 1 - Introduction to Jupyter Notebooks           3 - Introducing Jupyter widgets >>" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Overview of Big Earth Data repositories\n", 31 | "\n", 32 | "There is a lot of open environmental data available. The problem is often that people do not know where to find the data and how to access it.\n", 33 | "We will go through three Big Earth Data repositories today.\n", 34 | "\n", 35 | "\n", 36 | "* [2.1 Copernicus Climate Data Store (CDS)](#cds)\n", 37 | "* [2.2 Google Earth Engine (GEE)](#gee)\n", 38 | "* [2.3 Earth on Amazon Web Services (AWS)](#earth_aws)\n", 39 | "\n", 40 | "\n", 41 | "
\n", 42 | "\n", 43 | "\n", 44 | "\n" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "
" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "## 2.1 Copernicus Climate Data Store (CDS)" 59 | ] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "metadata": {}, 64 | "source": [ 65 | "\n", 66 | "The [Copernicus Climate Data Store (CDS)](cds.copernicus.eu) is a one-stop shop for information about the climate: past, present and future. It is operated by the [European Centre for Medium-Range Weather Forecasts(ECMWF)](https://ecmwf.int).\n", 67 | "\n", 68 | "\n", 69 | "It consists of two parts:\n", 70 | "* Access to Climate Datasets via a [web interface](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset) or programmatically via the [Climate Data Store API](https://cds.climate.copernicus.eu/api-how-to)\n", 71 | "* Analyse and visualise climate data with the [Climate Data Store toolbox (Python interface)](https://cds.climate.copernicus.eu/user/login?destination=/toolbox-user)\n", 72 | "\n", 73 | "Data are natively available in GRIB and NetCDF." 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "### Data available on the CDS (a selection)" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "The climate data store has a wide variety of climate data, e.g.:\n", 88 | "* **ERA5 climate reanalysis**\n", 89 | "* **Seasonal forecasts**\n", 90 | "* **Climate projections**\n", 91 | "* **Sectoral climate indices**\n", 92 | "\n", 93 | "Have a look and browse through [all the publicly available datasets on the CDS](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset). \n", 94 | "\n", 95 | "ECMWF has many more publicly available datasets, e.g. data on flood, fire risk and air quality (provided by the [Copernicus Atmosphere Monitoring Service](https://atmosphere.copernicus.eu). Have a look at [ECMWF's public datasets](https://apps.ecmwf.int/datasets/).\n" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "### Example to download ERA5 data in GRIB" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "Required non-standard libraries:\n", 110 | "* [cdsapi](https://pypi.org/project/cdsapi/)\n", 111 | "* [urllib](https://docs.python.org/3/library/urllib.html)" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": null, 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "import os, ssl\n", 121 | "if (not os.environ.get('PYTHONHTTPSVERIFY', '') and\n", 122 | " getattr(ssl, '_create_unverified_context', None)): \n", 123 | " ssl._create_default_https_context = ssl._create_unverified_context" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "import cdsapi\n", 133 | "import os\n", 134 | "import urllib\n", 135 | "import sys\n", 136 | "\n", 137 | "c = cdsapi.Client()\n", 138 | "\n", 139 | "os.chdir('./')\n", 140 | "\n", 141 | "def retrieve_func():\n", 142 | " data = c.retrieve(\n", 143 | " 'reanalysis-era5-single-levels',\n", 144 | " {\n", 145 | " 'product_type':'reanalysis',\n", 146 | " 'format':'netcdf',\n", 147 | " 'variable':'2m_temperature',\n", 148 | " 'year':'2019',\n", 149 | " 'month':'01',\n", 150 | " 'day':'01',\n", 151 | " 'time':'12:00',\n", 152 | " # Natively ERA5 data are on a 0-360 longitude grid\n", 153 | " 'area':'90/-180/-90/179.75'\n", 154 | " },\n", 155 | " 'download.nc')\n", 156 | " return data\n", 157 | "\n", 158 | "\n", 159 | "filename = \"era5_t2m_test.nc\"\n", 160 | "data = retrieve_func()\n", 161 | "urllib.request.urlretrieve(data.location, filename)\n", 162 | "\n", 163 | "sys.exit()" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "### Example to open a NetCDF file with xarray" 171 | ] 172 | }, 173 | { 174 | "cell_type": "markdown", 175 | "metadata": {}, 176 | "source": [ 177 | "Useful Python libraries to open NetCDF / GRIB files:\n", 178 | "* [xarray](http://xarray.pydata.org/en/stable/) - Natively supports NetCDF\n", 179 | "* [cfgrib](https://github.com/ecmwf/cfgrib) - A Python interface that supports GRIB engine for the xarray library\n", 180 | "* [netCDF4](http://unidata.github.io/netcdf4-python/netCDF4/index.html) - A Python interface to the netCDF4 C library" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | " import xarray as xr" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "test = xr.open_dataset('era5_test.nc')\n", 199 | "print(test)" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "### Good news for R users" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "metadata": {}, 212 | "source": [ 213 | "Koen Huefkens and Reto Stauffer just released the CRAN package [ecmwfr](https://cran.r-project.org/web/packages/ecmwfr/index.html), which is a programmatic interface to public data at ECMWF and on the CDS.\n", 214 | "\n", 215 | "* [ecmwfr](https://cran.r-project.org/web/packages/ecmwfr/index.html)" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "
" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "## 2.2 Google Earth Engine" 230 | ] 231 | }, 232 | { 233 | "cell_type": "markdown", 234 | "metadata": {}, 235 | "source": [ 236 | "\n", 237 | "[Google Earth Engine (GEE)](https://earthengine.google.com/) is a planetary-scale platform for Earth science data & analysis. \n", 238 | "\n", 239 | "There are several ways to work with the Google Earth Engine:\n", 240 | "* [Code Editor]( code.earthengine.google.com), a web-based IDE in Javascript\n", 241 | "* [Client libraries](https://github.com/google/earthengine-api) provide Javascript and Python wrapper functions for the Earth Engine API\n", 242 | "\n", 243 | "You have to sign up for GEE." 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": {}, 249 | "source": [ 250 | "### Data available on GEE" 251 | ] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": {}, 256 | "source": [ 257 | "Earth Engine's data archive includes:\n", 258 | "* **Weather and Climate Data**\n", 259 | " * A selection of ERA5 reanalysis **[SOON PUBLICLY AVAILABLE]**\n", 260 | " * TRMM precipitation\n", 261 | "* **Imagery**\n", 262 | " * Landsat\n", 263 | " * Sentinel\n", 264 | " * MODIS\n", 265 | " \n", 266 | "... and many more. Have a look yourself at the [Earth Engine Data Catalog](https://developers.google.com/earth-engine/datasets/catalog/)." 267 | ] 268 | }, 269 | { 270 | "cell_type": "markdown", 271 | "metadata": {}, 272 | "source": [ 273 | "### Example how to load an image from GEE and to interactively visualize it with ipyleaflet" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "metadata": {}, 279 | "source": [ 280 | "Required libraries:\n", 281 | "* [Earth Engine Python API](https://developers.google.com/earth-engine/python_install)\n", 282 | "* [ipyleaflet](https://ipyleaflet.readthedocs.io/en/latest/)" 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": {}, 289 | "outputs": [], 290 | "source": [ 291 | "%matplotlib inline" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "metadata": {}, 298 | "outputs": [], 299 | "source": [ 300 | "import ee" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "from ipyleaflet import Map, basemaps, basemap_to_tiles, FullScreenControl, Marker\n", 310 | "import ipyleaflet\n", 311 | "import ipywidgets\n", 312 | "import ipywidgets as widgets\n", 313 | "from IPython.display import display, clear_output" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "# Initialize ee with access token\n", 323 | "ee.Initialize()" 324 | ] 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": {}, 329 | "source": [ 330 | "Function below is taken from Tyler Erickson's [notebooks](https://github.com/tylere/EEUS2018-JupyterSession/blob/master/02%20-%20Interactive%20Maps.ipynb) for his Interactive Jupyter session at EEUS18." 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": null, 336 | "metadata": {}, 337 | "outputs": [], 338 | "source": [ 339 | "def GetTileLayerUrl(ee_image_object):\n", 340 | " map_id = ee.Image(ee_image_object).getMapId()\n", 341 | " tile_url_template = \"https://earthengine.googleapis.com/map/{mapid}/{{z}}/{{x}}/{{y}}?token={token}\"\n", 342 | " return tile_url_template.format(**map_id)" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "### Load an ERA5 image and get image information" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": {}, 356 | "outputs": [], 357 | "source": [ 358 | "img_test = ee.Image('projects/ecmwf/era5_monthly/200001')\n", 359 | "img_test.getInfo()" 360 | ] 361 | }, 362 | { 363 | "cell_type": "markdown", 364 | "metadata": {}, 365 | "source": [ 366 | "### Select one specific parameter" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": null, 372 | "metadata": {}, 373 | "outputs": [], 374 | "source": [ 375 | "t2m = img_test.select('t2m')\n", 376 | "tp = img_test.select('tp')" 377 | ] 378 | }, 379 | { 380 | "cell_type": "markdown", 381 | "metadata": {}, 382 | "source": [ 383 | "### Get image url for visualization" 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": null, 389 | "metadata": {}, 390 | "outputs": [], 391 | "source": [ 392 | "t2m_url = GetTileLayerUrl(t2m)\n", 393 | "tp_url = GetTileLayerUrl(tp)" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [ 402 | "map1 = ipyleaflet.Map(\n", 403 | " zoom=2,\n", 404 | " layout={'height':'500px'},\n", 405 | ")\n", 406 | "\n", 407 | "map1.add_layer(ipyleaflet.TileLayer(url=t2m_url))\n", 408 | "map1.add_layer(ipyleaflet.TileLayer(url=tp_url))\n", 409 | "\n", 410 | "# Adding some fance controls to the map, e.g. layers conrol, FullScreenControl\n", 411 | "map1.add_control(ipyleaflet.LayersControl())\n", 412 | "control = FullScreenControl()\n", 413 | "map1.add_control(control)\n", 414 | "\n", 415 | "map1" 416 | ] 417 | }, 418 | { 419 | "cell_type": "markdown", 420 | "metadata": {}, 421 | "source": [ 422 | "### Get image url and apply visualization params to it" 423 | ] 424 | }, 425 | { 426 | "cell_type": "code", 427 | "execution_count": null, 428 | "metadata": {}, 429 | "outputs": [], 430 | "source": [ 431 | "t2m_url = GetTileLayerUrl(t2m.visualize(min=250, max=310, palette=['#000080','#0000D9','#4000FF','#8000FF','#0080FF'\\\n", 432 | " ,'#00FFFF','#00FF80','#80FF00','#DAFF00','#FFFF00','#FFF500','#FFDA00','#FFB000','#FFA400','#FF4F00','#FF2500','#FF0A00','#FF00FF']))\n", 433 | "tp_url = GetTileLayerUrl(tp.visualize(min=0, max=1, palette=['#FFFFFF', '#00FFFF', '#0080FF', '#DA00FF', '#FFA400','#FF0000']))" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": null, 439 | "metadata": {}, 440 | "outputs": [], 441 | "source": [ 442 | "map2 = ipyleaflet.Map(\n", 443 | " zoom=2,\n", 444 | " layout={'height':'500px'},\n", 445 | ")\n", 446 | "\n", 447 | "map2.add_layer(ipyleaflet.TileLayer(url=t2m_url))\n", 448 | "map2.add_layer(ipyleaflet.TileLayer(url=tp_url))\n", 449 | "\n", 450 | "# Adding the layers control to the map.\n", 451 | "map2.add_control(ipyleaflet.LayersControl())\n", 452 | "control = FullScreenControl()\n", 453 | "map2.add_control(control)\n", 454 | "\n", 455 | "map2" 456 | ] 457 | }, 458 | { 459 | "cell_type": "markdown", 460 | "metadata": {}, 461 | "source": [ 462 | "
" 463 | ] 464 | }, 465 | { 466 | "cell_type": "markdown", 467 | "metadata": {}, 468 | "source": [ 469 | "## 2.3 Earth on AWS" 470 | ] 471 | }, 472 | { 473 | "cell_type": "markdown", 474 | "metadata": {}, 475 | "source": [ 476 | "[Earth on AWS](\"https://aws.amazon.com/earth/\") is a registry of open geospatial datasets on Amazon Web Services.\n", 477 | "* [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) is the Amazon Web Services (AWS) SDK for Python in order e.g. to access data on AWS S3 storage\n", 478 | "\n", 479 | "A more detailed example to access ERA5 data from a S3 cloud storage bucket is available [here](https://github.com/planet-os/notebooks/blob/master/aws/era5-s3-via-boto.ipynb), generated by Intertrust Technologies Corporation." 480 | ] 481 | }, 482 | { 483 | "cell_type": "code", 484 | "execution_count": null, 485 | "metadata": {}, 486 | "outputs": [], 487 | "source": [ 488 | "import boto3\n", 489 | "import botocore" 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": null, 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "era5_bucket = 'era5-pds'\n", 499 | "\n", 500 | "# No AWS keys required\n", 501 | "client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))" 502 | ] 503 | }, 504 | { 505 | "cell_type": "code", 506 | "execution_count": null, 507 | "metadata": {}, 508 | "outputs": [], 509 | "source": [ 510 | "paginator = client.get_paginator('list_objects')\n", 511 | "result = paginator.paginate(Bucket=era5_bucket, Delimiter='/')\n", 512 | "for prefix in result.search('CommonPrefixes'):\n", 513 | " print(prefix.get('Prefix'))" 514 | ] 515 | }, 516 | { 517 | "cell_type": "markdown", 518 | "metadata": {}, 519 | "source": [ 520 | "
\n", 521 | "© 2019 | Julia Wagemann\n", 522 | "\"Creative" 523 | ] 524 | } 525 | ], 526 | "metadata": { 527 | "kernelspec": { 528 | "display_name": "Python 3", 529 | "language": "python", 530 | "name": "python3" 531 | }, 532 | "language_info": { 533 | "codemirror_mode": { 534 | "name": "ipython", 535 | "version": 3 536 | }, 537 | "file_extension": ".py", 538 | "mimetype": "text/x-python", 539 | "name": "python", 540 | "nbconvert_exporter": "python", 541 | "pygments_lexer": "ipython3", 542 | "version": "3.7.0" 543 | } 544 | }, 545 | "nbformat": 4, 546 | "nbformat_minor": 2 547 | } 548 | -------------------------------------------------------------------------------- /03_Introduction_Jupyter_widgets.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Short course **\"Interactive analysis of Big Earth Data with Jupyter Notebooks\"** | EGU 2019" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# 3 Introducing Jupyter widgets\n", 15 | "### Make your data analysis and visualisation interactive" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "
\n", 23 | "<< 2 - Geospatial data access           4 - Getting to know the Jupyter environment >>" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Jupyter widgets - an overview" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "Widgets can be used to build interactive GUIs for you notebooks. Get more information about widgets in the official [readthedocs](https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20List.html#Selection-widgets)." 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "### Some widget examples (before we get real)" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "# Load the widgets\n", 54 | "import ipywidgets as widgets" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "#### Numeric widgets" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "widgets.IntSlider(\n", 71 | " value=7,\n", 72 | " min=0,\n", 73 | " max=10,\n", 74 | " step=1,\n", 75 | " description='Test:',\n", 76 | " disabled=False,\n", 77 | " continuous_update=False,\n", 78 | " orientation='vertical', #horizontal\n", 79 | " readout=True,\n", 80 | " readout_format='d'\n", 81 | ")" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "widgets.IntProgress(\n", 91 | " value=7,\n", 92 | " min=0,\n", 93 | " max=10,\n", 94 | " step=1,\n", 95 | " description='Loading:',\n", 96 | " bar_style='warning', # 'success', 'info', 'warning', 'danger' or ''\n", 97 | " orientation='horizontal'\n", 98 | ")" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "#### Boolean widgets" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "widgets.ToggleButton(\n", 115 | " value=False,\n", 116 | " description='Click me',\n", 117 | " disabled=False,\n", 118 | " button_style='info', # 'success', 'info', 'warning', 'danger' or ''\n", 119 | " tooltip='Description',\n", 120 | " icon='check'\n", 121 | ")" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "widgets.Checkbox(\n", 131 | " value=False,\n", 132 | " description='Check me',\n", 133 | " disabled=False\n", 134 | ")" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "#### Selection widgets" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "widgets.RadioButtons(\n", 151 | " options=['Jupyter is great', 'Jupyter is average', 'Do not understand the hype'],\n", 152 | "# value='pineapple',\n", 153 | " description='Survey:',\n", 154 | " disabled=False\n", 155 | ")" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": null, 161 | "metadata": {}, 162 | "outputs": [], 163 | "source": [ 164 | "widgets.ToggleButtons(\n", 165 | " options=['Slow', 'Regular', 'Fast'],\n", 166 | " description='Speed:',\n", 167 | " disabled=False,\n", 168 | " button_style='', # 'success', 'info', 'warning', 'danger' or ''\n", 169 | " tooltips=['Description of slow', 'Description of regular', 'Description of fast'],\n", 170 | "# icons=['check'] * 3\n", 171 | ")" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "#### String widgets" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "widgets.Text(\n", 188 | " value='Hello World',\n", 189 | " placeholder='Type something',\n", 190 | " description='String:',\n", 191 | " disabled=False\n", 192 | ")" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "widgets.Textarea(\n", 202 | " value='Hello World',\n", 203 | " placeholder='Type something',\n", 204 | " description='String:',\n", 205 | " disabled=False\n", 206 | ")" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "metadata": {}, 212 | "source": [ 213 | "
" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "## Example - Interactive application of climate graphs" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "Required libraries:\n", 228 | "* [Ipyleaflet](https://ipyleaflet.readthedocs.io/en/latest/)\n", 229 | "* [Plotly](https://plot.ly/) for interactive visualization [Registration required]\n", 230 | "* [Widgets](https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20Basics.html)" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": null, 236 | "metadata": {}, 237 | "outputs": [], 238 | "source": [ 239 | "from ipyleaflet import Map, basemaps, basemap_to_tiles, FullScreenControl, Marker\n", 240 | "import ipyleaflet\n", 241 | "\n", 242 | "from IPython.display import display, clear_output\n", 243 | "import ipywidgets as widgets\n", 244 | "import numpy as np\n", 245 | "import ee\n", 246 | "\n", 247 | "import plotly.plotly as py\n", 248 | "import plotly.graph_objs as go" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "metadata": {}, 255 | "outputs": [], 256 | "source": [ 257 | "def GetTileLayerUrl(ee_image_object):\n", 258 | " map_id = ee.Image(ee_image_object).getMapId()\n", 259 | " tile_url_template = \"https://earthengine.googleapis.com/map/{mapid}/{{z}}/{{x}}/{{y}}?token={token}\"\n", 260 | " return tile_url_template.format(**map_id)" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": null, 266 | "metadata": {}, 267 | "outputs": [], 268 | "source": [ 269 | "ee.Initialize()" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": {}, 275 | "source": [ 276 | "### Load ERA5 monthly ImageCollection " 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "era5_monthly = ee.ImageCollection('projects/ecmwf/era5_monthly')" 286 | ] 287 | }, 288 | { 289 | "cell_type": "code", 290 | "execution_count": null, 291 | "metadata": {}, 292 | "outputs": [], 293 | "source": [ 294 | "era5_monthly.getInfo()" 295 | ] 296 | }, 297 | { 298 | "cell_type": "markdown", 299 | "metadata": {}, 300 | "source": [ 301 | "### Process mean precipitation for each month based on entire time series" 302 | ] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "execution_count": null, 307 | "metadata": {}, 308 | "outputs": [], 309 | "source": [ 310 | "era5_monthly_img = era5_monthly.limit(1).first()\n", 311 | "collection_img_proj = era5_monthly_img.select(0).projection()" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "metadata": {}, 318 | "outputs": [], 319 | "source": [ 320 | "months = range(1,13)\n", 321 | "\n", 322 | "# Store images in a list\n", 323 | "img_list = []\n", 324 | "for i in months:\n", 325 | " collection_filtered = era5_monthly.filter(ee.Filter.calendarRange(i,i, 'month'))\n", 326 | " collection_red = collection_filtered.reduce(ee.Reducer.mean())\n", 327 | " \n", 328 | " # if reducer function is applied to an image collection, the output does not have any projection information, as collection can contain \\ \n", 329 | " # images with different projection information. Thus, one can set the projection to each image\n", 330 | " collection_red_proj = collection_red.setDefaultProjection(collection_img_proj)\n", 331 | " img_list.append(collection_red_proj)\n", 332 | " \n", 333 | "img_list[0].getInfo()" 334 | ] 335 | }, 336 | { 337 | "cell_type": "markdown", 338 | "metadata": {}, 339 | "source": [ 340 | "### Build an Image Collection of the resulting image list" 341 | ] 342 | }, 343 | { 344 | "cell_type": "code", 345 | "execution_count": null, 346 | "metadata": {}, 347 | "outputs": [], 348 | "source": [ 349 | "meanMonths_collection = ee.ImageCollection.fromImages(img_list)\n", 350 | "meanMonths_collection.getInfo()" 351 | ] 352 | }, 353 | { 354 | "cell_type": "code", 355 | "execution_count": null, 356 | "metadata": {}, 357 | "outputs": [], 358 | "source": [ 359 | "t2m_tmp = meanMonths_collection.select('t2m_mean').first()\n", 360 | "tp_tmp = meanMonths_collection.select('tp_mean').first()" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": null, 366 | "metadata": {}, 367 | "outputs": [], 368 | "source": [ 369 | "img_t2m = GetTileLayerUrl(t2m_tmp.visualize(min=250, max=330, palette=['#000080','#0000D9','#4000FF','#8000FF','#0080FF','#00FFFF','#00FF80','#80FF00','#DAFF00','#FFFF00','#FFF500','#FFDA00','#FFB000','#FFA400','#FF4F00','#FF2500','#FF0A00','#FF00FF']))\n", 370 | "img_tp = GetTileLayerUrl(tp_tmp.visualize(min=0, max=1, palette=['#FFFFFF', '#00FFFF', '#0080FF', '#DA00FF', '#FFA400','#FF0000']))" 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": null, 376 | "metadata": {}, 377 | "outputs": [], 378 | "source": [ 379 | "def click(b):\n", 380 | " point = ee.Geometry.Point(lon,lat)\n", 381 | " tp_point = meanMonths_collection.select('tp_mean').getRegion(point,500).getInfo()\n", 382 | " t2m_point = meanMonths_collection.select('t2m_mean').getRegion(point,500).getInfo()\n", 383 | " \n", 384 | " header_tp = tp_point[0]\n", 385 | " data_tp = tp_point[1:]\n", 386 | " ydata_tp = [row[4]*1000 for row in data_tp]\n", 387 | " \n", 388 | " header_t2m = t2m_point[0]\n", 389 | " data_t2m = t2m_point[1:]\n", 390 | " ydata_t2m = [row[4]-273.2 for row in data_t2m]\n", 391 | "\n", 392 | " tp = go.Bar(\n", 393 | " x=['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun','Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'],\n", 394 | " y=ydata_tp,\n", 395 | " name='Total precipitation in mm',\n", 396 | " marker=dict(\n", 397 | " color='rgb(204,204,204)',\n", 398 | " ))\n", 399 | " \n", 400 | " t2m = go.Scatter(\n", 401 | " x=['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun','Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'],\n", 402 | " y=ydata_t2m,\n", 403 | " name=\"2m air temperature in deg C\",\n", 404 | " yaxis='y2')\n", 405 | "\n", 406 | " data = [tp,t2m]\n", 407 | " layout = go.Layout(\n", 408 | " title='Climate graph at location '+ str(round(lat,2)) + ' / '+ str(round(lon,2)) + ' (lat/lon)',\n", 409 | " yaxis=dict(\n", 410 | " title=\"Total precipitation in mm\"\n", 411 | " ),\n", 412 | " yaxis2=dict(\n", 413 | " title=\"2 m air temperature in degC\",\n", 414 | " overlaying='y',\n", 415 | " side='right',\n", 416 | " range=[0,max(ydata_t2m)+2]\n", 417 | " )\n", 418 | " )\n", 419 | "\n", 420 | " fig = go.Figure(data=data, layout=layout)\n", 421 | " with out:\n", 422 | " clear_output(wait=True)\n", 423 | " display(py.iplot(fig,filename='test'))\n" 424 | ] 425 | }, 426 | { 427 | "cell_type": "code", 428 | "execution_count": null, 429 | "metadata": {}, 430 | "outputs": [], 431 | "source": [ 432 | "map1 = ipyleaflet.Map(\n", 433 | " zoom=2,\n", 434 | " layout={'height':'500px'},\n", 435 | ")\n", 436 | "\n", 437 | "map1.add_layer(ipyleaflet.TileLayer(url=img_t2m))\n", 438 | "map1.add_layer(ipyleaflet.TileLayer(url=img_tp))\n", 439 | "map1.add_control(ipyleaflet.LayersControl())\n", 440 | "\n", 441 | "control = FullScreenControl()\n", 442 | "map1.add_control(control)\n", 443 | "\n", 444 | "map1" 445 | ] 446 | }, 447 | { 448 | "cell_type": "code", 449 | "execution_count": null, 450 | "metadata": {}, 451 | "outputs": [], 452 | "source": [ 453 | "def handle_click(**kwargs):\n", 454 | " if kwargs.get('type') == 'click':\n", 455 | " global lat, lon\n", 456 | " mark = ipyleaflet.Marker(location=kwargs.get('coordinates'))\n", 457 | " map1.add_layer(mark)\n", 458 | " location = mark.location\n", 459 | " lat, lon = location[0], location[1] \n", 460 | "\n", 461 | "map1.on_interaction(handle_click)" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "metadata": {}, 468 | "outputs": [], 469 | "source": [ 470 | "out=widgets.Output()\n", 471 | "button=widgets.Button(description='Plot climate graph')\n", 472 | "button.on_click(click)\n", 473 | "display(out)\n", 474 | "display(button)" 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": {}, 480 | "source": [ 481 | "
\n", 482 | "© 2019 | Julia Wagemann\n", 483 | "\"Creative" 484 | ] 485 | } 486 | ], 487 | "metadata": { 488 | "kernelspec": { 489 | "display_name": "Python 3", 490 | "language": "python", 491 | "name": "python3" 492 | }, 493 | "language_info": { 494 | "codemirror_mode": { 495 | "name": "ipython", 496 | "version": 3 497 | }, 498 | "file_extension": ".py", 499 | "mimetype": "text/x-python", 500 | "name": "python", 501 | "nbconvert_exporter": "python", 502 | "pygments_lexer": "ipython3", 503 | "version": "3.7.0" 504 | } 505 | }, 506 | "nbformat": 4, 507 | "nbformat_minor": 2 508 | } 509 | -------------------------------------------------------------------------------- /04_Jupyter_Environment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 4 Getting to know the Jupyter environment\n", 8 | "### nbviewer, Binder, JupyterLab, JupyterHub, ..." 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "
\n", 16 | "<< 3 - Introducing Jupyter widgets           Index >>" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## 1.1. Sharing static Jupyter Notebooks" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "* [nbviewer](https://nbviewer.jupyter.org/) \n", 31 | "* GitHub" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "## 1.2. Reproducible Jupyter Notebooks" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "### mybinder" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "\n", 53 | "[Binder](https://mybinder.org/) allows you to open notebooks hosted on a Git repo in an executable environment, making the code immediately reproducible by anyone, anywhere.\n", 54 | "\n", 55 | "Binder builds a Docker image of the repo where the notebooks are hosted." 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "### JupyterHub" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "\n", 70 | "[JupyterHub](https://jupyter.org/hub) brings the power of notebooks to groups of users. It gives users access to computational environments and resources without burdening the users with installation and maintenance tasks.\n", 71 | "\n", 72 | "Users - including students, researchers, and data scientists - can get their work done in their own workspaces on shared resources which can be managed efficiently by system administrators.\n", 73 | "\n", 74 | "JupyterHub runs in the cloud or on your own hardware, and makes it possible to serve a pre-configured data science environment to any user in the world. It is customizable and scalable, and is suitable for small and large teams, academic courses, and large-scale infrastructure.\n", 75 | "\n", 76 | "Two tutorials on how to set up your own JupyterHub are recommended:\n", 77 | "* [Zero to JupyterHub with Kubernetes](https://z2jh.jupyter.org/en/latest/)\n", 78 | "* [The littlest JupyterHub](https://tljh.jupyter.org/en/latest/)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "## 1.3 Jupyterlab - a Jupyter development environment" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "[JupyterLab](https://jupyterlab.readthedocs.io/en/stable/) is the next-generation web-based user interface for Project Jupyter. It allows you to work with documents and activities such as Jupyter notebooks, text editors, terminals, and custom components in a flexible, integrated, and extensible manner.\n", 93 | "\n", 94 | "\n", 95 | "" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "
\n", 103 | "© 2019 | Julia Wagemann\n", 104 | "\"Creative" 105 | ] 106 | } 107 | ], 108 | "metadata": { 109 | "kernelspec": { 110 | "display_name": "Python 3", 111 | "language": "python", 112 | "name": "python3" 113 | }, 114 | "language_info": { 115 | "codemirror_mode": { 116 | "name": "ipython", 117 | "version": 3 118 | }, 119 | "file_extension": ".py", 120 | "mimetype": "text/x-python", 121 | "name": "python", 122 | "nbconvert_exporter": "python", 123 | "pygments_lexer": "ipython3", 124 | "version": "3.7.0" 125 | } 126 | }, 127 | "nbformat": 4, 128 | "nbformat_minor": 2 129 | } 130 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Interactive Analysis of Big Earth Data with Jupyter Notebooks 2 | A short course given at EGU 2019 | 10 April 2019 3 | 4 | [Link to nbviewer](https://nbviewer.jupyter.org/github/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/index.ipynb) 5 | 6 | ## Outline 7 | * [Introducing Jupyter Notebooks - Data analysis made simple](https://github.com/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/01_Introduction_Jupyter_Notebooks.ipynb) 8 | * [Data access with Jupyter Notebooks from different data repositories](https://github.com/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/02_Geospatial_Data_Access.ipynb) 9 | * [Introducing Jupyter widgets - Make your data analysis and visualisation interactive](https://github.com/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/03_Introduction_Jupyter_widgets.ipynb) 10 | * [Jupyterlab, JupyterHub, ... - Getting to know the Jupyter environment](https://github.com/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/04_Jupyter_Environment.ipynb) 11 | 12 | 13 | ## At the end of this short course 14 | * you **hopefully have an idea about the benefits of Jupyter notebooks** and understand the Jupyter environment better 15 | * you got an **overview of different data platforms to retrieve large volumes of environmental data** (meteorological, climate, Earth Observation) 16 | * You got an **overview of different python libraries to efficiently access and process Big Earth Data** 17 | * You were shown some **options to make your analysis / visualization interactive** 18 | 19 | 20 | ## Required libraries: 21 | * [xarray](http://xarray.pydata.org/en/stable/) 22 | * [cdsapi](https://pypi.org/project/cdsapi/) (requires registration) 23 | * [Earth Engine Python API](https://github.com/google/earthengine-api) (requires registration) 24 | * [Plotly](https://github.com/plotly/plotly.py) (requires registration) 25 | * [widgets](https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20Basics.html) 26 | * [ipyleaflet](https://ipyleaflet.readthedocs.io/en/latest/) 27 | 28 | ## Links to open data repositories: 29 | * [Copernicus Climate Data Store](https://cds.climate.copernicus.eu) 30 | * [Google Earth Engine](https://earthengine.google.com/) 31 | * [Earth on AWS](https://aws.amazon.com/earth/) 32 | * [ECMWF Public datasets](https://apps.ecmwf.int/datasets/) 33 | -------------------------------------------------------------------------------- /era5_test.nc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/era5_test.nc -------------------------------------------------------------------------------- /img/2019_esowc_banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/2019_esowc_banner.png -------------------------------------------------------------------------------- /img/aws_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/aws_logo.png -------------------------------------------------------------------------------- /img/binder_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/binder_logo.png -------------------------------------------------------------------------------- /img/cds_logo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/cds_logo.jpg -------------------------------------------------------------------------------- /img/data_services.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/data_services.png -------------------------------------------------------------------------------- /img/gee_logo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/gee_logo.jpg -------------------------------------------------------------------------------- /img/gee_logo_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/gee_logo_2.png -------------------------------------------------------------------------------- /img/jupyterhub.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/jupyterhub.png -------------------------------------------------------------------------------- /img/jupyterlab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwagemann/2019_egu_workshop_jupyter_notebooks/eeafe0e029e4457e5c8f6464418b42c9400c47b4/img/jupyterlab.png -------------------------------------------------------------------------------- /index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "### Short course\n", 8 | "# Interactive analysis of Big Earth Data with Jupyter Notebooks" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "Short course created for EGU 2019
\n", 16 | "Wednesday, 10 April 2019, 16:15 - 17:55 | Room -2.62" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "### Presenter\n", 24 | "Julia Wagemann | PhD candidate at University of Marburg and Visiting Scientist at [ECMWF](https://ecmwf.int)\n", 25 | "\n", 26 | "Twitter: [@JuliaWagemann](https://twitter.com/JuliaWagemann)
\n", 27 | "GitHub: [@jwagemann](https://github.com/jwagemann)\n", 28 | "\n", 29 | "### Short course material\n", 30 | "On GitHub: [https://github.com/jwagemann/2019_egu_workshop_jupyter_notebooks](https://github.com/jwagemann/2019_egu_workshop_jupyter_notebooks)
\n", 31 | "View it with Nbviewer: [https://nbviewer.jupyter.org/github/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/index.ipynb](https://nbviewer.jupyter.org/github/jwagemann/2019_egu_workshop_jupyter_notebooks/blob/master/index.ipynb) \n", 32 | "
" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "## Outline" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "
\n", 47 | "\n", 48 | "|Time|Content|\n", 49 | "|---------|-----------|\n", 50 | "|16:15 - 16:35| [Introducing Jupyter Notebooks - Data analysis made simple](./01_Introduction_Jupyter_Notebooks.ipynb) |\n", 51 | "|16:35 - 17:05 | [Data access with Jupyter Notebooks from different data repositories](./02_Geospatial_Data_Access.ipynb) |\n", 52 | "|17:05 - 17:35 | [Introducing Jupyter widgets - Make your data analysis and visualisation interactive](./03_Introduction_Jupyter_widgets.ipynb) |\n", 53 | "|17:35 - 17:55 | [Jupyterlab, JupyterHub, ... - Getting to know the Jupyter environment](./04_Jupyter_Environment.ipynb)|\n", 54 | "\n", 55 | "
" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "## At the end of this short course" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "* you hopefully have an idea about the **benefits of Jupyter notebooks** and understand the Jupyter environment better\n", 70 | "* you got an **overview of different data platforms to retrieve large volumes of environmental data** (meteorological, climate, Earth Observation)\n", 71 | "* You got an **overview of different python libraries to efficiently access and process Big Earth Data**\n", 72 | "* You were shown some **options to make your analysis / visualization interactive**" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "
" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "## Python developers - Come and team up with ECMWF mentors during the summer " 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "\n", 94 | "\n", 95 | "
\n", 96 | "\n", 97 | "#### More information\n", 98 | "* [GitHub](https://github.com/esowc/challenges_2019) - https://github.com/esowc/challenges_2019\n", 99 | "* [Website](https://www.ecmwf.int/en/learning/workshops/ecmwf-summer-weather-code-2019) - https://www.ecmwf.int/en/learning/workshops/ecmwf-summer-weather-code-2019\n", 100 | "\n", 101 | "**Deadline for proposal submission: 21 April 2019**" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "
" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "## Further resources" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "#### Jupyter \n", 123 | "* Project Jupyter website\n", 124 | "* Why Jupyter is Data Scientist's computational notebook of choice\n", 125 | "\n", 126 | "\n", 127 | "#### Data platforms\n", 128 | "* [Copernicus Climate Data Store](https://cds.copernicus.eu)\n", 129 | "* [Google Earth Engine](https://earthengine.google.com)\n", 130 | "* [Earth on AWS](https://aws.amazon.com/earth/) " 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "
\n", 138 | "© 2019 | Julia Wagemann\n", 139 | "\"Creative" 140 | ] 141 | } 142 | ], 143 | "metadata": { 144 | "kernelspec": { 145 | "display_name": "Python 3", 146 | "language": "python", 147 | "name": "python3" 148 | }, 149 | "language_info": { 150 | "codemirror_mode": { 151 | "name": "ipython", 152 | "version": 3 153 | }, 154 | "file_extension": ".py", 155 | "mimetype": "text/x-python", 156 | "name": "python", 157 | "nbconvert_exporter": "python", 158 | "pygments_lexer": "ipython3", 159 | "version": "3.7.0" 160 | } 161 | }, 162 | "nbformat": 4, 163 | "nbformat_minor": 2 164 | } 165 | --------------------------------------------------------------------------------