├── dutchrain.gif ├── README.md └── knmi-precip-radar-script.ipynb /dutchrain.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/berthubert/knmi-precipitation-radar-scripts/main/dutchrain.gif -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | In this repository I've deposited some rough instructions to get you started 2 | using the KNMI precipitation radar data. KNMI is the Dutch national weather 3 | service, and they do a lot of cool research. In addition, they publish a ton 4 | of open data. 5 | 6 | > Many thanks to [Russ Garrett](https://russ.garrett.co.uk/) who helped me 7 | > get started with HDF5 image data. His explanation how to retrieve NASA DSCOVR EPIC 8 | > images and data is [a thing of 9 | > beauty](https://russss.github.io/epic-image-processing/index.html) 10 | 11 | # The data 12 | "Gridded files of radar reflectivities at 1500 m over the Netherlands and 13 | surrounding area measured by two radars in Herwijnen and Den Helder. Time 14 | interval is 5 minutes." 15 | 16 | The dataset is described [here on the relevant KNMI open data 17 | page](https://dataplatform.knmi.nl/dataset/radar-reflectivity-composites-2-0). 18 | Some more in-depth stuff, in Dutch only, [is 19 | here](https://www.knmidata.nl/data-services/knmi-producten-overzicht/radar-producten/data-product-1). 20 | 21 | KNMI publishes [many more 22 | datasets](https://dataplatform.knmi.nl/organization/knmi) as well. 23 | 24 | There is also information on [how to access the data through the 25 | API](https://developer.dataplatform.knmi.nl/). Some functionality requires 26 | an [API 27 | key](https://developer.dataplatform.knmi.nl/get-started#obtain-an-api-key), 28 | which the system can generate for you in realtime, but you can also use two 29 | generic keys provided there. 30 | 31 | # Actually accessing radar reflectivity data 32 | This is a two-step process. First, get a KNMI API key, or use one of the 33 | generic keys, and put it in `knmi.key`. 34 | 35 | Then run something like: 36 | 37 | ``` 38 | wget --header "Authorization:$(cat knmi.key)" \ 39 | https://api.dataplatform.knmi.nl/open-data/v1/datasets/radar_reflectivity_composites/versions/2.0/files/RAD_NL25_PCP_NA_202208151620.h5/url 40 | ``` 41 | 42 | There is a new file every 5 minutes, and the timestamp '202208151620' is the 43 | 15th of August 2022, 16:20 UTC. This call requests the URL to actually retrieve 44 | this file, and this is returned as JSON, in the field 45 | `temporaryDownloadUrl`. You could retrieve the actual file like this (if 46 | you have `wget` and `jq`): 47 | 48 | ``` 49 | wget $(jq -r .temporaryDownloadUrl < url) 50 | ``` 51 | 52 | # HDF5 53 | The radar reflectivity data is not "just a picture", it is a HDF5 file, a 54 | format often used for scientific or space data. HDF5 is complex stuff, and 55 | it can do many things. 56 | 57 | If you are into Python, [h5py](https://www.h5py.org/) works well with the 58 | KNMI data and with Matplotlib. 59 | 60 | In [knmi-precip-radar-script.ipynb](knmi-precip-radar-script.ipynb) you'll find a simple 61 | Jupyter notebook that reads an 62 | hour of data from the KNMI API using the operational test key, and turns 63 | that into this fun animation: 64 | 65 | ![](dutchrain.gif) 66 | 67 | -------------------------------------------------------------------------------- /knmi-precip-radar-script.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "id": "aeb3a321", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "%matplotlib notebook\n", 11 | "%precision 3\n", 12 | "\n", 13 | "from pandas.plotting import register_matplotlib_converters\n", 14 | "register_matplotlib_converters()\n", 15 | "\n", 16 | "import matplotlib\n", 17 | "import matplotlib.pyplot as plt\n", 18 | "import datetime\n", 19 | "import numpy as np\n", 20 | "from matplotlib.animation import FuncAnimation\n", 21 | "from datetime import timedelta\n", 22 | "import h5py\n", 23 | "from matplotlib.pyplot import imshow\n", 24 | "import matplotlib.animation as animation\n", 25 | "h5py.enable_ipython_completer()\n", 26 | "from matplotlib import cm\n", 27 | "from matplotlib.colors import ListedColormap, LinearSegmentedColormap\n", 28 | "import urllib.parse\n", 29 | "import urllib.request\n", 30 | "import json\n", 31 | "import shutil" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "id": "8c512e2e", 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "# KNMI operational test key from https://developer.dataplatform.knmi.nl/get-started#make-api-calls\n", 42 | "key='eyJvcmciOiI1ZTU1NGUxOTI3NGE5NjAwMDEyYTNlYjEiLCJpZCI6IjI4ZWZlOTZkNDk2ZjQ3ZmE5YjMzNWY5NDU3NWQyMzViIiwiaCI6Im11cm11cjEyOCJ9'\n", 43 | " \n", 44 | "\n", 45 | "def getRadarData(key, tstamp):\n", 46 | " url = 'https://api.dataplatform.knmi.nl/open-data/v1/datasets/radar_reflectivity_composites/versions/2.0/files/RAD_NL25_PCP_NA_'+tstamp+'.h5/url'\n", 47 | " headers = {'Authorization': key}\n", 48 | "\n", 49 | " req = urllib.request.Request(url, headers=headers)\n", 50 | " with urllib.request.urlopen(req) as response:\n", 51 | " meta = response.read()\n", 52 | " \n", 53 | " realurl=json.loads(meta)[\"temporaryDownloadUrl\"]\n", 54 | " req = urllib.request.Request(realurl)\n", 55 | " fname=tstamp+\".hf5\"\n", 56 | " print(fname)\n", 57 | " with urllib.request.urlopen(req) as response:\n", 58 | " with open(fname, 'wb') as location:\n", 59 | " shutil.copyfileobj(response, location)\n", 60 | " \n" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "id": "1f19f226", 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "now=datetime.datetime.utcnow()\n", 71 | "now = now - datetime.timedelta(hours=1, minutes=5)\n", 72 | "now -= datetime.timedelta(minutes=now.minute%5)\n", 73 | "\n", 74 | "now.strftime(\"%Y%m%d%H%M\")\n", 75 | "start=now\n", 76 | "files=[]\n", 77 | "for n in range(0,12):\n", 78 | " tstamp=start.strftime(\"%Y%m%d%H%M\")\n", 79 | " files.append(tstamp+\".hf5\")\n", 80 | " getRadarData(key, tstamp)\n", 81 | " start += datetime.timedelta(minutes=5)" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "id": "25948a6b", 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "img = h5py.File(files[0])\n", 92 | "list(img) # show what is in there" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "id": "a46c7e82", 99 | "metadata": { 100 | "scrolled": false 101 | }, 102 | "outputs": [], 103 | "source": [ 104 | "# read the first file\n", 105 | "img = h5py.File(files[0])\n", 106 | "# build the KNMI suggested palette\n", 107 | "cmap=np.array(img[\"visualisation1\"][\"color_palette\"])\n", 108 | "knmimap=ListedColormap(cmap/256.0)\n", 109 | "\n", 110 | "# Show the thing\n", 111 | "plt.figure()\n", 112 | "plt.imshow(img[\"image1\"][\"image_data\"], cmap=knmimap)" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "id": "b26b6b46", 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "# Now let's make an animation!\n", 123 | "\n", 124 | "fig, ax = plt.subplots()\n", 125 | "\n", 126 | "def update(fname):\n", 127 | " # clear the axis each frame\n", 128 | " ax.clear()\n", 129 | " ax.set_xlabel(\"KM\")\n", 130 | " ax.set_ylabel(\"KM\")\n", 131 | " ax.grid()\n", 132 | " img = h5py.File(fname)\n", 133 | " cmap=np.array(img[\"visualisation1\"][\"color_palette\"])\n", 134 | " knmimap=ListedColormap(cmap/256.0)\n", 135 | " \n", 136 | " ax.imshow(img[\"image1\"][\"image_data\"], cmap=knmimap)\n", 137 | " # replot things\n", 138 | " ax.set_title(\"KNMI precipitation radar data from \"+fname)\n", 139 | " \n", 140 | " \n", 141 | "ani = animation.FuncAnimation(fig, update, frames=files, interval=500)\n", 142 | "ani.save('dutchrain.gif', writer='imagemagick', fps=2)\n" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "id": "ae435785", 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "# scan a little deeper in a file\n", 153 | "for a in list(img):\n", 154 | " print(a,\": \",list(img[a]))" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "id": "4cddf4a7", 161 | "metadata": {}, 162 | "outputs": [], 163 | "source": [ 164 | "list(img[\"visualisation1\"][\"color_palette\"].attrs.items())" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "id": "d5532cf8", 171 | "metadata": { 172 | "scrolled": false 173 | }, 174 | "outputs": [], 175 | "source": [ 176 | "arr = np.ma.fix_invalid(np.array(img[\"image1\"]['image_data']), fill_value=0)\n", 177 | "plt.figure()\n", 178 | "plt.hist(arr.ravel(), bins=255)" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "id": "13fa4faf", 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [] 188 | } 189 | ], 190 | "metadata": { 191 | "kernelspec": { 192 | "display_name": "Python 3 (ipykernel)", 193 | "language": "python", 194 | "name": "python3" 195 | }, 196 | "language_info": { 197 | "codemirror_mode": { 198 | "name": "ipython", 199 | "version": 3 200 | }, 201 | "file_extension": ".py", 202 | "mimetype": "text/x-python", 203 | "name": "python", 204 | "nbconvert_exporter": "python", 205 | "pygments_lexer": "ipython3", 206 | "version": "3.10.4" 207 | } 208 | }, 209 | "nbformat": 4, 210 | "nbformat_minor": 5 211 | } 212 | --------------------------------------------------------------------------------