├── README.md ├── session_1 ├── data │ ├── AKL_aero_rain_monthly.xlsx │ ├── Daily_clim_data_Auckland.csv │ ├── ISO_datetime.xls │ ├── Monthly_clim_data_Auckland.csv │ ├── Monty_Python.mp4 │ ├── NCEP2_airtemp2m_2014_anoms.nc │ ├── NIWA_SOI.csv │ ├── SOI_time_series.xlsx │ ├── ascii_table.txt │ ├── clusters_monthly.mat │ ├── ersst3b.nino.mth.81-10.ascii │ ├── rainfall_calibration.xlsx │ ├── random_2darray.mat │ ├── soi_nino.csv │ └── year_month_day.xlsx └── notebooks │ ├── IPython_notebook.html │ ├── IPython_notebook.ipynb │ ├── Jupyter_notebook.html │ ├── Jupyter_notebook.ipynb │ ├── Matplotlib_Basemap.html │ ├── Matplotlib_Basemap.ipynb │ ├── Numpy.html │ ├── Numpy.ipynb │ ├── Pandas.html │ ├── Pandas.ipynb │ ├── Scipy.html │ ├── Scipy.ipynb │ ├── images │ ├── array_3x5x8.png │ ├── broadcast_2D.png │ ├── broadcast_scalar.png │ ├── ipython_console.png │ ├── ipython_dashboard.png │ ├── lost_wormhole.jpg │ ├── poisson_pmf.png │ ├── presentation.png │ ├── python.png │ ├── python_console.png │ ├── qtconsole.png │ ├── split-apply-combine.png │ └── www.png │ ├── introduction_python.html │ ├── introduction_python.ipynb │ ├── load_style.py │ ├── print_upper.py │ ├── resources.html │ ├── resources.ipynb │ ├── talk.css │ ├── talktools.py │ ├── test.html │ └── test.ipynb └── session_2 ├── data ├── 3B42RT_daily.2014.07.23.nc ├── Daily_clim_data_Auckland.csv ├── ISO_datetime.xls ├── NIWA_SOI.csv ├── SOI_time_series.xlsx ├── ersst3b.nino.mth.81-10.ascii ├── soi_nino.csv ├── sst_nino3.dat └── year_month_day.xlsx └── notebooks ├── IPython_widgets.html ├── IPython_widgets.ipynb ├── Interactive_plots.html ├── Interactive_plots.ipynb ├── Numpy.ipynb ├── bearcart.html ├── conda.html ├── conda.ipynb ├── data.json ├── figure.png ├── images ├── clim_sst.png ├── seas_clim_sst.png └── split-apply-combine.png ├── iris_cartopy.ipynb ├── load_style.py ├── load_style.pyc ├── log_lines.html ├── map_notebook.py ├── map_notebook.pyc ├── mpld3_figure.html ├── python.ipynb ├── rickshaw.min.css ├── rickshaw.min.js ├── runipy.ipynb ├── sin.html ├── sklearn.html ├── sklearn.ipynb ├── sklearn_EOF_decomposition.html ├── sklearn_EOF_decomposition.ipynb ├── sklearn_kmeans.html ├── sklearn_kmeans.ipynb ├── talk.css ├── talktools.py ├── talktools.pyc ├── wavelet_analysis.ipynb ├── xray.html └── xray.ipynb /README.md: -------------------------------------------------------------------------------- 1 | #Python for Data Analysis and Visualisation 2 | 3 | [NIWA](http://www.niwa.co.nz), Wellington, Wednesday 22 and Thursday 23 April 2015 4 | 5 | Contact: 6 | 7 | Nicolas Fauchereau 8 | 9 | + [@gmail](mailto:Nicolas.Fauchereau@gmail.com) 10 | + [@niwa](mailto:Nicolas.Fauchereau@niwa.co.nz) 11 | 12 |
13 | 14 | ### Table of contents 15 | 16 | - [The Anaconda python distribution](#the-anaconda-python-distribution) 17 | - [Installation of Some additional libraries](#installation-of-additional-libraries) 18 | - [Basemap](#basemap) 19 | - [Bokeh](#bokeh) 20 | - [Seaborn](#seaborn) 21 | - [mplD3](#mplD3) 22 | - [bearcart](#bearcart) 23 | - [folium](#folium) 24 | - [Running the IPython notebooks](#running-the-ipython-notebooks) 25 | - [Troubleshooting](#troubleshooting) 26 | - [Rendered notebooks](#rendered-notebooks) 27 | 28 |
29 | 30 | ## The Anaconda python distribution 31 | 32 | For this tutorial, I **strongly** recommend installing the **Anaconda Python distribution**. It is a completely free enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing. It includes the python interpreter itself, the python standard library as well as a set of packages exposing data structures and methods for data manipulation and scientific computing and visualization. In particular it provides [Numpy](http://www.numpy.org/), [Scipy](http://www.scipy.org/), [Pandas](http://pandas.pydata.org/), [Matplotlib](http://matplotlib.org/), [scikit-learn](http://scikit-learn.org/stable/) and [statmodels](http://statsmodels.sourceforge.net/), i.e. all the main packages we will be using during the tutorial. The full list of packages is available at: 33 | 34 | [http://docs.continuum.io/anaconda/pkgs.html](http://docs.continuum.io/anaconda/pkgs.html) 35 | 36 | The Anaconda python distribution (**NOTE**: select the version shippint with Python 3.5, as of 8 April 2016, Anaconda version 4.0) must be downloaded from: 37 | 38 | [http://continuum.io/downloads](http://continuum.io/downloads) 39 | 40 | For your platform. 41 | 42 | Once you have installed Anaconda, you can update to the latest compatible versions of all the pre-installed packages by running: 43 | 44 | ``` 45 | $ conda update conda 46 | ``` 47 | 48 | Then 49 | 50 | ``` 51 | $ conda update anaconda 52 | ``` 53 | 54 | In a terminal. 55 | 56 | You also need to install [pip](https://github.com/pypa/pip) to install packages from the [Python Package Index](http://pypi.python.org/pypi). 57 | 58 | ``` 59 | $ conda install pip 60 | ``` 61 | 62 | ## Installation of additional libraries 63 | 64 | ### netcdf4 65 | 66 | [netcdf4](https://github.com/Unidata/netcdf4-python) allows you to read and write netcdf files (version 3 and 4 supported), install it by: 67 | 68 | ``` 69 | $ conda install netcdf4 70 | ``` 71 | 72 | ### Basemap 73 | 74 | **Basemap** is a graphic library for plotting (static, publication quality) geographical maps (see [http://matplotlib.org/basemap/](http://matplotlib.org/basemap/)). **Basemap** is available directly in **Anaconda** using the conda package manager, install with: 75 | 76 | ``` 77 | $ conda install basemap 78 | ``` 79 | 80 | ### Bokeh 81 | 82 | [Bokeh]() is a new interactive plotting library developed by the team behind **anaconda**: it is thus installable with conda (if not already installed): 83 | 84 | ``` 85 | $ conda install bokeh 86 | ``` 87 | 88 | ### Seaborn 89 | 90 | [seaborn](http://web.stanford.edu/~mwaskom/software/seaborn/) is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics. You should be able to install it with ```conda``` as well: 91 | 92 | ``` 93 | $ conda install seaborn 94 | ``` 95 | 96 | ### mplD3 97 | 98 | [mplD3](http://mpld3.github.io/) aims at *bringing matplotlib to the browser*. It has been developed by Jake VanDerPlas. It is installable using ```pip```: 99 | 100 | ``` 101 | $ pip install mpld3 102 | ``` 103 | 104 | ### bearcart 105 | 106 | [bearcart](https://github.com/wrobstory/bearcart) has been developed by [Rob Story](http://wrobstory.github.io/) and provides an interface to the rickshaw JavaScript library. It is also installable via ```pip```: 107 | 108 | ``` 109 | $ pip install bearcart 110 | ``` 111 | 112 | ### folium 113 | 114 | [folium](https://github.com/wrobstory/folium) has been also been developed by [Rob Story](http://wrobstory.github.io/) to provide an interface to the [leaflet.js](http://leafletjs.com/) JavaScript mapping library. Install with: 115 | 116 | ``` 117 | $ pip install folium 118 | ``` 119 | 120 |
121 | 122 | ### xarray 123 | 124 | [xarray](https://github.com/xarray/xarray) (previously *xray*) is a library aimed at bringing the power of Pandas to multidimensional labelled arrays, such as the ones usually associated with geophysical quantities varying along time and space dimensions (e.g. [time, latitudes, longitudes], [time, level, latitudes, longitudes], etc) and supports reading and writing netcdf files. It can be installed via `conda`: 125 | 126 | ``` 127 | $ conda install xarray 128 | ``` 129 | 130 | ## Running the Jupyter notebooks 131 | 132 | The material of the tutorial is in the form of [Jupyter notebooks](http://ipython.org/notebook.html). In a nutshell a Jupyter notebook is a web-based (i.e. running in the browser) interactive computational environment where you can combine Python code execution, text, mathematics, plots and rich media into a single document, which makes it an ideal medium for teaching and exploring code. 133 | 134 | 135 | After uncompressing the archive of the repo (or after cloning it with ```git```), navigate to the corresponding directory (containing the ```*.ipynb``` files, e.g. `session_1/notebooks`) and type: 136 | 137 | ``` 138 | $ jupyter notebook 139 | ``` 140 | 141 | That should bring up the Jupyter notebook dashboard (looking as below), you should be ready to go ! 142 | 143 | ![](http://nbviewer.ipython.org/github/nicolasfauchereau/Python-for-data-analysis-and-visualisation/blob/master/session_1/notebooks/images/ipython_dashboard.png) 144 | 145 | You should see in particular a ```test.ipynb``` notebook: please run it to make sure all the necessary libraries have been installed correctly. If you followed the instructions above (install the [anaconda python distribution](http://continuum.io/downloads)) it should be fine, this test notebook is mostly intended for those who have a *custom* python installation. 146 | 147 | ## Troubleshooting 148 | 149 | You might run into some problems installing additional libraries via `conda` or `pip` and / or running the IPython notebooks, especially on Windows machines behind a proxy, here are a few solutions that may work: 150 | 151 | **1. Proxy settings for conda:** 152 | 153 | create a `.condarc` file (the '.' is important) in your HOME directory (on windows it should be `C:\Users\YOU`) and add the following lines: 154 | 155 | ``` 156 | proxy_servers: 157 | http: http://url:port 158 | https: http://url:port 159 | ``` 160 | 161 | **2. specify proxy when using pip** 162 | 163 | If you are running into issues installing libraries via pip, try specifying the proxy to use at the command line, e.g. 164 | 165 | ``` 166 | pip install --proxy=http://url:port bearcart 167 | ``` 168 | 169 | **3. Set-up system-wide proxy settings** 170 | 171 | + On Macs: in your `${HOME}/.bash_profile`, insert these lines 172 | 173 | ``` 174 | export http_proxy=http://url:port 175 | export https_proxy=http://url:port 176 | 177 | ``` 178 | 179 | + On Linux machines, do the same as above in your `${HOME}/.bashrc` 180 | 181 | + On Windows machines: 182 | 183 | + As an administrator go to `Control Panel | System | Advanced Systems Settings | Advanced Tab | Environment Variables | System Variables | New` and set 184 | 185 | ``` 186 | HTTP_PROXY=http://url:port/ 187 | HTTPS_PROXY=https://url:port/ 188 | ``` 189 | 190 | + You can also do that in a command window by typing (the `$` represents the prompt) 191 | 192 | ``` 193 | $ SET HTTP_PROXY=http://url:port/ 194 | $ SET HTTPS_PROXY=http://url:port/ 195 | ``` 196 | 197 | **4. use Firefox instead of internet explorer to open the notebooks** 198 | 199 | The IPython notebook is an interactive web-based 'notebook', where executable python code can be weaved with rich comments, graphic outputs etc, which make it ideal for presenting interactive tutorials. When (in a command prompt) you navigate to the directory where you have downloaded the notebooks and type (the $ sign represent the prompt): 200 | 201 | ``` 202 | $ ipython notebook 203 | ``` 204 | 205 | a 'dashboard' with the list of notebooks should come up in your browser ... now if you are on windows, chances are that your default browser is Internet Explorer, which is generally bad news. If you encounter problems (blank page, notebooks not loading, kernel interruptions etc), it's probably because of Internet Explorer. What I suggest is that you download Firefox for windows and make it the default browser to open IPython notebooks. To do that you need (once firefox is installed) to do the following : 206 | 207 | i) in a command prompt type (again $ is the prompt): 208 | 209 | ``` 210 | $ ipython profile create default 211 | ``` 212 | 213 | it should create a bunch of configuration files in the following directory: 214 | 215 | ``` 216 | C:\Users\YOU\.ipython\profile_default 217 | ``` 218 | 219 | go and edit the `ipython_notebook_config.py` file 220 | 221 | search for the line 222 | 223 | ``` 224 | #c.NotebookApp.browser ='' 225 | ``` 226 | 227 | and replace it by: 228 | 229 | ``` 230 | import webbrowser 231 | webbrowser.register('firefox', None, webbrowser.GenericBrowser('C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe')) 232 | c.NotebookApp.browser = 'firefox' 233 | ``` 234 | 235 | **5. Specify localhost when calling the IPython notebook** 236 | 237 | On some configurations, you might also need to call: 238 | 239 | ``` 240 | $ ipython notebook --ip=127.0.0.1 241 | ``` 242 | 243 | To specify that the browser should connect to *localhost* 244 | 245 | **6. Clear the cache** 246 | 247 | If you are still running into issues (notably dashboard or IPython notebook not displaying correctly), try clearing the cache of your browser 248 | 249 | **7. Use an `incognito` window** 250 | 251 | If all else fails (!), one thing that has been reported working is: 252 | 253 | + launch the `ipython notebook` in no-browser mode: 254 | 255 | ``` 256 | ipython notebook --no-browser 257 | ``` 258 | 259 | You should see an output in the terminal looking like: 260 | 261 | ``` 262 | ... 263 | The IPython Notebook is running at: http://localhost:8888/ 264 | ... 265 | ``` 266 | 267 | Note that the URL and port could be different in your case. 268 | 269 | Open an `incognito` window from your browser and copy the URL (`http://localhost:8888/`) in the address bar 270 | 271 | ## Rendered notebooks 272 | 273 | **SESSION 1** 274 | 275 | + [resources and acknowledgments](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/resources.html) 276 | 277 | + [IPython notebook 278 | overview](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/IPython_notebook.html) 279 | 280 | + [Python language basics](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/introduction_python.html) 281 | 282 | + [Numpy](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/Numpy.html) 283 | 284 | + [Scipy](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/Scipy.html) 285 | 286 | + [Matplotlib and Basemap](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/Matplotlib_Basemap.html) 287 | 288 | + [Pandas](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_1/notebooks/Pandas.html) 289 | 290 | **SESSION 2** 291 | 292 | + [A brief intro to the IPython notebook widgets](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/IPython_widgets.html) 293 | 294 | + [Interactive plots in the browser](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/Interactive_plots.html) 295 | 296 | + [managing your packages and environments with conda](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/conda.html) 297 | 298 | + [xray](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/xray.html) 299 | 300 | + [brief intro to scikit-learn](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/sklearn.html) 301 | 302 | + [EOF decomposition of Pacific SSTs with scikit-learn](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/sklearn_EOF_decomposition.html) 303 | 304 | + [k-means clustering of Pacific SSTs with scikit-learn](https://cdn.rawgit.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/master/session_2/notebooks/sklearn_kmeans.html) 305 | -------------------------------------------------------------------------------- /session_1/data/AKL_aero_rain_monthly.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/AKL_aero_rain_monthly.xlsx -------------------------------------------------------------------------------- /session_1/data/ISO_datetime.xls: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/ISO_datetime.xls -------------------------------------------------------------------------------- /session_1/data/Monty_Python.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/Monty_Python.mp4 -------------------------------------------------------------------------------- /session_1/data/NCEP2_airtemp2m_2014_anoms.nc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/NCEP2_airtemp2m_2014_anoms.nc -------------------------------------------------------------------------------- /session_1/data/NIWA_SOI.csv: -------------------------------------------------------------------------------- 1 | Year,Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec 2 | 1876,1.1,1.0,0.1,1.0,0.8,1.8,-0.5,1.3,1.0,-0.8,-0.3,-0.4 3 | 1877,-0.9,-0.6,-0.4,-0.7,0.5,-1.6,-1.0,-0.8,-1.8,-1.5,-1.3,-1.4 4 | 1878,-0.8,-1.9,-1.5,-0.7,0.3,-0.2,1.7,1.3,1.7,1.1,1.4,1.7 5 | 1879,1.3,1.3,1.3,1.2,0.3,1.7,2.3,2.3,1.8,1.5,0.9,-0.6 6 | 1880,1.1,0.7,1.4,0.6,1.3,1.0,0.2,1.5,0.7,0.5,0.6,-0.3 7 | 1881,-0.7,-0.5,0.2,0.1,-0.3,-0.4,-0.5,-1.1,-1.4,-2.3,0.6,0.9 8 | 1882,-0.6,-0.1,0.5,0.2,0.8,-1.1,-2.1,-2.5,-1.5,-0.2,0.2,0.9 9 | 1883,0.6,0.8,-2.4,1.4,1.5,0.4,-1.0,0.2,-0.9,0.5,0.4,-1.6 10 | 1884,-1.2,-0.5,1.0,-1.2,0.2,1.0,-0.3,-0.5,-0.7,0.4,-0.2,-1.4 11 | 1885,-1.6,0.1,0.5,0.1,-0.3,-1.4,-0.5,-0.9,-0.4,-1.7,-1.6,0.4 12 | 1886,-0.0,0.1,0.3,0.5,0.7,0.6,0.8,1.4,1.3,1.3,0.9,1.4 13 | 1887,1.2,1.0,1.0,1.0,-0.3,0.6,0.5,0.5,0.4,0.5,-0.6,0.4 14 | 1888,-0.3,-0.2,-1.1,-2.0,-0.8,-1.5,-1.7,-0.8,-1.0,-1.4,-1.3,-0.3 15 | 1889,-2.5,-0.2,-2.6,0.1,-0.1,2.3,0.2,0.2,1.0,0.4,2.2,2.1 16 | 1890,2.1,1.0,1.4,0.7,0.5,0.7,-0.2,-0.3,0.9,0.4,0.2,-0.0 17 | 1891,1.6,-0.3,-0.9,0.5,0.1,-0.1,-0.6,-0.8,-1.1,0.1,-0.5,-0.5 18 | 1892,0.3,-0.9,1.1,0.7,1.1,2.1,0.8,0.6,0.6,0.8,-0.1,0.3 19 | 1893,1.1,0.7,-0.1,0.2,-0.2,1.2,1.5,0.8,0.5,0.8,0.2,0.1 20 | 1894,1.8,0.9,0.6,-0.1,-0.4,-0.1,-0.2,-0.5,-0.2,0.2,0.6,-0.1 21 | 1895,0.6,0.2,0.0,-0.5,-0.7,-0.4,0.0,-0.6,-0.4,-0.5,-0.9,-0.4 22 | 1896,0.2,0.4,-0.6,-0.7,-4.0,-3.0,-2.1,-2.2,-1.9,-1.8,-1.2,-1.5 23 | 1897,-1.2,-0.7,-1.6,-1.5,-1.6,0.1,-0.2,0.1,-0.0,0.2,-0.8,0.9 24 | 1898,0.7,0.5,1.9,1.1,-0.1,-0.1,0.7,0.2,0.3,-0.1,-0.3,-0.1 25 | 1899,1.3,0.8,1.4,0.5,-0.6,-1.0,-0.5,-1.0,-0.2,0.6,1.5,-0.4 26 | 1900,-0.7,-0.6,-2.4,-1.5,-0.6,2.7,1.1,0.8,-1.7,-1.7,-0.7,-0.6 27 | 1901,0.0,0.2,1.0,0.5,0.1,2.1,1.5,1.0,-1.6,-2.1,-0.9,-0.3 28 | 1902,1.7,-0.2,1.2,0.8,0.9,0.4,0.2,-0.8,-1.8,-0.7,-0.4,-0.4 29 | 1903,-0.9,-0.9,1.8,1.7,0.9,0.0,0.7,0.1,0.8,0.4,0.0,1.5 30 | 1904,1.4,1.4,1.0,2.9,1.0,-0.6,-0.9,0.1,-0.0,0.1,-1.7,0.2 31 | 1905,-0.9,-1.5,-2.9,-3.7,-3.6,-3.1,-2.1,-0.7,-0.7,-0.5,-1.8,-1.4 32 | 1906,-0.3,-0.7,-0.5,-0.7,0.2,-0.3,0.7,1.6,1.8,0.9,2.0,0.4 33 | 1907,0.5,0.1,0.0,0.5,1.1,0.9,-0.4,-0.8,-0.0,0.1,-0.3,0.8 34 | 1908,-1.0,0.7,0.1,1.6,0.0,-0.1,0.3,0.6,1.7,0.8,0.2,-0.6 35 | 1909,-0.2,-0.3,0.0,-1.2,0.3,2.4,1.1,1.0,0.0,0.4,0.8,0.4 36 | 1910,0.6,1.3,1.3,0.6,0.2,2.3,2.1,1.0,1.5,1.0,1.8,1.5 37 | 1911,0.3,0.1,0.4,0.3,-0.7,-1.1,-1.3,-1.2,-0.9,-1.1,-0.8,-0.2 38 | 1912,-0.9,-1.6,-0.8,-1.8,-1.2,-0.5,0.0,-0.7,-0.4,-0.8,0.2,-0.9 39 | 1913,-0.3,-0.5,0.2,-0.4,-0.7,-0.3,-0.1,-0.7,-1.0,-0.9,-1.2,-0.8 40 | 1914,-0.5,0.2,1.0,-1.2,0.1,-1.6,-1.8,-1.7,-1.3,-0.8,-1.2,-0.2 41 | 1915,-2.1,-0.2,-1.9,-1.5,-1.1,0.8,1.5,0.8,0.7,0.2,-1.5,0.9 42 | 1916,0.6,-0.3,-0.6,0.1,0.8,1.0,2.7,1.6,0.4,0.6,0.9,1.5 43 | 1917,0.5,0.9,1.8,2.1,2.3,2.2,2.9,3.5,2.9,1.5,2.0,2.2 44 | 1918,1.5,1.5,-0.1,1.6,1.1,-0.4,-1.4,-0.4,-0.9,-0.5,0.0,-0.9 45 | 1919,-1.4,-1.0,-1.2,-0.1,-0.6,-1.0,-0.9,-0.7,-0.6,-1.0,-1.2,-1.0 46 | 1920,0.2,-0.2,-0.4,0.1,-0.1,0.8,1.0,0.6,0.4,-0.4,-0.1,0.9 47 | 1921,1.1,0.6,0.9,-0.5,0.3,2.3,0.3,-0.7,0.4,0.9,0.8,0.7 48 | 1922,0.8,0.8,0.6,-0.4,-0.4,0.7,0.3,-0.1,0.4,0.6,0.8,1.1 49 | 1923,0.6,0.4,0.9,0.9,0.3,0.2,-1.1,-1.8,-1.5,-0.6,-1.3,0.1 50 | 1924,-0.5,0.1,0.3,-1.2,1.3,0.9,0.8,1.1,0.7,0.8,1.1,0.4 51 | 1925,0.6,1.2,1.5,1.4,0.0,-0.4,-1.3,-1.0,-0.7,-1.3,-1.0,-0.8 52 | 1926,-0.5,-1.3,-1.2,-0.5,-0.1,-0.6,-0.1,-0.7,0.1,0.4,0.0,0.5 53 | 1927,0.5,0.1,1.8,0.7,0.7,0.9,0.7,-0.5,-0.1,-0.4,-0.8,0.7 54 | 1928,-1.0,0.9,1.4,1.2,-0.1,-0.7,0.0,1.0,0.7,0.9,0.2,1.1 55 | 1929,1.6,1.6,0.5,0.5,-1.1,0.2,0.2,0.1,-0.1,0.8,1.0,0.5 56 | 1930,1.3,0.7,0.2,-0.2,0.3,-0.5,-0.4,-0.1,-0.7,0.4,0.1,-0.2 57 | 1931,0.7,-1.4,0.6,0.9,1.4,2.0,1.0,0.1,0.4,-1.3,-0.5,0.4 58 | 1932,0.2,-0.3,-0.2,-0.1,0.4,-0.4,-0.5,-0.7,-0.9,-0.4,-0.5,0.2 59 | 1933,-1.1,0.4,-0.1,0.4,0.7,-0.3,0.4,-0.0,0.1,0.4,0.6,0.7 60 | 1934,0.7,-0.0,0.1,0.7,-0.6,1.2,0.3,-2.2,-0.7,0.4,1.2,-0.3 61 | 1935,0.7,-0.4,1.2,0.4,-0.5,-0.1,0.0,0.2,0.6,0.7,0.3,-0.5 62 | 1936,-0.2,0.0,0.2,2.1,0.6,-0.1,0.5,-0.8,0.2,-0.0,-1.4,-0.0 63 | 1937,1.0,-0.5,0.7,0.3,0.1,0.4,-0.5,0.4,0.0,-0.2,-0.3,0.6 64 | 1938,0.8,0.3,-0.3,0.4,1.4,1.9,1.9,1.3,0.7,1.2,0.1,1.3 65 | 1939,1.7,0.7,1.2,1.0,0.0,-0.1,0.9,-0.0,-1.0,-1.4,-0.8,-0.9 66 | 1940,0.0,-0.4,-1.0,-0.7,-1.3,-1.9,-1.5,-1.8,-2.0,-1.8,-0.7,-3.0 67 | 1941,-0.9,-1.4,-1.0,-0.9,-0.5,-1.4,-2.1,-1.9,-0.9,-2.0,-1.0,-0.9 68 | 1942,-1.3,-0.3,-0.5,-0.4,0.6,0.9,-0.1,0.4,0.8,0.8,-0.5,1.3 69 | 1943,1.0,0.9,0.4,1.3,0.4,-0.7,0.3,0.8,0.5,0.9,0.3,-0.9 70 | 1944,-0.8,0.3,0.6,-0.4,0.0,-0.3,-0.9,0.4,0.2,-0.8,-0.7,0.3 71 | 1945,0.5,0.5,1.3,-0.5,0.1,0.9,0.4,1.2,0.8,0.2,-0.4,0.6 72 | 1946,-0.2,0.4,-0.1,-0.7,-1.0,-0.9,-1.0,-0.4,-1.6,-1.2,-0.2,-0.6 73 | 1947,-0.5,-0.4,1.2,-0.3,-1.2,0.4,1.0,0.8,1.1,-0.2,0.8,0.4 74 | 1948,-0.3,-0.3,-0.4,0.4,0.5,-0.4,0.1,-0.4,-0.8,0.6,0.4,-0.6 75 | 1949,-0.7,0.2,0.6,0.2,-0.5,-1.1,-0.1,-0.4,0.1,0.5,-0.7,0.7 76 | 1950,0.5,1.6,1.8,1.6,0.9,2.8,2.2,1.3,0.6,1.7,1.1,2.2 77 | 1951,1.7,0.8,-0.1,0.0,-0.5,0.6,-0.8,-0.0,-0.7,-0.8,-0.4,-0.4 78 | 1952,-0.9,-0.7,0.1,-0.7,0.7,0.8,0.4,-0.3,-0.4,0.2,-0.1,-1.4 79 | 1953,0.2,-0.6,-0.5,0.1,-3.0,-0.1,-0.1,-1.7,-1.3,-0.0,-0.3,-0.5 80 | 1954,0.6,-0.3,-0.0,0.7,0.6,-0.1,0.5,1.1,0.4,0.2,0.3,1.2 81 | 1955,-0.5,1.3,0.3,-0.1,1.4,1.7,2.0,1.5,1.3,1.5,1.4,0.8 82 | 1956,1.1,1.1,1.0,1.1,1.9,1.3,1.3,1.1,-0.0,1.8,0.1,0.9 83 | 1957,0.6,-0.2,-0.0,0.2,-1.1,-0.1,0.1,-0.9,-1.1,-0.1,-1.2,-0.4 84 | 1958,-1.6,-0.6,-0.1,0.2,-0.7,0.1,0.3,0.8,-0.4,-0.2,-0.5,-0.7 85 | 1959,-0.8,-1.3,0.9,0.4,0.4,-0.5,-0.5,-0.5,-0.0,0.4,1.0,0.7 86 | 1960,0.1,-0.2,0.6,0.8,0.6,-0.1,0.5,0.7,0.6,-0.1,0.6,0.6 87 | 1961,-0.2,0.5,-2.0,1.0,0.2,-0.2,0.3,0.1,0.0,-0.5,0.6,1.3 88 | 1962,1.7,0.5,-0.1,0.2,1.3,0.6,0.0,0.5,0.4,1.0,0.4,-0.0 89 | 1963,1.0,0.2,0.8,0.7,0.4,-0.9,-0.1,-0.2,-0.6,-1.3,-1.0,-1.3 90 | 1964,-0.4,-0.0,0.9,1.3,0.4,0.8,0.7,1.5,1.3,1.2,0.2,-0.4 91 | 1965,-0.4,0.1,0.3,-1.0,0.1,-1.2,-2.3,-1.1,-1.5,-1.1,-1.8,0.1 92 | 1966,-1.2,-0.4,-1.3,-0.5,-0.8,0.2,-0.1,0.4,-0.3,-0.2,-0.1,-0.5 93 | 1967,1.5,1.1,0.8,-0.1,-0.2,0.8,0.2,0.6,0.4,-0.0,-0.5,-0.6 94 | 1968,0.4,0.8,-0.2,-0.1,1.6,1.3,0.8,0.1,-0.3,-0.2,-0.4,0.1 95 | 1969,-1.3,-0.6,0.2,-0.7,-0.5,0.0,-0.7,-0.4,-1.1,-1.1,-0.1,0.3 96 | 1970,-1.0,-1.0,0.2,-0.3,0.3,1.1,-0.5,0.4,1.2,1.0,1.8,1.7 97 | 1971,0.3,1.4,1.9,2.1,1.0,0.4,0.2,1.5,1.5,1.7,0.6,0.1 98 | 1972,0.4,0.7,0.3,-0.4,-1.5,-1.1,-1.9,-0.8,-1.5,-1.1,-0.4,-1.3 99 | 1973,-0.3,-1.2,0.1,-0.1,0.4,1.3,0.7,1.3,1.3,0.9,3.0,1.6 100 | 1974,2.1,1.4,2.0,1.1,1.2,0.4,1.3,0.7,1.2,0.8,-0.2,-0.2 101 | 1975,-0.5,0.5,1.2,1.4,0.7,1.7,2.2,2.1,2.2,1.7,1.3,1.9 102 | 1976,1.2,1.1,1.3,0.2,0.3,0.1,-1.3,-1.2,-1.3,0.3,0.9,-0.4 103 | 1977,-0.4,0.7,-0.9,-0.7,-1.0,-1.7,-1.5,-1.2,-1.0,-1.3,-1.5,-1.2 104 | 1978,-0.3,-2.2,-0.5,-0.6,1.7,0.7,0.7,0.2,0.0,-0.6,-0.3,-0.2 105 | 1979,-0.4,0.6,-0.2,-0.4,0.5,0.7,-0.8,-0.5,0.1,-0.2,-0.5,-0.8 106 | 1980,0.3,0.1,-0.8,-1.0,-0.2,-0.4,-0.1,0.2,-0.6,-0.2,-0.4,-0.2 107 | 1981,0.3,-0.3,-1.6,-0.4,0.9,1.3,1.0,0.6,0.7,-0.5,0.2,0.4 108 | 1982,1.0,0.0,0.3,-0.2,-0.7,-1.9,-1.9,-2.3,-2.2,-2.0,-3.1,-2.2 109 | 1983,-3.0,-3.0,-2.7,-1.4,0.7,-0.2,-0.7,0.1,0.9,0.4,-0.1,-0.1 110 | 1984,0.2,0.5,-0.5,0.3,0.1,-0.8,0.3,0.3,0.1,-0.5,0.3,-0.2 111 | 1985,-0.3,0.6,-0.1,1.4,0.4,-0.9,-0.2,0.9,-0.0,-0.5,-0.2,0.1 112 | 1986,0.8,-1.0,0.1,0.2,-0.5,1.2,0.3,-0.7,-0.6,0.6,-1.4,-1.5 113 | 1987,-0.6,-1.1,-1.6,-2.1,-2.0,-1.9,-1.9,-1.4,-1.2,-0.5,-0.2,-0.5 114 | 1988,-0.1,-0.5,0.3,-0.0,1.1,-0.3,1.2,1.5,1.9,1.4,2.0,1.0 115 | 1989,1.3,0.8,0.7,2.0,1.6,0.8,1.0,-0.6,0.5,0.7,-0.3,-0.6 116 | 1990,-0.1,-1.6,-0.8,0.1,1.4,0.2,0.6,-0.5,-0.8,0.2,-0.6,-0.3 117 | 1991,0.5,0.0,-1.0,-1.0,-1.8,-0.5,-0.1,-0.7,-1.7,-1.3,-0.8,-1.8 118 | 1992,-2.5,-0.8,-2.3,-1.5,0.2,-1.2,-0.7,0.2,0.0,-1.7,-0.8,-0.6 119 | 1993,-0.8,-0.7,-0.8,-1.8,-0.7,-1.5,-1.1,-1.5,-0.8,-1.3,-0.0,0.1 120 | 1994,-0.1,0.0,-1.0,-1.9,-1.2,-1.0,-1.8,-1.7,-1.8,-1.4,-0.8,-1.3 121 | 1995,-0.4,-0.3,0.4,-1.3,-0.8,-0.1,0.5,0.1,0.3,-0.1,0.0,-0.6 122 | 1996,0.9,0.1,0.7,0.8,0.2,1.5,0.7,0.5,0.6,0.4,-0.1,0.6 123 | 1997,0.4,1.2,-0.8,-1.3,-2.1,-2.3,-0.9,-1.9,-1.5,-1.7,-1.6,-1.0 124 | 1998,-2.3,-1.7,-2.7,-2.1,0.2,1.1,1.5,1.0,1.0,1.1,1.1,1.3 125 | 1999,1.6,0.8,0.9,1.8,0.2,0.2,0.5,0.2,-0.1,0.9,1.2,1.2 126 | 2000,0.5,1.1,1.0,1.6,0.5,-0.5,-0.3,0.6,0.9,0.9,2.1,0.7 127 | 2001,0.9,1.0,0.7,0.1,-0.8,0.3,-0.3,-0.8,0.1,-0.2,0.6,-1.0 128 | 2002,0.3,0.7,-0.5,-0.2,-1.3,-0.5,-0.7,-1.4,-0.8,-0.7,-0.7,-1.2 129 | 2003,-0.2,-0.7,-0.6,-0.4,-0.6,-1.1,0.3,-0.1,-0.3,-0.2,-0.4,0.9 130 | 2004,-1.1,0.8,0.1,-1.2,1.4,-1.4,-0.7,-0.7,-0.3,-0.4,-1.0,-0.9 131 | 2005,0.2,-2.6,0.1,-0.9,-1.3,0.4,0.1,-0.7,0.3,1.1,-0.3,-0.0 132 | 2006,1.3,-0.0,1.4,1.5,-0.8,-0.5,-0.9,-1.5,-0.6,-1.5,-0.2,-0.4 133 | 2007,-0.7,-0.3,-0.1,-0.1,-0.1,0.6,-0.4,0.3,0.1,0.5,0.9,1.4 134 | 2008,1.4,1.9,1.2,0.5,-0.3,0.6,0.3,0.9,1.3,1.3,1.6,1.3 135 | 2009,1.0,1.3,0.1,0.9,-0.4,-0.1,0.2,-0.5,0.3,-1.4,-0.7,-0.8 136 | 2010,-1.0,-1.3,-1.0,1.5,1.1,0.3,2.1,1.9,2.4,1.8,1.5,2.6 137 | 2011,2.0,2.0,2.1,2.3,0.3,0.1,1.1,0.2,1.1,0.7,1.3,2.2 138 | 2012,1.0,0.2,0.3,-0.5,-0.1,-1.0,-0.1,-0.5,0.2,0.2,0.3,-0.7 139 | 2013,-0.1,-0.3,1.1,0.1,0.9,1.5,0.9,-0.0,0.3,-0.2,0.8,-0.0 140 | 2014,1.2,-0.1,-1.2,0.9,0.6,-0.1,-0.3,-1.1,-0.8,-0.8,-1.0,-0.6 141 | 2015,-0.7,0.0,-1.0,,,,,,,,, 142 | -------------------------------------------------------------------------------- /session_1/data/SOI_time_series.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/SOI_time_series.xlsx -------------------------------------------------------------------------------- /session_1/data/ascii_table.txt: -------------------------------------------------------------------------------- 1 | 0 1 2 3 4 5 6 7 8 9 2 | 10 11 12 13 14 15 16 17 18 19 3 | 20 21 22 23 24 25 26 27 28 29 4 | 30 31 32 33 34 35 36 37 38 39 5 | 40 41 42 43 44 45 46 47 48 49 6 | 50 51 52 53 54 55 56 57 58 59 7 | 60 61 62 63 64 65 66 67 68 69 8 | 70 71 72 73 74 75 76 77 78 79 9 | 80 81 82 83 84 85 86 87 88 89 10 | 90 91 92 93 94 95 96 97 98 99 11 | -------------------------------------------------------------------------------- /session_1/data/clusters_monthly.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/clusters_monthly.mat -------------------------------------------------------------------------------- /session_1/data/rainfall_calibration.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/rainfall_calibration.xlsx -------------------------------------------------------------------------------- /session_1/data/random_2darray.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/random_2darray.mat -------------------------------------------------------------------------------- /session_1/data/soi_nino.csv: -------------------------------------------------------------------------------- 1 | ,SOI,nino 2 | 1950-01-01,0.570492283299991,-1.84 3 | 1950-02-01,1.642897110809131,-1.63 4 | 1950-03-01,1.7458488331453068,-1.3 5 | 1950-04-01,1.8953161777858345,-1.43 6 | 1950-05-01,0.7486727622207335,-1.72 7 | 1950-06-01,2.7129664106468625,-1.17 8 | 1950-07-01,2.083015745855139,-0.86 9 | 1950-08-01,1.147656550641308,-0.95 10 | 1950-09-01,0.6255069712545226,-1.04 11 | 1950-10-01,1.6266674525145242,-0.83 12 | 1950-11-01,1.1832335684352566,-1.1 13 | 1950-12-01,2.292121930820888,-1.19 14 | 1951-01-01,1.8208863288896782,-1.21 15 | 1951-02-01,0.8171986367883551,-1.05 16 | 1951-03-01,-0.3925486877673553,-0.61 17 | 1951-04-01,-0.16633800818652664,-0.56 18 | 1951-05-01,-0.6887789412432129,-0.23 19 | 1951-06-01,0.4058768645852223,-0.23 20 | 1951-07-01,-0.8100616789440915,0.44 21 | 1951-08-01,-0.22267962922930465,0.43 22 | 1951-09-01,-0.7918979882384753,0.69 23 | 1951-10-01,-0.9335385188134022,0.64 24 | 1951-11-01,-0.4823151495967204,0.74 25 | 1951-12-01,-0.5011449382800363,0.43 26 | 1952-01-01,-0.9925002736870584,0.18 27 | 1952-02-01,-0.9799098066686018,-0.04 28 | 1952-03-01,-0.20925747168915493,-0.01 29 | 1952-04-01,-1.0097419933570428,0.08 30 | 1952-05-01,0.5889559062803456,0.07 31 | 1952-06-01,0.6622201474810141,-0.36 32 | 1952-07-01,0.3471692909756007,-0.56 33 | 1952-08-01,-0.5652636741968604,-0.47 34 | 1952-09-01,-0.4221401727186116,-0.29 35 | 1952-10-01,0.06556625048530937,-0.25 36 | 1952-11-01,-0.20472369659140405,-0.37 37 | 1952-12-01,-1.541773771866638,-0.15 38 | 1953-01-01,0.2578937719025693,0.18 39 | 1953-02-01,-0.7856278127813214,0.36 40 | 1953-03-01,-0.8813252639760055,0.35 41 | 1953-04-01,-0.07262645427866783,0.4 42 | 1953-05-01,-3.2442486362903296,0.38 43 | 1953-06-01,-0.3631529841019589,0.33 44 | 1953-07-01,-0.10286497510424714,0.24 45 | 1953-08-01,-2.004116663060937,0.23 46 | 1953-09-01,-1.4081610141050784,0.31 47 | 1953-10-01,-0.12176589375816801,0.23 48 | 1953-11-01,-0.3435194230940228,0.23 49 | 1953-12-01,-0.6106848154997048,0.3 50 | 1954-01-01,0.6746917870991513,0.36 51 | 1954-02-01,-0.5427753204222762,0.39 52 | 1954-03-01,-0.3314516157413348,-0.12 53 | 1954-04-01,0.7707775308917418,-0.93 54 | 1954-05-01,0.4292390503397759,-0.85 55 | 1954-06-01,-0.2777052231366949,-0.76 56 | 1954-07-01,0.4114599004155998,-0.91 57 | 1954-08-01,0.9421061236607589,-1.2 58 | 1954-09-01,0.3790017609079234,-1.45 59 | 1954-10-01,0.06556625048523837,-1.28 60 | 1954-11-01,0.28106134616795875,-1.07 61 | 1954-12-01,1.1967231586244518,-1.32 62 | 1955-01-01,-0.5757022584904762,-1.13 63 | 1955-02-01,1.4000446184500306,-0.93 64 | 1955-03-01,0.0962278884412254,-0.95 65 | 1955-04-01,-0.35376111600224425,-1.14 66 | 1955-05-01,1.3076817580123188,-1.25 67 | 1955-06-01,1.6021455180987232,-1.13 68 | 1955-07-01,1.8901439175352144,-0.98 69 | 1955-08-01,1.421723786615399,-1.33 70 | 1955-09-01,1.3650226022943903,-1.19 71 | 1955-10-01,1.4393353082710467,-2.13 72 | 1955-11-01,1.4608250214405731,-2.45 73 | 1955-12-01,0.8133335883556744,-2.1 74 | 1956-01-01,1.2477890579944146,-1.39 75 | 1956-02-01,1.1086216276191652,-0.83 76 | 1956-03-01,0.8293927527541659,-0.71 77 | 1956-04-01,1.239335300431036,-0.83 78 | 1956-05-01,1.7868323258335737,-0.68 79 | 1956-06-01,1.1749067132725004,-0.7 80 | 1956-07-01,1.2472378231353694,-0.91 81 | 1956-08-01,1.010622932654223,-1.02 82 | 1956-09-01,-0.05238235719867773,-0.92 83 | 1956-10-01,1.7515555486768186,-0.95 84 | 1956-11-01,0.07286775641391231,-1.1 85 | 1956-12-01,0.9228734655753428,-0.98 86 | 1957-01-01,0.6225920351995712,-0.58 87 | 1957-02-01,-0.3970638250068711,-0.25 88 | 1957-03-01,-0.33145161574126536,0.32 89 | 1957-04-01,0.11479665353694328,0.55 90 | 1957-05-01,-1.2477879370347977,0.68 91 | 1957-06-01,-0.3631529841019589,0.59 92 | 1957-07-01,0.09000685321567721,0.83 93 | 1957-08-01,-1.1819149551385857,0.82 94 | 1957-09-01,-1.1616558037584093,0.71 95 | 1957-10-01,-0.2466539899205336,0.7 96 | 1957-11-01,-1.3844873718640185,1.06 97 | 1957-12-01,-0.5559148768898705,1.4 98 | 1958-01-01,-1.826096304080163,1.7 99 | 1958-02-01,-0.8827688097249616,1.59 100 | 1958-03-01,-0.3925486877674248,1.01 101 | 1958-04-01,0.11479665353694328,0.55 102 | 1958-05-01,-0.8484957971836917,0.44 103 | 1958-06-01,-0.1068097012062642,0.36 104 | 1958-07-01,0.2185880720956755,0.2 105 | 1958-08-01,0.6680388876865897,-0.03 106 | 1958-09-01,-0.4221401727186116,-0.32 107 | 1958-10-01,-0.3090980380017164,-0.18 108 | 1958-11-01,-0.6211108760994181,0.16 109 | 1958-12-01,-0.8845345085487515,0.09 110 | 1959-01-01,-0.940400521787478,0.47 111 | 1959-02-01,-1.6113262868021525,0.42 112 | 1959-03-01,0.7071986087019859,0.22 113 | 1959-04-01,0.3959313152605197,0.14 114 | 1959-05-01,0.26952219439938785,-0.11 115 | 1959-06-01,-0.7903917889280843,-0.35 116 | 1959-07-01,-0.4886086317441689,-0.72 117 | 1959-08-01,-0.7022972921839449,-0.61 118 | 1959-09-01,-0.05238235719867773,-0.65 119 | 1959-10-01,0.31534244280996954,-0.36 120 | 1959-11-01,1.044437841932559,-0.41 121 | 1959-12-01,0.7037937111360681,-0.25 122 | 1960-01-01,0.049494764304307964,-0.42 123 | 1960-02-01,-0.3970638250068711,-0.56 124 | 1960-03-01,0.4017132485716057,-0.36 125 | 1960-04-01,0.8644890847996006,-0.29 126 | 1960-05-01,0.5090974783100153,-0.29 127 | 1960-06-01,-0.3631529841019589,-0.45 128 | 1960-07-01,0.475750509855599,-0.21 129 | 1960-08-01,0.531005269699583,-0.08 130 | 1960-09-01,0.6255069712545226,-0.23 131 | 1960-10-01,-0.18420994183942174,-0.43 132 | 1960-11-01,0.6280506624246239,-0.39 133 | 1960-12-01,0.5394838953065654,-0.28 134 | 1961-01-01,-0.2631037470931137,-0.29 135 | 1961-02-01,0.4772051474856697,-0.22 136 | 1961-03-01,-2.592043280706108,-0.33 137 | 1961-04-01,1.0519121926153183,-0.29 138 | 1961-05-01,0.10980533845890897,-0.06 139 | 1961-06-01,-0.4486007450671257,0.17 140 | 1961-07-01,0.2185880720956024,-0.12 141 | 1961-08-01,-0.1541628202357623,-0.5 142 | 1961-09-01,0.009243945387989588,-0.85 143 | 1961-10-01,-0.6213182784075593,-0.85 144 | 1961-11-01,0.6280506624246239,-0.53 145 | 1961-12-01,1.306263035844058,-0.49 146 | 1962-01-01,1.872986080789199,-0.55 147 | 1962-02-01,0.3800641505420295,-0.55 148 | 1962-03-01,-0.3925486877673553,-0.53 149 | 1962-04-01,0.11479665353694328,-0.57 150 | 1962-05-01,1.2278233300419883,-0.68 151 | 1962-06-01,0.4058768645852223,-0.51 152 | 1962-07-01,-0.03857436566424799,-0.35 153 | 1962-08-01,0.325454842718956,-0.41 154 | 1962-09-01,0.4406280634945207,-0.75 155 | 1962-10-01,0.9397829236216554,-0.73 156 | 1962-11-01,0.4198570726705775,-0.85 157 | 1962-12-01,-0.11775536801125885,-0.94 158 | 1963-01-01,1.0393900503961533,-0.81 159 | 1963-02-01,0.13721165818303946,-0.44 160 | 1963-03-01,0.585004464649806,-0.01 161 | 1963-04-01,0.6770659769839896,0.07 162 | 1963-05-01,0.26952219439938785,-0.2 163 | 1963-06-01,-1.13218283278914,0.0 164 | 1963-07-01,-0.10286497510432024,0.62 165 | 1963-08-01,-0.4282300562097758,0.7 166 | 1963-09-01,-0.6070190804785435,0.64 167 | 1963-10-01,-1.4330909034627934,0.77 168 | 1963-11-01,-1.1068959188587022,0.87 169 | 1963-12-01,-1.4322338946470317,1.02 170 | 1964-01-01,-0.419403002791795,0.76 171 | 1964-02-01,-0.202781831119646,0.44 172 | 1964-03-01,0.7071986087019859,-0.22 173 | 1964-04-01,1.520469962154506,-0.73 174 | 1964-05-01,0.26952219439938785,-0.97 175 | 1964-06-01,0.6622201474810141,-1.03 176 | 1964-07-01,0.6686223381755233,-0.86 177 | 1964-08-01,1.353206977621857,-1.19 178 | 1964-09-01,1.3650226022943903,-1.34 179 | 1964-10-01,1.1895591159463157,-1.25 180 | 1964-11-01,0.14226561966534002,-1.34 181 | 1964-12-01,-0.5011449382800363,-1.27 182 | 1965-01-01,-0.419403002791795,-0.89 183 | 1965-02-01,-0.008499837232420808,-0.46 184 | 1965-03-01,0.0962278884412254,-0.21 185 | 1965-04-01,-1.4782997628962304,-0.18 186 | 1965-05-01,-0.04991151748147908,0.15 187 | 1965-06-01,-1.4739738766500985,0.5 188 | 1965-07-01,-2.2244550866237804,0.74 189 | 1965-08-01,-1.3874653821192124,1.14 190 | 1965-09-01,-1.531413619278343,1.2 191 | 1965-10-01,-1.2457587592192452,1.47 192 | 1965-11-01,-2.009068141126,1.56 193 | 1965-12-01,-0.008215490791652597,1.38 194 | 1966-01-01,-1.30509878508448,1.08 195 | 1966-02-01,-0.5913458188941515,0.83 196 | 1966-03-01,-1.7977813443671464,0.89 197 | 1966-04-01,-0.8223188855414317,0.45 198 | 1966-05-01,-0.9283542251539312,-0.24 199 | 1966-06-01,-0.021361940240903127,0.08 200 | 1966-07-01,-0.10286497510424714,0.14 201 | 1966-08-01,0.2569380337254137,-0.28 202 | 1966-09-01,-0.29888756754527696,-0.34 203 | 1966-10-01,-0.3715420860828991,-0.39 204 | 1966-11-01,-0.13532583334005524,-0.32 205 | 1966-12-01,-0.6106848154997048,-0.48 206 | 1967-01-01,1.6124873212913573,-0.49 207 | 1967-02-01,1.1571921260909854,-0.48 208 | 1967-03-01,0.646101536675896,-0.66 209 | 1967-04-01,-0.35376111600224425,-0.89 210 | 1967-05-01,-0.3693452293624368,-0.42 211 | 1967-06-01,0.5767723865157501,-0.07 212 | 1967-07-01,0.15429746265567634,-0.11 213 | 1967-08-01,0.4624884607060406,-0.31 214 | 1967-09-01,0.4406280634945908,-0.72 215 | 1967-10-01,-0.12176589375816801,-0.6 216 | 1967-11-01,-0.5517130128479903,-0.47 217 | 1967-12-01,-0.774994631329083,-0.59 218 | 1968-01-01,0.4662927795008308,-0.89 219 | 1968-02-01,0.8171986367883551,-1.04 220 | 1968-03-01,-0.5758399038456252,-0.95 221 | 1968-04-01,-0.35376111600224425,-0.71 222 | 1968-05-01,1.4673986139527069,-0.76 223 | 1968-06-01,1.1749067132724034,0.0 224 | 1968-07-01,0.7329129476154493,0.33 225 | 1968-08-01,-0.1541628202357623,0.19 226 | 1968-09-01,-0.36051387013194425,0.07 227 | 1968-10-01,-0.3090980380017164,0.2 228 | 1968-11-01,-0.4823151495966415,0.65 229 | 1968-12-01,0.04655444781818165,0.73 230 | 1969-01-01,-1.4613980407831613,0.88 231 | 1969-02-01,-0.8827688097249616,1.12 232 | 1969-03-01,-0.025966255610885062,0.69 233 | 1969-04-01,-1.0097419933570428,0.6 234 | 1969-05-01,-0.6887789412433036,0.59 235 | 1969-06-01,-0.19225746217152812,0.3 236 | 1969-07-01,-0.6814804600641663,0.02 237 | 1969-08-01,-0.6337804831904027,0.23 238 | 1969-09-01,-1.1616558037584093,0.51 239 | 1969-10-01,-1.308202807300428,0.72 240 | 1969-11-01,-0.13532583334005524,0.61 241 | 1969-12-01,0.2108642636476844,0.61 242 | 1970-01-01,-1.0966997774862186,0.38 243 | 1970-02-01,-1.271332797499467,0.2 244 | 1970-03-01,-0.02596625561095452,0.02 245 | 1970-04-01,-0.5411842238179618,0.16 246 | 1970-05-01,0.1896637664291484,-0.14 247 | 1970-06-01,0.9185634303767088,-0.55 248 | 1970-07-01,-0.5528992411841679,-0.8 249 | 1970-08-01,0.2569380337254137,-1.07 250 | 1970-09-01,1.2417699971210556,-0.96 251 | 1970-10-01,0.9397829236215844,-1.0 252 | 1970-11-01,1.946610064199936,-0.9 253 | 1970-12-01,1.6896526061128978,-1.22 254 | 1971-01-01,0.3099935238021495,-1.58 255 | 1971-02-01,1.4486151169218506,-1.43 256 | 1971-03-01,1.9291400492235071,-1.23 257 | 1971-04-01,2.5512970551407395,-1.02 258 | 1971-05-01,0.9083896181612124,-0.85 259 | 1971-06-01,0.1495335816895276,-0.99 260 | 1971-07-01,0.15429746265567634,-0.99 261 | 1971-08-01,1.421723786615399,-0.94 262 | 1971-09-01,1.5499015100543223,-0.88 263 | 1971-10-01,1.689111500595636,-1.13 264 | 1971-11-01,0.6280506624246239,-1.04 265 | 1971-12-01,0.046554447818243916,-1.14 266 | 1972-01-01,0.4141930276013098,-0.83 267 | 1972-02-01,0.6714871413728949,-0.41 268 | 1972-03-01,0.035130816415135435,-0.12 269 | 1972-04-01,-0.6348957777258207,0.13 270 | 1972-05-01,-1.6470800768859044,0.35 271 | 1972-06-01,-1.3885261156848343,0.54 272 | 1972-07-01,-1.8387114299838585,0.9 273 | 1972-08-01,-1.1133981461450428,1.17 274 | 1972-09-01,-1.5930399218650104,1.27 275 | 1972-10-01,-1.2457587592192452,1.68 276 | 1972-11-01,-0.4823151495966415,1.96 277 | 1972-12-01,-1.4870038332568036,2.12 278 | 1973-01-01,-0.3152034989926347,1.77 279 | 1973-02-01,-1.5627557883302772,1.17 280 | 1973-03-01,-0.14816039966306496,0.39 281 | 1973-04-01,-0.26004956209438546,-0.41 282 | 1973-05-01,0.26952219439938785,-0.79 283 | 1973-06-01,1.1749067132725004,-0.95 284 | 1973-07-01,0.6043317287355242,-1.33 285 | 1973-08-01,1.147656550641308,-1.47 286 | 1973-09-01,1.3033962997077229,-1.51 287 | 1973-10-01,0.8773388755404726,-1.76 288 | 1973-11-01,3.195771602723978,-2.24 289 | 1973-12-01,1.6348826675030013,-2.25 290 | 1974-01-01,2.2897840959857216,-2.15 291 | 1974-02-01,1.497185615393671,-1.71 292 | 1974-03-01,2.0513341932756872,-1.42 293 | 1974-04-01,1.239335300431036,-1.17 294 | 1974-05-01,1.0681064741016004,-1.07 295 | 1974-06-01,0.1495335816895276,-0.98 296 | 1974-07-01,1.1829472136953705,-0.8 297 | 1974-08-01,0.531005269699505,-0.56 298 | 1974-09-01,1.1801436945343884,-0.59 299 | 1974-10-01,0.752450779378107,-0.87 300 | 1974-11-01,-0.27412155984267395,-1.11 301 | 1974-12-01,-0.2820651838407616,-0.99 302 | 1975-01-01,-0.5236025065909553,-0.49 303 | 1975-02-01,0.3800641505420847,-0.61 304 | 1975-03-01,1.0737810408584565,-0.89 305 | 1975-04-01,1.6141815160623647,-0.81 306 | 1975-05-01,0.5889559062802547,-0.94 307 | 1975-06-01,1.5166977571334588,-1.4 308 | 1975-07-01,2.0830157458552123,-1.3 309 | 1975-08-01,2.0383750675572023,-1.46 310 | 1975-09-01,2.2277908385075227,-1.62 311 | 1975-10-01,1.6891115005957067,-1.92 312 | 1975-11-01,1.3220292949379544,-1.64 313 | 1975-12-01,1.9087323605521105,-1.93 314 | 1976-01-01,1.2998888098939356,-1.95 315 | 1976-02-01,1.1571921260910405,-1.25 316 | 1976-03-01,1.2570722569366566,-0.72 317 | 1976-04-01,0.11479665353694328,-0.8 318 | 1976-05-01,0.1896637664291484,-0.64 319 | 1976-06-01,-0.10680970120616706,-0.35 320 | 1976-07-01,-1.2600959450240126,-0.02 321 | 1976-08-01,-1.4559821911127546,0.16 322 | 1976-09-01,-1.4081610141050085,0.3 323 | 1976-10-01,0.19045434664760394,0.7 324 | 1976-11-01,0.9056421154299404,0.68 325 | 1976-12-01,-0.5011449382800363,0.48 326 | 1977-01-01,-0.419403002791795,0.64 327 | 1977-02-01,0.6229166429010747,0.32 328 | 1977-03-01,-1.3090047681585657,0.24 329 | 1977-04-01,-1.1034535472649016,-0.18 330 | 1977-05-01,-1.1679295090645587,0.06 331 | 1977-06-01,-1.986660442441585,0.27 332 | 1977-07-01,-1.4529677733440098,0.28 333 | 1977-08-01,-1.4559821911127546,0.0 334 | 1977-09-01,-1.0384031985851447,0.38 335 | 1977-10-01,-1.4330909034627226,0.59 336 | 1977-11-01,-1.6620788248693348,0.54 337 | 1977-12-01,-1.3226940174273631,0.68 338 | 1978-01-01,-0.3152034989926939,0.65 339 | 1978-02-01,-2.6798772531818638,0.34 340 | 1978-03-01,-0.8813252639760055,-0.09 341 | 1978-04-01,-0.9160304394492906,-0.54 342 | 1978-05-01,1.6271154698931856,-0.55 343 | 1978-06-01,0.4913246255504862,-0.53 344 | 1978-07-01,0.6043317287355242,-0.47 345 | 1978-08-01,-0.017129202248677682,-0.7 346 | 1978-09-01,0.009243945387989588,-0.62 347 | 1978-10-01,-0.7462063745698538,-0.38 348 | 1978-11-01,-0.3435194230940228,-0.3 349 | 1978-12-01,-0.2820651838406993,-0.2 350 | 1979-01-01,-0.419403002791795,-0.22 351 | 1979-02-01,0.5257756459574898,-0.22 352 | 1979-03-01,-0.5758399038456252,0.15 353 | 1979-04-01,-0.6348957777258207,0.07 354 | 1979-05-01,0.34938062236953643,-0.17 355 | 1979-06-01,0.4913246255504862,-0.14 356 | 1979-07-01,-0.8100616789441646,-0.33 357 | 1979-08-01,-0.7022972921838672,-0.16 358 | 1979-09-01,0.07087024797458684,0.33 359 | 1979-10-01,-0.3715420860828991,0.22 360 | 1979-11-01,-0.6211108760994181,0.35 361 | 1979-12-01,-0.9940743857684201,0.4 362 | 1980-01-01,0.3620932757017297,0.43 363 | 1980-02-01,-0.057070335704185675,0.2 364 | 1980-03-01,-1.1868106241063858,0.05 365 | 1980-04-01,-1.4782997628963368,-0.01 366 | 1980-05-01,-0.369345229362346,0.07 367 | 1980-06-01,-0.6194962669976535,0.3 368 | 1980-07-01,-0.16715558454431936,0.07 369 | 1980-08-01,-0.017129202248677682,-0.31 370 | 1980-09-01,-0.6070190804785435,-0.26 371 | 1980-10-01,-0.30909803800178737,-0.27 372 | 1980-11-01,-0.4823151495967204,-0.11 373 | 1980-12-01,-0.2820651838406993,-0.1 374 | 1981-01-01,0.3099935238021495,-0.5 375 | 1981-02-01,-0.4942048219505113,-0.7 376 | 1981-03-01,-2.1032667044975266,-0.58 377 | 1981-04-01,-0.6348957777257141,-0.48 378 | 1981-05-01,0.7486727622207335,-0.5 379 | 1981-06-01,1.0894589523072369,-0.34 380 | 1981-07-01,0.9257847759354468,-0.62 381 | 1981-08-01,0.4624884607060406,-0.68 382 | 1981-09-01,0.6871332738411199,-0.29 383 | 1981-10-01,-0.6213182784075593,-0.26 384 | 1981-11-01,0.14226561966534002,-0.33 385 | 1981-12-01,0.32040414086729063,-0.22 386 | 1982-01-01,1.0393900503960942,0.02 387 | 1982-02-01,-0.10564083417600577,-0.12 388 | 1982-03-01,0.035130816415135435,0.03 389 | 1982-04-01,-0.44747266991010304,0.11 390 | 1982-05-01,-0.8484957971836917,0.51 391 | 1982-06-01,-2.2430037253372794,0.63 392 | 1982-07-01,-1.9030020394238576,0.37 393 | 1982-08-01,-2.6892847529962047,0.71 394 | 1982-09-01,-2.2709292503181406,1.38 395 | 1982-10-01,-2.1824194804367743,1.89 396 | 1982-11-01,-3.39702540615274,1.98 397 | 1982-12-01,-2.472862728233571,2.25 398 | 1983-01-01,-3.3369891091676323,2.26 399 | 1983-02-01,-3.6027167241462794,1.92 400 | 1983-03-01,-3.3863052170451384,1.43 401 | 1983-04-01,-1.9468575324355244,0.94 402 | 1983-05-01,0.5889559062802547,0.91 403 | 1983-06-01,-0.4486007450671257,0.58 404 | 1983-07-01,-0.7457710695040924,-0.1 405 | 1983-08-01,-0.1541628202357623,-0.42 406 | 1983-09-01,0.9336384841877892,-0.64 407 | 1983-10-01,0.31534244280996954,-1.03 408 | 1983-11-01,-0.20472369659140405,-1.19 409 | 1983-12-01,-0.17252530662109308,-0.87 410 | 1984-01-01,0.15369426810340905,-0.53 411 | 1984-02-01,0.4286346490139048,-0.11 412 | 1984-03-01,-0.8813252639760055,-0.28 413 | 1984-04-01,0.20850820744480209,-0.5 414 | 1984-05-01,-0.04991151748147908,-0.6 415 | 1984-06-01,-1.0467350718239732,-0.74 416 | 1984-07-01,0.2185880720956024,-0.44 417 | 1984-08-01,0.11990441573840695,-0.26 418 | 1984-09-01,0.13249655056125414,-0.39 419 | 1984-10-01,-0.6213182784075593,-0.64 420 | 1984-11-01,0.28106134616795875,-1.13 421 | 1984-12-01,-0.3368351224505958,-1.31 422 | 1985-01-01,-0.3673032508922741,-1.1 423 | 1985-02-01,0.5257756459574898,-0.72 424 | 1985-03-01,-0.4536457597934453,-0.8 425 | 1985-04-01,1.6141815160623647,-0.87 426 | 1985-05-01,0.26952219439938785,-0.82 427 | 1985-06-01,-1.13218283278914,-0.77 428 | 1985-07-01,-0.23144619398424546,-0.64 429 | 1985-08-01,0.7365556966801321,-0.65 430 | 1985-09-01,-0.05238235719867773,-0.74 431 | 1985-10-01,-0.6837623264887421,-0.56 432 | 1985-11-01,-0.27412155984267395,-0.35 433 | 1985-12-01,0.04655444781818165,-0.5 434 | 1986-01-01,0.8830907946973535,-0.63 435 | 1986-02-01,-1.2713327974994122,-0.57 436 | 1986-03-01,-0.14816039966306496,-0.34 437 | 1986-04-01,0.11479665353694328,-0.15 438 | 1986-05-01,-0.6887789412433036,-0.34 439 | 1986-06-01,1.0040111913419727,-0.13 440 | 1986-07-01,0.2185880720956755,0.1 441 | 1986-08-01,-0.9763645281580364,0.28 442 | 1986-09-01,-0.6070190804785435,0.53 443 | 1986-10-01,0.5026745870534469,0.81 444 | 1986-11-01,-1.5926809616179858,0.94 445 | 1986-12-01,-1.6513136490863063,1.06 446 | 1987-01-01,-0.6799017622896365,1.15 447 | 1987-02-01,-1.4656147913866924,1.19 448 | 1987-03-01,-2.1032667044975266,1.16 449 | 1987-04-01,-2.790261517606041,0.89 450 | 1987-05-01,-2.2060890726774898,0.79 451 | 1987-06-01,-2.2430037253373767,0.98 452 | 1987-07-01,-1.8387114299838585,1.24 453 | 1987-08-01,-1.6615326180933037,1.55 454 | 1987-09-01,-1.2232821063450765,1.59 455 | 1987-10-01,-0.6837623264887421,1.29 456 | 1987-11-01,-0.27412155984267395,1.2 457 | 1987-12-01,-0.6654547541094767,0.91 458 | 1988-01-01,-0.10680449139437327,0.82 459 | 1988-02-01,-0.6884868158377364,0.19 460 | 1988-03-01,0.035130816415204894,0.12 461 | 1988-04-01,-0.16633800818663316,-0.23 462 | 1988-05-01,0.9882480461314518,-0.94 463 | 1988-06-01,-0.5340485060323896,-1.48 464 | 1988-07-01,1.1186566042554442,-1.62 465 | 1988-08-01,1.421723786615399,-1.22 466 | 1988-09-01,1.9812856281608533,-1.11 467 | 1988-10-01,1.3768912601897931,-1.97 468 | 1988-11-01,2.0854057907026333,-2.01 469 | 1988-12-01,0.9776434041851769,-2.01 470 | 1989-01-01,1.456188065592676,-1.98 471 | 1989-02-01,0.7686281383165351,-1.49 472 | 1989-03-01,0.5239073926237856,-1.21 473 | 1989-04-01,2.363873947325022,-0.96 474 | 1989-05-01,1.4673986139527069,-0.74 475 | 1989-06-01,0.6622201474810141,-0.46 476 | 1989-07-01,0.9257847759354468,-0.48 477 | 1989-08-01,-0.8393309101709517,-0.5 478 | 1989-09-01,0.502254366081188,-0.36 479 | 1989-10-01,0.6275626832158124,-0.48 480 | 1989-11-01,-0.3435194230941017,-0.44 481 | 1989-12-01,-0.720224692719311,-0.11 482 | 1990-01-01,-0.10680449139437327,-0.05 483 | 1990-02-01,-1.951319776104783,0.2 484 | 1990-03-01,-1.1868106241063163,0.06 485 | 1990-04-01,-0.07262645427877436,0.14 486 | 1990-05-01,1.3076817580123188,0.17 487 | 1990-06-01,-0.021361940240903127,-0.06 488 | 1990-07-01,0.5400411192955982,0.14 489 | 1990-08-01,-0.7022972921839449,0.23 490 | 1990-09-01,-0.8535242908251427,0.13 491 | 1990-10-01,0.06556625048530937,0.22 492 | 1990-11-01,-0.690508739350688,0.15 493 | 1990-12-01,-0.4463749996702021,0.28 494 | 1991-01-01,0.570492283299991,0.39 495 | 1991-02-01,-0.10564083417600577,0.29 496 | 1991-03-01,-1.431198912210676,0.06 497 | 1991-04-01,-1.4782997628962304,0.31 498 | 1991-05-01,-1.9665137887668618,0.45 499 | 1991-06-01,-0.7049440279629174,0.72 500 | 1991-07-01,-0.16715558454424628,0.89 501 | 1991-08-01,-0.9763645281580364,0.8 502 | 1991-09-01,-1.7779188296249422,0.59 503 | 1991-10-01,-1.4330909034627934,0.83 504 | 1991-11-01,-0.8987023291047344,1.2 505 | 1991-12-01,-1.9799332807451877,1.63 506 | 1992-01-01,-2.763891838272369,1.69 507 | 1992-02-01,-1.125621302084062,1.6 508 | 1992-03-01,-2.958625712862577,1.33 509 | 1992-04-01,-2.134280640251136,1.26 510 | 1992-05-01,0.029946910488669563,1.1 511 | 1992-06-01,-1.4739738766500985,0.66 512 | 1992-07-01,-0.6814804600641663,0.28 513 | 1992-08-01,-0.01712920224859979,0.03 514 | 1992-09-01,0.009243945387989588,-0.16 515 | 1992-10-01,-1.870199240031002,-0.39 516 | 1992-11-01,-0.8987023291047344,-0.13 517 | 1992-12-01,-0.774994631329083,0.03 518 | 1993-01-01,-0.888300769887898,0.2 519 | 1993-02-01,-0.9799098066686018,0.37 520 | 1993-03-01,-1.1868106241063858,0.38 521 | 1993-04-01,-2.415415301974605,0.7 522 | 1993-05-01,-0.8484957971836917,0.79 523 | 1993-06-01,-1.8157649205110569,0.39 524 | 1993-07-01,-1.067224116704088,0.31 525 | 1993-08-01,-1.7985662360803103,0.08 526 | 1993-09-01,-0.8535242908251427,0.31 527 | 1993-10-01,-1.4955349515439054,0.22 528 | 1993-11-01,-0.06592797008870642,0.19 529 | 1993-12-01,-0.008215490791652597,0.1 530 | 1994-01-01,-0.15890424329401268,0.07 531 | 1994-02-01,-0.10564083417600577,0.14 532 | 1994-03-01,-1.431198912210676,0.2 533 | 1994-04-01,-2.602838409790323,0.33 534 | 1994-05-01,-1.3276463650050374,0.36 535 | 1994-06-01,-1.217630593754404,0.4 536 | 1994-07-01,-1.7744208205439325,0.39 537 | 1994-08-01,-2.004116663060937,0.47 538 | 1994-09-01,-1.8395451322116096,0.42 539 | 1994-10-01,-1.557978999625088,0.75 540 | 1994-11-01,-0.8987023291046555,1.22 541 | 1994-12-01,-1.4322338946469697,1.33 542 | 1995-01-01,-0.419403002791795,1.07 543 | 1995-02-01,-0.4456343234786912,0.84 544 | 1995-03-01,0.15732496046724587,0.58 545 | 1995-04-01,-1.8531459785276656,0.37 546 | 1995-05-01,-0.9283542251539312,-0.04 547 | 1995-06-01,-0.2777052231366949,0.11 548 | 1995-07-01,0.4114599004155268,-0.04 549 | 1995-08-01,-0.08564601124221999,-0.51 550 | 1995-09-01,0.2557491557345888,-0.62 551 | 1995-10-01,-0.2466539899206045,-0.71 552 | 1995-11-01,0.003469893162642392,-0.91 553 | 1995-12-01,-0.7749946313291453,-0.82 554 | 1996-01-01,0.9351905465969336,-0.84 555 | 1996-02-01,-0.0570703357042409,-0.85 556 | 1996-03-01,0.4628103205976956,-0.6 557 | 1996-04-01,0.8644890847996006,-0.38 558 | 1996-05-01,0.10980533845890897,-0.26 559 | 1996-06-01,1.3458022352029313,-0.2 560 | 1996-07-01,0.6686223381755233,-0.26 561 | 1996-08-01,0.325454842718956,-0.18 562 | 1996-09-01,0.6255069712545226,-0.39 563 | 1996-10-01,0.31534244280996954,-0.3 564 | 1996-11-01,-0.13532583334005524,-0.3 565 | 1996-12-01,0.5942538339163996,-0.56 566 | 1997-01-01,0.46629277950089,-0.58 567 | 1997-02-01,1.2057626245628603,-0.37 568 | 1997-03-01,-1.1868106241063858,-0.2 569 | 1997-04-01,-1.8531459785276656,0.18 570 | 1997-05-01,-2.285947500647638,0.72 571 | 1997-06-01,-2.6702425301635992,1.17 572 | 1997-07-01,-0.9386428978240898,1.58 573 | 1997-08-01,-2.2781838990350294,1.76 574 | 1997-09-01,-1.5930399218650104,2.06 575 | 1997-10-01,-1.9326432881121136,2.35 576 | 1997-11-01,-1.7314766881206836,2.39 577 | 1997-12-01,-1.1583842015979229,2.36 578 | 1998-01-01,-2.555492830674108,2.31 579 | 1998-02-01,-2.145601769992008,1.88 580 | 1998-03-01,-3.4474022890711584,1.33 581 | 1998-04-01,-2.790261517606041,0.85 582 | 1998-05-01,0.029946910488760345,0.56 583 | 1998-06-01,0.9185634303767088,-0.28 584 | 1998-07-01,1.4401096514553668,-0.76 585 | 1998-08-01,0.8735893146672166,-1.01 586 | 1998-09-01,1.0568910893610537,-1.17 587 | 1998-10-01,1.0022269717027672,-1.33 588 | 1998-11-01,1.1832335684353354,-1.25 589 | 1998-12-01,1.251493097234286,-1.57 590 | 1999-01-01,1.7166868250905176,-1.71 591 | 1999-02-01,0.7200576398447149,-1.25 592 | 1999-03-01,0.768295680728076,-0.9 593 | 1999-04-01,2.0827392856015523,-0.88 594 | 1999-05-01,0.10980533845890897,-0.92 595 | 1999-06-01,-0.021361940241000268,-1.03 596 | 1999-07-01,0.4757505098555259,-1.0 597 | 1999-08-01,0.05138760674486463,-1.11 598 | 1999-09-01,-0.11400865978527497,-1.06 599 | 1999-10-01,0.8148948274592898,-1.17 600 | 1999-11-01,1.2526314316866054,-1.54 601 | 1999-12-01,1.1967231586244518,-1.72 602 | 2000-01-01,0.570492283299991,-1.79 603 | 2000-02-01,1.1571921260909854,-1.6 604 | 2000-03-01,0.8293927527541659,-1.06 605 | 2000-04-01,1.895316177785941,-0.81 606 | 2000-05-01,0.34938062236962725,-0.81 607 | 2000-06-01,-0.7049440279629174,-0.72 608 | 2000-07-01,-0.3600274128642437,-0.59 609 | 2000-08-01,0.3939716517124983,-0.47 610 | 2000-09-01,0.9336384841877892,-0.51 611 | 2000-10-01,0.8773388755404726,-0.67 612 | 2000-11-01,2.2242015172052523,-0.73 613 | 2000-12-01,0.6490237725262339,-0.86 614 | 2001-01-01,0.9872902984965138,-0.74 615 | 2001-02-01,1.0600511291474004,-0.58 616 | 2001-03-01,0.5239073926237161,-0.44 617 | 2001-04-01,0.021085099629084454,-0.41 618 | 2001-05-01,-0.9283542251539312,-0.21 619 | 2001-06-01,0.06408582072426366,-0.08 620 | 2001-07-01,-0.2957368034242446,0.1 621 | 2001-08-01,-1.1133981461450428,0.03 622 | 2001-09-01,0.07087024797465691,-0.08 623 | 2001-10-01,-0.3090980380017164,-0.12 624 | 2001-11-01,0.6280506624246239,-0.26 625 | 2001-12-01,-1.1583842015978605,-0.36 626 | 2002-01-01,0.3099935238021495,-0.18 627 | 2002-02-01,0.6229166429011299,0.0 628 | 2002-03-01,-0.8202281919499155,0.15 629 | 2002-04-01,-0.44747266991010304,0.2 630 | 2002-05-01,-1.4873632209454255,0.52 631 | 2002-06-01,-0.7903917889281814,0.79 632 | 2002-07-01,-0.7457710695041655,0.75 633 | 2002-08-01,-1.7300494270868458,0.73 634 | 2002-09-01,-0.8535242908252128,0.92 635 | 2002-10-01,-0.8710944707322195,1.16 636 | 2002-11-01,-0.7599066026020368,1.4 637 | 2002-12-01,-1.3226940174273631,1.4 638 | 2003-01-01,-0.2110039951935336,1.01 639 | 2003-02-01,-0.9313393081967816,0.8 640 | 2003-03-01,-1.003519408028116,0.52 641 | 2003-04-01,-0.6348957777257141,-0.05 642 | 2003-05-01,-0.7686373692134523,-0.49 643 | 2003-06-01,-1.3885261156848343,-0.12 644 | 2003-07-01,0.28287868153567464,0.38 645 | 2003-08-01,-0.35971324721631137,0.4 646 | 2003-09-01,-0.29888756754527696,0.28 647 | 2003-10-01,-0.3090980380017164,0.55 648 | 2003-11-01,-0.4823151495967204,0.4 649 | 2003-12-01,0.8681035269655086,0.37 650 | 2004-01-01,-1.2529990331849,0.24 651 | 2004-02-01,0.7200576398447149,0.23 652 | 2004-03-01,-0.20925747168915493,0.07 653 | 2004-04-01,-1.7594344246198068,0.12 654 | 2004-05-01,1.3076817580123188,0.12 655 | 2004-06-01,-1.6448693985805292,0.26 656 | 2004-07-01,-0.6814804600641663,0.59 657 | 2004-08-01,-0.9763645281580364,0.75 658 | 2004-09-01,-0.36051387013194425,0.77 659 | 2004-10-01,-0.4964301822452647,0.77 660 | 2004-11-01,-1.1068959188587022,0.65 661 | 2004-12-01,-1.048844324378254,0.71 662 | 2005-01-01,0.2057940200029892,0.64 663 | 2005-02-01,-3.1655822379000087,0.34 664 | 2005-03-01,-0.20925747168915493,0.31 665 | 2005-04-01,-1.2908766550806192,0.23 666 | 2005-05-01,-1.4873632209455163,0.39 667 | 2005-06-01,0.1495335816895276,0.3 668 | 2005-07-01,0.09000685321560413,0.17 669 | 2005-08-01,-0.907847719164494,0.19 670 | 2005-09-01,0.3173754583212561,-0.02 671 | 2005-10-01,1.0022269717027672,-0.07 672 | 2005-11-01,-0.4129172863453716,-0.6 673 | 2005-12-01,-0.1177553680113211,-0.88 674 | 2006-01-01,1.404088313693096,-1.01 675 | 2006-02-01,-0.15421133264782586,-0.68 676 | 2006-03-01,1.3181693289627463,-0.52 677 | 2006-04-01,1.7078930699702235,-0.25 678 | 2006-05-01,-1.0082126531241706,0.0 679 | 2006-06-01,-0.7049440279629174,0.14 680 | 2006-07-01,-0.8743522883840907,0.09 681 | 2006-08-01,-1.8670830450739304,0.33 682 | 2006-09-01,-0.6070190804785435,0.49 683 | 2006-10-01,-1.6828670957874536,0.69 684 | 2006-11-01,-0.2741215598427529,1.08 685 | 2006-12-01,-0.5011449382799741,1.18 686 | 2007-01-01,-0.7841012660887968,0.71 687 | 2007-02-01,-0.4456343234786912,0.16 688 | 2007-03-01,-0.3925486877674248,-0.07 689 | 2007-04-01,-0.35376111600224425,-0.25 690 | 2007-05-01,-0.2894868013921973,-0.36 691 | 2007-06-01,0.4058768645853194,-0.15 692 | 2007-07-01,-0.4243180223042429,-0.34 693 | 2007-08-01,0.11990441573840695,-0.63 694 | 2007-09-01,0.07087024797465691,-0.85 695 | 2007-10-01,0.4402305389722641,-1.03 696 | 2007-11-01,0.9056421154300192,-1.26 697 | 2007-12-01,1.3610329744538925,-1.3 698 | 2008-01-01,1.560387569391777,-1.59 699 | 2008-02-01,2.031461098583526,-1.64 700 | 2008-03-01,1.1348781128844767,-1.16 701 | 2008-04-01,0.4896428691683785,-0.83 702 | 2008-05-01,-0.4492036573326762,-0.72 703 | 2008-06-01,0.4058768645852223,-0.55 704 | 2008-07-01,0.2185880720956755,-0.28 705 | 2008-08-01,0.8050725056736743,-0.12 706 | 2008-09-01,1.3650226022943204,-0.12 707 | 2008-10-01,1.2520031640274982,-0.12 708 | 2008-11-01,1.6690186111946197,-0.41 709 | 2008-12-01,1.251493097234286,-0.81 710 | 2009-01-01,1.0393900503960942,-0.95 711 | 2009-02-01,1.3514741199782103,-0.74 712 | 2009-03-01,-0.20925747168915493,-0.54 713 | 2009-04-01,0.958200638707566,-0.16 714 | 2009-05-01,-0.5290620853028248,0.24 715 | 2009-06-01,-0.3631529841019589,0.43 716 | 2009-07-01,0.15429746265560326,0.53 717 | 2009-08-01,-0.7022972921839449,0.6 718 | 2009-09-01,0.3173754583212561,0.71 719 | 2009-10-01,-1.6204230477062709,0.99 720 | 2009-11-01,-0.8293044658533856,1.54 721 | 2009-12-01,-0.9393044471585856,1.79 722 | 2010-01-01,-1.096699777486159,1.58 723 | 2010-02-01,-1.6598967852739177,1.32 724 | 2010-03-01,-1.431198912210676,1.0 725 | 2010-04-01,1.707893069970117,0.64 726 | 2010-05-01,0.9882480461313612,0.11 727 | 2010-06-01,0.06408582072426366,-0.46 728 | 2010-07-01,2.0187251364152132,-0.9 729 | 2010-08-01,1.8328246405764972,-1.3 730 | 2010-09-01,2.4742960488541224,-1.52 731 | 2010-10-01,1.7515555486768897,-1.52 732 | 2010-11-01,1.5996207479432707,-1.42 733 | 2010-12-01,2.730281439699499,-1.41 734 | 2011-01-01,2.1855845921866206,-1.59 735 | 2011-02-01,2.1286020955271665,-1.15 736 | 2011-03-01,2.1735283373278667,-0.96 737 | 2011-04-01,2.832431716864209,-0.6 738 | 2011-05-01,0.1896637664291484,-0.25 739 | 2011-06-01,-0.10680970120616706,-0.05 740 | 2011-07-01,1.054365994815372,-0.19 741 | 2011-08-01,0.05138760674486463,-0.39 742 | 2011-09-01,1.1185173919477212,-0.65 743 | 2011-10-01,0.6275626832158124,-0.8 744 | 2011-11-01,1.3220292949379544,-1.03 745 | 2011-12-01,2.292121930820888,-1.08 746 | 2012-01-01,1.0393900503960942,-0.83 747 | 2012-02-01,0.08864115971116418,-0.67 748 | 2012-03-01,0.0962278884412254,-0.41 749 | 2012-04-01,-0.8223188855414317,-0.27 750 | 2012-05-01,-0.2894868013921066,-0.2 751 | 2012-06-01,-1.2176305937543068,0.0 752 | 2012-07-01,-0.16715558454424628,0.09 753 | 2012-08-01,-0.7022972921839449,0.28 754 | 2012-09-01,0.1941228531479215,0.78 755 | 2012-10-01,0.12801029856649215,0.58 756 | 2012-11-01,0.28106134616795875,0.47 757 | 2012-12-01,-0.8297645699389795,-0.46 758 | 2013-01-01,-0.10680449139437327,-0.79 759 | 2013-02-01,-0.5427753204223313,-0.63 760 | 2013-03-01,1.0126839688323663,-0.45 761 | 2013-04-01,0.021085099629084454,0.01 762 | 2013-05-01,0.828531190190973,-0.12 763 | 2013-06-01,1.3458022352029313,-0.3 764 | 2013-07-01,0.7972035570555216,-0.33 765 | 2013-08-01,-0.22267962922922674,-0.37 766 | 2013-09-01,0.3173754583212561,-0.27 767 | 2013-10-01,-0.30909803800178737,-0.14 768 | 2013-11-01,0.8362442521785914,-0.19 769 | 2013-12-01,-0.1177553680113211,-0.43 770 | -------------------------------------------------------------------------------- /session_1/data/year_month_day.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/data/year_month_day.xlsx -------------------------------------------------------------------------------- /session_1/notebooks/images/array_3x5x8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/array_3x5x8.png -------------------------------------------------------------------------------- /session_1/notebooks/images/broadcast_2D.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/broadcast_2D.png -------------------------------------------------------------------------------- /session_1/notebooks/images/broadcast_scalar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/broadcast_scalar.png -------------------------------------------------------------------------------- /session_1/notebooks/images/ipython_console.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/ipython_console.png -------------------------------------------------------------------------------- /session_1/notebooks/images/ipython_dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/ipython_dashboard.png -------------------------------------------------------------------------------- /session_1/notebooks/images/lost_wormhole.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/lost_wormhole.jpg -------------------------------------------------------------------------------- /session_1/notebooks/images/poisson_pmf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/poisson_pmf.png -------------------------------------------------------------------------------- /session_1/notebooks/images/presentation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/presentation.png -------------------------------------------------------------------------------- /session_1/notebooks/images/python.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/python.png -------------------------------------------------------------------------------- /session_1/notebooks/images/python_console.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/python_console.png -------------------------------------------------------------------------------- /session_1/notebooks/images/qtconsole.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/qtconsole.png -------------------------------------------------------------------------------- /session_1/notebooks/images/split-apply-combine.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/split-apply-combine.png -------------------------------------------------------------------------------- /session_1/notebooks/images/www.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_1/notebooks/images/www.png -------------------------------------------------------------------------------- /session_1/notebooks/load_style.py: -------------------------------------------------------------------------------- 1 | from IPython.display import display, HTML 2 | import requests 3 | 4 | def load_style(s): 5 | """Load a CSS stylesheet in the notebook by URL or filename. 6 | 7 | Examples:: 8 | 9 | %load_style mystyle.css 10 | %load_style http://ipynbstyles.com/otherstyle.css 11 | """ 12 | if s.startswith('http'): 13 | r =requests.get(s) 14 | style = r.text 15 | else: 16 | with open(s, 'r') as f: 17 | style = f.read() 18 | s = ''.format(style=style) 19 | display(HTML(s)) 20 | 21 | def load_ipython_extension(ip): 22 | """Load the extension in IPython.""" 23 | ip.register_magic_function(load_style) -------------------------------------------------------------------------------- /session_1/notebooks/print_upper.py: -------------------------------------------------------------------------------- 1 | #!/Users/nicolasf/anaconda/bin/python 2 | # This is a python script 3 | 4 | import sys # I import the sys module, part of the Python standard library 5 | 6 | X = sys.argv[1:] # reading the command line arguments, X is list 7 | 8 | X = " ".join(map(str,X)) # transform everything into a string 9 | 10 | print(X.upper()) # printing the content, uppercase if applicable -------------------------------------------------------------------------------- /session_1/notebooks/resources.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "data": { 12 | "text/html": [ 13 | "" 86 | ], 87 | "text/plain": [ 88 | "" 89 | ] 90 | }, 91 | "metadata": {}, 92 | "output_type": "display_data" 93 | } 94 | ], 95 | "source": [ 96 | "%load_ext load_style\n", 97 | "%load_style talk.css" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "# resources" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 2, 110 | "metadata": { 111 | "collapsed": false 112 | }, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/html": [ 117 | "
" 180 | ], 181 | "text/plain": [ 182 | "" 183 | ] 184 | }, 185 | "execution_count": 2, 186 | "metadata": {}, 187 | "output_type": "execute_result" 188 | } 189 | ], 190 | "source": [ 191 | "import addutils.toc ; addutils.toc.js(ipy_notebook=True)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "### Object Oriented Programming in Python " 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "+ [www.google.com](http://www.google.com)\n", 206 | "+ [A good tutorial by Alan Gauld](http://www.freenetpages.co.uk/hp/alan.gauld/tutclass.htm)\n", 207 | "+ The [classes](http://docs.python.org/2/tutorial/classes.html) entry from the official Python doc \n", 208 | "+ A short [pdf](http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/PfES_3.pdf) from Andrew Walker's [Python for Earth Scientists](http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/) course\n" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": {}, 214 | "source": [ 215 | "### Python as a glue: wrapping of C, C++, Fortran or the Cython module " 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "+ [using python as a *glue*](http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html)\n", 223 | "+ [SWIG (Simplified Wrapper and Interface Generator)](http://www.swig.org/Doc1.3/Python.html)\n", 224 | "+ [Cython](http://cython.org)\n", 225 | "+ [f2py](http://www.f2py.com/)\n", 226 | "+ [A notebook on using C and Fortran with Python](http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-6A-Fortran-and-C.ipynb)" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "metadata": {}, 232 | "source": [ 233 | "### Real-time acquisition, physical programming" 234 | ] 235 | }, 236 | { 237 | "cell_type": "markdown", 238 | "metadata": {}, 239 | "source": [ 240 | "+ [pyserial](http://pyserial.sourceforge.net): Module that encapsulates the access for the serial port\n", 241 | "+ [Real World Instrumentation, Automated Data Acquisition and Control Systems](http://www.amazon.com/Real-World-Instrumentation-Python-Acquisition/dp/0596809565): A book available on Amazon.\n", 242 | "+ [Programming Raspberry Pi: Getting Started with Python](http://www.monkmakes.com/?page_id=63): book (not free) with code (free)\n", 243 | "+ [PyVISA](https://pyvisa.readthedocs.org/en/latest/): The PyVISA package enables you to control all kinds of measurement equipment through various busses (GPIB, RS232, USB) with Python programs" 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": {}, 249 | "source": [ 250 | "### Image processing" 251 | ] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": {}, 256 | "source": [ 257 | "+ PIL [(Python Image Library)](http://www.pythonware.com/products/pil/): Image processing capabilities to your Python interpreter.\n", 258 | "+ [pillow](http://python-pillow.github.io/): A more user-friendly `fork` of PIL\n", 259 | "+ [scikit-image](http://scikit-image.org): more Image processing, built on top of Numpy / Scipy." 260 | ] 261 | }, 262 | { 263 | "cell_type": "markdown", 264 | "metadata": {}, 265 | "source": [ 266 | "### Symbolic maths" 267 | ] 268 | }, 269 | { 270 | "cell_type": "markdown", 271 | "metadata": {}, 272 | "source": [ 273 | "+ The [SymPy](www.sympy.org) library is a Python library for symbolic mathematics. It supports polynomials, calculus, solving equations, etc\n", 274 | "+ The [sage](http://www.sagemath.org/) software: `Mission: Creating a viable free open source alternative to Magma, Maple, Mathematica and Matlab`" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "### Geospatial statistics" 282 | ] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "metadata": {}, 287 | "source": [ 288 | "+ [PySAL](http://pysal.readthedocs.org/en/v1.6/): PySAL is a cross-platform library of spatial analysis functions written in Python. It is intended to support the development of high level applications for spatial analysis.\n", 289 | "+ [GeoPandas](http://geopandas.org/): A project based on [Pandas](http://pandas.pydata.org/) to make working with geospatial data in python easier\n", 290 | "+ [Rasterio](https://github.com/mapbox/rasterio): Clean and fast and geospatial raster I/O with Numpy support, developed by the team at [https://www.mapbox.com/](https://www.mapbox.com/)\n", 291 | "+ [Pyproj](https://code.google.com/p/pyproj/): Performs cartographic transformations and geodetic computations. Wrapper around the Proj version 4 library\n", 292 | "+ [Python GDAL/OGR](https://pypi.python.org/pypi/GDAL/): Python bindings + tools around the Geospatial Data Abstraction Library\n", 293 | "+ [Python GIS resources](http://pythongisresources.wordpress.com/interpolation): a blog on geospatial python\n", 294 | "+ [High Performance Geostatistics Library](http://hpgl.mit-ufa.com/): A library written in C++ / Python implementing geostatistical algorithms (*e.g.* kriging, correlograms, etc)" 295 | ] 296 | }, 297 | { 298 | "cell_type": "markdown", 299 | "metadata": {}, 300 | "source": [ 301 | "### Biology, ecology" 302 | ] 303 | }, 304 | { 305 | "cell_type": "markdown", 306 | "metadata": {}, 307 | "source": [ 308 | "+ [Biopython](http://biopython.org/wiki/Main_Page): Biopython is a set of freely available tools for biological computation written in Python by an international team of developers. \n", 309 | "+ [GeoEco](https://pypi.python.org/pypi/GeoEco): Open source geoprocessing toolbox designed for coastal and marine researchers and GIS analysts who work with spatially-explicit ecological and oceanographic data. For Windows (> XP) only.\n", 310 | "+ [Galaxy](http://galaxyproject.org/): Galaxy is a scientific workflow, data integration and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming experience." 311 | ] 312 | }, 313 | { 314 | "cell_type": "markdown", 315 | "metadata": {}, 316 | "source": [ 317 | "### Computational Fluid Dynamics and PDE solvers" 318 | ] 319 | }, 320 | { 321 | "cell_type": "markdown", 322 | "metadata": {}, 323 | "source": [ 324 | "+ [CFD Python: 12 steps to Navier-Stokes](http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/)\n", 325 | "+ [github repository for the above](https://github.com/barbagroup/CFDPython)\n", 326 | "+ [Python bindings to OpenFOAM](http://openfoamwiki.net/index.php/Contrib_PyFoam)\n", 327 | "+ [FiPy: A Finite Volume PDE Solver Using Python](http://matforge.org/fipy/)\n", 328 | "+ [\"Practical Numerical Methods with Python\" MOOC](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about)" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": {}, 334 | "source": [ 335 | "### Python on the GPU " 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "metadata": {}, 341 | "source": [ 342 | "+ [PyCUDA](http://mathema.tician.de/software/pycuda/): PyCUDA lets you access Nvidia‘s CUDA parallel computation API from Python" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "### Parallel computing with Python / IPython" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "+ [IPython parallel introduction](http://ipython.org/ipython-doc/2/parallel/parallel_intro.html)\n", 357 | "+ [Interactive Parallel computing with IPython](https://www.youtube.com/watch?v=y4hgalfhc1Y&index=49&list=PLYx7XA2nY5GfuhCvStxgbynFNrxr3VFog): tutorials by Min. Raglan-Kelley at Scipy2014 (3 parts in the youtube list)\n", 358 | "+ [IPython parallel for distributed computing](https://www.youtube.com/watch?v=eyUeYnT-18s&index=32&list=PLs4CJRBY5F1Jm7H1dlesRvEgr-28QGXpR): A talk at PyCON Australia by Nathan Faggian\n", 359 | "+ [IPython & Jupyter in depth: high productivity interactive and parallel python](https://www.youtube.com/watch?v=05fA_DXgW-Y): A talk at PyCON 2015 by Thomas Kluyver and Kyle Kelley" 360 | ] 361 | }, 362 | { 363 | "cell_type": "markdown", 364 | "metadata": {}, 365 | "source": [ 366 | "### Signal processing in Python" 367 | ] 368 | }, 369 | { 370 | "cell_type": "markdown", 371 | "metadata": {}, 372 | "source": [ 373 | "+ [scipy.signal](http://docs.scipy.org/doc/scipy/reference/tutorial/signal.html)\n", 374 | "+ [Python for signal processing blog](http://python-for-signal-processing.blogspot.co.nz/)\n", 375 | "+ [Python for signal processing github repository](https://github.com/unpingco/Python-for-Signal-Processing)\n", 376 | "+ [Python for signal processing book](http://www.springer.com/engineering/signals/book/978-3-319-01341-1)" 377 | ] 378 | }, 379 | { 380 | "cell_type": "markdown", 381 | "metadata": {}, 382 | "source": [ 383 | "### Python for Matlab and R users " 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": {}, 389 | "source": [ 390 | "+ [Moving from MATLAB matrices to NumPy arrays - A Matrix Cheatsheet](http://sebastianraschka.com/Articles/2014_matlab_vs_numpy.html)\n", 391 | "\n", 392 | "+ [NumPy for Matlab Users](http://wiki.scipy.org/NumPy_for_Matlab_Users)\n" 393 | ] 394 | }, 395 | { 396 | "cell_type": "markdown", 397 | "metadata": {}, 398 | "source": [ 399 | "### Some URLs and blogs" 400 | ] 401 | }, 402 | { 403 | "cell_type": "markdown", 404 | "metadata": {}, 405 | "source": [ 406 | "+ [Python Scientific Lectures Notes](http://scipy-lectures.github.io): Tutorial material on the scientific Python ecosystem, a quick introduction to central tools and techniques.\n", 407 | "\n", 408 | "+ [Python for earth scientists](http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/): A two afternoons course by Andrew Walker (University of Bristol) on Python in the earth sciences. \n", 409 | "\n", 410 | "+ [oceanpython](http://oceanpython.org/): Python for oceanography \n", 411 | "\n", 412 | "+ [PyAOS](http://pyaos.johnny-lin.com/): Python for the Atmospheric and Oceanic Sciences \n", 413 | "\n", 414 | "+ [python4oceanographers](http://ocefpaf.github.io/): Learn python with examples applied to marine sciences.\n", 415 | "\n", 416 | "+ [Pythonic perambulations](http://jakevdp.github.io): A blog by Jake VanderPlas" 417 | ] 418 | }, 419 | { 420 | "cell_type": "markdown", 421 | "metadata": {}, 422 | "source": [ 423 | "### Github repositories: notebooks and accompanying material" 424 | ] 425 | }, 426 | { 427 | "cell_type": "markdown", 428 | "metadata": {}, 429 | "source": [ 430 | "+ [A gallery of interesting IPython Notebooks](https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks)\n", 431 | "\"...a curated collection of IPython notebooks that are notable for some reason.\"\n", 432 | "\n", 433 | "+ [Scikit-learn tutorial](https://github.com/jakevdp/sklearn_scipy2013): Files and other info associated with the Scipy 2013 scikit-learn tutorial developed by Gaël Varoquaux, Olivier Grisel and Jake VanderPlas.\n", 434 | "\n", 435 | "+ [Statistical Analysis tutorial](https://github.com/fonnesbeck/statistical-analysis-python-tutorial) from Chris. Fonnesbeck. \n", 436 | "\n", 437 | "+ [Bayesian Statistical Analysis in Python](https://github.com/fonnesbeck/scipy2014_tutorial): Ipython notebooks for the Scipy 2014 tutorial on Bayesian data analysis with Python, by Chris. Fonnesbeck. \n", 438 | "\n", 439 | "+ [AstroML: Machine Learning and Data Mining for Astronomy](http://www.astroml.org): A library and tutorial by Jake VanderPlas and co-authors, accompanying the book [Statistics, Data Mining, and Machine Learning in Astronomy](http://www.amazon.com/Statistics-Mining-Machine-Learning-Astronomy/dp/0691151687/)\n", 440 | "\n", 441 | "+ [Scientific Python lectures from Robert Johansson](https://github.com/jrjohansson/scientific-python-lectures)\n", 442 | "\n", 443 | "+ [Data Science in Python](http://blog.yhathq.com/posts/data-science-in-python-tutorial.html): A series of annotated notebooks on data science (i.e. geared towards machine learning) in python\n", 444 | "\n", 445 | "+ [http://earthpy.org/](http://earthpy.org/): EarthPy is a collection of IPython notebooks with examples of Earth Science related Python code\n", 446 | "\n", 447 | "+ [data science notebooks): IPython notebooks for data science, continually updated Data Science Python Notebooks: Spark, Hadoop MapReduce, HDFS, AWS, Kaggle, scikit-learn, matplotlib, pandas, NumPy, SciPy." 448 | ] 449 | }, 450 | { 451 | "cell_type": "markdown", 452 | "metadata": {}, 453 | "source": [ 454 | "### Some books " 455 | ] 456 | }, 457 | { 458 | "cell_type": "markdown", 459 | "metadata": {}, 460 | "source": [ 461 | "+ [Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do): From Wes McKinney (Developer of [Pandas](www.pandas.pydata.org))\n", 462 | "\n", 463 | "+ [Think stats](http://www.greenteapress.com/thinkstats/): Probability and statistics for programmers, from Allen Downey, pdf available for free. \n", 464 | "\n", 465 | "+ [Think complexity](http://www.greenteapress.com/compmod/): Complexity science (graphs, cellular automata, agent-based models), from Allen Downey, pdf available for free.\n", 466 | "\n", 467 | "+ [Python in hydrology](http://www.greenteapress.com/pythonhydro/pythonhydro.html): A book freely available in pdf, from Sat Kumar Tomer.\n", 468 | "\n", 469 | "+ [Programming collective intelligence](http://shop.oreilly.com/product/9780596529321.do): By Toby Segaran, Good intro on (general) Machine Learning algorithms.\n", 470 | "\n", 471 | "+ [Machine Learning in action](http://www.manning.com/pharrington/): By Peter Harrington.\n", 472 | "\n", 473 | "+ [Introduction to Python for Econometrics, Statistics and Numerical Analysis: Second Edition](http://www.kevinsheppard.com/images/0/09/Python_introduction.pdf): By Kevin Sheppard, Oxford Uni.\n", 474 | "\n", 475 | "+ [A Hands-On Introduction to Using Python in the Atmospheric and Oceanic Sciences](http://www.johnny-lin.com/pyintro/): By [Johnny Lin](http://www.johnny-lin.com), Professor of physics and head of the Climate Research Group at North Park University.\n", 476 | "\n", 477 | "+ [principles of planetary climate](http://geosci.uchicago.edu/~rtp1/PrinciplesPlanetaryClimate/index.html): by Ray Pierrehumbert. Excellent book on the physics of planetary climates, with freely downloadable python code to follow the examples given in the book. " 478 | ] 479 | }, 480 | { 481 | "cell_type": "markdown", 482 | "metadata": {}, 483 | "source": [ 484 | "### Some interesting libraries, built on top of the main Scientific stack" 485 | ] 486 | }, 487 | { 488 | "cell_type": "markdown", 489 | "metadata": {}, 490 | "source": [ 491 | "+ [PYMC](http://pymc-devs.github.io/pymc/): By Chris Fonnesbeck, Bayesian statistical models and fitting algorithms, including Markov chain Monte Carlo. \n", 492 | "\n", 493 | "+ [Seaborn](http://www.stanford.edu/~mwaskom/software/seaborn/): Statistical data visualization, by Michael Waskom. Its [graphical representation of linear models](http://www.stanford.edu/~mwaskom/software/seaborn/linear_models.html) is particularly interesting. \n", 494 | "\n", 495 | "+ [ggplot](https://github.com/yhat/ggplot/): For [R](http://cran.r-project.org) users, a 'port' of the [ggplot2](http://ggplot2.org) package to Python, see [here](http://blog.yhathq.com/posts/ggplot-0.4-released.html) for what's new in the latest release. \n", 496 | "\n", 497 | "+ [coards](https://pypi.python.org/pypi/coards): A COARDS compliant time parser. See also [netcdftime](http://netcdf4-python.googlecode.com/svn/trunk/docs/netcdftime.netcdftime-module.html) which is part of the [NetCDF4 module](https://code.google.com/p/netcdf4-python/)\n", 498 | "\n", 499 | "+ [seawater](http://www.imr.no/~bjorn/python/seawater/index.html): Similar to the MATLAB toolboxes SEAWATER from CSIRO and parts of OCEANS from Woods Hole Institute.\n", 500 | "\n", 501 | "+ [fluid](https://pypi.python.org/pypi/fluid): Procedures to study fluids on Python, focused for oceanography, meteorology and related sciences.\n", 502 | "\n", 503 | "+ [kyPyWavelet](https://github.com/regeirk/kPyWavelet): Continuous wavelet transform module for Python *ala* Torrence and Compo. Some manual edits were necessary to make it work for me ... \n", 504 | "\n", 505 | "+ [pyresample](https://code.google.com/p/pyresample/): Resampling (reprojection) of geospatial image data in Python\n", 506 | "\n", 507 | "+ [Rpy2](http://rpy.sourceforge.net/rpy2.html): calling R from Python" 508 | ] 509 | }, 510 | { 511 | "cell_type": "markdown", 512 | "metadata": {}, 513 | "source": [ 514 | "### Some articles on open and reproducible research" 515 | ] 516 | }, 517 | { 518 | "cell_type": "markdown", 519 | "metadata": {}, 520 | "source": [ 521 | "+ [Reproducible Research in Computational Science](http://dx.doi.org/10.1126/science.1213847), Roger D. Peng, Science 334, 1226 (2011).\n", 522 | "\n", 523 | "+ [Shining Light into Black Boxes](http://dx.doi.org/10.1126/science.1218263), A. Morin et al., Science 336, 159-160 (2012).\n", 524 | " \n", 525 | "+ [The case for open computer programs](http://dx.doi.org/doi:10.1038/nature10836), D.C. Ince, Nature 482, 485 (2012).\n", 526 | "\n", 527 | "+ [Best practices for scientific computing](http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001745): Paper in PLOS Biology exposing some of the tools and methods to build better Scientific software.\n" 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "## Acknowledgments" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": {}, 540 | "source": [ 541 | "In building this material, I have liberally '*borrowed*' from lecture notes, online notebooks, video recording of talks, code examples, articles, etc, that are freely available online, I would like to acknowledge in particular (list not exhaustive and in no particular order): \n", 542 | "\n", 543 | "+ [Fernando Perez](http://blog.fperez.org): Lead developer of [IPython](www.ipython.org), great advocate of [open science](http://en.wikipedia.org/wiki/Open_science) and reproducible research\n", 544 | "+ Brian Granger: Lead developer of the [IPython notebook](http://ipython.org/notebook.html)\n", 545 | "+ [Wes McKinney](http://blog.wesmckinney.com): Lead developer of Pandas, author of [Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do)\n", 546 | "+ [Christopher Fonnesbeck](http://biostat.mc.vanderbilt.edu/wiki/Main/ChrisFonnesbeck): developer of [PYMC](http://pymc-devs.github.io/pymc/) (Bayesian statistical models and fitting algorithms, including Markov chain Monte Carlo)\n", 547 | "+ [Jake VanderPlas](http://www.astro.washington.edu/users/vanderplas/): Astronomer, contributor to [scikit-learn](http://scikit-learn.org)\n", 548 | "+ [Gaël Varoquaux](http://gael-varoquaux.info): One of the main contributors to [scikit-learn](http://scikit-learn.org)\n", 549 | "+ [Olivier Grisel](http://ogrisel.com): Also main contributor to [scikit-learn](http://scikit-learn.org)\n", 550 | "+ [Travis Oliphant](http://technicaldiscovery.blogspot.co.nz): One of the lead developers of Numpy / Scipy, and founder of [Continuum Analytics](http://continuum.io).\n", 551 | "+ J.R. Johansson http://dml.riken.jp/~rob/\n", 552 | "+ [Thomas Wiecki](http://twiecki.github.io/): developer of [PYMC3](https://github.com/pymc-devs/pymc), a fork of PYMC. \n", 553 | "+ [Skipper Seabold](http://jseabold.net/): The main developer of [statsmodels](http://statsmodels.sourceforge.net/)\n", 554 | "+ [Kevin Sheppard](http://www.kevinsheppard.com/wiki/Main_Page) from University of Oxford: author or the book [Introduction to Python for Econometrics, Statistics and Numerical Analysis: Second Edition](http://www.kevinsheppard.com/images/0/09/Python_introduction.pdf). \n", 555 | "\n", 556 | "And finally [John Hunter](http://blog.fperez.org/2013/07/in-memoriam-john-d-hunter-iii-1968-2012.html), He was the founder and lead developer of [Matplotlib](http://matplotlib.org), a pivotal library to make Python a viable free and open-source alternative to commercial scientific software, and very sadly passed away in 2012.\n", 557 | "\n", 558 | "I encourage you - if you carry on using Python for science - to look up these people on google, have a look at their github repositories and the projects they contribute to, and follow them on twitter, some of them (*e.g.* Jake VanderPlas) have also very informative blogs. You can also donate to the [John Hunter memorial fund](http://numfocus.org/johnhunter.html) as a way to give back to an important contributor to the Python scientific community.\n", 559 | "\n", 560 | "---" 561 | ] 562 | }, 563 | { 564 | "cell_type": "markdown", 565 | "metadata": {}, 566 | "source": [ 567 | "
\n", 568 | "![](files/images/python.png)\n", 569 | "
" 570 | ] 571 | }, 572 | { 573 | "cell_type": "code", 574 | "execution_count": 4, 575 | "metadata": { 576 | "collapsed": false 577 | }, 578 | "outputs": [ 579 | { 580 | "name": "stdout", 581 | "output_type": "stream", 582 | "text": [ 583 | "[NbConvertApp] Converting notebook resources.ipynb to html\n", 584 | "[NbConvertApp] Writing 229088 bytes to resources.html\n" 585 | ] 586 | } 587 | ], 588 | "source": [ 589 | "!ipython nbconvert resources.ipynb --to html" 590 | ] 591 | } 592 | ], 593 | "metadata": { 594 | "kernelspec": { 595 | "display_name": "Python 2", 596 | "language": "python", 597 | "name": "python2" 598 | }, 599 | "language_info": { 600 | "codemirror_mode": { 601 | "name": "ipython", 602 | "version": 2 603 | }, 604 | "file_extension": ".py", 605 | "mimetype": "text/x-python", 606 | "name": "python", 607 | "nbconvert_exporter": "python", 608 | "pygments_lexer": "ipython2", 609 | "version": "2.7.10" 610 | } 611 | }, 612 | "nbformat": 4, 613 | "nbformat_minor": 0 614 | } 615 | -------------------------------------------------------------------------------- /session_1/notebooks/talk.css: -------------------------------------------------------------------------------- 1 | 2 | .rendered_html { 3 | font-family: "proxima-nova", helvetica; 4 | font-size: 130%; 5 | line-height: 1.5; 6 | } 7 | 8 | .rendered_html h1 { 9 | margin: 0.25em 0em 0.5em; 10 | color: #015C9C; 11 | text-align: center; 12 | line-height: 1.2; 13 | page-break-before: always; 14 | } 15 | 16 | .rendered_html h2 { 17 | margin: 1.1em 0em 0.5em; 18 | color: #26465D; 19 | line-height: 1.2; 20 | } 21 | 22 | .rendered_html h3 { 23 | margin: 1.1em 0em 0.5em; 24 | color: #002845; 25 | line-height: 1.2; 26 | } 27 | 28 | .rendered_html li { 29 | line-height: 1.5; 30 | } 31 | 32 | /*.prompt { 33 | font-size: 120%; 34 | }*/ 35 | 36 | .CodeMirror-lines { 37 | font-size: 110%; 38 | } 39 | 40 | /*.output_area { 41 | font-size: 120%; 42 | }*/ 43 | 44 | /*#notebook { 45 | background-image: url('files/images/witewall_3.png'); 46 | }*/ 47 | 48 | h1.bigtitle { 49 | margin: 4cm 1cm 4cm 1cm; 50 | font-size: 300%; 51 | } 52 | 53 | h3.point { 54 | font-size: 200%; 55 | text-align: center; 56 | margin: 2em 0em 2em 0em; 57 | #26465D 58 | } 59 | 60 | .logo { 61 | margin: 20px 0 20px 0; 62 | } 63 | 64 | a.anchor-link { 65 | display: none; 66 | } 67 | 68 | h1.title { 69 | font-size: 250%; 70 | } 71 | -------------------------------------------------------------------------------- /session_1/notebooks/talktools.py: -------------------------------------------------------------------------------- 1 | 2 | from IPython.display import HTML, display, YouTubeVideo 3 | 4 | def prefix(url): 5 | prefix = '' if url.startswith('http') else 'http://' 6 | return prefix + url 7 | 8 | 9 | def simple_link(url, name=None): 10 | name = url if name is None else name 11 | url = prefix(url) 12 | return '%s' % (url, name) 13 | 14 | 15 | def html_link(url, name=None): 16 | return HTML(simple_link(url, name)) 17 | 18 | 19 | # Utility functions 20 | def website(url, name=None, width=800, height=450): 21 | html = [] 22 | name = url if name == 'auto' else name 23 | if name: 24 | html.extend(['
', 25 | simple_link(url, name), 26 | '
'] ) 27 | 28 | html.append('" 164 | ], 165 | "text/plain": [ 166 | "" 167 | ] 168 | }, 169 | "execution_count": 3, 170 | "metadata": {}, 171 | "output_type": "execute_result" 172 | } 173 | ], 174 | "source": [ 175 | "HTML('')" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "##
conda" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "[conda](http://conda.pydata.org/docs/) is a package and **environment management system** akin to virtualenv" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "example, I want to test the [iris](http://scitools.org.uk/iris/index.html) and [cartopy](http://scitools.org.uk/cartopy/index.html) packages.\n", 197 | "\n", 198 | "These are developed by the [UK Met. Office](http://scitools.org.uk/).\n", 199 | "\n", 200 | "+ [iris](http://scitools.org.uk/iris/index.html) is a Python package for analysing and visualising meteorological and oceanographic data sets\n", 201 | "+ [cartopy](http://scitools.org.uk/cartopy/index.html) is a Python package for advanced map generation with a simple matplotlib interface.\n", 202 | "\n", 203 | "problem is that they require older versions of numpy and other libraries, and I don't want to mess up my main environment" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "first create a conda environment named *iris*, with the minimum: python and ipython [+ libraries for the notebook]\n", 211 | "\n", 212 | " ᐅ conda create -n iris python ipython ipython-notebook\n", 213 | "\n", 214 | "then: \n", 215 | "\n", 216 | " ᐅ source activate iris\n", 217 | "\n", 218 | "and to install (in the `iris` environment) using the binstar service (see below first)\n", 219 | "\n", 220 | " ᐅ conda install -c https://conda.binstar.org/scitools cdat-lite\n", 221 | " ᐅ conda install -c https://conda.binstar.org/scitools iris\n", 222 | " ᐅ conda install -c https://conda.binstar.org/scitools cartopy" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "A (quick and dirty) exploration of iris and cartopy can be found in [this notebook](./iris_cartopy.ipynb)\n", 230 | "\n", 231 | "IMPORTANT: **you need to launch a new instance of the jupyter notebook from the `iris` environment**: \n", 232 | "\n", 233 | " ~/Documents/talks_seminars/Python_NIWA_April2015/session_2/notebookss ᐅ source activate iris\n", 234 | "\n", 235 | " (iris)~/Documents/talks_seminars/Python_NIWA_April2015/session_2/notebooks ᐅ jupyter notebook " 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "some useful conda commands: \n", 243 | "\n", 244 | "+ list the existing environments\n", 245 | " \n", 246 | "```\n", 247 | "ᐅ conda env list\n", 248 | "``` \n", 249 | "\n", 250 | " or \n", 251 | "\n", 252 | "```\n", 253 | "ᐅ conda info -e\n", 254 | "``` \n", 255 | "\n", 256 | "+ remove an environment\n", 257 | "\n", 258 | "```\n", 259 | "ᐅ conda env remove -n envname\n", 260 | "```" 261 | ] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": {}, 266 | "source": [ 267 | "## anaconda.org and conda-forge" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "[anaconda.org](http://anaconda.org/) is a free service for hosting public packages for pip and conda, you need to register (for free) and you can install third-party packages not part of the anaconda distribution prepared and compiled for your platform" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "see also [https://conda-forge.github.io/](https://conda-forge.github.io/) for another service" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "metadata": { 288 | "collapsed": true 289 | }, 290 | "outputs": [], 291 | "source": [] 292 | } 293 | ], 294 | "metadata": { 295 | "kernelspec": { 296 | "display_name": "Python 3", 297 | "language": "python", 298 | "name": "python3" 299 | }, 300 | "language_info": { 301 | "codemirror_mode": { 302 | "name": "ipython", 303 | "version": 3 304 | }, 305 | "file_extension": ".py", 306 | "mimetype": "text/x-python", 307 | "name": "python", 308 | "nbconvert_exporter": "python", 309 | "pygments_lexer": "ipython3", 310 | "version": "3.5.1" 311 | }, 312 | "latex_envs": { 313 | "bibliofile": "biblio.bib", 314 | "cite_by": "apalike", 315 | "current_citInitial": 1, 316 | "eqLabelWithNumbers": true, 317 | "eqNumInitial": 0 318 | } 319 | }, 320 | "nbformat": 4, 321 | "nbformat_minor": 0 322 | } 323 | -------------------------------------------------------------------------------- /session_2/notebooks/figure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_2/notebooks/figure.png -------------------------------------------------------------------------------- /session_2/notebooks/images/clim_sst.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_2/notebooks/images/clim_sst.png -------------------------------------------------------------------------------- /session_2/notebooks/images/seas_clim_sst.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_2/notebooks/images/seas_clim_sst.png -------------------------------------------------------------------------------- /session_2/notebooks/images/split-apply-combine.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_2/notebooks/images/split-apply-combine.png -------------------------------------------------------------------------------- /session_2/notebooks/load_style.py: -------------------------------------------------------------------------------- 1 | from IPython.display import display, HTML 2 | import requests 3 | 4 | def load_style(s): 5 | """Load a CSS stylesheet in the notebook by URL or filename. 6 | 7 | Examples:: 8 | 9 | %load_style mystyle.css 10 | %load_style http://ipynbstyles.com/otherstyle.css 11 | """ 12 | if s.startswith('http'): 13 | r =requests.get(s) 14 | style = r.text 15 | else: 16 | with open(s, 'r') as f: 17 | style = f.read() 18 | s = ''.format(style=style) 19 | display(HTML(s)) 20 | 21 | def load_ipython_extension(ip): 22 | """Load the extension in IPython.""" 23 | ip.register_magic_function(load_style) -------------------------------------------------------------------------------- /session_2/notebooks/load_style.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_2/notebooks/load_style.pyc -------------------------------------------------------------------------------- /session_2/notebooks/map_notebook.py: -------------------------------------------------------------------------------- 1 | from IPython.display import HTML 2 | import folium 3 | 4 | def inline_map(map): 5 | """ 6 | Embeds the HTML source of the map directly into the IPython notebook. 7 | 8 | This method will not work if the map depends on any files (json data). Also this uses 9 | the HTML5 srcdoc attribute, which may not be supported in all browsers. 10 | """ 11 | map._build_map() 12 | return HTML(''.format(srcdoc=map.HTML.replace('"', '"'))) 13 | 14 | def embed_map(map, path="map.html"): 15 | """ 16 | Embeds a linked iframe to the map into the IPython notebook. 17 | 18 | Note: this method will not capture the source of the map into the notebook. 19 | This method should work for all maps (as long as they use relative urls). 20 | """ 21 | map.create_map(path=path) 22 | return HTML(''.format(path=path)) 23 | -------------------------------------------------------------------------------- /session_2/notebooks/map_notebook.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nicolasfauchereau/Python-for-data-analysis-and-visualisation/e8a56a80faa0ced7b7c7cf4dd98cecd351d8a9d8/session_2/notebooks/map_notebook.pyc -------------------------------------------------------------------------------- /session_2/notebooks/mpld3_figure.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 6 | 7 |
8 | -------------------------------------------------------------------------------- /session_2/notebooks/python.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "a = ['a','b','c']\n", 12 | "b = [1,2,3]" 13 | ] 14 | } 15 | ], 16 | "metadata": { 17 | "kernelspec": { 18 | "display_name": "Python 2", 19 | "language": "python", 20 | "name": "python2" 21 | }, 22 | "language_info": { 23 | "codemirror_mode": { 24 | "name": "ipython", 25 | "version": 2 26 | }, 27 | "file_extension": ".py", 28 | "mimetype": "text/x-python", 29 | "name": "python", 30 | "nbconvert_exporter": "python", 31 | "pygments_lexer": "ipython2", 32 | "version": "2.7.9" 33 | } 34 | }, 35 | "nbformat": 4, 36 | "nbformat_minor": 0 37 | } 38 | -------------------------------------------------------------------------------- /session_2/notebooks/rickshaw.min.css: -------------------------------------------------------------------------------- 1 | .rickshaw_graph .detail{pointer-events:none;position:absolute;top:0;z-index:2;background:rgba(0,0,0,.1);bottom:0;width:1px;transition:opacity .25s linear;-moz-transition:opacity .25s linear;-o-transition:opacity .25s linear;-webkit-transition:opacity .25s linear}.rickshaw_graph .detail.inactive{opacity:0}.rickshaw_graph .detail .item.active{opacity:1}.rickshaw_graph .detail .x_label{font-family:Arial,sans-serif;border-radius:3px;padding:6px;opacity:.5;border:1px solid #e0e0e0;font-size:12px;position:absolute;background:#fff;white-space:nowrap}.rickshaw_graph .detail .item{position:absolute;z-index:2;border-radius:3px;padding:.25em;font-size:12px;font-family:Arial,sans-serif;opacity:0;background:rgba(0,0,0,.4);color:#fff;border:1px solid rgba(0,0,0,.4);margin-left:1em;margin-top:-1em;white-space:nowrap}.rickshaw_graph .detail .item.active{opacity:1;background:rgba(0,0,0,.8)}.rickshaw_graph .detail .item:before{content:"\25c2";position:absolute;left:-.5em;color:rgba(0,0,0,.7);width:0}.rickshaw_graph .detail .dot{width:4px;height:4px;margin-left:-4px;margin-top:-3px;border-radius:5px;position:absolute;box-shadow:0 0 2px rgba(0,0,0,.6);background:#fff;border-width:2px;border-style:solid;display:none;background-clip:padding-box}.rickshaw_graph .detail .dot.active{display:block}.rickshaw_graph{position:relative}.rickshaw_graph svg{display:block;overflow:hidden}.rickshaw_graph .x_tick{position:absolute;top:0;bottom:0;width:0;border-left:1px dotted rgba(0,0,0,.2);pointer-events:none}.rickshaw_graph .x_tick .title{position:absolute;font-size:12px;font-family:Arial,sans-serif;opacity:.5;white-space:nowrap;margin-left:3px;bottom:1px}.rickshaw_annotation_timeline{height:1px;border-top:1px solid #e0e0e0;margin-top:10px;position:relative}.rickshaw_annotation_timeline .annotation{position:absolute;height:6px;width:6px;margin-left:-2px;top:-3px;border-radius:5px;background-color:rgba(0,0,0,.25)}.rickshaw_graph .annotation_line{position:absolute;top:0;bottom:-6px;width:0;border-left:2px solid rgba(0,0,0,.3);display:none}.rickshaw_graph .annotation_line.active{display:block}.rickshaw_graph .annotation_range{background:rgba(0,0,0,.1);display:none;position:absolute;top:0;bottom:-6px}.rickshaw_graph .annotation_range.active{display:block}.rickshaw_graph .annotation_range.active.offscreen{display:none}.rickshaw_annotation_timeline .annotation .content{background:#fff;color:#000;opacity:.9;padding:5px 5px;box-shadow:0 0 2px rgba(0,0,0,.8);border-radius:3px;position:relative;z-index:20;font-size:12px;padding:6px 8px 8px;top:18px;left:-11px;width:160px;display:none;cursor:pointer}.rickshaw_annotation_timeline .annotation .content:before{content:"\25b2";position:absolute;top:-11px;color:#fff;text-shadow:0 -1px 1px rgba(0,0,0,.8)}.rickshaw_annotation_timeline .annotation.active,.rickshaw_annotation_timeline .annotation:hover{background-color:rgba(0,0,0,.8);cursor:none}.rickshaw_annotation_timeline .annotation .content:hover{z-index:50}.rickshaw_annotation_timeline .annotation.active .content{display:block}.rickshaw_annotation_timeline .annotation:hover .content{display:block;z-index:50}.rickshaw_graph .y_axis,.rickshaw_graph .x_axis_d3{fill:none}.rickshaw_graph .y_ticks .tick,.rickshaw_graph .x_ticks_d3 .tick{stroke:rgba(0,0,0,.16);stroke-width:2px;shape-rendering:crisp-edges;pointer-events:none}.rickshaw_graph .y_grid .tick,.rickshaw_graph .x_grid_d3 .tick{z-index:-1;stroke:rgba(0,0,0,.20);stroke-width:1px;stroke-dasharray:1 1}.rickshaw_graph .y_grid path,.rickshaw_graph .x_grid_d3 path{fill:none;stroke:none}.rickshaw_graph .y_ticks path,.rickshaw_graph .x_ticks_d3 path{fill:none;stroke:#808080}.rickshaw_graph .y_ticks text,.rickshaw_graph .x_ticks_d3 text{opacity:.5;font-size:12px;pointer-events:none}.rickshaw_graph .x_tick.glow .title,.rickshaw_graph .y_ticks.glow text{fill:black;color:#000;text-shadow:-1px 1px 0 rgba(255,255,255,.1),1px -1px 0 rgba(255,255,255,.1),1px 1px 0 rgba(255,255,255,.1),0px 1px 0 rgba(255,255,255,.1),0px -1px 0 rgba(255,255,255,.1),1px 0 0 rgba(255,255,255,.1),-1px 0 0 rgba(255,255,255,.1),-1px -1px 0 rgba(255,255,255,.1)}.rickshaw_graph .x_tick.inverse .title,.rickshaw_graph .y_ticks.inverse text{fill:white;color:#fff;text-shadow:-1px 1px 0 rgba(0,0,0,.8),1px -1px 0 rgba(0,0,0,.8),1px 1px 0 rgba(0,0,0,.8),0px 1px 0 rgba(0,0,0,.8),0px -1px 0 rgba(0,0,0,.8),1px 0 0 rgba(0,0,0,.8),-1px 0 0 rgba(0,0,0,.8),-1px -1px 0 rgba(0,0,0,.8)}.rickshaw_legend{font-family:Arial;font-size:12px;color:#fff;background:#404040;display:inline-block;padding:12px 5px;border-radius:2px;position:relative}.rickshaw_legend:hover{z-index:10}.rickshaw_legend .swatch{width:10px;height:10px;border:1px solid rgba(0,0,0,.2)}.rickshaw_legend .line{clear:both;line-height:140%;padding-right:15px}.rickshaw_legend .line .swatch{display:inline-block;margin-right:3px;border-radius:2px}.rickshaw_legend .label{margin:0;white-space:nowrap;display:inline;font-size:inherit;background-color:transparent;color:inherit;font-weight:400;line-height:normal;padding:0;text-shadow:none}.rickshaw_legend .action:hover{opacity:.6}.rickshaw_legend .action{margin-right:.2em;font-size:10px;opacity:.2;cursor:pointer;font-size:14px}.rickshaw_legend .line.disabled{opacity:.4}.rickshaw_legend ul{list-style-type:none;margin:0;padding:0;margin:2px;cursor:pointer}.rickshaw_legend li{padding:0 0 0 2px;min-width:80px;white-space:nowrap}.rickshaw_legend li:hover{background:rgba(255,255,255,.08);border-radius:3px}.rickshaw_legend li:active{background:rgba(255,255,255,.2);border-radius:3px} -------------------------------------------------------------------------------- /session_2/notebooks/runipy.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext load_style\n", 12 | "%load_style talk.css" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "# [runipy](https://github.com/paulgb/runipy)" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "
\n", 27 | "ATTENTION: If you have an old installation of **Runipy**, it is very unlikely to work with the new (format v4) IPython notebook (latest version 3.0 of IPython). \n", 28 | "\n", 29 | "I recommend that you install the latest development version of **runipy** from the [github repository](https://github.com/paulgb/runipy) with the following: \n", 30 | "\n", 31 | "```\n", 32 | "pip install https://github.com/paulgb/runipy/archive/master.zip\n", 33 | "```\n", 34 | "\n", 35 | "Another solution (for IPython version 3) is to use **nbconvert**, which now comes with an option to run a notebook in place and save the outputs in the same notebook, the (rather verbose) syntax is: \n", 36 | "\n", 37 | "\n", 38 | "```\n", 39 | "ᐅ ipython nbconvert notebook_to_execute.ipynb --ExecutePreprocessor.enabled=True --to notebook --output notebook_to_execute.ipynb\n", 40 | "```\n", 41 | "\n", 42 | "However it does not seem to handle passing environment variables as **runipy** does ... \n", 43 | "\n", 44 | "see [http://ipython.org/ipython-doc/dev/whatsnew/version3.html](http://ipython.org/ipython-doc/dev/whatsnew/version3.html) for the list of changes in the version 3.0 of IPython and the IPython notebook, see in particular the [backwards incompatible changes](http://ipython.org/ipython-doc/dev/whatsnew/version3.html#backwards-incompatible-changes) section.\n", 45 | "\n", 46 | "
" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "The following comes from the doc of **Runipy** and goes through its major use cases" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "The [IPython notebook](http://ipython.org/notebook.html) provides an interactive interface to a Python\n", 61 | "interpreter.\n", 62 | "\n", 63 | "* **Literate programming**: the IPython notebook is an ideal format for\n", 64 | " writing \"literate\" programs, in which the code is part of a larger\n", 65 | " multi-media document. ``runipy`` lets you run such programs directly,\n", 66 | " without first converting to a pure Python script.\n", 67 | " \n", 68 | "* **Report generation**: ``runipy`` can run the notebook and convert it\n", 69 | " into HTML in one go, making it an easy way to automate reports.\n", 70 | " \n", 71 | "* **Data pipeline**: if you use IPython notebooks to create a data\n", 72 | " pipeline, ``runipy`` lets you automate that pipeline without losing\n", 73 | " the notebook formatting.\n", 74 | "\n", 75 | "### Requirements\n", 76 | "\n", 77 | "``runipy`` currently supports IPython versions 2.3.x and the current\n", 78 | "development version of 3.x.\n", 79 | "\n", 80 | "### Installation\n", 81 | "\n", 82 | "\n", 83 | "The easiest way to install ``runipy`` is with ``pip``:\n", 84 | "\n", 85 | " $ pip install runipy\n", 86 | "\n", 87 | "### Command-line use\n", 88 | "\n", 89 | "\n", 90 | "To run a ``.ipynb`` file as a script, run:\n", 91 | "\n", 92 | " $ runipy MyNotebook.ipynb\n", 93 | "\n", 94 | "To save the output of each cell back to the notebook file, run:\n", 95 | "\n", 96 | " $ runipy -o MyNotebook.ipynb\n", 97 | " \n", 98 | "**NOTE**: *The notebook seems to be automatically converted back to version 3 if it was in version 4.*\n", 99 | "\n", 100 | "To save the notebook output as a *new* notebook, run:\n", 101 | "\n", 102 | " $ runipy MyNotebook.ipynb OutputNotebook.ipynb\n", 103 | "\n", 104 | "To run a ``.ipynb`` file and generate an ``HTML`` report, run:\n", 105 | "\n", 106 | " $ runipy MyNotebook.ipynb --html report.html\n", 107 | "\n", 108 | "### Passing Arguments\n", 109 | "\n", 110 | "\n", 111 | "You can pass arguments to the notebook through environment variables.\n", 112 | "The use of environment variables is OS- and shell- dependent, but in a\n", 113 | "typical UNIX-like environment they can be passed on the command line\n", 114 | "before the program name:\n", 115 | "\n", 116 | " $ myvar=value runipy MyNotebook.ipynb\n", 117 | "\n", 118 | "Then in the notebook, to access myvar:\n", 119 | "\n", 120 | " from os import environ\n", 121 | " myvar = environ['myvar']\n", 122 | "\n", 123 | "``environ`` is just a ``dict``, so you can use ``.get()`` to fall back\n", 124 | "on a default value:\n", 125 | "\n", 126 | " from os import environ\n", 127 | " myvar = environ.get('myvar', 'default!')\n", 128 | "\n", 129 | "### Stdin / Stdout\n", 130 | "\n", 131 | "``runipy`` can read stdin and stdout and sit in a UNIX pipeline:\n", 132 | "\n", 133 | "::\n", 134 | "\n", 135 | " $ runipy --stdout < MyNotebook.ipynb > OutputNotebook.ipynb\n", 136 | "\n", 137 | " $ cat MyNotebook.ipynb | runipy --stdout > OutputNotebook.ipynb\n", 138 | "\n", 139 | "### Programmatic use\n", 140 | "\n", 141 | "It is also possible to run IPython notebooks from Python, using:\n", 142 | "\n", 143 | " from runipy.notebook_runner import NotebookRunner\n", 144 | " from IPython.nbformat.current import read\n", 145 | "\n", 146 | " notebook = read(open(\"MyNotebook.ipynb\"), 'json')\n", 147 | " r = NotebookRunner(notebook)\n", 148 | " r.run_notebook()\n", 149 | "\n", 150 | "and you can enable ``pylab`` with:\n", 151 | "\n", 152 | " r = NotebookRunner(notebook, pylab=True)\n", 153 | "\n", 154 | "The notebook is stored in the object and can be saved using:\n", 155 | "\n", 156 | " from IPython.nbformat.current import write\n", 157 | " write(r.nb, open(\"MyOtherNotebook.ipynb\", 'w'), 'json')\n", 158 | "\n", 159 | "run\\_notebook() takes two optional arguments. The first,\n", 160 | "skip\\_exceptions, takes a boolean value (False by default). If True,\n", 161 | "exceptions will be ignored and the notebook will continue to execute\n", 162 | "cells after encountering an exception. The second argument is\n", 163 | "progress\\_callback, which must be either None or a function that takes\n", 164 | "one argument. This function is called after execution of each cell with\n", 165 | "the 0-based index of the cell just evaluated. This can be useful for\n", 166 | "tracking progress of long-running notebooks\n" 167 | ] 168 | } 169 | ], 170 | "metadata": { 171 | "kernelspec": { 172 | "display_name": "Python 2", 173 | "language": "python", 174 | "name": "python2" 175 | }, 176 | "language_info": { 177 | "codemirror_mode": { 178 | "name": "ipython", 179 | "version": 2 180 | }, 181 | "file_extension": ".py", 182 | "mimetype": "text/x-python", 183 | "name": "python", 184 | "nbconvert_exporter": "python", 185 | "pygments_lexer": "ipython2", 186 | "version": "2.7.9" 187 | } 188 | }, 189 | "nbformat": 4, 190 | "nbformat_minor": 0 191 | } 192 | -------------------------------------------------------------------------------- /session_2/notebooks/sklearn_EOF_decomposition.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext load_style\n", 12 | "%load_style talk.css" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "# PCA of SST anomalies in the Pacific with scikit-learn" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": { 26 | "collapsed": false 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "import pandas as pd\n", 31 | "import numpy as np\n", 32 | "from numpy import ma\n", 33 | "from matplotlib import pyplot as plt\n", 34 | "from mpl_toolkits.basemap import Basemap" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": { 41 | "collapsed": false 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "%matplotlib inline" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "## load the SST data" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "The file (74 Mb) can be downloaded at `ftp://ftp.niwa.co.nz/incoming/fauchereaun/ersst.realtime.nc`" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "collapsed": false 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "import xray" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": { 77 | "collapsed": false 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "dset = xray.open_dataset('../data/ersst.realtime.nc')" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "collapsed": false 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "dset" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "### selects the period 1980 - 2014 and the tropical Pacific domain" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": { 106 | "collapsed": false 107 | }, 108 | "outputs": [], 109 | "source": [ 110 | "dsub = dset.sel(time=slice('1980','2014'), zlev=0, lat=slice(-40,40), lon=slice(120,290))" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": { 117 | "collapsed": false 118 | }, 119 | "outputs": [], 120 | "source": [ 121 | "lat = dsub['lat'].values\n", 122 | "lon = dsub['lon'].values\n", 123 | "sst = dsub['anom'].values" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": { 130 | "collapsed": false 131 | }, 132 | "outputs": [], 133 | "source": [ 134 | "sst.shape" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "### reshape in 2D (time, space)" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": { 148 | "collapsed": false 149 | }, 150 | "outputs": [], 151 | "source": [ 152 | "X = np.reshape(sst, (sst.shape[0], len(lat) * len(lon)), order='F')" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "metadata": { 159 | "collapsed": false 160 | }, 161 | "outputs": [], 162 | "source": [ 163 | "np.any(np.isnan(X))" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "### Mask the land points" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": null, 176 | "metadata": { 177 | "collapsed": false 178 | }, 179 | "outputs": [], 180 | "source": [ 181 | "type(X)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": { 188 | "collapsed": false 189 | }, 190 | "outputs": [], 191 | "source": [ 192 | "X = ma.masked_array(X, np.isnan(X))" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": { 199 | "collapsed": false 200 | }, 201 | "outputs": [], 202 | "source": [ 203 | "type(X)" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "metadata": { 210 | "collapsed": false 211 | }, 212 | "outputs": [], 213 | "source": [ 214 | "land = X.sum(0).mask" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": { 221 | "collapsed": false 222 | }, 223 | "outputs": [], 224 | "source": [ 225 | "ocean = -land" 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": {}, 231 | "source": [ 232 | "### keep only oceanic grid-points" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "metadata": { 239 | "collapsed": false 240 | }, 241 | "outputs": [], 242 | "source": [ 243 | "X = X[:,ocean]" 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": {}, 249 | "source": [ 250 | "### Standardize SST using the fit and transform methods of the `sklearn.preprocessing.scaler.StandardScaler`" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": null, 256 | "metadata": { 257 | "collapsed": false 258 | }, 259 | "outputs": [], 260 | "source": [ 261 | "from sklearn import preprocessing\n", 262 | "scaler = preprocessing.StandardScaler()" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": null, 268 | "metadata": { 269 | "collapsed": false 270 | }, 271 | "outputs": [], 272 | "source": [ 273 | "scaler_sst = scaler.fit(X)" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "metadata": {}, 279 | "source": [ 280 | "### Once the scaler object has been 'trained' on the data, we can save it as a pickle object" 281 | ] 282 | }, 283 | { 284 | "cell_type": "code", 285 | "execution_count": null, 286 | "metadata": { 287 | "collapsed": false 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "from sklearn.externals import joblib" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "metadata": { 298 | "collapsed": false 299 | }, 300 | "outputs": [], 301 | "source": [ 302 | "joblib.dump(scaler_sst, '../data/scaler_sst.pkl', compress=9)" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": null, 308 | "metadata": { 309 | "collapsed": false 310 | }, 311 | "outputs": [], 312 | "source": [ 313 | "scaler_sst = joblib.load('../data/scaler_sst.pkl')" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "### scales: use the `transform` method of the scaler object" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": null, 326 | "metadata": { 327 | "collapsed": false 328 | }, 329 | "outputs": [], 330 | "source": [ 331 | "X = scaler_sst.transform(X)" 332 | ] 333 | }, 334 | { 335 | "cell_type": "markdown", 336 | "metadata": {}, 337 | "source": [ 338 | "### verify that mean = 0 and std = 1" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": { 345 | "collapsed": false 346 | }, 347 | "outputs": [], 348 | "source": [ 349 | "X.mean()" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": { 356 | "collapsed": false 357 | }, 358 | "outputs": [], 359 | "source": [ 360 | "X.std()" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": null, 366 | "metadata": { 367 | "collapsed": false 368 | }, 369 | "outputs": [], 370 | "source": [ 371 | "X.shape" 372 | ] 373 | }, 374 | { 375 | "cell_type": "markdown", 376 | "metadata": {}, 377 | "source": [ 378 | "### EOF decomposition " 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": null, 384 | "metadata": { 385 | "collapsed": false 386 | }, 387 | "outputs": [], 388 | "source": [ 389 | "from sklearn.decomposition import pca" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "#### instantiates the PCA object" 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": null, 402 | "metadata": { 403 | "collapsed": false 404 | }, 405 | "outputs": [], 406 | "source": [ 407 | "skpca = pca.PCA()" 408 | ] 409 | }, 410 | { 411 | "cell_type": "markdown", 412 | "metadata": {}, 413 | "source": [ 414 | "#### fit" 415 | ] 416 | }, 417 | { 418 | "cell_type": "code", 419 | "execution_count": null, 420 | "metadata": { 421 | "collapsed": false 422 | }, 423 | "outputs": [], 424 | "source": [ 425 | "skpca.fit(X)" 426 | ] 427 | }, 428 | { 429 | "cell_type": "markdown", 430 | "metadata": {}, 431 | "source": [ 432 | "### Now saves the (fitted) PCA object for reuse in operations" 433 | ] 434 | }, 435 | { 436 | "cell_type": "code", 437 | "execution_count": null, 438 | "metadata": { 439 | "collapsed": false 440 | }, 441 | "outputs": [], 442 | "source": [ 443 | "joblib.dump(skpca, '../data/EOF.pkl', compress=9)" 444 | ] 445 | }, 446 | { 447 | "cell_type": "code", 448 | "execution_count": null, 449 | "metadata": { 450 | "collapsed": false 451 | }, 452 | "outputs": [], 453 | "source": [ 454 | "from matplotlib import style\n", 455 | "style.use('fivethirtyeight')" 456 | ] 457 | }, 458 | { 459 | "cell_type": "code", 460 | "execution_count": null, 461 | "metadata": { 462 | "collapsed": false 463 | }, 464 | "outputs": [], 465 | "source": [ 466 | "style.available" 467 | ] 468 | }, 469 | { 470 | "cell_type": "code", 471 | "execution_count": null, 472 | "metadata": { 473 | "collapsed": false 474 | }, 475 | "outputs": [], 476 | "source": [ 477 | "f, ax = plt.subplots(figsize=(6,6))\n", 478 | "ax.plot(skpca.explained_variance_ratio_[0:10]*100)\n", 479 | "ax.plot(skpca.explained_variance_ratio_[0:10]*100,'ro')" 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": {}, 485 | "source": [ 486 | "### keep number of PC sufficient to explain 70 % of the original variance " 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "metadata": { 493 | "collapsed": false 494 | }, 495 | "outputs": [], 496 | "source": [ 497 | "ipc = np.where(skpca.explained_variance_ratio_.cumsum() >= 0.70)[0][0]" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": null, 503 | "metadata": { 504 | "collapsed": false 505 | }, 506 | "outputs": [], 507 | "source": [ 508 | "ipc" 509 | ] 510 | }, 511 | { 512 | "cell_type": "markdown", 513 | "metadata": {}, 514 | "source": [ 515 | "### The Principal Components (PCs) are obtained by using the `transform` method of the `pca` object (`skpca`)" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": null, 521 | "metadata": { 522 | "collapsed": false 523 | }, 524 | "outputs": [], 525 | "source": [ 526 | "PCs = skpca.transform(X)" 527 | ] 528 | }, 529 | { 530 | "cell_type": "code", 531 | "execution_count": null, 532 | "metadata": { 533 | "collapsed": false 534 | }, 535 | "outputs": [], 536 | "source": [ 537 | "PCs = PCs[:,:ipc]" 538 | ] 539 | }, 540 | { 541 | "cell_type": "markdown", 542 | "metadata": {}, 543 | "source": [ 544 | "### The Empirical Orthogonal Functions (EOFs) are contained in the `components_` attribute of the `pca` object (`skpca`)" 545 | ] 546 | }, 547 | { 548 | "cell_type": "code", 549 | "execution_count": null, 550 | "metadata": { 551 | "collapsed": false 552 | }, 553 | "outputs": [], 554 | "source": [ 555 | "EOFs = skpca.components_" 556 | ] 557 | }, 558 | { 559 | "cell_type": "code", 560 | "execution_count": null, 561 | "metadata": { 562 | "collapsed": false 563 | }, 564 | "outputs": [], 565 | "source": [ 566 | "EOFs = EOFs[:ipc,:]" 567 | ] 568 | }, 569 | { 570 | "cell_type": "code", 571 | "execution_count": null, 572 | "metadata": { 573 | "collapsed": false 574 | }, 575 | "outputs": [], 576 | "source": [ 577 | "EOFs.shape" 578 | ] 579 | }, 580 | { 581 | "cell_type": "markdown", 582 | "metadata": {}, 583 | "source": [ 584 | "### we can the reconstruct the 2D fields (maps)" 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": null, 590 | "metadata": { 591 | "collapsed": false 592 | }, 593 | "outputs": [], 594 | "source": [ 595 | "EOF_recons = np.ones((ipc, len(lat) * len(lon))) * -999." 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": null, 601 | "metadata": { 602 | "collapsed": false 603 | }, 604 | "outputs": [], 605 | "source": [ 606 | "for i in xrange(ipc): \n", 607 | " EOF_recons[i,ocean] = EOFs[i,:]" 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": null, 613 | "metadata": { 614 | "collapsed": false 615 | }, 616 | "outputs": [], 617 | "source": [ 618 | "EOF_recons = ma.masked_values(np.reshape(EOF_recons, (ipc, len(lat), len(lon)), order='F'), -999.)" 619 | ] 620 | }, 621 | { 622 | "cell_type": "code", 623 | "execution_count": null, 624 | "metadata": { 625 | "collapsed": false 626 | }, 627 | "outputs": [], 628 | "source": [ 629 | "EOF_recons.shape" 630 | ] 631 | }, 632 | { 633 | "cell_type": "code", 634 | "execution_count": null, 635 | "metadata": { 636 | "collapsed": false 637 | }, 638 | "outputs": [], 639 | "source": [ 640 | "plt.imshow(EOF_recons[0,:,:], origin='lower', interpolation='nearest', aspect='auto')\n", 641 | "plt.colorbar();" 642 | ] 643 | }, 644 | { 645 | "cell_type": "markdown", 646 | "metadata": {}, 647 | "source": [ 648 | "### scale the Principal Components" 649 | ] 650 | }, 651 | { 652 | "cell_type": "code", 653 | "execution_count": null, 654 | "metadata": { 655 | "collapsed": false 656 | }, 657 | "outputs": [], 658 | "source": [ 659 | "from sklearn.preprocessing import StandardScaler" 660 | ] 661 | }, 662 | { 663 | "cell_type": "code", 664 | "execution_count": null, 665 | "metadata": { 666 | "collapsed": false 667 | }, 668 | "outputs": [], 669 | "source": [ 670 | "scaler_PCs = StandardScaler()" 671 | ] 672 | }, 673 | { 674 | "cell_type": "code", 675 | "execution_count": null, 676 | "metadata": { 677 | "collapsed": false 678 | }, 679 | "outputs": [], 680 | "source": [ 681 | "scaler_PCs.fit(PCs)" 682 | ] 683 | }, 684 | { 685 | "cell_type": "code", 686 | "execution_count": null, 687 | "metadata": { 688 | "collapsed": false 689 | }, 690 | "outputs": [], 691 | "source": [ 692 | "PCs_std = scaler_PCs.transform(PCs)" 693 | ] 694 | }, 695 | { 696 | "cell_type": "code", 697 | "execution_count": null, 698 | "metadata": { 699 | "collapsed": false 700 | }, 701 | "outputs": [], 702 | "source": [ 703 | "joblib.dump(scaler_PCs, '../data/scaler_PCs.pkl')" 704 | ] 705 | }, 706 | { 707 | "cell_type": "code", 708 | "execution_count": null, 709 | "metadata": { 710 | "collapsed": false 711 | }, 712 | "outputs": [], 713 | "source": [ 714 | "PCdf = pd.DataFrame(PCs_std, index = dsub['time'], \\\n", 715 | " columns = [\"EOF%s\" % (x) for x in xrange(1, PCs_std.shape[1] +1)])" 716 | ] 717 | }, 718 | { 719 | "cell_type": "code", 720 | "execution_count": null, 721 | "metadata": { 722 | "collapsed": false 723 | }, 724 | "outputs": [], 725 | "source": [ 726 | "PCdf.head()" 727 | ] 728 | }, 729 | { 730 | "cell_type": "code", 731 | "execution_count": null, 732 | "metadata": { 733 | "collapsed": false 734 | }, 735 | "outputs": [], 736 | "source": [ 737 | "PCdf.to_csv('../data/EOF_ERSST_PCs.csv')" 738 | ] 739 | }, 740 | { 741 | "cell_type": "code", 742 | "execution_count": null, 743 | "metadata": { 744 | "collapsed": false 745 | }, 746 | "outputs": [], 747 | "source": [ 748 | "from scipy.signal import detrend" 749 | ] 750 | }, 751 | { 752 | "cell_type": "code", 753 | "execution_count": null, 754 | "metadata": { 755 | "collapsed": false 756 | }, 757 | "outputs": [], 758 | "source": [ 759 | "f, ax = plt.subplots(figsize=(12,6))\n", 760 | "PCdf.ix[:,0].plot(ax=ax, color='k', label='PC1')\n", 761 | "#ax.set_xlabel('period', fontsize=18)\n", 762 | "ax.plot(PCdf.index, detrend(PCdf.ix[:,0].values), 'r', label='PC1 (trend removed)')\n", 763 | "ax.grid('off')\n", 764 | "ax.legend(loc=1); " 765 | ] 766 | }, 767 | { 768 | "cell_type": "code", 769 | "execution_count": null, 770 | "metadata": { 771 | "collapsed": false 772 | }, 773 | "outputs": [], 774 | "source": [ 775 | "!ipython nbconvert sklearn_EOF_decomposition.ipynb --to html" 776 | ] 777 | } 778 | ], 779 | "metadata": { 780 | "kernelspec": { 781 | "display_name": "Python 2", 782 | "language": "python", 783 | "name": "python2" 784 | }, 785 | "language_info": { 786 | "codemirror_mode": { 787 | "name": "ipython", 788 | "version": 2 789 | }, 790 | "file_extension": ".py", 791 | "mimetype": "text/x-python", 792 | "name": "python", 793 | "nbconvert_exporter": "python", 794 | "pygments_lexer": "ipython2", 795 | "version": "2.7.9" 796 | } 797 | }, 798 | "nbformat": 4, 799 | "nbformat_minor": 0 800 | } 801 | -------------------------------------------------------------------------------- /session_2/notebooks/sklearn_kmeans.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext load_style\n", 12 | "%load_style talk.css" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "# k-means clustering of SST anomalies in the Pacific with scikit-learn" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": { 26 | "collapsed": false 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "%matplotlib inline\n", 31 | "from matplotlib import pyplot as plt" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": { 38 | "collapsed": false 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "import os\n", 43 | "import numpy as np\n", 44 | "import pandas as pd\n", 45 | "from mpl_toolkits.basemap import Basemap as bm" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": { 52 | "collapsed": false 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "import xray" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "### defines a function to plot a field (must be 2D)" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": { 70 | "collapsed": false 71 | }, 72 | "outputs": [], 73 | "source": [ 74 | "def plot_field(X, lat, lon, vmin, vmax, step, cmap=plt.get_cmap('jet'), ax=False, title=False, grid=False):\n", 75 | " if not ax: \n", 76 | " f, ax = plt.subplots(figsize=(10, (X.shape[0] / float(X.shape[1])) * 10))\n", 77 | " m.ax = ax\n", 78 | " im = m.contourf(lons, lats, X, np.arange(vmin, vmax+step, step), latlon=True, cmap=cmap, extend='both', ax=ax)\n", 79 | " m.drawcoastlines()\n", 80 | " if grid: \n", 81 | " m.drawmeridians(np.arange(0, 360, 60), labels=[0,0,0,1])\n", 82 | " m.drawparallels([-40, 0, 40], labels=[1,0,0,0])\n", 83 | " m.colorbar(im)\n", 84 | " if title: \n", 85 | " ax.set_title(title)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "### reads the PCs (obtained before, see [sklearn_EOF_decomposition.ipynb](./sklearn_EOF_decomposition.ipynb))" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": { 99 | "collapsed": false 100 | }, 101 | "outputs": [], 102 | "source": [ 103 | "PCs = pd.read_csv('../data/EOF_ERSST_PCs.csv', index_col=0, parse_dates=True)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "collapsed": false 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "PCs.head()" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "### import the KMeans class from scikit-learn" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": { 128 | "collapsed": false 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "from sklearn.cluster import KMeans" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "#### How many clusters do we want ? " 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "metadata": { 146 | "collapsed": false 147 | }, 148 | "outputs": [], 149 | "source": [ 150 | "nclusters = 6" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "#### instantiates the k-means class" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": { 164 | "collapsed": false 165 | }, 166 | "outputs": [], 167 | "source": [ 168 | "KMeans?" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": { 175 | "collapsed": false 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "kmeans = KMeans(init='k-means++', n_clusters=nclusters, n_init=10)" 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "#### fit to the data" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "metadata": { 193 | "collapsed": false 194 | }, 195 | "outputs": [], 196 | "source": [ 197 | "X = PCs.values" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": { 204 | "collapsed": false 205 | }, 206 | "outputs": [], 207 | "source": [ 208 | "type(X)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "collapsed": false 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "kmeans.fit(X) " 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "#### the classes (clusters) are contained in the .labels_ attributes" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "metadata": { 233 | "collapsed": false 234 | }, 235 | "outputs": [], 236 | "source": [ 237 | "kmeans.labels_" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": {}, 243 | "source": [ 244 | "### Calculate composite anomalies for each of the clusters" 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": null, 250 | "metadata": { 251 | "collapsed": false 252 | }, 253 | "outputs": [], 254 | "source": [ 255 | "ncfname = '../data/ersst.realtime.nc'" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "collapsed": false 263 | }, 264 | "outputs": [], 265 | "source": [ 266 | "dset = xray.open_dataset(ncfname)" 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "execution_count": null, 272 | "metadata": { 273 | "collapsed": false 274 | }, 275 | "outputs": [], 276 | "source": [ 277 | "dsub = dset.sel(time=slice('1980','2014'),lat=slice(-40,40), lon=slice(120,290))" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": { 284 | "collapsed": false 285 | }, 286 | "outputs": [], 287 | "source": [ 288 | "dsub" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": null, 294 | "metadata": { 295 | "collapsed": false 296 | }, 297 | "outputs": [], 298 | "source": [ 299 | "lat = dsub['lat'].values\n", 300 | "lon = dsub['lon'].values\n", 301 | "lons, lats = np.meshgrid(lon, lat)" 302 | ] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "execution_count": null, 307 | "metadata": { 308 | "collapsed": false 309 | }, 310 | "outputs": [], 311 | "source": [ 312 | "labels = pd.DataFrame(kmeans.labels_, index=dsub['time'], columns=['cluster'])" 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": null, 318 | "metadata": { 319 | "collapsed": false 320 | }, 321 | "outputs": [], 322 | "source": [ 323 | "labels.head()" 324 | ] 325 | }, 326 | { 327 | "cell_type": "code", 328 | "execution_count": null, 329 | "metadata": { 330 | "collapsed": false 331 | }, 332 | "outputs": [], 333 | "source": [ 334 | "pd.unique(labels.cluster)" 335 | ] 336 | }, 337 | { 338 | "cell_type": "code", 339 | "execution_count": null, 340 | "metadata": { 341 | "collapsed": false 342 | }, 343 | "outputs": [], 344 | "source": [ 345 | "c = 0" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": null, 351 | "metadata": { 352 | "collapsed": false 353 | }, 354 | "outputs": [], 355 | "source": [ 356 | "clussub = labels.query('cluster == {}'.format(c))" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": { 363 | "collapsed": false 364 | }, 365 | "outputs": [], 366 | "source": [ 367 | "clussub.index" 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": null, 373 | "metadata": { 374 | "collapsed": false 375 | }, 376 | "outputs": [], 377 | "source": [ 378 | "cluster = dsub.sel(time=clussub.index).mean('time')" 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": null, 384 | "metadata": { 385 | "collapsed": false 386 | }, 387 | "outputs": [], 388 | "source": [ 389 | "plt.imshow(cluster['anom'][0,::-1,:])" 390 | ] 391 | }, 392 | { 393 | "cell_type": "code", 394 | "execution_count": null, 395 | "metadata": { 396 | "collapsed": false 397 | }, 398 | "outputs": [], 399 | "source": [ 400 | "m = bm(projection='cyl',llcrnrlat=-40,urcrnrlat=40,\\\n", 401 | " llcrnrlon=120,urcrnrlon=290,\\\n", 402 | " lat_ts=0,resolution='c')" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": null, 408 | "metadata": { 409 | "collapsed": false 410 | }, 411 | "outputs": [], 412 | "source": [ 413 | "f, axes = plt.subplots(nrows=3,ncols=2, figsize=(14,10))\n", 414 | "f.subplots_adjust(hspace=0.1, wspace=0.1)\n", 415 | "axes = axes.flatten()\n", 416 | "for c in xrange(nclusters):\n", 417 | " index = labels.query('cluster == {}'.format(c))\n", 418 | " cluster = dsub.sel(time=index.index).mean('time')\n", 419 | " ax = axes[c]\n", 420 | " plot_field(cluster['anom'][0,:,:], lats, lons, -2, 2, 0.1, \\\n", 421 | " ax=ax, cmap=plt.get_cmap('RdBu_r'), \\\n", 422 | " title=\"Cluster #{}: {} months\".format(c+1, len(index)))" 423 | ] 424 | }, 425 | { 426 | "cell_type": "code", 427 | "execution_count": null, 428 | "metadata": { 429 | "collapsed": false 430 | }, 431 | "outputs": [], 432 | "source": [ 433 | "!ipython nbconvert sklearn_kmeans.ipynb --to html" 434 | ] 435 | } 436 | ], 437 | "metadata": { 438 | "kernelspec": { 439 | "display_name": "Python 2", 440 | "language": "python", 441 | "name": "python2" 442 | }, 443 | "language_info": { 444 | "codemirror_mode": { 445 | "name": "ipython", 446 | "version": 2 447 | }, 448 | "file_extension": ".py", 449 | "mimetype": "text/x-python", 450 | "name": "python", 451 | "nbconvert_exporter": "python", 452 | "pygments_lexer": "ipython2", 453 | "version": "2.7.9" 454 | } 455 | }, 456 | "nbformat": 4, 457 | "nbformat_minor": 0 458 | } 459 | -------------------------------------------------------------------------------- /session_2/notebooks/talk.css: -------------------------------------------------------------------------------- 1 | 2 | .rendered_html { 3 | font-family: "proxima-nova", helvetica; 4 | font-size: 130%; 5 | line-height: 1.5; 6 | } 7 | 8 | .rendered_html h1 { 9 | margin: 0.25em 0em 0.5em; 10 | color: #015C9C; 11 | text-align: center; 12 | line-height: 1.2; 13 | page-break-before: always; 14 | } 15 | 16 | .rendered_html h2 { 17 | margin: 1.1em 0em 0.5em; 18 | color: #26465D; 19 | line-height: 1.2; 20 | } 21 | 22 | .rendered_html h3 { 23 | margin: 1.1em 0em 0.5em; 24 | color: #002845; 25 | line-height: 1.2; 26 | } 27 | 28 | .rendered_html li { 29 | line-height: 1.5; 30 | } 31 | 32 | /*.prompt { 33 | font-size: 120%; 34 | }*/ 35 | 36 | .CodeMirror-lines { 37 | font-size: 110%; 38 | } 39 | 40 | /*.output_area { 41 | font-size: 120%; 42 | }*/ 43 | 44 | /*#notebook { 45 | background-image: url('files/images/witewall_3.png'); 46 | }*/ 47 | 48 | h1.bigtitle { 49 | margin: 4cm 1cm 4cm 1cm; 50 | font-size: 300%; 51 | } 52 | 53 | h3.point { 54 | font-size: 200%; 55 | text-align: center; 56 | margin: 2em 0em 2em 0em; 57 | #26465D 58 | } 59 | 60 | .logo { 61 | margin: 20px 0 20px 0; 62 | } 63 | 64 | a.anchor-link { 65 | display: none; 66 | } 67 | 68 | h1.title { 69 | font-size: 250%; 70 | } 71 | -------------------------------------------------------------------------------- /session_2/notebooks/talktools.py: -------------------------------------------------------------------------------- 1 | 2 | from IPython.display import HTML, display, YouTubeVideo 3 | 4 | def prefix(url): 5 | prefix = '' if url.startswith('http') else 'http://' 6 | return prefix + url 7 | 8 | 9 | def simple_link(url, name=None): 10 | name = url if name is None else name 11 | url = prefix(url) 12 | return '%s' % (url, name) 13 | 14 | 15 | def html_link(url, name=None): 16 | return HTML(simple_link(url, name)) 17 | 18 | 19 | # Utility functions 20 | def website(url, name=None, width=800, height=450): 21 | html = [] 22 | name = url if name == 'auto' else name 23 | if name: 24 | html.extend(['
', 25 | simple_link(url, name), 26 | '
'] ) 27 | 28 | html.append('