├── .gitignore ├── AdvancedPython ├── Packaging │ ├── guido.jpg │ ├── install_tools.png │ ├── packages1.rst │ ├── packages2.rst │ └── packaging_plus.rst ├── PythonForDataScience │ ├── Matplotlib.ipynb │ ├── NumPy.ipynb │ ├── Sklearn introduction I.ipynb │ ├── Sklearn introduction II.ipynb │ ├── ipython.png │ ├── ipython.rst │ ├── ipython_screener.png │ ├── scipy.rst │ └── sklearn.rst └── StandardLibrary │ ├── Concurrency exercises.ipynb │ ├── Python2vs3.rst │ ├── SpeedupPython.ipynb │ ├── Standard Library.html │ ├── concurrency.rst │ ├── decorators.rst │ ├── iterators.rst │ ├── metaclass.rst │ ├── pep.rst │ ├── persistance.rst │ └── testing.rst ├── LICENSE.txt ├── README.md └── ScriptingLanguagesIntroduction ├── Exercises ├── basic_exercises.rst └── zoo.png ├── LuaIntro ├── 10-lua_basics.rst ├── 11-lua_basics.rst ├── 12-lua_patterns.rst ├── 13-lua_functions.rst └── 14-lua_oop.rst └── PythonIntro ├── 1-introduction.rst ├── 2-python_basics.rst ├── 3-python_data_structures.rst ├── 4-python-functions-1.rst ├── 4-python-functions-2.rst ├── 5-python-modules.rst ├── 6-python_re.rst ├── 7-python_oop.rst ├── 8-python_2vs3.rst ├── std_streams.png ├── tiobe.png ├── types.gif ├── unicode.png └── unicode2.png /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | -------------------------------------------------------------------------------- /AdvancedPython/Packaging/guido.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/AdvancedPython/Packaging/guido.jpg -------------------------------------------------------------------------------- /AdvancedPython/Packaging/install_tools.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/AdvancedPython/Packaging/install_tools.png -------------------------------------------------------------------------------- /AdvancedPython/Packaging/packages1.rst: -------------------------------------------------------------------------------- 1 | Introduciton to package management 2 | ================================== 3 | 4 | -------------------------------------------------------------------------- 5 | 6 | Python Package Index 7 | -------------------- 8 | 9 | also known as ``pypi`` or "The cheeseshop" is a repository of open-source third-party packages. 10 | 11 | http://pypi.python.org/pypi 12 | 13 | -------------------------------------------------------------------------- 14 | 15 | Packages 16 | -------- 17 | 18 | A package can be either a **Wheel** or an **Egg**, but Wheel is currently considered the standard for built and binary packaging for Python. 19 | 20 | http://python-packaging-user-guide.readthedocs.org/en/latest/wheel_egg/ 21 | 22 | -------------------------------------------------------------------------- 23 | 24 | ``easy_install`` 25 | ---------------- 26 | 27 | ... is a deprecated tool for installing packages from the Python 28 | 29 | * automatically downloads, builds, installs packages 30 | * looks in the `Python Packaging Index `_ 31 | * installs a Python Egg 32 | 33 | -------------------------------------------------------------------------- 34 | 35 | ``pip`` 36 | -------- 37 | 38 | ... is an enhanced tool for installing packages from the Python. 39 | 40 | .. code:: bash 41 | 42 | $ pip install SomePackage # latest version 43 | $ pip install SomePackage==1.0.4 # specific version 44 | $ pip install "SomePackage>=1.0.4" # minimum version 45 | $ pip install --upgrade SomePackage # upgrade 46 | 47 | -------------------------------------------------------------------------- 48 | 49 | * ``pip`` is part of the standard library from v3.4 50 | * All packages are downloaded before installation. Partially-completed installation doesn’t occur as a result. 51 | * The code is relatively concise and cohesive, making it easier to use programmatically. 52 | * Native support for other version control systems (Git, Mercurial and Bazaar) 53 | * Uninstallation of packages. 54 | * Simple to define fixed sets of requirements and reliably reproduce a set of packages. 55 | 56 | 57 | -------------------------------------------------------------------------- 58 | 59 | * ``install`` -- install package 60 | * ``--upgrade`` 61 | * ``--user`` -- uses ``$PYTHONUSERBASE`` 62 | * ``uninstall`` -- remove package 63 | * ``list`` -- list installed packages 64 | * ``search`` -- search for packages 65 | * ``show`` -- package details 66 | * ``freeze`` -- output installed packages in a requirement file 67 | * ``wheel`` -- build a wheel file from a requirements file 68 | 69 | -------------------------------------------------------------------------- 70 | 71 | Dependency management of ``pip``: 72 | 73 | ``requirements.txt`` 74 | 75 | .. code-block:: none 76 | 77 | MyApp 78 | Framework==0.9.4 79 | Library>=0.2 80 | 81 | ``$ pip install -r requirements.txt`` 82 | 83 | ``$ pip freeze > requirements.txt`` 84 | 85 | -------------------------------------------------------------------------- 86 | 87 | Virtual environments 88 | -------------------- 89 | 90 | Isolated working environments containing a copy of a Python interpreter. 91 | 92 | * no need to install packages globally 93 | * one can manage different versions of the same package 94 | 95 | Available tools: 96 | 97 | * ``venv`` -- boundled in Python 3.4 98 | * ``virtualenv`` -- http://virtualenv.readthedocs.org/en/latest/ 99 | 100 | ----------------------------------------------------- 101 | 102 | Usage: 103 | 104 | .. code-block:: bash 105 | 106 | $ virtualenv mytest 107 | $ source mytest/bin/activate 108 | (mytest)$ deactivate 109 | 110 | $ pyvenv mytest 111 | $ source mytest/bin/activate 112 | (mytest)$ deactivate 113 | 114 | 115 | -------------------------------------------------------------------------- 116 | 117 | ``conda`` 118 | --------- 119 | 120 | * is an open source platform independent package management system and environment management system for installing multiple versions of software packages and their dependencies and switching easily between them. 121 | * is also an environment manager application. A conda environment is a directory that contains a specific collection of conda packages that you have installed. 122 | 123 | -------------------------------------------------------------------------- 124 | 125 | ``conda`` package management 126 | '''''''''''''''''''''''''''' 127 | 128 | .. code:: bash 129 | 130 | $ conda install SomePackage 131 | $ conda search SomePackage 132 | $ conda update SomePackage 133 | $ conda remove SomePackage 134 | 135 | -------------------------------------------------------------------------- 136 | 137 | ``conda`` environments 138 | '''''''''''''''''''''' 139 | 140 | .. code:: bash 141 | 142 | $ conda-env create --name # Creates environment based on a environment file 143 | $ conda-env remove --name # Export a given environment 144 | $ conda-env export # Export a given environment 145 | $ conda-env list # Lists conda environments 146 | 147 | 148 | Usage: 149 | 150 | .. code:: bash 151 | 152 | $ conda-env create mytest 153 | $ source activate 154 | (mytest)$ source deactivate 155 | 156 | -------------------------------------------------------------------------- 157 | 158 | Demo 159 | ---- 160 | 161 | ``$ conda create -n py35 python=3.5`` 162 | -------------------------------------------------------------------------------- /AdvancedPython/Packaging/packages2.rst: -------------------------------------------------------------------------------- 1 | Packaging 2 | --------- 3 | 4 | ----------------------------------------------------- 5 | 6 | * What is a distribution/package? 7 | * How to create? 8 | * How to distribute? 9 | * How to install? 10 | 11 | https://packaging.python.org 12 | 13 | ----------------------------------------------------- 14 | 15 | Goals 16 | ~~~~~ 17 | 18 | * package creation including metadata, Python code & binary dependencies 19 | * distribute 20 | * install 21 | * dependency management 22 | 23 | ----------------------------------------------------- 24 | 25 | Introduction 26 | ~~~~~~~~~~~~~~ 27 | 28 | 29 | A bit of history 30 | ^^^^^^^^^^^^^^^^^^ 31 | 32 | * 2000 ``distutils`` introduced in Python 1.6 33 | * no dependency management 34 | * no consistant way to reproduce installation 35 | * not all metadata is handled 36 | * still the default packaging library included in the std. lib. 37 | * 2001 PEP 241: metadata of distributions 38 | * 2002-2003 PEP 301: PyPI 39 | 40 | ----------------------------------------------------- 41 | 42 | * 2004 ``setuptools`` 43 | * based on ``distutils`` 44 | * no way to uninstall packages 45 | * dependency management 46 | * introduces the Egg format and ``easy_install`` 47 | * is not part of Python's standard library 48 | 49 | ----------------------------------------------------- 50 | 51 | * 2008 ``distribute`` was introduced as a fork of ``setuptools`` 52 | * ... was merged back to setuptools 0.7 53 | * ``distribute2``: the aim was to replace distutils with all the functionalities introduced in previous attempts 54 | * included in Python 3.3 beta release 55 | * in 2012 it gets abandoned 56 | 57 | ----------------------------------------------------- 58 | 59 | * 2013 PyPA team maintains the "Python Packaging User Guide" 60 | * 2014 ``pip`` evolves 61 | * does not require setuptools 62 | * available as a wheel 63 | * included in Python 3.4 64 | * 2014 ``pyvenv`` is available in Python 3.4 65 | 66 | ----------------------------------------------------- 67 | 68 | Read more: 69 | 70 | * https://packaging.python.org/en/latest/history.html 71 | * http://stackoverflow.com/questions/6344076/differences-between-distribute-distutils-setuptools-and-distutils2 72 | 73 | ----------------------------------------------------- 74 | 75 | Rule of thumb 76 | ^^^^^^^^^^^^^^^^^^ 77 | 78 | https://docs.python.org/3.4/installing/index.html 79 | 80 | * ``pip`` is the preferred installer program 81 | * ``pyvenv`` is the standard tool for creating virtual environments 82 | * ``setuptools`` is the preferred way of creating distribution packages 83 | * PyPI is used for publishing open source Python packages 84 | * Wheels are a sort of standard 85 | 86 | ----------------------------------------------------- 87 | 88 | Glossary 89 | ^^^^^^^^^ 90 | 91 | Distribution package 92 | 93 | * a versioned archive file containing packages, modules (aka. package, distribution) 94 | * end user can download and install it 95 | 96 | Built distribution 97 | 98 | * contains files and metadata 99 | * can be directly deployed to the target OS 100 | * Python files do not have to be precompiled 101 | 102 | ----------------------------------------------------- 103 | 104 | Source distribution 105 | 106 | * contains only metadata and source files 107 | * a build distribution can be built from them 108 | * can be generated with ``python setup.py sdist`` 109 | 110 | Binary distribution 111 | 112 | * a specific build distribution 113 | * contains compiled extensions as well 114 | 115 | ----------------------------------------------------- 116 | 117 | 118 | Package formats 119 | ~~~~~~~~~~~~~~~~ 120 | 121 | ----------------------------------------------------- 122 | 123 | Eggs 124 | ^^^^^^^^^ 125 | 126 | * ~ Java jars: binary format (zip file) 127 | * ``python setup.py bdist`` 128 | * bundles additional metadata 129 | * enables the system to satisfy dependencies on runtime 130 | * plugin mechanism 131 | * may contain C extensions => not really platform independent 132 | * zero installation (copy the .egg file) 133 | 134 | ----------------------------------------------------- 135 | 136 | **Wheels** 137 | ^^^^^^^^^^^ 138 | This is becoming the new standard: (PEP 427) 139 | 140 | * zip format archive (.whl) 141 | * ``bdist_wheel`` 142 | * installation format (do not include .pyc files) 143 | * implements PEP 345,376 (metadata, database format) 144 | * support multiple Python versions 145 | * versioned 146 | * intended to replace the Egg format 147 | 148 | ----------------------------------------------------- 149 | 150 | **Universal Wheels** are wheels that are pure python (i.e. contains no compiled extensions) and support Python 2 and 3. This is a wheel that can be installed anywhere by pip. 151 | 152 | .. code-block:: bash 153 | 154 | $ python setup.py bdist_wheel --universal 155 | 156 | ----------------------------------------------------- 157 | 158 | **Pure Python Wheels** that are not “universal” are wheels that are pure python (i.e. contains no compiled extensions), but don’t natively support both Python 2 and 3. 159 | 160 | .. code-block:: bash 161 | 162 | $ python setup.py bdist_wheel 163 | 164 | ----------------------------------------------------- 165 | 166 | **Platform Wheels** are wheels that are specific to a certain platform like linux, OSX, or Windows, usually due to containing compiled extensions. 167 | 168 | .. code-block:: bash 169 | 170 | $ python setup.py bdist_wheel 171 | 172 | Naming of platform wheels: https://www.python.org/dev/peps/pep-0425/ 173 | 174 | ----------------------------------------------------- 175 | 176 | You'll need a ``setup.py`` for creating a distribution package... 177 | 178 | 179 | .. code-block:: bash 180 | 181 | python setup.py sdist # Source distribution 182 | 183 | python setup.py bdist # Egg 184 | 185 | python setup.py bdist_wheel # Wheel 186 | 187 | python setup.py build # compile 188 | 189 | python setup.py install # install package 190 | 191 | python setup.py install --user # install package to the user home dir. 192 | 193 | python setup.py develop # symlink package 194 | 195 | ----------------------------------------------------- 196 | 197 | Installation tools 198 | ~~~~~~~~~~~~~~~~~~~ 199 | 200 | ``easy_install`` 201 | 202 | Let's you automatically download (from PyPI), build and install packages (which are Eggs) 203 | 204 | ``pip`` 205 | 206 | An **enhanced** tool for installing and managing Python packages involving repository support, uninstall, upgrade, ... 207 | 208 | ----------------------------------------------------- 209 | 210 | .. image:: install_tools.png 211 | 212 | ----------------------------------------------------- 213 | 214 | Prefer ``pip`` 215 | ^^^^^^^^^^^^^^^ 216 | 217 | * All packages are downloaded before installation. Partially-completed installation doesn’t occur as a result. 218 | * Care is taken to present useful output on the console. 219 | * The reasons for actions are kept track of. For instance, if a package is being installed, pip keeps track of why that package was required. 220 | * Meaningful error messages should be useful. 221 | 222 | ----------------------------------------------------- 223 | 224 | * The code is relatively concise and cohesive, making it easier to use programmatically. 225 | * Native support for other version control systems (Git, Mercurial and Bazaar) 226 | * Uninstallation of packages. 227 | * Simple to define fixed sets of requirements and reliably reproduce a set of packages. 228 | 229 | ----------------------------------------------------- 230 | 231 | Creating packages 232 | ~~~~~~~~~~~~~~~~~~ 233 | 234 | 1. ``distutils`` 235 | * part of the std. lib. 236 | * lacks features 237 | * deprecated 238 | 2. ``setuptools`` 239 | * improves on ``distutils`` 240 | 241 | ----------------------------------------------------- 242 | 243 | Components 244 | ~~~~~~~~~~~ 245 | 246 | (We use ``setuptools``) 247 | 248 | * a top-level package (not always required) 249 | * ``setup.py`` 250 | 251 | Additionally, 252 | 253 | * ``README`` 254 | * ``setup.cfg`` 255 | 256 | ----------------------------------------------------- 257 | 258 | Directory structure 259 | ^^^^^^^^^^^^^^^^^^^^ 260 | 261 | .. code-block:: text 262 | 263 | sampleproject/ 264 | |-- DESCRIPTION.rst 265 | |-- README.rst 266 | |-- setup.py 267 | |-- setup.cfg 268 | |-- simple 269 | | |-- __init__.py 270 | | |-- __main__.py 271 | |-- tests 272 | |-- |-- __init__.py 273 | |-- |-- test_simple.py 274 | 275 | 276 | ----------------------------------------------------- 277 | 278 | ``setup.py`` 279 | ^^^^^^^^^^^^^^^^^^^^ 280 | 281 | .. code-block:: python 282 | 283 | from setuptools import setup, find_packages 284 | from os import path 285 | 286 | here = path.abspath(path.dirname(__file__)) 287 | 288 | with open(path.join(here, 'DESCRIPTION.rst'), encoding='utf-8') as f: 289 | long_description = f.read() 290 | 291 | ----------------------------------------------------- 292 | 293 | .. code-block:: python 294 | 295 | setup( 296 | name='sample', 297 | version='1.2.0', 298 | description='A sample Python project', 299 | long_description=long_description, 300 | url='https://github.com/pypa/sampleproject', 301 | author='The Python Packaging Authority', 302 | author_email='pypa-dev@googlegroups.com', 303 | license='MIT', 304 | classifiers=[ 305 | 'Development Status :: 3 - Alpha', 306 | 'Intended Audience :: Developers', 307 | 'Topic :: Software Development :: Build Tools', 308 | 'License :: OSI Approved :: MIT License', 309 | 'Programming Language :: Python :: 3.4', 310 | ], 311 | 312 | ----------------------------------------------------- 313 | 314 | .. code-block:: python 315 | 316 | keywords='sample setuptools development', 317 | packages=find_packages(exclude=['contrib', 'docs', 'tests*']), 318 | install_requires=['peppercorn'], 319 | package_data={ 320 | 'sample': ['package_data.dat'], 321 | }, 322 | entry_points={ 323 | 'console_scripts': [ 324 | 'myscript=simple:__main__.main', 325 | ], 326 | }, 327 | ) 328 | 329 | https://pythonhosted.org/setuptools/setuptools.html#basic-use 330 | 331 | ----------------------------------------------------- 332 | 333 | Some conventions: 334 | 335 | * name: PEP426 336 | * version: PEP440 337 | 338 | An example project: https://github.com/pypa/sampleproject 339 | 340 | ----------------------------------------------------- 341 | 342 | 343 | Building with ``setup.py`` 344 | ^^^^^^^^^^^^^^^^^^^^^^^^^ 345 | 346 | .. code-block:: bash 347 | 348 | # installs a locally editable version 349 | # (same as ``pip install -e``) 350 | $ python setup.py develop 351 | # creates a source distribution 352 | $ python setup.py sdist 353 | # creates a (platform) wheel distribution 354 | $ python setup.py bdist_wheel 355 | # creates a universal dist. which works with both 356 | # Python 2 and 3 and do not have any C extension 357 | $ python setup.py bdist_wheel --universal 358 | 359 | 360 | ----------------------------------------------------- 361 | 362 | Upload to PyPI 363 | ^^^^^^^^^^^^^^^ 364 | 365 | https://packaging.python.org/en/latest/distributing.html#uploading-your-project-to-pypi 366 | 367 | ``$ python setup.py upload`` 368 | 369 | ``$ twine upload`` 370 | 371 | ``$ python setup.py sdist bdist_wheel upload`` 372 | 373 | ----------------------------------------------------- 374 | 375 | 376 | Future 377 | ~~~~~~ 378 | 379 | https://packaging.python.org/en/latest/future.html 380 | -------------------------------------------------------------------------------- /AdvancedPython/Packaging/packaging_plus.rst: -------------------------------------------------------------------------------- 1 | ``argparse`` 2 | ------------ 3 | 4 | The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of ``sys.argv```. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments. 5 | 6 | ------------------------------------------------------ 7 | 8 | .. code-block:: python 9 | 10 | import argparse 11 | parser = argparse.ArgumentParser(description='This is a sample program') 12 | parser.add_argument("echo") 13 | args = parser.parse_args() 14 | 15 | print(args.echo) 16 | 17 | ``$ python sample.py echo`` 18 | 19 | ------------------------------------------------------ 20 | 21 | Examples 22 | ~~~~~~~~ 23 | 24 | Positional arguments 25 | ~~~~~~~~~~~~~~~~~~~~~~ 26 | 27 | .. code-block:: python 28 | 29 | parser.add_argument("square", type=int, 30 | help="display a square of a given number") 31 | 32 | parser.add_argument('dir', nargs='?', default=os.getcwd()) 33 | 34 | ------------------------------------------------------ 35 | 36 | Optional arguments 37 | ~~~~~~~~~~~~~~~~~~~~~~ 38 | 39 | .. code-block:: python 40 | 41 | parser.add_argument("-v", "--verbosity", type=int, 42 | help="increase output verbosity") 43 | 44 | parser.add_argument("-v", "--verbose", 45 | help="increase output verbosity", 46 | action="store_true") 47 | 48 | ------------------------------------------------------ 49 | 50 | Package components 51 | ------------------ 52 | 53 | ``__init__.py`` 54 | ~~~~~~~~~~~~~~~~~ 55 | 56 | * packages are directories having a ``__init__.py`` 57 | * it was necessary for packages before Python 3.3 58 | * since then it became optional: any directory can be a package thus can be imported 59 | * rule of thumb: use ``__init__.py`` if you want create a package 60 | 61 | ------------------------------------------------------ 62 | 63 | ``__main__.py`` 64 | ~~~~~~~~~~~~~~~~~ 65 | 66 | .. code-block:: None 67 | 68 | myproject/ 69 | |-- simple 70 | |-- __main__.py 71 | 72 | file: __main__.py 73 | 74 | print("hello world") 75 | 76 | $ python -m simple 77 | hello world 78 | 79 | 80 | 81 | ------------------------------------------------------ 82 | 83 | Relative imports 84 | ~~~~~~~~~~~~~~~~~ 85 | 86 | * ``from . import something`` 87 | * ``from .. import something`` 88 | * ``from ... import something`` 89 | 90 | But ``__name__`` is used to resolve the hierarchy! 91 | 92 | ------------------------------------------------------ 93 | 94 | ``~$ python -m package.subpackage1.moduleX`` 95 | 96 | .. code-block:: none 97 | 98 | ~/package/ 99 | __init__.py 100 | subpackage1/ 101 | __init__.py 102 | moduleX.py <== 103 | moduleY.py 104 | subpackage2/ 105 | __init__.py 106 | moduleZ.py 107 | moduleA.py 108 | 109 | ------------------------------------------------------ 110 | 111 | .. code-block:: python 112 | 113 | from .moduleY import spam 114 | from .moduleY import spam as ham 115 | from . import moduleY 116 | from ..subpackage1 import moduleY 117 | from ..subpackage2.moduleZ import eggs 118 | from ..moduleA import foo 119 | from ...package import bar 120 | 121 | ------------------------------------------------------ 122 | 123 | Rule of thumb 124 | ~~~~~~~~~~~~~ 125 | 126 | Use absolute imports! 127 | ''''''''''''''''''''''''''' 128 | 129 | 130 | .. code-block:: python 131 | 132 | from package.subpackage1.moduleY import spam 133 | from package.subpackage1.moduleY import spam as ham 134 | from package.subpackage1 import moduleY 135 | from package.subpackage1 import moduleY 136 | from package.subpackage2.moduleZ import eggs 137 | from package.moduleA import foo 138 | from package import bar 139 | 140 | 141 | 142 | ------------------------------------------------------ 143 | 144 | RsT 145 | ~~~ 146 | 147 | ReStructuredText is a plaintext markup syntax used as the default documentation format for Python projects. 148 | 149 | http://docutils.sourceforge.net/rst.html 150 | 151 | http://docutils.sourceforge.net/docs/user/rst/quickref.html 152 | -------------------------------------------------------------------------------- /AdvancedPython/PythonForDataScience/Sklearn introduction II.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Building a pipeline" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": { 14 | "collapsed": false 15 | }, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "Populating the interactive namespace from numpy and matplotlib\n" 22 | ] 23 | } 24 | ], 25 | "source": [ 26 | "%pylab inline" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 2, 32 | "metadata": { 33 | "collapsed": true 34 | }, 35 | "outputs": [], 36 | "source": [ 37 | "import sklearn\n", 38 | "from sklearn.linear_model import LogisticRegression\n", 39 | "from sklearn.datasets import load_digits\n", 40 | "from sklearn.pipeline import Pipeline\n", 41 | "from sklearn.decomposition import PCA" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": 3, 47 | "metadata": { 48 | "collapsed": true 49 | }, 50 | "outputs": [], 51 | "source": [ 52 | "digits = load_digits()\n", 53 | "X_digits = digits.data\n", 54 | "y_digits = digits.target" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 4, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "logistic = LogisticRegression()\n", 66 | "pca = PCA()\n", 67 | "pipe = Pipeline(steps=[('pca', pca), ('logistic', logistic)])" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 5, 73 | "metadata": { 74 | "collapsed": false 75 | }, 76 | "outputs": [ 77 | { 78 | "data": { 79 | "text/plain": [ 80 | "Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('logistic', LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n", 81 | " intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n", 82 | " penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n", 83 | " verbose=0, warm_start=False))])" 84 | ] 85 | }, 86 | "execution_count": 5, 87 | "metadata": {}, 88 | "output_type": "execute_result" 89 | } 90 | ], 91 | "source": [ 92 | "pipe.fit(X_digits, y_digits)" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 6, 98 | "metadata": { 99 | "collapsed": false 100 | }, 101 | "outputs": [ 102 | { 103 | "data": { 104 | "text/plain": [ 105 | "array([0])" 106 | ] 107 | }, 108 | "execution_count": 6, 109 | "metadata": {}, 110 | "output_type": "execute_result" 111 | } 112 | ], 113 | "source": [ 114 | "pipe.predict(X_digits[:1])" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "## Finding the best model" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 7, 127 | "metadata": { 128 | "collapsed": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "from sklearn.grid_search import GridSearchCV" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": 8, 138 | "metadata": { 139 | "collapsed": false 140 | }, 141 | "outputs": [ 142 | { 143 | "data": { 144 | "text/plain": [ 145 | "GridSearchCV(cv=5, error_score='raise',\n", 146 | " estimator=Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('logistic', LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n", 147 | " intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n", 148 | " penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n", 149 | " verbose=0, warm_start=False))]),\n", 150 | " fit_params={}, iid=True, n_jobs=8,\n", 151 | " param_grid={'logistic__class_weight': [None, 'balanced'], 'logistic__C': array([ 1.00000e-04, 1.00000e-02, 1.00000e+00]), 'logistic__penalty': ['l1', 'l2'], 'pca__n_components': [20, 40, 64]},\n", 152 | " pre_dispatch='2*n_jobs', refit=True, scoring=None, verbose=0)" 153 | ] 154 | }, 155 | "execution_count": 8, 156 | "metadata": {}, 157 | "output_type": "execute_result" 158 | } 159 | ], 160 | "source": [ 161 | "n_components = [20, 40, 64] # number of compomentens in PCA \n", 162 | "Cs = np.logspace(-4, 0, 3, 4) # Inverse of regularization strength\n", 163 | "penalty = [\"l1\", \"l2\"] # Norm used by the Logistic regression penalization\n", 164 | "class_weight = [None, \"balanced\"] # Weights associatied with clases\n", 165 | "\n", 166 | "estimator = GridSearchCV(pipe,\n", 167 | " {\"pca__n_components\": n_components,\n", 168 | " \"logistic__C\": Cs,\n", 169 | " \"logistic__class_weight\": class_weight,\n", 170 | " \"logistic__penalty\": penalty\n", 171 | " }, n_jobs=8, cv=5)\n", 172 | "estimator.fit(X_digits, y_digits)" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 9, 178 | "metadata": { 179 | "collapsed": false, 180 | "scrolled": true 181 | }, 182 | "outputs": [ 183 | { 184 | "data": { 185 | "text/plain": [ 186 | "[mean: 0.09905, std: 0.00076, params: {'logistic__class_weight': None, 'pca__n_components': 20, 'logistic__penalty': 'l1', 'logistic__C': 0.0001},\n", 187 | " mean: 0.09905, std: 0.00076, params: {'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 0.0001},\n", 188 | " mean: 0.09905, std: 0.00076, params: {'logistic__class_weight': None, 'pca__n_components': 64, 'logistic__penalty': 'l1', 'logistic__C': 0.0001},\n", 189 | " mean: 0.87702, std: 0.03613, params: {'logistic__class_weight': None, 'pca__n_components': 20, 'logistic__penalty': 'l2', 'logistic__C': 0.0001},\n", 190 | " mean: 0.88592, std: 0.03789, params: {'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l2', 'logistic__C': 0.0001},\n", 191 | " mean: 0.88703, std: 0.03829, params: {'logistic__class_weight': None, 'pca__n_components': 64, 'logistic__penalty': 'l2', 'logistic__C': 0.0001},\n", 192 | " mean: 0.09905, std: 0.00076, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 20, 'logistic__penalty': 'l1', 'logistic__C': 0.0001},\n", 193 | " mean: 0.09905, std: 0.00076, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 0.0001},\n", 194 | " mean: 0.09905, std: 0.00076, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 64, 'logistic__penalty': 'l1', 'logistic__C': 0.0001},\n", 195 | " mean: 0.87590, std: 0.03502, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 20, 'logistic__penalty': 'l2', 'logistic__C': 0.0001},\n", 196 | " mean: 0.88592, std: 0.03789, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 40, 'logistic__penalty': 'l2', 'logistic__C': 0.0001},\n", 197 | " mean: 0.88703, std: 0.03829, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 64, 'logistic__penalty': 'l2', 'logistic__C': 0.0001},\n", 198 | " mean: 0.89482, std: 0.03603, params: {'logistic__class_weight': None, 'pca__n_components': 20, 'logistic__penalty': 'l1', 'logistic__C': 0.01},\n", 199 | " mean: 0.89538, std: 0.03581, params: {'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 0.01},\n", 200 | " mean: 0.89538, std: 0.03581, params: {'logistic__class_weight': None, 'pca__n_components': 64, 'logistic__penalty': 'l1', 'logistic__C': 0.01},\n", 201 | " mean: 0.88481, std: 0.03741, params: {'logistic__class_weight': None, 'pca__n_components': 20, 'logistic__penalty': 'l2', 'logistic__C': 0.01},\n", 202 | " mean: 0.90484, std: 0.03247, params: {'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l2', 'logistic__C': 0.01},\n", 203 | " mean: 0.90262, std: 0.03276, params: {'logistic__class_weight': None, 'pca__n_components': 64, 'logistic__penalty': 'l2', 'logistic__C': 0.01},\n", 204 | " mean: 0.89482, std: 0.03603, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 20, 'logistic__penalty': 'l1', 'logistic__C': 0.01},\n", 205 | " mean: 0.89649, std: 0.03544, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 0.01},\n", 206 | " mean: 0.89649, std: 0.03544, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 64, 'logistic__penalty': 'l1', 'logistic__C': 0.01},\n", 207 | " mean: 0.88481, std: 0.03862, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 20, 'logistic__penalty': 'l2', 'logistic__C': 0.01},\n", 208 | " mean: 0.90484, std: 0.03247, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 40, 'logistic__penalty': 'l2', 'logistic__C': 0.01},\n", 209 | " mean: 0.90373, std: 0.03219, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 64, 'logistic__penalty': 'l2', 'logistic__C': 0.01},\n", 210 | " mean: 0.91653, std: 0.02974, params: {'logistic__class_weight': None, 'pca__n_components': 20, 'logistic__penalty': 'l1', 'logistic__C': 1.0},\n", 211 | " mean: 0.92209, std: 0.03039, params: {'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 1.0},\n", 212 | " mean: 0.91820, std: 0.02980, params: {'logistic__class_weight': None, 'pca__n_components': 64, 'logistic__penalty': 'l1', 'logistic__C': 1.0},\n", 213 | " mean: 0.91820, std: 0.02660, params: {'logistic__class_weight': None, 'pca__n_components': 20, 'logistic__penalty': 'l2', 'logistic__C': 1.0},\n", 214 | " mean: 0.91820, std: 0.03286, params: {'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l2', 'logistic__C': 1.0},\n", 215 | " mean: 0.91653, std: 0.03731, params: {'logistic__class_weight': None, 'pca__n_components': 64, 'logistic__penalty': 'l2', 'logistic__C': 1.0},\n", 216 | " mean: 0.91653, std: 0.02974, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 20, 'logistic__penalty': 'l1', 'logistic__C': 1.0},\n", 217 | " mean: 0.92154, std: 0.03069, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 1.0},\n", 218 | " mean: 0.91820, std: 0.02980, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 64, 'logistic__penalty': 'l1', 'logistic__C': 1.0},\n", 219 | " mean: 0.91875, std: 0.02700, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 20, 'logistic__penalty': 'l2', 'logistic__C': 1.0},\n", 220 | " mean: 0.91708, std: 0.03293, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 40, 'logistic__penalty': 'l2', 'logistic__C': 1.0},\n", 221 | " mean: 0.91597, std: 0.03789, params: {'logistic__class_weight': 'balanced', 'pca__n_components': 64, 'logistic__penalty': 'l2', 'logistic__C': 1.0}]" 222 | ] 223 | }, 224 | "execution_count": 9, 225 | "metadata": {}, 226 | "output_type": "execute_result" 227 | } 228 | ], 229 | "source": [ 230 | "estimator.grid_scores_" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": 10, 236 | "metadata": { 237 | "collapsed": false 238 | }, 239 | "outputs": [ 240 | { 241 | "name": "stdout", 242 | "output_type": "stream", 243 | "text": [ 244 | "0.922092376183\n", 245 | "{'logistic__class_weight': None, 'pca__n_components': 40, 'logistic__penalty': 'l1', 'logistic__C': 1.0}\n" 246 | ] 247 | } 248 | ], 249 | "source": [ 250 | "print(estimator.best_score_)\n", 251 | "print(estimator.best_params_)" 252 | ] 253 | }, 254 | { 255 | "cell_type": "markdown", 256 | "metadata": {}, 257 | "source": [ 258 | "## Exercise" 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": {}, 264 | "source": [ 265 | "Find the best model for the diabetes dataset\n", 266 | "\n", 267 | "http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html#sklearn.datasets.load_diabetes" 268 | ] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "execution_count": null, 273 | "metadata": { 274 | "collapsed": true 275 | }, 276 | "outputs": [], 277 | "source": [] 278 | } 279 | ], 280 | "metadata": { 281 | "kernelspec": { 282 | "display_name": "Python 3", 283 | "language": "python", 284 | "name": "python3" 285 | }, 286 | "language_info": { 287 | "codemirror_mode": { 288 | "name": "ipython", 289 | "version": 3 290 | }, 291 | "file_extension": ".py", 292 | "mimetype": "text/x-python", 293 | "name": "python", 294 | "nbconvert_exporter": "python", 295 | "pygments_lexer": "ipython3", 296 | "version": "3.5.1" 297 | } 298 | }, 299 | "nbformat": 4, 300 | "nbformat_minor": 0 301 | } 302 | -------------------------------------------------------------------------------- /AdvancedPython/PythonForDataScience/ipython.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/AdvancedPython/PythonForDataScience/ipython.png -------------------------------------------------------------------------------- /AdvancedPython/PythonForDataScience/ipython.rst: -------------------------------------------------------------------------------- 1 | IPython / Jupyter 2 | ------------------ 3 | 4 | ------------------------------------------------------------ 5 | 6 | Overview 7 | ~~~~~~~~~ 8 | 9 | **I** nteractive **Python** is an enhanced Python console, with 10 | 11 | * tab completion (in many contexts!) 12 | * object exploration 13 | * integrated debugger 14 | * history management 15 | * system shell integration 16 | * *magic functions* 17 | 18 | ``%quickref`` 19 | 20 | ------------------------------------------------------------ 21 | 22 | IPython notebook 23 | ~~~~~~~~~~~~~~~~~ 24 | 25 | * **interactive Pthon console in your browser** 26 | * can display inline html and images 27 | * you can take notes 28 | * export functions 29 | 30 | ------------------------------------------------------------ 31 | 32 | .. image:: 33 | ipython_screener.png 34 | 35 | ------------------------------------------------------------ 36 | 37 | Architecture 38 | ~~~~~~~~~~~~ 39 | 40 | .. image:: 41 | ipython.png 42 | 43 | ------------------------------------------------------------ 44 | 45 | Object exploration 46 | ~~~~~~~~~~~~~~~~~~ 47 | 48 | .. code-block:: ipython 49 | 50 | In [3]: import os 51 | In [4]: ?os 52 | Type: module 53 | String Form: 54 | File: /usr/lib/python2.7/os.py 55 | Docstring: 56 | OS routines for Mac, NT, or Posix depending on what system we're on. 57 | 58 | This exports: 59 | - all functions from posix, nt, os2, or ce, e.g. unlink, stat, etc. 60 | - os.path is one of the modules posixpath, or ntpath 61 | - os.name is 'posix', 'nt', 'os2', 'ce' or 'riscos' 62 | ... 63 | 64 | ------------------------------------------------------------ 65 | 66 | History 67 | ~~~~~~~ 68 | 69 | .. code-block:: txt 70 | 71 | _i, _ii, _iii : Previous, next previous, next next previous input 72 | _i4, _ih[2:5] : Input history line 4, lines 2-4 73 | exec _i81 : Execute input history line #81 again 74 | %rep 81 : Edit input history line #81 75 | _, __, ___ : previous, next previous, next next previous output 76 | _dh : Directory history 77 | _oh : Output history 78 | %hist : Command history. '%hist -g foo' search history for 'foo' 79 | 80 | 81 | ------------------------------------------------------------ 82 | 83 | Autocall 84 | ~~~~~~~~~ 85 | 86 | .. code-block:: txt 87 | 88 | f 1,2 : f(1,2) # Off by default, enable with %autocall magic. 89 | /f 1,2 : f(1,2) (forced autoparen) 90 | ,f 1 2 : f("1","2") 91 | ;f 1 2 : f("1 2") 92 | 93 | 94 | ----------------------------------------------------------- 95 | 96 | Shell integration 97 | ~~~~~~~~~~~~~~~~~~ 98 | 99 | Several commands are available by default (e.g ``ls, pwd, cd, rm, mkdir``) 100 | 101 | .. code-block:: txt 102 | 103 | !cp a.txt b/ : System command escape by os.system() 104 | cp ${f}.txt $bar : Variable expansion 105 | files = !ls /usr : Capture sytem command output 106 | files.s : "a b c" 107 | files.l : ['a','b','c'] 108 | files.n : 'a\nb\nc' 109 | 110 | 111 | 112 | ------------------------------------------------------------ 113 | 114 | Magic functions 115 | ~~~~~~~~~~~~~~~~~~ 116 | 117 | * starts with ``%``, but it can be omitted 118 | 119 | Managing history: 120 | 121 | .. code-block:: txt 122 | 123 | In[1]: %save myfile 3-6 6-8 124 | The following commands were written to file `myfile.py` 125 | ... 126 | In[2]: %load myfile 127 | ... 128 | In[3]: %run myscript.py 129 | ... 130 | In[4]: %macro mymacro 3-4 131 | 132 | 133 | ------------------------------------------------------------ 134 | 135 | Other magic functions 136 | ~~~~~~~~~~~~~~~~~~~~~~ 137 | 138 | .. code-block:: txt 139 | 140 | %alias: 141 | Define an alias for a system command. 142 | %edit: 143 | Bring up an editor and execute the resulting code. 144 | %lsmagic: 145 | List currently available magic functions. 146 | %bookmark: 147 | Manage IPython's bookmark system. 148 | %timeit: 149 | Time execution of a Python statement or expression 150 | 151 | 152 | ------------------------------------------------------------ 153 | 154 | Debugging 155 | ~~~~~~~~~ 156 | 157 | ``%run -d theprogram.py`` 158 | 159 | Profiling 160 | ~~~~~~~~~ 161 | 162 | ``%run -p theprogram.py`` 163 | 164 | -------------------------------------------------------------------------- 165 | 166 | DEMO 167 | ==== 168 | -------------------------------------------------------------------------------- /AdvancedPython/PythonForDataScience/ipython_screener.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/AdvancedPython/PythonForDataScience/ipython_screener.png -------------------------------------------------------------------------------- /AdvancedPython/PythonForDataScience/scipy.rst: -------------------------------------------------------------------------------- 1 | SciPy 2 | ====== 3 | 4 | -------------------------------------------------------- 5 | 6 | "... a Python ecosystem of software fot mathematics, science, engineering." 7 | 8 | Core packages: 9 | 10 | * **NumPy** 11 | * **IPython** 12 | * **matplotlib** 13 | * SymPy 14 | * SciPy library 15 | * pandas 16 | * nose 17 | 18 | In addition, many others are built on top of it... 19 | 20 | --------------------------------------------------------- 21 | 22 | SciPy library 23 | ------------- 24 | * ``fftpack`` - discrete fourier transform algorithms 25 | * ``integrate`` - integration routines 26 | * ``interpolate`` - interpolation tools 27 | * ``linalg`` - linear algebra routines 28 | * ``optimize`` - optimization tools 29 | * ``signal`` - signal processing tools 30 | * ``sparse`` - sparse matrices 31 | * ``stats`` - statistical functions 32 | * ``io`` - data input and output 33 | * ``special`` - definitions of many usual math functions 34 | * ``weave`` - C/C++ integration 35 | 36 | --------------------------------------------------------- 37 | 38 | ``scipy.sparse`` 39 | ----------------- 40 | 41 | Data structure for representing sparse matrices 42 | 43 | Compressed-Sparse-Row matrix 44 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 45 | 46 | .. code-block:: python 47 | 48 | """converts to CSR matrix""" 49 | X_csr = sparse.csr_matrix(X) 50 | 51 | """converts back to a dense array""" 52 | X_1 = X_csr.toarray() 53 | 54 | * Suppports matrix algebra (e.g. ``X_csr.dot(y)``) 55 | 56 | --------------------------------------------------------- 57 | 58 | List-in-List matrix 59 | ~~~~~~~~~~~~~~~~~~~ 60 | 61 | * effeicient element insertion 62 | 63 | .. code-block:: python 64 | 65 | """converts to LIL matrix""" 66 | X_csr = sparse.lil_matrix(X) 67 | 68 | """converts back to a dense array""" 69 | X_1 = X_lil.toarray() 70 | 71 | --------------------------------------------------------- 72 | 73 | Other data types 74 | ~~~~~~~~~~~~~~~~~ 75 | 76 | * CSC (compressed sparse column) 77 | * BSR (block sparse row) 78 | * COO (coordinate) 79 | * DIA (diagonal) 80 | * DOK (dictionary of keys) 81 | 82 | --------------------------------------------------------- 83 | 84 | ``scipy.linalg`` 85 | ----------------- 86 | 87 | http://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html 88 | 89 | * contains ``numpy.linalg`` and extends it 90 | * might be faster (as it is compiled with BAS support) 91 | 92 | --------------------------------------------------------- 93 | 94 | 95 | ``scipy.stats`` 96 | --------------- 97 | 98 | * 80< continuous random variables & 10 discrete random variables 99 | * statistical function 100 | * ... 101 | 102 | 103 | --------------------------------------------------------- 104 | 105 | ``scipy.io`` 106 | ------------- 107 | 108 | Contains methods for reading and writing data 109 | 110 | http://docs.scipy.org/doc/scipy/reference/tutorial/io.html 111 | 112 | .. code-block:: python 113 | 114 | import scipy.io as sio 115 | mat_contents = sio.loadmat('my_file.mat') 116 | 117 | vect = np.arange(10) 118 | sio.savemat('np_vector.mat', {'vect':vect}) 119 | 120 | --------------------------------------------------------- 121 | 122 | ``scipy.weave`` 123 | --------------- 124 | 125 | http://docs.scipy.org/doc/scipy/reference/tutorial/weave.html 126 | 127 | Contains tools for including C/C++ code within Python. 128 | 129 | * ``weave.inline()`` - compiles and executes C/C++ code on the fly 130 | * ``weave.blitz()`` - compiles NumPy Python expression for fast execution 131 | * ``weave.ext_tools`` - classes for generating extension modules 132 | 133 | --------------------------------------------------------- 134 | 135 | 136 | ``SymPy`` 137 | --------------- 138 | 139 | http://sympy.org/en/index.html 140 | 141 | * polynomials 142 | * calculus 143 | * equation solving 144 | * combinatorics, discrete math 145 | * geometry ... 146 | 147 | .. code-block:: python 148 | 149 | expr = (x + y)**5 150 | expand(expr) 151 | """x^5+5x^4y+10x^3y^2+10x^2y^3+5xy^4+y^5""" 152 | 153 | .. code-block:: python 154 | 155 | l1 = Line(p1, p2) 156 | l2 = Line(p3, p4) 157 | px = intersection(l1, l2) 158 | 159 | 160 | --------------------------------------------------------- 161 | 162 | ``Pandas`` 163 | --------------- 164 | 165 | Tools for data analysis and modelling 166 | 167 | * I/O tools for CSV/Excel/SQL/HDF5... 168 | * data alignment tools with handling of missing data 169 | * aggregating with a group by engine 170 | * merging & joining 171 | * time series functionality 172 | * ... 173 | 174 | http://pandas.pydata.org/ 175 | 176 | --------------------------------------------------------- 177 | 178 | ``Other packages`` 179 | ------------------- 180 | 181 | Many other packages built on this library. 182 | 183 | Examples: 184 | 185 | * Mayavi - powerful 3D visualization 186 | * Chaco - another plotting tool for embedded interactive plotting 187 | * Cython - an extended Python language (extensions, integration, speed-up) 188 | 189 | --------------------------------------------------------- 190 | 191 | ``Scikits`` 192 | ------------------- 193 | 194 | http://scikits.appspot.com/scikits 195 | 196 | Extra packages for specific functionality 197 | 198 | * scikit-image 199 | * scikit-bio 200 | * scikit-learn 201 | * cuda 202 | 203 | --------------------------------------------------------- 204 | 205 | SciPy vs. Matlab 206 | ----------------- 207 | 208 | http://wiki.scipy.org/NumPy_for_Matlab_Users 209 | 210 | http://www.pyzo.org/python_vs_matlab.html 211 | 212 | -------------------------------------------------------------------------------- /AdvancedPython/PythonForDataScience/sklearn.rst: -------------------------------------------------------------------------------- 1 | Scikit-learn (sklearn) 2 | ====================== 3 | 4 | -------------------------------------------------------------------------------- 5 | 6 | Installation 7 | ------------ 8 | 9 | ``$ sudo pip[3] install scikit-learn`` 10 | 11 | ``$ conda install scikit-learn`` 12 | 13 | ``$ sudo apt-get install python-sklearn`` 14 | 15 | -------------------------------------------------------------------------------- 16 | 17 | Overview 18 | --------- 19 | 20 | 1. Problem setting 21 | 2. Supervised learning 22 | 3. Unsupervised learning 23 | 4. Validation 24 | 25 | http://github.com/jakevdp/sklearn_pycon2014 26 | 27 | -------------------------------------------------------------------------------- 28 | 29 | Problem / Data 30 | -------------- 31 | 32 | * Data is represented in a 2D array ``[n_samples x n_features]`` 33 | * numpy array 34 | * Optional: target features/classes 35 | 36 | -------------------------------------------------------------------------------- 37 | 38 | Supervised learning 39 | ------------------- 40 | 41 | Classification 42 | ~~~~~~~~~~~~~~~ 43 | 44 | * SVM 45 | * Log-linear models 46 | * Nearest neighbours 47 | * Naive Bayes 48 | * Decision Trees 49 | * etc. 50 | 51 | Regression 52 | ~~~~~~~~~~ 53 | 54 | ... 55 | 56 | 57 | -------------------------------------------------------------------------------- 58 | 59 | Unsupervised learning 60 | ---------------------- 61 | 62 | * dimensionality reduction 63 | * clustering 64 | * matric factorization 65 | * neural network models 66 | * ... 67 | 68 | -------------------------------------------------------------------------------- 69 | 70 | 71 | Estimator interface 72 | ------------------- 73 | 74 | ... is a uniform interface 75 | 76 | Notation: ``X`` is a 2D array of data, ``y`` is an array of target labels 77 | 78 | -------------------------------------------------------------------------------- 79 | 80 | Supervised case 81 | ~~~~~~~~~~~~~~~ 82 | 83 | * ``model.fit(X, y)`` - fit the training data 84 | * ``model.predict(X_new)`` - predicts labels 85 | * ``model.predict_proba(X_new)`` - predicts labels with estimated probs 86 | * ``model.score(X_test, y_test)`` - calculates average accuracy 87 | 88 | -------------------------------------------------------------------------------- 89 | 90 | Unsupervised case 91 | ~~~~~~~~~~~~~~~~~ 92 | 93 | * ``model.fit(X)`` - fit the data 94 | * ``model.transform(X)`` - transform the data 95 | * ``model.fit_transform(X)`` - fit & transform the data 96 | 97 | -------------------------------------------------------------------------------- 98 | 99 | Validation 100 | ---------- 101 | 102 | ``sklearn.metrics`` contains methods for computing 103 | 104 | * Classification metrics 105 | * Multilabel rakning metrics 106 | * Regression metrics 107 | * Clustering metrics 108 | * Pairwise metrics 109 | 110 | -------------------------------------------------------------------------------- 111 | 112 | Other components 113 | ----------------- 114 | 115 | ... and much more ... 116 | 117 | * data preprocessing 118 | * feature selection 119 | * model selection 120 | * pipelining estimators 121 | * ensemble learning 122 | * semi-supervised learning 123 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/Concurrency exercises.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Benchmarking your code" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 5, 13 | "metadata": { 14 | "collapsed": false 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "def fun():\n", 19 | " max(range(1000))" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "Using magic functions of Jupyter and `timeit`\n", 27 | "\n", 28 | "* https://docs.python.org/3.5/library/timeit.html\n", 29 | "* https://ipython.org/ipython-doc/3/interactive/magics.html#magic-time" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 10, 35 | "metadata": { 36 | "collapsed": false, 37 | "scrolled": true 38 | }, 39 | "outputs": [ 40 | { 41 | "name": "stdout", 42 | "output_type": "stream", 43 | "text": [ 44 | "10000 loops, best of 3: 27.8 µs per loop\n" 45 | ] 46 | } 47 | ], 48 | "source": [ 49 | "%%timeit\n", 50 | "fun()" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 7, 56 | "metadata": { 57 | "collapsed": false 58 | }, 59 | "outputs": [ 60 | { 61 | "name": "stdout", 62 | "output_type": "stream", 63 | "text": [ 64 | "CPU times: user 44 µs, sys: 1 µs, total: 45 µs\n", 65 | "Wall time: 47 µs\n" 66 | ] 67 | } 68 | ], 69 | "source": [ 70 | "%%time\n", 71 | "fun()" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "## Exercises" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "1. What is the fastest way to download 100 pages from index.hu?\n", 86 | "2. How to calculate the factors of 1000 random integers effectively using `factorize_naive` function below?" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 28, 92 | "metadata": { 93 | "collapsed": false 94 | }, 95 | "outputs": [ 96 | { 97 | "data": { 98 | "text/plain": [ 99 | "" 100 | ] 101 | }, 102 | "execution_count": 28, 103 | "metadata": {}, 104 | "output_type": "execute_result" 105 | } 106 | ], 107 | "source": [ 108 | "import requests\n", 109 | "def get_page(url): \n", 110 | " response = requests.request(url=url, method=\"GET\")\n", 111 | " return response\n", 112 | "get_page(\"http://index.hu\")" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 12, 118 | "metadata": { 119 | "collapsed": true 120 | }, 121 | "outputs": [], 122 | "source": [ 123 | "def factorize_naive(n):\n", 124 | " \"\"\" A naive factorization method. Take integer 'n', return list of\n", 125 | " factors.\n", 126 | " \"\"\"\n", 127 | " if n < 2:\n", 128 | " return []\n", 129 | " factors = []\n", 130 | " p = 2\n", 131 | "\n", 132 | " while True:\n", 133 | " if n == 1:\n", 134 | " return factors\n", 135 | "\n", 136 | " r = n % p\n", 137 | " if r == 0:\n", 138 | " factors.append(p)\n", 139 | " n = n // p\n", 140 | " elif p * p >= n:\n", 141 | " factors.append(n)\n", 142 | " return factors\n", 143 | " elif p > 2:\n", 144 | " # Advance in steps of 2 over odd numbers\n", 145 | " p += 2\n", 146 | " else:\n", 147 | " # If p == 2, get to 3\n", 148 | " p += 1\n", 149 | " assert False, \"unreachable\"" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [] 160 | } 161 | ], 162 | "metadata": { 163 | "kernelspec": { 164 | "display_name": "Python 3", 165 | "language": "python", 166 | "name": "python3" 167 | }, 168 | "language_info": { 169 | "codemirror_mode": { 170 | "name": "ipython", 171 | "version": 3 172 | }, 173 | "file_extension": ".py", 174 | "mimetype": "text/x-python", 175 | "name": "python", 176 | "nbconvert_exporter": "python", 177 | "pygments_lexer": "ipython3", 178 | "version": "3.5.1" 179 | } 180 | }, 181 | "nbformat": 4, 182 | "nbformat_minor": 0 183 | } 184 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/Python2vs3.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | Python 2 vs 3 3 | ============= 4 | 5 | -------------------------------------------------------------------------------- 6 | 7 | Basics 8 | ------ 9 | 10 | * Python 3 is incompatible with 2.x 11 | * due to code clean-up 12 | * some changes have been backported: 13 | ``from __future__ import ...`` 14 | * forced name conventions 15 | 16 | -------------------------------------------------------------------------------- 17 | 18 | Details: 19 | 20 | * http://docs.python.org/2/library/__future__.html 21 | * http://docs.python.org/3.0/whatsnew/3.0.html 22 | * http://wiki.python.org/moin/Python2orPython3 23 | * http://docs.python.org/2/library/2to3.html 24 | 25 | -------------------------------------------------------------------------------- 26 | 27 | ``print`` 28 | --------- 29 | 30 | * became a function 31 | * ``from __future__ import print_function`` 32 | 33 | .. code-block:: python 34 | 35 | Old: print "The answer is", 2*2 36 | New: print("The answer is", 2*2) 37 | 38 | Old: print x, # Trailing comma suppresses newline 39 | New: print(x, end=" ") # Appends a space instead of a newline 40 | 41 | Old: print # Prints a newline 42 | New: print() # You must call the function! 43 | 44 | print("It is", 2**32, "!", sep="") 45 | 46 | ----- 47 | 48 | Views and Iterators vs Lists 49 | ---------------------------- 50 | 51 | * ``dict.keys()``, ``dict.items()`` was lists but became views 52 | * ``dict.iterkeys(), iteritems() and itervalues()`` not supperted anymore 53 | * ``map(), filter()`` return iterators 54 | * ``range()`` became a generator 55 | * ``zip()`` became an iterator 56 | 57 | ---- 58 | 59 | Ordering, comparisions 60 | ---------------------- 61 | 62 | * ``1 < ''``, ``0>None`` raise ``TypeError`` 63 | * ``sort()`` and ``sorted()`` no longer accepts ``cmp`` argument, thus ``key`` should be used 64 | 65 | Integers 66 | -------- 67 | 68 | * ``long`` renamed to ``int`` 69 | * ``1/2`` equals ``0.5`` 70 | * ``1.0//2.0`` equals ``0.0`` 71 | * there is no upper bould for an integer 72 | * ``sys.maxint`` is used as a theoretic maximum 73 | 74 | ------ 75 | 76 | Text 77 | ------- 78 | Text vs. data: 79 | 80 | * all text is Unicode with type ``str`` 81 | * data is stored in ``bytes`` 82 | * conversion between bytes to strings: ``encode(), decode()`` 83 | * ``basestring`` type is deprecated 84 | 85 | * ``u".."`` is no longer valid 86 | 87 | ---- 88 | 89 | Text 90 | ---- 91 | 92 | Internal changes: 93 | 94 | * all API functions uses Unicode strings 95 | * default source encoding is UTF-8 96 | * non-ASCII letters are allowed in identifiers 97 | 98 | * ``open()`` use an encoding to map files 99 | * ``StringIO`` became ``io.StringIO`` 100 | 101 | ------ 102 | 103 | New syntax 104 | ----------- 105 | 106 | .. code-block:: python 107 | 108 | # Extended unpacking 109 | a,b, *rest = some_sequence 110 | *rest, a,b = some_sequence 111 | 112 | # dict comprehension 113 | {k:v for k,v in stuff.items()} 114 | 115 | # set comprehension 116 | {k for k in stuff} 117 | 118 | ---- 119 | 120 | Changed syntax 121 | -------------- 122 | * new Metaclass syntax "``class C(metaclass=M)``" 123 | * ``raw_input()`` became ``input()`` 124 | * new style classes are default: 125 | "``class A: pass``" and 126 | "``class A(object): pass``" are equal 127 | 128 | ----- 129 | 130 | * ``nonlocal`` statement: reach an outer but non-blobal 131 | * ``True``, ``False``, ``None`` are reserved words 132 | 133 | ----- 134 | 135 | Exceptions 136 | ---------- 137 | 138 | Simplified ``except``, thus only allowed constructions are: 139 | 140 | .. code-block:: python 141 | 142 | except ValueError as e: 143 | pass 144 | 145 | except (ValueError, TypeError) as e: 146 | pass 147 | 148 | Simplified ``raise`` statement: "``raise Exception(args)``" 149 | 150 | ---- 151 | 152 | Others 153 | ------ 154 | * ``long`` and ``int`` types were merged 155 | * updated integer literals 156 | * library changes 157 | * applied naming conventions 158 | * "``%``" string formatting is deprecated, use the ``format`` function: 159 | ``"The story of {0}, {1}, and {c}".format(a, b, c=d)`` 160 | 161 | http://dev.pocoo.org/~gbrandl/py3 162 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/concurrency.rst: -------------------------------------------------------------------------------- 1 | Concurrency 2 | =========== 3 | 4 | ------------------------------------------------------------ 5 | 6 | Basic concepts 7 | --------------- 8 | 9 | * multitasking 10 | * simultaneous task execution 11 | * CPU bound tasks 12 | * I/O bound tasks 13 | * shared memory 14 | * threads 15 | * processes 16 | * distributed computing 17 | 18 | ------------------------------------------------------------ 19 | 20 | Things to consider 21 | 22 | * Performance of the 23 | * application 24 | * programmer 25 | 26 | * whether your problem is CPU or IO bound 27 | 28 | ------------------------------------------------------------ 29 | 30 | ``threading`` 31 | ------------- 32 | 33 | https://docs.python.org/3/library/threading.html 34 | 35 | "..this module is loosely based on Java’s threading model..." 36 | 37 | ------------------------------------------------------------ 38 | 39 | ``Thread`` class 40 | * ``is_alive()`` 41 | * ``join()`` 42 | * ``start()`` 43 | * ``run()`` # Contains the implementation of the thread 44 | * ``setDaemon()`` 45 | 46 | ------------------------------------------------------------ 47 | 48 | .. code-block:: python 49 | 50 | import threading 51 | 52 | class MyThread(threading.Thread) 53 | def run(self): 54 | """thread worker function""" 55 | print('Worker') 56 | 57 | t = MyThread() 58 | t.start() 59 | 60 | 61 | ------------------------------------------------------------ 62 | 63 | .. code-block:: python 64 | 65 | import threading 66 | 67 | def worker(): 68 | """thread worker function""" 69 | print('Worker') 70 | 71 | t = threading.Thread(target=worker) 72 | t.start() 73 | 74 | ------------------------------------------------------------ 75 | 76 | Accessing to shared data 77 | -------------------------- 78 | 79 | Thread synchronization primitives: 80 | 81 | * Lock 82 | * RLcok 83 | * Sempahore 84 | * Event 85 | * Condition 86 | 87 | ------------------------------------------------------------ 88 | 89 | ``Lock`` 90 | --------- 91 | 92 | Mutex exclusion lock 93 | 94 | .. code-block:: python 95 | 96 | x_lock = threading.Lock() 97 | 98 | x_lock.acquire() 99 | # ..critical section 100 | x_lock.relase() 101 | 102 | ------------------------------------------------------------ 103 | 104 | ... or with context managers: 105 | 106 | 107 | .. code-block:: python 108 | 109 | x_lock = threading.Lock() 110 | 111 | with x_lock: 112 | # critical section 113 | 114 | ------------------------------------------------------------ 115 | 116 | ``RLock`` 117 | --------- 118 | 119 | Reentrant mutex lock: 120 | 121 | * it can be reacquired multiple times **by the same thread** 122 | * only one thread is allowed to execute 123 | 124 | ------------------------------------------------------------ 125 | 126 | ``Semaphore`` 127 | ------------ 128 | 129 | * ``acquire()`` -- wait if the count is 0 130 | * ``relase()`` -- increment the count 131 | * they can be called in any order by any thread 132 | 133 | ------------------------------------------------------------ 134 | 135 | ``Event`` 136 | --------- 137 | 138 | Used for notifying threads about an event. 139 | 140 | * Easy wating instead of busy loops 141 | * Broadcasting massages 142 | 143 | .. code-block:: python 144 | 145 | e = threading.Event() 146 | e.isSet() # Returns True if the event is True 147 | e.set() # Sets event to True 148 | e.clear() # Sets event ot False 149 | e.wait() # Waits until event becomes True 150 | 151 | ------------------------------------------------------------ 152 | 153 | ``Condition`` 154 | -------------- 155 | 156 | Is a combination of locking and signalling 157 | 158 | .. code-block:: python 159 | 160 | cv = threading.Condition() # create a condition with an optional lock parameter 161 | cv.acquire() # Acquire the underlying lock 162 | cv.release() # Release the underlying lock 163 | cv.wait() # Wait for condition 164 | cv.notify(n=1) # Wake up one thread witing on this condition 165 | cv.notifyAll() # Wake up all threads waiting on this condition 166 | 167 | ------------------------------------------------------------ 168 | 169 | .. code-block:: python 170 | 171 | """Producer thread""" 172 | # ... generate item 173 | condition.acquire() 174 | # ... add item to resource 175 | condition.notify() # signal that a new item is available 176 | condition.release() 177 | 178 | """Consumer thread""" 179 | condition.acquire() 180 | while True: 181 | # ... get item from resource 182 | if item: 183 | break 184 | condition.wait() # sleep until item becomes available 185 | condition.release() 186 | # ... process item 187 | 188 | ------------------------------------------------------------ 189 | 190 | ... but 191 | ------- 192 | 193 | * deadlocks are easy to create 194 | * non-deterministic scheduling 195 | * ... 196 | 197 | ------------------------------------------------------------ 198 | 199 | ``Queue`` 200 | -------------- 201 | 202 | * **thread safe** 203 | * Instead of sharing the memory, you can send data between threads. 204 | * Queues are a convenient way to organize applications in a producer/consumer model. 205 | 206 | ------------------------------------------------------------ 207 | 208 | Usage: 209 | ~~~~~~ 210 | 211 | .. code-block:: python 212 | 213 | from Queue import Queue 214 | q = Queue([maxsize]) # Create a queue 215 | q.put(item) # Put an item on the queue 216 | q.get() # Get an item from the queue 217 | q.empty() # Check if empty 218 | q.full() # Check if full 219 | 220 | q.task_done() # Signal that work is done 221 | q.join() # Wait for all work to be done 222 | 223 | ------------------------------------------------------------ 224 | 225 | Global Interpreter Lock 226 | ------------------------ 227 | 228 | **But Python has a Global Interpreter Lock (GIL)** 229 | * thus only one thread can be executed at the same time 230 | * not efficient on CPU intense tasks 231 | * however, it can only help on I/O bounded tasks 232 | * threads are necessary for GUI apps 233 | 234 | ------------------------------------------------------------ 235 | 236 | * A thread is a real system thread 237 | * managed by the host OS 238 | * only one python thread can be executed in the interpreter 239 | * whenever a thread runs it holds the GIL 240 | 241 | ------------------------------------------------------------ 242 | 243 | Sulutions 244 | --------- 245 | 246 | * message passing: 247 | 1. Create several Python applications 248 | 2. Implement a Message passing protocol 249 | 250 | * use processes from the ``multiprocessing`` module 251 | 252 | ------------------------------------------------------------ 253 | 254 | ``multiprocessing`` 255 | -------------------- 256 | 257 | * The ``multiprocessing`` module provides access to processes instead threads. 258 | * Meaning, that there are no shared memory spaces! 259 | * You can use the **same API shown for threads** 260 | 261 | ------------------------------------------------------------ 262 | 263 | .. code-block:: python 264 | 265 | import multiprocessing 266 | 267 | class MyProcess(multiprocessing.Process): 268 | def run(self): 269 | print("Hello") 270 | 271 | Easy way: 272 | 273 | .. code-block:: python 274 | 275 | import multiprocessing 276 | 277 | def worker(): 278 | """thread worker function""" 279 | print('Worker') 280 | 281 | t = multiprocessing.Process(target=worker) 282 | t.start() 283 | 284 | 285 | ------------------------------------------------------------ 286 | 287 | 288 | 289 | ``Pipe`` 290 | --------- 291 | 292 | Pipes are a pair of connection objects for sending and receiving objects 293 | 294 | ``(c1, c2) = multiprocessing.Pipe()`` 295 | 296 | 297 | .. code-block:: python 298 | 299 | c.send(obj) # Send an object 300 | c.recv() # Receive an object 301 | c.send_bytes(buffer) # Send a buffer of bytes 302 | c.recv_bytes([max]) # Receive a buffer of bytes 303 | c.poll([timeout]) # Check for data 304 | 305 | ------------------------------------------------------------ 306 | 307 | .. code-block:: python 308 | 309 | import multiprocessing 310 | 311 | def consumer(p1, p2): 312 | x = p2.recv() 313 | print(x) 314 | 315 | p1, p2 = multiprocessing.Pipe() 316 | cons = multiprocessing.Process( 317 | target=consumer, 318 | args=(p1, p2) 319 | ) 320 | cons.start() 321 | item = "alma" 322 | p1.send(item) 323 | p1.close() 324 | cons.join() 325 | 326 | ------------------------------------------------------------ 327 | 328 | ``Queue`` 329 | --------- 330 | 331 | * Processes can be also used with queues. 332 | * They are implemented on top of pipes 333 | 334 | .. code-block:: python 335 | 336 | from multiprocessing import Queue 337 | q = Queue() 338 | q.put(item) # Put an item on the queue 339 | item = q.get() # Get an item from the queue 340 | 341 | ------------------------------------------------------------ 342 | 343 | ``Pool`` 344 | ~~~~~~~~ 345 | 346 | "One can create a pool of processes which will carry out tasks submitted to it with the Pool class." 347 | 348 | Processes can be started using: 349 | 350 | * ``apply(func[, args[, kwds]])`` 351 | * ``apply_async(func[, args[, kwds[, callback[, error_callback]]]])`` 352 | * ``map(func, iterable[, chunksize])`` 353 | * ``map_async(func, iterable[, chunksize[, callback[, error_callback]]])`` 354 | 355 | Async methods yields an ``AsyncResult`` object 356 | 357 | ------------------------------------------------------------ 358 | 359 | .. code-block:: python 360 | 361 | from multiprocessing.pool import Pool 362 | 363 | def f(x): 364 | print(x) 365 | return x*x 366 | 367 | with Pool(processes=4) as pool: 368 | result = pool.apply_async(f, (10,)) 369 | print(result.get(timeout=1)) 370 | 371 | print(pool.map(f, range(10))) 372 | 373 | res = pool.map_async(f, range(10)) 374 | pool.close() 375 | pool.join() 376 | print(res.get()) 377 | 378 | ------------------------------------------------------------ 379 | 380 | 381 | ``concurrent.futures`` 382 | ---------------------- 383 | 384 | http://python.org/dev/peps/pep-3148/ 385 | 386 | "provides a high-level interface for asynchronously executing callables ... using": 387 | 388 | * Executors 389 | * ``ThreadPoolExecutor`` vs ``ProcessPoolExecutor`` 390 | * ``submit, map`` is used for starting execution 391 | 392 | ------------------------------------------------------------ 393 | 394 | * ``Future`` objects 395 | * is returned when a process/thread is executed 396 | * it can be used to monitor the execution 397 | * ``cancel, done, result, exception, add_done_callback`` 398 | 399 | 400 | ------------------------------------------------------------ 401 | 402 | .. code-block:: python 403 | 404 | import concurrent.futures 405 | CORES = 4 406 | 407 | def hard(): 408 | print(99*99) 409 | 410 | monitors = [] 411 | with concurrent.futures.ProcessPoolExecutor( 412 | max_workers=CORES) as ex: 413 | for num in range(CORES): 414 | f = ex.submit(hard) 415 | monitors.append(f) 416 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/decorators.rst: -------------------------------------------------------------------------------- 1 | Decorators 2 | -------------- 3 | 4 | ------------------------------------------------------------ 5 | 6 | 7 | Higher order functions 8 | ~~~~~~~~~~~~~~~~~~~~~~~ 9 | 10 | .. code-block:: python 11 | 12 | def my_higher_order() 13 | def myfun(x,y): 14 | return x,y 15 | 16 | return myfun 17 | 18 | f = my_higher_order() 19 | f(1,2) #3 20 | 21 | my_higher_order()(1,2) # 3 22 | 23 | ------------------------------------------------------------ 24 | 25 | Let's make ``addn`` functions: 26 | 27 | .. code-block:: python 28 | 29 | def get_adder(n) 30 | return lambda x: x+n 31 | 32 | add2 = get_adder(2) 33 | print(add2(1)) #3 34 | 35 | Make a linear *function* factory: 36 | 37 | .. code-block:: python 38 | 39 | def linear(a,b): 40 | def result(x): return a*x + b 41 | return result 42 | 43 | line_21 = linear(2,1) 44 | print(line_21(5)) # 11 45 | 46 | 47 | 48 | ------------------------------------------------------------ 49 | 50 | Closures 51 | ~~~~~~~~ 52 | 53 | * an inner function remembers its namespace (as above) 54 | 55 | .. code-block:: python 56 | 57 | def outer(x): 58 | def inner(): 59 | # x is part of the outer's namespace 60 | print(x) 61 | return inner 62 | 63 | print1 = outer(1) 64 | print2 = outer(2) 65 | print1() # 1 66 | print2() # 2 67 | 68 | ------------------------------------------------------------ 69 | 70 | 71 | Decorators 72 | ~~~~~~~~~~ 73 | 74 | https://docs.python.org/3/glossary.html#term-decorator 75 | 76 | * is a *callable* that takes a function as an argument and return one as replacement 77 | * usually applied with the ``@wrapper`` syntax 78 | 79 | ------------------------------------------------------------ 80 | 81 | .. code-block:: python 82 | 83 | def makebold(fn): 84 | def wrapped(): 85 | return "" + fn() + "" 86 | return wrapped 87 | 88 | def makeitalic(fn): 89 | def wrapped(): 90 | return "" + fn() + "" 91 | return wrapped 92 | 93 | @makebold 94 | @makeitalic 95 | def hello(): 96 | return "hello world" 97 | 98 | print(hello()) 99 | # hello world 100 | 101 | ------------------------------------------------------------ 102 | 103 | .. code-block:: python 104 | 105 | def bold(fun): 106 | def wrapped(*args, **kwargs): 107 | res = fun(*args, **kwargs) 108 | return "{}".format(res) 109 | return wrapped 110 | 111 | 112 | @bold 113 | def myfun(out1, out2): 114 | return out1 + " " + out2 115 | 116 | 117 | print(myfun("alma", out2="szilva")) 118 | 119 | ------------------------------------------------------------ 120 | 121 | Example decorators from the std. lib. 122 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 123 | 124 | Builtins: 125 | 126 | * ``classmethod`` 127 | * ``staticmethod`` 128 | * ``contextlib.contextmanager`` 129 | * ``functools.*`` 130 | * Properties 131 | 132 | Others examples: https://wiki.python.org/moin/PythonDecoratorLibrary 133 | 134 | ------------------------------------------------------------ 135 | 136 | 137 | ``functools`` 138 | ^^^^^^^^^^^^^^ 139 | 140 | * ``total_ordering(cls)`` -- "Given a class defining one or more rich comparison ordering methods, this class decorator supplies the rest." 141 | * ``functools.wraps(wrapped[, assigned][, updated])`` 142 | 143 | .. code-block:: python 144 | 145 | def makebold(fn): 146 | @wraps(fn) 147 | def wrapped(): 148 | return "" + fn() + "" 149 | return wrapped 150 | 151 | 152 | ------------------------------------------------------------ 153 | 154 | 155 | .. code-block:: python 156 | 157 | from contextlib import contextmanager 158 | 159 | @contextmanager 160 | def tag(name): 161 | print "<%s>" % name 162 | yield 163 | print "" % name 164 | 165 | >>> with tag("h1"): 166 | ... print "foo" 167 | ... 168 |

169 | foo 170 |

171 | 172 | ------------------------------------------------------------ 173 | 174 | 175 | Properties 176 | ^^^^^^^^^^ 177 | 178 | .. code-block:: python 179 | 180 | class C(object): 181 | def __init__(self): 182 | self._x = None 183 | 184 | def getx(self): 185 | return self._x 186 | 187 | def setx(self, value): 188 | self._x = value 189 | 190 | def delx(self): 191 | del self._x 192 | 193 | x = property(getx, setx, delx, "I'm the 'x' property.") 194 | 195 | ----------------------------------------- 196 | 197 | .. code-block:: python 198 | 199 | class C(object): 200 | def __init__(self): 201 | self._x = None 202 | 203 | @property 204 | def x(self): 205 | """I'm the 'x' property.""" 206 | return self._x 207 | 208 | @x.setter 209 | def x(self, value): 210 | self._x = value 211 | 212 | @x.deleter 213 | def x(self): 214 | del self._x 215 | 216 | ----------------------------------------- 217 | 218 | Class decorators 219 | ~~~~~~~~~~~~~~~~ 220 | 221 | * a class decorator is a callable that takes a class and returns a class 222 | 223 | .. code-block:: python 224 | 225 | def my_dec(cls): 226 | cls.y = 0 227 | return cls 228 | 229 | @my_dec 230 | class A(object): pass 231 | 232 | print A.y # 0 233 | 234 | 235 | ------------------------------------------------------------ 236 | 237 | .. code-block:: python 238 | 239 | class MyDec(object): 240 | def __new__(self, cls): 241 | cls.x = 1 242 | return cls 243 | 244 | 245 | @MyDec 246 | class A(object): pass 247 | 248 | print A.x # 1 249 | 250 | 251 | ------------------------------------------------------------ 252 | 253 | Class of decorators 254 | ~~~~~~~~~~~~~~~~~~~~ 255 | 256 | .. code-block:: python 257 | 258 | class Add(): 259 | def __init__(self, num): 260 | self.__num = num 261 | 262 | def __call__(self, fun): 263 | return lambda x : fun(x) + self.__num 264 | 265 | @Add(1) 266 | @Add(2) 267 | def pow2(x): 268 | return x*x 269 | 270 | print(pow2(2)) # 7 271 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/iterators.rst: -------------------------------------------------------------------------------- 1 | Iterators 2 | --------- 3 | 4 | ------------------------------------------------------------ 5 | 6 | Iterator type 7 | ~~~~~~~~~~~~~ 8 | 9 | An iterator is an object implementing the iterator protocol: 10 | https://docs.python.org/3/library/stdtypes.html#iterator-types 11 | 12 | * allows to loop just once 13 | * ``__next__() / next()`` -- get next element 14 | * ``StopIteration`` is raised when the iteration is over 15 | 16 | ------------------------------------------------------------ 17 | 18 | .. code-block:: python 19 | 20 | >>> nums = [1,2,3] 21 | >>> iter(nums) 22 | 23 | >>> nums.__iter__() 24 | 25 | >>> it = iter(nums) 26 | >>> next(it) # next(obj) simply calls obj.next() 27 | 1 28 | >>> it.__next__() 29 | 2 30 | >>> next(it) 31 | 3 32 | >>> next(it) 33 | Traceback (most recent call last): 34 | File "", line 1, in 35 | StopIteration 36 | 37 | ------------------------------------------------------------ 38 | 39 | ``for`` can be used to iterate through an iterator 40 | 41 | .. code-block:: python 42 | 43 | >>> for f in iter([1,2,3]): print(f) 44 | ... 45 | 1 46 | 2 47 | 3 48 | 49 | All sequences and container types implements this protocol: 50 | 51 | .. code-block:: python 52 | 53 | >>> f = open('/etc/fstab') 54 | >>> f is f.__iter__() 55 | True 56 | 57 | ------------------------------------------------------------ 58 | 59 | 60 | Generators 61 | ~~~~~~~~~~ 62 | 63 | http://docs.python.org/3/tutorial/classes.html#generators 64 | 65 | Generator functions allow to declare a function that behaves like an iterator 66 | 67 | * In practice, "a generator is a function containing the keyword ``yield``." 68 | * when it is called, the instructions are executed till the the next ``yield`` 69 | * cf. PEP 250 70 | 71 | ------------------------------------------------------------ 72 | 73 | .. code-block:: python 74 | 75 | >>> def f(): 76 | ... yield 3 77 | ... yield 4 78 | >>> gen = f() 79 | >>> next(gen) 80 | 3 81 | >>> next(gen) 82 | 4 83 | >>> next(gen) 84 | -- finished -- 85 | Traceback (most recent call last): 86 | ... 87 | StopIteration 88 | 89 | 90 | ------------------------------------------------------------ 91 | 92 | And your code can be more effective: 93 | 94 | .. code-block:: python 95 | 96 | def firstn(n): 97 | num, nums = 0, [] 98 | while num < n: 99 | nums.append(num) 100 | num += 1 101 | return nums 102 | 103 | def improved_firstn(n): 104 | num = 0 105 | while num < n: 106 | yield num 107 | num += 1 108 | 109 | ------------------------------------------------------------ 110 | 111 | Generator expressions 112 | ~~~~~~~~~~~~~~~~~~~~~ 113 | 114 | https://docs.python.org/3/tutorial/classes.html#generator-expressions 115 | 116 | The syntax is similar to the list comprehension but will use parentheses 117 | 118 | .. code-block:: python 119 | 120 | genpow = (i*i for i in range(10)) 121 | 122 | """ 123 | for x in genpow: 124 | if x > 100: 125 | print(x) 126 | break 127 | """ 128 | 129 | ------------------------------------------------------------ 130 | 131 | Advanced generator topics 132 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 133 | 134 | * `Bidirectional communication `_ 135 | 136 | * `Chaining generators `_ 137 | 138 | ------------------------------------------------------------ 139 | 140 | 141 | Bidirectional communication 142 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 143 | 144 | .. code-block:: python 145 | 146 | def test_gen(): 147 | k = 1 148 | s = 0 149 | for i in range(10): 150 | s += k 151 | k_ = yield s 152 | k = k_ or k 153 | 154 | g = test_gen() 155 | print(next(g)) 156 | print(next(g)) 157 | g.send(10) 158 | print(next(g)) 159 | 160 | 161 | ------------------------------------------------------------ 162 | 163 | 164 | 165 | Iterators in the std. lib. 166 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 167 | 168 | * ``range()`` 169 | * ``map()`` 170 | * ``filter()`` 171 | * ``reversed()`` 172 | * ``zip()`` 173 | * ``enumerate()`` 174 | 175 | ------------------------------------------------------------ 176 | 177 | 178 | The ``itertools`` module 179 | ~~~~~~~~~~~~~~~~~~~~~~~~ 180 | 181 | "The module standardizes a core set of fast, memory efficient tools that are useful by themselves or in combination" 182 | 183 | https://docs.python.org/3/library/itertools.html 184 | 185 | ------------- 186 | 187 | .. code-block:: python 188 | 189 | 190 | count(10) # 10, 11, 12, ... 191 | cycle([1,2,3]) # 1 2 3 1 2 ... 192 | repeat('a', 3) # a a a 193 | 194 | takewhile(lambda x: x<5, [1,4,6,4,1]) # 1 4 195 | dropwhile(lambda x: x<5, [1,4,6,4,1]) # 6 4 1 196 | chain('ABC', 'DEF') # A B C D E F 197 | accumulate([1,2,3,4,5]) # 1 3 6 10 15 198 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/metaclass.rst: -------------------------------------------------------------------------------- 1 | Other advanced concepts 2 | ------------------------ 3 | 4 | ------------------------------------------------------------------------- 5 | 6 | 7 | Python types 8 | ~~~~~~~~~~~~ 9 | 10 | * everything is an object 11 | 12 | 13 | .. code-block:: python 14 | >>> class A: pass 15 | >>> type(A) 16 | 17 | >>> A.__class__ 18 | 19 | 20 | * classes are first-class objects 21 | * => can be created / deleted anytime 22 | * ``type`` 23 | * is the class of all python classes 24 | * ``print A.__class__ # `` 25 | * is a (callable) object 26 | * ``MyClass=type(name, bases, dict)`` 27 | 28 | ------------------------------------------------------------------------- 29 | 30 | Metaclasses 31 | ~~~~~~~~~~~ 32 | 33 | Is the class of a class. 34 | 35 | * metaclasses create classes, while classes create instances 36 | * the default metaclass of classes is the ``type`` 37 | 38 | In practice, a class is created: 39 | * instantiating a class using metaclass available through the ``__metaclass__`` attribute 40 | * or calling the builtin ``type(name, bases, dct)`` method 41 | 42 | ------------------------------------------------------------------------- 43 | 44 | Call ``type()`` with three parameters return a new class: 45 | 46 | .. code-block:: python 47 | 48 | B = type("B", (object,), 49 | {"f": lambda self,x : x+1}) 50 | b = B() 51 | b.f(1) # 2 52 | 53 | ------------------------------------------------------------------------- 54 | 55 | One can implement a metaclass, inheriting form the ``type`` class 56 | 57 | .. code-block:: python 58 | 59 | class CustomMetaclass(type): 60 | def __init__(cls, name, bases, dct): 61 | print "Creating class %s using CustomMetaclass" 62 | % name 63 | super(CustomMetaclass, cls).__init__(name, bases, dct) 64 | 65 | class BaseClass(metaclass = CustomMetaclass): pass 66 | 67 | class Subclass1(BaseClass): pass 68 | 69 | # Creating class BaseClass using CustomMetaclass 70 | # Creating class Subclass1 using CustomMetaclass 71 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/pep.rst: -------------------------------------------------------------------------------- 1 | Python Enhancement Proposals (PEPs) 2 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3 | 4 | ------------------------------------------------------------------------- 5 | 6 | **Guido van Rossum** is known as a "Benevolent Dictator For Life" (BDFL), meaning that he continues to oversee the Python development process, making decisions where necessary. 7 | 8 | .. image:: guido.jpg 9 | ------------------------------------------------------------------------- 10 | 11 | "A PEP is a design document providing information to the Python community, or describing a new feature for Python or its processes or environment ... " 12 | 13 | "... (are) the primary mechanisms for proposing major new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python." 14 | 15 | "...maintained as text files in a versioned repository, their revision history is the historical record of the feature proposal..." 16 | 17 | * https://www.python.org/dev/peps/ 18 | * https://wiki.python.org/moin/Topically%20Organized%20PEP%20List 19 | 20 | 21 | ------------------------------------------------------------------------- 22 | 23 | PEP20 -- The Zen of Python 24 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 25 | 26 | .. code-block:: none 27 | 28 | Beautiful is better than ugly. 29 | Explicit is better than implicit. 30 | Simple is better than complex. 31 | Complex is better than complicated. 32 | Flat is better than nested. 33 | Sparse is better than dense. 34 | Readability counts. 35 | Special cases aren't special enough to break the rules. 36 | Although practicality beats purity. 37 | Errors should never pass silently. 38 | Unless explicitly silenced. 39 | 40 | ------------------------------------------------------------------------- 41 | 42 | .. code-block:: none 43 | 44 | In the face of ambiguity, refuse the temptation to guess. 45 | There should be one-- and preferably only one --obvious way to do it. 46 | Although that way may not be obvious at first unless you're Dutch. 47 | Now is better than never. 48 | Although never is often better than *right* now. 49 | If the implementation is hard to explain, it's a bad idea. 50 | If the implementation is easy to explain, it may be a good idea. 51 | Namespaces are one honking great idea -- let's do more of those! 52 | 53 | 54 | ------------------------------------------------------------------------- 55 | 56 | PEP8 57 | ^^^^^ 58 | 59 | *Style Guide for Python code* 60 | 61 | * Indentation: 4 spaces 62 | * Line length: 79 chars 63 | * Blank lines: 2 for top level methods class and 1 for methods inside classes 64 | * Encoding: UTF-8 / ASCII 65 | * separate ``import`` statements 66 | 67 | ------------------------------------------------------------------------- 68 | 69 | * Naming conventions: 70 | * ``do_something(var_one, var_two)`` 71 | * ``ClassName`` 72 | * ``CONSTANT_VAR`` 73 | * etc. 74 | 75 | A tool for checking your code: https://pypi.python.org/pypi/pep8 76 | 77 | ------------------------------------------------------------------------- 78 | 79 | Docstrings 80 | ^^^^^^^^^^ 81 | `PEP 257 `_ (and some others) 82 | 83 | "A docstring is a **string literal that occurs as the first statement** in a *module, function, class, or method* definition. Such a docstring becomes the **__doc__** special attribute of that object." 84 | 85 | 86 | ------------------------------------------------------------------------- 87 | 88 | One-line docstring 89 | ''''''''''''''''''' 90 | 91 | * one sentence 92 | * using triple quotes (``"""``) 93 | * no newline 94 | 95 | .. code-block:: python 96 | 97 | def kos_root(): 98 | """Return the pathname of the KOS root directory.""" 99 | global _kos_root 100 | if _kos_root: return _kos_root 101 | 102 | ------------------------------------------------------------------------- 103 | 104 | Multi-line docstring 105 | '''''''''''''''''''' 106 | 107 | Is composed of a summary, a separating new line, and optional lines describing parameters, etc. 108 | 109 | .. code-block:: python 110 | 111 | def complex(real=0.0, imag=0.0): 112 | """Form a complex number. 113 | 114 | Keyword arguments: 115 | real -- the real part (default 0.0) 116 | imag -- the imaginary part (default 0.0) 117 | """ 118 | if imag == 0.0 and real == 0.0: 119 | return complex_zero 120 | ... 121 | 122 | ------------------------------------------------------------------------- 123 | 124 | Recommended readings 125 | ^^^^^^^^^^^^^^^^^^^^^ 126 | 127 | * https://google-styleguide.googlecode.com/svn/trunk/pyguide.html 128 | * http://docs.python-guide.org/en/latest/writing/style/ 129 | 130 | ------------------------------------------------------------------------- 131 | 132 | ``doctest`` 133 | ^^^^^^^^^^^^ 134 | 135 | Lets you test your code by running examples embedded in your docstrings. 136 | 137 | It evaluates statements started with ``" >>>"`` and compares their results with the next lines. 138 | 139 | .. code-block:: python 140 | 141 | def multiply(a, b): 142 | """ 143 | >>> multiply(4, 3) 144 | 12 145 | >>> multiply('a', 3) 146 | 'aaa' 147 | """ 148 | return a * b 149 | 150 | ------------------------------------------------------------------------- 151 | 152 | Running doctests 153 | ''''''''''''''''' 154 | 155 | .. code-block:: bash 156 | 157 | $ python3 -m doctest mydoctest.py 158 | $ python3 -m doctest -v mydoctest.py # verbose 159 | 160 | or 161 | 162 | .. code-block:: python 163 | 164 | import doctest 165 | 166 | """Your code""" 167 | 168 | if __name__ == "__main__": 169 | doctest.testmod() 170 | 171 | ------------------------------------------------------------------------- 172 | 173 | Context managers 174 | ~~~~~~~~~~~~~~~~~ 175 | 176 | A context manager (`PEP 343 `_) is an object with ``__enter__`` and ``__exit__`` methods which can be used in the with statement. It permits e.g. the extraction of the exception handling structure into a class. 177 | 178 | ------------------------------------------------------------------------- 179 | 180 | .. code-block:: python 181 | 182 | with manager as var: 183 | do_something(var) 184 | 185 | # is in the simplest case equivalent to 186 | 187 | var = manager.__enter__() 188 | try: 189 | do_something(var) 190 | finally: 191 | manager.__exit__() 192 | 193 | ------------------------------------------------------------------------- 194 | 195 | Context managers are implemented in the std. library in several places, such as 196 | * ``file`` 197 | * ``ftplib`` 198 | * ``tempfile``, etc. 199 | 200 | An example for files: 201 | 202 | .. code-block:: python 203 | 204 | with open('/tmp/file', 'a') as f: 205 | # the file gets closed whatever happens 206 | f.write('more contents\n') 207 | 208 | ------------------------------------------------------------------------- 209 | 210 | Type hints 211 | ~~~~~~~~~~ 212 | 213 | https://www.python.org/dev/peps/pep-0484/ 214 | 215 | .. code-block:: python 216 | 217 | def greeting(name: str) -> str: 218 | return 'Hello ' + name 219 | 220 | ------------------------------------------------------------------------- 221 | 222 | .. code-block:: python 223 | 224 | Url = str 225 | def retry(url: Url, retry_count: int) -> None: 226 | pass 227 | 228 | 229 | 230 | from typing import TypeVar, Iterable, Tuple 231 | 232 | T = TypeVar('T', int, float, complex) 233 | Vector = Iterable[Tuple[T, T]] 234 | 235 | def inproduct(v: Vector) -> T: 236 | return sum(x*y for x, y in v) 237 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/persistance.rst: -------------------------------------------------------------------------------- 1 | Data persistance 2 | ---------------- 3 | 4 | ----------------------------------------------------- 5 | 6 | ``pickle`` 7 | ------------ 8 | 9 | The pickle module implements binary protocols for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation. 10 | 11 | ----------------------------------------------------- 12 | 13 | What can be pickled? 14 | ~~~~~~~~~~~~~~~~~~~~ 15 | 16 | * ``None``, ``True``, and ``False`` 17 | * numbers 18 | * strings, bytes, bytearrays 19 | * functions defined at the top level of a module (using def, not lambda) 20 | * built-in functions defined at the top level of a module 21 | * classes that are defined at the top level of a module 22 | 23 | ----------------------------------------------------- 24 | 25 | * tuples, lists, sets, and dictionaries containing only picklable objects 26 | * instances of such classes whose ``__dict__`` or the result of calling ``__getstate__()`` is picklable 27 | 28 | ----------------------------------------------------- 29 | 30 | How? 31 | ~~~~ 32 | 33 | .. code-block:: python 34 | 35 | pickle.dump(obj, file) 36 | pickle.load(file) 37 | 38 | bytes_object = pickle.dumps(obj) 39 | pickle.loads(bytes_object) 40 | 41 | ----------------------------------------------------- 42 | 43 | .. code-block:: python 44 | 45 | import pickle 46 | 47 | # An arbitrary collection of objects supported by pickle. 48 | data = { 49 | 'a': [1, 2.0, 3, 4+6j], 50 | 'b': ("character string", b"byte string"), 51 | 'c': {None, True, False} 52 | } 53 | 54 | with open('data.pickle', 'wb') as f: 55 | pickle.dump(data, f) 56 | 57 | ----------------------------------------------------- 58 | 59 | .. code-block:: python 60 | 61 | import pickle 62 | 63 | with open('data.pickle', 'rb') as f: 64 | # The protocol version used is detected automatically, so we do not 65 | # have to specify it. 66 | data = pickle.load(f) 67 | ----------------------------------------------------- 68 | 69 | ``shelve`` 70 | ------------ 71 | 72 | A “shelf” is a persistent, dictionary-like object. 73 | 74 | * backed by ``pickle`` 75 | * keys are ordinally strings 76 | * the *values* in a shelf can be essentially arbitrary Python objects — anything that the pickle module can handle 77 | 78 | ----------------------------------------------------- 79 | 80 | * works like an ordinary dictionary 81 | * implements context manager 82 | 83 | .. code-block:: python 84 | 85 | with shelve.open('spam') as db: 86 | db['eggs'] = 'eggs' 87 | 88 | 89 | ----------------------------------------------------- 90 | 91 | ``marshal`` 92 | ------------ 93 | 94 | It contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues. 95 | 96 | * Udocumented on purpose. 97 | * This is not a general “persistence” module. 98 | * The marshal module exists mainly to support reading and writing the “pseudo-compiled” code for Python modules of .pyc files. 99 | 100 | ----------------------------------------------------- 101 | 102 | JSON support 103 | ------------ 104 | 105 | Supports a ``pickle`` like serialization and deserialization of objects to JSON format. 106 | 107 | 108 | .. code-block:: python 109 | 110 | import json 111 | 112 | json.dump(obj, file) 113 | json_dict = json.load(file) 114 | 115 | formatted_string = json.dumps(obj) 116 | json_dict = json.loads(formatted_string) 117 | 118 | ----------------------------------------------------- 119 | 120 | CSV support 121 | ------------ 122 | 123 | The csv module implements classes to read and write tabular data in CSV format. 124 | 125 | ----------------------------------------------------- 126 | 127 | .. code-block:: python 128 | 129 | import csv 130 | 131 | with open('eggs.csv', 'w', newline='') as csvfile: 132 | spamwriter = csv.writer(csvfile, delimiter=' ', 133 | quotechar='|') 134 | spamwriter.writerow(['Spam'] * 5 + ['Baked Beans']) 135 | spamwriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam']) 136 | 137 | with open('eggs.csv', newline='') as csvfile: 138 | spamreader = csv.reader(csvfile, 139 | delimiter=' ', 140 | quotechar='|') 141 | for row in spamreader: 142 | print(', '.join(row)) 143 | # Spam, Spam, Spam, Spam, Spam, Baked Beans 144 | # Spam, Lovely Spam, Wonderful Spam 145 | 146 | 147 | ----------------------------------------------------- 148 | 149 | .. code-block:: python 150 | 151 | import csv 152 | with open('names.csv', 'w') as csvfile: 153 | fieldnames = ['first_name', 'last_name'] 154 | writer = csv.DictWriter(csvfile, 155 | fieldnames=fieldnames) 156 | 157 | writer.writeheader() 158 | writer.writerow({'first_name': 'Baked', 159 | 'last_name': 'Beans'}) 160 | writer.writerow({'first_name': 'Lovely', 161 | 'last_name': 'Spam'}) 162 | 163 | with open('names.csv') as csvfile: 164 | reader = csv.DictReader(csvfile) 165 | for row in reader: 166 | print(row['first_name'], row['last_name']) 167 | 168 | 169 | 170 | ----------------------------------------------------- 171 | 172 | XML support 173 | ------------ 174 | 175 | * ``xml.dom`` DOM API definition 176 | * ``xml.sax`` SAX2 classes 177 | * ``xml.etree`` ElementTree API, a simple and lightweight XML processor 178 | 179 | https://docs.python.org/3.5/library/xml.etree.elementtree.html#module-xml.etree.ElementTree 180 | 181 | ----------------------------------------------------- 182 | 183 | DOM example 184 | ~~~~~~~~~~~ 185 | 186 | .. code-block:: python 187 | 188 | from xml.dom import minidom 189 | 190 | doc = minidom.parse("staff.xml") 191 | # doc.getElementsByTagName returns NodeList 192 | name = doc.getElementsByTagName("name")[0] 193 | staffs = doc.getElementsByTagName("staff") 194 | for staff in staffs: 195 | sid = staff.getAttribute("id") 196 | nickname = staff.getElementsByTagName("nickname")[0] 197 | salary = staff.getElementsByTagName("salary")[0] 198 | 199 | ----------------------------------------------------- 200 | 201 | SAX example 202 | ~~~~~~~~~~~~ 203 | 204 | .. code-block:: python 205 | 206 | import xml.sax 207 | 208 | class InkscapeSvgHandler(xml.sax.ContentHandler): 209 | def startElement(self, name, attrs): 210 | if name == "svg": 211 | for (k,v) in attrs.items(): 212 | print k + " " + v 213 | 214 | parser = xml.sax.make_parser() 215 | parser.setContentHandler(InkscapeSvgHandler()) 216 | parser.parse(open("svg.xml","r")) 217 | -------------------------------------------------------------------------------- /AdvancedPython/StandardLibrary/testing.rst: -------------------------------------------------------------------------------- 1 | Testing 2 | ------- 3 | 4 | ------------------------------------------------------------ 5 | 6 | ``assert`` 7 | ~~~~~~~~~~ 8 | 9 | * you can insert debugging assertions to any program 10 | 11 | ``assert expression_to_check [, expression_to_raise]`` 12 | 13 | .. code-block:: python 14 | 15 | >>> x=21 16 | >>> assert x==21 17 | >>> assert x==42, ":(" 18 | Traceback (most recent call last): 19 | File "", line 1, in 20 | AssertionError: :( 21 | 22 | 23 | 24 | ------------------------------------------------------------ 25 | 26 | ``unittest`` 27 | ~~~~~~~~~~~~ 28 | 29 | https://docs.python.org/3/library/unittest.html 30 | 31 | Basic concepts 32 | * test fixture -- the preparations/cleanups needed to perform one or more tests 33 | * test case -- checks for a specific response to a particular set of inputs 34 | * test suite -- is a collection of test cases, test suites 35 | * test runner -- orchestrates the execution of tests and provides the outcome to the user 36 | 37 | ------------------------------------------------------------ 38 | 39 | In practice 40 | ^^^^^^^^^^^ 41 | 42 | * ``TestCase`` -- a test case is created with subclassing 43 | * ``test_someMethod`` -- a test is performed by defining a method 44 | * ``setUp()`` -- is run before each test 45 | * ``tearDown()`` -- is run after each test 46 | * `assertions methods `_ 47 | * ``assertEqual``, ``assertNotEqual`` 48 | * ``assertTrue``, ``assertFalse`` 49 | * ``assertIs``, ``assertIsNot`` 50 | * ``assertIn``, ``assertNotIn`` 51 | * ... 52 | 53 | 54 | ------------------------------------------------------------ 55 | 56 | A test (suite/case) can be run with: ``unittest.main()`` or 57 | 58 | .. code-block:: bash 59 | 60 | $ python -m unittest test_module1 test_module2 61 | $ python -m unittest test_module.TestClass 62 | $ python -m unittest test_module.TestClass.test_method 63 | 64 | 65 | ------------------------------------------------------------ 66 | 67 | Example 68 | ^^^^^^^ 69 | .. code-block:: python 70 | 71 | import random 72 | import unittest 73 | 74 | class TestSequenceFunctions(unittest.TestCase): 75 | 76 | def setUp(self): 77 | self.seq = list(range(10)) 78 | 79 | def test_shuffle(self): 80 | # make sure the shuffled sequence does not lose any elements 81 | random.shuffle(self.seq) 82 | self.seq.sort() 83 | self.assertEqual(self.seq, list(range(10))) 84 | 85 | # should raise an exception for 86 | # an immutable sequence 87 | self.assertRaises(TypeError, 88 | random.shuffle, (1,2,3)) 89 | 90 | ------------------------------------------------------------ 91 | 92 | .. code-block:: python 93 | 94 | def test_choice(self): 95 | element = random.choice(self.seq) 96 | self.assertTrue(element in self.seq) 97 | 98 | def test_sample(self): 99 | with self.assertRaises(ValueError): 100 | random.sample(self.seq, 20) 101 | for element in random.sample(self.seq, 5): 102 | self.assertTrue(element in self.seq) 103 | 104 | if __name__ == '__main__': 105 | unittest.main() 106 | 107 | 108 | 109 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Attribution 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution 4.0 International Public License 58 | 59 | By exercising the Licensed Rights (defined below), You accept and agree 60 | to be bound by the terms and conditions of this Creative Commons 61 | Attribution 4.0 International Public License ("Public License"). To the 62 | extent this Public License may be interpreted as a contract, You are 63 | granted the Licensed Rights in consideration of Your acceptance of 64 | these terms and conditions, and the Licensor grants You such rights in 65 | consideration of benefits the Licensor receives from making the 66 | Licensed Material available under these terms and conditions. 67 | 68 | 69 | Section 1 -- Definitions. 70 | 71 | a. Adapted Material means material subject to Copyright and Similar 72 | Rights that is derived from or based upon the Licensed Material 73 | and in which the Licensed Material is translated, altered, 74 | arranged, transformed, or otherwise modified in a manner requiring 75 | permission under the Copyright and Similar Rights held by the 76 | Licensor. For purposes of this Public License, where the Licensed 77 | Material is a musical work, performance, or sound recording, 78 | Adapted Material is always produced where the Licensed Material is 79 | synched in timed relation with a moving image. 80 | 81 | b. Adapter's License means the license You apply to Your Copyright 82 | and Similar Rights in Your contributions to Adapted Material in 83 | accordance with the terms and conditions of this Public License. 84 | 85 | c. Copyright and Similar Rights means copyright and/or similar rights 86 | closely related to copyright including, without limitation, 87 | performance, broadcast, sound recording, and Sui Generis Database 88 | Rights, without regard to how the rights are labeled or 89 | categorized. For purposes of this Public License, the rights 90 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 91 | Rights. 92 | 93 | d. Effective Technological Measures means those measures that, in the 94 | absence of proper authority, may not be circumvented under laws 95 | fulfilling obligations under Article 11 of the WIPO Copyright 96 | Treaty adopted on December 20, 1996, and/or similar international 97 | agreements. 98 | 99 | e. Exceptions and Limitations means fair use, fair dealing, and/or 100 | any other exception or limitation to Copyright and Similar Rights 101 | that applies to Your use of the Licensed Material. 102 | 103 | f. Licensed Material means the artistic or literary work, database, 104 | or other material to which the Licensor applied this Public 105 | License. 106 | 107 | g. Licensed Rights means the rights granted to You subject to the 108 | terms and conditions of this Public License, which are limited to 109 | all Copyright and Similar Rights that apply to Your use of the 110 | Licensed Material and that the Licensor has authority to license. 111 | 112 | h. Licensor means the individual(s) or entity(ies) granting rights 113 | under this Public License. 114 | 115 | i. Share means to provide material to the public by any means or 116 | process that requires permission under the Licensed Rights, such 117 | as reproduction, public display, public performance, distribution, 118 | dissemination, communication, or importation, and to make material 119 | available to the public including in ways that members of the 120 | public may access the material from a place and at a time 121 | individually chosen by them. 122 | 123 | j. Sui Generis Database Rights means rights other than copyright 124 | resulting from Directive 96/9/EC of the European Parliament and of 125 | the Council of 11 March 1996 on the legal protection of databases, 126 | as amended and/or succeeded, as well as other essentially 127 | equivalent rights anywhere in the world. 128 | 129 | k. You means the individual or entity exercising the Licensed Rights 130 | under this Public License. Your has a corresponding meaning. 131 | 132 | 133 | Section 2 -- Scope. 134 | 135 | a. License grant. 136 | 137 | 1. Subject to the terms and conditions of this Public License, 138 | the Licensor hereby grants You a worldwide, royalty-free, 139 | non-sublicensable, non-exclusive, irrevocable license to 140 | exercise the Licensed Rights in the Licensed Material to: 141 | 142 | a. reproduce and Share the Licensed Material, in whole or 143 | in part; and 144 | 145 | b. produce, reproduce, and Share Adapted Material. 146 | 147 | 2. Exceptions and Limitations. For the avoidance of doubt, where 148 | Exceptions and Limitations apply to Your use, this Public 149 | License does not apply, and You do not need to comply with 150 | its terms and conditions. 151 | 152 | 3. Term. The term of this Public License is specified in Section 153 | 6(a). 154 | 155 | 4. Media and formats; technical modifications allowed. The 156 | Licensor authorizes You to exercise the Licensed Rights in 157 | all media and formats whether now known or hereafter created, 158 | and to make technical modifications necessary to do so. The 159 | Licensor waives and/or agrees not to assert any right or 160 | authority to forbid You from making technical modifications 161 | necessary to exercise the Licensed Rights, including 162 | technical modifications necessary to circumvent Effective 163 | Technological Measures. For purposes of this Public License, 164 | simply making modifications authorized by this Section 2(a) 165 | (4) never produces Adapted Material. 166 | 167 | 5. Downstream recipients. 168 | 169 | a. Offer from the Licensor -- Licensed Material. Every 170 | recipient of the Licensed Material automatically 171 | receives an offer from the Licensor to exercise the 172 | Licensed Rights under the terms and conditions of this 173 | Public License. 174 | 175 | b. No downstream restrictions. You may not offer or impose 176 | any additional or different terms or conditions on, or 177 | apply any Effective Technological Measures to, the 178 | Licensed Material if doing so restricts exercise of the 179 | Licensed Rights by any recipient of the Licensed 180 | Material. 181 | 182 | 6. No endorsement. Nothing in this Public License constitutes or 183 | may be construed as permission to assert or imply that You 184 | are, or that Your use of the Licensed Material is, connected 185 | with, or sponsored, endorsed, or granted official status by, 186 | the Licensor or others designated to receive attribution as 187 | provided in Section 3(a)(1)(A)(i). 188 | 189 | b. Other rights. 190 | 191 | 1. Moral rights, such as the right of integrity, are not 192 | licensed under this Public License, nor are publicity, 193 | privacy, and/or other similar personality rights; however, to 194 | the extent possible, the Licensor waives and/or agrees not to 195 | assert any such rights held by the Licensor to the limited 196 | extent necessary to allow You to exercise the Licensed 197 | Rights, but not otherwise. 198 | 199 | 2. Patent and trademark rights are not licensed under this 200 | Public License. 201 | 202 | 3. To the extent possible, the Licensor waives any right to 203 | collect royalties from You for the exercise of the Licensed 204 | Rights, whether directly or through a collecting society 205 | under any voluntary or waivable statutory or compulsory 206 | licensing scheme. In all other cases the Licensor expressly 207 | reserves any right to collect such royalties. 208 | 209 | 210 | Section 3 -- License Conditions. 211 | 212 | Your exercise of the Licensed Rights is expressly made subject to the 213 | following conditions. 214 | 215 | a. Attribution. 216 | 217 | 1. If You Share the Licensed Material (including in modified 218 | form), You must: 219 | 220 | a. retain the following if it is supplied by the Licensor 221 | with the Licensed Material: 222 | 223 | i. identification of the creator(s) of the Licensed 224 | Material and any others designated to receive 225 | attribution, in any reasonable manner requested by 226 | the Licensor (including by pseudonym if 227 | designated); 228 | 229 | ii. a copyright notice; 230 | 231 | iii. a notice that refers to this Public License; 232 | 233 | iv. a notice that refers to the disclaimer of 234 | warranties; 235 | 236 | v. a URI or hyperlink to the Licensed Material to the 237 | extent reasonably practicable; 238 | 239 | b. indicate if You modified the Licensed Material and 240 | retain an indication of any previous modifications; and 241 | 242 | c. indicate the Licensed Material is licensed under this 243 | Public License, and include the text of, or the URI or 244 | hyperlink to, this Public License. 245 | 246 | 2. You may satisfy the conditions in Section 3(a)(1) in any 247 | reasonable manner based on the medium, means, and context in 248 | which You Share the Licensed Material. For example, it may be 249 | reasonable to satisfy the conditions by providing a URI or 250 | hyperlink to a resource that includes the required 251 | information. 252 | 253 | 3. If requested by the Licensor, You must remove any of the 254 | information required by Section 3(a)(1)(A) to the extent 255 | reasonably practicable. 256 | 257 | 4. If You Share Adapted Material You produce, the Adapter's 258 | License You apply must not prevent recipients of the Adapted 259 | Material from complying with this Public License. 260 | 261 | 262 | Section 4 -- Sui Generis Database Rights. 263 | 264 | Where the Licensed Rights include Sui Generis Database Rights that 265 | apply to Your use of the Licensed Material: 266 | 267 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 268 | to extract, reuse, reproduce, and Share all or a substantial 269 | portion of the contents of the database; 270 | 271 | b. if You include all or a substantial portion of the database 272 | contents in a database in which You have Sui Generis Database 273 | Rights, then the database in which You have Sui Generis Database 274 | Rights (but not its individual contents) is Adapted Material; and 275 | 276 | c. You must comply with the conditions in Section 3(a) if You Share 277 | all or a substantial portion of the contents of the database. 278 | 279 | For the avoidance of doubt, this Section 4 supplements and does not 280 | replace Your obligations under this Public License where the Licensed 281 | Rights include other Copyright and Similar Rights. 282 | 283 | 284 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 285 | 286 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 287 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 288 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 289 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 290 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 291 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 292 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 293 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 294 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 295 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 296 | 297 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 298 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 299 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 300 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 301 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 302 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 303 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 304 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 305 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 306 | 307 | c. The disclaimer of warranties and limitation of liability provided 308 | above shall be interpreted in a manner that, to the extent 309 | possible, most closely approximates an absolute disclaimer and 310 | waiver of all liability. 311 | 312 | 313 | Section 6 -- Term and Termination. 314 | 315 | a. This Public License applies for the term of the Copyright and 316 | Similar Rights licensed here. However, if You fail to comply with 317 | this Public License, then Your rights under this Public License 318 | terminate automatically. 319 | 320 | b. Where Your right to use the Licensed Material has terminated under 321 | Section 6(a), it reinstates: 322 | 323 | 1. automatically as of the date the violation is cured, provided 324 | it is cured within 30 days of Your discovery of the 325 | violation; or 326 | 327 | 2. upon express reinstatement by the Licensor. 328 | 329 | For the avoidance of doubt, this Section 6(b) does not affect any 330 | right the Licensor may have to seek remedies for Your violations 331 | of this Public License. 332 | 333 | c. For the avoidance of doubt, the Licensor may also offer the 334 | Licensed Material under separate terms or conditions or stop 335 | distributing the Licensed Material at any time; however, doing so 336 | will not terminate this Public License. 337 | 338 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 339 | License. 340 | 341 | 342 | Section 7 -- Other Terms and Conditions. 343 | 344 | a. The Licensor shall not be bound by any additional or different 345 | terms or conditions communicated by You unless expressly agreed. 346 | 347 | b. Any arrangements, understandings, or agreements regarding the 348 | Licensed Material not stated herein are separate from and 349 | independent of the terms and conditions of this Public License. 350 | 351 | 352 | Section 8 -- Interpretation. 353 | 354 | a. For the avoidance of doubt, this Public License does not, and 355 | shall not be interpreted to, reduce, limit, restrict, or impose 356 | conditions on any use of the Licensed Material that could lawfully 357 | be made without permission under this Public License. 358 | 359 | b. To the extent possible, if any provision of this Public License is 360 | deemed unenforceable, it shall be automatically reformed to the 361 | minimum extent necessary to make it enforceable. If the provision 362 | cannot be reformed, it shall be severed from this Public License 363 | without affecting the enforceability of the remaining terms and 364 | conditions. 365 | 366 | c. No term or condition of this Public License will be waived and no 367 | failure to comply consented to unless expressly agreed to by the 368 | Licensor. 369 | 370 | d. Nothing in this Public License constitutes or may be interpreted 371 | as a limitation upon, or waiver of, any privileges and immunities 372 | that apply to the Licensor or You, including from the legal 373 | processes of any jurisdiction or authority. 374 | 375 | 376 | ======================================================================= 377 | 378 | Creative Commons is not a party to its public 379 | licenses. Notwithstanding, Creative Commons may elect to apply one of 380 | its public licenses to material it publishes and in those instances 381 | will be considered the “Licensor.” The text of the Creative Commons 382 | public licenses is dedicated to the public domain under the CC0 Public 383 | Domain Dedication. Except for the limited purpose of indicating that 384 | material is shared under a Creative Commons public license or as 385 | otherwise permitted by the Creative Commons policies published at 386 | creativecommons.org/policies, Creative Commons does not authorize the 387 | use of the trademark "Creative Commons" or any other trademark or logo 388 | of Creative Commons without its prior written consent including, 389 | without limitation, in connection with any unauthorized modifications 390 | to any of its public licenses or any other arrangements, 391 | understandings, or agreements concerning use of licensed material. For 392 | the avoidance of doubt, this paragraph does not form part of the 393 | public licenses. 394 | 395 | Creative Commons may be contacted at creativecommons.org. 396 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Teaching materials for Scripting Languages and Advanced Python courses 2 | 3 | This repository contains slides, tutorials and exercises for two university lectures taught in the Pázmány Péter Catholic University of Budapest. 4 | * "Introduction to Scripting Languages" -- fall semesters in 2012-2015 5 | * "Advanced Python" / "Python for Data Science" -- spring semesters 2015-2016 6 | 7 | Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License. 8 | 9 | (c) György Orosz 2016 10 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/Exercises/basic_exercises.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | Exercises 3 | ============= 4 | 5 | 1. Basics 6 | --------- 7 | 8 | #. What are the values of the following expressions? (Why?) 9 | 10 | a) ``"a" + 1`` 11 | b) ``type("3.14")``; ``type(type("3.14"))`` 12 | c) ``type``; ``type(type)`` 13 | d) ``2.0/3.0``; ``2.0/3``; ``2/3.0``; ``2/3``; 14 | e) ``2//3``; ``2.5//2.1``; ``2.5%2.1`` 15 | f) ``"a"<"b"``; ``"a"<"á"``; ``"á"<"b"`` 16 | g) ``None``; ``type(None)`` 17 | h) ``bool(1)``; ``bool(0.1)``; ``bool(0)`` 18 | i) ``bool("a")``; ``bool("")``; ``bool(" ")`` 19 | j) ``bool(bool)``; ``bool(None)`` 20 | k) ``True or 2``; ``True and 2`` 21 | l) ``1 or 2``; ``1 and 2``; ``0 and 1``; ``0 or 1`` 22 | m) ``-1 and ("empty" and 0.0) or str(type(8 is (7+1)))[8:-3] is "boo"`` 23 | 24 | 25 | #. Read two strings from the user and print their concatenation, without the first character of each. The strings will be at least length 1. E.g.: ``'Hello', 'There' → 'ellohere'`` 26 | 27 | #. Write a script that decides on a string (which is typed by the user) whether it is a palindrome. 28 | 29 | #. Print "True" if the strings "cat" and "dog" appear the same number of times in the given string. 30 | 31 | .. code-block:: none 32 | 33 | $ python myscript.py < echo 'catcat' 34 | False 35 | $ python myscript.py < echo '1cat1cadodog' 36 | True 37 | 38 | #. Transform a non-empty string (read from the user) like "Code" to another such as "CCoCodCode", "Program" to "PPrProProgProgrPrograProgram" etc. 39 | 40 | #. Given 2 strings, ``a`` and ``b``. Count the number of the positions where they contain the same two character long substring. E.g. "xxcaazz" and "xxbaaz" yields 3, since the "xx", "aa", and "az" substrings appear in the same place in both strings. 41 | 42 | #. Create a simple script that works like a calculator. It reads expressions from the user until "q" is inputted, then evaluates it and prints the results to the output. 43 | * The script should handle simple expressions containing only one operator. 44 | 45 | #. Implement a script that evaluates Roman numbers that are given by the user. 46 | * After printing the value, ask the user for the next task, till she types "quit". 47 | * Check the input before the evaluation! 48 | 49 | 2. Data structures 50 | ------------------ 51 | 52 | #. "One liners" -- use comprehension! 53 | 54 | #. Create a list of numbers in which the elements are less than 1000 and are not divisible by neither 2 or 5. 55 | #. Create a list with the even powers of 2 that are less than 100. 56 | #. Create a list containing a the coordinates of a three dimensional (3x3x3) matrix. (``(1,1,1), (1,1,2) ... (3,3,3)``) 57 | #. Transform a string by omitting the lowercase characters. 58 | #. Calculate the symmetric difference of two sets. 59 | #. Split a multiline string into list of lists representing words in lines. (``E.g. [["First", "line"], ["Second", "line"]]``) 60 | 61 | 62 | #. A sparse matrix is a matrix with a lot of 0 elements. We want to save memory by storing only the nonzero elements. Using the Python dictionary datatype, create a representation of sparse matrices, with the following interface: 63 | 64 | * ``m_new()`` -- returns an all-zero matrix 65 | * ``m_set(m, i, j, e)`` -- sets the value at row ``i``, column ``j`` to ``e`` 66 | * ``m_get(m, i, j)`` -- returns the value at position ``(i,j)`` 67 | 68 | Define the scalar multiplication as well: ``s_mul(mlambda, m)`` that returns a **new** matrix. 69 | 70 | #. Write a script that takes a word from the user and prints a dictionary that counts the number of times each letter appears. 71 | 72 | a. Read the string as a command line argument, instead of interactively. 73 | 74 | #. A sparse vector is a vector whose entries are almost all zero, like [1, 0, 0, 0, 0, 0, 0, 2, 0]. Storing all those zeros wastes memory and dictionaries are commonly used to keep track of just the nonzero entries. For example, the vector shown earlier can be represented as {0:1, 7:2}, since the vector it is meant to represent has the value 1 at index 0 and the value 2 at index 7. Write a function / script that converts a dictionary to its sparse vector representation and vica versa. 75 | 76 | #. Write a function ``removeCommonElements(t1, t2)`` that takes in 2 tuples as arguments and returns a sorted tuple containing elements that are only found in one of them. 77 | 78 | 3. Functions 79 | ------------ 80 | 81 | #. Create a function: given a non-negative number ``num``, return ``True`` if ``num`` is one of the multipliers of 10. 82 | 83 | #. We want to make a row of bricks that is goal inches long. We have a number of small bricks (1 inch each) and big bricks (5 inches each). Return ``True`` if it is possible to make the goal by choosing from the given bricks. This is a little harder than it looks and can be done without any loops. 84 | 85 | .. code-block:: python 86 | 87 | >>> make_bricks(3, 1, 8) 88 | True 89 | >>> make_bricks(3, 1, 9) 90 | False 91 | >>> make_bricks(3, 2, 10) 92 | True 93 | 94 | #. Return the "centered" average of an array of integers, which we'll say is the mean average of the values, except ignoring the largest and smallest values in the array. If there are multiple copies of the smallest value, ignore just one copy, and likewise for the largest value. Use int division to produce the final average. You may assume that the array's length 3 or more. 95 | 96 | .. code-block:: python 97 | 98 | >>> centered_average([1, 2, 3, 4, 100]) 99 | 3 100 | >>> centered_average([1, 1, 5, 5, 10, 8, 7]) 101 | 5 102 | 103 | #. Given an list of integers, return ``True`` if the array contains a 2 next to a 2 somewhere. 2 should be a (default) parameter. 104 | 105 | .. code-block:: python 106 | 107 | >>> has_x([1, 2, 2]) 108 | True 109 | >>> has_x([1, 2, 1, 2]) 110 | False 111 | >>> has_x([1, 3, 3, 2], 3) 112 | True 113 | 114 | #. Write a function ``shiftByTwo(*args)`` that accepts a variable number of arguments and returns a tuple with the argument list shifted to the right by two indices. See the example below. 115 | 116 | .. code-block:: 117 | 118 | >>> shiftByTwo(1,2,3,4,5,6) 119 | (5, 6, 1, 2, 3, 4) 120 | 121 | #. Write a function ``sortByIndex(aList)`` that takes in a list of tuples in the following format: (index, value) and returns a new tuple with its elements sorted based by their index. 122 | 123 | .. code-block:: python 124 | 125 | >>> sortByIndex([(2,'Programming'), (3, 'is'), (1, 'Python'), (4, 'Fun')]) 126 | ('Python', 'Programming', 'is', 'Fun') 127 | 128 | #. Write a recursive function ``countX`` that takes a string and returns the number of uppercase 'X' characters in the string. 129 | 130 | #. Write a function ``numbersInbetween(start, end)`` that takes in two numbers and returns a comma-separated string with all the numbers in between the start and end number inclusive of both numbers. 131 | 132 | .. code-block:: python 133 | 134 | >>> numbersInbetween(5, 10) 135 | '5,6,7,8,9,10' 136 | >>> numbersInbetween(5, 0) 137 | 'Invalid' 138 | 139 | #. Write a recursive function that traverses the tree given below and appends a new left node with the name ``42`` to each leaf node! 140 | 141 | .. code-block:: python 142 | 143 | Tree = { 144 | 'name': 'animals', 145 | 'left_branch': { 146 | 'name': 'birds', 147 | 'left_branch': { 148 | 'name': 'seed eaters', 149 | 'left_branch': { 150 | 'name': 'house finch', 151 | 'left_branch': None, 152 | 'right_branch': None, 153 | }, 154 | 'right_branch': { 155 | 'name': 'white crowned sparrow', 156 | 'left_branch': None, 157 | 'right_branch': None, 158 | }, 159 | }, 160 | 'right_branch': { 161 | 'name': 'insect eaters', 162 | 'left_branch': { 163 | 'name': 'hermit thrush', 164 | 'left_branch': None, 165 | 'right_branch': None, 166 | }, 167 | 'right_branch': { 168 | 'name': 'black headed phoebe', 169 | 'left_branch': None, 170 | 'right_branch': None, 171 | }, 172 | }, 173 | }, 174 | 'right_branch': None, 175 | } 176 | 177 | #. Implement a function called ``transmogr`` that returns all the values from an iterable that satisfy a predicate. Optionally, it applies a series of transforms to each returned value. The function takes these arguments: 178 | 179 | #. values -- A list of values. Actually, it could be any iterable. 180 | 181 | #. predicate -- A function that takes a single argument, performs a test on that value, and returns True or False. 182 | 183 | #. transforms -- (optional) A list of functions. Apply each function in this list and returns the resulting value. So, for example, if the function is called like this: `` transmogr([11, 22], p, [f, g])`` where ``f``, ``g`` and ``p`` are functions, ``p(11)==True`` and ``p(22)==False``, then the returned value should equal ``[g(f(11))]`` 184 | 185 | *Optional: Implement this exercise as a generator function. (Apply all the transformations before yielding!)* 186 | 187 | 188 | #. Implement the higher order functions ``map()``, ``filter()`` and ``reduce()``. They are built-ins but writing them by yourself might be a good exercise. You should use alternative function names, so that you can compare your solutions to the built-ins. 189 | 190 | #. Using the higher order function ``filter()``, define a function ``filter_long_words(words, n)`` that takes a list of words and an integer ``n`` and returns the list of words that are longer than ``n``. 191 | a. use ``n=5`` as default parameter 192 | 193 | #. Using the higher order function ``reduce()``, write a function ``max_in_list(nums)`` that takes a list of numbers and returns the largest one. 194 | 195 | #. A memoized function is a function that remembers the returned values for all arguments it was previously called with. It does not calculate the result for the same arguments twice. Instead, it returns the remembered result. This is useful for expensive calculations. Your task is to write a function, that takes a regular function as argument and returns the memoized version. 196 | 197 | #. Use builtin functions to: 198 | #. Count the number of characters in a list. 199 | #. Implement the following metric: list ``l1`` is *greater* than ``l2`` (with the same size), if at least half of the elements of ``l1`` is greater than their counterparts in ``l2`` (having the same index) 200 | #. Count whitespaces in a string. 201 | #. Create a dictionary from a set of strings (keys) with the number of their uppercase characters (as values). 202 | #. Calculate the value of *pi* iteratively. 203 | 204 | 205 | #. Generate randomly a set of quadratic functions and find which has the maximal value in the discrete interval [0..10]. 206 | 207 | #. We want to make a row of bricks that is goal inches long. We have a number of small bricks (1 inch each) and big bricks (5 inches each). Return ``True`` if it is possible to make the goal by choosing from the given bricks. This is a little harder than it looks and can be done without any loops. 208 | 209 | .. code-block:: python 210 | 211 | >>> make_bricks(3, 1, 8) 212 | True 213 | >>> make_bricks(3, 1, 9) 214 | False 215 | >>> make_bricks(3, 2, 10) 216 | True 217 | 218 | #. Return the "centered" average of an array of integers, which we'll say is the mean average of the values, except ignoring the largest and smallest values in the array. If there are multiple copies of the smallest value, ignore just one copy, and likewise for the largest value. Use int division to produce the final average. You may assume that the array's length 3 or more. 219 | 220 | .. code-block:: python 221 | 222 | >>> centered_average([1, 2, 3, 4, 100]) 223 | 3 224 | >>> centered_average([1, 1, 5, 5, 10, 8, 7]) 225 | 5 226 | 227 | 4. I/O and moduls 228 | ------ 229 | 230 | #. Implement the Unix ``sort`` command: the program reads lines from a file (if it is given) or from the standard input then prints them in alphabetical order. 231 | 232 | #. Implement the Unix ``tr`` command: the program reads lines from a file (if it is given as the last parameter) or from the standard input then replaces the first set of characters with the second set of characters. 233 | 234 | .. code-block:: guess 235 | 236 | $ echo "Hello" | tr "lo" "10" 237 | He110 238 | 239 | 240 | 241 | #. Implement the following Unix commands (as before): 242 | 243 | .. TODO:nem ismerik az emberek a unixos parancsokat - Marci 244 | 245 | a) ``uniq`` with the optional parameter ``-c`` 246 | b) ``less`` 247 | c) ``head``, ``tail`` with the optional ``-n `` parameter 248 | d) ``cut`` with the ``-f`` and ``-d`` options 249 | 250 | #. From the standard input *recode* the Hungarian accents in the following way: 251 | 252 | .. code-block:: guess 253 | 254 | ó -> o' ő -> o" ö -> o: ü -> u: ű -> u" é -> e' á -> a' í -> i' 255 | 256 | #. In cryptography, a Caesar cipher is a very simple encryption techniques in which each letter in the plain text is replaced by a letter some fixed number of positions down the alphabet. For example, with a shift of 3, A would be replaced by D, B would become E, and so on. The method is named after Julius Caesar, who used it to communicate with his generals. *ROT-13* ("rotate by 13 places") is a widely used example of a Caesar cipher where the shift is 13. In Python, the key for ROT-13 may be represented by means of the following format in a text file: 257 | 258 | .. code-block:: 259 | 260 | 'a':'n', 'b':'o', 'c':'p', 'd':'q', 'e':'r', 'f':'s', 'g':'t', 'h':'u', 261 | 'i':'v', 'j':'w', 'k':'x', 'l':'y', 'm':'z', 'n':'a', 'o':'b', 'p':'c', 262 | 'q':'d', 'r':'e', 's':'f', 't':'g', 'u':'h', 'v':'i', 'w':'j', 'x':'k', 263 | 'y':'l', 'z':'m', 'A':'N', 'B':'O', 'C':'P', 'D':'Q', 'E':'R', 'F':'S', 264 | 'G':'T', 'H':'U', 'I':'V', 'J':'W', 'K':'X', 'L':'Y', 'M':'Z', 'N':'A', 265 | 'O':'B', 'P':'C', 'Q':'D', 'R':'E', 'S':'F', 'T':'G', 'U':'H', 'V':'I', 266 | 'W':'J', 'X':'K', 'Y':'L', 'Z':'M' 267 | 268 | Your task in this exercise is to implement an encoder/decoder of ROT-13. Once you're done, you will be able to read the following secret message: ``"Pnrfne pvcure? V zhpu cersre Pnrfne fnynq!"s`` 269 | Note that since English has 26 characters, your ROT-13 program will be able to both encode and decode texts written in English. 270 | 271 | #. A *hapax legomenon* (often abbreviated to hapax) is a word which occurs only once in either the written record of a language, the works of an author, or in a single text. Define a function ``hapax(file_path)``that reads a text file and returns all of the hapaxes. 272 | a. Make sure your program ignores capitalization. 273 | 274 | #. In a game of Lingo, there is a hidden word, five characters long. The object of the game is to find this word by guessing, and in return receive two kinds of clues: 275 | 276 | 1. the characters that are fully correct, with respect to identity as well as to position, 277 | 2. the characters that are indeed present in the word, but which are placed in the wrong position. 278 | 279 | Write a program with which one can play Lingo. Use square brackets to mark characters correct in the sense of 1), and ordinary parentheses to mark characters correct in the sense of 2). (Words to be guessed are stored in a text file.) Assuming, for example, that the program conceals the word "tiger", you should be able to interact with it in the following way: 280 | 281 | .. code-block:: none 282 | 283 | >>> import lingo 284 | snake 285 | Clue: snak(e) 286 | fiest 287 | Clue: f[i](e)s(t) 288 | times 289 | Clue: [t][i]m[e]s 290 | tiger 291 | Clue: [t][i][g][e][r] 292 | 293 | 294 | #. From the `names.html`_ file create a ``male_names.txt`` and a ``female_names.txt`` containing the most popular given names in 2011. 295 | 296 | * Use UTF-8 files. 297 | * Sort the names by their popularity. 298 | * Take the input and output file names command line arguments. 299 | 300 | #. Create a "local search engine" that finds the text file in a directory structure that is the most relevant according to a search query. (Relevancy is calculated: ``# query words occurance / # words in the document``) 301 | 302 | .. code-block:: bash 303 | 304 | $ search.py "Barack Obama" ./ 305 | ./subdir/presidents.txt 0.0236 306 | 307 | a) Make your app work recursively with the ``-r`` option! (you can use ``argparse`` module) 308 | b) Print the first ``n`` (5) documents with the ``-n 5`` option. 309 | 310 | #. **Truecase model:** Truecasing is the task to find out the ordinary case of a word. It is useful for NLP. The model is to count the occurence of the different case form of the words in a dictionary. The key of the dictionary should be the lowercased word. The value of the dictionary is the sorted order of the word forms. 311 | #. Create a script file (``truecase_train.py``) which contains the training function of a truecaser. The input is a filename contains the training data (English side of `parallel corpora `_) and a model file name where to store the created model file. 312 | #. Create a script file (``truecase.py``) which contains the truecaser function. The input of this function is the name of the model file and the name of the input and output files 313 | #. Create a run.py which imports the previous two files and do a training and a truecasing. 314 | 315 | The required dictionary during the training has the following structure: 316 | 317 | .. code-block:: json 318 | 319 | { 320 | "cat": { 321 | "CAT": 1, 322 | "Cat": 5, 323 | "cat": 10 324 | }, 325 | "chicago": { 326 | "Chicago": 3 327 | }, 328 | "sad": { 329 | "SAD": 4, 330 | "sad": 8 331 | } 332 | } 333 | 334 | From this dictionary we have to select the highest probable forms for all words. 335 | 336 | 337 | 5. Regular expressions 338 | ---------------------- 339 | 340 | #. What do the following regular expressions mean? 341 | a) ``[.]`` 342 | b) ``\**`` 343 | c) ``^[*]+$`` 344 | d) ``^\*+$`` 345 | e) ``^#`` 346 | f) ``^$`` 347 | g) ``*.*`` 348 | h) ``.{1,8}\..{3}`` 349 | i) ``b[ae]n?`` 350 | j) ``^(not|to|be)`` 351 | k) ``([()])`` 352 | l) ``]+>`` 353 | 354 | #. Solve the following interactive exercises 355 | a) numbers http://regexone.com/example/0 356 | b) phone numbers http://regexone.com/example/1 357 | c) e-mail address http://regexone.com/example/2 358 | d) html tags http://regexone.com/example/3 359 | e) special file names http://regexone.com/example/4 360 | f) extracting information from a log file http://regexone.com/example/6 361 | 362 | #. Write a regular expression that matches 363 | #. URLs, 364 | #. email addresses at PPKE ITK, 365 | #. Python lists that are not nested, 366 | #. bold parts of an HTML file, 367 | #. lines that have more than five **words**. 368 | 369 | 370 | #. Collect the URLs from the `Pythagorean_theorem.html `_. 371 | 372 | a) How many of them are referreing to an element which is inside the document? 373 | b) Print them sorting by the number of their occurance! 374 | 375 | #. Define a simple "spell check" function ``correct()`` that takes a string and transforms it as described below: 376 | 1. two or more occurrences of the space character is compressed into one 377 | 2. inserts an extra space after a period if the period is followed by a letter. 378 | 379 | E.g. ``correct("This is very funny and cool.Indeed!")`` should return ``"This is very funny and cool. Indeed!"`` 380 | 381 | .. tokenizacio fogalmat nem tanultuk; Unicode literal legujabb Python 3-ban kerult vissza, ez talan fontos !Javítva - így már érthetőbb? 382 | 383 | #. In natural language processing, tokenisation is the process of splitting a sentence into tokens (that are either words or punctuation marks). Do a basic word tokenisation script for Hungarian texts which splits tokens with spaces. 384 | 385 | .. code-block:: python 386 | 387 | >> word_tokenize(u"Josh, this is a (very) nice day!") 388 | u"Jush ⬛, this is a (⬛ very ⬛) nice day ⬛!" 389 | 390 | #. To translate numbers, dates or URL is a challenge for machine translation systems. These words greatly increase the vocabulary size because MT systems should learn all numburs one-by-one. The solution is placeholders. The task is to create a script which mark all matched numbers in parallel data and change them to a symbol. 391 | 392 | E.g. ``I live in the 2nd floor. A 2. emeleten lakom.`` 393 | should become 394 | ``I live in the ⬛num_1⬛ ⬛nd foor ⬛. A ⬛num_1⬛ ⬛. emeleten lakom ⬛.`` 395 | 396 | E.g. `` The date is 26/03/2019. Ma 2019. 03. 26.-a van.`` 397 | should become 398 | ``The date is ⬛date_1⬛ ⬛. Ma ⬛date_1⬛ ⬛. ⬛-⬛ a van ⬛. 399 | 400 | #. Create a script which shortens a python script by removing all lines that are empty or contain only comments. 401 | 402 | 403 | 6. OOP 404 | ------ 405 | 406 | .. Utolso reszfeladat eleg sokfelekeppen ertelmezheto ! Raktam példát a végére, így érthetőbb? 407 | 408 | #. Implement a ``Point`` class that represents two dimensional points. 409 | A. Basics 410 | * define a constructor that takes two optional parameters with 0 default values, and assigns these values to ``x`` and ``y`` 411 | * define a ``set(x,y)``, a ``get_x()`` and a ``get_y()`` method 412 | B. Operators (http://docs.python.org/3/reference/datamodel.html#special-method-names) 413 | * implement the plus operator 414 | * define a ``__str__`` method 415 | * implement ``__setitem__`` and ``__getitem__`` methods 416 | 417 | .. code-block:: python 418 | 419 | >>> p1 = Point(1,2) 420 | >>> p2 = Point(3,5) 421 | >>> print(p1+p2) 422 | Point(4, 7) 423 | >>> print(p1[0]+p1[1]+p2[0]+p2[1]) 424 | 11 425 | 426 | .. file belinkelese - Marci 427 | #. Download the file `BadKangaroo.py ` and find the error in the code. 428 | 429 | .. Design Patterns kovetkezo feleves targy, magyarazat szukseges, vagy ki is hagyhatjuk ezt a peldat 430 | #. Implement the following design patterns: 431 | * Singleton (with lazy initialization) http://en.wikipedia.org/wiki/Singleton_pattern 432 | * Composite http://en.wikipedia.org/wiki/Composite_pattern 433 | 434 | #. Implement a class for rational (``Rat``) numbers. Define the following functions: 435 | * add operator, 436 | * substraction operator, 437 | * constructor, 438 | * string representation, 439 | * multiplication operator\*, 440 | * divide operator\*. 441 | 442 | .. code-block:: python 443 | 444 | x = Rat(2,3) 445 | y = Rat(2,6) 446 | print(x) # 2/3 447 | print(y) # 1/3 (!) 448 | z = x+y 449 | print(z) # 1/1 450 | 451 | 452 | .. TODO Python 3-ra atirni (print fv stb.) tenylegesen tesztelni - Marci 453 | #. Create a sparse vector representation (use dictionaries). Multiplication of two vectors should mean scalar product. Make your code compile against the following tests: 454 | 455 | .. code-block:: python 456 | 457 | ####### Basics ####### 458 | 459 | x = SparseVector() 460 | x[1] = 1.0 461 | x[3] = 3.0 462 | x[5] = 5.0 463 | print('len(x)', len(x)) 464 | for i in range(len(x)): 465 | print('...', i, x[i]) 466 | 467 | y = SparseVector() 468 | y[1] = 10.0 469 | y[2] = 20.0 470 | y[3] = 30.0 471 | 472 | print(x) # [1.0, 0.0, 3.0, 0.0, 5.0] 473 | 474 | ####### Full exercise ####### 475 | 476 | print('x + y', x + y) 477 | print('y + x', y + x) 478 | 479 | print('x * y', x * y) 480 | print('y * x', y * x) 481 | 482 | ####### Extra ########## 483 | 484 | z = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5] 485 | 486 | print('x + z', x + z) 487 | print('x * z', x * z) 488 | print('z + x', z + x) 489 | 490 | #. Implement an addition (``Add``) functor class. 491 | a) With ``+``, ``*`` operators and string representation: 492 | 493 | .. code-block:: python 494 | 495 | a2 = Add(2) 496 | print(a2(1)) # 3 497 | a1 = Add(1) 498 | print(a1) # "+1" 499 | a3 = a1+a2 500 | print(a3(1)) # 4 501 | a4 = a2*a2 502 | print(a4(1)) # 5 503 | 504 | 505 | b) With comparison and subtraction operators: 506 | 507 | .. code-block:: python 508 | 509 | print(a2 == a2, a1 < a2, a4 < a2) # True, True, False 510 | print(a1-a2) # "+-1' 511 | print(a2.plus) # 2 512 | a2.plus = -1 # AttributeError 513 | 514 | #. **Zoo animal hierarchy.** Consider the class tree shown in the figure. 515 | Implement six classes which model the taxonomy in the figure. (Use Python inheritance!) Then, add a ``speak()`` method to each class, this should print a unique message. Implement a ``reply()`` method as well in the top-level ``Animal`` superclass that calls ``self.speak`` to invoke the category-specific message printer. Finally, remove the speak method from your Hacker class so that it picks up the default above it. When you’re finished, your classes should work this way: 516 | 517 | .. image:: zoo.png 518 | 519 | .. code-block:: python 520 | 521 | >>> from zoo import Cat, Hacker 522 | >>> spot = Cat() 523 | >>> spot.reply() # Animal.reply; calls Cat.speak 524 | meow 525 | >>> data = Hacker() # Animal.reply; calls Primate.speak 526 | >>> data.reply() 527 | Hello world! 528 | 529 | #. Implement an ordered dictionary class! An instance object must remember the insertion order of the elements, thus when the elements are enumerated the (key,value) pairs should appear in the order of their insertion. (Hint: https://docs.python.org/3/library/stdtypes.html#dict.items) 530 | 531 | .. code-block:: python 532 | 533 | od = MyOrderedDict() 534 | od[1] = "a" 535 | od["hello"] = 0 536 | od[None] = 3.14 537 | 538 | for k,v in od.items(): 539 | print(k,v) 540 | 541 | """ 542 | Result: 543 | 1 "a" 544 | "hello" 0 545 | None 3.14 546 | """ 547 | 548 | 7. Lua 549 | ------ 550 | 551 | #. Create a linked list representation using tables. 552 | 553 | .. code-block:: lua 554 | 555 | -- "a"->"l"->"m"->"a" 556 | print(list.element) 557 | print(list.next.element) 558 | 559 | a) Store strings by reading the values from the standard input. 560 | b) If no more string is given print the list! 561 | c) Make your list double-linked. 562 | 563 | #. Create an application that reads lines from the std. input and writes them back but ignores Lua-style comments. 564 | 565 | #. Find assignments from a ``.lua`` file, then print each variable's value. (Use string matching!) 566 | 567 | a) 568 | .. code-block:: lua 569 | 570 | a = 42 571 | x,y = 1,3 572 | x,y = y,x 573 | b) 574 | .. code-block:: lua 575 | 576 | a,b,c = 42, 0 577 | a,b,c = 1,2,3,4 578 | 579 | #. Implement a table that accesses indices in an *ignorcase* way. 580 | 581 | .. code-block:: lua 582 | 583 | tbl.x = 42 584 | print(tbl.x, tbl.X) -- 42, 42 585 | tbl.X = 24 586 | print(tbl.x, tbl.X) -- 24, 24 587 | 588 | #. Create a type for rational numbers using metatables by implementing multiplication and division operations. 589 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/Exercises/zoo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/ScriptingLanguagesIntroduction/Exercises/zoo.png -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/LuaIntro/10-lua_basics.rst: -------------------------------------------------------------------------------- 1 | ========== 2 | Lua basics 3 | ========== 4 | 5 | ---------------- 6 | 7 | Introduction 8 | ------------ 9 | 10 | History 11 | ~~~~~~~ 12 | * 1993, Petronas 13 | * *moon* 14 | * ancestors: Simple Object Language, Data Entry Language 15 | 16 | Application 17 | ~~~~~~~~~~~ 18 | * testing C/C++ codes 19 | * iOS_ / game development 20 | .. _iOS: http://www.luanova.org/ioswithlua 21 | * embedded data processing 22 | * scripting applications 23 | 24 | 25 | ------ 26 | 27 | Principles 28 | ---------- 29 | 30 | * simplicity (in language and in its implementation) 31 | * efficiency (fast interpreter) 32 | * portability (ANSI C compatible) 33 | * can be embedded (simple C API) 34 | * low cost embedding 35 | 36 | Install 37 | ------- 38 | * version **5.1** 39 | * Windows: http://code.google.com/p/luaforwindows/ 40 | * http://lua-users.org/wiki/LuaBinaries 41 | * http://www.lua.org/demo.html 42 | 43 | IDE: http://lua-users.org/wiki/LuaIntegratedDevelopmentEnvironments 44 | 45 | ------- 46 | 47 | Docs 48 | ---- 49 | 50 | * http://www.lua.org/docs.html 51 | * http://www.lua.org/pil/ 52 | * http://nyelvek.inf.elte.hu/leirasok/Lua/ 53 | * http://lua-users.org/wiki 54 | 55 | Textbooks in library 56 | ~~~~~~~~~~~~~~~~~~~~~ 57 | 58 | * Jung, Brown - Beginning Lua Programming 59 | * Ierusalimschy - Programming in Lua 60 | 61 | ---------- 62 | 63 | Typing 64 | ------- 65 | 66 | * weak typing 67 | * ``type`` function helps 68 | 69 | Types 70 | ~~~~~ 71 | 72 | * ``nil`` 73 | * ``boolean`` 74 | * ``number`` (~ double) 75 | * ``string`` (8 bytes) 76 | * ``function`` 77 | * ``userdata`` 78 | * ``table`` 79 | * ``thread`` 80 | 81 | 82 | ---------------- 83 | 84 | 85 | Basics 86 | ------ 87 | 88 | Interpreter 89 | ~~~~~~~~~~~ 90 | 91 | * exit: ``CTRL`` + ``Z`` or ``C`` 92 | * history: ↑ ↓ 93 | 94 | ``nil`` 95 | ~~~~~ 96 | * not initialized variables 97 | * logical value is ``false`` 98 | 99 | Comments 100 | ~~~~~~~~ 101 | 102 | * ``-- single line comment`` 103 | * ``--[[ multi-line comment ]]`` 104 | 105 | ---------------- 106 | 107 | ``number`` 108 | ---------- 109 | 110 | Literals 111 | ~~~~~~~~~ 112 | 113 | * ``5e2`` 114 | * ``500`` 115 | * ``500.0`` 116 | * ``0x1f4`` 117 | 118 | Operations 119 | ~~~~~~~~~~ 120 | 121 | * float operations 122 | * ``+ - * / ^`` 123 | * from Lua 5.1 ``%`` as well 124 | 125 | ``1/0=?`` 126 | 127 | ---------------- 128 | 129 | ``string`` 130 | ------------ 131 | Literals 132 | ~~~~~~~~~ 133 | * ``"don't"`` and ``'"hello"'`` works like Python 134 | * escape sequences ``"\n"``,``"\t"`` 135 | * multi-line strings with ```[[```` and ``]]`` 136 | * no special characters 137 | 138 | Operations 139 | ~~~~~~~~~~ 140 | * concatenate: ``"hello"..'world'..[[!]]`` 141 | * length: ``#"four"`` 142 | * automatic conversion: ``string`` ⇄ ``number`` 143 | 144 | http://lua-users.org/wiki/StringLibraryTutorial 145 | 146 | ---------------- 147 | 148 | ``boolean`` 149 | ~~~~~~~~~~~~~ 150 | 151 | * literals: ``true``, ``false`` 152 | * every object has a boolean value 153 | * ``nil`` → ``false`` 154 | * others → ``true`` 155 | 156 | Operations 157 | ~~~~~~~~~~ 158 | 159 | * ``==``, ``~=``: types must match, a ``string`` is compared by its content 160 | * ``<``, ``>``: used on ``string`` and ``number`` types 161 | * ``and or not`` 162 | * lazy evaluation 163 | * evaluated to a subexpression value 164 | 165 | ---------------- 166 | 167 | Statements 168 | ---------- 169 | 170 | * assignment 171 | * ``x = 42`` 172 | * ``x,y = y,x`` 173 | * Garbage collection 174 | * sequence 175 | * ``','``, ``';'``, whitespace 176 | 177 | Example 178 | ~~~~~~~ 179 | 180 | .. code-block:: lua 181 | 182 | h,w = "Hello", 'World' 183 | out = "h.." "..w.."!"; print(out) 184 | 185 | 186 | ---------------- 187 | 188 | Compound statements 189 | -------------------- 190 | 191 | ``if`` 192 | ~~~~~~ 193 | 194 | .. code-block:: lua 195 | 196 | if N == 1 then 197 | print("N is one") 198 | elseif N == 2 then 199 | print("N is two") 200 | else 201 | print("N is unknown") 202 | end 203 | 204 | * ``elseif`` and ``else`` are optional 205 | * ``then`` and ``end`` is obligatory 206 | * indentation is not important 207 | 208 | ---------------- 209 | 210 | ``while`` 211 | ~~~~~~~~~ 212 | 213 | .. code-block:: lua 214 | 215 | c = 1 216 | while c <= 10 do 217 | print(c) 218 | c = c + 1 219 | end 220 | 221 | ``break`` 222 | '''''''''' 223 | 224 | * breaks the loop 225 | * must be the last statement in a block 226 | 227 | Blocks 228 | ~~~~~~ 229 | 230 | .. code-block:: lua 231 | 232 | do 233 | 234 | end 235 | 236 | ---------------- 237 | 238 | ``for`` 239 | ~~~~~~~~~~ 240 | 241 | .. code-block:: lua 242 | 243 | for c = 1,10,2 do 244 | print(c) 245 | end 246 | 247 | -- 1,3,5,7,9 248 | 249 | * loop variable cannot be modified 250 | 251 | ``repeat`` 252 | ~~~~~~~~~~~~ 253 | 254 | .. code-block:: lua 255 | 256 | i = 5 257 | repeat 258 | print(i) i=i-1 259 | until i<0 -- 5,4,3,2,1,0 260 | 261 | * the condition is evaluated after the expression 262 | 263 | 264 | 265 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/LuaIntro/11-lua_basics.rst: -------------------------------------------------------------------------------- 1 | =================== 2 | Lua basics 3 | =================== 4 | 5 | ---------------- 6 | 7 | 8 | Visibility 9 | ---------- 10 | 11 | * every variable is global 12 | * ``local`` makes a variable local (in a block) 13 | * variable of the ``for`` is local 14 | * local variable hides the global one with the same name 15 | * the visibility of a local one starts after its declaration 16 | 17 | .. code-block:: lua 18 | 19 | var = 42 20 | do 21 | local var = var -- ??? 22 | var = 24 23 | print(var) 24 | end 25 | print(var) 26 | 27 | --------- 28 | 29 | 30 | String indexing 31 | --------------- 32 | 33 | ``string.sub(s, i [, j])`` 34 | 35 | - indexing starts from ``1`` 36 | - closed interval from both sides 37 | 38 | .. code-block:: lua 39 | 40 | string.sub("Hello Lua user", 7) -- Lua user 41 | string.sub("Hello Lua user", 7, 9) -- Lua 42 | string.sub("Hello Lua user", -8) -- Lua user 43 | string.sub("Hello Lua user", -8, 9) -- Lua 44 | string.sub("Hello Lua user", -8, -6) -- Lua 45 | 46 | -------------- 47 | 48 | Strings 49 | -------- 50 | 51 | - ``string.len("Alma") -- "4"`` 52 | - ``string.format("%d %s", 2, "alma") -- "2 alma"`` 53 | - ``string.upper("Alma") -- "ALMA"`` 54 | - ``string.lower("Alma") -- "alma""`` 55 | - ``string.reverse("Alma") -- " amlA"`` 56 | 57 | Shorter forms: 58 | 59 | .. code-block :: lua 60 | 61 | mystring = "alma" 62 | print(#mystring==mystring:len()) -- true 63 | print(mystring:reverse()) -- "amla" 64 | -- etc. 65 | 66 | http://www.lua.org/manual/5.1/manual.html#5.4 67 | 68 | -------------- 69 | 70 | Math 71 | ---- 72 | 73 | - ``math.cos(num)``, ``math.sin(num)``, ... 74 | - ``math.ceil(num)``, ``math.floor(num)`` -- round 75 | - ``math.exp(x)``, ``math.pow(x,y)`` -- ``e^x`` and ``x^y`` 76 | - ``math.sqrt(x)`` -- ``x^(0.5)`` 77 | - ``math.huge``, ``math.pi`` -- constants 78 | - ``math.log(x)``, ``math.log10(x)`` -- logarithm 79 | - ``math.max(...), math.min(...)`` -- maximum / minimum of the parameters 80 | - ``math.random ([m [, n]])`` -- random number between ``1..m`` **or** ``m..n`` 81 | 82 | http://www.lua.org/manual/5.1/manual.html#5.6 83 | 84 | ---- 85 | 86 | Tables 87 | ------ 88 | 89 | * associative arrays 90 | * can be indexed by any values except ``nil`` 91 | * can contain any type of value 92 | * ``nil`` **as value** means, the element is deleted 93 | 94 | .. code-block:: lua 95 | 96 | empty = {} 97 | myvar = "x" 98 | myint = 42 99 | many = { ["here"]="where", 100 | there=empty; [{}]=0, [42]=24, 101 | [myint]= myvar;} 102 | 103 | * ``[]`` is obligatory for numbers and variables 104 | * string quotes can be omitted 105 | * one can separate with ``,`` or ``;`` 106 | 107 | ------- 108 | 109 | Tables 110 | ------ 111 | 112 | .. code-block:: lua 113 | 114 | print(many["here"]) -- where 115 | many["here"]=0 116 | print(many.here) -- 0 117 | print(many[{}]) -- 0 118 | 119 | 120 | for k,v in pairs(many) do -- generalized for 121 | print(k,v) 122 | end 123 | 124 | * do not modify your table during a loop 125 | 126 | 127 | --------------- 128 | 129 | Tables as "arrays" 130 | ------------------- 131 | 132 | .. code-block:: lua 133 | 134 | array = {'a', 'b', 'c'} 135 | print (#array) -- 3 136 | print(array[1], array[2], array[3]) -- 1 2 3 137 | 138 | for k,v in ipairs(array) do 139 | print(k,v) 140 | end 141 | 142 | array.x = 6 143 | print (#array) -- 3 144 | 145 | - implicit indexing (positive integers from ``1``) 146 | - array and table syntax can be mixed 147 | - ``ipairs(...)`` is used for positive integer indexes 148 | 149 | 150 | 151 | -------------- 152 | 153 | Further "array" methods 154 | ---------------------- 155 | 156 | - ``table.concat(table [, sep [, i [, j]]])`` -- concatenate table content 157 | - ``table.sort(table [, comp])`` -- order with ``comp`` function 158 | - ``table.insert(table, [pos,] value)`` -- "array" insertion 159 | - ``table.remove(table [, pos])`` -- "array" deletion 160 | - ``table.maxn(table)`` -- returns the index of the greatest value 161 | - ``unpack (list, i, j)`` - returns the elements from the given table between the indeces 162 | 163 | .. code-block:: lua 164 | 165 | > a={1,2,3, alma = 1} 166 | > =#a 167 | 3 168 | 169 | http://www.lua.org/manual/5.1/manual.html#5.5 170 | 171 | --------------- 172 | 173 | Standard IO 174 | ------------- 175 | 176 | .. code-block:: lua 177 | 178 | print("Your name:") 179 | line = io.read() 180 | io.write("Hello "..line.."!\n") 181 | 182 | ----------- 183 | 184 | I/O 185 | ---- 186 | 187 | Predefined files 188 | ~~~~~~~~~~~~~~~~~~ 189 | 190 | - ``io.input()``, ``io.output()`` 191 | - ``io.stdin``, ``io.stdout``, ``io.stderr`` 192 | 193 | Operations 194 | ~~~~~~~~~~~ 195 | 196 | - ``io.read([format])`` -- \`\ ``io.input():read()`` 197 | 198 | - ``"*n"``, ``"*a"``, ``"l"``, ``num`` 199 | - returns ``nil`` if fails 200 | - *buffer!* 201 | 202 | - ``io.write(str)`` -- \`\ ``io.output():write(str)``, ... 203 | 204 | -------------- 205 | 206 | 207 | Files 208 | ------ 209 | 210 | - ``file = io.open(filename [, mode])`` : open 211 | 212 | - ``mode``: ``r``, ``w``, ``a``,... 213 | - ``nil`` or file 214 | 215 | - ``io.close(file)``, ``io.flush(file)`` 216 | - ``for line in io.lines(filename) do ... end``: open, iterate, close 217 | - ``io.tmpfile()`` 218 | 219 | - ``io.type(object)`` 220 | - ``file:read()`` 221 | 222 | 223 | ------------- 224 | 225 | ``read`` 226 | ---------- 227 | 228 | - ``file:read("*all")`` 229 | - ``file:read("*line")`` 230 | - ``file:read("*number")`` 231 | - ``n=42; file:read(n)`` 232 | 233 | Command line arguments 234 | ---------------------- 235 | 236 | .. code-block:: lua 237 | 238 | print(arg[0]) -- script name 239 | print(arg[1]) -- first parameter 240 | print(arg[2]) -- ... 241 | 242 | print (#arg) -- number of parameters 243 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/LuaIntro/12-lua_patterns.rst: -------------------------------------------------------------------------------- 1 | Pattern matching in Lua 2 | =================================== 3 | 4 | -------------- 5 | 6 | Similarity (Regexps) 7 | --------------------- 8 | 9 | - subset of the the standard regexps 10 | - basic workflow 11 | 12 | - single character matches itself 13 | - characters with special meaning: ``^$()%.[]*+-?`` 14 | - ``[^ ]`` character sets 15 | - ``*,+,?`` multiple match 16 | 17 | - capture groups 18 | - ``^`` and ``$`` match the start / end of the string 19 | 20 | http://www.lua.org/manual/5.1/manual.html#5.4.1 21 | 22 | -------------- 23 | 24 | Differences(Regexps) 25 | --------------------- 26 | 27 | - escape character ``%`` 28 | - ``[ ]`` predefined sets can be used inside 29 | - ``-`` non greedy matching 30 | - ``%n`` refers to a capturing group 31 | 32 | Character classes 33 | ----------------- 34 | 35 | .. code-block:: none 36 | 37 | %a letters 38 | %d digits 39 | %l lower characters 40 | %s whitespace 41 | %w alfanumeric characters 42 | ... 43 | 44 | -------------- 45 | 46 | Operations 47 | ---------- 48 | 49 | - ``string.match(s, pattern [, init])`` 50 | - matches a string with a pattern 51 | - returns ``nil`` or the matching string(s) 52 | 53 | - ``string.gmatch (s, pattern)`` 54 | - returns an iterator function 55 | 56 | .. code-block:: lua 57 | 58 | s = "hello world from Lua" 59 | for w in string.gmatch(s, "%a+") do 60 | print(w) 61 | end 62 | t = {}, s = "from=world, to=Lua" 63 | for k, v in string.gmatch(s, "(%w+)=(%w+)") do 64 | t[k] = v 65 | end 66 | 67 | -------------- 68 | 69 | Operations 70 | ---------- 71 | 72 | - ``string.find (s, pattern [, init [, plain]])`` 73 | 74 | - returns the beginning and the end of the match 75 | - in case of ``plain = true`` simple string matching 76 | - with **capture groups** it also returns the matched strings 77 | 78 | - ``string.gsub (s, pattern, repl [, n])`` 79 | 80 | - returns a new string and the number of times when it replaced a substring 81 | - possible values of ``repl`` parameter: 82 | 83 | - **string** (%n, %0, %%) 84 | - **table** -- keys are patterns values are replacements 85 | - **functions** -- called when the pattern matches 86 | 87 | -------------- 88 | 89 | .. code-block:: lua 90 | 91 | x = string.gsub("hello world", "(%w+)", "%1 %1") 92 | -- x="hello hello world world" 93 | 94 | x = string.gsub("hello world", "%w+", "%0 %0", 1) 95 | -- x="hello hello world" 96 | 97 | local t = {name="lua", version="5.1"} 98 | x = string.gsub("$name-$version.tar.gz", "%$(%w+)", t) 99 | -- x="lua-5.1.tar.gz" 100 | 101 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/LuaIntro/13-lua_functions.rst: -------------------------------------------------------------------------------- 1 | Functions in Lua 2 | ================ 3 | 4 | ------------------- 5 | 6 | Basics 7 | ------ 8 | 9 | .. code-block:: lua 10 | 11 | function myPrinter(name) 12 | print "hello " .. name 13 | end 14 | 15 | - the only tool for abstraction 16 | - first-class values 17 | - function parameters are locals 18 | - parenthesis can be left in case of one parameter 19 | 20 | -------------- 21 | 22 | Function definitions 23 | -------------------- 24 | 25 | Two possible way of defining functions 26 | 27 | .. code-block:: lua 28 | 29 | function f() 30 | 31 | end 32 | 33 | f = function() 34 | 35 | end 36 | 37 | * builtin names can be overridden 38 | 39 | -------------- 40 | 41 | Parameter passing 42 | ----------------- 43 | 44 | - only positional 45 | - can be called with any number of parameters 46 | - if more than needed: useless parameters are dropped 47 | - if less: uninitialized parameter has ``nil`` values 48 | 49 | .. code-block:: lua 50 | 51 | function f(x,y) 52 | print(x,y) 53 | end 54 | f() -- nil nil 55 | f(1) -- 1 nil 56 | f(1,2,3) -- 1 2 57 | 58 | Vararg functions 59 | ~~~~~~~~~~~~~~~~ 60 | 61 | .. code-block:: lua 62 | 63 | function f(...) -- any number of parameters 64 | a,b = ... 65 | print(a,b,...) 66 | end 67 | f(1,2,3) -- 1 2 1 2 3 68 | 69 | -------------- 70 | 71 | Simulating named parameter 72 | -------------------------- 73 | 74 | .. code-block:: lua 75 | 76 | function foo(args) 77 | print (args.first .. args.last) 78 | end 79 | 80 | foo{first="hello", last="world"} 81 | 82 | ------------- 83 | 84 | 85 | Parameter passing 86 | ----------------- 87 | 88 | * passing by value 89 | * values are objects that are assigned to names 90 | * some types are immutable 91 | 92 | .. code-block:: lua 93 | 94 | function f(a) 95 | a = a .. "!" 96 | end 97 | x = "Hello" 98 | f(x) 99 | print(x) -- Hello 100 | 101 | -------------- 102 | 103 | Parameter passing 104 | ----------------- 105 | 106 | * but some are mutable 107 | 108 | .. code-block:: lua 109 | 110 | function f(t) 111 | t["x"] = 42 112 | end 113 | mt = {} 114 | f(mt) 115 | print(mt["x"]) -- 42 116 | 117 | 118 | -------------- 119 | 120 | Default parameters 121 | ------------------- 122 | 123 | .. code-block:: lua 124 | 125 | function incCount (n) 126 | n = n or 1 127 | count = count + n 128 | end 129 | 130 | Function overloading 131 | --------------------- 132 | 133 | .. code-block:: lua 134 | 135 | function f(a) 136 | print(a) 137 | end 138 | 139 | function f() 140 | print(42) 141 | end 142 | 143 | f("b") -- 42 144 | 145 | -------------- 146 | 147 | Return values 148 | ------------------ 149 | 150 | - ``return`` -- last statement of a block 151 | - without ``return`` or with *empty* ``return`` there is **no return value** 152 | 153 | .. code-block:: lua 154 | 155 | function doNothing() end -- nothing 156 | function doNothing2() return end -- nothing 157 | function getNil() return nil end -- nil 158 | 159 | More than one return value 160 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 161 | 162 | .. code-block:: lua 163 | 164 | function returnArgs(a,b,c) 165 | return a,b,c 166 | end 167 | 168 | -------------- 169 | 170 | Arguments vs. return values 171 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 172 | 173 | .. code-block:: lua 174 | 175 | print(returnArgs(1,2,3)) -- 1 2 3 176 | print("a",returnArgs(1,2,3)) -- "a" 1 2 3 177 | print(returnArgs(1,2,3), "a") -- 1 "a" 178 | print(returnArgs(1,2,3), returnArgs(4,5,6)) -- 1 4 5 6 179 | print(returnArgs(1,2,3), (returnArgs(4,5,6))) -- 1 4 180 | print(doNothing(),2) -- ??? 181 | print(doNothing(), doNothing()) -- ??? 182 | 183 | 184 | - brackets makes returning values treated as there was only one 185 | - if the last parameter a function call: all the returned values are used 186 | - if not, just the first value is used 187 | 188 | -------------- 189 | 190 | Closures 191 | --------- 192 | 193 | - functions that has inner state (*local variables*) 194 | 195 | .. code-block:: lua 196 | 197 | function makeAdd(n) -- in is local for the inner fun. 198 | add = function(m) return n+m end 199 | return add 200 | end 201 | plusone = makeAdd(1) 202 | print(plusOne(1)) -- 2 203 | print (makeAdd(1)(2)) -- 3 204 | 205 | function makeCounter() 206 | local count = 0 -- local for the inner fun. 207 | return function() count = count +1; print(count) end 208 | end 209 | 210 | counter = makeCounter() 211 | counter() counter() counter() -- 1 2 3 212 | 213 | -------------- 214 | 215 | Recursion 216 | ---------- 217 | 218 | - similar to any other imperative language 219 | - *stack overflow* error may occur 220 | - optimized tail recursion 221 | - only with using ``return`` 222 | 223 | .. code-block:: lua 224 | 225 | function rec() 226 | return rec() -- tail recursion 227 | end 228 | 229 | rec() 230 | 231 | -------------- 232 | 233 | Chunks 234 | ------- 235 | 236 | - a code fraction that can be interpreted 237 | 238 | - any line typed in the interpreter (eg. ``for``, ``if``) (plus the following lines if needed) 239 | - a Lua file 240 | 241 | - interpreted => anonymus function 242 | - may contain a ``return`` 243 | 244 | .. code-block:: lua 245 | 246 | > return "hello" 247 | hello 248 | > = "hello" 249 | hello 250 | 251 | ---------- 252 | 253 | Modules 254 | ======= 255 | 256 | - usage *policies* 257 | - ``require`` and ``module`` 258 | - a module is a library, that is loaded with ``require`` and defines a global namespace (that is a table...) 259 | - every exported entity is part of this namespace (table) 260 | - first-class values 261 | - RHS or LHS of an expression 262 | 263 | - it is not a substantive part of the language just a table 264 | 265 | -------------- 266 | 267 | .. code-block:: lua 268 | 269 | -- mod.lua 270 | module(..., package.seeall) 271 | function foo() end 272 | myVar = 42 273 | 274 | .. code-block:: lua 275 | 276 | -- testmod.lua 277 | require "mod" 278 | mod.foo() 279 | 280 | -- testmod2.lua 281 | local m = require "mod" 282 | local f = m.foo 283 | f() 284 | 285 | -------------- 286 | 287 | Modules 288 | ------------------- 289 | 290 | - simplified statement: ``module(..., package.seeall)`` 291 | 292 | - the module name is the filename 293 | - globals are exported 294 | 295 | http://www.lua.org/pil/15.html 296 | 297 | 298 | ``require`` 299 | ~~~~~~~~~~~~ 300 | 301 | - ``require`` forces the interpreter to interpret the module 302 | - a module can be loaded only once 303 | - ``package.loaded`` table contains the loaded modules 304 | 305 | -------------- 306 | 307 | Loading modules 308 | ----------------- 309 | 310 | - there is no directory representation in ANSI C 311 | - *patterns* are used for describe a path 312 | 313 | - ``package.path`` 314 | - ``"?;?.lua;/usr/local/lua/?/?.lua"`` 315 | - ``'?'`` -- modul name, ``';'`` -- separator 316 | 317 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/LuaIntro/14-lua_oop.rst: -------------------------------------------------------------------------------- 1 | OOP in Lua 2 | ========== 3 | 4 | ---- 5 | 6 | Tables and functions 7 | -------------------- 8 | 9 | .. code-block:: lua 10 | 11 | t = {} 12 | 13 | function t.f(x) -- (*) 14 | print(x) 15 | end 16 | 17 | t:f() -- table: 0x12520c0 (**) 18 | t.f() -- nil 19 | 20 | 21 | * ``(*)`` is the same as 22 | * ``t.f = function(x) print(x) end`` or 23 | * ``function t:f() print(t) end`` 24 | * ``(**)`` means ``t.f(t)`` 25 | 26 | ----- 27 | 28 | Objects 29 | ------- 30 | 31 | .. code-block:: lua 32 | 33 | cplx = {re=1, im=1} 34 | 35 | function cplx.add(s, o) 36 | s.re, s.im= s.re + o.re, s.im+o.im 37 | end 38 | 39 | function cplx.str(s) 40 | return s.re.."+"..s.im.."i" 41 | end 42 | 43 | cplx:add(cplx) 44 | print(cplx:str()) 45 | 46 | * ``self`` can be used instead of ``s`` 47 | * ``:`` syntax also works 48 | 49 | ----- 50 | 51 | Simple *classes* 52 | --------------- 53 | 54 | .. code-block:: lua 55 | 56 | Cplx = {} 57 | 58 | function Cplx.new() 59 | return {re = 1, im = 1} 60 | end 61 | 62 | function Cplx.add(s, o) 63 | return {re = s.re + o.re, im =s.im+o.im} 64 | end 65 | 66 | function Cplx.str(s) 67 | return s.re.."+"..s.im.."i" end 68 | 69 | x = Cplx.new() 70 | y = Cplx.add(x,x) 71 | print(Cplx.str(y)) 72 | 73 | ----- 74 | 75 | OOP with Closure approach 76 | ------------------------------------------ 77 | 78 | .. code-block:: lua 79 | 80 | Cplx = { 81 | -- constructor 82 | new = function() 83 | local self = {re=1, im=1} 84 | 85 | -- add method 86 | self.add = function(o) 87 | return {re =self.re + o.re, 88 | im = self.im + o.im} 89 | end 90 | 91 | return self 92 | end, 93 | 94 | -- static methods 95 | get_origo = function() return {re=0, im=0} end, 96 | str = function(s) return s.re.."+"..s.im.."i" end 97 | } 98 | 99 | ---- 100 | 101 | .. code-block:: lua 102 | 103 | x = Cplx.new() 104 | y = x:add(x) 105 | print(Cplx.str(y)) -- 2+2i 106 | print(Cplx.str(Cplx.get_origo())) -- 0+0i 107 | 108 | ----- 109 | 110 | 111 | Metatables, metamethods 112 | ----------------------- 113 | 114 | * metamethods are associated with events, that occur when an operation is executed (e.g.: addition, comparision...) 115 | * only tables and userdata types can have metamethods 116 | * metatable is a regular table containing metamethods 117 | 118 | .. code-block:: lua 119 | 120 | x = {re=1, im =2} 121 | print(x+x) -- error 122 | 123 | mt = { 124 | __add = function(a,b) 125 | return {re=a.re+b.re, im=a.im + b.im} 126 | end 127 | } 128 | setmetatable(x, mt) 129 | 130 | y = x+x; print(y.re, y.im) -- 2 4 131 | 132 | * ``getmetatable`` is used to retrieve an object's metatable 133 | 134 | ----- 135 | 136 | Metatables 137 | ---------- 138 | 139 | * but ``y+y`` won't work 140 | 141 | .. code-block:: lua 142 | 143 | mt = { -- new metatable definition 144 | __add = function(a,b) 145 | return setmetatable({re=a.re+b.re, im=a.im + b.im}, mt) 146 | end 147 | } 148 | 149 | z = y+y 150 | print(z.re, z.im) -- 4 8 151 | 152 | ----- 153 | 154 | Metamethods 155 | ----------- 156 | http://lua-users.org/wiki/MetatableEvents 157 | 158 | .. code-block:: lua 159 | 160 | __index -- read a value at a key 161 | __newindex -- assign a value for a key 162 | __call -- call the table as a function 163 | __metatable -- result of the getmetatable call 164 | __tostring -- print 165 | __unm -- '-' 166 | __add -- '+' 167 | __sub -- '-' 168 | __mul -- '*' 169 | __div -- '/' 170 | __pow -- '^' 171 | __concat -- '..' 172 | __eq -- '==' 173 | __lt -- '<' 174 | __gt -- '>' 175 | 176 | ------- 177 | 178 | Prototypes 179 | ---------- 180 | 181 | .. code-block:: lua 182 | 183 | x = {s="hello"} 184 | y = {} 185 | setmetatable(y, {__index = x}) 186 | print(y.s) 187 | 188 | ---- 189 | 190 | OOP with metatables 191 | ------------------- 192 | 193 | .. code-block:: lua 194 | 195 | Cplx = {} 196 | 197 | function Cplx:new() 198 | return setmetatable({re=1, im=1}, self) 199 | end 200 | 201 | function Cplx:__add(other) 202 | return setmetatable({re=self.re+other.re, 203 | im=self.im+other.im}, 204 | self) 205 | end 206 | 207 | function Cplx:__tostring() 208 | return self.re.."+"..self.im.."i" 209 | end 210 | 211 | x = Cplx:new() 212 | y=x+x 213 | print(y) 214 | 215 | ----- 216 | 217 | .. code-block:: lua 218 | 219 | Cplx = {} 220 | 221 | function Cplx:new() 222 | return setmetatable({re=1, im=1}, Cplx) 223 | end 224 | 225 | function Cplx:__add(other) 226 | return setmetatable({re=self.re+other.re, im=self.im+other.im}, Cplx) 227 | end 228 | 229 | function Cplx:__tostring() 230 | return self.re.."+"..self.im.."i" 231 | end 232 | 233 | setmetatable(Cplx, 234 | {__call = function() 235 | return Cplx:new() end}) 236 | x = Cplx() 237 | y=x+x 238 | print(y) 239 | 240 | ----- 241 | 242 | OOP Questions 243 | -------------- 244 | 245 | * instances 246 | * methods, attributes 247 | * static methods / attributes 248 | * (multiple-) inheritance 249 | * information hiding 250 | * polymorphism 251 | 252 | http://lua-users.org/wiki/ObjectOrientedProgramming 253 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/1-introduction.rst: -------------------------------------------------------------------------------- 1 | =================== 2 | Scripting languages 3 | =================== 4 | 5 | ---- 6 | 7 | About the lectures 8 | ------------------ 9 | 10 | ------- 11 | 12 | Aims 13 | ---- 14 | 15 | * to get a general picture about scripting languages 16 | * basic knowledge in programming Python / Lua 17 | * ability 18 | * to learn new scripting languages 19 | * to use new technologies 20 | 21 | ---- 22 | 23 | Topics 24 | -------- 25 | 26 | * Scripting languages in general 27 | * Python language 28 | * basics 29 | * data structures, functions 30 | * modules, packages, I/O 31 | * OOP 32 | * Pattern matching 33 | * (GUI) 34 | * Lua 35 | * basics 36 | * *OOP* 37 | 38 | ---- 39 | 40 | Schedule 41 | -------- 42 | 43 | 11 lectures in the semester, not involving: 44 | 45 | * 10 April -- out of office 46 | * 17,24 April -- Holiday 47 | 48 | Test: first week of June 49 | 50 | ----- 51 | 52 | Grading 53 | ------------- 54 | 55 | * presence on classes 56 | * 1 final test (programming) 57 | * 1-2 programming task (this can be your own project as well) 58 | * regular homeworks (necessary!) 59 | 60 | Contact 61 | ------- 62 | http://users.itk.ppke.hu/~oroszgy 63 | 64 | oroszgy@itk.ppke.hu 65 | 66 | ---- 67 | 68 | Revision 69 | -------- 70 | 71 | Definitions 72 | ~~~~~~~~~~~ 73 | 74 | * compiler 75 | * compiled programming languages 76 | * assembly 77 | * interpreter 78 | * interpreted languages 79 | * bytecode 80 | * virtual machine 81 | * garbage collector 82 | 83 | ---- 84 | 85 | Strong vs. weak type system 86 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 87 | 88 | Strong 89 | ====== 90 | 91 | "there are no loopholes in the type system" 92 | 93 | Weak 94 | ==== 95 | "the type system can be subverted (invalidating any guarantees)" 96 | 97 | Read `this! `_ 98 | 99 | ----- 100 | 101 | Dynamic vs. static type system 102 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 103 | 104 | Static 105 | ======= 106 | 107 | "the type of each value is checked before the code being executed" (C/C++, JAVA, ...) 108 | 109 | Dynamic 110 | ======= 111 | 112 | "the type of values are checked during the execution" (PHP, Python, ...) 113 | 114 | -------- 115 | 116 | Examples 117 | ~~~~~~~~ 118 | 119 | C++ 120 | === 121 | * static typing 122 | * weak type system (e. g. casting a pointers to int!) 123 | * compiler(s) 124 | 125 | JAVA 126 | ==== 127 | * static type system 128 | * strongly typed 129 | * compiled (into bytecode) 130 | * virtual machine executes the bytecode 131 | 132 | ----- 133 | 134 | Examples 135 | ~~~~~~~~ 136 | 137 | Python 138 | ====== 139 | * dynamically typed 140 | * strongly typed 141 | * interpreter interprets the bytecode 142 | 143 | ----- 144 | 145 | Scripting languages 146 | ------------------- 147 | 148 | ------ 149 | 150 | 151 | General attributes 152 | ----------------------- 153 | 154 | Historical purpose: running applications with parameters 155 | 156 | Nowadays: 157 | 158 | * rapid developing, prototyping 159 | * productivity 160 | * easy integration 161 | * simple syntax 162 | * interpreted -- interactive interpeter 163 | * dynamic typing 164 | * weak type system 165 | * automatic memory handling (GC) 166 | 167 | ---- 168 | 169 | Applications 170 | ------------ 171 | * OS facilities 172 | * Web 173 | * rapid prototyping in sciences 174 | * AI 175 | * NLP 176 | * bioinforamtics 177 | * scripting 178 | * games 179 | * graphics 180 | * GUI 181 | * gluing 182 | 183 | 184 | ---- 185 | 186 | Scripting languages 187 | ------------------- 188 | 189 | * PHP, JavaScript, ActionScript 190 | * R, Ruby 191 | * Scala, Groovy 192 | * Perl, Python, Lua 193 | * Lisp, 194 | * Rexx, TCL, BASH 195 | * ... 196 | 197 | ---- 198 | 199 | Importance 200 | ----------------------------------- 201 | 202 | .. image:: tiobe.png 203 | :scale: 80% -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/2-python_basics.rst: -------------------------------------------------------------------------------- 1 | Python basics 2 | ============= 3 | 4 | ---- 5 | 6 | Python 7 | ------ 8 | 9 | * general purpose programming language 10 | * high level 11 | * dynamic but strong typing 12 | * platform independent 13 | * object-oriented 14 | * partly support for functional paradigm 15 | * garbage collector 16 | 17 | ---- 18 | 19 | Duck typing 20 | ~~~~~~~~~~~~ 21 | 22 | *"When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck."* -- James WhitComb Riley 23 | 24 | .. code-block:: python 25 | 26 | def f(something): 27 | for e in something: 28 | print(e) 29 | 30 | f(open("input.txt")) 31 | f([1,2,3]) 32 | f("hello") 33 | 34 | *"Simply stated: provided you can perform the job, we don't care who your parents are."* 35 | 36 | ----- 37 | 38 | Usage 39 | -------- 40 | 41 | * general purpose 42 | * system programming 43 | * rapid GUI modelling 44 | * prototyping 45 | * text processing 46 | * research 47 | * *scipy* 48 | * *numpy* 49 | * *nltk* 50 | * ... 51 | * mobile development 52 | 53 | ---- 54 | 55 | Install the interpreter! 56 | ------------------------- 57 | 58 | ---- 59 | 60 | ``import this`` 61 | --------------- 62 | 63 | .. sourcecode:: none 64 | 65 | Beautiful is better than ugly. 66 | Explicit is better than implicit. 67 | Simple is better than complex. 68 | Complex is better than complicated. 69 | Flat is better than nested. 70 | Sparse is better than dense. 71 | Readability counts. 72 | Special cases aren't special enough to break the rules. 73 | Although practicality beats purity. 74 | Errors should never pass silently. 75 | Unless explicitly silenced. 76 | In the face of ambiguity, 77 | refuse the temptation to guess. 78 | There should be one -- and preferably only one 79 | -- obvious way to do it. 80 | 81 | 82 | ---- 83 | 84 | ``import this`` 85 | --------------- 86 | 87 | .. sourcecode:: none 88 | 89 | Although that way may not be obvious at first 90 | unless you're Dutch. 91 | Now is better than never. 92 | Although never is often better than *right* now. 93 | If the implementation is hard to explain, 94 | it's a bad idea. 95 | If the implementation is easy to explain, 96 | it may be a good idea. 97 | Namespaces are one honking great idea -- 98 | let's do more of those! 99 | 100 | 101 | ---- 102 | 103 | Development environment 104 | ----------------------- 105 | 106 | We use Python **3.x** (and not 2.x)! 107 | 108 | IDE: 109 | 110 | * **Pycharm** (Free for academic users) 111 | * **Notepad++** 112 | * Spyder 113 | * IDLE 114 | * *Eclipse+PyDev* 115 | 116 | Documentation 117 | ~~~~~~~~~~~~~ 118 | * http://docs.python.org/3/ 119 | * http://www.diveintopython3.org/ 120 | * Programming in Python 3 121 | 122 | ---- 123 | 124 | Interpreting 125 | --------------- 126 | 127 | * several interpreters available 128 | * **CPython**, PyPy, Stackless Python 129 | * Jython, Pyjamas, IronPython, RPython 130 | 131 | Cython 132 | ~~~~~~ 133 | "bytecode interpreter":: 134 | 135 | hello.py --> hello.pyc -- |VM| 136 | generating bytecode 137 | interpreting 138 | 139 | ---- 140 | 141 | Basic types 142 | ---------------- 143 | 144 | * everything is an object, thus has methods and attributes 145 | * ``object`` is the base class 146 | * names/identifiers instead of variables ``=>`` everything is a *reference* 147 | * mutable vs. immutable objects 148 | * ``type(x)``, ``dir(x)`` type of an object and all accessible methods, attributes 149 | 150 | ---- 151 | 152 | Numbers 153 | ------- 154 | * ``int`` type for integers 155 | * ``float(3.14)`` 156 | * ``complex(1+2j)`` 157 | * constructor / conversion: ``int()`` / ``float()`` / ``complex()`` 158 | * operators: "``+,-,/,//,*,%, **``" and ``pow(x,y), divmod(x,y)`` 159 | * additional ``math`` library, bitwise operations 160 | 161 | ---- 162 | 163 | Boolean 164 | ------- 165 | 166 | * subtype of ``int`` 167 | * literals: ``True, False`` 168 | * basic operations: ``and or not`` 169 | * constructor / conversion: ``bool()`` 170 | * "``<,>,==,!=, <=, >=, is, is not``" operators result boolean 171 | * everything can be evaluated as a boolean 172 | 173 | ``NoneType`` type 174 | ----------------- 175 | 176 | * the only literal is "``None``" 177 | * always evaluates to "``False``" 178 | 179 | ---------------------------- 180 | 181 | Equality 182 | ~~~~~~~~ 183 | 184 | As expected: 185 | 186 | .. code-block:: python 187 | 188 | a = 1000 189 | b = 999 +1 190 | >>> a is b 191 | False 192 | >>> a == b 193 | True 194 | 195 | But: 196 | 197 | .. code-block:: python 198 | 199 | c = 2 200 | b = 1 201 | >>> b + 1 is c 202 | True 203 | 204 | Try: 205 | 206 | .. code-block:: python 207 | 208 | a = "I have a cat" 209 | b = "I have a cat" 210 | >>> a is b 211 | False 212 | 213 | a=sys.intern("I have a cat") 214 | b=sys.intern("I have a cat") 215 | >>> a is b 216 | True 217 | 218 | https://realpython.com/python-is-identity-vs-equality/ 219 | 220 | ------ 221 | 222 | Strings 223 | -------- 224 | 225 | http://docs.python.org/3/library/string.html 226 | 227 | .. code-block:: python 228 | 229 | b = 'hi!' 230 | a = "Hello World!" 231 | len(a) # 12 232 | a[6] # "W" 233 | str(1) # "1" 234 | b = ' :)', a+b # "Hello World! :)" 235 | a.find("o") # 5 236 | a.rfind("o") # 7 237 | a.split() # ['Hello', 'World!'] 238 | a.lower() # 'hello world!' 239 | a.upper() # 'HELLO WORLD!' 240 | a.replace("World", "ITK") #' Hello ITK!' 241 | print("Hello {}. {}!".format(1, "Arthur")) 242 | 243 | * multi-line strings with ``"""`` or ``'''`` 244 | 245 | 246 | ---------------------------- 247 | 248 | 249 | Slicing 250 | ~~~~~~~~~~ 251 | 252 | .. code-block:: python 253 | 254 | myStr = "Hello" 255 | 256 | :: 257 | 258 | +---+---+---+---+---+ 259 | | H | e | l | l | o | 260 | +---+---+---+---+---+ 261 | 0 1 2 3 4 5 262 | -5 -4 -3 -2 -1 263 | 264 | .. code-block:: python 265 | 266 | myStr[1:] 267 | myStr[:-1] 268 | myStr[2:5] 269 | myStr[2:-2] 270 | myStr[:] 271 | 272 | ---------------------------- 273 | 274 | 275 | Type hierarchy of Python (2.x) 276 | ------------------------------ 277 | 278 | .. image:: types.gif 279 | :scale: 100% 280 | 281 | http://docs.python.org/3/reference/datamodel.html#types 282 | 283 | ---- 284 | 285 | Indentation 286 | ------------ 287 | 288 | **Blocks are marked with colons (:) and with the indentation itself!** 289 | 290 | * Wrong indentation yields error! 291 | * Do not mix tabs and spaces! 292 | 293 | **But** not all sort of whitespace are significant: 294 | 295 | .. code-block:: python 296 | 297 | mylist = [ 298 | "some string", 299 | "and another", 300 | "and finally this", 301 | ] 302 | 303 | mystring = 'this is ' \ 304 | 'a very long string ' \ 305 | 'that is split' \ 306 | 'across multiple lines' 307 | 308 | 309 | 310 | ---- 311 | 312 | Statements 313 | ---------- 314 | 315 | * ``pass`` 316 | * assignment with "``=``", and "``x,y = y,x``" also works 317 | * use a modul: ``import modul`` 318 | * ``del o`` 319 | 320 | * read std. input: ``input()`` function 321 | 322 | ------ 323 | 324 | Comments 325 | -------- 326 | 327 | .. code-block:: python 328 | 329 | 330 | 331 | a = 42 # After the hashmark... 332 | 333 | # This is a single line comment. 334 | 335 | """ This is a multi- 336 | line comments 337 | """ 338 | 339 | """ 340 | This works as well. 341 | """ 342 | 343 | ---------------------------- 344 | 345 | Sequence of statements 346 | ----------------------- 347 | 348 | * separate statements with new line or "``,``" "``;``" 349 | * one statement is in one line except if ends ``\`` 350 | 351 | .. code-block:: python 352 | 353 | a = 42 # nice number 354 | c,b = 1,2 355 | a = s.split( \ 356 | " ") 357 | 358 | 359 | ----------------------------------------- 360 | 361 | 362 | ``if`` 363 | ------ 364 | 365 | .. code-block:: python 366 | 367 | if conditon1: 368 | ... 369 | elif conditon2: 370 | ... 371 | else: 372 | ... 373 | 374 | * no ``switch`` or ``case`` statement 375 | * no need of brackets 376 | * no restriction for the condition 377 | 378 | ---- 379 | 380 | Evaluating 381 | ~~~~~~~~~~ 382 | 383 | * everything can be evaluated as boolean 384 | * lazy strategy 385 | * the results of "``and or not``" are not ``True`` or ``False``, rather the dominating expression 386 | * ``if x == True: ...`` ~ ``if a: ...`` 387 | * ``if x != None: ...`` ~ ``if not x: ...`` 388 | 389 | 390 | ----------------------------------------- 391 | 392 | Loops 393 | -------- 394 | 395 | .. code-block:: python 396 | 397 | while condition: 398 | ... 399 | else: 400 | # optional block 401 | 402 | for e in iterable_element: 403 | ... 404 | else: 405 | # optional block 406 | 407 | Others: 408 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 409 | ``"else"`` is only executed when the ``condition`` becomes ``False`` 410 | 411 | * ``break`` -- break the loop 412 | * ``continue`` -- skip an iteration step 413 | 414 | ---- 415 | 416 | 417 | Python program structure 418 | ------------------------ 419 | 420 | Command line processing in Linux: ``#!/usr/bin/env python`` 421 | 422 | .. code-block:: python 423 | 424 | #!/usr/bin/env python 425 | 426 | if __name__ == "__main__": 427 | print("Hello World!") 428 | 429 | 430 | 431 | 432 | 433 | 434 | 435 | 436 | 437 | 438 | 439 | 440 | 441 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/3-python_data_structures.rst: -------------------------------------------------------------------------------- 1 | Basic data structures in Python 2 | =============================== 3 | 4 | ---- 5 | 6 | ``list`` 7 | -------- 8 | 9 | a **mutable** sequence `type `_ 10 | https://realpython.com/python-lists-tuples/ 11 | 12 | .. code-block:: python 13 | 14 | empty1, empty2 = [], list() 15 | a = [True, 1, "alma", 5.0, []] 16 | 1 in a # True 17 | a[2] = "szilva" 18 | a[2] # "szilva" 19 | a.append(-1) # add an element to the end 20 | a.extend([1,2]) # extends the list with another 21 | a + b # creates a new list 22 | del a[3] # remove the item at the index 23 | range(0,5) # [0, 1, 2, 3, 4] 24 | range(5) # same as above 25 | l = list("abc") # ["a", "b", "c"] 26 | " ".join(l) # "a b c" 27 | [1,2,3,4,5,6,7,8,9,10][1::2] # [2,4,6,8,10] 28 | 29 | What is the difference between ``sorted(l)`` vs. ``l.sort()``? 30 | 31 | -------- 32 | 33 | 34 | ``range`` 35 | ---------- 36 | 37 | an **immutable** sequence `type `_ of **numbers** 38 | 39 | .. code-block:: python 40 | 41 | list(range(3)) # [0,1,2] 42 | list(range(1,3)) # [1,2] 43 | r = list(range(0,5,2)) # [0,2,4] 44 | 10 in r # False 45 | r[1] # 5 46 | r[1:] # range(2,5,2) 47 | 48 | ----- 49 | 50 | 51 | 52 | ``set`` 53 | -------- 54 | 55 | "unordered collection of distinct objects" 56 | https://realpython.com/python-sets/ 57 | 58 | * **mutable** `type `_ 59 | * each element is `hashable `_ 60 | 61 | .. code-block:: python 62 | 63 | s = {1,2,3} 64 | s = set([1,2,3]) # create new 65 | s.add(x) # add element 66 | s.remove(x) # remove element 67 | x in s # contains? 68 | s | t # union 69 | s & t # intersect 70 | s - t # symmetric difference 71 | s <= t # is subset? 72 | 73 | * but ``frozenset`` is an **immutable** type 74 | 75 | ----------- 76 | 77 | ``tuple`` 78 | ---------- 79 | 80 | "fix number of items in a definite order" 81 | 82 | * **immutable** sequence type 83 | 84 | .. code-block:: python 85 | 86 | t0 = () 87 | t = (1,"a", None, ()) 88 | t[1] # "a" 89 | tuple([1,2,3]) # (1,2,3) 90 | list((1,2,3)) # [1,2,3] 91 | 92 | t0, t1, t2, t3 = t 93 | a, *b = t # 1, ["a", None, ()] 94 | a, b, *c = t # 1, "a", [None, ()] 95 | a, *b, c = t # 1, ["a", None], () 96 | 97 | ------------- 98 | 99 | ``dict`` 100 | -------- 101 | 102 | 103 | "efficient for storing of key-value pairs" 104 | https://realpython.com/python-dicts/ 105 | 106 | * **mutable** type 107 | * with `many methods `_ 108 | * ~ hash-table, associative array, dictionary, table, mapping types 109 | * a key value must be `hashable `_ 110 | * access methods returns `view objects `_ 111 | ---- 112 | 113 | Basic methods: 114 | 115 | .. code-block:: python 116 | 117 | dt = {}; dt2 = dict() 118 | d = {1 : 2, "a" : 1.0, None : "alma"} 119 | dt[1] = "alma"; print d["a"] 120 | del d["a"] # remove element with key 121 | d.keys() # view of keys 122 | d.values() # view of values 123 | x in d # contains key? 124 | 125 | for key,value in d.items(): # view object of (key,value) tuples 126 | print key,value 127 | 128 | 129 | ---------- 130 | 131 | List comprehension 132 | ------------------ 133 | 134 | https://www.datacamp.com/community/tutorials/python-list-comprehension 135 | 136 | * compact syntax 137 | * generate lists on demand 138 | * process an existing list easily 139 | 140 | General formalism: 141 | 142 | .. code-block:: python 143 | 144 | [ expression(e1, e2...) 145 | for e1 in container1 if condition1 146 | for e2 in container2 if condition2 147 | ... ] 148 | 149 | ---- 150 | 151 | Examples: 152 | 153 | .. code-block:: python 154 | 155 | """ even cube numbers """ 156 | [x*x for x in range(10) if x % 2 == 0 ] 157 | 158 | """ 3x3 matrix coordinates """ 159 | [ [ (x,y) for y in range(3) ] for x in range(3) ] 160 | 161 | ----- 162 | 163 | ``set`` comprehension 164 | --------------------- 165 | 166 | Examples: 167 | 168 | .. code-block:: python 169 | 170 | """ set of short words""" 171 | { w for w in words if len(words) < 5} 172 | 173 | """ intersection of two lists""" 174 | { e1 for e1 in list1 for e2 in list2 if e1==e2 } 175 | 176 | ---- 177 | 178 | ``dict`` comprehension 179 | ----------------------- 180 | 181 | Examples: 182 | 183 | .. code-block:: python 184 | 185 | """ generate a word index """ 186 | mdcit = { word:index for index, word in enumerate(words)} 187 | 188 | """ transform an existing dictionary""" 189 | newdict = { k-1: v.lower() for k,v in mdict.items()} 190 | 191 | 192 | ------------ 193 | 194 | Command line arguments 195 | ---------------------- 196 | 197 | ``sys.argv`` is a **list** which contains the command line arguments 198 | 199 | .. code-block:: python 200 | 201 | # hello.py 202 | 203 | import sys 204 | print("Hello", sys.argv[1]) 205 | 206 | Running from the shell: 207 | 208 | .. code-block:: none 209 | 210 | $ python hello.py "John" 211 | Hello John 212 | 213 | 214 | 215 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/4-python-functions-1.rst: -------------------------------------------------------------------------------- 1 | Functions in Python 2 | =================== 3 | 4 | ---- 5 | 6 | Defining functions 7 | -------------------- 8 | 9 | * there is no distinction between methods and functions *(side-effect)* 10 | * all functions are objects 11 | 12 | Syntax: 13 | 14 | * ``def`` statement is used to create function 15 | * semicolon(``':'`` ) is necessary to start the body 16 | * indentation is important! 17 | * no type definition! (dynamically typed language) 18 | 19 | ---- 20 | 21 | Examples: 22 | 23 | .. code-block:: python 24 | 25 | def myadd(par1, par2): 26 | return par1 + par2 27 | 28 | def mysum(mylist): 29 | s = 0 30 | for e in mylist: 31 | s+=e 32 | return e 33 | 34 | def donothing(par1): pass 35 | 36 | print(myadd(1,mysum([2,3,4]))) 37 | 38 | ---- 39 | 40 | But, what happens? 41 | 42 | .. code-block:: python 43 | 44 | def myfun(a,b): 45 | print(a) 46 | 47 | def myfun(a,b): 48 | print(b) 49 | 50 | # The output is ... 51 | myfun 52 | myfun(1) 53 | myfun = 42 54 | myfun 55 | myfun(1) 56 | 57 | 58 | ---- 59 | 60 | 61 | Returning value 62 | ----------------------------- 63 | 64 | * ``return`` statement is used 65 | * the default return value is ``None`` 66 | 67 | .. code-block:: python 68 | 69 | def f(): 70 | pass 71 | 72 | print(f()) # None 73 | 74 | * returning several values is possible 75 | 76 | .. code-block:: python 77 | 78 | def multifun(a,b): 79 | return a+b, a-b 80 | 81 | print(f(3,2)) # (5,1) 82 | 83 | 84 | 85 | ---------------------------------- 86 | 87 | 88 | Passing parameters 1. 89 | ---------------------- 90 | 91 | * parameters are local variables 92 | * dynamically typed 93 | * their references are passed 94 | * mutable vs immutable case! 95 | 96 | 97 | ---- 98 | 99 | Examples: 100 | 101 | .. code-block:: python 102 | 103 | def f(x): 104 | x=42 105 | print(x) 106 | a=0 107 | f(a) # 42 108 | print(a) # 0 109 | 110 | """This is the so called duck-typing""" 111 | def foo(x): 112 | x+="a" 113 | 114 | a = "hello"; 115 | l = [42,0] 116 | foo(a); print(a) # "hello" 117 | foo(l); print(l) # [42, 0, "a"] 118 | 119 | 120 | 121 | ---------------------------------- 122 | 123 | Passing parameters 2. 124 | ---------------------- 125 | 126 | * passing by order 127 | * default parameters: 128 | 129 | .. code-block:: python 130 | 131 | def inc(x, plus = 1): 132 | return x + plus 133 | 134 | * parameter passing by keywords: 135 | 136 | .. code-block:: python 137 | 138 | inc(plus=5, x=6) 139 | 140 | * passing by order and by keyword can be mixed 141 | 142 | ---------------------------------- 143 | 144 | Default parameter 145 | ~~~~~~~~~~~~~~~~~ 146 | 147 | What is the expected behavior? 148 | 149 | .. code-block:: python 150 | 151 | def function(data=[]): 152 | data.append(1) 153 | return data 154 | 155 | print(function()) # [1] 156 | print(function()) # [1, 1] 157 | print(function()) # [1, 1, 1] 158 | 159 | But: 160 | 161 | .. code-block:: python 162 | 163 | def function(data=None): 164 | data = data or [] 165 | data.append(1) 166 | return data 167 | 168 | print(function()) # [1] 169 | print(function()) # [1] 170 | 171 | ------------------- 172 | 173 | Scope and lifetime 174 | ------------------ 175 | 176 | The lifetime of an object depends on the GC. 177 | 178 | A variable scope is its namespace (and the enclosed ones): 179 | 180 | * names organized in namespaces 181 | * modules and functions defines a namespaces 182 | * a namespace can be global or local 183 | * every name is only accessible from its namespace 184 | * globals, builtins might be hidden 185 | * name resolution order: local → *enclosing* → global → builtin 186 | 187 | ----------------------------------- 188 | 189 | Local and global variables 190 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 191 | 192 | * local variables: 193 | * arguments of functions 194 | * variable defined in the body of a code block (function, loop, selection) 195 | * loop variable 196 | * variables declared in the module called globals (but may be hidden) 197 | 198 | .. code-block:: python 199 | 200 | var = 42 201 | for var in range(5): 202 | print(var) # 0, 1, 2, 3, 4 203 | -------- 204 | 205 | 206 | ``global`` statement can be used to change the value of a global variable 207 | 208 | .. code-block:: python 209 | 210 | x = 42 211 | 212 | def h1(y): 213 | x = y 214 | 215 | def h2(y): 216 | global x; 217 | x = y 218 | 219 | h1(1); print(x) # 42 220 | h2(1); print(x) # 1 221 | 222 | 223 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/4-python-functions-2.rst: -------------------------------------------------------------------------------- 1 | Advanced functions 2 | =================== 3 | 4 | ---------------------------------- 5 | 6 | 7 | Function as parameter 8 | ---------------------- 9 | 10 | * ``def`` is a statement, that creates an object 11 | * the name of the object is the function name 12 | * any object can be passed as a parameter 13 | 14 | .. code-block:: python 15 | 16 | def calc(x): 17 | return (x + 2) / 10 18 | 19 | def f(do, with): 20 | return do(with) 21 | 22 | print(f) # 23 | print(f(calc, 8)) # 1 24 | 25 | ---------------------------------- 26 | 27 | Nested Functions 28 | ~~~~~~~~~~~~~~~~~ 29 | 30 | * ``def`` is just a statement 31 | * you can create an object in the body of a function 32 | 33 | .. code-block:: python 34 | 35 | def printer(*args): 36 | def nice_printer(x): 37 | print ("--%s--"%(x,)) 38 | for a in args: 39 | nice_printer(a) 40 | 41 | printer(1,2,3) 42 | 43 | ------------------- 44 | 45 | 46 | Anonymous functions 47 | ------------------- 48 | 49 | ``lambda`` is a statement for defining function objects 50 | 51 | 52 | 53 | Syntax: 54 | 55 | .. code-block:: python 56 | 57 | f = lambda parameters: expression(parameters) 58 | 59 | print(f) # at 0x7f57dba11320> 60 | 61 | f(param) 62 | 63 | 64 | ------------- 65 | 66 | Both type of functions do the same: 67 | 68 | .. code-block:: python 69 | 70 | def calc1(x): return (x + 2) / 10 71 | calc2 = lambda x: (x + 2) / 10 72 | print (calc1(42) == calc2(42)) # True 73 | 74 | Sometimes ``lambda`` can shorten your code: 75 | 76 | .. code-block:: python 77 | 78 | def f(do, _with): return do(_with) 79 | print (f(lambda x: (x + 2) / 10, 8)) # 1 80 | 81 | A more complex example: 82 | 83 | .. code-block:: python 84 | 85 | f = lambda *x: [ e+1 for e in x] 86 | print(f(1,2,3)) # [2,3,4] 87 | 88 | ---- 89 | 90 | Local and global variables 91 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 92 | 93 | * local variables: 94 | * arguments of functions 95 | * variable defined in the body of a code block (function, loop, selection) 96 | * loop variable 97 | * variables declared in the module called globals (but may be hidden) 98 | 99 | .. code-block:: python 100 | 101 | var = 42 102 | for var in range(5): 103 | print(var) # 0, 1, 2, 3, 4 104 | 105 | -------- 106 | 107 | 108 | ``global`` statement can be used to change the value of a global variable 109 | 110 | .. code-block:: python 111 | 112 | x = 42 113 | 114 | def h1(y): 115 | x = y 116 | 117 | def h2(y): 118 | global x; 119 | x = y 120 | 121 | h1(1); print(x) # 42 122 | h2(1); print(x) # 1 123 | 124 | -------- 125 | 126 | Functions parameters 127 | ---------------------- 128 | 129 | Call 130 | ~~~~~~ 131 | 132 | * ``fun(val)`` -- passing by position 133 | * ``fun(name=val)`` -- passing by keyword 134 | 135 | Define 136 | ~~~~~~ 137 | 138 | * ``def fun(name)`` 139 | * ``def fun(name=val)`` -- default value 140 | * ``def fun(*name)`` -- rest of the positional parameters 141 | * ``def fun(**name)`` -- rest of the parameters by keywords 142 | 143 | ---------------------------------- 144 | 145 | Examples: 146 | 147 | .. code-block:: python 148 | 149 | def mysum(*args): 150 | s = 0 151 | for val in args: # args is a tuple 152 | s+=val 153 | return s 154 | 155 | print(mysum(1,2,3)) # 6 156 | 157 | def add(**kwargs): 158 | # kwargs is a dictionary 159 | if "x" in kwargs and "y" in kwargs: 160 | return kwargs["x"] + kwargs["y"] 161 | 162 | print(add(x=3, y=6)) # 9 163 | 164 | --------------- 165 | 166 | .. code-block:: python 167 | 168 | def complicated(first, second=None, *args, **kwargs): 169 | print(first, second) 170 | print(args) 171 | print(kwargs) 172 | 173 | complicated(42, "non-empty", 1,2,3, x=0, y=1) 174 | 175 | """ 176 | The result is: 177 | 42 non-empty 178 | (1, 2, 3) 179 | {'y': 1, 'x': 0} 180 | """ 181 | 182 | 183 | --------------- 184 | 185 | 186 | Built-in functions 187 | ------------------ 188 | 189 | Efficiency counts! 190 | 191 | * ``len()`` 192 | * ``max()``, ``min()`` 193 | 194 | * ``eval()`` 195 | 196 | * ``all(), filter(), map()`` 197 | * ``reduce(), sum()`` 198 | * ``zip()`` 199 | * ``next()`` 200 | * ... 201 | 202 | http://docs.python.org/3/library/functions.html 203 | 204 | ---- 205 | 206 | Built-in functions 1. 207 | --------------------- 208 | 209 | ``map`` 210 | ~~~~~~~ 211 | 212 | Applies a function for each element of a list 213 | 214 | .. code-block:: python 215 | 216 | >>> map(str.lower, ["A", "B"]) 217 | ['a', 'b'] 218 | >>> map(max, [[4,2,0], [1,3,5]]) 219 | [4, 5] 220 | 221 | ``filter`` 222 | ~~~~~~~~~~ 223 | 224 | Constructs a new iterable from another keeping elements whose function returns ``True`` 225 | 226 | .. code-block:: python 227 | 228 | >>> filter(str.islower, ["a", "A", "1"]) 229 | ['a'] 230 | >>> filter(lambda x: x, [0, 1, [], None, {}]) 231 | [1] 232 | 233 | ---- 234 | 235 | Built-in functions 2. 236 | --------------------- 237 | 238 | ``sum`` 239 | ~~~~~~~ 240 | 241 | * summarize all elements of an iterable 242 | 243 | .. code-block:: python 244 | 245 | >>> sum(range(5)) 246 | 10 247 | >>> sum([[1], [2,3], [4,5,6]], []) 248 | [1, 2, 3, 4, 5, 6] 249 | 250 | ``reduce`` 251 | ~~~~~~~~~~ 252 | 253 | * apply a function of two arguments cumulatively 254 | 255 | .. code-block:: python 256 | 257 | >>> from functools import reduce 258 | >>> reduce(lambda x,y: x*y, range(1,5)) 259 | 24 260 | 261 | Evaluated as: ``((((1*2)*3)*4)*5)`` 262 | 263 | ---- 264 | 265 | Built-in functions 3. 266 | --------------------- 267 | 268 | ``zip`` 269 | ~~~~~~~ 270 | 271 | Creates an iterator of tuples from iteratables 272 | 273 | .. code-block:: python 274 | 275 | >>> x = [1, 2, 3] 276 | >>> y = ["a", "b", "c"] 277 | >>> zipped = zip(x, y) 278 | >>> list(zipped) 279 | [(1, "a"), (2, "b"), (3, "c")] 280 | 281 | The other direction: 282 | 283 | .. code-block:: python 284 | 285 | >>> x2, y2 = zip(*zipped) 286 | >>> x == list(x2) and y == list(y2) 287 | True 288 | 289 | ---- 290 | 291 | 292 | Random numbers 293 | --------------- 294 | 295 | http://docs.python.org/3/library/random.html 296 | 297 | .. code-block:: python 298 | 299 | from random import * 300 | 301 | print(choice(["a", None, 1])) # either of the three element 302 | 303 | print(random()) # a float number from [0.0, 1.0) 304 | 305 | print(randint(1,10)) #an integer from [1,10] 306 | 307 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/5-python-modules.rst: -------------------------------------------------------------------------------- 1 | Modules, packages 2 | ======================== 3 | 4 | Modules 5 | ------- 6 | 7 | * all ``.py`` files are modules (compiling units) 8 | * interpreting a module means *interpreting the given file* 9 | * ``import`` statement interprets a module and adds its element to the namespace 10 | * they are accessible by their name (e. g. ``modul.foo()``) 11 | * one can import a module if its path is in the ``sys.path`` 12 | 13 | ----- 14 | 15 | .. code-block:: python 16 | 17 | """mymath.py""" 18 | def myadd(a,b): 19 | return a+b 20 | def mymul(a,b): 21 | return a*b 22 | 23 | """myapp.py""" 24 | import mymath 25 | mymath.myadd(1,2) 26 | 27 | from mymath import myadd 28 | myadd(1,2) 29 | mymath.mymul(1,2) # Error 30 | 31 | from mymath import * 32 | myadd(1,2) 33 | mymul(1,2) 34 | 35 | 36 | ------------- 37 | 38 | 39 | But the *__main__* environment is skipped during importing 40 | 41 | .. code-block:: python 42 | 43 | print("this is run and printed") 44 | 45 | if __name__ == "__main__": 46 | print("While this condition is not True, " \ 47 | "thus the interpreter skip this section.") 48 | 49 | --------------- 50 | 51 | - All the names from a module is exported not started with an "_". 52 | - One can overwrite the exported names with using "__all__". 53 | 54 | .. code-block:: python 55 | 56 | """mymodule.py""" 57 | 58 | __all__ = ["_NOT_SO_HIDDEN_NAME"] 59 | 60 | _NOT_SO_HIDDEN_NAME = 42 61 | 62 | 63 | --------------- 64 | 65 | Packages 66 | -------- 67 | 68 | Packages are directories, containing at least one module named ``__init__.py`` 69 | 70 | Usage: 71 | 72 | * ``import package`` 73 | * interprets the ``package/__init__.py`` 74 | * ``import package.module`` 75 | * interprets the ``package/__init__.py`` 76 | * interprets the ``package/module.py`` 77 | 78 | ------------------------------------------------------------ 79 | 80 | ``import`` and ``from`` 81 | ------------------------ 82 | 83 | * assigns the *module* object to a name 84 | * one can import : 85 | * **from** a module or a package 86 | * a module, a class, a function or any **name** 87 | 88 | * any imported object is accessible by its qualified name 89 | 90 | .. code-block:: python 91 | 92 | import something 93 | from mymodule import myobject as mo 94 | from package.submodule import myclass 95 | from anything import * 96 | # ... 97 | 98 | mo.do() 99 | a = anything.Anything() 100 | print something.MyClass.static_foo() 101 | #... 102 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/6-python_re.rst: -------------------------------------------------------------------------------- 1 | Regular expressions in Python 2 | ============================= 3 | 4 | ------------------------------ 5 | 6 | Regular expressions 7 | --------------------- 8 | 9 | * each character match itself, except some specials:: 10 | 11 | . ^ $ * + ? { } [ ] \ | ( ) 12 | 13 | 14 | +------------+-----------------------------+---------------+ 15 | | Operator | Meaning | Example | 16 | +============+=============================+===============+ 17 | | ``.`` | match any character | ``k.r`` | 18 | +------------+-----------------------------+---------------+ 19 | | ``\`` | escape character | ``\.`` | 20 | +------------+-----------------------------+---------------+ 21 | | ``[]`` | define a character class | ``b[ae]n`` | 22 | +------------+-----------------------------+---------------+ 23 | | ``[first-``| character class between the | ``[0-9]`` | 24 | | ``last]`` | ``first`` and the ``last`` | ``[a-z]`` | 25 | +------------+-----------------------------+---------------+ 26 | | ``[^]`` | not matching the class | ``[^<>]`` | 27 | +------------+-----------------------------+---------------+ 28 | | ``?`` | matches 0 or 1 time | ``fája?`` | 29 | +------------+-----------------------------+---------------+ 30 | | ``+`` | matches at least once | ``ps+zt!`` | 31 | +------------+-----------------------------+---------------+ 32 | | ``*`` | matches 0 or more times | ``bo.*ka`` | 33 | +------------+-----------------------------+---------------+ 34 | 35 | ---------------------- 36 | 37 | +------------+-----------------------------+---------------+ 38 | | Operator | Meaning | Example | 39 | +============+=============================+===============+ 40 | | ``{m,n}`` | matches ``m< >> re.split("\d+", "1a2b3c") 141 | ['', 'a', 'b', 'c'] 142 | 143 | 144 | ---------------------------- 145 | 146 | Flags 147 | ~~~~~~~~~ 148 | Most of the functions accepts them through ``flags`` parameter: 149 | * ``re.I`` - ignore case 150 | * ``re.A`` - predefined classes matches ASCII strings 151 | * ``re.L`` - predefined classes matches became locale dependent (deprecated) 152 | * ``re.S`` - ``.`` character matches all (including newline) 153 | * ``re.M`` - ``^`` and ``$`` work also for lines 154 | * ``re.X`` - whitespaces are ignored, ``#`` can be used to comment 155 | 156 | -------------------------------------------------------------------------------- 157 | 158 | Efficiency 159 | ~~~~~~~~~~~~~~~~~~ 160 | 161 | * each ``match``, ``search``, .... call interprets the *regex* 162 | * sometimes useful for compile it 163 | * creates a ``Pattern`` object 164 | 165 | 166 | .. code-block:: python 167 | 168 | import re 169 | import sys 170 | myregexp = re.compile("\\d+") 171 | for line in sys.stdin: 172 | if myregexp.match(line): 173 | print line 174 | 175 | * ``re.escape`` can be used to escape a string 176 | 177 | .. code-block:: python 178 | 179 | >>> re.escape("^.*\\(][)") 180 | '\\^\\.\\*\\\\\\(\\]\\[\\)' 181 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/7-python_oop.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | OOP in Python 3 | ============= 4 | 5 | -------------------------------------------------------------------------------- 6 | 7 | Basics 8 | ~~~~~~ 9 | 10 | http://docs.python.org/3/tutorial/classes.html 11 | 12 | * everything is an object 13 | * classes as well, but 14 | * defines a new namespace 15 | * are callable 16 | * ``object`` is their base class 17 | * instances 18 | * are created by classes 19 | * inherit the namespace of the class 20 | * there is no interfaces or abstract classes 21 | 22 | -------------------------------------------------- 23 | 24 | Class statement 25 | ~~~~~~~~~~~~~~~ 26 | 27 | .. code-block:: python 28 | 29 | class T(object): 30 | def do(self): print("Hello") 31 | class T(object): pass 32 | 33 | t = T() 34 | t.do() # AttributeError 35 | 36 | class X(object): 37 | class Y(object): 38 | class Z(object): pass 39 | 40 | x = X() 41 | y = x.Y() 42 | z = X.Y.Z() 43 | 44 | --------------------------------------------- 45 | 46 | Basic syntax 47 | ~~~~~~~~~~~~ 48 | 49 | .. code-block:: python 50 | 51 | class Vehicle(object): 52 | vehicle_name = "train" 53 | 54 | def add_passanger(self, name): 55 | self.last_passanger = name 56 | 57 | def get_last_passanger(self): 58 | return self.last_passanger 59 | 60 | my_train = Vehicle() 61 | print(my_train.vehicle_name) 62 | 63 | my_train.add_passanger("John") 64 | print(my_train.last_passanger) 65 | print(my_train.get_last_passanger()) 66 | 67 | -------------------------------------------------- 68 | 69 | 70 | Instance and class attributes 71 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 72 | 73 | .. code-block:: python 74 | 75 | >>> class TestData(object): 76 | ... spam = 42 77 | ... 78 | >>> x = TestData() 79 | >>> y = TestData() 80 | >>> x.myAttr = 1; print(x.myAttr) 81 | 1 82 | >>> TestData.spam = 99 83 | >>> x.spam, y.spam, TestData.spam 84 | (99, 99, 99) 85 | >>> x.spam = 88 86 | >>> x.spam, y.spam, TestData.spam 87 | (88, 99, 99) 88 | 89 | -------------------------------------------------------------------------------- 90 | 91 | 92 | Methods 93 | ~~~~~~~ 94 | 95 | * a method is just another object... 96 | * is bound to an instance object 97 | * the first parameter is ``self` referring to the instance object (implicit parameter) 98 | * ``Vehicle.add_passanger(my_train, "John")`` call is translated to ``my_train.add_passanger("John")`` 99 | * accessing method from another one: ``self.method()`` 100 | 101 | .. code-block:: python 102 | 103 | my_train.x = "42" 104 | print my_train.x 105 | my_train.my_foo = lambda x: x+1 106 | print(my_train.my_foo(0)) 107 | 108 | -------------------------------------------------- 109 | 110 | Constructor, destructor 111 | ~~~~~~~~~~~~~~~~~~~~~~~~ 112 | 113 | * implicit constructor (with 1 parameter) 114 | * implicit destructor (does nothing) 115 | 116 | .. code-block:: python 117 | 118 | class Train(object): 119 | def __init__(self, capacity): 120 | self.capacity = capacity 121 | 122 | def __del__(self): 123 | pass 124 | 125 | t = T(1) 126 | del t 127 | 128 | -------------------------------------------------------------------------------- 129 | 130 | Object creation and initialization 131 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 132 | 133 | * ``__init__(self)`` 134 | * instance level method 135 | * initialize the object 136 | * does not return anything 137 | * called after the object creation 138 | 139 | * ``__new__(cls)`` 140 | * class level method 141 | * returns the new object 142 | 143 | 144 | -------------------------------------------------------------------------------- 145 | 146 | Class-level methods 147 | ~~~~~~~~~~~~~~~~~~~~ 148 | 149 | * can be created with decorators 150 | * creating a classmethod lets you access the class as well 151 | 152 | .. code-block:: python 153 | 154 | class T(object): 155 | @staticmethod 156 | def static_method(): 157 | print("Hello!") 158 | 159 | @classmethod 160 | def class_method(cls): 161 | print(cls.__name__) 162 | 163 | o = T() 164 | T.static_method() # "Hello!" 165 | o.static_method() # "Hello!" 166 | 167 | T.class_method() # "T" 168 | o.class_method() # "T" 169 | 170 | -------------------------------------------------------------------------------- 171 | 172 | More on methods 173 | ~~~~~~~~~~~~~~~ 174 | 175 | * overloading? 176 | * no abstract methods: ``raise NotImplementedError()`` 177 | * special methods: http://docs.python.org/3/reference/datamodel.html#special-method-names 178 | * numeric operators 179 | * comparisons 180 | * emulating containers, callables, ... 181 | 182 | 183 | ---------------------------------------------------------- 184 | 185 | Visibility 186 | ~~~~~~~~~~ 187 | * every attribute is visible and accessible by default 188 | * you may start the *non API* attributes with ``_`` 189 | * name mangling helps: ``__name`` is transformed to ``_Classname__name`` 190 | 191 | .. code-block:: python 192 | 193 | class Test(object): 194 | def __hidden(self): print("You won!") 195 | 196 | t = Test() 197 | t.__hidden() # ERROR! 198 | t._Test__hidden() # "You won!" 199 | 200 | Controlling attribute access 201 | '''''''''''''''''''''''''''' 202 | 203 | * ``__setattr__`` 204 | * ``__getattr__`` 205 | * ``__delattr__`` 206 | 207 | ------------------------------------------ 208 | 209 | 210 | Inheritance 211 | ~~~~~~~~~~~ 212 | 213 | .. code-block:: python 214 | 215 | class A(object): 216 | def __init__(self): 217 | print("init A"); self.__x = "a" 218 | 219 | class B1(A): 220 | def __init__(self): 221 | print("init B1"); self.__x = "b" 222 | 223 | class B2(A): 224 | def __init__(self): 225 | A.__init__(self); print("init B2") 226 | 227 | class B3(A): pass 228 | 229 | b1 = B1(), # init B1 230 | b2 = B2() # init A init B2 231 | b3 = B3() # init A 232 | 233 | -------------------------------------------------------------------------------- 234 | 235 | Multiple inheritance 236 | ~~~~~~~~~~~~~~~~~~~~~ 237 | 238 | .. code-block:: python 239 | 240 | class C1(B1,B2): 241 | def get_x(self): return self.__x 242 | class C2(B2, B1): 243 | def get_x(self): return self.__x 244 | 245 | c1 = C1() # init B1 246 | c2 = C2() # init A init B2 247 | print(c1.get_x(), c2.get_x()) # AtributeError 248 | 249 | #====BUT===== 250 | 251 | class C1(B1,B2): 252 | def get_x(self): return self._B1__x 253 | class C2(B2, B1): 254 | def get_x(self): return self._A__x 255 | 256 | c1, c2 = C1(), C2() 257 | print(c1.get_x(), c2.get_x()) # b a 258 | 259 | --------------------------- 260 | 261 | ``super`` 262 | ~~~~~~~~~ 263 | 264 | .. code-block:: python 265 | 266 | class T(object): 267 | a = 0 268 | class A(T): 269 | pass 270 | class B(T): 271 | a = 2 272 | class C(A,B): 273 | pass 274 | c = C() 275 | 276 | super(C,c).a # 2 277 | 278 | * uses the method resolution order 279 | * the MRO of ``C`` is ``[C, A, B, T, object]`` 280 | * with ``super`` there is no need for explicitly name the parent 281 | * returns a proxy object 282 | 283 | --------------------------- 284 | 285 | 286 | Python type hierarchy 287 | ~~~~~~~~~~~~~~~~~~~~~ 288 | 289 | * Method resolution order: http://www.python.org/download/releases/3/mro/ 290 | * Python type hierarchy: http://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy 291 | * built-in types can be used as base classes 292 | 293 | .. code-block:: python 294 | 295 | def MyList(list): pass 296 | l = MyList() 297 | l.append(1) 298 | print(l) 299 | 300 | 301 | 302 | --------------------------- 303 | 304 | Exceptions 305 | ~~~~~~~~~~~~~~~~~~~~ 306 | 307 | * one can define its own exception class 308 | * ``BaseException`` or ``Exception`` used as a base class 309 | * name convention ``SomethingError`` 310 | 311 | .. code-block:: python 312 | 313 | class MyError(Exception): pass 314 | 315 | 316 | class CustomError(Exception): 317 | def __init__(self, value): 318 | self.parameter = value 319 | def __str__(self): 320 | return repr(self.parameter) 321 | 322 | 323 | --------------------------- 324 | 325 | Docstrings 326 | ~~~~~~~~~~~ 327 | 328 | http://www.python.org/dev/peps/pep-0257/ 329 | http://google-styleguide.googlecode.com/svn/trunk/pyguide.html 330 | 331 | * first statement (that is a string) in a module, function, class, package 332 | * thus becames the ``__doc__`` attribute 333 | * use always ``"""triple double quotes"""`` 334 | * can be one-line or multiline 335 | 336 | 337 | -------- 338 | 339 | Style guides 340 | ~~~~~~~~~~~~~ 341 | 342 | * http://www.python.org/dev/peps/pep-0008/ 343 | * https://github.com/quantifiedcode/python-anti-patterns 344 | -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/8-python_2vs3.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | Python 2 vs 3 3 | ============= 4 | 5 | --------------------- 6 | 7 | Basics 8 | ------ 9 | 10 | * Python 3 is incompatible with 2.x 11 | * due code clean-up 12 | * some changes have been backported: 13 | ``from __future__ import ...`` 14 | * forced name conventions 15 | 16 | Details: 17 | 18 | * http://docs.python.org/2/library/__future__.html 19 | * http://docs.python.org/3.0/whatsnew/3.0.html 20 | * http://wiki.python.org/moin/Python2orPython3 21 | * http://docs.python.org/2/library/2to3.html 22 | 23 | Hands on with: http://ideone.com/ 24 | 25 | ------ 26 | 27 | ``print`` 28 | --------- 29 | * became a function 30 | * ``from __future__ import print_function`` 31 | 32 | .. code-block:: python 33 | 34 | Old: print "The answer is", 2*2 35 | New: print("The answer is", 2*2) 36 | 37 | Old: print x, # Trailing comma suppresses newline 38 | New: print(x, end=" ") # Appends a space instead of a newline 39 | 40 | Old: print # Prints a newline 41 | New: print() # You must call the function! 42 | 43 | print("It is", 2**32, "!", sep="") 44 | 45 | ----- 46 | 47 | Views and Iterators vs Lists 48 | ---------------------------- 49 | * become views: ``dict.keys()``, ``dict.items()`` 50 | * ``dict.iterkeys(), iteritems() and tiervalues()`` not supperted 51 | * ``map(), filter()`` return iterators 52 | * ``range()`` is a generator 53 | * ``zip()`` returns an iterator 54 | 55 | ---- 56 | 57 | Ordering, comparisions 58 | ---------------------- 59 | * ``1 < ''``, ``0>None`` raise ``TypeError`` 60 | * ``sort()`` and ``sorted()`` no longer accepts ``cmp`` argument, thus ``key`` should be used 61 | 62 | Integers 63 | -------- 64 | * ``long`` renamed to ``int`` 65 | * ``1/2`` equals ``0.5`` 66 | * ``1.0//2.0`` equals ``0.0`` 67 | * there is no upper bould for an integer 68 | * ``sys.maxint`` is used as a theoretic maximum 69 | 70 | ------ 71 | 72 | Text 73 | ------- 74 | Text vs. data: 75 | 76 | * all text is Unicode with type ``str`` 77 | * data is stored in ``bytes`` 78 | * conversion between bytes to strings: ``encode(), decode()`` 79 | * ``basestring`` type is deprecated 80 | 81 | * previously used ``u".."`` strings are easily converted wit ``2to3`` 82 | * but this syntax is no longer valid 83 | 84 | ---- 85 | 86 | Text 87 | ---- 88 | 89 | Internal changes: 90 | 91 | * all API functions uses Unicode strings 92 | * default source encoding is UTF-8 93 | * non-ASCII letters are allowed in identifiers 94 | 95 | * ``open()`` use an encoding to map files 96 | * ``StringIO`` became ``io.StringIO`` 97 | 98 | ------ 99 | 100 | New syntax 101 | ----------- 102 | * ``nonlocal`` statement: reach an outer but non-blobal 103 | * extended unpacking: 104 | * ``a,b, *rest = some_sequence`` 105 | * ``*rest, a,b = some_sequence`` 106 | * dictionary comprehensions 107 | ``{k:v for k,v in stuff}`` 108 | * set comprehensions 109 | ``{k for k in stuff}`` 110 | 111 | ---- 112 | 113 | Changed syntax 114 | -------------- 115 | * new Metaclass syntax "``class C(metaclass=M)``" 116 | * ``raw_input()`` became ``input()`` 117 | * new style classes are default: 118 | "``class A: pass``" and 119 | "``class A(object): pass``" are equal 120 | * ``True``, ``False``, ``None`` are reserved words 121 | 122 | ----- 123 | 124 | Exceptions 125 | ---------- 126 | 127 | Simplified ``except``, thus only allowed constructions are: 128 | 129 | .. code-block:: python 130 | 131 | except ValueError as e: 132 | pass 133 | 134 | except (ValueError, TypeError) as e: 135 | pass 136 | 137 | Simplified ``raise`` statement: "``raise Exception(args)``" 138 | 139 | ---- 140 | 141 | Others 142 | ------ 143 | * ``long`` and ``int`` types were merged 144 | * updated integer literals 145 | * library changes 146 | * applied naming conventions 147 | * "``%``" string formatting is deprecated, use the ``format`` function: 148 | ``"The story of {0}, {1}, and {c}".format(a, b, c=d)`` 149 | 150 | http://dev.pocoo.org/~gbrandl/py3 -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/std_streams.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/ScriptingLanguagesIntroduction/PythonIntro/std_streams.png -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/tiobe.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/ScriptingLanguagesIntroduction/PythonIntro/tiobe.png -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/types.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/ScriptingLanguagesIntroduction/PythonIntro/types.gif -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/unicode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/ScriptingLanguagesIntroduction/PythonIntro/unicode.png -------------------------------------------------------------------------------- /ScriptingLanguagesIntroduction/PythonIntro/unicode2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oroszgy/ScriptingLanguagesCourseMaterials/d836adfeed86ca9d6998412fa0d54e4f478332ad/ScriptingLanguagesIntroduction/PythonIntro/unicode2.png --------------------------------------------------------------------------------