├── .github └── workflows │ └── gh-pages-deploy.yml ├── .gitignore ├── .python-version ├── README.md ├── mkdocs.yml ├── programming-notes ├── .nav.yml ├── general │ ├── CLI-Cheat-Sheet.pdf │ ├── batch-script.md │ ├── contributing-oss.md │ ├── git-cheatsheet-EN-grey.pdf │ ├── git.md │ ├── github-git-cheat-sheet.pdf │ ├── image-metadata.md │ ├── os-env-vars.md │ ├── powershell.md │ ├── production │ │ ├── deploy-django-azure.md │ │ ├── deploy-django-heroku.md │ │ ├── deploy-django.md │ │ ├── s3-django-setup.md │ │ └── vps-providers.md │ ├── python-raspbian.md │ ├── regex.md │ ├── vscode-shortcut.md │ ├── vscode.md │ ├── web-dev-resources.md │ └── windows-terminal.md ├── html-css │ ├── css-frameworks.md │ ├── css-reset.md │ ├── css-selectors.md │ ├── seo-meta-tags.md │ └── sticky-footer.md ├── index.md ├── javascript │ ├── DataTables.md │ ├── ajax.md │ ├── basics.md │ ├── cache-data.md │ ├── create-react-app.md │ ├── expiration-localstorage.md │ ├── filter-search.md │ ├── modern-js.md │ └── react.md ├── latex │ ├── beamer.md │ ├── bibliography.md │ ├── derivatives.md │ ├── figures.md │ └── latex-intro.md ├── linux │ ├── crontab.md │ ├── django-production-server.md │ ├── keep-scripts-running-ssh.md │ ├── keep-ssh-running.md │ ├── path.md │ ├── raspberrypi.md │ ├── server-setup.md │ └── ubuntu-vps.md ├── matlab │ ├── artificial-data.md │ ├── doe.md │ ├── figures-publication.md │ ├── integrate.md │ ├── links.md │ ├── strings.md │ └── surface-fit.md └── python │ ├── .nav.yml │ ├── anaconda.md │ ├── basemap.md │ ├── basics │ ├── abstract-classes.md │ ├── arg-parse.md │ ├── args-kwargs.md │ ├── builtin-functions.md │ ├── classes.md │ ├── custom-exceptions.md │ ├── data-structures.md │ ├── dataclasses.md │ ├── decorators.md │ ├── imports.md │ ├── lambda-functions.md │ ├── magic-methods.md │ ├── packages.md │ ├── slice-notation.md │ ├── snippets.md │ └── string-formatting.md │ ├── beautifulsoup4.md │ ├── boilerplate.md │ ├── config-files.md │ ├── create-exe.md │ ├── csv.md │ ├── datetime.md │ ├── django │ ├── django-pyinstaller.md │ ├── django-rest-api.md │ ├── django-snippets.md │ ├── django-uv-workflow.md │ └── django.md │ ├── email.md │ ├── flask.md │ ├── ghp-import.md │ ├── gui.md │ ├── image-manipulation.md │ ├── json.md │ ├── jupyter.md │ ├── logging.md │ ├── machine-learning │ ├── ml-pipeline.md │ ├── ml-save-model.md │ ├── ml-sklearn.md │ └── ml-workflows.md │ ├── matlab.md │ ├── matplotlib.md │ ├── msgpack.md │ ├── mysql.md │ ├── pandas.md │ ├── pathlib.md │ ├── profiling.md │ ├── progress-bar.md │ ├── project-set-up.md │ ├── python-packaging.md │ ├── python-version.md │ ├── resources │ ├── data-science-libs.md │ ├── hosting.md │ ├── tutorials.md │ └── web-dev.md │ ├── schedule.md │ ├── scientific-figures.md │ ├── seaborn.md │ ├── sphinx.md │ ├── sql-alchemy.md │ ├── store-dicts.md │ ├── stravalib.md │ ├── streamlit.md │ ├── type-hints.md │ ├── uv.md │ └── virtual-environments.md ├── pyproject.toml └── uv.lock /.github/workflows/gh-pages-deploy.yml: -------------------------------------------------------------------------------- 1 | name: gh-pages-deploy 2 | on: 3 | push: 4 | branches: 5 | - master 6 | - main 7 | permissions: 8 | contents: write 9 | jobs: 10 | deploy: 11 | runs-on: ubuntu-latest 12 | steps: 13 | - uses: actions/checkout@v4 14 | - name: Configure Git Credentials 15 | run: | 16 | git config user.name github-actions[bot] 17 | git config user.email 41898282+github-actions[bot]@users.noreply.github.com 18 | - uses: actions/setup-python@v5 19 | with: 20 | python-version: 3.x 21 | - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV 22 | - uses: actions/cache@v4 23 | with: 24 | key: mkdocs-material-${{ env.cache_id }} 25 | path: .cache 26 | restore-keys: | 27 | mkdocs-material- 28 | - run: pip install mkdocs-material mkdocs-awesome-nav 29 | - run: mkdocs gh-deploy --force -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | __pycache__/ 3 | 4 | 5 | .venv/ 6 | site/ 7 | -------------------------------------------------------------------------------- /.python-version: -------------------------------------------------------------------------------- 1 | 3.11 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Programming Notes 2 | 3 | A collection of programming notes, snippets and examples. By José Aniceto. 4 | 5 | [Click here to check the website](https://janiceto.github.io/programming-notes/) 6 | 7 | Below is a list of the most used pages. 8 | 9 | ## Python 10 | 11 | - [Simple Machine Learning workflow](/programming-notes/python/ml-workflows) 12 | - [Scientific quality figures](/programming-notes/python/scientific-figures) 13 | - [Python packages and modules](/programming-notes/python/packages) 14 | 15 | 16 | ## MATLAB 17 | - [Preparing figures for publication](programming-notes/matlab/figures-publication.md) 18 | - [Surface fitting with MATLAB](/matlab/surface-fit.md) 19 | - [Working with strings](/matlab/strings.md) 20 | - [Links and Resources](/matlab/links.md) 21 | - [Integrating from data points](/matlab/integrate.md) 22 | - [Design of Experiments](/matlab/doe.md) 23 | 24 | ## GENERAL 25 | - [VSCode shortcuts for Windows](/general/vscode-shortcut.md) 26 | - [Git Version Control System](/general/git.md) 27 | - [Deploy a Django app to Heroku](/general/deploy-django-heroku.md) 28 | - [Deploy Django app to Azure App Service](/general/deploy-django-azure.md) 29 | - [Create a Windows batch script](/general/batch-script.md) 30 | - [Regular expressions](/general/regex.md) 31 | - [Online Resources For Web Developers](/general/web-dev-resources.md) 32 | - [Installing Python 3.7 on Raspbian](/general/python-raspbian.md) 33 | - [Deploy a Django app in an Ubuntu VPS (DigitalOcean)](/general/deploy-django.md) 34 | - [Set up Amazon S3 to serve Django Static and Media files](/general/s3-django-setup.md) 35 | - [Powershell](/general/powershell.md) 36 | - [VPS providers](/general/vps-providers.md) 37 | - [Hiding Passwords and Secret Keys in Environment Variables](/general/os-env-vars.md) 38 | - [Guide to contributing on GitHub](/general/contributing-oss.md) 39 | - [Visual Studio Code setup](/general/vscode.md) 40 | - [Editing photos metadata](/general/image-metadata.md) 41 | - [Windows terminal](/general/windows-terminal.md) 42 | 43 | ## LINUX 44 | - [Set up a Django production server with gunicorn](/linux/django-production-server.md) 45 | - [Initial Server Setup with Ubuntu 18.04](/linux/server-setup.md) 46 | - [Deploy a Flask Application on an Ubuntu VPS (DigitalOcean)](/linux/ubuntu-vps.md) 47 | - [Set `$PATH` variable in Ubuntu](/linux/path.md) 48 | - [Keep processes running and closing SSH connection](/linux/keep-scripts-running-ssh.md) 49 | - [Linux Crontab](/linux/crontab.md) 50 | - [Keep SSH sessions running after disconnecting](/linux/keep-ssh-running.md) 51 | - [Set up server in Raspberry Pi](/linux/raspberrypi.md) 52 | 53 | ## LATEX 54 | - [Bibliography management](/latex/bibliography.md) 55 | - [Derivatives](/latex/derivatives.md) 56 | - [An Introduction to LaTeX](/latex/latex-intro.md) 57 | - [Create LaTeX presentations with Beamer](/latex/beamer.md) 58 | - [Working with figures](/latex/figures.md) 59 | 60 | ## JAVASCRIPT 61 | - [Basic Javascript concepts](/javascript/basics.md) 62 | - [Filter, sort and layout](/javascript/filter-search.md) 63 | - [React](/javascript/react.md) 64 | - [DataTables JS Library](/javascript/DataTables.md) 65 | - [Create ReactJS app and deploy to Github pages](/javascript/create-react-app.md) 66 | - [Making AJAX request](/javascript/ajax.md) 67 | - [Modern JavaScript](/javascript/modern-js.md) 68 | - [Set expiration time (TTL) for LocalStorage](/javascript/expiration-localstorage.md) 69 | - [Caching data](/javascript/cache-data.md) 70 | 71 | ## HTML CSS 72 | - [CSS selectors](/html-css/css-selectors.md) 73 | - [CSS frameworks](/html-css/css-frameworks.md) 74 | - [Meta tags for SEO](/html-css/seo-meta-tags.md) 75 | - [Sticky footer](/html-css/sticky-footer.md) 76 | - [Simple CSS reset](/html-css/css-reset.md) 77 | 78 | ## PYTHON 79 | 80 | 81 | 82 | These notes contain excerpts, code snippets and examples from various sources including, but not limited to, Python Docs, The Hitchhiker’s Guide to Python, Real Python and Stack Overflow users. 83 | -------------------------------------------------------------------------------- /mkdocs.yml: -------------------------------------------------------------------------------- 1 | # Project information 2 | site_name: Programming Notes 3 | site_url: https://janiceto.github.io/programming-notes/ 4 | site_author: José Aniceto 5 | site_description: >- 6 | A collection of programming notes, snippets and examples. Mainly focused on Python and MATLAB. 7 | 8 | # Repository 9 | repo_name: jAniceto/programming-notes 10 | repo_url: https://github.com/jAniceto/programming-notes 11 | edit_uri: edit/master/ 12 | 13 | # Configuration 14 | docs_dir: programming-notes 15 | 16 | theme: 17 | name: material 18 | palette: 19 | primary: blue grey 20 | accent: blue grey 21 | features: 22 | - navigation.tabs 23 | - content.action.edit 24 | # - navigation.footer 25 | - search.suggest 26 | 27 | plugins: 28 | - search 29 | - awesome-nav 30 | 31 | markdown_extensions: 32 | - pymdownx.highlight: 33 | anchor_linenums: true 34 | line_spans: __span 35 | pygments_lang_class: true 36 | - pymdownx.inlinehilite 37 | - pymdownx.snippets 38 | - pymdownx.superfences 39 | 40 | extra: 41 | social: 42 | - icon: fontawesome/brands/github 43 | link: https://github.com/jAniceto 44 | generator: false -------------------------------------------------------------------------------- /programming-notes/.nav.yml: -------------------------------------------------------------------------------- 1 | nav: 2 | - Home: index.md 3 | - Python: python 4 | - MATLAB: matlab 5 | - JavaScript: javascript 6 | - HTML & CSS: html-css 7 | - LaTeX: latex 8 | - Linux: linux 9 | - general 10 | 11 | append_unmatched: true 12 | 13 | sort: 14 | sections: first -------------------------------------------------------------------------------- /programming-notes/general/CLI-Cheat-Sheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jAniceto/programming-notes/46b52dc8653e98f5678d94935d26c562e34d61db/programming-notes/general/CLI-Cheat-Sheet.pdf -------------------------------------------------------------------------------- /programming-notes/general/batch-script.md: -------------------------------------------------------------------------------- 1 | # Create a Windows batch script 2 | 3 | A batch file is a text file with the extension **.bat** that contains one or more command line commands to be run consecutively. 4 | 5 | ## Some usefull commands: 6 | 7 | **@echo** - This parameter will allow you to view your working script in the command prompt. This parameter is useful for viewing your working code. If any issues arise from the batch file, you will be able to view the issues associated with your script using the echo function. Adding a following off to this parameter will allow you to quickly close your script after it has finished. 8 | 9 | **title** - This will provide a title for your batch script in your Command Prompt window. 10 | 11 | **cls** - Clears your command prompt, best used when extraneous code can make what you’re accessing had to find. 12 | 13 | **rem** OR **::** - Shorthand for remark. Rem statements are not entered into your code. Instead, they are used to explain and give information regarding the code. 14 | 15 | **%%a** - Each file in the folder. 16 | 17 | **(“.\”)** - The root folder. When using the command prompt, one must direct the prompt to a particular directory before changing a files name, deleting a file, and so on. With batch files, you only need to paste your .bat file into the directory of your choosing. 18 | 19 | **pause** - Allows a break in the logical chain of your .bat file. This allows for users to read over command lines before proceeding with the code. The phrase “Press any key to continue…” will denote a pause. 20 | 21 | **start “” [website]** - Will head to a website of your choice using your default web browser. 22 | 23 | A complete list can be found [here](https://en.wikibooks.org/wiki/Windows_Batch_Scripting). 24 | 25 | ## Useful batch files: 26 | 27 | ### Start Flask server 28 | 29 | The following script starts a Flask development server provided the batch file is located in the project root directory. 30 | 31 | ``` 32 | echo off 33 | cd %localhost% 34 | call venv\Scripts\activate.bat 35 | start python dashboard\__init__.py 36 | pause 37 | ``` 38 | -------------------------------------------------------------------------------- /programming-notes/general/git-cheatsheet-EN-grey.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jAniceto/programming-notes/46b52dc8653e98f5678d94935d26c562e34d61db/programming-notes/general/git-cheatsheet-EN-grey.pdf -------------------------------------------------------------------------------- /programming-notes/general/git.md: -------------------------------------------------------------------------------- 1 | # Git Version Control System 2 | 3 | ## Instalation 4 | 5 | On a Debian-based distribution, such as Ubuntu 6 | 7 | ``` 8 | $ sudo apt update 9 | $ sudo apt install git-all 10 | ``` 11 | 12 | On Windows download and install from [git-scm.com](https://git-scm.com/download/win). 13 | 14 | Configure your user name and email address: 15 | 16 | ``` 17 | $ git config --global user.name "exampleusername" 18 | $ git config --global user.email "example@gmail.com" 19 | ``` 20 | 21 | ## Creating repositories 22 | 23 | ### Create a new local repository 24 | 25 | ```git 26 | $ cd path/to/desired/directory 27 | $ git init 28 | ``` 29 | 30 | ### Clone an existing repository 31 | 32 | Grab the `https` or `ssh` path from the remote (Github, GitLab, etc...) and then run: 33 | 34 | ```git 35 | $ git clone ssh://user@domain.com/repo.git 36 | ``` 37 | 38 | ## Working locally 39 | 40 | Check the status of our repository: 41 | 42 | ```git 43 | $ git status 44 | ``` 45 | 46 | When we create a new file `git` becomes aware of it but it is still not tracking changes to that file. To instruct `git` to track a file run: 47 | 48 | ```git 49 | $ git add file1.txt 50 | ``` 51 | 52 | Now `git` is tracking the file but changes haven't been recorded as a commit yet. A commit is a record of a change. This change will be permanently recorded in our history, and can be reverted to at a later date. To make a commit we use `git commit`. We also add a message which should be a short desciption of changes made. 53 | 54 | ```git 55 | $ git commit -m "Added stuff on file 1" 56 | ``` 57 | 58 | The `-m` flag allows us add a short message to each commit. A good commit message should give a brief statement of any changes made to the file. 59 | 60 | To commit all local changes in tracked files (however this is generally discouredged): 61 | 62 | ```git 63 | $ git commit -a -m "Changed something" 64 | ``` 65 | 66 | We can lookup recent changes made to our repository with: 67 | 68 | ```git 69 | $ git log 70 | ``` 71 | 72 | 73 | ## Working with remote repository 74 | 75 | To push your local changes to the remote repository we use the following command: 76 | 77 | ```git 78 | $ git push origin master 79 | ``` 80 | 81 | To pull the current version of the repository from the remote server use: 82 | 83 | ```git 84 | $ git pull 85 | ``` 86 | 87 | If your default branch is different than master, you will need to specify the branch name: 88 | 89 | ``` 90 | $ git pull origin my_default_branch_name 91 | ``` 92 | 93 | 94 | ## Remove a file from version control 95 | 96 | Remove the file from the Git repository but not from the filesystem: 97 | 98 | ```git 99 | $ git rm --cached file1.txt 100 | ``` 101 | 102 | If you want to remove a whole folder, you need to remove all files in it recursively. 103 | 104 | ```git 105 | $ git rm -r --cached folder_name 106 | ``` 107 | 108 | Then push changes to remote repo: 109 | 110 | ```git 111 | $ git commit -m "Removed stuff" 112 | $ git push origin branch_name 113 | ``` 114 | -------------------------------------------------------------------------------- /programming-notes/general/github-git-cheat-sheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jAniceto/programming-notes/46b52dc8653e98f5678d94935d26c562e34d61db/programming-notes/general/github-git-cheat-sheet.pdf -------------------------------------------------------------------------------- /programming-notes/general/image-metadata.md: -------------------------------------------------------------------------------- 1 | # Editing photos metadata 2 | 3 | You can do it in one line in the Terminal using `jhead`. 4 | 5 | For example, adjust all EXIF times forward by 1 hour: 6 | ``` 7 | jhead -ta+1:00 *.jpg 8 | ``` 9 | 10 | Useful links: 11 | 12 | [jhead download page](http://www.sentex.net/~mwandel/jhead/) 13 | 14 | [jhead documentation](http://www.sentex.net/~mwandel/jhead/usage.html) 15 | -------------------------------------------------------------------------------- /programming-notes/general/os-env-vars.md: -------------------------------------------------------------------------------- 1 | # Hiding Passwords and Secret Keys in Environment Variables 2 | 3 | ### Adding a Environment Variable (Windows) 4 | 5 | System Menu > `Advanced System Settings` > `Environment Variables...` > in User Variables click `New...` > insert variable name (VAR_NAME) and value (VAR_VALUE) 6 | 7 | When coding it may be necessary to restart your IDE or text editor. 8 | 9 | ### Adding a Environment Variable (Linux and Mac) 10 | 11 | Open the terminal: 12 | 13 | ``` 14 | $ cd 15 | $ nano .bash_profile 16 | ``` 17 | 18 | To the file add the line and save: 19 | ``` 20 | export VAR_NAME="VAR_VALUE" 21 | ``` 22 | 23 | When coding it may be necessary to restart your IDE or text editor. 24 | 25 | ### Acessing your Environment Variables in Python 26 | 27 | ```python 28 | import os 29 | 30 | var1 = os.environ.get('VAR_NAME') 31 | ``` 32 | -------------------------------------------------------------------------------- /programming-notes/general/powershell.md: -------------------------------------------------------------------------------- 1 | # Powershell 2 | 3 | ## Usefull commands 4 | 5 | Open Explorer in current location: 6 | ```powershell 7 | ii . 8 | ``` 9 | 10 | which is short for: 11 | ```powershell 12 | Invoke-Item . 13 | ``` 14 | 15 | ## Costumization 16 | 17 | Using `Oh My Posh`: 18 | 19 | Install with: 20 | ```powershell 21 | Install-Module oh-my-posh -Scope CurrentUser 22 | ``` 23 | 24 | List themes: 25 | ```powershell 26 | Get-PoshThemes 27 | ``` 28 | 29 | To apply a theme, edit `$PROFILE` and add the following line. 30 | ```powershell 31 | Set-PoshPrompt -Theme jandedobbeleer 32 | ``` 33 | 34 | Once added, reload your profile for the changes to take effect. 35 | ```powershell 36 | . $PROFILE 37 | ``` 38 | -------------------------------------------------------------------------------- /programming-notes/general/production/deploy-django-azure.md: -------------------------------------------------------------------------------- 1 | # Deploy Django app to Azure App Service 2 | 3 | ## Run a cronjob in Azure App Service (Linux) 4 | 5 | SSH into the container and copy the current startup script by typing the following: 6 | 7 | ``` 8 | $ cp /opt/startup/startup.sh /home 9 | ``` 10 | 11 | Edit the `startup.sh` under `/home/startup.sh` and add your changes to the top of the file after `#!/bin/sh`. In the sample below, I'll be installing cron to run a cronjob. 12 | 13 | ```sh 14 | # Installing cron 15 | apt-get update -qq && apt-get install cron -yqq 16 | service cron start 17 | mkdir /home/BackupLogs 18 | (crontab -l 2>/dev/null; echo "*/5 * * * * cp /home/LogFiles/*.log /home/BackupLogs")|crontab 19 | ``` 20 | 21 | Save the file. 22 | 23 | In the Azure Portal configurations, add `/home/startup.sh` as the Startup Command and restart the site. 24 | 25 | ![](https://i.stack.imgur.com/WMuGX.png) 26 | 27 | Note, there are two pitfalls to this approach: 28 | - The script must be executable, so either install w/ unix and chmod 755 start.sh or use a git command (see SO). 29 | - The 3pp (here crontab) is installed on every startup, thus you depend on external servers/repositories when starting the webapp. 30 | 31 | 32 | References for this section: 33 | 34 | - https://stackoverflow.com/questions/57654279/how-to-run-cronjobs-on-an-azure-linux-hosted-webapp 35 | - https://azureossd.github.io/2020/01/23/custom-startup-for-nodejs-python/index.html 36 | 37 | 38 | ## References 39 | 40 | - [Deploying a Django App to Azure App Service](https://testdriven.io/blog/django-azure-app-service/) 41 | - [Deploy a Python (Django or Flask) web app with PostgreSQL in Azure](https://learn.microsoft.com/en-us/azure/app-service/tutorial-python-postgresql-app?tabs=django%2Cwindows) 42 | -------------------------------------------------------------------------------- /programming-notes/general/production/deploy-django-heroku.md: -------------------------------------------------------------------------------- 1 | # Deploy a Django app to Heroku 2 | 3 | ### Index: 4 | * [Requirements](#requirements) 5 | * [Install the Heroku CLI](#install-the-heroku-cli) 6 | * [Deploying Python and Django Apps on Heroku](#deploying-python-and-django-apps-on-heroku) 7 | * [Configure Django apps for Heroku](#configure-django-apps-for-heroku) 8 | * [Deploy to Heroku](#deploy-to-heroku) 9 | * [References](#references) 10 | 11 | 12 | ### Requirements: 13 | * Git installed 14 | * Python >3.6 15 | * Heroku CLI (see step 1) 16 | 17 | 18 | ## 1) Install the Heroku CLI 19 | 20 | Install the Heroku Command Line Interface (CLI) for your platform from [here](https://devcenter.heroku.com/articles/getting-started-with-python#set-up). Once installed, you can use the `heroku` command from your command shell. To log in to the Heroku CLI use: 21 | 22 | ```bash 23 | $ heroku login 24 | ``` 25 | 26 | 27 | ## 2) Deploying Python and Django apps on Heroku 28 | 29 | Heroku automatically identifies your app as a Python app if any of the following files are present in its root directory: 30 | * `Pipfile` 31 | * `setup.py` 32 | * `requirements.txt` 33 | 34 | When you deploy to Heroku, the dependencies you specify are automatically installed. If you’re using Django, the `collectstatic` command also runs automatically during the deployment process. For this to work properly make sure you install the Django-Heroku Python package (step 3). 35 | 36 | 37 | ## 3) Configure Django apps for Heroku 38 | 39 | Create a `Procfile` (no extension) and add the following content where `myproject` is the name of your Django app. 40 | ``` 41 | web: gunicorn myproject.wsgi 42 | ``` 43 | 44 | Install `gunicorn`: 45 | ```bash 46 | $ pipenv install gunicorn 47 | ``` 48 | 49 | Install the `django-heroku` package 50 | ```bash 51 | $ pipenv install django-heroku 52 | ``` 53 | 54 | Add the following import statement to the top of `settings.py`: 55 | ```python 56 | import django_heroku 57 | ``` 58 | Then add the following to the bottom of `settings.py`: 59 | ```python 60 | # Activate Django-Heroku. 61 | django_heroku.settings(locals()) 62 | ``` 63 | 64 | 65 | ## 4) Deploy to Heroku 66 | 67 | Using the Heroku CLI lets create our app and the database. 68 | 69 | ```bash 70 | $ heroku login 71 | $ heroku create 72 | ``` 73 | 74 | Now lets push our code to Heroku: 75 | ```bash 76 | $ git add . 77 | $ git commit -m "Ready to heroku this." 78 | $ git push heroku master 79 | ``` 80 | 81 | Note: If you wish to depoly from a local branch other than `master`, e.g. `testbranch` then use: 82 | ```bash 83 | $ git push heroku testbranch:master 84 | ``` 85 | 86 | Finally, migrate your Database to the Heroku app: 87 | 88 | ```bash 89 | $ heroku run python manage.py migrate 90 | ``` 91 | 92 | You should now be able to see your app in the Heroku Dashboard as well as a Dyno web process with the `ON` indication. 93 | 94 | 95 | ## 5) Import local database to Heroku (Optional) 96 | 97 | If you are using a postgresql database locally you can easily import it to your newly created Heroku app. First create a backup of your local DB: 98 | 99 | ```bash 100 | pg_dump -U USERNAME DATABASE > pg_db_dump.sql 101 | ``` 102 | 103 | To import it to Heroku run: 104 | 105 | ```bash 106 | heroku pg:psql --app APPNAME < pg_db_dump.sql 107 | ``` 108 | 109 | ### References: 110 | * [Getting Started on Heroku with Python](https://devcenter.heroku.com/articles/getting-started-with-python) 111 | * [Deploying Python and Django Apps on Heroku](https://devcenter.heroku.com/articles/deploying-python) 112 | * [Configuring Django Apps for Heroku](https://devcenter.heroku.com/articles/django-app-configuration) 113 | * [django-heroku Github](https://github.com/heroku/django-heroku) 114 | * [Deploying with Git](https://devcenter.heroku.com/articles/git) 115 | -------------------------------------------------------------------------------- /programming-notes/general/production/vps-providers.md: -------------------------------------------------------------------------------- 1 | # VPS providers 2 | 3 | When you need to host your projects, you have several options available: 4 | 5 | **Shared hosting**, where you share a server with lots of other users and have very limited control over the server. **Dedicated hosting**, where you have complete control over a server, a very flexible option, but requires you to manage the whole server, worry about hardware failures and backups, and is typically expensive. A third option that is in between these two choices is a **Virtual Private Server**, or **VPS**. 6 | 7 | VPSs virtualized nature provides several advantages such as instant reinstallation of the operating system, quick recovery and backup, and fast and simple hardware upgrades (the hardware is virtual, after all). 8 | 9 | Here is a list of several popular VPS providers: 10 | 11 | * [hetzner.co](https://www.hetzner.com/cloud) - from 4.29 €/month, based in Germany 12 | * [digitalocean.com](https://www.digitalocean.com/pricing/) - from 5 $/month + tax (anual) 13 | * [linode.com](https://www.linode.com) - from 5 $/month + tax 14 | * [heroku.com](https://www.heroku.com/pricing) from 7 $/month + tax (anual) 15 | * [Google Cloudm](https://console.cloud.google.com/compute) 16 | * [AWS EC2](https://aws.amazon.com/ec2/) 17 | 18 | ### Portugal based providers: 19 | 20 | * [ovh.pt](https://www.ovh.pt/vps/vps-ssd.xml) - from 3 €/month + tax (anual) 21 | * [flexi.pt](https://www.flexi.pt/vps-servidor-virtual) - from 3 €/month + tax (anual) 22 | -------------------------------------------------------------------------------- /programming-notes/general/python-raspbian.md: -------------------------------------------------------------------------------- 1 | # Installing Python 3.7 on Raspbian 2 | 3 | As of January 2018, Raspbian does not yet include the latest Python release, Python 3.6. This means we will have to build it ourselves, and here is how to do it. There is also an ansible role attached that automates it all for you. 4 | 5 | 1) Install the required build-tools (some might already be installed on your system). 6 | 7 | ``` 8 | $ sudo apt-get update 9 | $ sudo apt-get install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev 10 | ``` 11 | 12 | If one of the packages cannot be found, try a newer version number (e.g. `libdb5.4-dev` instead of `libdb5.3-dev`). 13 | 14 | 2) Download and install Python 3.6. When downloading the source code, select the most recent release of Python 3.6, available on the [official site](https://www.python.org/downloads/source/). Adjust the file names accordingly. 15 | 16 | ``` 17 | $ wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz 18 | $ tar xf Python-3.7.0.tar.xz 19 | $ cd Python-3.7.0 20 | $ ./configure 21 | $ make 22 | $ sudo make altinstall 23 | ``` 24 | 25 | 3) Optionally: Delete the source code and uninstall the previously installed packages. When uninstalling the packages, make sure you only remove those that were not previously installed on your system. Also, remember to adjust version numbers if necesarry. 26 | 27 | ``` 28 | $ sudo rm -r Python-3.7.0 29 | $ rm Python-3.7.0.tar.xz 30 | $ sudo apt-get --purge remove build-essential tk-dev 31 | $ sudo apt-get --purge remove libncurses5-dev libncursesw5-dev libreadline6-dev 32 | $ sudo apt-get --purge remove libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev 33 | $ sudo apt-get --purge remove libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev 34 | $ sudo apt-get autoremove 35 | $ sudo apt-get clean 36 | ``` 37 | 38 | ### References 39 | * https://liudr.wordpress.com/2016/02/04/install-python-on-raspberry-pi-or-debian/ 40 | * https://gist.github.com/BMeu/af107b1f3d7cf1a2507c9c6429367a3b 41 | -------------------------------------------------------------------------------- /programming-notes/general/regex.md: -------------------------------------------------------------------------------- 1 | # Regular expressions 2 | 3 | ## Regex for finding URLs 4 | 5 | Regex if you want to ensure URL starts with HTTP/HTTPS: 6 | 7 | ```regex 8 | https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*) 9 | ``` 10 | 11 | If you do not require HTTP protocol: 12 | 13 | ```regex 14 | [-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*) 15 | ``` 16 | 17 | #### Exemple usage in VSCode to find a URL and convert to Markdown link like, `[link](link)`: 18 | 19 | Find: `(https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*))` 20 | 21 | Replace: `[$1]($1)` 22 | 23 | Use `()` to create a group and `$1` to reference that group. 24 | -------------------------------------------------------------------------------- /programming-notes/general/vscode-shortcut.md: -------------------------------------------------------------------------------- 1 | # VSCode shortcuts for Windows 2 | 3 | ## Open/View 4 | 5 | Open Command Pallete 6 | 7 | ```bash 8 | Shift+Ctrl+P 9 | ``` 10 | 11 | Access Settings 12 | 13 | ```bash 14 | Ctrl+, 15 | ``` 16 | 17 | Toggle Terminal 18 | 19 | ```bash 20 | Ctrl+` 21 | ``` 22 | 23 | Create New Terminal 24 | ``` 25 | Shift+Ctrl+` 26 | ``` 27 | 28 | Toggle Sidebar 29 | 30 | ```bash 31 | Ctrl+B 32 | ``` 33 | 34 | Open New Window/Instance 35 | 36 | ```bash 37 | Shift+Ctrl+N 38 | ``` 39 | 40 | Close Window 41 | 42 | ```bash 43 | Ctrl+W 44 | ``` 45 | 46 | ## Working With Files 47 | 48 | Sidebar Focus 49 | 50 | ```bash 51 | Shift+Ctrl+E 52 | ``` 53 | 54 | Open File/Folder From Sidebar 55 | ``` 56 | Ctrl+Down 57 | ``` 58 | 59 | Change File Tabs 60 | 61 | ```bash 62 | Ctrl+PageUP 63 | ``` 64 | 65 | Quick File Open 66 | 67 | ```bash 68 | Ctrl+P 69 | ``` 70 | 71 | Open File From Explorer 72 | 73 | ```bash 74 | Ctrl+O 75 | ``` 76 | 77 | New File 78 | 79 | ```bash 80 | Ctrl+N 81 | ``` 82 | 83 | Save 84 | 85 | ```bash 86 | Ctrl+S 87 | ``` 88 | 89 | Save As 90 | 91 | ```bash 92 | Shift+Ctrl+S 93 | ``` 94 | 95 | Close File 96 | 97 | ```bash 98 | Ctrl+W 99 | ``` 100 | 101 | Delete File 102 | ``` 103 | Ctrl+Delete 104 | ``` 105 | 106 | Reopen Files 107 | ``` 108 | Shift+Ctrl+T 109 | ``` 110 | 111 | Zoom 112 | 113 | ```bash 114 | Ctrl++ # Zoom in 115 | Ctrl+- # Zoom out 116 | ``` 117 | 118 | Spilt Editor 119 | 120 | ```bash 121 | Ctrl+\ 122 | ``` 123 | 124 | ## Code Editing 125 | 126 | Go To Start & End Of Line 127 | 128 | ```bash 129 | Ctrl+Right 130 | Ctrl+Left 131 | 132 | home 133 | end 134 | ``` 135 | 136 | Move By Word 137 | 138 | ```bash 139 | Alt+Right 140 | Alt+Left 141 | ``` 142 | 143 | Go To Start & End Of File 144 | 145 | ```bash 146 | Ctrl+Home 147 | Ctrl+End 148 | ``` 149 | 150 | Cut, Copy & Past Line 151 | 152 | ```bash 153 | Ctrl+X # Cut 154 | Ctrl+C # Copy 155 | Ctrl+V # Paste 156 | ``` 157 | 158 | Move Line Up & Down 159 | 160 | ```bash 161 | Alt+Up 162 | Alt+Down 163 | ``` 164 | 165 | Copy Line Up & Down 166 | 167 | ```bash 168 | Shift+Alt+Up 169 | Shift+Alt+Down 170 | ``` 171 | 172 | Remove Line 173 | 174 | ```bash 175 | Shift+Ctrl+K 176 | ``` 177 | 178 | Insert Line 179 | 180 | ```bash 181 | Ctrl+Enter # Insert below 182 | Shift+Ctrl+Enter # Insert above 183 | ``` 184 | 185 | Jump To Matching Bracket 186 | 187 | ```bash 188 | Shift+Ctrl+\ 189 | ``` 190 | 191 | Add Line Comment 192 | 193 | ```bash 194 | Ctrl+/ 195 | ``` 196 | 197 | Add Block Comment 198 | 199 | ```bash 200 | Shift+Alt+A 201 | ``` 202 | 203 | Highlight Code 204 | 205 | ```bash 206 | Shift+Any Direction 207 | ``` 208 | 209 | Select Next Match 210 | 211 | ```bash 212 | Ctrl+D 213 | ``` 214 | 215 | De-select Match 216 | 217 | ```bash 218 | Ctrl+U 219 | ``` 220 | 221 | Add Cursor 222 | 223 | ```bash 224 | Alt+Click 225 | ``` 226 | 227 | Go to Entity (Functions, vars, etc) 228 | 229 | ```bash 230 | Ctrl+Shift+O 231 | ``` 232 | 233 | 234 | ## References 235 | 236 | * [Official list of commands for Windows](https://code.visualstudio.com/shortcuts/keyboard-shortcuts-windows.pdf) 237 | -------------------------------------------------------------------------------- /programming-notes/general/vscode.md: -------------------------------------------------------------------------------- 1 | # Visual Studio Code setup 2 | 3 | 4 | ## Themes 5 | - [Material Theme](https://marketplace.visualstudio.com/items?itemName=Equinusocio.vsc-material-theme) 6 | - [Mariana Pro](https://marketplace.visualstudio.com/items?itemName=rickynormandeau.mariana-pro) 7 | - [One Dark Pro](https://marketplace.visualstudio.com/items?itemName=zhuangtongfa.Material-theme) - Atom's iconic One Dark theme 8 | 9 | 10 | ## General use extensions 11 | - [Code Runner](https://marketplace.visualstudio.com/items?itemName=formulahendry.code-runner) - Run C, C++, Java, JS, PHP, Python, Perl, Ruby, Go and much more 12 | - [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker) - Spelling checker for source code 13 | 14 | 15 | ## Python extensions 16 | - [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Linting, Debugging, Intellisense, formatting, etc 17 | - [Python Docstring Generator](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring) - Automatically generates detailed docstrings for python functions 18 | - [Jupyter](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter) - Jupyter Notebook support 19 | 20 | 21 | ## LaTeX extensions 22 | - [LaTeX Workshop](https://marketplace.visualstudio.com/items?itemName=James-Yu.latex-workshop) - LaTeX typesetting efficiency with preview, compile, autocomplete, colorize, and more 23 | - [LaTeX Utilities](https://marketplace.visualstudio.com/items?itemName=tecosaur.latex-utilities) - Add-on to LaTeX Workshop that provides some extra, non-essential, features. 24 | 25 | 26 | ## HTML/CSS/Javascript 27 | - [Live Server](https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer) - Launch a development local Server with live reload feature for static & dynamic pages 28 | - [Debugger for Chrome](https://marketplace.visualstudio.com/items?itemName=msjsdiag.debugger-for-chrome) - Debug your JavaScript code in the Chrome browser 29 | - [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) - Code formatter 30 | 31 | 32 | ## Markdown 33 | - [markdownlint](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint) - Markdown linting and style checking 34 | -------------------------------------------------------------------------------- /programming-notes/general/web-dev-resources.md: -------------------------------------------------------------------------------- 1 | # Online Resources For Web Developers 2 | 3 | ## GENERATORS: 4 | * Text Content Generator - http://www.lipsum.com 5 | * Favicon Generator - http://tools.dynamicdrive.com/favicon 6 | * Data Generator - https://mockaroo.com/ 7 | * Mobile Mockup Generator - https://mockuphone.com 8 | * Logo Generator - https://www.logaster.com 9 | * UUID Generator - https://www.uuidgenerator.net/ 10 | * Hash Generator - https://passwordsgenerator.net/sha256-hash-generator/ 11 | 12 | ## FONTS 13 | * Google Fonts - https://fonts.google.com/ 14 | * Font Joy - https://fontjoy.com/ 15 | * Font Pair - https://fontpair.co/ 16 | * HTML symbols - https://www.toptal.com/designers/htmlarrows/ 17 | 18 | ## UI Kits 19 | - Wired elements - https://github.com/rough-stuff/wired-elements 20 | 21 | ## IMAGE RESOURCES: 22 | * Free Stock Images - https://www.pexels.com 23 | * Free Stock Images With Great API - https://unsplash.com/ 24 | * Free- Vectors, mockups - https://www.freepik.com 25 | * Dummy Image Placeholders - https://source.unsplash.com/ 26 | * Dummy Image Placeholders - https://placeholder.com 27 | * Free Icons - https://www.iconfinder.com 28 | * Icons and Photos - https://thenounproject.com/ 29 | 30 | ## COLOR & DESIGN: 31 | * Generate Color Palette from Image - https://www.canva.com/color-palette/ 32 | * Color Palette Generator - https://coolors.co/app 33 | * Color Pallete Generator - http://colormind.io/ 34 | * Create Color Schemes - https://color.hailpixel.com 35 | * Get Color Schemes of Websites - http://stylifyme.com 36 | * Create Gradients - https://uigradients.com 37 | * CSS Button Generator - http://css3buttongenerator.com 38 | * SVG/CSS Background Patterns - https://www.heropatterns.com/ 39 | * Color restrictive palletes (Pixel Art) - https://lospec.com/ 40 | 41 | ## DATA VIZ 42 | - Color palete selection - https://colorbrewer2.org 43 | - Color palete selection - https://projects.susielu.com/viz-palette 44 | 45 | ## IMAGE COMPRESSION & CONVERSION: 46 | * Compress All Images - https://compressor.io/compress 47 | * Compress JPG - http://jpeg-optimizer.com/ 48 | * Compress PNG - https://tinypng.com/ 49 | * Convert images (any format) - https://convertio.co/ 50 | 51 | ## CODE OPTIMIZATION: 52 | * Minify JS & CSS - http://minifier.org 53 | * Code Optimization Tools - https://codebeautify.org 54 | * Code Diff Checker - https://www.diffchecker.com 55 | 56 | ## CONVERTERS: 57 | * ES6+ & JSX Compiler - https://babeljs.io/repl 58 | * Sass Converter - https://www.sassmeister.com/ 59 | * Less Converter & More - http://www.webtoolkitonline.com 60 | * Markdown Editor - https://stackedit.io 61 | * Jade Converter - http://www.html2jade.org/ 62 | 63 | ## VALIDATION & COMPATIBILITY: 64 | * Validate HTML - https://validator.w3.org 65 | * Validate CSS - https://jigsaw.w3.org/css-validator 66 | * Check Browser Compatibility - https://caniuse.com/ 67 | * ES6+ Compatibility Table - https://kangax.github.io/compat-table/es6/ 68 | 69 | ## IN BROWSER CODING: 70 | * Client Side Code - https://codepen.io 71 | * Client Side Code - https://jsfiddle.net 72 | * Client Side Code - http://liveweave.com 73 | * Client Side Code - https://codesandbox.io 74 | * Server Side Code - https://repl.it 75 | 76 | ## SNIPPET TOOLS: 77 | * Snippet Manager - https://gist.github.com 78 | * Snippet Manager - https://pastebin.com 79 | 80 | ## WIREFRAME: 81 | * In Browser Wireframing - https://app.moqups.com 82 | * Very Basic In Browser Wireframing - https://wireframe.cc 83 | * Wireframes and prototyping - https://www.justinmind.com/ 84 | 85 | ## RESPONSIVENESS: 86 | * Device Testing - http://www.responsinator.com 87 | * What's My Browser Size - https://www.webpagefx.com/tools/whats-my-browser-size/ 88 | 89 | ## SPEED TEST: 90 | * Speed & Performance Testing - https://tools.keycdn.com/speed 91 | * Pingdom Speed Test - https://tools.pingdom.com/ 92 | 93 | ## OTHER: 94 | * Public API Resources - https://github.com/toddmotto/public-apis?utm_source=mybridge&utm_medium=blog&utm_campaign=read_more 95 | * HTML Entity Lookup - http://entity-lookup.leftlogic.com/ 96 | 97 | ### References: 98 | * https://gist.github.com/bradtraversy/61171a9b81586f5bc4c0ca1e2beb59ab 99 | * https://www.youtube.com/watch?v=4JC8ahZneYU 100 | -------------------------------------------------------------------------------- /programming-notes/general/windows-terminal.md: -------------------------------------------------------------------------------- 1 | # Windows terminal 2 | 3 | ## Costumization 4 | - Windows Terminal themes - [windowsterminalthemes.dev](https://windowsterminalthemes.dev/) 5 | - Oh My Posh - prompt theme engine - [ohmyposh.dev](https://ohmyposh.dev/) 6 | - Starship - customizable prompt - [starship.rs](https://starship.rs/) 7 | 8 | ## References 9 | - [A guide to customising the Windows Terminal](https://freshman.tech/windows-terminal-guide/#change-the-default-profile) 10 | -------------------------------------------------------------------------------- /programming-notes/html-css/css-frameworks.md: -------------------------------------------------------------------------------- 1 | # CSS frameworks 2 | 3 | Full fledged: 4 | - [Bootsrap](https://getbootstrap.com/) 5 | - [Foundation](https://foundation.zurb.com/) 6 | - [PureCSS](https://purecss.io/) 7 | - [Bulma](https://bulma.io/) 8 | - [Semantic UI](https://semantic-ui.com/) 9 | - [Materialize](https://materializecss.com/) 10 | 11 | Lightweight: 12 | - [UI kit](https://getuikit.com/) 13 | - [Milligram](https://milligram.io/) 14 | - [Spectre](https://picturepan2.github.io/spectre/) 15 | 16 | Minimal Bloilerplates: 17 | - [Skeleton](http://getskeleton.com/) 18 | 19 | Other: 20 | - [AgnosticUI](https://www.agnosticui.com/) - Accessible CSS component primitives that work with React, Vue 3, Svelte, and Angular! 21 | - [Tailwind](https://tailwindcss.com/) 22 | - [Primitive UI](https://taniarascia.github.io/primitive) 23 | -------------------------------------------------------------------------------- /programming-notes/html-css/css-reset.md: -------------------------------------------------------------------------------- 1 | # Simple CSS reset 2 | 3 | ```css 4 | html { 5 | box-sizing: border-box; 6 | font-size: 16px; 7 | } 8 | 9 | *, *:before, *:after { 10 | box-sizing: inherit; 11 | } 12 | 13 | body, h1, h2, h3, h4, h5, h6, p, ol, ul { 14 | margin: 0; 15 | padding: 0; 16 | font-weight: normal; 17 | } 18 | 19 | img { 20 | max-width: 100%; 21 | height: auto; 22 | } 23 | ``` 24 | 25 | The CSS above will: 26 | 27 | - Reset is the `box-sizing: border-box`, as this will ensure consistent and predictable sizing. The default value of content-box doesn't account for the padding or border. This is probably the most important point. 28 | 29 | - Dont' bold headings by default with `font-weight: normal`. 30 | 31 | - Set the default font size to be 16px. Everything else can be specified in rem units and it will be based on those 16px. Then, if you want to adjust globally to make the text a little larger overall, you can change the base rule for something like 17 or 18px. 32 | 33 | - Make images responsive by default. 34 | -------------------------------------------------------------------------------- /programming-notes/html-css/css-selectors.md: -------------------------------------------------------------------------------- 1 | # CSS selectors 2 | 3 | In CSS, selectors are patterns used to select the element(s) you want to style. 4 | 5 | The "CSS" column indicates in which CSS version the property is defined (CSS1, CSS2, or CSS3). 6 | 7 | Selector | Example | Example description | CSS 8 | ----- | ----- | ----- | ----- 9 | .class | .intro | Selects all elements with class="intro" | 1 10 | #id | #firstname | Selects the element with id="firstname" | 1 11 | \* | \* | Selects all elements | 2 12 | element | p | Selects all \

elements | 1 13 | element,element | div, p | Selects all \

elements and all \

elements | 1 14 | element element | div p | Selects all \

elements inside \

elements | 1 15 | element>element | div > p | Selects all \

elements where the parent is a \

element | 2 16 | element+element | div + p | Selects all \

elements that are placed immediately after \

elements | 2 17 | element1~element2 | p ~ ul | Selects every \
    element that are preceded by a \

    element | 3 18 | :active | a:active | Selects the active link | 1 19 | ::after | p::after | Insert something after the content of each \

    element | 2 20 | ::before | p::before | Insert something before the content of each \

    element | 2 21 | :checked | input:checked | Selects every checked element | 3 22 | :disabled | input:disabled | Selects every disabled element | 3 23 | :empty | p:empty | Selects every \

    element that has no children (including text nodes) | 3 24 | :enabled | input:enabled | Selects every enabled element | 3 25 | :first-child | p:first-child | Selects every \

    element that is the first child of its parent | 2 26 | ::first-letter | p::first-letter | Selects the first letter of every \

    element | 1 27 | ::first-line | p::first-line | Selects the first line of every \

    element | 1 28 | :first-of-type | p:first-of-type | Selects every \

    element that is the first \

    element of its parent | 3 29 | :focus | input:focus | Selects the input element which has focus | 2 30 | :hover | a:hover | Selects links on mouse over | 1 31 | :in-range | input:in-range | Selects input elements with a value within a specified range | 3 32 | :invalid | input:invalid | Selects all input elements with an invalid value | 3 33 | :lang(language) | p:lang(it) | Selects every \

    element with a lang attribute equal to "it" (Italian) | 2 34 | :last-child | p:last-child | Selects every \

    element that is the last child of its parent | 3 35 | :last-of-type | p:last-of-type | Selects every \

    element that is the last \

    element of its parent | 3 36 | :link | a:link | Selects all unvisited links | 1 37 | :not(selector) | :not(p) | Selects every element that is not a \

    element | 3 38 | :nth-child(n) | p:nth-child(2) | Selects every \

    element that is the second child of its parent | 3 39 | :nth-last-child(n) | p:nth-last-child(2) | Selects every \

    element that is the second child of its parent, counting from the last child | 3 40 | :nth-last-of-type(n) | p:nth-last-of-type(2) | Selects every \

    element that is the second \

    element of its parent, counting from the last child | 3 41 | :nth-of-type(n) | p:nth-of-type(2) | Selects every \

    element that is the second \

    element of its parent | 3 42 | :only-of-type | p:only-of-type | Selects every \

    element that is the only \

    element of its parent | 3 43 | :only-child | p:only-child | Selects every \

    element that is the only child of its parent | 3 44 | :optional | input:optional | Selects input elements with no "required" attribute | 3 45 | :out-of-range | input:out-of-range | Selects input elements with a value outside a specified range | 3 46 | :read-only | input:read-only | Selects input elements with the "readonly" attribute specified | 3 47 | :read-write | input:read-write | Selects input elements with the "readonly" attribute NOT specified | 3 48 | :required | input:required | Selects input elements with the "required" attribute specified | 3 49 | :root | :root | Selects the document's root element | 3 50 | ::selection | ::selection | Selects the portion of an element that is selected by a user | 51 | :target | #news:target | Selects the current active #news element (clicked on a URL containing that anchor name) | 3 52 | :valid | input:valid | Selects all input elements with a valid value | 3 53 | :visited | a:visited | Selects all visited links | 1 54 | 55 | -------------------------------------------------------------------------------- /programming-notes/html-css/seo-meta-tags.md: -------------------------------------------------------------------------------- 1 | # Meta tags for SEO 2 | 3 | ### Title tag 4 | 5 | Title tags are usually used by search engines to determine the subject of a particular page and display it in SERPs. 6 | 7 | ```html 8 | Your Fantastic Title 9 | ``` 10 | 11 | 12 | ### Meta description tag 13 | 14 | Meta description is a short paragraph of text in the HTML section of a page. It is usually displayed in a SERP snippet after website's title and URL. 15 | 16 | ```html 17 | 18 | ``` 19 | 20 | 21 | ### Open Graph tags 22 | 23 | Open Graph (OG) tags meta tags in HTML section of a page that allow any webpage to become a rich object in social networks. 24 | 25 | ```html 26 | 27 | 28 | 29 | 30 | ``` 31 | 32 | 33 | ### Robots tag 34 | 35 | ```html 36 | 37 | ``` 38 | 39 | 40 | ### Canonical tag 41 | 42 | When you have a few pages with identical content, you can use a canonical tag to tell search engines which page should be prioritized. 43 | 44 | ```html 45 | 46 | ``` 47 | 48 | ### References 49 | 50 | [https://www.link-assistant.com/news/html-tags-for-seo.html](https://www.link-assistant.com/news/html-tags-for-seo.html) 51 | -------------------------------------------------------------------------------- /programming-notes/html-css/sticky-footer.md: -------------------------------------------------------------------------------- 1 | # Sticky footer 2 | 3 | Get the footer to stick to the bottom of the window even when there is not enough content to fill the page. 4 | 5 | HTML: 6 | ```html 7 | 8 |

    …
    9 |
    …
    10 |
    …
    11 | 12 | ``` 13 | 14 | CSS: 15 | ```css 16 | body { 17 | display: flex; 18 | min-height: 100vh; 19 | flex-direction: column; 20 | } 21 | 22 | main { 23 | flex: 1; 24 | } 25 | ``` 26 | 27 | ### References 28 | - [https://philipwalton.github.io/solved-by-flexbox/demos/sticky-footer/](https://philipwalton.github.io/solved-by-flexbox/demos/sticky-footer/) 29 | 30 | Other: 31 | - [https://css-tricks.com/couple-takes-sticky-footer/](https://css-tricks.com/couple-takes-sticky-footer/) 32 | - [https://getbootstrap.com/docs/4.0/examples/sticky-footer/](https://getbootstrap.com/docs/4.0/examples/sticky-footer/) 33 | -------------------------------------------------------------------------------- /programming-notes/index.md: -------------------------------------------------------------------------------- 1 | # Programming Notes 2 | 3 | A collection of programming notes, snippets and examples. By José Aniceto. 4 | 5 | [Click here to check the website](https://janiceto.github.io/programming-notes/) 6 | 7 | Below is a list of the most used pages. 8 | 9 | ## Python 10 | 11 | - [Simple Machine Learning workflow](/python/ml-workflows) 12 | - [Scientific quality figures](/python/scientific-figures) 13 | - [Python packages and modules](/python/packages) -------------------------------------------------------------------------------- /programming-notes/javascript/DataTables.md: -------------------------------------------------------------------------------- 1 | # DataTables JS Library 2 | 3 | [Documentation here](https://datatables.net/manual/index) 4 | 5 | ### Requirements: 6 | - JQuery 7 | 8 | ### Usage: 9 | Include the CSS and JS code for DataTables to the head and bottom of the body of your website, respectively. CSS is not required. 10 | 11 | ```html 12 | 13 | 14 | 15 | ``` 16 | 17 | Use the single function bellow to call to initialise the tabl. Do not forget to add an id tag to the table. 18 | ```javascript 19 | $(document).ready(function(){ 20 | $('#myTable').DataTable(); 21 | }); 22 | ``` 23 | 24 | Add options like so: 25 | ```javascript 26 | $(document).ready(function() { 27 | $('#myTable').DataTable( { 28 | "columnDefs": [ 29 | { "orderable": false, "targets": [2,3,4] } 30 | ], 31 | "paging": false, 32 | "order": [[ 0, "asc" ]] 33 | } ); 34 | } ); 35 | ``` 36 | -------------------------------------------------------------------------------- /programming-notes/javascript/ajax.md: -------------------------------------------------------------------------------- 1 | # Making AJAX request 2 | 3 | ## Send a Request To a Server (Pure Javascript) 4 | 5 | ```javascript 6 | // Create a new XMLHttpRequest 7 | var request = new XMLHttpRequest(); 8 | 9 | // Handle state changes for the request. 10 | request.onreadystatechange = function() { 11 | if (request.readyState == 4 && request.status == 200) { 12 | // Do something like: 13 | var jsonData = JSON.parse(request.responseText); 14 | } 15 | }; 16 | 17 | // Set up and make the request 18 | request.open("GET", "ajax_info.txt", true); 19 | request.send(); 20 | ``` 21 | 22 | Where: 23 | 24 | * `open(method, url, async)`: Specifies the type of request. *method*: the type of request: GET or POST; *url*: the server (file) location; *async*: true (asynchronous) or false (synchronous) 25 | 26 | * `send()`: Sends the request to the server (used for GET) 27 | 28 | * `send(string)`: Sends the request to the server (used for POST). For instance: `xhttp.send("fname=Henry&lname=Ford");` 29 | 30 | * `onreadystatechange`: defines a function to be executed when the request receives an answer 31 | -------------------------------------------------------------------------------- /programming-notes/javascript/basics.md: -------------------------------------------------------------------------------- 1 | # Basic Javascript concepts 2 | 3 | ## Variables 4 | 5 | There are three main keywords used to define variables in JavaScript: 6 | - `const`: defines a constant variable that cannot be redefined later 7 | - `let`: defines a variable is local to the scope of the innermost pair of curly braces surrounding it 8 | - `var`: defines a variable that is local to the function it is defined in 9 | 10 | Here is an example showcasing these different ways to define variables: 11 | 12 | ```javascript 13 | // This variable exists even outside the loop 14 | if (true) { 15 | var message = 'Hello!'; 16 | } 17 | 18 | alert(message); 19 | ``` 20 | 21 | Because `var` was used to define message, there will be no errors running this code. 22 | 23 | ```javascript 24 | // This variable does not exist outside the loop 25 | if (true) { 26 | let message = 'Hello!'; 27 | } 28 | 29 | alert(message); 30 | ``` 31 | 32 | Because `let` was used to define message, it cannot be passed to alert, which is outside the scope of message. If this were in an HTML page, when the page was opened, no alert would pop up. If the console were opened in the browser, there would be an `Uncaught ReferenceError`. 33 | 34 | ```javascript 35 | // The value of const variables cannot change 36 | const message = 'Hello!'; 37 | message = 'Goodbye!'; 38 | 39 | alert(message); 40 | ``` 41 | 42 | Similar to the last example, no alert will pop up. In the console, there would be an `Uncaught TypeError`, since there was an attempt to redefine a variable defined with `const`. 43 | 44 | ## Template literals (formatting strings) 45 | 46 | ```javascript 47 | const name = 'Daniel'; 48 | 49 | alert(`Hello ${name}!`); 50 | ``` 51 | 52 | 53 | ## Arrow functions 54 | 55 | Since functions, especially anonymous functions, are so common in JavaScript, ES6 has introduced a new syntax for functions called arrow notation that allows for the definition of so-called arrow functions. 56 | 57 | ```javascript 58 | () => { 59 | alert('Hello, world!'); 60 | } 61 | 62 | x => { 63 | alert(x); 64 | } 65 | 66 | x => x * 2; 67 | ``` 68 | 69 | An arrow function is defined without using the word function, but rather just with a pair of parentheses enclosing any arguments the function takes, followed by an arrow, and finally the function body, enclosed in curly braces. Functions with only one argument can be defined without the use of parentheses enclosing the argument list. Functions that have only one line in the body can drop the curly braces and have the body on the same line as the argument list and arrow. Here's an example: 70 | 71 | ```javascript 72 | document.addEventListener('DOMContentLoaded', () => { 73 | // Have each button change the color of the heading 74 | document.querySelectorAll('.color-change').forEach(button => { 75 | button.onclick = () => { 76 | document.querySelector('#hello').style.color = button.dataset.color; 77 | }; 78 | }); 79 | }); 80 | ``` 81 | -------------------------------------------------------------------------------- /programming-notes/javascript/cache-data.md: -------------------------------------------------------------------------------- 1 | # Caching data 2 | 3 | You have three options: 4 | 5 | 1. Cookies: https://developer.mozilla.org/en-US/docs/DOM/document.cookie 6 | 2. DOMStorage (sessionStorage or localStorage): https://developer.mozilla.org/en-US/docs/DOM/Storage 7 | 3. If your users are logged in, you could persist data in your server's DB that is keyed to a user (or group) 8 | 9 | 10 | ### Using localStorage (persistent over sessions) 11 | 12 | Writing : 13 | ```javascript 14 | localStorage['myKey'] = 'somestring'; // only strings 15 | ``` 16 | 17 | Reading : 18 | ```javascript 19 | var myVar = localStorage['myKey'] || 'defaultValue'; 20 | ``` 21 | 22 | If you need to store complex structures, you might serialize them in JSON. For example : 23 | 24 | Reading : 25 | ```javascript 26 | var stored = localStorage['myKey']; 27 | if (stored) myVar = JSON.parse(stored); 28 | else myVar = {a:'test', b: [1, 2, 3]}; 29 | ``` 30 | 31 | Writing : 32 | ```javascript 33 | localStorage['myKey'] = JSON.stringify(myVar); 34 | ``` 35 | 36 | Note that you may use more than one key. They'll all be retrieved by all pages on the same domain. 37 | 38 | Unless you want to be compatible with IE7, you have no reason to use the obsolete and small cookies. 39 | 40 | 41 | ### References 42 | https://stackoverflow.com/questions/14266730/js-how-to-cache-a-variable 43 | -------------------------------------------------------------------------------- /programming-notes/javascript/create-react-app.md: -------------------------------------------------------------------------------- 1 | # Create ReactJS app and deploy to Github pages 2 | 3 | ## Requirements 4 | - **NodeJS** Install from [here](https://nodejs.org/en/). Check version with: `$ node --version` 5 | - **npm** Check version with: `$ npm --version` 6 | 7 | ## Create React app 8 | 9 | You can `npm` install `create-react-app` globally with: 10 | ``` 11 | $ npm install -g create-react-app 12 | $ create-react-app my-app 13 | ``` 14 | 15 | Alternatively, you can use `npx` (a tool to execute packages) to create the app withput installing `create-react-app`. This way is recommended by the docs. 16 | 17 | ``` 18 | $ npx create-react-app my-app 19 | $ cd my-app 20 | $ npm start 21 | ``` 22 | 23 | This will lunch a server where you can see your app. 24 | 25 | 26 | ## Deploying to Github Pages 27 | 28 | ### 1) Install the gh-pages package as a "dev-dependency" of the app 29 | ``` 30 | $ npm install gh-pages --save-dev 31 | ``` 32 | 33 | ### 2) Create an empty repository on Github 34 | 35 | Go to github.com and create a repo. We will name it named `react-gh-pages`. 36 | 37 | ### 3) Modify the package.json file 38 | 39 | At the top level, add a homepage property: 40 | ``` 41 | "homepage": "https://gitname.github.io/react-gh-pages" 42 | ``` 43 | 44 | In the existing scripts property, add the following: 45 | ``` 46 | "scripts": { 47 | //... 48 | "predeploy": "npm run build", 49 | "deploy": "gh-pages -d build" 50 | } 51 | ``` 52 | 53 | ### 4) Add the GitHub repository as a "remote" in your local git repository 54 | ``` 55 | $ git remote add origin https://github.com/gitname/react-gh-pages.git 56 | ``` 57 | 58 | ### 5) Generate a production build of your app, and deploy it to GitHub Pages 59 | ``` 60 | $ npm run deploy 61 | ``` 62 | 63 | The app is now accessible at https://gitname.github.io/react-gh-pages/ 64 | 65 | 66 | ### 6) Optionally, commit your source code to the "master" branch and push your commit to GitHub. 67 | ``` 68 | $ git add . 69 | $ git commit -m "Create a React app and publish it to GitHub Pages" 70 | $ git push origin master 71 | ``` 72 | 73 | So, the `master` branch helds the source code, and the `gh-pages` branch helds the built app code. 74 | 75 | 76 | ## References 77 | 78 | - [react-gh-pages](https://github.com/gitname/react-gh-pages) 79 | - [Create React App](https://create-react-app.dev/docs/getting-started) 80 | -------------------------------------------------------------------------------- /programming-notes/javascript/expiration-localstorage.md: -------------------------------------------------------------------------------- 1 | # Set expiration time (TTL) for LocalStorage 2 | 3 | It's not possible to specify expiration for items saved to a browser LocalStorage. However, we can use Javascript to add a TTL (Time To Live) to invalidate items in localStorage after a certain period of time elapses. 4 | 5 | ## Storing items with expiration time 6 | 7 | Set a key in localStorage, and store the expiration time along with it: 8 | 9 | ```javascript 10 | function setWithExpiry(key, value, ttl) { 11 | const now = new Date() 12 | 13 | // item is an object which contains the original value as well as the time when it's supposed to expire 14 | const item = { 15 | value: value, 16 | expiry: now.getTime() + ttl, 17 | } 18 | localStorage.setItem(key, JSON.stringify(item)) 19 | } 20 | ``` 21 | 22 | ## Storing items with expiration time 23 | 24 | Verify the expiration time while retrieving items from localStorage: 25 | 26 | ```javascript 27 | function getWithExpiry(key) { 28 | const itemStr = localStorage.getItem(key) 29 | // if the item doesn't exist, return null 30 | if (!itemStr) { 31 | return null 32 | } 33 | const item = JSON.parse(itemStr) 34 | const now = new Date() 35 | // compare the expiry time of the item with the current time 36 | if (now.getTime() > item.expiry) { 37 | // If the item is expired, delete the item from storage and return null 38 | localStorage.removeItem(key) 39 | return null 40 | } 41 | return item.value 42 | } 43 | ``` 44 | 45 | ## References 46 | 47 | - [How to Set Expiry Time (TTL) for LocalStorage With Javascript](https://www.sohamkamani.com/blog/javascript-localstorage-with-ttl-expiry/) 48 | -------------------------------------------------------------------------------- /programming-notes/latex/beamer.md: -------------------------------------------------------------------------------- 1 | # Create LaTeX presentations with Beamer 2 | 3 | ## Templates 4 | 5 | A good theme compilation can be found at: [The Ultimate Beamer Theme List](https://github.com/martinbjeldbak/ultimate-beamer-theme-list) 6 | 7 | Here are a few select examples. 8 | 9 | 10 | [Metropolis](https://github.com/matze/mtheme) 11 | 12 | --- 13 | 14 | [Execushares](https://github.com/hamaluik/Beamer-Theme-Execushares) 15 | 16 | --- 17 | 18 | ![](https://writelatex.s3.amazonaws.com/published_ver/39692.jpeg?X-Amz-Expires=14400&X-Amz-Date=20241128T133038Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWJBOALPNFPV7PVH5/20241128/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=68013045953bba2c202ba2f41c2fb8b14c0faed1bb03d7578136f088e39af0f9) 19 | 20 | [wildcat](https://www.overleaf.com/latex/templates/wildcat/knynymwgrxxj) 21 | 22 | --- 23 | 24 | [ant-center-brief](https://www.overleaf.com/latex/templates/ant-center-brief/rhkgyzdnkhhn) 25 | 26 | --- 27 | 28 | ![](https://writelatex.s3.amazonaws.com/published_ver/40062.jpeg?X-Amz-Expires=14400&X-Amz-Date=20241128T132615Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWJBOALPNFPV7PVH5/20241128/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=afc753ac67a33c1c97627e18176f3c4abcb12567e11c29412757c8f815619462) 29 | 30 | [uic-presentation-template](https://www.overleaf.com/latex/templates/uic-presentation-template/dgjbtyvtgqcg) 31 | -------------------------------------------------------------------------------- /programming-notes/latex/bibliography.md: -------------------------------------------------------------------------------- 1 | # Bibliography management 2 | 3 | ## Using `bibtext` (default) 4 | 5 | ```latex 6 | \bibliographystyle{stylename} 7 | \bibliography{bibfile} 8 | ``` 9 | 10 | Where `bibfile` is the name of the bibliography .bib file, without the extension, and `stylename` is the bibliography style. A simple numeric style is `unsrt` 11 | 12 | 13 | ## Using `biblatext` package 14 | 15 | `biblatext` is the most complete and flexible bibliography tool in the LaTeX world. 16 | 17 | ```latex 18 | \documentclass[12pt]{article} 19 | 20 | \usepackage[backend=biber,style=numeric,sorting=ynt]{biblatex} 21 | \addbibresource{journals.bib,phd-references.bib} 22 | 23 | \begin{document} 24 | 25 | \cite{robertson2007} 26 | \cite{earnshaw1842} 27 | 28 | \printbibliography 29 | 30 | \end{document} 31 | ``` 32 | 33 | Where journals.bib and phd-references.bib are BibTeX databases 34 | 35 | ## The bibliography database file (`.bib`) 36 | 37 | ```bibtex 38 | @article{einstein, 39 | author = "Albert Einstein", 40 | title = "{Zur Elektrodynamik bewegter K{\"o}rper}. ({German}) 41 | [{On} the electrodynamics of moving bodies]", 42 | journal = "Annalen der Physik", 43 | volume = "322", 44 | number = "10", 45 | pages = "891--921", 46 | year = "1905", 47 | DOI = "http://dx.doi.org/10.1002/andp.19053221004", 48 | keywords = "physics" 49 | } 50 | 51 | @book{dirac, 52 | title={The Principles of Quantum Mechanics}, 53 | author={Paul Adrien Maurice Dirac}, 54 | isbn={9780198520115}, 55 | series={International series of monographs on physics}, 56 | year={1981}, 57 | publisher={Clarendon Press}, 58 | keywords = {physics} 59 | } 60 | 61 | @online{knuthwebsite, 62 | author = "Donald Knuth", 63 | title = "Knuth: Computers and Typesetting", 64 | url = "http://www-cs-faculty.stanford.edu/~uno/abcde.html", 65 | keywords = "latex,knuth" 66 | } 67 | 68 | @inbook{knuth-fa, 69 | author = "Donald E. Knuth", 70 | title = "Fundamental Algorithms", 71 | publisher = "Addison-Wesley", 72 | year = "1973", 73 | chapter = "1.2", 74 | keywords = "knuth,programming" 75 | } 76 | ... 77 | ``` 78 | 79 | ## References 80 | 81 | - [Bibtex bibliography styles - Overleaf](https://www.overleaf.com/learn/latex/bibtex_bibliography_styles) 82 | - [Bibliography management with biblatex - Overleaf](https://www.overleaf.com/learn/latex/Bibliography_management_with_biblatex) 83 | -------------------------------------------------------------------------------- /programming-notes/latex/derivatives.md: -------------------------------------------------------------------------------- 1 | # Derivatives 2 | 3 | The [`esdiff`](https://ctan.org/pkg/esdiff?lang=en) package has handy macros for derivatives and partial derivatives, taking care of indices. 4 | 5 | ```latex 6 | \documentclass{article} 7 | 8 | \usepackage[utf8]{inputenc} 9 | \usepackage[T1]{fontenc} 10 | \usepackage{mathtools} 11 | \usepackage[thinc]{esdiff} 12 | 13 | \begin{document} 14 | 15 | First order derivative: df/dx 16 | 17 | \begin{equation} 18 | \diff{f}{x} 19 | \end{equation} 20 | 21 | Fourth order derivative: d4f/dx4|x=1 22 | 23 | \begin{equation} 24 | \diff*[4]{f}{x}{x = 1} 25 | \end{equation} 26 | 27 | First order partial derivative: 28 | 29 | \begin{equation} 30 | \diffp{f}{x} 31 | \end{equation} 32 | 33 | Second order crossed partial derivative 34 | 35 | \begin{equation} 36 | \diffp{g}{tu} 37 | \end{equation} 38 | 39 | \end{document} 40 | ``` 41 | 42 | 43 | ## References: 44 | 45 | - [Is there a short hand command to write derivatives?](https://tex.stackexchange.com/questions/412439/is-there-a-short-hand-command-to-write-derivatives) 46 | -------------------------------------------------------------------------------- /programming-notes/latex/figures.md: -------------------------------------------------------------------------------- 1 | # Working with figures 2 | 3 | ## Simple figure 4 | 5 | ```latex 6 | \begin{figure}[!htbp] 7 | \centering 8 | \includegraphics{path/to/figure.pdf} 9 | \caption{Figure caption here.} 10 | \label{fig:fig-label} 11 | \end{figure} 12 | ``` 13 | 14 | The `!htbp` forces the image position. 15 | 16 | 17 | ## Figures side by side 18 | 19 | ```latex 20 | \usepackage{graphicx} 21 | \usepackage{caption} 22 | \usepackage{subcaption} 23 | 24 | 25 | \begin{figure} 26 | \centering 27 | \begin{subfigure}{0.45\textwidth} 28 | \includegraphics{fig1.pdf} 29 | \label{fig:fig1} 30 | \caption{fig1 caption} 31 | \end{subfigure} 32 | 33 | \begin{subfigure}{0.45\textwidth} 34 | \centering 35 | \includegraphics{fig2.pdf} 36 | \label{fig:fig2} 37 | \caption{fig2 caption} 38 | \end{subfigure} 39 | \caption{global caption} 40 | \end{figure} 41 | ``` 42 | -------------------------------------------------------------------------------- /programming-notes/latex/latex-intro.md: -------------------------------------------------------------------------------- 1 | # An Introduction to LaTeX 2 | 3 | ## Managing large documents 4 | 5 | For smaller projects it is okay to keep everything in a single `.tex` file. For more involved projects this approach quickly becomes cumbersome. The `\include` command makes it possible to break your document down into smaller chunks. Working with smaller chunks is more manageable. An example structure for a thesis project could look like the following: 6 | 7 | ``` 8 | thesis/ 9 | |-- thesis.tex 10 | |-- chapters/ 11 | |-- chapter_1.tex 12 | |-- chapter_2.tex 13 | |-- chapter_3.tex 14 | |-- internal/ 15 | |-- preamble.tex 16 | |-- fig/ 17 | |-- science.png 18 | |-- references.bib 19 | ``` 20 | 21 | Example `thesis.tex`: 22 | 23 | ```latex 24 | \documentclass[12pt]{report} 25 | 26 | \include{internal/preamble} 27 | 28 | \begin{document} 29 | 30 | \include{chapters/chapter _1} 31 | \include{chapters/chapter _2} 32 | \include{chapters/chapter _3} 33 | 34 | \bibliography{references} 35 | 36 | \end{document} 37 | ``` 38 | 39 | Example `internal/preamble.tex`: 40 | 41 | ```latex 42 | % Preamble, packages, commands, etc . 43 | \ usepackage{microtype} 44 | \ usepackage{booktabs} 45 | \ usepackage{cleveref} 46 | \ usepackage{graphicx} 47 | 48 | % Make it easier to include figures 49 | \graphicspath{{fig/}} 50 | ``` 51 | 52 | Example `chapters/chapter_1.tex`: 53 | 54 | ```python 55 | \chapter{Literature review} 56 | \label{cha:lit _ review} 57 | 58 | Here's stuff others did which I don't really understand \ldots 59 | ``` 60 | 61 | ## Custom commands 62 | 63 | ### Simple macros 64 | 65 | Used to simplify repetitive and/or complex formatting. Usually specified in the preamble: 66 | 67 | ```latex 68 | \newcommand{\name}{definition} 69 | ``` 70 | 71 | ### Macros with parameters 72 | 73 | ```latex 74 | \newcommand{\name}[#params]{definition} 75 | ``` 76 | 77 | For example: 78 | 79 | ```latex 80 | \newcommand{\bb}[1]{\mathbb{#1}} 81 | ``` 82 | 83 | 84 | ### Macros with default parameters 85 | 86 | ```latex 87 | \newcommand{\name}[# params][default #1]{def.} 88 | ``` 89 | 90 | For example: 91 | 92 | ```latex 93 | \newcommand{\plusbinomial}[3][2]{(#2 + #3)^#1} 94 | ``` 95 | 96 | ## References 97 | 98 | - [Preparing your thesis with LaTeX](https://jwalton.info/assets/teaching/latex/slides.pdf) 99 | - [Thesis based on Tufte style](http://www.ccs.neu.edu/home/turon/thesis.pdf) 100 | - [ClassicThesis – A “classically styled” thesis package](https://ctan.org/pkg/classicthesis?lang=en) 101 | - [Entry types which BibTeX understands](https://bib-it.sourceforge.net/help/fieldsAndEntryTypes.php#Entries) 102 | - [Plot publication-quality figures with matplotlib and LaTeX](https://jwalton.info/Embed-Publication-Matplotlib-Latex/) 103 | -------------------------------------------------------------------------------- /programming-notes/linux/crontab.md: -------------------------------------------------------------------------------- 1 | # Linux Crontab 2 | 3 | To view currently active crontab entries: `$ sudo crontab -l` 4 | 5 | To enter the edit the crontab enter the following command in the terminal: `$ sudo crontab -e` 6 | 7 | **Note:** Don't use `sudo` in a cron job. Instead edit root's crontab instead of your own, e.g. `sudo crontab -e` and then enter commands without `sudo`. 8 | 9 | At the bottom of the file enter one line for each task in the following format: 10 | 11 | ``` 12 | Minute Hour Day of Month Month Day of Week Command 13 | (0-59) (0-23) (1-31) (1-12 or Jan-Dec) (0-6 or Sun-Sat) 14 | ``` 15 | 16 | The command must use a complete link. Instead of the first five fields, one of eight special strings may appear: 17 | 18 | string | meaning 19 | --- | --- 20 | @reboot | Run once, at startup. 21 | @yearly | Run once a year, "0 0 1 1 *". 22 | @annually | (same as @yearly) 23 | @monthly | Run once a month, "0 0 1 * *". 24 | @weekly | Run once a week, "0 0 * * 0". 25 | @daily | Run once a day, "0 0 * * *". 26 | @midnight | (same as @daily) 27 | @hourly | Run once an hour, "0 * * * *". 28 | 29 | #### Example 1: Run a python script every day a 16:15 30 | ``` 31 | 15 16 * * * sudo python /home/pi/projects/script1.py 32 | ``` 33 | 34 | #### Example 2: Run a python script every five days a 18:30 35 | ``` 36 | 30 18 */5 * * sudo python /home/pi/projects/script1.py 37 | ``` 38 | 39 | ### Crontab log 40 | 41 | On a default installation the cron jobs get logged to 42 | ``` 43 | $ /var/log/syslog 44 | ``` 45 | 46 | You can see just cron jobs in that logfile by running 47 | ``` 48 | $ grep CRON /var/log/syslog 49 | ``` 50 | 51 | 52 | ### Helpfull links: 53 | 54 | https://crontab.guru 55 | -------------------------------------------------------------------------------- /programming-notes/linux/django-production-server.md: -------------------------------------------------------------------------------- 1 | # Set up a Django production server with gunicorn 2 | 3 | We are going to set up a production ready Django server using Gunicorn and manage it using systemd. 4 | 5 | 6 | ## 1) Install Gunicorn 7 | 8 | Inside the virtual environment with Django and all other required packages for your project run: 9 | 10 | ``` 11 | $ pip install gunicorn 12 | ``` 13 | 14 | ## 2) Configure Gunicorn 15 | 16 | In your project directory, create a file named `gunicorn_config.py` with the Gunicorn configuration for your Django project: 17 | 18 | ```python 19 | import multiprocessing 20 | 21 | bind = "127.0.0.1:8000" # Replace with your desired IP and port 22 | workers = multiprocessing.cpu_count() * 2 + 1 23 | worker_class = "gthread" 24 | threads = 2 25 | timeout = 60 26 | ``` 27 | 28 | To test that Gunicorn can serve your Django app run: 29 | 30 | ``` 31 | $ gunicorn -c gunicorn_config.py myproject.wsgi 32 | ``` 33 | 34 | If Gunicorn starts without errors, it's working correctly. You can stop it by pressing `Ctrl + C`. 35 | 36 | ## 3) Create a `systemd` service 37 | 38 | Create a `systemd` service file to manage the Gunicorn process. Use a text editor to create a file named `myproject_gunicorn.service` in the `/etc/systemd/system/` directory: 39 | 40 | ``` 41 | $ sudo nano /etc/systemd/system/myproject_gunicorn.service 42 | ``` 43 | 44 | Add the following content to the file, adjusting the paths and configuration as neede. Replace `your_username`, `your_group`, `/path/to/your/project`, and `/path/to/your/virtualenv` with your actual information. 45 | 46 | ``` 47 | [Unit] 48 | Description=Gunicorn daemon for myproject 49 | After=network.target 50 | 51 | [Service] 52 | User=your_username 53 | Group=your_group 54 | WorkingDirectory=/path/to/your/project 55 | ExecStart=/path/to/your/virtualenv/bin/gunicorn -c gunicorn_config.py myproject.wsgi 56 | Restart=always 57 | 58 | [Install] 59 | WantedBy=multi-user.target 60 | ``` 61 | 62 | ## 4) Enable and start the `systemd` service 63 | 64 | Enable the systemd service and start Gunicorn: 65 | 66 | ``` 67 | $ sudo systemctl enable myproject_gunicorn 68 | $ sudo systemctl start myproject_gunicorn 69 | ``` 70 | 71 | Check the status of the Gunicorn service to make sure it's running without errors: 72 | 73 | ``` 74 | $ sudo systemctl status myproject_gunicorn 75 | ``` 76 | 77 | If everything is configured correctly, your Django application should now be running in production using Gunicorn and managed by systemd. You can access it through the specified IP and port. Make sure to configure your web server (e.g., Nginx or Apache) as a reverse proxy to forward requests to Gunicorn for better security and performance in a production environment. 78 | 79 | ## References 80 | 81 | - [https://dev.to/karthiknarayan/setting-up-django-for-production-using-gunicorn-on-linux-37ce](https://dev.to/karthiknarayan/setting-up-django-for-production-using-gunicorn-on-linux-37ce) 82 | -------------------------------------------------------------------------------- /programming-notes/linux/keep-scripts-running-ssh.md: -------------------------------------------------------------------------------- 1 | # Keep processes running and closing SSH connection 2 | 3 | The are several alternatives. Here we cover `nohup` and `tmux`. 4 | 5 | ## Using `nohup` 6 | 7 | ``` 8 | $ nohup long-running-command & 9 | ``` 10 | It logs `stdout` to `nohup.log`. 11 | 12 | 13 | ## Using `tmux` 14 | 15 | 1) SSH into the remote machine. 16 | 17 | 2) Install `tmux`: 18 | ``` 19 | $ apt install tmux 20 | ``` 21 | 22 | 3) Start `tmux`: 23 | ``` 24 | $ tmux 25 | ``` 26 | 27 | 4) Start the process you want inside the started `tmux` session. 28 | 29 | 5) Leave/detach the `tmux` session by typing `Ctrl+b` and then `d`. 30 | 31 | 6) You can now safely log off from the remote machine, your process will keep running inside `tmux`. 32 | 33 | 7) When you come back again and want to check the status of your process you can attach to your tmux session using: 34 | ``` 35 | $ tmux attach 36 | ``` 37 | 38 | If you want to have multiple sessions running side-by-side, you should name each session using `Ctrl+b` and `$`. 39 | 40 | You can get a list of the currently running sessions using: 41 | ``` 42 | $ tmux list-sessions 43 | ``` 44 | or 45 | ``` 46 | $ tmux ls 47 | ``` 48 | 49 | To attach to a running session with use: 50 | ``` 51 | $ tmux attach-session -t 52 | ``` 53 | -------------------------------------------------------------------------------- /programming-notes/linux/keep-ssh-running.md: -------------------------------------------------------------------------------- 1 | # Keep SSH sessions running after disconnecting 2 | 3 | ### Using nohup 4 | 5 | ``` 6 | $ nohup long-running-process & 7 | $ exit 8 | ``` 9 | 10 | ### Using GNU Screen 11 | 12 | ``` 13 | $ screen # to start a screen session 14 | $ run-a-process 15 | CTRL+a , d # to detatch from your screen session 16 | $ exit # to disconnect from the server, while run-a-process continues 17 | $ screen -r # to resume the screen session when you come back to your laptop 18 | ``` 19 | 20 | ### Using tmux 21 | ``` 22 | $ tmux # to start a screen session 23 | $ run-a-process 24 | Ctrl+b then d # to detatch from your session 25 | $ tmux attach # to resume the session when you come back to your laptop 26 | ``` 27 | -------------------------------------------------------------------------------- /programming-notes/linux/path.md: -------------------------------------------------------------------------------- 1 | # Set `$PATH` variable in Ubuntu 2 | 3 | The `$PATH` variable is one of the default environment variable in linux (ubuntu). It is used by the shell to look for executable files or commands. 4 | 5 | One way to permanently add a directory to `$PATH` environment variable is to use the `~/.profile` file. Add the following to the `~/.profile` file to add `myNewDir` to `$PATH` 6 | 7 | ``` 8 | $ export PATH=$PATH:/myNewDir 9 | $ source ~/.profile 10 | ``` 11 | 12 | ## The `source` command 13 | 14 | `source` is a bash shell built-in command that executes the content of the file passed as argument, **in the current shell**. It has a synonym in `.` (period). 15 | 16 | Note that `./` and `source` are not quite the same: 17 | 18 | - `./script` runs the script as an executable file, launching a new shell to run it 19 | - `source some_script` reads and executes commands from filename in the current shell environment 20 | - `source some_script` is the same as `. some_script` 21 | -------------------------------------------------------------------------------- /programming-notes/linux/raspberrypi.md: -------------------------------------------------------------------------------- 1 | # Set up server in Raspberry Pi 2 | 3 | ## Set up Django web app locally 4 | 5 | Find you IP: 6 | 7 | ``` 8 | $ ifconfig 9 | ``` 10 | 11 | Run: 12 | 13 | ``` 14 | $ python manage.py runserver 192.XXX.XX.XX:8000 15 | ``` 16 | 17 | You should be able to access your web app from any device in the local network. 18 | 19 | This will keep the web app running while the SSH connection is active. 20 | 21 | If you want to serve the app continously you can use `gunicorn`: 22 | 23 | ``` 24 | $ gunicorn --bind 192.XXX.XX.XX:8000 your_project.wsgi --daemon 25 | ``` 26 | 27 | 28 | ## References 29 | 30 | - [[stackoverflow](https://stackoverflow.com/questions/13654688/what-is-the-correct-way-to-leave-gunicorn-running)](https://stackoverflow.com/questions/13654688/what-is-the-correct-way-to-leave-gunicorn-running) 31 | -------------------------------------------------------------------------------- /programming-notes/linux/server-setup.md: -------------------------------------------------------------------------------- 1 | # Initial Server Setup with Ubuntu 18.04 2 | 3 | When first creating a new Ubuntu 18.04 server, there are a few configuration steps that you should take early on as part of the basic setup. 4 | 5 | ## 1) Creating a New User 6 | 7 | Once you are logged in as root, we're prepared to add the new user (synergix) account that we will use to log in from now on. 8 | 9 | `adduser synergix` 10 | 11 | Enter a strong password and, optionally, fill in any of the additional information if you would like. This is not required and you can just hit ENTER in any field you wish to skip. 12 | 13 | ## 2) Granting Administrative Privileges 14 | 15 | Set up what is known as "superuser" or root privileges for our normal account. This will allow our normal user to run commands with administrative privileges by putting the word `sudo` before each command. As **root**, run this command to add your new user to the sudo group: 16 | 17 | `usermod -aG sudo synergix` 18 | 19 | ## 3) Setting Up a Basic Firewall 20 | 21 | Ubuntu 18.04 servers can use the UFW firewall to make sure only connections to certain services are allowed. We can set up a basic firewall very easily using this application. 22 | 23 | Different applications can register their profiles with UFW upon installation. These profiles allow UFW to manage these applications by name. OpenSSH, the service allowing us to connect to our server now, has a profile registered with UFW. You can see this by typing: 24 | 25 | `ufw app list` 26 | 27 | Afterwards, we can enable the firewall by typing: 28 | 29 | `ufw enable` 30 | 31 | Type "y" and press ENTER to proceed. You can see that SSH connections are still allowed by typing: 32 | 33 | `ufw status` 34 | 35 | As the firewall is currently blocking all connections except for SSH, if you install and configure additional services, you will need to adjust the firewall settings to allow acceptable traffic in. More info here: https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands 36 | 37 | ## 4) Enabling External Access for Your Regular User 38 | 39 | Now that we have a regular user for daily use, we need to make sure we can SSH into the account directly. The process for configuring SSH access for your new user depends on whether your server's root account uses a password or SSH keys for authentication. 40 | 41 | #### If the Root Account Uses Password Authentication 42 | 43 | If you logged in to your root account using a password, then password authentication is enabled for SSH. You can SSH to your new user account by opening up a new terminal session and using SSH with your new username: 44 | 45 | `ssh synergix@your_server_ip` 46 | 47 | After entering your regular user's password, you will be logged in. You will be prompted for your regular user password when using sudo for the first time each session (and periodically afterwards). 48 | 49 | To enhance your server's security, it is recommended to set up SSH keys instead of using password authentication: https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-ubuntu-1804 50 | 51 | #### If the Root Account Uses SSH Key Authentication 52 | 53 | If you logged in to your root account using SSH keys, then password authentication is disabled for SSH. You will need to add a copy of your local public key to the new user's `~/.ssh/authorized_keys` file to log in successfully. 54 | 55 | Since your public key is already in the root account's `~/.ssh/authorized_keys` file on the server, we can copy that file and directory structure to our new user account in our existing session. 56 | 57 | The simplest way to copy the files with the correct ownership and permissions is with the `rsync` command. This will copy the root user's .ssh directory, preserve the permissions, and modify the file owners, all in a single command: 58 | 59 | `rsync --archive --chown=synergix:synergix ~/.ssh /home/synergix` 60 | 61 | Now, open up a new terminal session and using SSH with your new username: 62 | 63 | `ssh synergix@your_server_ip` 64 | 65 | You should be logged in to the new user account without using a password. 66 | 67 | ## 5) Other 68 | 69 | Chose the timezone by running: `sudo dpkg-reconfigure tzdata` 70 | 71 | ## References: 72 | * https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-18-04 73 | -------------------------------------------------------------------------------- /programming-notes/matlab/artificial-data.md: -------------------------------------------------------------------------------- 1 | # Create artificial data 2 | 3 | Sometimes we want to create artificial (synthetic) data so that we can use it in an example of in a test. For instance, imagine we have a script that fits a function to data and we want to test it on some data. We can generate that data artificially. 4 | 5 | ## General approach 6 | 7 | ```matlab 8 | % Define the range of x values 9 | x = linspace(0, 10, 100); 10 | 11 | % Define the true function (e.g., a quadratic) 12 | y_true = 2*x.^2 + 3*x + 5; 13 | 14 | % Add some random noise to simulate measurement errors 15 | noise = randn(size(x)) * 10; % Gaussian noise with std dev = 10 16 | y_noisy = y_true + noise; 17 | 18 | % Plot the noisy data 19 | figure; 20 | plot(x, y_noisy, 'o'); 21 | hold on; 22 | plot(x, y_true, '-'); 23 | legend('Noisy Data', 'Function'); 24 | xlabel('x'); 25 | ylabel('y'); 26 | ``` 27 | 28 | ## Types of noise 29 | 30 | The most common noise types you can add include Gaussian (normal), uniform, and Poisson. 31 | 32 | ### Gaussian noise 33 | 34 | Simulates measurement errors or natural variation in physical systems. 35 | 36 | ```matlab 37 | x = linspace(0, 10, 100); 38 | 39 | y_true = 2*x.^2 + 3*x + 5; % function 40 | 41 | sigma = 10; % standard deviation of noise 42 | noise = randn(size(x)) * sigma; 43 | y_noisy = y_true + noise; 44 | ``` 45 | 46 | Using `randn` gives standard normal values (mean 0 and standard deviation 1). `sigma` controls the intensity of the noise. 47 | 48 | 49 | ### Uniform Noise 50 | 51 | Every value within a range is equally likely. This is less realistic, but can be useful. 52 | 53 | ```matlab 54 | range = 5; % total range of uniform noise 55 | noise = (rand(size(x)) - 0.5) * range; 56 | y_noisy = y_true + noise; 57 | ``` 58 | 59 | Using `rand` generates values between [0, 1], we can then shift and scale to get the desired range. 60 | 61 | 62 | ### Poisson Noise 63 | 64 | Useful for count data. 65 | 66 | ```matlab 67 | y_true = 20 * exp(-0.5*x); % expected count values 68 | y_noisy = poissrnd(y_true); % Poisson-distributed noise 69 | ``` 70 | The `poissrnd` function takes the expected (mean) value for each x. The output is directly a Poisson-distributed version of the expected value. It requires the Statistics and Machine Learning Toolbox. 71 | -------------------------------------------------------------------------------- /programming-notes/matlab/doe.md: -------------------------------------------------------------------------------- 1 | # Design of Experiments 2 | 3 | **Note: Requires Matlab Statistics Toolbox.** 4 | 5 | The Statistics Toolbox offers a collection of DOE tools rather than a beginning-to-end DOE application. Here's how to use some of those tools. 6 | 7 | You can generate a full factorial design for four factors each taking two values as follows: 8 | ```matlab 9 | design = fullfact([2 2 2 2]) 10 | ``` 11 | 12 | For historical reasons, this codes the two levels as 1 and 2. The fracfact function codes them as -1 and 1. Here's how to use that function to get the equivalent design and its confounding pattern: 13 | ```matlab 14 | [design,confounding] = fracfact('a b c d') 15 | ``` 16 | 17 | Here's how to generate a 2^(4-1) design: 18 | ```matlab 19 | [design,confounding] = fracfact('a b c abc') 20 | ``` 21 | 22 | Here's how to change the limits in the first column to other limits that you specify (you could repeat for other columns or loop over them): 23 | ```matlab 24 | limits = [5 10]; 25 | design(:,1) = limits(1) + (limits(2)-limits(1))*(1+design(:,1))/2 26 | ``` 27 | 28 | Finally, you may want to randomize the order of runs when you carry out the experiment. Here's how to do that: 29 | ```matlab 30 | order = randperm(8) 31 | design = design(order,:) 32 | ``` 33 | -------------------------------------------------------------------------------- /programming-notes/matlab/figures-publication.md: -------------------------------------------------------------------------------- 1 | # Preparing figures for publication 2 | 3 | Here is a summary of the most important steps and commands necessary to obtain nice figures of your data that can be imported into the text editing program of your choice. 4 | 5 | ## Scaling 6 | 7 | Here is how to set the figure dimensions to (8 x 6) cm: 8 | 9 | ```matlab 10 | fig = figure; 11 | 12 | % create figure here 13 | 14 | fig.Units = 'centimeters'; 15 | fig.Position(3) = 8; 16 | fig.Position(4) = 6; 17 | ``` 18 | 19 | Alternatively, you can do the following and use the `gcf` command: 20 | 21 | ```matlab 22 | set(gcf, 'units', 'centimeters', 'position', [0 0 width height]) 23 | ``` 24 | 25 | The maximum width for one-column and two-column figures, respectively, is usually given by the journal you want to submit to. Or, if you are using a LaTeX, you can output the required widths with the `\the` command: 26 | 27 | ```latex 28 | \the\hsize 29 | ``` 30 | 31 | ## Formatting text 32 | 33 | Here we select the font Times and set the font size to 9. 34 | 35 | ```matlab 36 | set(fig.Children, 'FontName', 'Times', 'FontSize', 9); 37 | ``` 38 | 39 | 40 | ## Remove unnecessary white space 41 | 42 | As the white space surrounding the plot wastes a lot of the precious figure space, especially for small figures, it should be removed (or minimized) in the next step: 43 | 44 | ```matlab 45 | set(gca, 'LooseInset', max(get(gca,'TightInset'), 0.02)) 46 | ``` 47 | 48 | 49 | ## Exporting 50 | 51 | The figure can be exported to the desired graphics format. This can be done with the saveas command, but the print command allows for the definition of more attributes of the exported file. 52 | 53 | First, in order for the exported file to have the same size as the Matlab figure, it's necessary to first set the `PaperPositionMode` to automatic. Then we export the current figure to a png file with 600 dpi resolution: 54 | 55 | ```matlab 56 | fig.PaperPositionMode = 'auto'; 57 | print('img/my_figure', '-dpng', '-r600') 58 | ``` 59 | 60 | If using vector graphics, `-dpng` can be replaced by `-epsc` for colored eps or `-dsvg`. 61 | 62 | 63 | ## Summary 64 | 65 | Here is a simple copy-pastable template: 66 | 67 | ```matlab 68 | filename = 'something'; % output file name 69 | width = 8; % cm 70 | height = 5; % cm 71 | 72 | fig.Units = 'centimeters'; 73 | fig.Position(3) = width; 74 | fig.Position(4) = height; 75 | set(fig, 'LooseInset', max(get(gca,'TightInset'), 0.02)) 76 | set(fig, 'PaperPositionMode','auto', 'PaperUnits','centimeters', 'PaperSize',[fig.Position(3), fig.Position(4)]) 77 | print(filename, '-dpng', '-r600') % save as png using 600 dpi resolution 78 | print(filename, '-dtiff', '-r600') % save as svg using 600 dpi resolution 79 | print(filename, '-dpdf', '-r600') % save as pdf using 600 dpi resolution 80 | print(filename, '-dsvg', '-r600') % save as svg using 600 dpi resolution 81 | ``` 82 | -------------------------------------------------------------------------------- /programming-notes/matlab/integrate.md: -------------------------------------------------------------------------------- 1 | # Integrating from data points 2 | 3 | Consider the following data 4 | 5 | ```matlab 6 | t = [ ... ]; 7 | y = [ ... ]; 8 | ``` 9 | 10 | ## Matlab trapz function 11 | Calculate the area (integral) directly from the datapoints. 12 | 13 | ```matlab 14 | int_trapz = trapz(t,y) 15 | ``` 16 | 17 | ## Fiting a curve to data and then integrating 18 | 19 | ```matlab 20 | [fitobject,gof,output] = fit(t,y,'cubicspline'); 21 | ``` 22 | 23 | ### Calculate with Matlab integrate function from the fitted curve 24 | 25 | ```matlab 26 | int_fit_integrate = integrate(fitobject, t(end), 0) 27 | ``` 28 | 29 | ### Calculate with simpson38 function from the fitted curve 30 | 31 | ```matlab 32 | int_fit_simp38 = simpson38(fitobject, 0, t(end), 3*100) 33 | ``` 34 | 35 | 36 | ## Simpson's 3/8 rule 37 | 38 | ```matlab 39 | %% 40 | function integral = simpson38(f , a , b , n) 41 | %{ 42 | description: 43 | this function evaluates the integral of a mathematical function f 44 | between the limits a and b using the simpson's 1/3 formula given by, 45 | I = 3 * h / 8 * (f(a) + f(b) + 3 * f(x_interiors)) 46 | 47 | inputs: 48 | 1. f = the function which to be integrated. 49 | 2. a = lower limit of the integral. 50 | 3. b = upper limit of the integral. 51 | 4. n = the no of intervals to which the domain is to be split. note 52 | that n is always even for this case. 53 | note: n is an optional argument. if n is not specified, the function 54 | splits the interval into 60 pieces. 55 | outputs: 56 | 1. integral = this will give the value of the integral of the 57 | mathematical function f between the limits a and b. 58 | %} 59 | %adding robustness - argumental pre-check 60 | if nargin < 3 61 | error('not enough input arguments. minimum 3 arguments needed.') 62 | end 63 | 64 | if a > b 65 | error("lowerlimit can't be greater than upperlimit.") 66 | end 67 | 68 | if mod(n , 3) ~= 0 69 | error("n must be a multiple of 3.") 70 | end 71 | 72 | if nargin < 4 73 | n = 60; 74 | end 75 | 76 | %calculate h value 77 | h = (b - a) / n; 78 | 79 | %evaluate the function values 80 | x = linspace(a , b , n + 1); 81 | for i = 1 : n + 1 82 | fofx(i) = f(x(i)); 83 | end 84 | 85 | %split up the points 86 | g = fofx; 87 | g(1) = []; 88 | g(end) = []; 89 | inot3 = 0; %not a 3 multiple 90 | i3 = 0; %is a 3 multiple 91 | 92 | global g_3; 93 | for j = 1 : length(g) 94 | if mod(j , 3) == 0 95 | i3 = i3 + 1; 96 | g_3(i3) = g(j); 97 | else 98 | inot3 = inot3 + 1; 99 | g_not3(inot3) = g(j); 100 | end 101 | end 102 | 103 | if isempty(g_3) 104 | g_3 = 0; 105 | end 106 | 107 | %evaluate the integral 108 | integral = 3 * h / 8 * (fofx(1) + fofx(end) + 3 * sum(g_not3) + 2 * sum(g_3)); 109 | 110 | end 111 | ``` 112 | 113 | 114 | ## Composite Simpson's 3/8 rule 115 | 116 | ```matlab 117 | function int = simpson38comp(f, a, b, n) 118 | 119 | if nargin < 3 120 | error('Not enough input arguments.') 121 | end 122 | 123 | if a > b 124 | error("Lower limit can't be greater than upper limit.") 125 | end 126 | 127 | if mod(n , 3) ~= 0 128 | error("n must be a multiple of 3.") 129 | end 130 | 131 | h=(b-a)/n; 132 | 133 | so = 0; 134 | sm3 = 0; 135 | 136 | % (3h/8)*[(y0+yn)+2*(y3+y6+..+yn-3)+3*(y1+y2+y4+y5+...+yn-2+yn-1)] 137 | for k = 1:1:n-1 138 | x(k) = a+k*h; 139 | y(k) = f(x(k)); 140 | 141 | if rem(k,3) == 0 142 | sm3 = sm3+y(k); %sum of multiple of 3 terms 143 | else 144 | so = so+y(k); %sum of others terms 145 | end 146 | end 147 | 148 | int = (3*h/8)*(f(a)+f(b)+3*so+2*sm3); 149 | ``` 150 | -------------------------------------------------------------------------------- /programming-notes/matlab/links.md: -------------------------------------------------------------------------------- 1 | # Links and Resources 2 | 3 | - [MATLAB Style Guide](https://sites.google.com/site/matlabstyleguidelines/home) 4 | -------------------------------------------------------------------------------- /programming-notes/matlab/strings.md: -------------------------------------------------------------------------------- 1 | # Working with strings 2 | 3 | ## `fprintf` over multiple lines 4 | 5 | ```matlab 6 | fprintf( ['The results of test %d are such that %d of the ', ... 7 | 'cats are older than %d years old.\nThe results of test ', ... 8 | '%d are such that %d of the dogs are older than %d years ', ... 9 | 'old.\nThe results of test %d are such that %d of the ', ... 10 | 'fish are older than %d years old.\n'], t1, cats, age, ... 11 | t2, dogs, age, t3, fish, age); 12 | ``` 13 | -------------------------------------------------------------------------------- /programming-notes/matlab/surface-fit.md: -------------------------------------------------------------------------------- 1 | # Surface fitting with MATLAB 2 | 3 | ## `polyfitn` 4 | 5 | [polyfitn at File Exchange](https://www.mathworks.com/matlabcentral/fileexchange/34765-polyfitn) 6 | 7 | Polynomial modeling in 1 or n dimensions 8 | 9 | 10 | ## `gridfit` 11 | 12 | [gridfit at File Exchange](https://www.mathworks.com/matlabcentral/fileexchange/8998-surface-fitting-using-gridfit) 13 | 14 | Model 2-d surfaces from scattered data. 15 | -------------------------------------------------------------------------------- /programming-notes/python/.nav.yml: -------------------------------------------------------------------------------- 1 | # nav: 2 | # - "*/" 3 | # - "*.md" 4 | 5 | append_unmatched: true 6 | 7 | sort: 8 | sections: first -------------------------------------------------------------------------------- /programming-notes/python/basemap.md: -------------------------------------------------------------------------------- 1 | # Geo ploting with Basemap 2 | 3 | The matplotlib basemap toolkit is a library for plotting 2D data on maps in Python. 4 | 5 | ## Installation 6 | 7 | #### Option 1: 8 | 9 | The recommended installation method for Basemap is using Anaconda and the conda-forge channel. In the Anaconda Prompt run: 10 | ``` 11 | $ conda install -c anaconda basemap 12 | ``` 13 | 14 | You might also need to run the following command to install [PROJ](https://proj.org/install.html), which is a required dependency of Basemap: 15 | ``` 16 | $ conda install -c conda-forge proj 17 | ``` 18 | 19 | If the installation was sucessful you should now be able to run the following import in the Python (Anaconda) prompt without any errors: 20 | ```python 21 | from mpl_toolkits.basemap import Basemap 22 | ``` 23 | 24 | #### Option 2: 25 | 26 | If you are on Windows you can also install the binaries directly. This worked better for me than installing through Anaconda and conda-forge. Download the [Basemap](https://www.lfd.uci.edu/~gohlke/pythonlibs/#basemap) and [PROJ](https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyproj) binaries. Make sure you download the correct binary for your Python version. For instance, if you have Python 3.7 64-bit make sure to download the `pyproj‑2.4.2.post1‑cp37‑cp37m‑win_amd64.whl` and `basemap‑1.2.1‑cp37‑cp37m‑win_amd64.whl` files. The `cp37` indicates Python version and `amd64` the 64-bit version. You can now install both libraries with pip. 27 | 28 | ``` 29 | $ pip install pyproj‑2.4.2.post1‑cp37‑cp37m‑win_amd64.whl 30 | $ pip install basemap‑1.2.1‑cp37‑cp37m‑win_amd64.whl 31 | ``` 32 | 33 | Test the instalation by running: 34 | 35 | ```python 36 | from mpl_toolkits.basemap import Basemap 37 | ``` 38 | 39 | 40 | ## Ploting a simple map 41 | 42 | Let's plot a simple world map. 43 | 44 | ```python 45 | from mpl_toolkits.basemap import Basemap 46 | import matplotlib.pyplot as plt 47 | import numpy as np 48 | 49 | m = Basemap(projection='mill',llcrnrlat=-90, urcrnrlat=90, llcrnrlon=-180, urcrnrlon=180, resolution='c') 50 | 51 | # Draw map features 52 | m.drawcoastlines() 53 | m.fillcontinents(color='#072B57',lake_color='#FFFFFF') 54 | 55 | # Draw parallels and meridians 56 | m.drawparallels(np.arange(-90.,91.,30.)) 57 | m.drawmeridians(np.arange(-180.,181.,60.)) 58 | 59 | m.drawmapboundary(fill_color='#FFFFFF') 60 | plt.title("Basemap Example!") 61 | plt.show() 62 | ``` 63 | 64 | `projection='mill'` is the map projection. There are multiple projections availlable. Check [here](https://matplotlib.org/basemap/users/mapsetup.html). 65 | 66 | `llcrnrlat`, `llcrnrlon`, `urcrnrlat`, and `urcrnrlon` are the latitude and longitude values of the lower left and upper right corners of the map. 67 | 68 | There are three resolution levels: `resolution='c'` (used above) is crude resolution but faster render time. There is also `resolution='l'` for low resolution and `resolution='h'` for high resolution, which represent increasing map resolution and increased render time. Unless coastlines or lakes are really important to you crude resolution is usually enough. 69 | 70 | 71 | ## Drawing other map features: 72 | 73 | ```python 74 | m.drawcountries() # draw countries 75 | m.drawstates() # draw states 76 | m.drawrivers() # draw rivers 77 | m.bluemarble() # satellite style map 78 | ``` 79 | 80 | 81 | ## Ploting coordinates 82 | 83 | ```python 84 | from mpl_toolkits.basemap import Basemap 85 | import matplotlib.pyplot as plt 86 | 87 | m = Basemap(projection='mill',llcrnrlat=-90, urcrnrlat=90, llcrnrlon=-180, urcrnrlon=180, resolution='c') 88 | m.drawcoastlines() 89 | m.fillcontinents(color='#072B57',lake_color='#FFFFFF') 90 | m.drawmapboundary(fill_color='#FFFFFF') 91 | 92 | lat, lon = 29.7630, -95.3630 # define the coordinates 93 | x,y = m(lon, lat) # convert to Basemap system using your map 94 | m.plot(x, y, 'ro', markersize=20, alpha=.5) # plot and specify marker size and marker fill transparency 95 | 96 | plt.title("Basemap Example!") 97 | plt.show() 98 | ``` 99 | 100 | 101 | 102 | 103 | ## References: 104 | - [Basemap installation via Anaconda](https://anaconda.org/anaconda/basemap) 105 | - [PROJ installation instructions](https://proj.org/install.html) 106 | - [pythonprogramming.net](https://pythonprogramming.net/geographical-plotting-basemap-tutorial/) 107 | -------------------------------------------------------------------------------- /programming-notes/python/basics/abstract-classes.md: -------------------------------------------------------------------------------- 1 | # Abstract classes 2 | 3 | Abstract class is an extension of a basic class. Like a basic class, an abstract class has methods and state. Unlike a basic class, it inherits the `ABC` class and has at least one `abstractmethod`. That means we cannot create an instance directly from its constructor. We will create an abstract class and two concrete classes. 4 | 5 | ## Creating the classes 6 | 7 | First let's create an abstract definition of an employee. Any employee can work and relax. The way that one type of employee can work and relax is different from another type of employee. 8 | 9 | ```python 10 | from abc import ABC, abstractmethod 11 | 12 | class Employee(ABC): 13 | def __init__(self, name, title): 14 | self.name = name 15 | self.title = title 16 | 17 | def __str__(self): 18 | return self.name 19 | 20 | @abstractmethod 21 | def do_work(self): 22 | """Do something for work.""" 23 | raise NotImplementedError 24 | 25 | @abstractmethod 26 | def do_relax(self): 27 | """Do something to relax.""" 28 | raise NotImplementedError 29 | ``` 30 | 31 | Then we can create a concrete definition of an engineer. The Engineer class is concrete because it implements every `abstractmethod` that was not implemented above. Notice that we leverage the parent's constructor when creating this object. We also define `do_refactor` for an engineer, which is something that a manager prefers not to do. 32 | 33 | ```python 34 | class Engineer(Employee): 35 | def __init__(self, name, title, skill): 36 | super().__init__(name, title) 37 | self.skill = skill 38 | 39 | def do_work(self): 40 | return f"{self} is coding in {self.skill}" 41 | 42 | def do_relax(self): 43 | return f"{self} is watching YouTube" 44 | 45 | def do_refactor(self): 46 | """Do the hard work of refactoring code, unlike managers.""" 47 | return f"{self} is refactoring code" 48 | ``` 49 | 50 | Now we create a oncrete definition of a manager. The Manager class is concrete for the same reasons as the Engineer class is concrete. Notice that a manager has direct reports and has the responsibility of hiring people on the team, unlike an engineer. 51 | 52 | ```python 53 | class Manager(Employee): 54 | def __init__(self, name, title, direct_reports): 55 | super().__init__(name, title) 56 | self.direct_reports = direct_reports 57 | 58 | def do_work(self): 59 | return f"{self} is meeting up with {len(self.direct_reports)} reports" 60 | 61 | def do_relax(self): 62 | return f"{self} is taking a trip to the Bahamas" 63 | 64 | def do_hire(self): 65 | """Do the hard work of hiring employees, unlike engineers.""" 66 | return f"{self} is hiring employees" 67 | ``` 68 | 69 | ## Using the classes 70 | 71 | Declare two engineers: 72 | 73 | ```python 74 | engineer_john = Engineer("John Doe", "Software Engineer", "Android") 75 | engineer_jane = Engineer("Jane Doe", "Software Engineer", "iOS") 76 | ``` 77 | 78 | These engineers are employees but not managers: 79 | ```python 80 | assert all(isinstance(engineer, Employee) for engineer in engineers) 81 | assert all(not isinstance(engineer, Manager) for engineer in engineers) 82 | ``` 83 | 84 | Engineers can work, relax and refactor: 85 | ```python 86 | assert engineer_john.do_work() == "John Doe is coding in Android" 87 | assert engineer_john.do_relax() == "John Doe is watching YouTube" 88 | assert engineer_john.do_refactor() == "John Doe is refactoring code" 89 | ``` 90 | 91 | Declare manager with engineers as direct reports: 92 | ```python 93 | manager_max = Manager("Max Doe", "Engineering Manager", engineers) 94 | ``` 95 | 96 | Managers are employees but not engineers: 97 | ```python 98 | assert isinstance(manager_max, Employee) 99 | assert not isinstance(manager_max, Engineer) 100 | ``` 101 | 102 | Managers can work, relax and hire: 103 | ```python 104 | assert manager_max.do_work() == "Max Doe is meeting up with 2 reports" 105 | assert manager_max.do_relax() == "Max Doe is taking a trip to the Bahamas" 106 | assert manager_max.do_hire() == "Max Doe is hiring employees" 107 | ``` 108 | -------------------------------------------------------------------------------- /programming-notes/python/basics/args-kwargs.md: -------------------------------------------------------------------------------- 1 | # *args and **kwargs 2 | 3 | ### A normal function: 4 | 5 | ```python 6 | def func1(one, two) 7 | print(one) 8 | print(two) 9 | 10 | func1('arg one', 'arg two') # Correct 11 | func1('arg one') # Error 12 | func1('arg one', 'arg two', 'arg three') # Error 13 | ``` 14 | 15 | ### *args usage: 16 | 17 | ```python 18 | def func2(*args) 19 | for stuff in args: 20 | print(stuff) 21 | 22 | my_list = ['green', 'yellow', 'blue', 'red'] 23 | 24 | func2(*my_list) # Correct 25 | func2('green', 'yellow', 'blue', 'red') # Correct 26 | ``` 27 | 28 | 29 | ```python 30 | def func3(one, two, *args) 31 | print(one) 32 | print(two) 33 | for stuff in args: 34 | print(stuff) 35 | 36 | my_list = ['green', 'yellow', 'blue', 'red'] 37 | 38 | func3('required one', 'required two', *my_list) # Correct 39 | ``` 40 | 41 | 42 | ### **kwargs usage: 43 | 44 | ```python 45 | def func4(**kwargs) 46 | for key, value in kwargs.items(): 47 | print(key) 48 | print(value) 49 | 50 | my_dict = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'} 51 | 52 | func4(**kwargs) # Correct 53 | func4(key1 = 'value1', key2 = 'value2', key3 = 'value3') # Correct 54 | ``` 55 | -------------------------------------------------------------------------------- /programming-notes/python/basics/builtin-functions.md: -------------------------------------------------------------------------------- 1 | # Sorting lists 2 | 3 | `sorted()` returns a **new** sorted list, leaving the original list unaffected. `list.sort()` sorts the list **in-place**, mutating the list indices, and returns None (like all in-place operations). 4 | 5 | `sorted()` works on any iterable, not just lists. Strings, tuples, dictionaries (you'll get the keys), generators, etc., returning a list containing all elements, sorted. 6 | 7 | * Use `list.sort()` when you want to mutate the list, sorted() when you want a new sorted object back. Use `sorted()` when you want to sort something that is an iterable, not a list yet. 8 | * For lists, `list.sort()` is faster than `sorted()` because it doesn't have to create a copy. For any other iterable, you have no choice. 9 | * No, you cannot retrieve the original positions. Once you called `list.sort()` the original order is gone. 10 | 11 | #### Useful links: 12 | https://stackoverflow.com/questions/22442378/what-is-the-difference-between-sortedlist-vs-list-sort-python 13 | https://stackoverflow.com/questions/18761776/sort-list-of-dictionaries-by-multiple-keys-with-different-ordering 14 | https://stackoverflow.com/questions/16082954/python-how-to-sort-a-list-of-dictionaries-by-several-values 15 | -------------------------------------------------------------------------------- /programming-notes/python/basics/classes.md: -------------------------------------------------------------------------------- 1 | # Classes and OOP (Object Oriented Programming) 2 | 3 | 4 | ```python 5 | class Foo: 6 | a = 0 # <-- class variable 7 | 8 | def __init__(self, b=0): 9 | self.b = b # <-- instance variable 10 | 11 | def bar(self): # <- instance method 12 | return self.a + self.b 13 | 14 | @classmethod 15 | def foo(cls, c=12): # <-- class method 16 | return cls(c).bar() 17 | 18 | def foobar(): # <-- function 19 | print(Foo.foo()) 20 | ``` 21 | 22 | A function and a method are not the same thing, and a class and an instance are not the same thing, which is why a class variable/method and an instance variable/method are very definitely not the same things. 23 | 24 | A class is like a blueprint, it tells you about some thing you want to make. An instance is the thing that gets made. 25 | 26 | For example, if you write up a blueprint for an airplane, the blueprint is like when you define a class. The airplanes that get made from that blueprint are like instances of a class. 27 | 28 | Defining a class looks like this: 29 | 30 | ```python 31 | class Airplane: 32 | pass 33 | ``` 34 | 35 | (Normally you would have some more code instead of `pass`. I'll explain that later.) 36 | 37 | Now once you define a class you can create instances of a class like this, `Airplane()`. For example, 38 | 39 | ```python 40 | airplane1 = Airplane() 41 | airplane2 = Airplane() 42 | ``` 43 | 44 | Here we created two instances of the Airplane class and put them in the variables `airplane1` and `airplane2`. The important thing here is that you can change `airplane1` without affecting `airplane2`. They're two separate instances. 45 | 46 | Okay now let's go back and talk about what goes inside a class. Let's take our Airplane class and fill it out: 47 | ```python 48 | class Airplane: 49 | def __init__(self): 50 | print "A new instance got made!" 51 | ``` 52 | So what's going on here? `__init__` is a function that gets run when you create an instance. That's it! So if you go back to where we created the two Airplane instances, 53 | ```python 54 | airplane1 = Airplane() 55 | airplane2 = Airplane() 56 | ``` 57 | 58 | what would happen is, "A new instance got made!" would be printed out twice. 59 | 60 | What about the `self` parameter? I think this would be easier to understand if we added a new method. 61 | 62 | ```python 63 | class Airplane: 64 | def __init__(self): 65 | print "A new instance got made!" 66 | def fly(self): 67 | print "I'm flying!" 68 | ``` 69 | 70 | So if you wanted to call this method you'd do something like this: `airplane1.fly()`. Actually this is the same thing as this: `Airplane.fly(airplane1)`. Both of these would do the same thing, i.e. print out "I'm flying!". So `airplane1` is the instance that we used to call our `fly` method. This instance is what gets passed to `self`. 71 | 72 | 73 | ### Classes 74 | 75 | ```python 76 | # Function, this lives outside a class 77 | def add(a, b): 78 | return a + b 79 | 80 | 81 | class Adder(): 82 | 83 | # __init__ roughly equivilant to a constructor in other languages 84 | def __init__(self, a, b): 85 | # Adding attributes to this instance of our class 86 | self.a = a 87 | self.b = b 88 | 89 | # Method, This belongs to an instance of a class and must have self as first argument. self refers to an instance of a class 90 | def add(self): 91 | return self.a + self.b 92 | 93 | # CLass Method, belongs to a class and is shared by every instance of the class, must have the class as first argument 94 | @classmethod 95 | def class_add(cls, x, y): 96 | return x + y 97 | 98 | # Static Method, Only belongs to the class for organisation, can't reference class or instance attributes 99 | @staticmethod 100 | def static_add(c, d): 101 | return c + d 102 | 103 | 104 | # Let's start with an instance initialized with values for a and b 105 | foo = Adder(1, 2) 106 | # we call the add method of foo which refernces the values we initialized the class with 107 | print ('Instance result', foo.add()) 108 | 109 | # Class method doesnt require an instance, note no () after Adder, we arent creating an instance, just referencing the class 110 | bar = Adder.class_add(1, 2) 111 | print ('Class result', bar) 112 | 113 | # Static, similar to classmethod we don't need to create an instance 114 | baz = Adder.static_add(1, 2) 115 | print ('Static result', baz) 116 | 117 | # Finally we'll just use our add function 118 | spam = add(1, 2) 119 | print ('Function result', spam) 120 | ``` 121 | 122 | 123 | ### References: 124 | * [https://www.reddit.com/r/learnpython/comments/1cpu7x/explain_classes_init_and_self_like_im_five/](https://www.reddit.com/r/learnpython/comments/1cpu7x/explain_classes_init_and_self_like_im_five/) 125 | -------------------------------------------------------------------------------- /programming-notes/python/basics/custom-exceptions.md: -------------------------------------------------------------------------------- 1 | # Custom exceptions 2 | 3 | Exceptions are a way to deal with errors in your code. 4 | 5 | For example imagine that you have a function that saves some data to the disk. And when the program executes, the disk happens to be full. In that case, an exception is "raised" (or "thrown"). Imagine it like an alternative result-value from your function. 6 | 7 | In areas where the function is called, you can "catch" those exceptions and do something sensible. For example, in the case of a disk full, show an error to the user that says: "Error: Disk is full" or something similar. 8 | 9 | **Any exception that is not caught will cause the program to crash and exit**. 10 | 11 | So clearly, handling (catching) them and doing something other than crashing is nicer for the end-user. 12 | 13 | Python comes with [some standard/builtin exceptions](https://docs.python.org/3/library/exceptions.html#exception-hierarchy). Each exception has a well defined reason why it might be thrown. 14 | 15 | If those "builtin" exceptions are not appropriate for your own program, you can define your own. This is done by creating a new class inheriting from "Exception". 16 | 17 | For example, let's say your program that creates files on disk only gives a user the permission to store 1GB of data. Everything above "exceeds the quota". The disk is still not full, so that error is not appropriate. So we need something else. 18 | 19 | In that case we can create such a subclass: 20 | 21 | ```python 22 | class QuotaExceeded(Exception): 23 | pass 24 | ``` 25 | 26 | And then use it: 27 | 28 | ```python 29 | def store_file(user, file): 30 | quota = get_quota(user) 31 | used_diskspace = get_used_diskspace(user) 32 | filesieze = get_filesize(file) 33 | if (used_diskspace + filesize) > quota: 34 | raise QuotaExceeed( 35 | f"The file {file} would exceed the disk quota for {user}. " 36 | f"Currently {used_diskspace} out of {quota} is used. " 37 | f"Filesize: {filesize}" 38 | ) 39 | store_file_to_disk(user, file) 40 | ``` 41 | 42 | The `raise` line in the if-block will immediately stop executing that function and "throw" that custom `QuotaExceeded` error. It can then be caught using the `try/except` block: 43 | 44 | ```python 45 | try: 46 | store_file(john_doe, uploaded_file) 47 | except QuotaExceeded as exc: 48 | print(exc) 49 | ``` 50 | 51 | If that `try/except` block would not exist, the program would crash. 52 | 53 | Sometimes you want your program to crash out instead of being in an "unknown/unpredictable" state. That's when exceptions come in handy. 54 | 55 | To add to the example above, you can make custom exceptions more useful by adding a docstring and by storing useful context-values in the exception itself. For example: 56 | 57 | ```python 58 | class QuotaExceeded(Exception): 59 | """ 60 | This exception is raised if an operation would cause more disk-space 61 | to be used than is allowed for a given user. 62 | """ 63 | def __init__(self, file, user, used_diskspace, quota, filesize): 64 | super().__init__( 65 | f"The file {file} would exceed the disk quota for {user}. " 66 | f"Currently {used_diskspace} out of {quota} is used. " 67 | f"Filesize: {filesize}" 68 | ) 69 | self.file = file 70 | self.user = user 71 | self.used_diskspace = used_diskspace 72 | self.quota = quota 73 | self.filesize = filesize 74 | 75 | def store_file(user, file): 76 | quota = get_quota(user) 77 | used_diskspace = get_used_diskspace(user) 78 | filesieze = get_filesize(file) 79 | if (used_diskspace + filesize) > quota: 80 | raise QuotaExceeed(file, user, used_diskspace, quota, filesize) 81 | store_file_to_disk(user, file) 82 | ``` 83 | 84 | And finally, exceptions are one of those topics where people like to argue about whether or not they are good. There are valid arguments for both ways. To avoid unnecessary digression and complexity, all I will say is that you could rewrite the code above with `if` statements to something like this: 85 | 86 | ```python 87 | def store_file(user, file): 88 | quota = get_quota(user) 89 | used_diskspace = get_used_diskspace(user) 90 | filesieze = get_filesize(file) 91 | if (used_diskspace + filesize) > quota: 92 | return ( 93 | f"The file {file} would exceed the disk quota for {user}. " 94 | f"Currently {used_diskspace} out of {quota} is used. " 95 | f"Filesize: {filesize}" 96 | ) 97 | store_file_to_disk(user, file) 98 | return "" 99 | 100 | error_message = store_file(john_doe, uploaded_file) 101 | if error_message: 102 | print(error_message) 103 | ``` 104 | -------------------------------------------------------------------------------- /programming-notes/python/basics/data-structures.md: -------------------------------------------------------------------------------- 1 | # Data structures 2 | Python offers several data structures such as: 3 | - Strings 4 | - Lists 5 | - Tuples 6 | - Dictionaries 7 | - Sets 8 | - Deques 9 | 10 | We'll cover some of these below. 11 | 12 | 13 | ## Tuples 14 | Tuples are an ordered collection of values that cannot be modified at runtime. This module shows how tuples are created, iterated, accessed and combined. 15 | 16 | This is a tuple of integers: 17 | ```python 18 | immutable = (1, 2, 3, 4) 19 | ``` 20 | 21 | It can be indexed like a list: 22 | ```python 23 | assert immutable[0] == 1 24 | assert immutable[-1] == 4 25 | ``` 26 | 27 | It can be sliced like a list: 28 | ```python 29 | assert immutable[1:3] == (2, 3) 30 | assert immutable[3:4] == (4,) 31 | assert immutable[1::2] == (2, 4) 32 | assert immutable[::-1] == (4, 3, 2, 1) 33 | ``` 34 | 35 | It can be iterated over like a list: 36 | ```python 37 | for ix, number in enumerate(immutable): 38 | assert immutable[ix] == number 39 | ``` 40 | 41 | But its contents cannot be changed. As an alternative, we can create new tuples from existing tuples 42 | ```python 43 | bigger_immutable = immutable + (5, 6) 44 | assert bigger_immutable == (1, 2, 3, 4, 5, 6) 45 | smaller_immutable = immutable[0:2] 46 | assert smaller_immutable == (1, 2) 47 | ``` 48 | 49 | We use tuples when the number of items is consistent. An example where this can help is a 2D game with X and Y coordinates. Using a tuple with two numbers can ensure that the number of coordinates doesn't change to one, three, four, etc. 50 | ```python 51 | moved_count = 0 52 | pos_x, pos_y = (0, 0) 53 | for i in range(1, 5, 2): 54 | moved_count += 1 55 | pos_x, pos_y = (pos_x + 10 * i, pos_y + 15 * i) 56 | assert moved_count == 2 57 | assert pos_x == 40 and pos_y == 60 58 | ``` 59 | 60 | ## Sets 61 | Sets are an unordered collection of unique values that can be modified at 62 | runtime. This module shows how sets are created, iterated, accessed, 63 | extended and shortened. 64 | 65 | Let's define one `set` for starters: 66 | ```python 67 | simple_set = {0, 1, 2} 68 | ``` 69 | 70 | A set is dynamic like a `list` and `tuple`: 71 | ```python 72 | simple_set.add(3) 73 | simple_set.remove(0) 74 | assert simple_set == {1, 2, 3} 75 | ``` 76 | 77 | Unlike a `list` and `tuple`, it is not an ordered sequence as it does not allow duplicates to be added: 78 | ```python 79 | for _ in range(5): 80 | simple_set.add(0) 81 | simple_set.add(4) 82 | assert simple_set == {0, 1, 2, 3, 4} 83 | ``` 84 | 85 | Now let's define two new `set` collections: 86 | ```python 87 | multiples_two = set() 88 | multiples_four = set() 89 | ``` 90 | 91 | Fill sensible values into the set using `add`: 92 | ```python 93 | for i in range(10): 94 | multiples_two.add(i * 2) 95 | multiples_four.add(i * 4) 96 | ``` 97 | 98 | As we can see, both sets have similarities and differences: 99 | ```python 100 | assert multiples_two == {0, 2, 4, 6, 8, 10, 12, 14, 16, 18} 101 | assert multiples_four == {0, 4, 8, 12, 16, 20, 24, 28, 32, 36} 102 | ``` 103 | 104 | We cannot decide in which order the numbers come out - so let's look for fundamental truths instead, such as divisibility against 2 and 4. We do this by checking whether the modulus of 2 and 4 yields 0 (i.e. no remainder from performing a division): 105 | ```python 106 | multiples_common = multiples_two.intersection(multiples_four) 107 | for number in multiples_common: 108 | assert number % 2 == 0 and number % 4 == 0 109 | ``` 110 | 111 | We can compute exclusive multiples: 112 | ```python 113 | multiples_two_exclusive = multiples_two.difference(multiples_four) 114 | multiples_four_exclusive = multiples_four.difference(multiples_two) 115 | assert len(multiples_two_exclusive) > 0 116 | assert len(multiples_four_exclusive) > 0 117 | ``` 118 | 119 | Numbers in this bracket are greater than 2 * 9 and less than 4 * 10: 120 | ```python 121 | for number in multiples_four_exclusive: 122 | assert 18 < number < 40 123 | ``` 124 | 125 | By computing a set union against the two sets, we have all integers in this program: 126 | ```python 127 | multiples_all = multiples_two.union(multiples_four) 128 | ``` 129 | 130 | Check if set A is a subset of set B: 131 | ```python 132 | assert multiples_four_exclusive.issubset(multiples_four) 133 | assert multiples_four.issubset(multiples_all) 134 | ``` 135 | 136 | Check if set A is a subset and superset of itself: 137 | ```python 138 | assert multiples_all.issubset(multiples_all) 139 | assert multiples_all.issuperset(multiples_all) 140 | ``` 141 | 142 | Check if set A is a superset of set B: 143 | ```python 144 | assert multiples_all.issuperset(multiples_two) 145 | assert multiples_two.issuperset(multiples_two_exclusive) 146 | ``` 147 | 148 | 149 | ## References 150 | - [Ultimate Python - Tuples](https://github.com/huangsam/ultimate-python/blob/master/ultimatepython/data_structures/tuple.py) 151 | - [Ultimate Python - Sets](https://github.com/huangsam/ultimate-python/blob/master/ultimatepython/data_structures/set.py) 152 | -------------------------------------------------------------------------------- /programming-notes/python/basics/dataclasses.md: -------------------------------------------------------------------------------- 1 | # Dataclasses 2 | 3 | Since version 3.7, Python offers data classes. There are several advantages over regular classes or other alternatives like returning multiple values or dictionaries: 4 | - a data class requires a minimal amount of code 5 | - you can compare data classes because `__eq__` is implemented for you 6 | - you can easily print a data class for debugging because `__repr__` is implemented as well 7 | - data classes require type hints, reduced the chances of bugs 8 | 9 | Example: 10 | 11 | ```python 12 | from dataclasses import dataclass 13 | 14 | @dataclass 15 | class Card: 16 | rank: str 17 | suit: str 18 | 19 | card = Card("Q", "hearts") 20 | 21 | print(card == card) 22 | # True 23 | 24 | print(card.rank) 25 | # 'Q' 26 | 27 | print(card) 28 | Card(rank='Q', suit='hearts') 29 | ``` 30 | -------------------------------------------------------------------------------- /programming-notes/python/basics/decorators.md: -------------------------------------------------------------------------------- 1 | # Decorators 2 | 3 | Decorators are wrappers around a function that modify the behavior of the function in a certain way. Let’s create our own decorator: 4 | 5 | ```python 6 | def print_argument(func): 7 | def wrapper(the_number): 8 | print("Argument for", 9 | func.__name__, 10 | "is", the_number) 11 | return func(the_number) 12 | return wrapper 13 | @print_argument 14 | def add_one(x): 15 | return x + 1 16 | print(add_one(1)) 17 | ``` 18 | 19 | Inside `print_argument`, we define a wrapper function. This function prints the argument and the name of the called function. Next, it executes the actual function and returns its result as if the function was called regularly. With `@print_argument` we apply our decorator to a function. The output of our little script will be: 20 | 21 | ```python 22 | # Argument for add_one is 1 23 | # 2 24 | ``` 25 | -------------------------------------------------------------------------------- /programming-notes/python/basics/lambda-functions.md: -------------------------------------------------------------------------------- 1 | # Annonymous functions (Lambda functions) 2 | 3 | Sometimes, naming a function is not worth the trouble. For example when you’re sure the function will only be used once. For such cases, Python offers us anonymous functions, also called lambda functions. 4 | 5 | A lambda function can be assigned to a variable, creating a concise way of defining a function: 6 | 7 | ```python 8 | add_one = lambda x: x + 1 9 | add_one(3) 10 | # 4 11 | ``` 12 | 13 | It gets more interesting when you need to use a function as an argument. In such cases, the function is often used only once. As you may know, `map` applies a function to all elements of an iterable object. We can use a lambda when calling map: 14 | 15 | ```python 16 | numbers = [1, 2, 3, 4] 17 | times_two = map(lambda x: x * 2, numbers) 18 | list(times_two) 19 | # [2, 4, 6, 8] 20 | ``` 21 | 22 | In fact, this is a pattern that you’ll see often. When you need to apply a relatively simple operation on each element of an iterable object, using `map()` in combination with a lambda function is concise and efficient. 23 | -------------------------------------------------------------------------------- /programming-notes/python/basics/packages.md: -------------------------------------------------------------------------------- 1 | # Python packages and modules 2 | 3 | ## Modules 4 | 5 | A python “module” is a single namespace, with a collection of values: 6 | 7 | - functions 8 | - constants 9 | - class definitions 10 | - etc. 11 | 12 | A module usually corresponds to a single file: `something.py` 13 | 14 | ## Packages 15 | 16 | A package is essentially a module, except it can have other modules (and indeed other packages) inside it. 17 | 18 | A package usually corresponds to a directory with a file in it called `__init__.py` and any number of python files or other package directories: 19 | 20 | ``` 21 | a_package 22 | __init__.py 23 | module_a.py 24 | a_sub_package 25 | __init__.py 26 | module_b.py 27 | ``` 28 | 29 | The `__init__.py` can be totally empty or it can have python code in it. The code will be run when the package is imported. Modules inside packages are not automatically imported. So, with the above structure: 30 | 31 | ``` 32 | import a_package 33 | ``` 34 | 35 | will run the code in `a_package/__init__.py`. 36 | 37 | Any names defined in the `__init__.py` will be available in: 38 | 39 | ``` 40 | a_package.a_name 41 | ``` 42 | 43 | but: 44 | 45 | ``` 46 | a_package.module_a 47 | ``` 48 | 49 | will not exist. To get submodules, you need to explicitly import them: 50 | 51 | ``` 52 | import a_package.module_a 53 | ``` 54 | 55 | ## Python packaging tools - `setuptools` 56 | 57 | ### The `setup.py` file 58 | 59 | An example setup.py: 60 | 61 | ``` 62 | from setuptools import setup 63 | 64 | setup( 65 | name='PackageName', 66 | version='0.1.0', 67 | author='An Awesome Coder', 68 | author_email='aac@example.com', 69 | packages=['package_name', 'package_name.test'], 70 | scripts=['bin/script1','bin/script2'], 71 | url='http://pypi.python.org/pypi/PackageName/', 72 | license='LICENSE.txt', 73 | description='An awesome package that does something', 74 | long_description=open('README.txt').read(), 75 | install_requires=[ 76 | "Django >= 1.1.1", 77 | "pytest", 78 | ], 79 | ) 80 | ``` 81 | 82 | ### Running `setup.py` 83 | 84 | With a `setup.py` script defined, setuptools can do a lot: 85 | 86 | Build a source distribution (a tar archive of all the files needed to build and install the package): 87 | 88 | ``` 89 | python setup.py sdist 90 | ``` 91 | 92 | Build from source: 93 | 94 | ``` 95 | python setup.py build 96 | ``` 97 | 98 | And install: 99 | 100 | ``` 101 | python setup.py install 102 | ``` 103 | 104 | Install in "develop" or "editable" mode: 105 | ``` 106 | python setup.py develop 107 | ``` 108 | or 109 | ``` 110 | pip install -e . 111 | ``` 112 | 113 | ## References 114 | 115 | - [Making a Python Package](https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html) 116 | - [https://packaging.python.org/](https://packaging.python.org/) 117 | - [How To Package Your Python Code](https://python-packaging.readthedocs.io/en/latest/) 118 | - [Sample Python Project](https://github.com/pypa/sampleproject) 119 | -------------------------------------------------------------------------------- /programming-notes/python/basics/slice-notation.md: -------------------------------------------------------------------------------- 1 | # Slice notation 2 | 3 | The ASCII art diagram is helpful too for remembering how slices work: 4 | 5 | +---+---+---+---+---+---+ 6 | | P | y | t | h | o | n | 7 | +---+---+---+---+---+---+ 8 | 0 1 2 3 4 5 6 9 | -6 -5 -4 -3 -2 -1 10 | 11 | One way to remember how slices work is to think of the indices as pointing *between* characters, with the left edge of the first character numbered 0. Then the right edge of the last character of a string of *n* characters has index *n*. 12 | 13 | 14 | ```python 15 | a[start:end] # items start through end-1 16 | a[start:] # items start through the rest of the array 17 | a[:end] # items from the beginning through end-1 18 | a[:] # a copy of the whole array 19 | ``` 20 | 21 | There is also the `step` value, which can be used with any of the above: 22 | 23 | ```python 24 | a[start:end:step] # start through not past end, by step 25 | ``` 26 | 27 | The key point to remember is that the `:end` value represents the first value that is *not* in the selected slice. So, the difference beween `end` and `start` is the number of elements selected (if `step` is 1, the default). 28 | 29 | The other feature is that `start` or `end` may be a *negative* number, which means it counts from the end of the array instead of the beginning. So: 30 | 31 | ```python 32 | a[-1] # last item in the array 33 | a[-2:] # last two items in the array 34 | a[:-2] # everything except the last two items 35 | ``` 36 | 37 | Python is kind to the programmer if there are fewer items than you ask for. For example, if you ask for `a[:-2]` and `a` only contains one element, you get an empty list instead of an error. Sometimes you would prefer the error, so you have to be aware that this may happen. 38 | -------------------------------------------------------------------------------- /programming-notes/python/basics/string-formatting.md: -------------------------------------------------------------------------------- 1 | # String formatting in Python 2 | 3 | ## F-strings (Python >=3.6) 4 | 5 | F-strings are faster than the other string formatting methods and are easier to read and use. Here are some tricks you may not have known. 6 | 7 | ```python 8 | name = "Test" 9 | f'My app name is {name}.' # 'My app name is Test. 10 | ``` 11 | 12 | ### 1. Number formatting: 13 | 14 | You can do various formatting with numbers. 15 | 16 | ```python 17 | number = 150 18 | 19 | # decimal places to n -> .nf 20 | print(f"number: {number:.2f}") 21 | # number: 150.00 22 | 23 | # hex conversion 24 | print(f"hex: {number:#0x}") 25 | # hex: 0x96 26 | 27 | # binary conversion 28 | print(f"binary: {number:b}") 29 | # binary: 10010110 30 | 31 | # octal conversion 32 | print(f"octal: {number:o}") 33 | # octal: 226 34 | 35 | # scientific notation 36 | print(f"scientific: {number:e}") 37 | # scientific: 1.500000e+02 38 | 39 | # total number of characters 40 | print(f"Number: {number:09}") 41 | # Number: 000000150 42 | 43 | ratio = 1 / 2 44 | # percentage with 2 decimal places 45 | print(f"percentage = {ratio:.2%}") 46 | # percentage = 50.00% 47 | ``` 48 | 49 | ### 2. Stop writing print(f”var = {var}”) 50 | 51 | This is the debug feature with f-strings. 52 | 53 | ```python 54 | a, b = 5, 15 55 | print(f"a = {a}") # Doing this ? 56 | # a = 5 57 | 58 | # Do this instead. 59 | print(f"{a = }") 60 | # a = 5 61 | 62 | # Arithmatic operations 63 | print(f"{a + b = }") 64 | # a + b = 20 65 | 66 | # with formatting 67 | print(f"{a + b = :.2f}") 68 | # a + b = 20.00 69 | ``` 70 | 71 | ### 3. Date formatting 72 | 73 | You can do `strftime()` formattings from f-string. 74 | 75 | ```python 76 | import datetime 77 | 78 | today = datetime.datetime.now() 79 | print(f"datetime : {today}") 80 | # datetime : 2023-10-27 11:05:40.282314 81 | 82 | print(f"date time: {today:%m/%d/%Y %H:%M:%S}") 83 | # date time: 10/27/2023 11:05:40 84 | 85 | print(f"date: {today:%m/%d/%Y}") 86 | # date: 10/27/2023 87 | 88 | print(f"time: {today:%H:%M:%S %p}") 89 | # time: 11:05:40 AM 90 | ``` 91 | 92 | ## Old method (Python >=2.6) 93 | ```python 94 | '{} {}'.format('one', 'two') # Output: 'one two' 95 | '{} {}'.format(1, 2) # Output: '1 2' 96 | ``` 97 | 98 | ## Very old method 99 | ```python 100 | '%s %s' % ('one', 'two') # Output: 'one two' 101 | '%d %d' % (1, 2) # Output: '1 2' 102 | ``` 103 | 104 | ## Number formatting 105 | 106 | The following table shows various ways to format numbers using str.format(), including examples for both float formatting and integers. 107 | 108 | To run examples use `print("FORMAT".format(NUMBER))`. So to get the output of the first example, you would run: `print("{:.2f}".format(3.1415926))`. 109 | 110 | Number | Format | Output | Description 111 | --- | --- | --- | --- 112 | 3.1415926 | {:.2f} | 3.14 | 2 decimal places 113 | 3.1415926 | {:+.2f} | +3.14 | 2 decimal places with sign 114 | -1 | {:+.2f} | -1.00 | 2 decimal places with sign 115 | 2.71828 | {:.0f} | 3 | No decimal places 116 | 5 | {:0>2d} | 05 | Pad number with zeros (left padding, width 2) 117 | 5 | {:x<4d} | 5xxx | Pad number with x’s (right padding, width 4) 118 | 10 | {:x<4d} | 10xx | Pad number with x’s (right padding, width 4) 119 | 1000000 | {:,} | 1,000,000 | Number format with comma separator 120 | 0.25 | {:.2%} | 25.00% | Format percentage 121 | 1000000000 | {:.2e} | 1.00e+09 | Exponent notation 122 | 13 | {:10d} | 13 | Right aligned (default, width 10) 123 | 13 | {:<10d} | 13 | Left aligned (width 10) 124 | 13 | {:^10d} | 13 | Center aligned (width 10) 125 | -------------------------------------------------------------------------------- /programming-notes/python/beautifulsoup4.md: -------------------------------------------------------------------------------- 1 | # Web scrapping with Beautiful Soup 4 2 | 3 | ## Required modules 4 | ``` 5 | pip install beautifulsoup4 6 | pip install lxml 7 | pip instal html5lib 8 | pip install requests 9 | ``` 10 | 11 | ## Usage 12 | 13 | ```python 14 | from bs4 import BeautifulSoup 15 | import requests 16 | 17 | source = requests.get('http://example.com').text 18 | soup = BeautifulSoup(source, 'lxml') 19 | ``` 20 | 21 | ## References 22 | 23 | https://www.youtube.com/watch?v=ng2o98k983k 24 | -------------------------------------------------------------------------------- /programming-notes/python/boilerplate.md: -------------------------------------------------------------------------------- 1 | # Python boilerplates 2 | 3 | ## Simple Python boilerplate 4 | 5 | ```python 6 | #!/usr/bin/env python3 7 | """ 8 | Module Docstring 9 | """ 10 | 11 | __author__ = "Your Name" 12 | __version__ = "0.1.0" 13 | __license__ = "MIT" 14 | 15 | 16 | def main(): 17 | """ Main entry point of the app """ 18 | print("hello world") 19 | 20 | 21 | if __name__ == "__main__": 22 | """ This is executed when run from the command line """ 23 | main() 24 | ``` 25 | 26 | The first line is for executable scripts. 27 | 28 | ## Argparse boilerplate 29 | ```python 30 | """ 31 | Module Docstring 32 | """ 33 | 34 | __author__ = "Your Name" 35 | __version__ = "0.1.0" 36 | __license__ = "MIT" 37 | 38 | import argparse 39 | 40 | 41 | def main(args): 42 | """ Main entry point of the app """ 43 | print("hello world") 44 | print(args) 45 | 46 | 47 | if __name__ == "__main__": 48 | """ This is executed when run from the command line """ 49 | parser = argparse.ArgumentParser() 50 | 51 | # Required positional argument 52 | parser.add_argument("arg", help="Required positional argument") 53 | 54 | # Optional argument flag which defaults to False 55 | parser.add_argument("-f", "--flag", action="store_true", default=False) 56 | 57 | # Optional argument which requires a parameter (eg. -d test) 58 | parser.add_argument("-n", "--name", action="store", dest="name") 59 | 60 | # Optional verbosity counter (eg. -v, -vv, -vvv, etc.) 61 | parser.add_argument( 62 | "-v", 63 | "--verbose", 64 | action="count", 65 | default=0, 66 | help="Verbosity (-v, -vv, etc)") 67 | 68 | # Specify output of "--version" 69 | parser.add_argument( 70 | "--version", 71 | action="version", 72 | version="%(prog)s (version {version})".format(version=__version__)) 73 | 74 | args = parser.parse_args() 75 | main(args) 76 | ``` 77 | -------------------------------------------------------------------------------- /programming-notes/python/config-files.md: -------------------------------------------------------------------------------- 1 | # Configuration files 2 | 3 | Most interesting programs need some kind of configuration. For very simple tasks you might choose to write these configuration variables directly into the source code. But this is a bad idea when you want to distribute your code and allow users to change configurations. Here are a few alternatives on how to handle configuration files in Python. 4 | 5 | ## Using a `.ini` file [*](https://wiki.python.org/moin/ConfigParserExamples) 6 | 7 | Create a `myfile.ini` like: 8 | ```ini 9 | [SectionOne] 10 | Status: Single 11 | Name: Derek 12 | Value: Yes 13 | Age: 30 14 | Single: True 15 | 16 | [SectionTwo] 17 | FavoriteColor=Green 18 | [SectionThree] 19 | FamilyName: Johnson 20 | 21 | [Others] 22 | barList=item1,item2 23 | ``` 24 | 25 | Retrieve the data like: 26 | ```python 27 | >>> import ConfigParser 28 | >>> Config = ConfigParser.ConfigParser() 29 | >>> Config 30 | 31 | >>> Config.read("myfile.ini") 32 | ['c:\\tomorrow.ini'] 33 | >>> Config.sections() 34 | ['Others', 'SectionThree', 'SectionOne', 'SectionTwo'] 35 | >>> Config.options('SectionOne') 36 | ['Status', 'Name', 'Value', 'Age', 'Single'] 37 | >>> Config.get('SectionOne', 'Status') 38 | 'Single' 39 | ``` 40 | 41 | ## Using YAML [*](https://camel.readthedocs.io/en/latest/yamlref.html) 42 | YAML is a human friendly data serialization standard for all programming languages. 43 | 44 | Create a `config.yml` file 45 | ```yaml 46 | database: 47 | username: admin 48 | password: foobar # TODO get prod passwords out of config 49 | socket: /var/tmp/database.sock 50 | options: {use_utf8: true} 51 | memcached: 52 | host: 10.0.0.99 53 | workers: 54 | - host: 10.0.0.101 55 | port: 2301 56 | - host: 10.0.0.102 57 | port: 2302 58 | ``` 59 | 60 | Parse with: 61 | ```python 62 | import yaml 63 | config = yaml.safe_load(open("config.yml")) 64 | ``` 65 | 66 | ## Using a Python module 67 | 68 | Create a regular Python module, say `config.py`, like this: 69 | ```python 70 | truck = dict( 71 | color = 'blue', 72 | brand = 'ford', 73 | ) 74 | city = 'new york' 75 | cabriolet = dict( 76 | color = 'black', 77 | engine = dict( 78 | cylinders = 8, 79 | placement = 'mid', 80 | ), 81 | doors = 2, 82 | ) 83 | ``` 84 | 85 | Use it like this: 86 | ```python 87 | import config 88 | print(config.truck['color']) 89 | ``` 90 | 91 | ## Other alternatives 92 | - Using a JSON file. 93 | - Using .env files to write configuration as environmental variables. 94 | 95 | ## References 96 | - [What's the best practice using a settings file in Python?](https://stackoverflow.com/questions/5055042/whats-the-best-practice-using-a-settings-file-in-python/34354110#34354110) 97 | - [Configuration files in Python](https://martin-thoma.com/configuration-files-in-python/) 98 | - [From Novice to Expert: How to Write a Configuration file in Python](https://towardsdatascience.com/from-novice-to-expert-how-to-write-a-configuration-file-in-python-273e171a8eb3) 99 | -------------------------------------------------------------------------------- /programming-notes/python/create-exe.md: -------------------------------------------------------------------------------- 1 | # Create a standalone executable of a Python app 2 | 3 | ## Alternatives 4 | 5 | Most popular options: 6 | 7 | - [pyinstaller](https://pyinstaller.readthedocs.io/en/stable/) 8 | - [cx_Freeze](https://cx-freeze.readthedocs.io/en/latest/) 9 | - [Nuitka](https://nuitka.net/) 10 | 11 | Other alternatives: 12 | 13 | - [py2exe](https://www.py2exe.org/) - Based on `pyinstaller`. For Windows. 14 | - [py2app](https://py2app.readthedocs.io/en/latest/) - Based on `pyinstaller`. For MacOS X. 15 | - [PyOxidizer](https://pyoxidizer.readthedocs.io/en/stable/) 16 | 17 | 18 | 19 | ## Using `pyinstaller` 20 | 21 | It's pretty straight forward: 22 | 1. Create an entry-point script that calls your main function. 23 | 2. Install PyInstaller. 24 | 3. Run PyInstaller on your entry-point. 25 | 4. Test your new executable. 26 | 5. Ship your resulting `dist/` folder to users. 27 | 28 | ### Install 29 | 30 | To install `pyinstaller` on your pc run (more details can be found here: https://www.pyinstaller.org/): 31 | 32 | ``` 33 | pip install pyinstaller 34 | ``` 35 | 36 | ### For a single file program 37 | 38 | If your program is a single script use `cmd` to go to your program directory and to turn it into a exe folder run 39 | 40 | ``` 41 | pyinstaller myprogram.py 42 | ``` 43 | 44 | 45 | ### For a package 46 | 47 | Create an entrypoint outside your package folder that imports and runs your program. This will be the entry point. Call pyinstaller on the entrypoint script as above. 48 | 49 | ### Options 50 | 51 | Change the name of your executable: 52 | ``` 53 | pyinstaller myprogram.py --name MY_PROGRAM 54 | ``` 55 | 56 | Package your entire application into a single executable file: 57 | ``` 58 | pyinstaller myprogram.py --onefile 59 | ``` 60 | 61 | Insert additional data or binary files into your build: 62 | ``` 63 | pyinstaller myprogram.py --add-data data/file.csv 64 | ``` 65 | 66 | Exclude some modules from being included with your executable: 67 | ``` 68 | pyinstaller myprogram.py --exclude-module=pytest 69 | ``` 70 | 71 | Avoid automatically opening a console window for stdout logging: 72 | ``` 73 | pyinstaller myprogram.py -w 74 | ``` 75 | 76 | 77 | What I usually run: 78 | 79 | ``` 80 | pyinstaller -F --windowed --icon=myapp.ico myprogram.py 81 | ``` 82 | 83 | This makes it a single exe (without the folders with all dependables) but is slightly slower, minimizes the console window sine I have a GUI, and gives it a snazzy icon that I had saved (you have to copy over the icon, or have an exception incase it isn't found) 84 | 85 | 86 | ## References 87 | 88 | - [Using PyInstaller to Easily Distribute Python Applications](https://realpython.com/pyinstaller-python/) 89 | -------------------------------------------------------------------------------- /programming-notes/python/csv.md: -------------------------------------------------------------------------------- 1 | # CSV files 2 | 3 | CSV (Comma Separated Values) is a very popular import and export data format used in spreadsheets and databases. Each line in a CSV file is a data record. Each record consists of one or more fields, separated by commas. While CSV is a very simple data format, there can be many differecies, such as different delimiters, new lines, or quoting characters. 4 | 5 | ## Reading CSV files 6 | Reading of CSV files into memory: 7 | 8 | ``` 9 | # example1.csv 10 | 1/2/2014,5,8,red 11 | 1/3/2014,5,2,green 12 | 1/4/2014,9,1,blue 13 | ``` 14 | 15 | ```python 16 | import csv 17 | 18 | with open('example1.csv') as csvfile: 19 | readCSV = csv.reader(csvfile, delimiter=',') 20 | for row in readCSV: 21 | print(row) 22 | print(row[0]) 23 | print(row[0],row[1],row[2],) 24 | ``` 25 | 26 | Another example: reading a file with headers and data with specific encoding and delimiter. Print the header and create a data list (without the header line). 27 | 28 | ```python 29 | data = [] 30 | with open('data.csv', 'r', encoding='utf-8') as f: 31 | csvreader = csv.reader(f, delimiter=';') 32 | header = next(csvreader) # skip header 33 | print(header) 34 | for row in csvreader: 35 | data.append([float(val) for val in row]) 36 | ``` 37 | 38 | ### DictReader 39 | The `csv.DictReader` class operates like a regular reader but maps the information read into a dictionary. The keys for the dictionary can be passed in with the fieldnames parameter or inferred from the first row of the CSV file. 40 | 41 | ``` 42 | # example2.csv 43 | min,avg,max 44 | 1, 5.5, 10 45 | 2, 3.5, 5 46 | ``` 47 | 48 | ```python 49 | import csv 50 | 51 | with open('values.csv', 'r') as f: 52 | reader = csv.DictReader(f) 53 | for row in reader: 54 | print(row['min'], row['avg'], row['max']) 55 | ``` 56 | 57 | ## Writing to CSV files 58 | The `csv.writer()` method returns a writer object which converts the user's data into delimited strings on the given file-like object. 59 | 60 | ```python 61 | import csv 62 | 63 | nms = [[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12]] 64 | 65 | with open('numbers2.csv', 'w') as f: 66 | writer = csv.writer(f) 67 | for row in nms: 68 | writer.writerow(row) 69 | ``` 70 | 71 | ### DictWriter 72 | 73 | The csv.DictWriter class operates like a regular writer but maps Python dictionaries into CSV rows. The fieldnames parameter is a sequence of keys that identify the order in which values in the dictionary passed to the writerow() method are written to the CSV file. 74 | ```python 75 | import csv 76 | 77 | with open('names.csv', 'w') as f: 78 | fnames = ['first_name', 'last_name'] 79 | writer = csv.DictWriter(f, fieldnames=fnames) 80 | 81 | writer.writeheader() # writes the headers to the CSV file. 82 | writer.writerow({'first_name' : 'John', 'last_name': 'Smith'}) 83 | writer.writerow({'first_name' : 'Robert', 'last_name': 'Brown'}) 84 | writer.writerow({'first_name' : 'Julia', 'last_name': 'Griffin'}) 85 | ``` 86 | 87 | ## Quoting CSV Files 88 | With the CSV module, you can also perform a variety of quoting functions. 89 | 90 | They are: 91 | 92 | * `csv.QUOTE_ALL` - Quote everything, regardless of type. 93 | * `csv.QUOTE_MINIMAL` - Quote fields with special characters 94 | * `csv.QUOTE_NONNUMERIC` - Quote all fields that are not integers or floats 95 | * `csv.QUOTE_NONE` - Do not quote anything on output 96 | -------------------------------------------------------------------------------- /programming-notes/python/datetime.md: -------------------------------------------------------------------------------- 1 | # Datetime module 2 | 3 | The `datetime` module is a standard library module that supplies classes for manipulating dates and times. While date and time arithmetic is supported, the focus of the implementation is on efficient attribute extraction for output formatting and manipulation. 4 | 5 | The `datetime` module has three types for storing a point in time: 6 | 7 | * `date` for year, month, day of month 8 | * `time` for hours, minutes, seconds, microseconds, time zone info 9 | * `datetime` combines `date` and `time`. It has the methods `date()` and `time()` to get the corresponding date and time objects, and there's a handy `combine` function to combine date and time into a datetime. 10 | 11 | 12 | ## Parsing and formatting date strings 13 | 14 | `datetime.strptime` is the main routine for parsing strings into datetimes. It can handle all sorts of formats, with the format determined by a format string you give it: 15 | 16 | ```python 17 | from datetime import datetime 18 | 19 | datetime_object = datetime.strptime('Jun 1 2005 1:33PM', '%b %d %Y %I:%M%p') 20 | ``` 21 | 22 | `datetime.strftime` is the routine to format datetime objects to string. 23 | ```python 24 | import datetime 25 | 26 | t = datetime.datetime(2012, 2, 23, 0, 0) 27 | t.strftime('%m/%d/%Y') 28 | # output: '02/23/2012' 29 | ``` 30 | 31 | Format options available in the [datetime documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior) and at [strftime.org](http://strftime.org/). 32 | 33 | To summarize: 34 | * `strptime` = "string parse time" 35 | * `strftime` = "string format time" 36 | -------------------------------------------------------------------------------- /programming-notes/python/django/django-pyinstaller.md: -------------------------------------------------------------------------------- 1 | # Create a executable Django App 2 | 3 | Since PyInstaller 2.1, you can build an executable from a project using Django 2.1.8. PyInstaller will take care of a _lot_ of the magic needed to correctly build Django applications. For instance, it will parse all your files and find all the dotted-names strings that refer to module names (eg: within `settings.py`) and make sure that all those modules are packaged. 4 | 5 | You should be able to run PyInstaller over the `manage.py` script directly. 6 | 7 | The following example should clarify the required steps. 8 | 9 | ## Example 10 | 11 | Let's say that your Django project is called `mysite`. If you used the `django-admin.py` wizard, you probably have ended up with a [Django project directory structure](https://docs.djangoproject.com/en/1.8/intro/tutorial01/). PyInstaller expects a directory structure like this: 12 | ``` 13 | mysite/ 14 | manage.py 15 | mysite/ 16 | __init__.py 17 | settings.py 18 | urls.py 19 | wsgi.py 20 | ... 21 | ``` 22 | Use PyInstaller: 23 | ```console 24 | $ pyinstaller --name=mysite mysite/manage.py 25 | ``` 26 | Notice the use of the `--name` option to make sure the output is a packaged executable called `mysite` and not `manage` from the script name. 27 | 28 | Now you should have a frozen django app in the directory `./dist/mysite/`. You should be able to see an executable file `./dist/mysite/mysite.exe`. Use this file the same way as you would use `manage.py`. 29 | 30 | To run the built-in Django test server run: 31 | ```console 32 | $ ./dist/mysite/mysite.exe runserver localhost:8000 33 | ``` 34 | 35 | ## Multiple Settings Modules 36 | 37 | If you're using multiple settings files in your Django project, you'll need to do a bit more work to get pyinstaller to use the correct one. Say you had the following configuration: 38 | 39 | ``` 40 | mysite/ 41 | manage.py 42 | mysite/ 43 | __init__.py 44 | settings/ 45 | __init__.py 46 | base.py 47 | development.py 48 | production.py 49 | urls.py 50 | wsgi.py 51 | ``` 52 | 53 | In this case, if you want PyInstaller to use `mysite.settings.production`, you should modify `mysite/settings/__init__.py` to include the following: 54 | 55 | ```python 56 | from production import * 57 | ``` 58 | 59 | This is required because PyInstaller only looks for settings files at two locations: either `mysite/settings.py`, or `mysite/settings`. 60 | 61 | 62 | ## What if it does not work? 63 | 64 | Django uses a lot of magic under the hood, so some quirks are expected. If the application does not run, you should see the traceback in the console. If it is an `ImportError`, it is probably a missing module whose dependency PyInstaller was not able to automatically discover. 65 | 66 | The quickest workaround is to add an import statement for that module, for instance in the `manage.py` script. But please report the incident to the mailing list so that we can prepare a proper fix that everyone can benefit from. 67 | 68 | If this does not help, have a look at [[How to Report Bugs#before-submitting-a-report-make-sure-everything-is-packaged]] to track down the problem. 69 | 70 | ## References 71 | - [https://realpython.com/installable-django-app/](How to Write an Installable Django App) 72 | -------------------------------------------------------------------------------- /programming-notes/python/django/django-rest-api.md: -------------------------------------------------------------------------------- 1 | # Create a REST API with Django Rest Framework 2 | 3 | 4 | ## References: 5 | 6 | - [Django REST framework](https://www.django-rest-framework.org/) 7 | - [Full Stack React & Django](https://www.youtube.com/watch?v=Uyei2iDA4Hs) 8 | - [Modern Django: Part 1: Setting up Django and React](http://v1k45.com/blog/modern-django-part-1-setting-up-django-and-react/) 9 | - [Django REST with React](https://www.valentinog.com/blog/drf/#Django_REST_with_React_Django_and_React_together) 10 | - [Django Rest Framework Tutorial - Todo API](https://wsvincent.com/django-rest-framework-todo-api-tutorial/) 11 | - [Let’s build an API with Django REST Framework](https://medium.com/backticks-tildes/lets-build-an-api-with-django-rest-framework-32fcf40231e5) 12 | - [The Ultimate Tutorial for Django REST Framework](https://sunscrapers.com/blog/ultimate-tutorial-django-rest-framework-part-1/) 13 | -------------------------------------------------------------------------------- /programming-notes/python/django/django-uv-workflow.md: -------------------------------------------------------------------------------- 1 | # Django and UV workflow 2 | 3 | Create a project folder 4 | 5 | ``` 6 | mkdir django-uv-test 7 | ``` 8 | 9 | Update uv: 10 | 11 | ``` 12 | uv self update 13 | 14 | ``` 15 | 16 | Start uv project: 17 | 18 | ``` 19 | uv init django-project --python 3.11 20 | 21 | ``` 22 | 23 | Add packages: 24 | 25 | ``` 26 | cd django-project 27 | uv add Django django-extensions 28 | ``` 29 | 30 | 31 | Start Django project: 32 | 33 | ``` 34 | uv run django-admin startproject myproject . 35 | ``` 36 | 37 | 38 | Run Django management commands: 39 | 40 | ``` 41 | uv run manage.py runserver 42 | uv run manage.py startapp appname 43 | uv run manage.py makemigrations 44 | uv run manage.py migrate 45 | uv run manage.py shell_plus # from django-extensions 46 | ``` 47 | 48 | 49 | Working with development dependencies: 50 | 51 | ``` 52 | uv add --dev black 53 | uv run black path\to\file.py 54 | ``` 55 | 56 | 57 | Deploying the project to production excluding dev dependencies: 58 | 59 | ``` 60 | uv sync --no-dev 61 | ``` 62 | 63 | 64 | References 65 | 66 | - [https://www.youtube.com/watch?v=hm-rDxSMzSw](https://www.youtube.com/watch?v=hm-rDxSMzSw) 67 | -------------------------------------------------------------------------------- /programming-notes/python/email.md: -------------------------------------------------------------------------------- 1 | # Sending e-mails with Python 2 | 3 | In all examples I am using a gmail server to send emails. If you are not using a gmail address you need to insert the respective smtp server and possibly the port. 4 | 5 | - Gmail: smtp.gmail.com, port 587 6 | - Hotmail: smtp.live.com, port 587 7 | - Outlook: smtp-mail.outlook.com, port 587 8 | 9 | 10 | ### Very basic example 11 | This is a very barebones example. No subject is added to the email. 12 | 13 | ```python 14 | import smtplib 15 | 16 | sender = 'sender@gmail.com' 17 | pwd = 'senderpassword' # password for sender email 18 | receiver = 'receiver@gmail.com' 19 | msg = 'Hi there!' 20 | 21 | server = smtplib.SMTP('smtp.gmail.com', 587) # smtp server and port for the sender email 22 | server.ehlo() 23 | server.starttls() # starts a TLS encrypted connection 24 | server.login(sender, pwd) 25 | server.sendmail(sender, receiver, msg) 26 | server.close() 27 | 28 | print('Mail sent!') 29 | ``` 30 | 31 | --- 32 | 33 | ### More complete case 34 | This uses the Python email library to help compose the email. You can add the subject to the email. 35 | 36 | ```python 37 | import smtplib 38 | from email.message import EmailMessage 39 | 40 | sender = 'sender@gmail.com' 41 | pwd = 'senderpassword' # password for sender email 42 | receiver = 'receiver@gmail.com' 43 | message = 'Hi there!' 44 | 45 | msg = EmailMessage() 46 | msg.set_content(message) 47 | msg['Subject'] = subject 48 | msg['From'] = sender 49 | msg['To'] = receiver 50 | 51 | server = smtplib.SMTP('smtp.gmail.com', 587) 52 | server.ehlo() 53 | server.starttls() # starts a TLS encrypted connection 54 | server.login(sender, pwd) 55 | server.sendmail(sender, receiver, msg.as_string()) 56 | server.close() 57 | 58 | print('Mail sent!') 59 | ``` 60 | 61 | --- 62 | 63 | ### HTML email 64 | Here’s an example of how to create an HTML message with an alternative plain text version. 65 | 66 | ```python 67 | import smtplib 68 | 69 | from email.mime.multipart import MIMEMultipart 70 | from email.mime.text import MIMEText 71 | 72 | # me == my email address 73 | # you == recipient's email address 74 | me = "my@email.com" 75 | you = "your@email.com" 76 | 77 | # Create message container - the correct MIME type is multipart/alternative. 78 | msg = MIMEMultipart('alternative') 79 | msg['Subject'] = "Link" 80 | msg['From'] = me 81 | msg['To'] = you 82 | 83 | # Create the body of the message (a plain-text and an HTML version). 84 | text = "Hi!\nHow are you?\nHere is the link you wanted:\nhttp://www.python.org" 85 | html = """\ 86 | 87 | 88 | 89 |

    Hi!
    90 | How are you?
    91 | Here is the link you wanted. 92 |

    93 | 94 | 95 | """ 96 | 97 | # Record the MIME types of both parts - text/plain and text/html. 98 | part1 = MIMEText(text, 'plain') 99 | part2 = MIMEText(html, 'html') 100 | 101 | # Attach parts into message container. 102 | # According to RFC 2046, the last part of a multipart message, in this case 103 | # the HTML message, is best and preferred. 104 | msg.attach(part1) 105 | msg.attach(part2) 106 | 107 | # Send the message via local SMTP server. 108 | s = smtplib.SMTP('localhost') 109 | # sendmail function takes 3 arguments: sender's address, recipient's address 110 | # and message to send - here it is sent as one string. 111 | s.sendmail(me, you, msg.as_string()) 112 | s.quit() 113 | ``` 114 | -------------------------------------------------------------------------------- /programming-notes/python/flask.md: -------------------------------------------------------------------------------- 1 | # Building a Flask site 2 | 3 | ### Installing Flask and other usefull Python modules 4 | ``` 5 | pip install Flask 6 | pip install flask-mysqldb 7 | pip install Flask-WTF 8 | pip install passlib 9 | ``` 10 | 11 | ### Project file structure 12 | ``` 13 | /flask-site 14 | /venv 15 | /flask-site 16 | __init__.py 17 | /static 18 | style.css 19 | /templates 20 | layout.html 21 | index.html 22 | about.html 23 | ... 24 | ``` 25 | 26 | ### The `__init__.py` file 27 | 28 | ```python 29 | from flask import Flask, render_template, flash, redirect, url_for, session, request, logging 30 | 31 | app = Flask(__name__) 32 | 33 | # Home page route 34 | @app.route('/') 35 | def index(): 36 | return render_template('index.html') 37 | 38 | # About page route 39 | @app.route('/about') 40 | def about(): 41 | return render_template('about.html') 42 | 43 | if __name__ == '__main__': 44 | app.secret_key = 'secret123' 45 | app.run(debug=True) 46 | 47 | ``` 48 | -------------------------------------------------------------------------------- /programming-notes/python/ghp-import.md: -------------------------------------------------------------------------------- 1 | # Deploying to Github pages with `ghp-import` 2 | 3 | ## Example directory structure 4 | 5 | ``` 6 | myproject/ 7 | ├── myproject/ 8 | ├── jupyterbook/ 9 | ├── README.md 10 | ├── requirements.txt 11 | ├── build.bat 12 | ``` 13 | 14 | ## Building with `ghp-import` 15 | 16 | The build.bat file can then be used to run: 17 | 18 | ```batch 19 | jupyter-book build jupyterbook 20 | cd jupyterbook 21 | ghp-import -n -p -f _build/html 22 | cd .. 23 | ``` 24 | 25 | 26 | ## References 27 | 28 | - [GitHub Pages Import](https://github.com/c-w/ghp-import) 29 | -------------------------------------------------------------------------------- /programming-notes/python/gui.md: -------------------------------------------------------------------------------- 1 | # Graphical User Interfaces (GUIs) in Python 2 | 3 | ### For simple command line programs 4 | 5 | - [Gooey](https://github.com/chriskiehl/Gooey) - Turn (almost) any Python command line program into a full GUI application 6 | 7 | 8 | ### GUI libraries 9 | 10 | - [TkInter](https://wiki.python.org/moin/TkInter) - Python's de-facto standard GUI available in the standard library 11 | 12 | - [PySimpleGUI](https://pysimplegui.readthedocs.io/en/latest/) - Transforms tkinter, Qt, Remi, WxPython into portable people-friendly Pythonic interfaces. Easy for beginners, not really recommended for production ready app. 13 | 14 | - [Kivy](https://kivy.org/) - Cross-platform GUI library supporting both desktop operating systems (Windows, macOS, Linux) and mobile devices (Android, iOS). 15 | 16 | - [PyQt5](https://riverbankcomputing.com/software/pyqt/intro), [QtPy](https://pypi.org/project/QtPy/), [PySide](https://wiki.qt.io/PySide) - There are Python bindings available for the Qt toolkit (using either PyQt or PySide). PyQt is currently more mature than PySide, but you must buy a PyQt license if you want to write proprietary applications. PySide is free for all applications. 17 | 18 | - [DearPyGUI](https://github.com/hoffstadt/DearPyGui) - Has nice documentation, good tutorial made by dev himself, loaded with features but frequent updates change the functionality sometimes. Built with Dear ImGui. 19 | 20 | - [WX-Python](https://www.wxpython.org/#) - Hasn't been updated in a while. 21 | 22 | - [Toga](https://github.com/beeware/toga) - Still in beta. 23 | 24 | ### Other 25 | 26 | - [Streamlit](https://streamlit.io/docs/) - Makes it easy to build beautiful apps for machine learning. 27 | -------------------------------------------------------------------------------- /programming-notes/python/image-manipulation.md: -------------------------------------------------------------------------------- 1 | # Image manipulation in Python 2 | 3 | The **Python Imaging Library** (PIL) development has stagnated, with its last release in 2009. Luckily, there’s an actively-developed fork of PIL called **Pillow**. 4 | **Pillow** is easier to install, runs on all operating systems, and supports Python 3. 5 | 6 | ## Pillow 7 | 8 | * Pillow and PIL cannot co-exist in the same environment. Before installing Pillow, please uninstall PIL. 9 | * Pillow >= 2.0.0 supports Python versions 2.6, 2.7, 3.2, 3.3, 3.4 10 | * Before installing Pillow, you’ll have to install Pillow’s prerequisites. [Instructions here](https://pillow.readthedocs.io/en/3.0.0/installation.html). 11 | 12 | ### Instalation: 13 | 14 | **Windows**: 15 | 16 | `$ pip install Pillow` 17 | 18 | **Linux (Debian or Ubuntu)**: 19 | 20 | Make sure you have Python’s development libraries installed. In Debian or Ubuntu: `$ sudo apt-get install python3-dev python3-setuptools` 21 | Prerequisites are installed on Ubuntu 12.04 LTS or Raspian Wheezy 7.0 with: `$ sudo apt-get install libtiff4-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.5-dev tk8.5-dev python-tk` 22 | 23 | `$ pip install Pillow` 24 | 25 | ```python 26 | from PIL import Image, ImageFilter 27 | #Read image 28 | im = Image.open( 'image.jpg' ) 29 | #Display image 30 | im.show() 31 | 32 | #Applying a filter to the image 33 | im_sharp = im.filter( ImageFilter.SHARPEN ) 34 | #Saving the filtered image to a new file 35 | im_sharp.save( 'image_sharpened.jpg', 'JPEG' ) 36 | 37 | #Splitting the image into its respective bands, i.e. Red, Green, 38 | #and Blue for RGB 39 | r,g,b = im_sharp.split() 40 | 41 | #Viewing EXIF data embedded in image 42 | exif_data = im._getexif() 43 | exif_data 44 | ``` 45 | -------------------------------------------------------------------------------- /programming-notes/python/json.md: -------------------------------------------------------------------------------- 1 | # Working with JSON 2 | 3 | ## Saving 4 | 5 | ```python 6 | import json 7 | with open('data.json', 'w') as fp: 8 | json.dump(data, fp) 9 | ``` 10 | 11 | Supply extra arguments like sort_keys or indent to get a pretty result. The argument sort_keys will sort the keys alphabetically and indent will indent your data structure with indent=N spaces. 12 | 13 | ```python 14 | json.dump(data, fp, sort_keys=True, indent=4) 15 | ``` 16 | 17 | ## Loading 18 | ```python 19 | with open('data.json', 'r') as fp: 20 | data = json.load(fp) 21 | ``` 22 | 23 | #### json.dump() vs json.dumps(), json.load() vs json.loads() 24 | 25 | If you want to dump the JSON into a file/socket or whatever, then you should go for `dump()`. If you only need it as a string (for printing, parsing or whatever) then use `dumps()` (dump string). The functions with an `s` take string parameters. The others take file streams. 26 | 27 | ## Query JSON 28 | JMESPath is a query language for JSON, which allows you to obtain the data you need from a JSON document or dictionary easily. This library is available for Python, but also for many other programming languages, meaning that if you master the JMESPath query language, you can use it in many places. Here's some example code to get a feeling for what's possible: 29 | 30 | ```python 31 | import jmespath 32 | 33 | persons = { 34 | "persons": [ 35 | { "name": "erik", "age": 38 }, 36 | { "name": "john", "age": 45 }, 37 | { "name": "rob", "age": 14 } 38 | ] 39 | } 40 | jmespath.search('persons[*].age', persons) 41 | # [38, 45, 14] 42 | ``` 43 | 44 | ## References 45 | - [python.land/data-processing/working-with-json/jmespath](https://python.land/data-processing/working-with-json/jmespath) 46 | -------------------------------------------------------------------------------- /programming-notes/python/jupyter.md: -------------------------------------------------------------------------------- 1 | # Jupyter 2 | 3 | **Jupyter Notebook** is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more. 4 | 5 | **JupyterLab** is the next-generation user interface for Project Jupyter. It offers all the familiar building blocks of the classic Jupyter Notebook (notebook, terminal, text editor, file browser, rich outputs, etc.) in a flexible and powerful user interface. 6 | 7 | ## Setup Jupyter Notebook 8 | 9 | Install Jupyter Notebook with the following command. Alternatively, it is already packaged and ready to use with the Anaconda distribution. 10 | 11 | ``` 12 | pip install jupyter 13 | ``` 14 | 15 | Starting the Notebook Server: 16 | ``` 17 | jupyter notebook 18 | ``` 19 | 20 | Open a specific Notebook: 21 | ``` 22 | jupyter notebook notebook.ipynb 23 | ``` 24 | 25 | ## Setup JupyterLab 26 | 27 | Install JupyterLab with the following command. 28 | 29 | ``` 30 | pip install jupyterlab 31 | ``` 32 | 33 | Launch by running: 34 | ``` 35 | jupyter lab 36 | ``` 37 | 38 | ## Useful JupyterLab extensions 39 | - JupyterLab-LSP: adds code assistance capabilities to JupyterLab 40 | - Debugger: a proper debugger 41 | - jupyterlab-toc: auto-generates a table of contents in the left area when you have a notebook or markdown document open 42 | - jupyterlab_spellchecker: spellchecker for markdown cells 43 | 44 | [Here is a curated list of JupyterLab extensions](https://github.com/mauhai/awesome-jupyterlab) 45 | 46 | ## Useful IPython commands 47 | 48 | The `%matplotlib inline` command is an IPython magic function that sets the output of plotting commands to be displayed inline within frontends like the Jupyter notebook, directly below the code cell that produced it. The resulting plots will then also be stored in the notebook document. 49 | 50 | ``` 51 | %matplotlib inline 52 | ``` 53 | 54 | The `figsize` sets the figure size 55 | 56 | ``` 57 | from IPython.core.pylabtools import figsize 58 | 59 | figsize(10, 12) 60 | ``` 61 | 62 | Globaly set the scale of text and font for Seaborn 63 | 64 | ``` 65 | import seaborn as sns 66 | sns.set(font_scale=2, font='Palatino Linotype') 67 | ``` 68 | 69 | ## Add parent folder to path 70 | 71 | This is useful so you can import into the notebook utility functions from a parent directory. 72 | 73 | ```python 74 | # Add parent directory to Python path 75 | import os 76 | import sys 77 | module_path = os.path.abspath(os.path.join('..')) 78 | if module_path not in sys.path: 79 | sys.path.append(module_path) 80 | 81 | # Import function fun1 from modeule file1 in directory utils 82 | from utils.file1 import fun1 83 | ``` 84 | 85 | ### Here is another alternative. 86 | 87 | Supose you have the following diretory structure. 88 | 89 | ``` 90 | myproject/ 91 | |-- README 92 | |-- myproject/ 93 | |-- utils.py 94 | |-- notebooks/ 95 | |-- example.ipynb 96 | ``` 97 | 98 | We want to use our functions inside `utils.py` from our `example.ipynb` notebook. 99 | 100 | ```python 101 | import sys; sys.path.insert(0, '..') 102 | 103 | from myproject.utils import func1 104 | ``` 105 | 106 | 107 | 108 | ## References 109 | 110 | - [JupyterLab Documentation](https://jupyterlab.readthedocs.io/en/stable/index.html) 111 | -------------------------------------------------------------------------------- /programming-notes/python/logging.md: -------------------------------------------------------------------------------- 1 | # Logging 2 | 3 | Pyhton standard library comes with a powerful logging module. 4 | 5 | ## Usage 6 | 7 | By default, there are 5 standard levels indicating the severity of events: `DEBUG`, `INFO`, `WARNING`, `ERROR` and `CRITICAL`. 8 | The default logger configurarion will log all events with level `WARNING` or above. Example: 9 | 10 | ```python 11 | import logging 12 | 13 | logging.debug('This is a debug message') 14 | logging.info('This is an info message') 15 | logging.warning('This is a warning message') 16 | logging.error('This is an error message') 17 | logging.critical('This is a critical message') 18 | 19 | # Outputs: 20 | # WARNING:root:This is a warning message 21 | # ERROR:root:This is an error message 22 | # CRITICAL:root:This is a critical message 23 | ``` 24 | 25 | ### Configurating the logger 26 | 27 | ```python 28 | import logging 29 | 30 | logging.basicConfig(filename='log.txt', filemode='w', level=logging.DEBUG, format='%(name)s - %(levelname)s - %(message)s') 31 | logging.warning('This will log every log level to a file.') 32 | ``` 33 | 34 | Adding time info: 35 | 36 | ```python 37 | import logging 38 | 39 | logging.basicConfig(format='%(asctime)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S') 40 | logging.warning('Admin logged out') 41 | ``` 42 | 43 | ## References: 44 | 45 | - [Logging in Python](https://realpython.com/python-logging/) 46 | - [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html) 47 | -------------------------------------------------------------------------------- /programming-notes/python/machine-learning/ml-pipeline.md: -------------------------------------------------------------------------------- 1 | # Pipelines 2 | 3 | A pipeline chains together multiple steps, meaning the output of each step is used as input to the next step. 4 | 5 | ## Creating a pipeline 6 | 7 | Load data: 8 | ```python 9 | import pandas as pd 10 | import numpy as np 11 | 12 | train = pd.DataFrame({'feat1':[10, 20, np.nan, 2], 'feat2':[25., 20, 5, 3], 'label':['A', 'A', 'B', 'B']}) 13 | test = pd.DataFrame({'feat1':[30., 5, 15], 'feat2':[12, 10, np.nan]}) 14 | ``` 15 | 16 | Defining steps: 17 | ```python 18 | from sklearn.impute import SimpleImputer 19 | from sklearn.linear_model import LogisticRegression 20 | 21 | imputer = SimpleImputer() 22 | clf = LogisticRegression() 23 | ``` 24 | 25 | Create a 2-step pipeline. Impute missing values, then pass the results to the classifier: 26 | ```python 27 | from sklearn.pipeline import make_pipeline 28 | 29 | pipe = make_pipeline(imputer, clf) 30 | ``` 31 | 32 | Using the pipeline: 33 | ```python 34 | features = ['feat1', 'feat2'] 35 | X, y = train[features], train['label'] 36 | X_new = test[features] 37 | 38 | # pipeline applies the imputer to X before fitting the classifier 39 | pipe.fit(X, y) 40 | 41 | # pipeline applies the imputer to X_new before making predictions 42 | # note: pipeline uses imputation values learned during the "fit" step 43 | pipe.predict(X_new) 44 | ``` 45 | 46 | ## `make_pipeline` vs `Pipeline` 47 | 48 | `Pipeline` requires naming of steps while `make_pipeline` does not. 49 | 50 | With `make_pipeline`: 51 | ```python 52 | from sklearn.pipeline import make_pipeline 53 | 54 | pipe = make_pipeline(imputer, clf) 55 | ``` 56 | 57 | With `Pipeline`: 58 | ```python 59 | from sklearn.pipeline import Pipeline 60 | 61 | pipe = Pipeline([('preprocessor', imputer), ('classifier', clf)]) 62 | ``` 63 | 64 | ## Examine the intermediate steps in a Pipeline 65 | Use the `named_steps` attribute as `pipe.named_steps.STEP_NAME.ATTRIBUTE`: 66 | 67 | ```python 68 | pipe.named_steps.imputer.statistics_ 69 | # or 70 | pipe.named_steps.preprocessor.statistics_ 71 | ``` 72 | 73 | If using `make_pipeline` the name of the step is the name of the variable (here `imputer`). When using `Pipeline` the name is the assigned name when creating the pipeline (here `preprocessor`). 74 | 75 | ## Cross-validate and grid search an entire pipeline 76 | 77 | Cross-validate the entire pipeline (not just the model): 78 | ```python 79 | from sklearn.model_selection import cross_val_score 80 | 81 | cross_val_score(pipe, X, y, cv=5, scoring='accuracy').mean() 82 | ``` 83 | 84 | Find optimal tuning parameters for the entire pipeline: 85 | ```python 86 | # specify parameter values to search 87 | params = {} 88 | params['columntransformer__countvectorizer__min_df'] = [1, 2] 89 | params['logisticregression__C'] = [0.1, 1, 10] 90 | params['logisticregression__penalty'] = ['l1', 'l2'] 91 | 92 | # try all possible combinations of those parameter values 93 | from sklearn.model_selection import GridSearchCV 94 | grid = GridSearchCV(pipe, params, cv=5, scoring='accuracy') 95 | grid.fit(X, y); 96 | ``` 97 | 98 | Best score found during the search: 99 | ```python 100 | grid.best_score_ 101 | ``` 102 | 103 | Combination of parameters that produced the best score: 104 | ```python 105 | grid.best_params_ 106 | ``` 107 | 108 | ## Pipeline diagram 109 | 110 | Create interactive diagrams of Pipelines (and other estimators): 111 | ```python 112 | from sklearn import set_config 113 | set_config(display='diagram') 114 | 115 | pipe = make_pipeline(ct, selection, logreg) 116 | pipe 117 | ``` 118 | 119 | Export the diagram to an HTML file: 120 | ```python 121 | from sklearn.utils import estimator_html_repr 122 | 123 | with open('pipeline.html', 'w') as f: 124 | f.write(estimator_html_repr(pipe)) 125 | ``` 126 | 127 | ## Operate on part of a Pipeline 128 | 129 | Slice the Pipeline using Python's slicing notation: 130 | 131 | ```python 132 | # access step 0 (preprocessor) 133 | pipe[0].fit_transform(X) 134 | 135 | # access steps 0 and 1 (preprocessor and feature selector) 136 | pipe[0:2].fit_transform(X, y) 137 | 138 | # access step 1 (feature selector) 139 | pipe[1].get_support() 140 | ``` 141 | 142 | 143 | ## References 144 | - [scikit-learn tips](https://github.com/justmarkham/scikit-learn-tips) 145 | 146 | -------------------------------------------------------------------------------- /programming-notes/python/machine-learning/ml-save-model.md: -------------------------------------------------------------------------------- 1 | # Saving Scikit-learn model for reuse 2 | 3 | Saving the model with Pickle or Joblib allows you to recover the full Scikit-learn estimator object however some compatibility issues may occur if using different versions of sklearn. Other methods of saving models can deploy the model for prediction, usually by using tools supporting open model interchange formats but do not allow the recovery of the full Scikit-learn estimator object. 4 | 5 | ## Index 6 | * [Using Joblib](#using-joblib) 7 | * [Using Pickle](#using-pickle) 8 | * [Saving a complete ML pipeline](#saving-a-complete-ml-pipeline) 9 | 10 | 11 | ## Using Joblib 12 | 13 | Joblib is part of the SciPy ecosystem and provides utilities for pipelining Python jobs. It provides utilities for saving and loading Python objects that make use of NumPy data structures, efficiently. This can be useful for some machine learning algorithms that require a lot of parameters or store the entire dataset (like k-Nearest Neighbors). 14 | 15 | ```python 16 | from sklearn import svm 17 | from sklearn import datasets 18 | clf = svm.SVC() 19 | X, y= datasets.load_iris(return_X_y=True) 20 | clf.fit(X, y) 21 | SVC() 22 | ``` 23 | 24 | Saving and reusing the model: 25 | ```python 26 | # Save 27 | from joblib import dump 28 | dump(clf, 'filename.joblib') 29 | 30 | # Load 31 | from joblib import load 32 | clf = load('filename.joblib') 33 | ``` 34 | 35 | 36 | ## Using Pickle 37 | 38 | Pickle is the standard way of serializing objects in Python. You can use the pickle operation to serialize your machine learning algorithms and save the serialized format to a file. Later you can load this file to deserialize your model and use it to make new predictions. 39 | 40 | Train the model: 41 | 42 | ```python 43 | from sklearn import svm 44 | from sklearn import datasets 45 | 46 | clf = svm.SVC(gamma='scale') 47 | iris = datasets.load_iris() 48 | X, y = iris.data, iris.target 49 | clf.fit(X, y) 50 | ``` 51 | 52 | Saving and reusing the model: 53 | 54 | ```python 55 | import pickle 56 | 57 | # Save to file 58 | pickle.dump(clf, open('model.sav', 'wb')) 59 | 60 | # Load from file 61 | clf2 = pickle.load(open('model.sav', 'rb')) 62 | 63 | # Reuse 64 | clf2.predict(X[0:1]) 65 | 66 | y[0] 67 | ``` 68 | 69 | ## Saving a complete ML pipeline 70 | 71 | Often it is useful, not only saving the ML model, but also other required component lique the scaler used. In the following you can use `joblib` or `pickle`. The point is to create a pipeline so that you don't have to separately call the scaler. 72 | 73 | Create the pipeline: 74 | ```python 75 | from sklearn.pipeline import make_pipeline 76 | from sklearn.preprocessing import MinMaxScaler 77 | from sklearn.externals import joblib 78 | 79 | pipeline = make_pipeline(MinMaxScaler(),YOUR_ML_MODEL() ) 80 | 81 | model = pipeline.fit(X_train, y_train) 82 | ``` 83 | 84 | Save: 85 | ```python 86 | joblib.dump(model, 'filename.mod') 87 | ``` 88 | 89 | Load: 90 | ```python 91 | model = joblib.load('filename.mod') 92 | ``` 93 | 94 | 95 | ## References: 96 | - [Scikit-learn docs](https://scikit-learn.org/stable/modules/model_persistence.html) 97 | - [Machine Learning — How to Save and Load scikit-learn Models](https://medium.com/datadriveninvestor/machine-learning-how-to-save-and-load-scikit-learn-models-d7b99bc32c27) 98 | - [Save and Load Machine Learning Models in Python with scikit-learn](https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/) 99 | -------------------------------------------------------------------------------- /programming-notes/python/machine-learning/ml-workflows.md: -------------------------------------------------------------------------------- 1 | # Simple Machine Learning workflow 2 | 3 | This can be adapted to solve many ML problems. It has plenty of shortcomings, but can work surprisingly well as-is. Main shortcomings include: 4 | - Assumes all columns have proper data types 5 | - May include irrelevant or improper features 6 | - Does not handle text or date columns well 7 | - Does not include feature engineering 8 | - Ordinal encoding may be better 9 | - Other imputation strategies may be better 10 | - Numeric features may not need scaling 11 | - A different model may be better 12 | 13 | ## Load data 14 | 15 | ```python 16 | import pandas as pd 17 | 18 | cols = ['Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'] 19 | 20 | # Train data 21 | df = pd.read_csv('http://bit.ly/kaggletrain') 22 | X = df[cols] 23 | y = df['Survived'] 24 | 25 | # Test data 26 | df_new = pd.read_csv('http://bit.ly/kaggletest', nrows=10) 27 | X_new = df_new[cols] 28 | ``` 29 | 30 | ## ML imports 31 | 32 | ```python 33 | from sklearn.impute import SimpleImputer 34 | from sklearn.preprocessing import StandardScaler, OneHotEncoder 35 | from sklearn.compose import make_column_selector, make_column_transformer 36 | from sklearn.pipeline import make_pipeline 37 | from sklearn.linear_model import LogisticRegression 38 | from sklearn.model_selection import cross_val_score 39 | ``` 40 | 41 | ## Preprocessing 42 | 43 | ```python 44 | # set up preprocessing for numeric columns 45 | imp_median = SimpleImputer(strategy='median', add_indicator=True) 46 | scaler = StandardScaler() 47 | 48 | # set up preprocessing for categorical columns 49 | imp_constant = SimpleImputer(strategy='constant') 50 | ohe = OneHotEncoder(handle_unknown='ignore') 51 | 52 | # select columns by data type 53 | num_cols = make_column_selector(dtype_include='number') 54 | cat_cols = make_column_selector(dtype_exclude='number') 55 | 56 | # do all preprocessing 57 | preprocessor = make_column_transformer( 58 | (make_pipeline(imp_median, scaler), num_cols), 59 | (make_pipeline(imp_constant, ohe), cat_cols)) 60 | ``` 61 | 62 | ## Create pipeline 63 | ```python 64 | # create a pipeline 65 | pipe = make_pipeline(preprocessor, LogisticRegression()) 66 | 67 | # cross-validate the pipeline 68 | cross_val_score(pipe, X, y).mean() 69 | ``` 70 | 71 | ## Fitting and making predictions 72 | ```python 73 | # fit the pipeline and make predictions 74 | pipe.fit(X, y) 75 | pipe.predict(X_new) 76 | ``` 77 | -------------------------------------------------------------------------------- /programming-notes/python/matlab.md: -------------------------------------------------------------------------------- 1 | # Run Matlab programs in Python 2 | 3 | This example shows how to create a Python package using a MATLAB function. You can then pass the generated package to the developer who is responsible for integrating it into an application. The target system does not require a licensed copy of MATLAB. 4 | 5 | ## Requirements 6 | 7 | * A Matlab version with a license for MATLAB Compiler or MATLAB Compiler SDK to compile the Python package. 8 | * The target computer (where the final application will run) does not require Matlab or a Matlab Licence. However, it requires the Matlab Runtime (free, [download here](https://www.mathworks.com/products/compiler/matlab-runtime.html)) of the same version of the Matlab used to compile the application. For instance, if the package was compiled in Matlab 2018a the target computer must have Matlab Runtime version 2018a installed 9 | 10 | 11 | 12 | ## 1) Create a Python package for the Maltab Application 13 | 14 | For this example let's create a simple Matlab function and save it as `makesqr.m`: 15 | 16 | ```matlab 17 | function y = makesqr(x) 18 | 19 | y = magic(x); 20 | ``` 21 | 22 | Now lets create the Python package. Enter `libraryCompiler` at the MATLAB prompt to open the Library Compiler. To use the Library Compiler you need a license for MATLAB Compiler or MATLAB Compiler SDK. 23 | 24 | In the MATLAB Compiler project window, specify the main file of the MATLAB application that you want to deploy. Select whether to include the MATLAB Runtime installer in the generated application by selecting one of the two options in the Packaging Options section: 25 | 26 | * Runtime downloaded from web — Generates an installer that downloads the MATLAB Runtime and installs it along with the deployed MATLAB application. 27 | 28 | * Runtime included in package — Generates an application that includes the MATLAB Runtime installer. 29 | 30 | Fill in the remaining fields. For this example lets edit the Library Name field, replacing makesqr with `MagicSquarePkg`. 31 | 32 | To generate the packaged application, click Package. When the deployment process is complete, the output folder contains: 33 | 34 | * for_redistribution — Folder containing the file that installs the application and the MATLAB Runtime. 35 | 36 | * for_testing — Folder containing all the artifacts created by mcc, such as binaries and JAR, header, and source files for a specific target. Use these files to test the installation. 37 | 38 | * for_redistribution_files_only — Folder containing the files required for redistributing the application. Distribute these files to users who have MATLAB or MATLAB Runtime installed on their machines. 39 | 40 | * PackagingLog.txt — Log file generated by MATLAB Compiler. 41 | 42 | 43 | ## 2) Set up the target computer (Ubuntu server) 44 | 45 | 1) Make sure Python is installed. Python versions supported are versions 2.7, 3.5, and 3.6. For 64-bit MATLAB, the 64-bit Python version is requiered. 46 | 47 | 2) If necessary install setuptools for python: `sudo apt-get install python-setuptools` 48 | 49 | 3) Now we need to install the Matlab Runtime in the Ubuntu server. Download the it from the website at [Mathworks](https://www.mathworks.com/products/compiler/mcr). Make sure you download and install the version corresponding to the Matlab you used to create the Python package. 50 | 51 | 3) Start the MATLAB Runtime installer. 52 | On Ubuntu: `sudo ./install` 53 | On Debian: `gksudo ./install` 54 | 55 | 4) Follow the installation dialog instructions. 56 | 57 | ### Install the MATLAB Runtime Non-Interactively 58 | 59 | To install the MATLAB Runtime without having to interact with the installer dialog boxes (headless Ubuntu server for instance), use the MATLAB Runtime installer’s non-interactive `silent` mode. The installer runs as a background task and does not display any dialog boxes. 60 | Extract the contents of the MATLAB Runtime installer file to a temporary folder and run the MATLAB Runtime installer, specifying the -mode option: 61 | ``` 62 | ./install -mode silent -agreeToLicense yes 63 | ``` 64 | 65 | ## 3) Create and install the Python application that will use the Matlab function 66 | 67 | To install the Python application, open a command prompt in the `for_redistribution_files_only` folder and run the script: 68 | 69 | ``` 70 | python setup.py install 71 | ``` 72 | 73 | Create a file called getmagic.py, and include the following code in it. 74 | 75 | ```python 76 | import MagicSquarePkg 77 | 78 | myMagic = MagicSquarePkg.initialize() 79 | print(myMagic.makesqr(3)) 80 | myMagic.terminate() 81 | ``` 82 | 83 | At the system command prompt, navigate to the folder containing getmagic.py and run the application as follows: 84 | ``` 85 | python getmagic.py 86 | ``` 87 | 88 | ## Notes 89 | 90 | #### Uninstall MATLAB Runtime 91 | 92 | Exit the application and run the command: 93 | ``` 94 | rm -rf mcr_root 95 | ``` 96 | 97 | where `mcr_root` represents the name of your top-level MATLAB installation folder. 98 | 99 | 100 | ## References 101 | 102 | * https://www.mathworks.com/help/compiler_sdk/gs/create-a-python-application-with-matlab-code.html 103 | * https://www.mathworks.com/help/matlab/matlab-engine-for-python.html 104 | * https://www.mathworks.com/help/compiler/deployment-process.html 105 | -------------------------------------------------------------------------------- /programming-notes/python/msgpack.md: -------------------------------------------------------------------------------- 1 | # MessagePack serializer for Python 2 | 3 | MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. 4 | 5 | ## Install 6 | 7 | ``` 8 | $ pip install msgpack 9 | ``` 10 | 11 | ## Usage 12 | 13 | ```python 14 | import msgpack 15 | import json 16 | import random 17 | 18 | 19 | def msgpack_example_1(): 20 | example_dict = {i: random.random() for i in range(10000)} 21 | 22 | with open('json_file.json', 'w') as f: 23 | json.dump(example_dict, f) 24 | with open('json_file.json') as f: 25 | back_from_json = json.load(f) 26 | 27 | # Saving and loading 28 | with open('msgpack_file.msgpack', 'wb') as f: 29 | # f.write(msgpack.packb(example_dict)) 30 | f.write(msgpack.packb(example_dict, use_single_float=True)) 31 | with open('msgpack_file.msgpack', 'rb') as f: 32 | back_from_msgpack = msgpack.unpackb(f.read()) 33 | 34 | # Data integrity 35 | print(type(next(iter(back_from_json.keys())))) 36 | print(type(next(iter(back_from_msgpack.keys())))) 37 | 38 | 39 | def msgpack_example_2(): 40 | list_of_dicts = [{0: random.random()} for i in range(100)] 41 | with open('streamed.msgpack', 'wb') as f: 42 | for d in list_of_dicts: 43 | f.write(msgpack.packb(d)) 44 | 45 | with open('streamed.msgpack', 'rb') as f: 46 | loaded_list_of_dicts = [item for item in msgpack.Unpacker(f)] 47 | 48 | print(list_of_dicts[3][0], loaded_list_of_dicts[3][0]) 49 | 50 | if __name__ == '__main__': 51 | # msgpack_example_1() 52 | msgpack_example_2() 53 | ``` 54 | -------------------------------------------------------------------------------- /programming-notes/python/mysql.md: -------------------------------------------------------------------------------- 1 | # Using MySQL in Python 2 | 3 | ### Some basic MySQL commands 4 | 5 | In MySQL a `DATABASE` is composed by one or more `TABLE`s. Typically you create a database for each project. 6 | 7 | ##### To create a new MySQL database named `database_name` (the starting point of a new project using MySQL): 8 | 9 | ``` 10 | CREATE DATABASE database_name; 11 | ``` 12 | 13 | Show all databases: `SHOW DATABASES;` 14 | 15 | ##### To start using a database you must: 16 | 17 | `USE database_name;` 18 | 19 | ##### To create a new table named users (an example): 20 | 21 | ``` 22 | CREATE TABLE users(id INT(11) AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100), email VARCHAR(100), username VARCHAR(30), password VARCHAR(100), register_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP); 23 | ``` 24 | 25 | Show all tables: `SHOW TABLES;` 26 | 27 | Get info on a TABLE (in this example users): `DESCRIBE users;` 28 | 29 | ##### Add data to table via command: 30 | 31 | ``` 32 | INSERT INTO users(name, email, username, password) VALUES(x, x, x, x) 33 | ``` 34 | 35 | ##### Typical queries: 36 | 37 | `SELECT * FROM users;` Select all rows from users table 38 | 39 | 40 | ## Copying a database to another server 41 | 42 | 1. One the server where the DB is located make a backup file by running the following command (on Windows it may be necessary to `cd Program Files/MySQL/MySQL Server 5.1/bin` first: 43 | 44 | `mysqldump -u root -p database_name > C:\Users\USER\Desktop\database_name.sql` 45 | 46 | Alternatively: `mysqldump -u[root] -p[password] database_123 > C:\Users\USER\Desktop\database_123.sql` 47 | 48 | If getting an "Access Denied" it message probably means you are outputing to a directory where you have no permission to create files. 49 | 50 | 2. On the second server create a new database using the same database name: 51 | 52 | `mysql -u root -p` to start the MySQL shell 53 | 54 | `CREATE DATABASE database_123;` to create the new DB 55 | 56 | 3. Copy the created backup to the second server and import it: 57 | 58 | `mysql -u[root] -p[password] database_name < database_name.sql` 59 | 60 | 61 | 62 | ## Usage in Python 63 | 64 | #### Configuring in MySQL in Flask: 65 | ```python 66 | # Config MySQL 67 | app.config['MYSQL_HOST'] = 'localhost' 68 | app.config['MYSQL_USER'] = 'root' 69 | app.config['MYSQL_PASSWORD'] = 'mypassword' 70 | app.config['MYSQL_DB'] = 'flaskappdb' 71 | app.config['MYSQL_CURSORCLASS'] = 'DictCursor' # useful to return queries as dictionaries 72 | 73 | # Initialize MySQL 74 | mysql = MySQL(app) 75 | ``` 76 | 77 | Create cursor: `cursor = mysql.connection.cursor()` 78 | 79 | Execute a MySQL command: `cursor.execute( command )` 80 | 81 | Add data: `cursor.execute("INSERT INTO users(name, email, username, password) VALUES(%s, %s, %s, %s)", (x, x, x, x))` 82 | 83 | Commit to database: `mysql.connection.commit()` 84 | 85 | Close: `cursor.close()` 86 | 87 | #### Check DB for a login: 88 | ```python 89 | result = cursor.execute("SELECT * FROM users WHERE username = %s", [username])` 90 | data = cursor.fetchone() 91 | ``` 92 | -------------------------------------------------------------------------------- /programming-notes/python/pandas.md: -------------------------------------------------------------------------------- 1 | # Data manipulation with `pandas` 2 | 3 | `pandas` is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language. 4 | 5 | 6 | ## Instalation 7 | 8 | `pandas`, and other required libraries like `numpy` and the remaining scientific stack, comes pre-installed if you use the Anaconda Python distribution. 9 | 10 | To install via PyPI run: 11 | 12 | ``` 13 | $ pip install pandas 14 | ``` 15 | 16 | ## DataFrame indexing 17 | 18 | See [here](https://lucytalksdata.com/how-to-effectively-index-pandas-dataframes/). 19 | 20 | ## Useful snippets: 21 | 22 | Create a test dataframe: 23 | 24 | ```python 25 | df = pd.DataFrame(np.arange(12).reshape(3, 4), columns=['A', 'B', 'C', 'D']) 26 | 27 | # Output: 28 | # A B C D 29 | # 0 0 1 2 3 30 | # 1 4 5 6 7 31 | # 2 8 9 10 11 32 | ``` 33 | 34 | ### Droping (removing) a column: [^](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html) 35 | 36 | ```python 37 | df = df.drop(columns=['B', 'C']) 38 | 39 | # Output: 40 | # A D 41 | # 0 0 3 42 | # 1 4 7 43 | # 2 8 11 44 | ``` 45 | 46 | ### Copying a dataframe: [^](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.copy.html) 47 | 48 | Note that you can't copy a dataframe simply by `df2 = df`. Here df2 and df refer to the same object and any change in one will affect the other. To create a new copy of a dataframe you do: 49 | 50 | ```python 51 | df2 = df.copy() 52 | ``` 53 | 54 | ### Merge dataframes on common column [^](https://stackoverflow.com/questions/43297589/merge-two-data-frames-based-on-common-column-values-in-pandas) 55 | 56 | ```python 57 | merged_df = pd.merge(df1, df2, on="column_name") 58 | ``` 59 | 60 | Only rows for which common keys are found are kept in both data frames. In case you want to keep all rows from the left data frame and only add values from `df2` where a matching key is available, you can use `how="left"`. 61 | 62 | Alternatively: 63 | 64 | ```python 65 | dfinal = df1.merge(df2, on="column_name", how ='inner') 66 | ``` 67 | 68 | ### Applying a function to each row of a dataframe: 69 | 70 | Use `dataframe.apply()` to run a function on each row, `axis=1`, (or column, `axis=0`) in a dataframe. In the example below we create a new column in the dataframe where the value for each row is the row index + 10. 71 | 72 | ```python 73 | def function1(x): 74 | return x.name + 10 75 | 76 | df['new_column'] = df.apply(lambda row: function1(row), axis=1) 77 | ``` 78 | 79 | 80 | ## Advanced funtions 81 | 82 | ### Collinear variables: [^](https://stackoverflow.com/questions/29294983/how-to-calculate-correlation-between-all-columns-and-remove-highly-correlated-on/61938339) 83 | 84 | The function below finds collinear variables and removes them from dataframe. It also prints a report of the collinear pairs found. 85 | 86 | ```python 87 | def remove_collinear_features(x, threshold): 88 | ''' 89 | Objective: 90 | Remove collinear features in a dataframe with a correlation coefficient greater than the threshold. 91 | Removing collinear features can help a model to generalize and improves the interpretability of the model. 92 | 93 | Inputs: 94 | x: features dataframe 95 | threshold: any features with correlations greater than this value are removed 96 | 97 | Output: 98 | dataframe that contains only the non-highly-collinear features 99 | ''' 100 | 101 | # Calculate the correlation matrix 102 | corr_matrix = x.corr() 103 | iters = range(len(corr_matrix.columns) - 1) 104 | drop_cols = [] 105 | 106 | # Iterate through the correlation matrix and compare correlations 107 | for i in iters: 108 | for j in range(i+1): 109 | item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)] 110 | col = item.columns 111 | row = item.index 112 | val = abs(item.values) 113 | 114 | # If correlation exceeds the threshold 115 | if val >= threshold: 116 | # Print the correlated features and the correlation value 117 | print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2)) 118 | drop_cols.append(col.values[0]) 119 | 120 | # Drop one of each pair of correlated columns 121 | drops = set(drop_cols) 122 | x = x.drop(columns=drops) 123 | 124 | return x 125 | ``` 126 | 127 | ## References 128 | 129 | - [Pandas docs](https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html) 130 | - [Pandas DataFrame Indexing Explained](https://lucytalksdata.com/how-to-effectively-index-pandas-dataframes/) 131 | -------------------------------------------------------------------------------- /programming-notes/python/pathlib.md: -------------------------------------------------------------------------------- 1 | # Working with paths in Python using `pathlib` 2 | 3 | `pathlib` is a Python Standard Library module created to make it easier to work with paths in a file system. This module debuted in Python 3.4 and updated in Python 3.5. 4 | 5 | ## The anatomy of a `pathlib.Path` on Windows 6 | ```python 7 | from pathlib import Path 8 | 9 | path = Path(r'C:/Users/Me/projects/blog/config.tar.gz') 10 | 11 | path.drive 12 | # 'C:' 13 | 14 | path.root 15 | # '/' 16 | 17 | path.root 18 | # 'C:/' 19 | 20 | path.parent 21 | # WindowsPath('C:/Users/Me/projects/blog') 22 | 23 | path.name 24 | # 'config.tar.gz' 25 | 26 | path.stem 27 | # 'config.tar' 28 | 29 | path.suffix 30 | # '.gz' 31 | 32 | path.suffixes 33 | # ['.tar', '.gz'] 34 | ``` 35 | 36 | ## Working with paths 37 | 38 | ### Creating paths and directories 39 | 40 | When we convert a `WindowsPath` to string, Python adds backslashes. `repr` returns the path with forward slashses as it is represented on Windows. 41 | ```python 42 | path = Path(r'C:/Users/Me/projects/blog/config.tar.gz') 43 | 44 | str(path) 45 | # 'C:\\Users\\Me\\projects\\blog\\config.tar.gz' 46 | 47 | repr(path) 48 | # "WindowsPath('C:/Users/Me/projects/blog/config.tar.gz')" 49 | ``` 50 | 51 | Creating and joining paths: 52 | ```python 53 | Path('.', 'projects', 'python', 'source') 54 | # WindowsPath('projects/python/source') 55 | 56 | Path('.', 'projects', 'python') / Path('source') 57 | # WindowsPath('projects/python/source') 58 | 59 | Path('.', 'projects', 'python') / 'source' 60 | # WindowsPath('projects/python/source') 61 | ``` 62 | 63 | Creating directories: 64 | ```python 65 | path = Path('new_directory') 66 | path.mkdir() 67 | 68 | path.exists() 69 | # True 70 | ``` 71 | 72 | When you have a directory path and it already exists, Python raises `FileExistsError` if you call `Path.mkdir()` on it. 73 | 74 | Create parent directories recursively if not exists: 75 | ```python 76 | path = Path('new_parent_dir/sub_dir') 77 | path.mkdir(parents=True) 78 | ``` 79 | 80 | ### List all files and directories 81 | List all files and and directories: 82 | ```python 83 | path = Path('/home/Me/projects/pathlib') 84 | list(path.iterdir()) 85 | ``` 86 | 87 | List only directories: 88 | ```python 89 | [p for p in path.iterdir() if p.is_dir()] 90 | ``` 91 | 92 | List only files: 93 | ```python 94 | [p for p in path.iterdir() if p.is_file()] 95 | ``` 96 | 97 | 98 | ## References 99 | 100 | - [Python pathlib Cookbook: 57+ Examples to Master It (2021)](https://miguendes.me/python-pathlib) 101 | -------------------------------------------------------------------------------- /programming-notes/python/profiling.md: -------------------------------------------------------------------------------- 1 | # Profiling in Python 2 | 3 | Python includes a profiler called [cProfile](https://docs.python.org/3/library/profile.html#module-cProfile). It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations. 4 | 5 | You can call it from within your code, or from the interpreter, like this: 6 | 7 | ```python 8 | import cProfile 9 | cProfile.run('foo()') 10 | ``` 11 | 12 | Even more usefully, you can invoke the cProfile when running a script: 13 | 14 | ```python 15 | python -m cProfile myscript.py 16 | ``` 17 | 18 | And I get this: 19 | ``` 20 | 1007 function calls in 0.061 CPU seconds 21 | 22 | Ordered by: standard name 23 | ncalls tottime percall cumtime percall filename:lineno(function) 24 | 1 0.000 0.000 0.061 0.061 :1() 25 | 1000 0.051 0.000 0.051 0.000 myscript.py:2() 26 | 1 0.005 0.005 0.061 0.061 myscript.py:2() 27 | 1 0.000 0.000 0.061 0.061 {execfile} 28 | 1 0.002 0.002 0.053 0.053 {map} 29 | 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler objects} 30 | 1 0.000 0.000 0.000 0.000 {range} 31 | 1 0.003 0.003 0.003 0.003 {sum} 32 | ``` 33 | -------------------------------------------------------------------------------- /programming-notes/python/project-set-up.md: -------------------------------------------------------------------------------- 1 | # Project set-up 2 | 3 | ## Setting a project with Git, Poetry and Pycharm 4 | 5 | 1) Initialize a Git Repository something like the Github Desktop App or using Git by running: 6 | 7 | ``` 8 | $ git init your_project_name 9 | ``` 10 | 11 | 2) Create a `.gitignore` file and add all relevant exclusions. You can use something like [toptal.com/developers/gitignore](https://www.toptal.com/developers/gitignore). Here is an example `.gitignore` for [Python, Pycharm, and Windows](https://www.toptal.com/developers/gitignore/api/windows,pycharm,jupyternotebooks,python). 12 | 13 | 14 | 3) Set up Poetry: 15 | 16 | ``` 17 | $ cd your_project_name 18 | $ poetry init 19 | ``` 20 | 21 | Follow the prompts to set up your project. This will create a `pyproject.toml` file. 22 | 23 | 4) Install Dependencies: 24 | 25 | For example, to add requests as a dependency: 26 | 27 | ``` 28 | $ poetry add requests 29 | ``` 30 | 31 | To create a Virtual Environment: `$ poetry shell`. 32 | 33 | 5) Pycharm 34 | 35 | Open Pycharm and open the project folder. 36 | Then, configure the Python interpreter by selecting the one create by Poetry. 37 | -------------------------------------------------------------------------------- /programming-notes/python/python-packaging.md: -------------------------------------------------------------------------------- 1 | # Python packaging 2 | 3 | ## Creating a `setup.py` 4 | 5 | Example of a `setup.py` file: 6 | 7 | ```python 8 | from setuptools import find_packages, setup 9 | 10 | with open("README.md", "r", encoding="utf-8") as f: 11 | long_description = f.read() 12 | 13 | setup( 14 | name="marc", 15 | version="22.5.0", 16 | url="https://github.com/maxhumber/marc", 17 | description="Markov chain generator", 18 | long_description=long_description, 19 | long_description_content_type="text/markdown", 20 | author="Max Humber", 21 | author_email="max.humber@gmail.com", 22 | license="MIT", 23 | classifiers=[ 24 | "Development Status :: 5 - Production/Stable", 25 | "License :: OSI Approved :: MIT License", 26 | "Programming Language :: Python :: 3", 27 | ], 28 | packages=[""], 29 | package_dir={"": "python/src"}, 30 | extras_require={ 31 | "dev": [ 32 | "black>=22.3.0", 33 | ], 34 | }, 35 | python_requires=">=3.9", 36 | setup_requires=["setuptools>=62.1.0"], 37 | ) 38 | ``` 39 | 40 | ## Structure of package with CLI scripts 41 | 42 | Example structure: 43 | 44 | ``` 45 | my_package/ 46 | my_package/ 47 | __init__.py 48 | cli_scripts.py 49 | setup.py 50 | ``` 51 | 52 | Let's assume your `__init__.py` looks like this (as a side note, I'd recommend moving the classes defined in there to a separate file, and then simply importing that file in the `__init__`): 53 | 54 | ```python 55 | class Test(object): 56 | 57 | def __init__(self, a) 58 | self.a = a 59 | 60 | def __call__(self): 61 | print(self.a) 62 | ``` 63 | 64 | Now there is an additional file inside the package that utilizes the stuff you implemented in the package, let's call it `cli_scripts.py`: 65 | 66 | ```python 67 | import argparse 68 | 69 | from my_package import Test 70 | 71 | def parse_args(): 72 | parser = argparse.ArgumentParser() 73 | parser.add_argument("a", type=int, help="Just an example argument") 74 | return parser.parse_args() 75 | 76 | def demonstration(): 77 | args = parse_args() 78 | t = Test(a=args.a) 79 | t() 80 | ``` 81 | My suggestion now is to utilize the `console_scripts` entry point inside `setup.py`, which could now look something like this: 82 | 83 | ```python 84 | from setuptools import setup 85 | 86 | setup( 87 | name='testpy', 88 | version='0.1', 89 | description='Just an example', 90 | author='RunOrVeith', 91 | author_email='xyz@mailinator.com', 92 | packages=["my_package"], 93 | entry_points={ 94 | "console_scripts": ["mypkg=my_package.cli_scripts:demonstration"]}, 95 | install_requires=[], 96 | ) 97 | ``` 98 | Now when you run `pip install .` inside the top-level `my_package` folder, your package will be installed. The `entry_points` automatically generate an executable for you, that you can now call with the name you gave it inside `setup.py`, in this example `mypkg`. This means you can now run `mypkg 5` and this will call `demonstration()`. 99 | 100 | This way: 101 | 102 | - you do not need to handle any `__main__` files 103 | - you can give your script a meaningful name 104 | - you don't need to care whether your script is executable, or specify to run the script with python 105 | - you can have as many of these as you want inside the list `entry_points` 106 | - it forces you to write functions that you could also call from other modules, instead of the `__main__` 107 | -------------------------------------------------------------------------------- /programming-notes/python/python-version.md: -------------------------------------------------------------------------------- 1 | # Working with the Python version 2 | 3 | To check the Python version installed, open a terminal window and entering the following: 4 | ``` 5 | python ––version 6 | ``` 7 | 8 | ### Check for a minimum required Python version 9 | You can check for the Python version in your code, to make sure your users are not running your script with an incompatible version. Use this simple check: 10 | ```python 11 | if not sys.version_info > (2, 7): 12 | # berate your user for running a 10 year 13 | # python version 14 | elif not sys.version_info >= (3, 5): 15 | # Kindly tell your user (s)he needs to upgrade 16 | # because you're using 3.5 features 17 | ``` 18 | 19 | 20 | ### Check if 32 or 64 bit version 21 | 22 | ```python 23 | if sys.maxsize > 2**32: 24 | print('64 bit Python version') 25 | else: 26 | print('32 bit Python version') 27 | ``` 28 | -------------------------------------------------------------------------------- /programming-notes/python/resources/data-science-libs.md: -------------------------------------------------------------------------------- 1 | # Python Libraries for Data Science and Data Viz 2 | 3 | ## General use 4 | - [numpy](https://numpy.org/) 5 | - [pandas](https://pandas.pydata.org/) 6 | 7 | 8 | ## Specific 9 | - [scikit-learn](https://scikit-learn.org/stable/) - machine learning framework 10 | - [PyTorch](https://pytorch.org/) - machine learning framework 11 | - [keras](https://keras.io/) - deep learning (e.g., neural networks) 12 | - [TensorFlow](https://www.tensorflow.org/install/pip) - machine learning framework 13 | - [DEAP](https://deap.readthedocs.io/en/master/) - evolutionary computation framework (e.g., genetic algorithms) 14 | - [pydantic](https://pydantic-docs.helpmanual.io/) - data validation and settings management using python type annotations 15 | 16 | ## Data visualization 17 | 18 | ### Plotting libraries 19 | - [Matplotlib](https://matplotlib.org/): Python de facto 2D plotting library. 20 | - [seaborn](http://seaborn.pydata.org/): Based on matplotlib, it provides a high-level interface for drawing attractive and informative statistical graphics. 21 | - [Altair](https://altair-viz.github.io/): A declarative statistical visualization library for Python, based on Vega and Vega-Lite 22 | 23 | 24 | ### Web ready 25 | - [Bokeh ](https://docs.bokeh.org/en/latest/): Creates interactive, web-ready plots, which can be easily output as JSON objects, HTML documents, or interactive web applications. 26 | - [Plotly](https://plot.ly/python/): Makes interactive plots, but it offers some charts you won't find in most libraries, like contour plots, dendograms, and 3D charts. 27 | 28 | ### Geo 29 | - [Folium](https://python-visualization.github.io/folium/): Folium makes it easy to visualize data that’s been manipulated in Python on an interactive [`leaflet.js`](https://leafletjs.com/) map. 30 | - [Basemap](https://matplotlib.org/basemap/users/index.html): A matplotlib toolkit library for plotting 2D data on maps in Python. 31 | -------------------------------------------------------------------------------- /programming-notes/python/resources/hosting.md: -------------------------------------------------------------------------------- 1 | # Hosting scripts and services 2 | 3 | This list is mainly focused on Python. 4 | 5 | ## Hosting options: 6 | 7 | ### Free: 8 | - [Render](https://render.com) - disconects after 15 minutes of inactivity 9 | - [Railway](https://railway.app) 10 | - [Cyclic](https://cyclic.sh) - Javascript only 11 | - [Google App Engine](https://cloud.google.com/appengine/) (Free Tier) 12 | - [Amazon EC2](https://aws.amazon.com/ec2) (Free Tier) 13 | - [Python Anywhaere](https://www.pythonanywhere.com) (Free Tier) 14 | 15 | ### Paid: 16 | - [Heroku](https://www.heroku.com/) (7$/month Min) 17 | - [Digital Ocean](https://www.digitalocean.com) (5$/month Min, 50$ credit from Github Student Pack) 18 | - [Linode](https://www.linode.com/) (5$/month Min) 19 | 20 | ## Useful Software 21 | - [PuTTY](https://www.putty.org/) - for SSH access 22 | - [WinSCP](https://winscp.net/eng/index.php) - To transfer files to the server 23 | - [FileZilla](https://filezilla-project.org/) - An alternative to WinSCP 24 | 25 | ## Useful references 26 | [Running python scripts on a VPS](https://nikolak.com/deploying-python-code-to-vps/) 27 | -------------------------------------------------------------------------------- /programming-notes/python/resources/tutorials.md: -------------------------------------------------------------------------------- 1 | # Tutorials on various Python subjects 2 | 3 | #### Legend: 4 | - :blue_book: - Book (usually freely available). 5 | - :school: - Course material, usually provided by an University. 6 | - :newspaper: - Written tutorial. May be a blog, article, post or even a PDF document. 7 | - :tv: - Video tutorials. May be a playlist or a single video. 8 | 9 | 10 | ## Python language workings (general) 11 | 12 | - :tv: :blue_book: [Automate the Boring Stuff with Python](https://www.youtube.com/playlist?list=PL0-84-yl1fUnRuXGFe_F7qSH1LEnn9LkW) by Al Sweigart 13 | - :tv: [Python OOP Tutorials - Working with Classes](https://www.youtube.com/playlist?list=PL-osiE80TeTsqhIuOqKhwlXsIBIdSeYtc) by Corey Schafer 14 | - :tv: [Python Tutorials](https://www.youtube.com/playlist?list=PL-osiE80TeTt2d9bfVyTiXJA-UTHn6WwU) by Corey Schafer 15 | - :newspaper: [Full Stack Python](https://www.fullstackpython.com/blog.html) 16 | 17 | ## Python Libraries 18 | 19 | - :newspaper: [The Most Detailed Selenium WebDriver Tutorial With Python](https://www.lambdatest.com/blog/selenium-webdriver-with-python/?utm_source=Reddit&utm_medium=blog&utm_campaign=RP-011229-1&utm_term=OrganicPosting) by Himanshu Sheth 20 | - :newspaper: [PyQt Layouts: Create Professional-Looking GUI Applications](https://realpython.com/python-pyqt-layout/) by Real Python 21 | 22 | ## Data analysis 23 | Libraries like `Numpy`, `Pandas`, and `Matplotlib`. 24 | 25 | - :newspaper: [Python NumPy For Your Grandma](https://www.gormanalysis.com/blog/python-numpy-for-your-grandma/) by Ben Gorman 26 | - :newspaper: [Python Pandas For Your Grandpa](https://www.gormanalysis.com/blog/python-pandas-for-your-grandpa/) by Ben Gorman 27 | - :tv: [Data analysis in Python with pandas](https://www.youtube.com/playlist?list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y) by Data School 28 | - :tv: [Matplotlib Tutorial Series - Graphing in Python](https://www.youtube.com/playlist?list=PLQVvvaa0QuDfefDfXb9Yf0la1fPDKluPF) by sentdex 29 | 30 | 31 | ## Data Science and Machine Learning 32 | General Data Science theory as well as working with tools and frameworks like `scikit-learn`, `Keras` and `TensorFlow`. 33 | 34 | - :tv: [Intro to Data Science](https://www.youtube.com/playlist?list=PLMrJAkhIeNNQV7wi9r7Kut8liLFMWQOXn) by Steve Brunton 35 | - :school: :tv: [CS50's Introduction to Artificial Intelligence with Python 2020](https://www.youtube.com/playlist?list=PLhQjrBD2T382Nz7z1AEXmioc27axa19Kv) by Harvard 36 | - :tv: [Keras - Python Deep Learning Neural Network API](https://www.youtube.com/playlist?list=PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL) by deeplizard 37 | - :tv: [Machine Learning Basics](https://www.youtube.com/playlist?list=PLR0bgGon_WTJnvH92Ls_Vj4ueCi98PDW5) by Vinsloev Academy 38 | - :tv: [Neural Networks Explained from Scratch using Python](https://www.youtube.com/watch?v=9RN2Wr8xvro) by Bot Academy 39 | 40 | 41 | ## Web frameworks 42 | Mostly `Django` and `Flask`. 43 | 44 | - :tv::newspaper: [Python Flask From Scratch](https://www.youtube.com/playlist?list=PLillGF-RfqbbbPz6GSEM9hLQObuQjNoj_) by Traversy Media 45 | - :tv: [Flask Tutorial](https://www.youtube.com/playlist?list=PLei96ZX_m9sWQco3fwtSMqyGL-JDQo28l) by Chris Hawkes 46 | - :tv: [The Flask Mega-Tutorial](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world) by Miguel Grinberg 47 | - :tv: [Practical Flask Web Development Tutorials](https://www.youtube.com/playlist?list=PLQVvvaa0QuDc_owjTbIY4rbgXOFkUYOUB) by sentdex 48 | - :tv: [Flask Web Development with Python Tutorials](https://www.youtube.com/playlist?list=PL6gx4Cwl9DGDi9F_slcQK7knjtO8TUvUs) by thenewboston 49 | - :tv: [Django Tutorials](https://www.youtube.com/playlist?list=PL-osiE80TeTtoQCKZ03TU5fNfx2UY6U4p) by Corey Schafer 50 | - :tv: [The Best Django Tutorial](https://www.youtube.com/playlist?list=PLei96ZX_m9sWlZ9pgnJ6eix76lffAZ2_0) by Chris Hawkes 51 | - :tv: [Building a Website with Django](https://www.youtube.com/playlist?list=PLei96ZX_m9sV5865IlH-RktGZXYW7hQTB) by Chris Hawkes 52 | - :tv: [Django Girls Tutorial](https://tutorial.djangogirls.org/en/index.html) by Django Girls 53 | - :newspaper: [A Complete Beginner's Guide to Django](https://simpleisbetterthancomplex.com/series/beginners-guide/1.11/) by Vitor Freitas 54 | 55 | 56 | ## How To's 57 | Tutorials on building croncrete projects. 58 | 59 | - :newspaper: [Monitor your internet with python](https://pythonprogramming.org/monitor-your-internet-with-python/) by Joseph 60 | - :newspaper: [How to build your own Python Voice Assistant](https://thecodingpie.com/post/how-to-build-your-own-python-voice-assistant-thecodingpie/) by thecodingpie 61 | 62 | -------------------------------------------------------------------------------- /programming-notes/python/resources/web-dev.md: -------------------------------------------------------------------------------- 1 | # Pure Python web development 2 | 3 | ## 1) Fully-featured production grade frameworks 4 | 5 | ### [NiceGUI](https://nicegui.io/) 6 | 7 | - Built on FastAPI. 8 | - Clean and well-documented API. 9 | - Not just limited to web applications. The 1.12 release has made it possible to have desktop apps as well. 10 | 11 | ### [Datapane](https://docs.datapane.com/) 12 | 13 | - Can be embedded in Flask or Django 14 | 15 | ### [ReactPy](https://reactpy.dev/docs/index.html) 16 | 17 | - Official support for the following backends: Flask, FastAPI, Sanic, Tornado, Django, Jupyter, Plotly-Dash 18 | 19 | ### [Dash](https://dash.plotly.com/) 20 | 21 | - Built on plotly and ReactJS. 22 | 23 | ### [Flet](https://flet.dev/) 24 | 25 | - Build Flutter apps in Python. Flutter is an open source framework by Google for building beautiful, natively compiled, multi-platform applications from a single codebase. 26 | 27 | ### [Pynecone](https://pynecone.io/) 28 | 29 | - Compiles to a traditional React (NextJS flavor) app. 30 | - Excellent scalability. 31 | - Uses FastAPI. 32 | 33 | 34 | 35 | ## 2) Limited-scope frameworks 36 | 37 | ### [Streamlit](https://streamlit.io/) 38 | 39 | - Very powerful and very popular. 40 | 41 | ### [Gradio](https://gradio.app/) 42 | 43 | - Gradio can wrap almost any Python function with an easy-to-use user interface. 44 | 45 | ### [Shiny for Python](https://shiny.rstudio.com/py/) 46 | 47 | - Clean API and good documentation. 48 | - Lacks the features for complete web applications like authentication, authorization and sessioning. 49 | 50 | 51 | 52 | ## References 53 | 54 | - [Pure Python Web Development](https://metaperl.github.io/pure-python-web-development/intro.html) 55 | -------------------------------------------------------------------------------- /programming-notes/python/schedule.md: -------------------------------------------------------------------------------- 1 | # Schedule task in Python 2 | 3 | ## Using the `schedule` package 4 | 5 | Schedule lets you run Python functions (or any other callable) periodically at pre-determined intervals using a simple syntax. 6 | 7 | ### Install 8 | 9 | ``` 10 | $ pip install schedule 11 | ``` 12 | 13 | 14 | ### Usage 15 | 16 | ```python 17 | import time 18 | import schedule 19 | 20 | def test_function(): 21 | print(f'test called at {time.time()}') 22 | 23 | def test_function_2(): 24 | print(f'test 2 called at {time.time()}') 25 | 26 | if __name__ == '__main__': 27 | schedule.every(1).seconds.do(test_function) 28 | schedule.every(3).seconds.do(test_function_2) 29 | # schedule.every(1).days.do(daily_task) 30 | # schedule.every().thursday.at("10:00").do(day_time_task) 31 | 32 | while True: 33 | schedule.run_pending() 34 | ``` 35 | 36 | ## References 37 | 38 | - [schedule](https://schedule.readthedocs.io/en/stable/) 39 | -------------------------------------------------------------------------------- /programming-notes/python/scientific-figures.md: -------------------------------------------------------------------------------- 1 | # Scientific quality figures 2 | 3 | ## 1) Using `SciencePlots` 4 | 5 | [SciencePlots](https://github.com/garrettj403/SciencePlots/) is a Python pakage that provides Matplotlib styles for scientific figures. Install with: 6 | 7 | ``` 8 | $ pip install SciencePlots 9 | ``` 10 | 11 | To use, simple add the import and select the style: 12 | 13 | ```python 14 | import matplotlib.pyplot as plt 15 | import scienceplots 16 | 17 | plt.style.use('science') 18 | ``` 19 | 20 | To use any of the styles temporarily, you can use a constext manager: 21 | 22 | ```python 23 | with plt.style.context('science'): 24 | plt.figure() 25 | plt.plot(x, y) 26 | plt.show() 27 | ``` 28 | 29 | More information and available styles can be found [here](https://github.com/garrettj403/SciencePlots/). 30 | 31 | 32 | ## 2) Using a predifined stylesheet 33 | 34 | You can create and install a custom Matplotlib style. Install the styles by copying the respective `*.mplstyle` file into your Matplotlib style directory. To check where this is, in an interactive python console type: 35 | 36 | ```python 37 | import matplotlib 38 | print(matplotlib.get_configdir()) 39 | ``` 40 | 41 | You should get back something like `/home/johndoe/.matplotlib`. You can then put the `*.mplstyle` file in `/home/johndoe/.matplotlib/stylelib/` (you may need to create the stylelib directory). Here is a base configuration for a scientific style: 42 | 43 | ```mplstyle 44 | # Matplotlib style for scientific plotting 45 | # Base style 46 | 47 | # Set color cycle: blue, green, yellow, red, violet, gray 48 | axes.prop_cycle : cycler('color', ['0C5DA5', '00B945', 'FF9500', 'FF2C00', '845B97', '474747', '9e9e9e']) 49 | 50 | # Set default figure size 51 | figure.figsize : 3.5, 2.625 52 | 53 | # Set x axis 54 | xtick.direction : in 55 | xtick.major.size : 3 56 | xtick.major.width : 0.5 57 | xtick.minor.size : 1.5 58 | xtick.minor.width : 0.5 59 | xtick.minor.visible : True 60 | #xtick.top : True 61 | 62 | # Set y axis 63 | ytick.direction : in 64 | ytick.major.size : 3 65 | ytick.major.width : 0.5 66 | ytick.minor.size : 1.5 67 | ytick.minor.width : 0.5 68 | ytick.minor.visible : True 69 | #ytick.right : True 70 | 71 | # Set line widths 72 | axes.linewidth : 0.5 73 | grid.linewidth : 0.5 74 | lines.linewidth : 1. 75 | 76 | # Remove legend frame 77 | legend.frameon : False 78 | 79 | # Always save as 'tight' 80 | savefig.bbox : tight 81 | savefig.pad_inches : 0.05 82 | 83 | # Use serif fonts 84 | font.serif : cmr10, Computer Modern Serif, DejaVu Serif 85 | font.family : serif 86 | axes.formatter.use_mathtext : True 87 | mathtext.fontset : cm 88 | 89 | # Use LaTeX for math formatting 90 | text.usetex : True 91 | text.latex.preamble : \usepackage{amsmath} \usepackage{amssymb} 92 | ``` 93 | 94 | You can then use the styles in a similar fashion to that shown above. The following will apply the style globally: 95 | 96 | ```python 97 | plt.style.use('science') 98 | ``` 99 | 100 | You can also combine multiple styles by: 101 | 102 | ```python 103 | plt.style.use(['science', 'grid']) 104 | ``` 105 | 106 | To use styles on a figure by figure basis, use a contect manager: 107 | 108 | ```python 109 | with plt.style.context('science'): 110 | plt.figure() 111 | plt.plot(x, y) 112 | plt.show() 113 | ``` 114 | 115 | More styles [here](https://gist.github.com/jAniceto/e58cf2738b970de54910d886d91093ee). 116 | 117 | 118 | ## 3) Additional usefull formating 119 | 120 | 121 | ```python 122 | import matplotlib.pyplot as plt 123 | from matplotlib.offsetbox import AnchoredText 124 | 125 | with plt.style.context('science'): 126 | fig, ax = plt.subplots() 127 | 128 | ax = plot(x, y) 129 | 130 | # Labels 131 | ax.set_xlabel(r'$D_{12}$ $\mathrm{(cm^2 s^{-1})}$') # x-axis label using LaTeX 132 | ax.set_ylabel(r'Y axis label') # y-axis label using LaTeX 133 | 134 | # Ticks and grids 135 | ax.grid(False) # Remove grid 136 | ax.set_xlim([1, 11]) # Set x-axis limits 137 | ax.set_ylim([6.5, 8.5]) # Set y-axis limits 138 | 139 | ax.ticklabel_format(style='sci', axis='x', scilimits=(0,0)) # Set x-axis ticks to scientific notation 140 | ax.ticklabel_format(style='sci', axis='both', scilimits=(0,0)) # Set both axis ticks to scientific notation 141 | 142 | # Add a text box to a corner of the graph with e.g. 'a)' 143 | anchor_text = AnchoredText('a)', pad=0, loc='upper right', frameon=False, prop=dict(fontweight="bold", fontsize=12, fontfamily='serif')) 144 | ax.add_artist(anchor_text) 145 | 146 | plt.tight_layout() 147 | 148 | fig.set_size_inches(5.25, 3.95) # Set figure size in inches 149 | 150 | fig.savefig('fig1.png', dpi=300, bbox_inches='tight') # Save figure in png 151 | fig.savefig('fig1.pdf', dpi=300, bbox_inches='tight') # Save figure in pdf 152 | ``` 153 | 154 | -------------------------------------------------------------------------------- /programming-notes/python/seaborn.md: -------------------------------------------------------------------------------- 1 | # Seaborn 2 | 3 | ## Change overal asthetics 4 | 5 | Scaling plot elements, like line widths. In increasing size: 6 | 7 | ```python 8 | sns.set_context("paper") 9 | sns.set_context("notebook") # default 10 | sns.set_context("talk") 11 | sns.set_context("poster") 12 | ``` 13 | 14 | Scaling overall text size: 15 | 16 | ```python 17 | sns.set(font_scale=2) 18 | ``` 19 | 20 | ### References: 21 | * [https://seaborn.pydata.org/tutorial/aesthetics.html](https://seaborn.pydata.org/tutorial/aesthetics.html) 22 | -------------------------------------------------------------------------------- /programming-notes/python/sphinx.md: -------------------------------------------------------------------------------- 1 | # Creating documentation with Sphinx 2 | 3 | ## Set up 4 | 5 | Install Sphinx: 6 | 7 | ``` 8 | pip install sphinx 9 | ``` 10 | 11 | Generate the basic structure of Sphinx documentation. It is usually recommended to separate source and build directories. 12 | 13 | ``` 14 | sphinx-quickstart docs 15 | ``` 16 | 17 | Now we can build the HTML documentation with: 18 | 19 | ``` 20 | cd docs/ 21 | .\make.bat html 22 | ``` 23 | 24 | Note: `.\make.bat html` is for Powershell. `make html` for macOS, Linux or Windows command prompt 25 | 26 | Serve with Python built-in HTTP server: 27 | 28 | ``` 29 | python -m http.server 30 | ``` 31 | 32 | ## Using Markdown with Sphinx 33 | 34 | Convert RST files to Markdown: 35 | 36 | ``` 37 | pip install "rst-to-myst[sphinx]" 38 | rst2myst convert docs/**/*.rst 39 | ``` 40 | 41 | We can now delete the index.rst file. 42 | 43 | In the Sphinx `config.py` file we now need to add the following extension: 44 | 45 | ``` 46 | extensions = [ 47 | "myst_parser", 48 | ] 49 | ``` 50 | 51 | and install it: 52 | 53 | ``` 54 | pip install myst-parser 55 | ``` 56 | 57 | Now we can build the HTML from the Markdown files. 58 | 59 | ``` 60 | cd docs/ 61 | make html 62 | python -m http 63 | ``` 64 | 65 | ## Useful configurations 66 | 67 | To monitor changes in the docs and rebuild the HTML we can use `sphinx-autobuild`: 68 | 69 | ``` 70 | pip install sphinx-autobuild 71 | sphinx-autobuild docs/source/ docs/build/html 72 | ``` 73 | 74 | If you wish to include the README.md file in the docs, in the docs/source/index.md add the following directive: 75 | 76 | 77 | ```{include} ../../README.md 78 | :relative-images: 79 | ``` 80 | 81 | To add a warning: 82 | 83 | ```{warning} 84 | Warning text. 85 | ``` 86 | 87 | Other handy extensions: 88 | 89 | ``` 90 | extensions = [ 91 | 'myst_parser', 92 | 'sphinx.ext.duration', 93 | 'sphinx.ext.autosectionlabel', 94 | ] 95 | ``` 96 | 97 | Install another theme: 98 | 99 | ``` 100 | pip install furo 101 | ``` 102 | 103 | In the Sphinx `config.py`: 104 | 105 | ``` 106 | html_theme = 'furo' 107 | ``` 108 | 109 | Create a new page (`usage.md`) next to the `index.md` file. Link to a page: 110 | 111 | ``` 112 | Check page {doc}`usage`. 113 | Specifically the section {ref}`Instalation`. 114 | ``` 115 | 116 | ## Creating module pages automatically 117 | 118 | ```rst 119 | Joback Method 120 | ============= 121 | 122 | .. automodule:: chem_eng_kit.properties.joback 123 | :members: 124 | :exclude-members: _sum_group_property 125 | ``` 126 | 127 | ## Creating docstrings 128 | 129 | ### Code blocks 130 | 131 | ```python 132 | def foo(): 133 | ''' 134 | .. highlight:: python 135 | .. code-block:: python 136 | 137 | res = aFunction(something, goes, in) 138 | print(res.avalue) 139 | ''' 140 | ``` 141 | 142 | Inline code in reStructuredText: 143 | 144 | ```python 145 | def foo(): 146 | ''' 147 | bla `foo()` bla 148 | ''' 149 | ``` 150 | 151 | Inline code in Markdown: 152 | 153 | ```python 154 | def foo(): 155 | ''' 156 | bla ``foo()`` bla 157 | ''' 158 | ``` 159 | 160 | ### Math 161 | 162 | Displayed math: 163 | 164 | ```python 165 | def foo(): 166 | ''' 167 | .. math:: V_c [cm^3/mol] = 17.5 + \\sum V_{c,i} 168 | ''' 169 | ``` 170 | 171 | Inline math: 172 | 173 | ```python 174 | def foo(): 175 | ''' 176 | where :math:`N_{atom}` is the number of atoms 177 | ''' 178 | ``` 179 | -------------------------------------------------------------------------------- /programming-notes/python/store-dicts.md: -------------------------------------------------------------------------------- 1 | # Storing dictionaries for later use: JSON and Pickle 2 | 3 | ## JSON 4 | 5 | **Saving:** 6 | ```python 7 | import json 8 | with open('data.json', 'w') as fp: 9 | json.dump(data, fp) 10 | ``` 11 | 12 | Supply extra arguments like sort_keys or indent to get a pretty result. The argument sort_keys will sort the keys alphabetically and indent will indent your data structure with indent=N spaces. 13 | ```python 14 | json.dump(data, fp, sort_keys=True, indent=4) 15 | ``` 16 | 17 | **Loading:** 18 | ```python 19 | with open('data.json', 'r') as fp: 20 | data = json.load(fp) 21 | ``` 22 | 23 | #### json.dump() vs json.dumps(), json.load() vs json.loads() 24 | 25 | If you want to dump the JSON into a file/socket or whatever, then you should go for `dump()`. If you only need it as a string (for printing, parsing or whatever) then use `dumps()` (dump string). The functions with an `s` take string parameters. The others take file streams. 26 | 27 | --- 28 | 29 | ## Pickle 30 | 31 | **Saving:** 32 | ```python 33 | import cPickle as pickle 34 | with open('data.p', 'wb') as fp: 35 | pickle.dump(data, fp) 36 | ``` 37 | 38 | **Loading:** 39 | ```python 40 | with open('data.p', 'rb') as fp: 41 | data = pickle.load(fp) 42 | ``` 43 | -------------------------------------------------------------------------------- /programming-notes/python/stravalib.md: -------------------------------------------------------------------------------- 1 | # Interfacing with Strava API using stavalib 2 | 3 | ### Instalation: 4 | `$ pip install stravalib` 5 | 6 | ### Usage: 7 | 8 | ```python 9 | from stravalib import Client 10 | 11 | client = Client(access_token='fgd456fgs5dgs546dfg') 12 | athlete = client.get_athlete() # Get your athlete profile 13 | athlete2 = client.get_athlete(227615) # By providing an athlete ID you can access other people 14 | ``` 15 | 16 | To get a given activity, use the get_activity function and provide activity_id: 17 | ```python 18 | activity = client.get_activity(207650614) 19 | 20 | # Activity object has many basic properties such as type and distance. 21 | print("type={0.type} distance={1} km".format(activity, unithelper.kilometers(activity.distance))) 22 | ``` 23 | 24 | Activity information: 25 | ```python 26 | # Activities can have many streams, you can request desired stream types 27 | types = ['time', 'latlng', 'altitude', 'heartrate', 'temp', ] 28 | 29 | streams = client.get_activity_streams(123, types=types, resolution='medium') 30 | 31 | # Result is a dictionary object. The dict's key are the stream type. 32 | if 'altitude' in streams.keys(): 33 | print(streams['altitude'].data) 34 | ``` 35 | 36 | List of Activities: 37 | ```python 38 | for activity in client.get_activities(after = "2010-01-01T00:00:00Z", limit=5): # To get newest to oldest use before argument. 39 | print("{0.name} {0.moving_time}".format(activity)) 40 | ``` 41 | 42 | ##### Official documentation 43 | * [Strava API Docs](http://strava.github.io/api/) 44 | * [stravalib Docs](http://pythonhosted.org/stravalib/index.html) 45 | -------------------------------------------------------------------------------- /programming-notes/python/streamlit.md: -------------------------------------------------------------------------------- 1 | # Streamlit 2 | 3 | ## Instalation 4 | ``` 5 | $ pip install streamlit 6 | ``` 7 | 8 | ## Run 9 | ``` 10 | $ streamlit run myfile.py 11 | ``` 12 | 13 | ## References 14 | 15 | - [Streamlit Docs](https://docs.streamlit.io/library/get-started) 16 | - [Streamlit Cheat Sheet](https://docs.streamlit.io/library/cheatsheet) 17 | -------------------------------------------------------------------------------- /programming-notes/python/uv.md: -------------------------------------------------------------------------------- 1 | # Working with uv 2 | 3 | `uv` is a Python package and project manager. It is extremely fast, compared with alternatives. 4 | 5 | 6 | ## Installing 7 | 8 | On Windows, you can install `uv` using Powershell: 9 | 10 | ``` 11 | $ powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" 12 | ``` 13 | 14 | More info on other installation options can be found on the [uv docs](https://docs.astral.sh/uv/getting-started/installation/). 15 | 16 | To update uv: 17 | 18 | ``` 19 | $ uv self update 20 | ``` 21 | 22 | 23 | ## Starting a new project 24 | 25 | Starting a project with `uv`: 26 | 27 | ``` 28 | $ uv init project_name 29 | ``` 30 | 31 | `uv` creates a new directory called `project_name` and sets up a few project files. You can then enter the project folder. 32 | 33 | ``` 34 | $ cd project_name 35 | ``` 36 | 37 | Note: If you already have a project folder you can ask `uv` to initialize the project there: 38 | 39 | ``` 40 | $ cd project_name 41 | $ uv init 42 | ``` 43 | 44 | When starting a new project you can specify the Python version using: 45 | 46 | ``` 47 | $ uv init project_name --python 3.11 48 | ``` 49 | 50 | ## Installing packages 51 | 52 | To add packages: 53 | 54 | ``` 55 | $ cd project_name 56 | $ uv add requests flask 57 | ``` 58 | 59 | To remove packages: 60 | 61 | ``` 62 | $ uv remove flask 63 | ``` 64 | 65 | Working with development dependencies: 66 | 67 | ``` 68 | $ uv add --dev black 69 | $ uv run black path\to\file.py 70 | ``` 71 | 72 | 73 | ## Running scripts 74 | 75 | To run python files in you environment use: 76 | 77 | ``` 78 | $ uv run example.py 79 | ``` 80 | 81 | 82 | ## Working with tools 83 | 84 | Tools are pacakges that can perform several functions but are not part of your project. For instance, it is common to use a linter/formatter when developing (e.g., `ruff`). With `uv` you can use tools like `ruff` in different ways. 85 | 86 | ### Ways to use tools 87 | 88 | 1) Running a tool without installing it (it's installed in a temporary environment and deleted after use). 89 | ``` 90 | $ uv tool run ruff check 91 | ``` 92 | The following alias can also be used and results in the same: 93 | ``` 94 | $ uvx ruff check 95 | ``` 96 | 97 | 2) When a tool is used frequently it may be usefull to install it to a persistent environment and add it to the `PATH`. 98 | ``` 99 | $ uv tool install ruff 100 | ``` 101 | Now, whatever the project you are wirking in, you can run `ruff` by doing: 102 | ``` 103 | $ ruff check 104 | ``` 105 | 106 | 3) You can also install the tool in your project as a development dependency 107 | ``` 108 | $ uv add --dev ruff 109 | ``` 110 | And run it with 111 | ``` 112 | $ uv run ruff check 113 | ``` 114 | 115 | ### Upgrading tools 116 | 117 | Upgrade all tools with: 118 | ``` 119 | $ uv tool upgrade --all 120 | ``` 121 | 122 | Or a single tool with: 123 | ``` 124 | $ uv tool upgrade ruff 125 | ``` 126 | 127 | 128 | 129 | ## Working with `jupyter` 130 | 131 | If you're working within a project, you can start a Jupyter server with access to the project's virtual environment by: 132 | ``` 133 | $ uv run --with jupyter jupyter lab 134 | ``` 135 | 136 | Alternatively you can install Jupyter as a dev dependency in your project 137 | ``` 138 | $ uv add --dev jupyterlab 139 | $ uv run jupyter lab 140 | ``` 141 | 142 | 143 | ## Project entry points and command line interfaces 144 | 145 | To create a project CLI you need to configure entry point tables in the `pyproject.toml` and add a build system. For example, to declare a command called `hello` that invokes the `hello` function in the `example` module: 146 | 147 | In the `pyproject.toml` file: 148 | 149 | ```toml 150 | [build-system] 151 | requires = ["setuptools>=61.0", "wheel"] 152 | build-backend = "setuptools.build_meta" 153 | 154 | [project.scripts] 155 | hello = "example:hello" 156 | ``` 157 | 158 | You can then run the command with: 159 | ``` 160 | $ uv run hello 161 | ``` 162 | 163 | 164 | ## Deploying 165 | 166 | Deploying the project to production excluding dev dependencies: 167 | 168 | ``` 169 | $ uv sync --no-dev 170 | ``` 171 | 172 | ## References 173 | 174 | - [uv Documentation](https://docs.astral.sh/uv/) 175 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "programming-notes" 3 | version = "0.1.0" 4 | description = "Add your description here" 5 | readme = "README.md" 6 | requires-python = ">=3.11" 7 | dependencies = [ 8 | "mkdocs>=1.6.1", 9 | "mkdocs-awesome-nav>=3.1.2", 10 | "mkdocs-material>=9.6.14", 11 | ] 12 | --------------------------------------------------------------------------------