├── pydevtips ├── __init__.py ├── utils.py └── fftconvolve.py ├── examples ├── configs │ ├── signal │ │ ├── ExampleNumpy.yaml │ │ ├── ExampleZeros.yaml │ │ ├── transform │ │ │ └── power.yaml │ │ └── ExampleCustom.yaml │ ├── exp1.yaml │ └── defaults.yaml ├── joblib_parallel.py ├── cupy_fft.py ├── real_convolve.py └── numpy_speedup.py ├── docs ├── requirements.txt ├── images │ └── vscode_jupyter_notebook.png ├── source │ ├── utils.rst │ ├── index.rst │ ├── version_control.rst │ ├── badges.rst │ ├── fftconvolve.rst │ ├── remote_development.rst │ ├── conf.py │ ├── virtual_env.rst │ ├── testing.rst │ ├── reproducible.rst │ ├── clean_code.rst │ ├── documentation.rst │ └── packaging.rst ├── Makefile └── make.bat ├── .flake8 ├── .pre-commit-config.yaml ├── .readthedocs.yaml ├── OLD_setup.py ├── LICENSE ├── pyproject.toml ├── tests └── test_fftconvolve.py ├── profile └── fftconvolve.py ├── .github └── workflows │ ├── setuptools.yml │ └── poetry.yml ├── .gitignore ├── environment.yml └── README.rst /pydevtips/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /examples/configs/signal/ExampleNumpy.yaml: -------------------------------------------------------------------------------- 1 | _target_: numpy.random.randn -------------------------------------------------------------------------------- /examples/configs/signal/ExampleZeros.yaml: -------------------------------------------------------------------------------- 1 | _target_: real_convolve.ExampleZeros -------------------------------------------------------------------------------- /examples/configs/signal/transform/power.yaml: -------------------------------------------------------------------------------- 1 | _target_: real_convolve.PowerTransform 2 | pow: 2 -------------------------------------------------------------------------------- /examples/configs/signal/ExampleCustom.yaml: -------------------------------------------------------------------------------- 1 | _target_: real_convolve.ExampleCustom 2 | numpy_method: arange -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | sphinx==5.0.1 2 | sphinx_rtd_theme==0.4.3 3 | docutils==0.16 # >0.17 doesn't render bullets -------------------------------------------------------------------------------- /docs/images/vscode_jupyter_notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ebezzam/python-dev-tips/HEAD/docs/images/vscode_jupyter_notebook.png -------------------------------------------------------------------------------- /examples/configs/exp1.yaml: -------------------------------------------------------------------------------- 1 | # python examples/real_convolve.py -cn exp1 2 | defaults: 3 | - defaults 4 | - _self_ 5 | 6 | filter_len: 25 -------------------------------------------------------------------------------- /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | exclude = .git, __pycache__, build, dist 3 | ignore = E203, E266, E501, W503, F403, F401, C901 4 | max-line-length = 100 5 | max-complexity = 18 6 | select = B,C,E,F,W,T4,B9 -------------------------------------------------------------------------------- /docs/source/utils.rst: -------------------------------------------------------------------------------- 1 | Function example 2 | ================ 3 | 4 | .. GENERATE DOCUMENTATION FOR ALL FUNCTIONS 5 | .. .. automodule:: pydevtips.utils 6 | 7 | 8 | .. GENERATE INDIVIDUALLY 9 | .. autofunction:: pydevtips.utils.add 10 | 11 | -------------------------------------------------------------------------------- /examples/configs/defaults.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - signal: ExampleNumpy 3 | - _self_ 4 | 5 | hydra: 6 | job: 7 | chdir: True # change to output folder 8 | job_logging: 9 | formatters: 10 | simple: 11 | format: '[%(levelname)s] - %(message)s' 12 | 13 | seed: 0 14 | signal_len: 1000 15 | filter_len: 10 -------------------------------------------------------------------------------- /pydevtips/utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Utility functions. 3 | """ 4 | 5 | 6 | def add(a, b): 7 | """ 8 | Add two integers. 9 | 10 | Parameters 11 | ---------- 12 | a : int 13 | First integer. 14 | b : int 15 | Second integer. 16 | 17 | Returns 18 | ------- 19 | result : int 20 | Sum of inputs. 21 | 22 | """ 23 | assert isinstance(a, int) 24 | assert isinstance(b, int) 25 | return a + b 26 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/python-poetry/poetry 3 | rev: '1.8.4' # add version here 4 | hooks: 5 | - id: poetry-check 6 | - id: poetry-lock 7 | - id: poetry-export 8 | - id: poetry-install 9 | - repo: https://github.com/psf/black 10 | rev: 22.12.0 11 | hooks: 12 | - id: black 13 | language_version: python3 14 | - repo: https://github.com/PyCQA/flake8 15 | rev: 6.0.0 16 | hooks: 17 | - id: flake8 18 | - repo: https://github.com/pycqa/isort 19 | rev: 5.13.2 20 | hooks: 21 | - id: isort 22 | name: isort (python) -------------------------------------------------------------------------------- /.readthedocs.yaml: -------------------------------------------------------------------------------- 1 | # .readthedocs.yaml 2 | # Read the Docs configuration file 3 | # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details 4 | 5 | # Required 6 | version: 2 7 | 8 | # Set the version of Python and other tools you might need 9 | build: 10 | os: ubuntu-20.04 11 | tools: 12 | python: "3.9" 13 | 14 | # Build documentation in the docs/ directory with Sphinx 15 | sphinx: 16 | configuration: docs/source/conf.py 17 | 18 | # If using Sphinx, optionally build your docs in additional formats such as PDF 19 | # formats: 20 | # - pdf 21 | 22 | # Optionally declare the Python requirements required to build your docs 23 | python: 24 | install: 25 | - requirements: docs/requirements.txt 26 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = source 9 | BUILDDIR = build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /docs/source/index.rst: -------------------------------------------------------------------------------- 1 | .. pydevtips documentation master file, created by 2 | sphinx-quickstart on Fri Jan 13 17:55:12 2023. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | .. toctree:: 7 | :maxdepth: 2 8 | :caption: Contents: 9 | 10 | .. include:: ../../README.rst 11 | 12 | 13 | Contents 14 | -------- 15 | 16 | .. toctree:: 17 | 18 | packaging 19 | virtual_env 20 | version_control 21 | reproducible 22 | documentation 23 | clean_code 24 | testing 25 | remote_development 26 | badges 27 | 28 | .. toctree:: 29 | :caption: Code 30 | 31 | fftconvolve 32 | utils 33 | 34 | 35 | Indices and tables 36 | ================== 37 | 38 | * :ref:`genindex` 39 | * :ref:`modindex` 40 | * :ref:`search` 41 | -------------------------------------------------------------------------------- /OLD_setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.rst", "r", encoding="utf-8") as fh: 4 | long_description = fh.read() 5 | 6 | setuptools.setup( 7 | name="pydevtips", 8 | version="0.0.2", 9 | author="Eric Bezzam", 10 | author_email="ebezzam@gmail.com", 11 | description="Functions and scripts to demonstrate Python development tips.", 12 | long_description=long_description, 13 | long_description_content_type="text/x-rst", 14 | url="https://github.com/ebezzam/python-dev-tips", 15 | packages=setuptools.find_packages(), 16 | classifiers=[ 17 | "Programming Language :: Python :: 3", 18 | "Operating System :: OS Independent", 19 | ], 20 | python_requires=">=3.8", 21 | install_requires=[ 22 | "numpy", 23 | "scipy", 24 | "matplotlib", 25 | "hydra-core", 26 | "tqdm", 27 | ], 28 | include_package_data=True, 29 | ) 30 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=source 11 | set BUILDDIR=build 12 | 13 | if "%1" == "" goto help 14 | 15 | %SPHINXBUILD% >NUL 2>NUL 16 | if errorlevel 9009 ( 17 | echo. 18 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 19 | echo.installed, then set the SPHINXBUILD environment variable to point 20 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 21 | echo.may add the Sphinx directory to PATH. 22 | echo. 23 | echo.If you don't have Sphinx installed, grab it from 24 | echo.http://sphinx-doc.org/ 25 | exit /b 1 26 | ) 27 | 28 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 29 | goto end 30 | 31 | :help 32 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 33 | 34 | :end 35 | popd 36 | -------------------------------------------------------------------------------- /docs/source/version_control.rst: -------------------------------------------------------------------------------- 1 | Version control 2 | =============== 3 | 4 | Version control is important for many reasons. It allows you to keep track of 5 | changes, and to revert to previous versions if needed. It also allows you to 6 | try out new things without breaking a stable version of your code, which is 7 | especially useful if you're working with others. There are several version 8 | control systems (Git, Mercurial, SVN, etc.). For this project, `Git `_ 9 | is used. 10 | 11 | Independent of version control is choosing a platform to host your project, 12 | which can also help in the reproducibility of your code by allowing others to 13 | conveniently access your code. Example platforms for Git are GitHub, GitLab, 14 | and Bitbucket. GitHub is the most popular, and is used for this project. 15 | 16 | More on Git and GitHub can be found in this `tutorial `_. 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Eric Bezzam 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.poetry] 2 | name = "pydevtips" 3 | version = "0.0.4" 4 | description = "Functions and scripts to demonstrate Python development tips." 5 | authors = ["Eric Bezzam "] 6 | license = "MIT" 7 | 8 | # -- manually added -- 9 | readme = "README.rst" 10 | package-mode = true # https://python-poetry.org/docs/basic-usage/#operating-modes 11 | # -------------------- 12 | 13 | [tool.poetry.dependencies] 14 | python = "^3.10" 15 | numpy = "^2.1.2" 16 | scipy = "^1.14.1" 17 | matplotlib = "^3.9.2" 18 | hydra-core = "^1.3.2" 19 | tqdm = "^4.66.5" 20 | 21 | 22 | [tool.poetry.group.dev.dependencies] 23 | black = "^24.10.0" 24 | isort = "^5.13.2" 25 | flake8 = "^7.1.1" 26 | pytest = "^8.3.3" 27 | pre-commit = "^4.0.1" 28 | twine = "^5.1.1" 29 | 30 | [build-system] 31 | requires = ["poetry-core"] 32 | build-backend = "poetry.core.masonry.api" 33 | 34 | #### -- manually added (below) 35 | 36 | [project.urls] 37 | Homepage = "https://github.com/ebezzam/python-dev-tips" 38 | Issues = "https://github.com/ebezzam/python-dev-tips/issues" 39 | Documentation = "https://pydevtips.readthedocs.io" 40 | 41 | [tool.isort] 42 | profile = "black" 43 | 44 | [tool.black] 45 | line-length = 100 46 | include = '\.pyi?$' 47 | exclude = ''' 48 | /( 49 | \.git 50 | | build 51 | | dist 52 | )/ 53 | ''' 54 | -------------------------------------------------------------------------------- /tests/test_fftconvolve.py: -------------------------------------------------------------------------------- 1 | from pydevtips.fftconvolve import RFFTConvolve, FFTConvolve 2 | import numpy as np 3 | 4 | 5 | # create random signal 6 | n = 1000 7 | signal = np.random.randn(n) 8 | 9 | # create filter 10 | filter = np.random.randn(n) 11 | 12 | # reference output 13 | fft_naive = np.convolve(signal, filter, mode="full") 14 | 15 | 16 | def test_rfft(): 17 | 18 | # create object 19 | rfft_convolver = RFFTConvolve(filt=filter, length=len(signal)) 20 | 21 | # convolve 22 | rfft_out = rfft_convolver(signal) 23 | 24 | # check results 25 | assert np.allclose(rfft_out, fft_naive) 26 | 27 | 28 | def test_fft(): 29 | 30 | # create object 31 | fft_convolver = FFTConvolve(filter=filter, length=len(signal)) 32 | 33 | # convolve 34 | fft_out = fft_convolver(signal) 35 | 36 | # check results 37 | assert np.allclose(fft_out, fft_out) 38 | 39 | 40 | def test_fft_complex(): 41 | 42 | # create complex signal 43 | signal = np.random.randn(n) + 1j * np.random.randn(n) 44 | 45 | # create complex filter 46 | filter = np.random.randn(n) + 1j * np.random.randn(n) 47 | 48 | # create object 49 | fft_convolver = FFTConvolve(filter=filter, length=len(signal)) 50 | 51 | # convolve 52 | fft_out = fft_convolver(signal) 53 | 54 | # check results 55 | fft_naive = np.convolve(signal, filter, mode="full") 56 | assert np.allclose(fft_out, fft_naive) 57 | -------------------------------------------------------------------------------- /examples/joblib_parallel.py: -------------------------------------------------------------------------------- 1 | """ 2 | Example of using `joblib` to parallelize a loop. 3 | 4 | pip install joblib 5 | 6 | """ 7 | 8 | import numpy as np 9 | import time 10 | 11 | try: 12 | from joblib import Parallel, delayed 13 | except ImportError: 14 | print("Install joblib to run this example") 15 | exit() 16 | 17 | 18 | def f(seed, n, proc_time): 19 | np.random.seed(seed) 20 | x = np.random.randn(n) 21 | time.sleep(proc_time) # simulate some long running process 22 | return np.fft.fft(x) 23 | 24 | 25 | if __name__ == "__main__": 26 | 27 | # Create a list of inputs 28 | n = 2048 29 | n_exp = 100 30 | proc_time = 0.2 31 | n_cpu = 6 32 | 33 | # Compare processing time for serial and parallel processing 34 | 35 | # Serial processing 36 | start = time.perf_counter() 37 | outputs_ser = [] 38 | for seed in range(n_exp): 39 | outputs_ser.append(f(seed, n, proc_time)) 40 | serial_time = time.perf_counter() - start 41 | print(f"Serial processing took {serial_time} seconds") 42 | 43 | # Parallel processing 44 | start = time.perf_counter() 45 | outputs_par = Parallel(n_jobs=n_cpu)(delayed(f)(seed, n, proc_time) for seed in range(n_exp)) 46 | parallel_time = time.perf_counter() - start 47 | print(f"Parallel processing took {parallel_time} seconds") 48 | 49 | # Speed-up 50 | print(f"Speed-up: {serial_time / parallel_time}") 51 | -------------------------------------------------------------------------------- /docs/source/badges.rst: -------------------------------------------------------------------------------- 1 | Badges 2 | ====== 3 | 4 | Adding badges to your GitHub repository and documentation amounts to simply adding an image. 5 | 6 | For example, in a `reStructuredText `__ file: 7 | 8 | .. code:: rst 9 | 10 | .. image:: https://readthedocs.org/projects/pydevtips/badge/?version=latest 11 | :target: http://pydevtips.readthedocs.io/en/latest/ 12 | :alt: Documentation Status 13 | 14 | which when rendered will make the following badge: 15 | 16 | .. image:: https://readthedocs.org/projects/pydevtips/badge/?version=latest 17 | :target: http://pydevtips.readthedocs.io/en/latest/ 18 | :alt: Documentation Status 19 | 20 | You can also create `badges from GitHub Actions `__. 21 | 22 | .. code:: rst 23 | 24 | .. image:: https://github.com/ebezzam/python-dev-tips/actions/workflows/poetry.yml/badge.svg 25 | :target: https://github.com/ebezzam/python-dev-tips/blob/main/.github/workflows/poetry.yml 26 | :alt: Unit tests and formatting 27 | 28 | .. image:: https://github.com/ebezzam/python-dev-tips/actions/workflows/poetry.yml/badge.svg 29 | :target: https://github.com/ebezzam/python-dev-tips/blob/main/.github/workflows/poetry.yml 30 | :alt: Unit tests and formatting 31 | 32 | Finally, you can use `Badgen `__ or `Shields.io `__ to create custom badges. 33 | 34 | -------------------------------------------------------------------------------- /docs/source/fftconvolve.rst: -------------------------------------------------------------------------------- 1 | Class example 2 | ============= 3 | 4 | .. GENERATE DOCUMENTATION FOR ALL CLASSES 5 | .. .. automodule:: pydevtips.fftconvolve 6 | .. :member-order: bysource 7 | .. :show-inheritance: 8 | .. :special-members: __init__, __call__ 9 | 10 | 11 | .. GENERATE INDIVIDUALLY WITH CUSTOM TEXT 12 | 13 | There are two classes in this module for performing convolution in the frequency domain with a fixed filter. 14 | 15 | .. autosummary:: 16 | pydevtips.fftconvolve.RFFTConvolve 17 | pydevtips.fftconvolve.FFTConvolve 18 | 19 | Both inherit from the base class :class:`pydevtips.fftconvolve.FFTConvolveBase`, 20 | overwriting the abstract methods: :func:`pydevtips.fftconvolve.FFTConvolveBase._compute_filter_frequency_response` and 21 | :func:`pydevtips.fftconvolve.FFTConvolveBase.__call__`. 22 | 23 | RFFTConvolve 24 | ------------ 25 | 26 | .. autoclass:: pydevtips.fftconvolve.RFFTConvolve 27 | :member-order: bysource 28 | :show-inheritance: 29 | :special-members: __init__, __call__ 30 | 31 | FFTConvolve 32 | ----------- 33 | 34 | .. autoclass:: pydevtips.fftconvolve.FFTConvolve 35 | :member-order: bysource 36 | :show-inheritance: 37 | :special-members: __init__, __call__ 38 | 39 | 40 | FFTConvolveBase 41 | --------------- 42 | 43 | .. autoclass:: pydevtips.fftconvolve.FFTConvolveBase 44 | :member-order: bysource 45 | :show-inheritance: 46 | :members: 47 | :undoc-members: 48 | :special-members: __init__, __call__ 49 | :private-members: _compute_filter_frequency_response 50 | -------------------------------------------------------------------------------- /profile/fftconvolve.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from tqdm import tqdm 3 | import time 4 | from pydevtips.fftconvolve import RFFTConvolve, FFTConvolve 5 | 6 | 7 | seed = 0 8 | np.random.seed(seed) 9 | 10 | 11 | # create random signal 12 | n = 10000 13 | signal = np.random.randn(n) 14 | 15 | # create filter 16 | filter = np.random.randn(n) 17 | 18 | # create FFT objects 19 | rfft_convolver = RFFTConvolve(filt=filter, length=len(signal)) 20 | fft_convolver = FFTConvolve(filter=filter, length=len(signal)) 21 | 22 | # profile 23 | n_trials = 100 24 | 25 | # rfft 26 | print("rfft") 27 | rfft_convolved_signal = np.zeros_like(signal) 28 | start_time = time.perf_counter() 29 | for _ in tqdm(range(n_trials)): 30 | rfft_out = rfft_convolver(signal) 31 | proc_time_rfft = (time.perf_counter() - start_time) / n_trials 32 | 33 | # fft 34 | print("fft") 35 | fft_convolved_signal = np.zeros_like(signal) 36 | start_time = time.perf_counter() 37 | for _ in tqdm(range(n_trials)): 38 | fft_out = fft_convolver(signal) 39 | proc_time_fft = (time.perf_counter() - start_time) / n_trials 40 | 41 | # fft without initializing 42 | print("fft naive (without initializing)") 43 | for _ in tqdm(range(n_trials)): 44 | fft_naive_out = np.convolve(signal, filter, mode="full") 45 | proc_time_fft_naive = (time.perf_counter() - start_time) / n_trials 46 | 47 | # check results 48 | assert np.allclose(rfft_out, fft_out) 49 | assert np.allclose(rfft_out, fft_naive_out) 50 | print(f"rfft: {proc_time_rfft} s") 51 | print(f"fft: {proc_time_fft} s") 52 | print(f"fft naive: {proc_time_fft_naive} s") 53 | print(f"rfft is {proc_time_fft / proc_time_rfft:.2f} times faster than fft") 54 | print(f"fft is {proc_time_fft_naive / proc_time_fft:.2f} times faster than fft naive") 55 | print(f"rfft is {proc_time_fft_naive / proc_time_rfft:.2f} times faster than fft naive") 56 | -------------------------------------------------------------------------------- /.github/workflows/setuptools.yml: -------------------------------------------------------------------------------- 1 | name: pydevtips setuptools 2 | 3 | # HACK to not trigger GitHub action 4 | on: 5 | push: 6 | branches: 7 | - non-existant-branch 8 | 9 | 10 | jobs: 11 | build: 12 | 13 | runs-on: ${{ matrix.os }} 14 | strategy: 15 | fail-fast: false 16 | max-parallel: 12 17 | matrix: 18 | os: [ubuntu-latest, macos-latest, windows-latest] 19 | python-version: [3.8, 3.9, "3.10"] 20 | steps: 21 | - uses: actions/checkout@v3 22 | - name: Checkout submodules 23 | shell: bash 24 | run: | 25 | auth_header="$(git config --local --get http.https://github.com/.extraheader)" 26 | git submodule sync --recursive 27 | git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1 28 | - name: Set up Python ${{ matrix.python-version }} 29 | uses: actions/setup-python@v4 30 | with: 31 | python-version: ${{ matrix.python-version }} 32 | - name: Install dependencies and build package 33 | run: | 34 | python -m pip install --upgrade pip 35 | python -m pip install -e . 36 | - name: Lint with flake8 37 | run: | 38 | pip install flake8 39 | # stop the build if there are Python syntax errors or undefined names 40 | flake8 . --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901 41 | - name: Format with black 42 | run: | 43 | pip install black 44 | black *.py -l 100 45 | black examples/*.py -l 100 46 | black profile/*.py -l 100 47 | black pydevtips/*.py -l 100 48 | black tests/*.py -l 100 49 | - name: Test with pytest 50 | run: | 51 | pip install -U pytest 52 | pytest -------------------------------------------------------------------------------- /docs/source/remote_development.rst: -------------------------------------------------------------------------------- 1 | Remote development 2 | ================== 3 | 4 | In machine learning and scientific computing applications, you may find yourself in a situation where you need to develop and deploy code on a remote machine. 5 | This may limit your ability to visualize results as your interaction with the remote machine is typically limited to the command line. 6 | One way to get around this is to use a interactive development environment (IDE) that allows you to run code on a remote machine. 7 | With Visual Studio (VS) Code, you can use the `Remote - SSH `__ extension to connect to a remote machine and develop code on it. 8 | 9 | `This page `__ gives you pretty much everything you need to setup remote development with VS Code, most notably `connecting to a remote host `__. 10 | 11 | 12 | Using Jupyter notebooks to visualize plots 13 | ------------------------------------------ 14 | 15 | Running Jupyter notebooks on a remote machine through VS Code allows you to conveniently visualize results. 16 | 17 | First, be sure to install the ``ipykernel`` package in your virtual environment (on the remote machine). 18 | 19 | .. code-block:: bash 20 | 21 | pip install ipykernel 22 | 23 | When opening your notebook, select your virtual environment. 24 | 25 | .. image:: ../images/vscode_jupyter_notebook.png 26 | 27 | 28 | Running long scripts 29 | -------------------- 30 | 31 | When you close an SSH connection, the process running on the remote machine is killed. 32 | If you would like to run a long script on a remote machine, there are several command line tools that let you run a script even after logging out: 33 | 34 | * `screen `__ (Linux or `WSL `__ for Windows). 35 | * `tmux `__ (Linux, WSL, and macOS). 36 | * `nohup `__ (Linux or WSL). 37 | -------------------------------------------------------------------------------- /examples/cupy_fft.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | CuPy Example 4 | 5 | Installing: 6 | - check Cuda version, e.g. from Terminal run: `nvcc --version` or `nvidia-smi` 7 | - install corresponding version of cupy, e.g. `pip install cupy-cuda11x` 8 | 9 | Installation page: https://docs.cupy.dev/en/stable/install.html 10 | 11 | """ 12 | 13 | from importlib import util 14 | import os 15 | import numpy as np 16 | import scipy 17 | import time 18 | 19 | try: 20 | import cupy as cp 21 | import cupyx 22 | 23 | CUPY_AVAILABLE = True 24 | except ImportError: 25 | CUPY_AVAILABLE = False 26 | 27 | 28 | def get_array_module(x): 29 | """ 30 | Returns correct numerical module based on input. 31 | 32 | Parameters 33 | ---------- 34 | x : :obj:`numpy.ndarray` or :obj:`cupy.ndarray` 35 | Array 36 | Returns 37 | ------- 38 | mod : :obj:`func` 39 | Module to be used to process array (:mod:`numpy` or :mod:`cupy`) 40 | """ 41 | if CUPY_AVAILABLE: 42 | return cp.get_array_module(x) 43 | else: 44 | return np 45 | 46 | 47 | def fft2(x): 48 | """ 49 | Applies correct fft method based on input. 50 | 51 | Parameters 52 | ---------- 53 | x : :obj:`numpy.ndarray` or :obj:`cupy.ndarray` 54 | Array 55 | 56 | Returns 57 | ------- 58 | mod : :obj:`func` 59 | Module to be used to process array (:mod:`numpy` or :mod:`cupy`) 60 | """ 61 | if get_array_module(x) == np: 62 | func = scipy.fft.fft2 63 | else: 64 | func = cupyx.scipy.fft.fft2 65 | return func(x) 66 | 67 | 68 | # compare FFT computation 69 | n = 1024 70 | n_trials = 100 71 | x = np.random.rand(n, n) 72 | 73 | if CUPY_AVAILABLE: 74 | x_gpu = cp.asarray(x) 75 | print(x_gpu.device) 76 | else: 77 | x_gpu = x 78 | print("Cupy not available. Using numpy instead.") 79 | 80 | # numpy 81 | start = time.perf_counter() 82 | for _ in range(n_trials): 83 | fft2(x) 84 | time_cpu = time.perf_counter() - start 85 | print(f"CPU processing took {time_cpu} seconds") 86 | 87 | # cupy 88 | start = time.perf_counter() 89 | for _ in range(n_trials): 90 | fft2(x_gpu) 91 | time_gpu = time.perf_counter() - start 92 | print(f"GPU processing took {time_gpu} seconds") 93 | 94 | # speed-up 95 | print(f"Speed-up: {time_cpu / time_gpu}") 96 | -------------------------------------------------------------------------------- /.github/workflows/poetry.yml: -------------------------------------------------------------------------------- 1 | # Poetry GitHub Action: https://github.com/marketplace/actions/python-poetry-action 2 | name: pydevtips 3 | 4 | # on: [push, pull_request] 5 | on: 6 | # trigger on pushes and PRs to main 7 | push: 8 | branches: 9 | - main 10 | pull_request: 11 | branches: 12 | - main 13 | 14 | jobs: 15 | build: 16 | 17 | runs-on: ${{ matrix.os }} 18 | strategy: 19 | fail-fast: false 20 | max-parallel: 12 21 | matrix: 22 | os: [ubuntu-latest, macos-latest, windows-latest] 23 | python-version: ["3.10", "3.11"] 24 | poetry-version: ["1.8.4"] 25 | steps: 26 | - uses: actions/checkout@v4 27 | - name: Checkout submodules 28 | shell: bash 29 | run: | 30 | auth_header="$(git config --local --get http.https://github.com/.extraheader)" 31 | git submodule sync --recursive 32 | git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1 33 | - name: Set up Python ${{ matrix.python-version }} 34 | uses: actions/setup-python@v4 35 | with: 36 | python-version: ${{ matrix.python-version }} 37 | - name: Install poetry 38 | uses: abatilo/actions-poetry@v3.0.0 39 | with: 40 | poetry-version: ${{ matrix.poetry-version }} 41 | - name: Install dependencies 42 | run: | 43 | poetry install --with dev 44 | - name: Lint with flake8 45 | run: | 46 | # stop the build if there are Python syntax errors or undefined names 47 | poetry run flake8 pydevtips --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901 48 | poetry run flake8 examples --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901 49 | poetry run flake8 profile --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901 50 | poetry run flake8 tests --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901 51 | - name: Format with black 52 | run: | 53 | poetry run black pydevtips -l 100 54 | poetry run black examples -l 100 55 | poetry run black profile -l 100 56 | poetry run black tests -l 100 57 | - name: Test with pytest 58 | run: poetry run pytest -v -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | project_env/ 2 | .DS_Store 3 | dataset/ 4 | 5 | # from hydra 6 | outputs/ 7 | 8 | # Byte-compiled / optimized / DLL files 9 | __pycache__/ 10 | *.py[cod] 11 | *$py.class 12 | 13 | # C extensions 14 | *.so 15 | 16 | # Distribution / packaging 17 | poetry.lock 18 | .Python 19 | build/ 20 | develop-eggs/ 21 | dist/ 22 | downloads/ 23 | eggs/ 24 | .eggs/ 25 | lib/ 26 | lib64/ 27 | parts/ 28 | sdist/ 29 | var/ 30 | wheels/ 31 | pip-wheel-metadata/ 32 | share/python-wheels/ 33 | *.egg-info/ 34 | .installed.cfg 35 | *.egg 36 | MANIFEST 37 | 38 | # PyInstaller 39 | # Usually these files are written by a python script from a template 40 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 41 | *.manifest 42 | *.spec 43 | 44 | # Installer logs 45 | pip-log.txt 46 | pip-delete-this-directory.txt 47 | 48 | # Unit test / coverage reports 49 | htmlcov/ 50 | .tox/ 51 | .nox/ 52 | .coverage 53 | .coverage.* 54 | .cache 55 | nosetests.xml 56 | coverage.xml 57 | *.cover 58 | *.py,cover 59 | .hypothesis/ 60 | .pytest_cache/ 61 | 62 | # Translations 63 | *.mo 64 | *.pot 65 | 66 | # Django stuff: 67 | *.log 68 | local_settings.py 69 | db.sqlite3 70 | db.sqlite3-journal 71 | 72 | # Flask stuff: 73 | instance/ 74 | .webassets-cache 75 | 76 | # Scrapy stuff: 77 | .scrapy 78 | 79 | # Sphinx documentation 80 | docs/_build/ 81 | 82 | # PyBuilder 83 | target/ 84 | 85 | # Jupyter Notebook 86 | .ipynb_checkpoints 87 | 88 | # IPython 89 | profile_default/ 90 | ipython_config.py 91 | 92 | # pyenv 93 | .python-version 94 | 95 | # pipenv 96 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 97 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 98 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 99 | # install all needed dependencies. 100 | #Pipfile.lock 101 | 102 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 103 | __pypackages__/ 104 | 105 | # Celery stuff 106 | celerybeat-schedule 107 | celerybeat.pid 108 | 109 | # SageMath parsed files 110 | *.sage.py 111 | 112 | # Environments 113 | .env 114 | .venv 115 | env/ 116 | venv/ 117 | ENV/ 118 | env.bak/ 119 | venv.bak/ 120 | 121 | # Spyder project settings 122 | .spyderproject 123 | .spyproject 124 | 125 | # Rope project settings 126 | .ropeproject 127 | 128 | # mkdocs documentation 129 | /site 130 | 131 | # mypy 132 | .mypy_cache/ 133 | .dmypy.json 134 | dmypy.json 135 | 136 | # Pyre type checker 137 | .pyre/ 138 | -------------------------------------------------------------------------------- /examples/real_convolve.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import hydra 4 | import matplotlib.pyplot as plt 5 | import numpy as np 6 | from hydra.utils import instantiate 7 | 8 | from pydevtips.fftconvolve import RFFTConvolve 9 | 10 | 11 | class PowerTransform: 12 | def __init__(self, pow): 13 | self.pow = pow 14 | 15 | def __call__(self, x): 16 | return np.power(x, self.pow) 17 | 18 | 19 | class ExampleZeros: 20 | """ 21 | Wrapper over np.zeros 22 | """ 23 | 24 | def __init__(self, signal_len) -> None: 25 | self.data = np.zeros(signal_len) 26 | 27 | def max(self): 28 | return self.data.max() 29 | 30 | def __array__(self): 31 | return self.data 32 | 33 | def __len__(self): 34 | return len(self.data) 35 | 36 | 37 | class ExampleCustom: 38 | """ 39 | Wrapper over custom np.ndarray creation method with an optional transform 40 | """ 41 | 42 | def __init__(self, signal_len, numpy_method, transform=None) -> None: 43 | self.data = getattr(np, numpy_method)(signal_len) 44 | if transform is not None: 45 | self.data = transform(self.data) 46 | 47 | def max(self): 48 | return self.data.max() 49 | 50 | def __array__(self): 51 | return self.data 52 | 53 | def __len__(self): 54 | return len(self.data) 55 | 56 | 57 | @hydra.main(version_base=None, config_path="configs", config_name="defaults") 58 | def main(config): 59 | 60 | output_dir = os.getcwd() 61 | np.random.seed(config.seed) 62 | 63 | # Create a signal 64 | signal = instantiate(config.signal, config.signal_len) 65 | # Note that we added extra argument signal_len which was not defined in signal config 66 | print(f"Signal class, len and max: {type(signal), len(signal), signal.max()}") 67 | signal = np.array(signal) # for the following computations np.ndarray is required 68 | 69 | # Create a moving average filter (low pass) 70 | n = config.filter_len 71 | filter = np.ones(n) / n 72 | 73 | # initialize the convolver 74 | convolver = RFFTConvolve(filt=filter, length=len(signal)) 75 | 76 | # convolve 77 | convolved_signal = convolver(signal) 78 | 79 | # plot with three subplots 80 | _, ax = plt.subplots(3, 1, figsize=(10, 10)) 81 | ax[0].plot(signal) 82 | ax[0].set_title("Signal") 83 | ax[1].plot(filter) 84 | ax[1].set_title("Filter") 85 | ax[2].plot(convolved_signal) 86 | ax[2].set_title("Convolved Signal") 87 | 88 | plt.savefig(os.path.join(output_dir, "convolved_signal.png")) 89 | output_fp = os.path.join(output_dir, "convolved_signal.png") 90 | print(f"Saved convolved signal to {output_fp}") 91 | 92 | plt.show() 93 | 94 | 95 | if __name__ == "__main__": 96 | main() 97 | -------------------------------------------------------------------------------- /docs/source/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | # import os 14 | # import sys 15 | # sys.path.insert(0, os.path.abspath('.')) 16 | 17 | import datetime 18 | import os 19 | import sys 20 | from unittest import mock 21 | 22 | sys.path.insert(0, os.path.abspath(os.path.join("..", ".."))) 23 | 24 | MOCK_MODULES = ["matplotlib", "matplotlib.pyplot", "numpy"] 25 | for mod_name in MOCK_MODULES: 26 | sys.modules[mod_name] = mock.Mock() 27 | 28 | 29 | # -- Project information ----------------------------------------------------- 30 | 31 | project = "pydevtips" 32 | copyright = f"{datetime.date.today().year}, Eric Bezzam" 33 | author = "Eric Bezzam" 34 | 35 | 36 | # -- General configuration --------------------------------------------------- 37 | 38 | # Add any Sphinx extension module names here, as strings. They can be 39 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 40 | # ones. 41 | extensions = [ 42 | "sphinx_rtd_theme", 43 | "sphinx.ext.intersphinx", 44 | "sphinx.ext.autosummary", 45 | "sphinx.ext.autodoc", 46 | "sphinx.ext.napoleon", 47 | "sphinx.ext.autosectionlabel", 48 | ] 49 | 50 | autodoc_default_options = { 51 | "members": True, 52 | } 53 | autodoc_typehints = "none" # for cleaner sphinx output, don't show type hints 54 | 55 | intersphinx_mapping = { 56 | "python": ("https://docs.python.org/3/", None), 57 | "NumPy [latest]": ("https://docs.scipy.org/doc/numpy/", None), 58 | "matplotlib [stable]": ("https://matplotlib.org/stable/", None), 59 | } 60 | intersphinx_disabled_domains = ["std"] 61 | 62 | # Add any paths that contain templates here, relative to this directory. 63 | templates_path = ["_templates"] 64 | 65 | # List of patterns, relative to source directory, that match files and 66 | # directories to ignore when looking for source files. 67 | # This pattern also affects html_static_path and html_extra_path. 68 | exclude_patterns = [] 69 | 70 | 71 | # -- Options for HTML output ------------------------------------------------- 72 | 73 | # The theme to use for HTML and HTML Help pages. See the documentation for 74 | # a list of builtin themes. 75 | # 76 | html_theme = "sphinx_rtd_theme" 77 | 78 | # Add any paths that contain custom static files (such as style sheets) here, 79 | # relative to this directory. They are copied after the builtin static files, 80 | # so a file named "default.css" will overwrite the builtin "default.css". 81 | # html_static_path = ["_static"] 82 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: project_env_yml 2 | channels: 3 | - defaults 4 | dependencies: 5 | - bzip2=1.0.8=h80987f9_5 6 | - ca-certificates=2023.12.12=hca03da5_0 7 | - libffi=3.4.4=hca03da5_0 8 | - ncurses=6.4=h313beb8_0 9 | - openssl=3.0.13=h1a28f6b_0 10 | - pip=23.3.1=py311hca03da5_0 11 | - python=3.11.8=hb885b13_0 12 | - readline=8.2=h1a28f6b_0 13 | - setuptools=68.2.2=py311hca03da5_0 14 | - sqlite=3.41.2=h80987f9_0 15 | - tk=8.6.12=hb8d0fd4_0 16 | - wheel=0.41.2=py311hca03da5_0 17 | - xz=5.4.6=h80987f9_0 18 | - zlib=1.2.13=h5a0b063_0 19 | - pip: 20 | - aiohappyeyeballs==2.3.7 21 | - aiohttp==3.10.3 22 | - aiosignal==1.3.1 23 | - antlr4-python3-runtime==4.9.3 24 | - attrs==24.2.0 25 | - audioread==3.0.1 26 | - backports-tarfile==1.2.0 27 | - black==24.10.0 28 | - certifi==2024.8.30 29 | - cffi==1.17.0 30 | - cfgv==3.4.0 31 | - charset-normalizer==3.4.0 32 | - click==8.1.7 33 | - contourpy==1.3.0 34 | - cycler==0.12.1 35 | - datasets==2.21.0 36 | - decorator==5.1.1 37 | - dill==0.3.8 38 | - distlib==0.3.9 39 | - docutils==0.21.2 40 | - filelock==3.16.1 41 | - flake8==7.1.1 42 | - fonttools==4.54.1 43 | - frozenlist==1.4.1 44 | - fsspec==2024.6.1 45 | - huggingface-hub==0.24.5 46 | - hydra-core==1.3.2 47 | - identify==2.6.1 48 | - idna==3.10 49 | - importlib-metadata==8.5.0 50 | - iniconfig==2.0.0 51 | - isort==5.13.2 52 | - jaraco-classes==3.4.0 53 | - jaraco-context==6.0.1 54 | - jaraco-functools==4.1.0 55 | - jedi==0.19.1 56 | - joblib==1.4.2 57 | - keyring==25.4.1 58 | - kiwisolver==1.4.7 59 | - lazy-loader==0.4 60 | - librosa==0.10.2.post1 61 | - llvmlite==0.43.0 62 | - markdown-it-py==3.0.0 63 | - matplotlib==3.9.2 64 | - mccabe==0.7.0 65 | - mdurl==0.1.2 66 | - more-itertools==10.5.0 67 | - msgpack==1.0.8 68 | - multidict==6.0.5 69 | - multiprocess==0.70.16 70 | - mypy-extensions==1.0.0 71 | - nh3==0.2.18 72 | - nodeenv==1.9.1 73 | - numba==0.60.0 74 | - numpy==2.1.2 75 | - omegaconf==2.3.0 76 | - packaging==24.1 77 | - pandas==2.2.2 78 | - parso==0.8.4 79 | - pathspec==0.12.1 80 | - pillow==11.0.0 81 | - pkginfo==1.10.0 82 | - platformdirs==4.3.6 83 | - pluggy==1.5.0 84 | - pooch==1.8.2 85 | - pre-commit==4.0.1 86 | - pudb==2024.1.2 87 | - pyarrow==17.0.0 88 | - pycodestyle==2.12.1 89 | - pycparser==2.22 90 | - pydevtips==0.0.3 91 | - pyflakes==3.2.0 92 | - pygments==2.18.0 93 | - pyparsing==3.2.0 94 | - pytest==8.3.3 95 | - python-dateutil==2.9.0.post0 96 | - pytz==2024.1 97 | - pyyaml==6.0.2 98 | - readme-renderer==44.0 99 | - requests==2.32.3 100 | - requests-toolbelt==1.0.0 101 | - rfc3986==2.0.0 102 | - rich==13.9.2 103 | - scikit-learn==1.5.1 104 | - scipy==1.14.1 105 | - six==1.16.0 106 | - soundfile==0.12.1 107 | - soxr==0.4.0 108 | - threadpoolctl==3.5.0 109 | - tqdm==4.66.5 110 | - twine==5.1.1 111 | - typing-extensions==4.12.2 112 | - tzdata==2024.1 113 | - urllib3==2.2.3 114 | - urwid==2.6.15 115 | - urwid-readline==0.14 116 | - virtualenv==20.27.0 117 | - wcwidth==0.2.13 118 | - xxhash==3.5.0 119 | - yarl==1.9.4 120 | - zipp==3.20.2 121 | prefix: /Users/eric/anaconda3/envs/project_env 122 | -------------------------------------------------------------------------------- /examples/numpy_speedup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Example of vectorizing operations with `numpy` 3 | 4 | Blog: https://towardsdatascience.com/the-art-of-speeding-up-python-loop-4970715717c 5 | Blog: https://shihchinw.github.io/2019/03/performance-tips-of-numpy-ndarray.html 6 | 7 | """ 8 | 9 | import numpy as np 10 | import time 11 | from scipy.fft import rfft2 # depending on version of `numpy` float32 may not output complex64 12 | 13 | 14 | n_trials = 100 15 | 16 | """ Adding two vectors, compare nonvectorized and vectorized operations """ 17 | print("\n-- Adding two vectors") 18 | n = 1000 19 | a = np.random.randn(n) 20 | b = np.random.randn(n) 21 | 22 | # Nonvectorized 23 | start = time.perf_counter() 24 | for _ in range(n_trials): 25 | c = [] 26 | for i in range(len(a)): 27 | c.append(a[i] + b[i]) 28 | c = np.array(c) 29 | nonvectorized_time = (time.perf_counter() - start) / n_trials 30 | print(f"Nonvectorized time: {nonvectorized_time} seconds") 31 | 32 | # Vectorized 33 | start = time.perf_counter() 34 | for _ in range(n_trials): 35 | c_vec = a + b 36 | vectorized_time = (time.perf_counter() - start) / n_trials 37 | print(f"Vectorized time: {vectorized_time} seconds") 38 | 39 | assert np.allclose(c, c_vec) 40 | print(f"Speed-up: {nonvectorized_time / vectorized_time}") 41 | 42 | 43 | """ Broadcasting """ 44 | print("\n-- Broadcasting") 45 | n = 100 46 | a = np.random.randn(n, n) 47 | b = np.random.randn(n) # add a row vector to each row of a 48 | 49 | # Nonvectorized 50 | start = time.perf_counter() 51 | for _ in range(n_trials): 52 | c = [] 53 | for i in range(len(a)): 54 | c.append(a[i] + b) 55 | c = np.array(c) 56 | nonvectorized_time = (time.perf_counter() - start) / n_trials 57 | print(f"Nonvectorized time: {nonvectorized_time} seconds") 58 | 59 | # Vectorized 60 | start = time.perf_counter() 61 | for _ in range(n_trials): 62 | c_vec = a + b[np.newaxis, :] 63 | vectorized_time = (time.perf_counter() - start) / n_trials 64 | print(f"Vectorized time: {vectorized_time} seconds") 65 | 66 | assert np.allclose(c, c_vec) 67 | print(f"Speed-up: {nonvectorized_time / vectorized_time}") 68 | 69 | 70 | """ Apply function along axis (if memory allows) """ 71 | print("\n-- Apply function (e.g. fft) along axis") 72 | n = 512 73 | n_signals = 500 74 | a = np.random.randn(n, n_signals) 75 | 76 | # Nonvectorized 77 | start = time.perf_counter() 78 | for _ in range(n_trials): 79 | c = [] 80 | for i in range(n_signals): 81 | c.append(np.fft.fft(a[:, i])) 82 | c = np.array(c) 83 | nonvectorized_time = (time.perf_counter() - start) / n_trials 84 | print(f"Nonvectorized time: {nonvectorized_time} seconds") 85 | 86 | # Vectorized 87 | start = time.perf_counter() 88 | for _ in range(n_trials): 89 | c_vec = np.fft.fft(a, axis=0) 90 | vectorized_time = (time.perf_counter() - start) / n_trials 91 | print(f"Vectorized time: {vectorized_time} seconds") 92 | 93 | assert np.allclose(c.T, c_vec) 94 | print(f"Speed-up: {nonvectorized_time / vectorized_time}") 95 | 96 | 97 | """ Using smaller data types if possible """ 98 | print("\n-- Using smaller data types if possible") 99 | 100 | # float64 101 | a = np.random.randn(512, 512) 102 | start = time.perf_counter() 103 | for _ in range(n_trials): 104 | # b = np.fft.rfft2(a) 105 | b = rfft2(a) 106 | float64_time = (time.perf_counter() - start) / n_trials 107 | print(f"float64 time: {float64_time} seconds") 108 | 109 | # float32 110 | a32 = a.astype(np.float32) 111 | start = time.perf_counter() 112 | for _ in range(n_trials): 113 | # b32 = np.fft.rfft2(a32) 114 | b32 = rfft2(a32) 115 | float32_time = (time.perf_counter() - start) / n_trials 116 | print(f"float32 time: {float32_time} seconds") 117 | 118 | assert b32.dtype == np.complex64 119 | 120 | print(f"Speed-up: {float64_time / float32_time}") 121 | -------------------------------------------------------------------------------- /docs/source/virtual_env.rst: -------------------------------------------------------------------------------- 1 | Virtual environments 2 | ==================== 3 | 4 | Virtual environments are a way to isolate your project from the rest of your 5 | system. This is important because it allows you to install packages that are 6 | specific to your project, without affecting the rest of your system. 7 | 8 | Creating an environment 9 | ----------------------- 10 | 11 | There are several ways to create virtual environments. The most popular 12 | (and recommended) is with `Anaconda `_. 13 | After installing Anaconda or `Miniconda `_ (light version), 14 | you create a new environment like so: 15 | 16 | .. code:: bash 17 | 18 | # create new environment, press enter to accept 19 | # -- important to set python version, otherwise `python` executable may not exist 20 | # -- (would be `python3` instead) 21 | conda create -n project_env python=3.11 22 | 23 | # view available environments 24 | conda info --envs 25 | 26 | # activate environment 27 | conda activate project_env 28 | 29 | # deactivate environment 30 | (project_env) conda deactivate 31 | 32 | 33 | For machines really light on memory (e.g. Raspberry Pi), you can use 34 | `Virtualenv `_: 35 | 36 | .. code:: bash 37 | 38 | # install library if not already 39 | pip install virtualenv 40 | 41 | # create virtual environment (creates folder called project_env) 42 | python3 -m venv project_env 43 | 44 | # activate virtual environment 45 | source project_env/bin/activate 46 | 47 | # deactivate virtual environment 48 | (project_env) deactivate 49 | 50 | Note that when the virtual environment is activated, it will 51 | typically appear in parenthesis in the command line. 52 | 53 | In this project we recommend using Poetry, and while Poetry creates a virtual 54 | environment per project (as described `here `_), 55 | I typically stick to using one of the above methods for creating virtual environments 56 | (e.g. to use conda, to share environments with other projects, to use in notebooks). 57 | You can identify the location and info of the Poetry virtual environment by running: 58 | 59 | .. code:: bash 60 | 61 | poetry env info 62 | 63 | You can also configure to create the virtual environment within the project folder: 64 | 65 | .. code:: bash 66 | 67 | poetry config virtualenvs.in-project true 68 | 69 | 70 | Sharing your environment 71 | ------------------------ 72 | 73 | Inside your virtual environment, you can install packages specific to 74 | your project. It is highly recommended to keep track of the packages 75 | you install, so that others (including yourself) can easily recreate 76 | the same virtual environment. There are three common approaches to 77 | storing and keeping track of packages: 78 | 79 | * ``requirements.txt``: This is a simple text file that lists all the 80 | packages you have installed. You can create this file by running: 81 | 82 | .. code:: bash 83 | 84 | (project_env) pip freeze > requirements.txt 85 | 86 | You can then install all the packages in this file by running: 87 | 88 | .. code:: bash 89 | 90 | (project_env) pip install -r requirements.txt 91 | 92 | 93 | * ``environment.yml``: This is a YAML file that lists all the packages you have installed. You can create this file by running: 94 | 95 | .. code:: bash 96 | 97 | (project_env) conda env export > environment.yml 98 | 99 | You can simulatenously create the environment and install all the packages in this file by running: 100 | 101 | .. code:: bash 102 | 103 | conda env create -f environment.yml 104 | 105 | You can check that the environment was created by running: 106 | 107 | .. code:: bash 108 | 109 | conda env list 110 | 111 | The name of the environment is specified at the top of ``environment.yml``. 112 | 113 | Note that this approach is specific to Anaconda / Miniconda. More 114 | information can be found 115 | `here `_. 116 | -------------------------------------------------------------------------------- /pydevtips/fftconvolve.py: -------------------------------------------------------------------------------- 1 | from abc import abstractmethod 2 | 3 | import numpy as np 4 | 5 | 6 | class FFTConvolveBase: 7 | """Abstract class for FFT convolve.""" 8 | 9 | def __init__(self, filt, length) -> None: 10 | """ 11 | Base class for creating a convolver that uses the same filter. 12 | 13 | Parameters 14 | ---------- 15 | filt : :py:class:`~numpy.ndarray` 16 | Filter to convolve with. Must be real. 17 | length : int 18 | Length of the signal to convolve with. 19 | """ 20 | 21 | assert isinstance(filt, np.ndarray) 22 | self.filter = filt 23 | self.signal_length = length 24 | self.pad_length = len(filt) + length - 1 25 | self.filter_frequency_response = self._compute_filter_frequency_response() 26 | 27 | @abstractmethod 28 | def _compute_filter_frequency_response(self) -> np.ndarray: 29 | """ 30 | Compute the filter frequency response. 31 | 32 | Parameters 33 | ---------- 34 | filter : :py:class:`~numpy.ndarray` 35 | Filter to compute the frequency response for. 36 | 37 | Returns 38 | ------- 39 | filter_frequency_response : :py:class:`~numpy.ndarray` 40 | Filter frequency response. 41 | """ 42 | raise NotImplementedError 43 | 44 | @abstractmethod 45 | def __call__(self, signal) -> np.ndarray: 46 | """Apply the filter to the signal, in the frequency domain.""" 47 | pass 48 | 49 | 50 | class RFFTConvolve(FFTConvolveBase): 51 | """Real FFT convolve.""" 52 | 53 | def __init__(self, filt, length) -> None: 54 | """ 55 | Create convolver that uses a real-valued filter. 56 | 57 | Parameters 58 | ---------- 59 | filt : :py:class:`~numpy.ndarray` 60 | Filter to convolve with. Must be real. 61 | length : int 62 | Length of the signal to convolve with. 63 | """ 64 | 65 | # check real 66 | assert np.isreal(filt).all(), "Filter must be real." 67 | super(RFFTConvolve, self).__init__(filt, length) 68 | 69 | def _compute_filter_frequency_response(self): 70 | """Compute the filter frequency response.""" 71 | return np.fft.rfft(self.filter, n=self.pad_length) 72 | 73 | def __call__(self, signal) -> np.ndarray: 74 | """ 75 | Apply the real-valued filter to the signal, in the frequency domain. 76 | 77 | Parameters 78 | ---------- 79 | signal : :py:class:`~numpy.ndarray` 80 | Signal to convolve with. Must be real. 81 | 82 | Returns 83 | ------- 84 | result : :py:class:`~numpy.ndarray` 85 | Convolved signal. 86 | """ 87 | assert np.isreal(signal).all(), "Signal must be real." 88 | signal_frequency_response = np.fft.rfft(signal, n=self.pad_length) 89 | return np.fft.irfft( 90 | signal_frequency_response * self.filter_frequency_response, n=self.pad_length 91 | ) 92 | 93 | 94 | class FFTConvolve(FFTConvolveBase): 95 | """General FFT convolve.""" 96 | 97 | def __init__(self, filter, length) -> None: 98 | """ 99 | Create convolver that uses a fixed filter. 100 | 101 | Parameters 102 | ---------- 103 | filter : :py:class:`~numpy.ndarray` 104 | Filter to convolve with. Must be real. 105 | length : int 106 | Length of the signal to convolve with. 107 | """ 108 | super(FFTConvolve, self).__init__(filter, length) 109 | 110 | def _compute_filter_frequency_response(self): 111 | """Compute the filter frequency response.""" 112 | return np.fft.fft(self.filter, n=self.pad_length) 113 | 114 | def __call__(self, signal) -> np.ndarray: 115 | """ 116 | Apply the filter to the signal, in the frequency domain. 117 | 118 | Parameters 119 | ---------- 120 | signal : :py:class:`~numpy.ndarray` 121 | Signal to convolve with. 122 | 123 | Returns 124 | ------- 125 | result : :py:class:`~numpy.ndarray` 126 | Convolved signal. 127 | """ 128 | signal_frequency_response = np.fft.fft(signal, n=self.pad_length) 129 | return np.fft.ifft( 130 | signal_frequency_response * self.filter_frequency_response, n=self.pad_length 131 | ) 132 | -------------------------------------------------------------------------------- /docs/source/testing.rst: -------------------------------------------------------------------------------- 1 | Testing 2 | ======= 3 | 4 | *"If debugging is the process of removing bugs, then programming must 5 | be the process of putting them in."* 6 | 7 | *-- Edsger Dijkstra* 8 | 9 | But there are ways to make it easier to find bugs and prevent the introduction 10 | of new ones. This is where testing comes in. There are two types that I often use 11 | (discussed below): 12 | 13 | * `Assertion tests `__. 14 | * `Unit tests `__. 15 | 16 | We also the discuss the use of `GitHub Actions `__. 17 | for continuous integration (CI), namely the software practice of committing code 18 | to a shared repository where the project is built and tested. 19 | 20 | 21 | Assertion tests 22 | --------------- 23 | 24 | Assertion tests are lightweight Boolean checks that you can include in your code 25 | to check that certain conditions are met. For example, you can check that the 26 | input/output is what you expect. If the condition is not met, the 27 | test will fail and code execution will stop. 28 | 29 | For example, for :class:`pydevtips.fftconvolve.RFFTConvolve` we check that the 30 | input is indeed real: 31 | 32 | .. code:: python 33 | 34 | def __init__(self, filt, length) -> None: 35 | assert np.isreal(filt).all(), "Filter must be real." 36 | ... 37 | 38 | Unit tests 39 | ---------- 40 | 41 | Unit testing is a method to test small pieces of code, usually functions. With 42 | a large code base, having unit tests can ensure you don't break core functionality 43 | when you make changes. 44 | 45 | In Python, there is the `pytest `__ package 46 | that can be used to write and run unit tests. A common practice is to create a 47 | `tests` folder in the root of your project and write your tests there. The test 48 | functions should begin with `test_` so that ``pytest`` can find them. 49 | 50 | For example, for our FFT convolvers -- :class:`pydevtips.fftconvolve.RFFTConvolve` 51 | and :class:`pydevtips.fftconvolve.FFTConvolve` -- we can write unit tests to 52 | check that they are consistent with :py:func:`numpy.convolve`, as done in 53 | `this script `__. 54 | 55 | To run the unit tests: 56 | 57 | .. code:: bash 58 | 59 | # install in virtual environment (if not done already) 60 | # -- pytest in the dev group 61 | (project_env) poetry install --with dev 62 | 63 | # run tests 64 | (project_env) poetry run pytest 65 | 66 | # -- if not using Poetry 67 | # (project_env) pip install pytest 68 | # (project_env) pytest 69 | 70 | To run a specific test: 71 | 72 | .. code:: bash 73 | 74 | # inside virtual environment 75 | (project_env) poetry run pytest tests/test_fftconvolve.py::test_fft 76 | 77 | # -- if not using Poetry 78 | # (project_env) pytest tests/test_fftconvolve.py::test_fft 79 | 80 | 81 | Continuous integration with GitHub Actions 82 | ------------------------------------------ 83 | 84 | Continuous integration (CI) is the practice of automatically building and testing 85 | code whenever a change is made to the codebase. This is useful to ensure that 86 | the codebase is always in a working state. 87 | 88 | With GitHub Actions, you can set up a workflow that will run, e.g. on every push to 89 | the repository. This workflow can build the package, run the unit tests, build the 90 | documentions, etc for different versions of Python and operating systems. 91 | 92 | Workflows are defined in YAML files in the ``.github/workflows`` folder. For example, 93 | the workflow for this project is defined in `this file `__, 94 | whose code is shown below: 95 | 96 | .. literalinclude:: ../../.github/workflows/poetry.yml 97 | :caption: poetry.yml 98 | :linenos: 99 | 100 | The workflow performs the following: 101 | 102 | * (Lines 5-12) Triggers on pushes and pull requests to the ``main`` branch. 103 | * (Lines 21-24) Performs the test on all combinations of Python versions (3.10 and 3.11) and operating systems 104 | Ubuntu, Windows, and macOS. 105 | * (Lines 33-43) Installs Python, Poetry, and the package with its dependencies. 106 | * (Lines 44-56) Checks for code formatting and style and errors if it doesn't conform. 107 | *Make sure it matches the code formatting you've setup in your project, e.g. via* :ref:`pre-commit hooks `. 108 | * (Lines 57-58) Runs the unit tests. 109 | 110 | An older version of the workflow (not using Poetry but rather ``setup.py`` with 111 | ``setuptools``) can be found below: 112 | 113 | .. literalinclude:: ../../.github/workflows/setuptools.yml 114 | :caption: setuptools.yml (OLD WAY) 115 | :linenos: 116 | 117 | More information on configuring GitHub Actions can be found in `their documentation `__. 118 | -------------------------------------------------------------------------------- /docs/source/reproducible.rst: -------------------------------------------------------------------------------- 1 | Reproducible examples 2 | ===================== 3 | 4 | Examples and tutorials are a great way to showcase your software. However, 5 | for them to be useful, they need to be reproducible. There are a few ways 6 | you can ensure this is the case: 7 | 8 | * Providing clear instructions to install the software and its dependencies, 9 | as described :ref:`here`. 10 | * Providing clear instructions to run the example and download any necessary data, 11 | e.g. in the README or in the home page of the documentation. 12 | * Using notebooks and scripts to showcase your software. See :ref:`below` for more details. 14 | * Using configuration management tools, such as `Hydra `_, 15 | for separating the configuration of an experiment from the code. See 16 | :ref:`below` for more details. 17 | 18 | Using notebooks and scripts 19 | --------------------------- 20 | 21 | Jupyter notebooks are a great way to showcase your software. They allow you to 22 | combine code, text, and images in a single document. However, they can be 23 | prone to errors when not running cells in order. Moreover, they are a pain 24 | when it comes to version control, as they are not plain text files and lend 25 | to many lines of code being changed when only a few lines are actually 26 | changed. 27 | 28 | Scripts on the other hand are plain text files and are easy to version 29 | control. Moreover, they are less prone to errors as everything is run in 30 | order, and you are less prone to have missing or incorrectly defined 31 | variables. 32 | 33 | Both notebooks and scripts have their place. My recommendation is to: 34 | 35 | #. (Optionally) Use notebooks for interactive development. 36 | #. Convert the notebook into a script / scripts + files with utility functions, and continue development. 37 | #. Commit the script(s) to your repository. 38 | #. Create a polished notebook to showcase your software with code, text, and images. 39 | #. Commit the notebook to your repository. 40 | 41 | In this manner, you can avoid the annoyances of notebooks with version control, 42 | while still being able to showcase your software in a polished notebook. 43 | 44 | Separating configuration from code 45 | ---------------------------------- 46 | 47 | Relying on scripts can still lead to pesky little commits and code duplication, 48 | e.g. when you want to change a parameter or showcase different results from the 49 | same functionality. This is where configuration management tools, such as 50 | `Hydra `_, can come in handy. 51 | 52 | Hydra allow you to separate the configuration of an experiment from the code, 53 | through a separate configuration YAML file. This allows you to change the 54 | parameters inside the configuration file(s) without having to change the code. 55 | This is particularly useful when you want to demonstrate different scenarios, e.g. 56 | when you have have a script that trains a model and a configuration file that 57 | specifies the parameters of the model / training. Moreover, having a single 58 | configuration file that exposes all the relevant parameters can be much more 59 | convenient than having to dig through the code to find them. 60 | 61 | The script `examples/real_convolve.py `_ 62 | shows how to use Hydra, namely adding a decorate to the function: 63 | 64 | .. code-block:: python 65 | 66 | @hydra.main(version_base=None, config_path="configs", config_name="defaults") 67 | def main(config): 68 | # `config` is a dictionary with the parameters specified in the YAML file. 69 | # e.g. `config.seed` 70 | 71 | Running ``python examples/real_convolve.py`` will run the script with the default parameters 72 | from the file ``configs/defaults.yaml``. You can also specify a different configuration 73 | file, e.g. ``python examples/real_convolve.py -cn exp1`` will run the script with the 74 | file ``configs/exp1.yaml``. Instead of creating a new configuration file, you can also 75 | specify the parameters directly through the command line, e.g. 76 | ``python examples/real_convolve.py filter_len=15``. 77 | 78 | One handy thing about Hydra is that it creates a folder ``outputs`` with the 79 | timestamp of the run, and saves the configuration file used in that run. You 80 | can also save the results of the run in that folder: 81 | 82 | .. code-block:: python 83 | 84 | @hydra.main(version_base=None, config_path="configs", config_name="defaults") 85 | def main(config): 86 | # ... 87 | # Save the results 88 | np.save(os.path.join(os.getcwd(), "results.npy"), results) 89 | 90 | This makes Hydra a great tool for keeping tracking of experiment runs and their 91 | parameters, while limiting changes to the code (and having new code commits). 92 | Check out `this blog post `_ for more on Hydra. 93 | 94 | Setting the seed 95 | ---------------- 96 | 97 | Setting the seed is important for reproducibility. You can set the seed in the 98 | configuration file and use it the script like so: 99 | 100 | .. code-block:: python 101 | 102 | @hydra.main(version_base=None, config_path="configs", config_name="defaults") 103 | def main(config): 104 | 105 | # Set the seed for numpy 106 | np.random.seed(config.seed) 107 | # application-specific seed setting 108 | 109 | -------------------------------------------------------------------------------- /docs/source/clean_code.rst: -------------------------------------------------------------------------------- 1 | Clean(er) code 2 | ============== 3 | 4 | Below are a few tips for writing clean(er) code. 5 | 6 | Code formatting 7 | --------------- 8 | 9 | Code formatting is important for readability. It allows you to write code that 10 | is easy to follow (for your future self as much as others), and (least importantly) 11 | adheres to the `PEP8 `_ standard. 12 | 13 | However, remembering all the rules and manually formatting your code is not 14 | how we want to spend our time as developers. To this end, there are several 15 | tools that can help us with this task. The ones we use are: 16 | 17 | * `Black `_ which will reformat your code 18 | in-place to conform to the PEP8 standard. 19 | * `Flake8 `_ which is a *linter* that 20 | will check your code for errors and style violations, but not reformat it. For 21 | example, for me it has identified code where I have unused variables or 22 | scripts / functions that are too long. 23 | * `isort `_ which will sort your imports 24 | alphabetically and group them by type. 25 | 26 | There are many alternatives for there tools. An increasingly popular alternative 27 | is `ruff `_, which is written in Rust and is meant 28 | to replace Flake8, Black, and isort. 29 | 30 | While you can use these tools manually, it is much more convenient to use them 31 | as pre-commit hooks. This means that before you commit your code, these tools 32 | will be run automatically. If they find any errors, the commit will be aborted 33 | and you will have to fix the errors before you can commit again. Pre-commit 34 | helps you to automate this process and avoid commits that do not conform to 35 | the PEP8 standard (and commits that are just for formatting). 36 | 37 | A few files are needed to setup pre-commit hooks: 38 | 39 | * `.pre-commit-config.yaml `_: This file contains the configuration for the 40 | pre-commit hooks. It specifies which tools to use, and how to use them. 41 | * `.flake8 `_: This file contains the configuration for Flake8. It specifies 42 | e.g. which errors to ignore, and which line length to use. 43 | * `pyproject.toml `_: This file contains the configuration for Black and isort. It 44 | specifies e.g. which line length to use. 45 | 46 | You can then install the pre-commit hooks for your project by running the 47 | following commands: 48 | 49 | .. code:: bash 50 | 51 | # inside virtual environment 52 | # -- black, flake8, isort are in the dev group 53 | (project_env) poetry install --with dev 54 | 55 | # -- if not using Poetry 56 | # (project_env) pip install pre-commit black flake8 isort 57 | 58 | # Install git hooks 59 | (project_env) pre-commit install 60 | # pre-commit installed at .git/hooks/pre-commit 61 | 62 | More pre-commit hooks are available provided by `Poetry `_. 63 | 64 | 65 | Avoiding long ``if-else`` statements with object instantiation 66 | -------------------------------------------------------------- 67 | 68 | In :ref:`Reproducible examples` we presented Hydra for separating configuration from code. 69 | Another cool feature of Hydra is `object instantiating `_. 70 | Imagine you want to try different optimizers for your Deep Neural Network (DNN) or you want to try different DNNs in the same pipeline. 71 | Instead of doing ``if-else`` statements, you write one line of code and let Hydra choose the 72 | appropriate object class based on your configuration. See the script 73 | `examples/real_convolve.py `_ 74 | for the example. 75 | 76 | .. code-block:: python 77 | 78 | @hydra.main(version_base=None, config_path="configs", config_name="defaults") 79 | def main(config): 80 | # instantiate object from config 81 | signal = instantiate(config.signal) 82 | # application specific choice of object class 83 | 84 | ``instantiate`` function from ``hydra.utils`` allows you to define an object in a YAML file 85 | without being tied to a particular class. To do this, you need to define ``_target_`` in 86 | your config (see configs in ``configs/signal``) and object initialization arguments. Object class 87 | can be either defined in your project (``configs/signal/ExampleZeros``, ``configs/signal/ExampleCustom``) 88 | or taken from a package (``configs/signal/ExampleNumpy``). 89 | 90 | Note that here we use another Hydra feature: config grouping and splitting. Instead of writing 91 | configurations for all objects in the main config and copying configuration files, we create a sub-directory ``signal``, 92 | where all ``signal`` configs are defined. Now we can run the main config with the ``signal`` of 93 | our choice simply by specifying it in the command line. For example, ``python examples/real_convolve.py signal=ExampleNumpy`` 94 | or ``python examples/real_convolve.py signal=ExampleZeros``. 95 | 96 | If we need to define some of the arguments inside the code before creating an object, we can pass them directly to the ``instantiate`` function. 97 | For example, we did not define ``signal_len`` in the ``signal`` configuration file and passed it by hand: 98 | ``signal = instantiate(config.signal, config.signal_len)``. This is especially useful when you have positional-only arguments 99 | like ``numpy.random.randn`` in our example. Note that we can both define arguments in the configuration file and pass new ones to ``instantiate`` like we did for 100 | ``ExampleCustom``. 101 | 102 | Object instantiating is recursive, i.e. some of the arguments of the class can also be 103 | defined using ``_target_`` and they will be created automatically. For example, 104 | ``python examples/real_convolve.py signal=ExampleCustom +signal/transform=power`` defines the ``transform`` argument of 105 | the ``ExampleCustom`` class as the ``PowerTransform`` class. The ``+signal/transform=power`` in the command line 106 | means adding the ``transform`` argument to the current ``signal`` configuration from the ``power.yaml`` config defined 107 | in ``configs/signal/transform``. That is, you can have sub-sub-directories. The default values from sub-sub-directories 108 | can also be changed in the command-line: ``python examples/real_convolve.py signal=ExampleCustom +signal/transform=power signal.transform.pow=3`` 109 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | *************************************** 2 | pydevtips: Python Development Tips 3 | *************************************** 4 | 5 | .. image:: https://img.shields.io/badge/GitHub-100000?style=for-the-badge&logo=github&logoColor=white 6 | :target: https://github.com/ebezzam/python-dev-tips 7 | :alt: GitHub page 8 | 9 | .. image:: https://readthedocs.org/projects/pydevtips/badge/?version=latest 10 | :target: http://pydevtips.readthedocs.io/en/latest/ 11 | :alt: Documentation Status 12 | 13 | .. image:: https://github.com/ebezzam/python-dev-tips/actions/workflows/poetry.yml/badge.svg 14 | :target: https://github.com/ebezzam/python-dev-tips/blob/main/.github/workflows/poetry.yml 15 | :alt: Unit tests and formatting 16 | 17 | .. image:: https://img.shields.io/badge/YouTube-%23FF0000.svg?style=for-the-badge&logo=YouTube&logoColor=white 18 | :target: https://youtu.be/BmTBQicddqc?si=cnVcbo36EFNh6dg7 19 | :alt: Recording 20 | 21 | .. image:: https://img.shields.io/badge/Google_Slides-yellow 22 | :target: https://docs.google.com/presentation/d/1D1_JywMl2rjaeuVzpykPBOJsDIuwQKGOJB4EFZjej2s/edit#slide=id.g2eaa4b61f15_0_1346 23 | :alt: Slides 24 | 25 | 26 | .. .. |ss| raw:: html 27 | 28 | .. 29 | 30 | .. .. |se| raw:: html 31 | .. 32 | 33 | | 34 | 35 | Reproducibility is important for software: *if it's not reproducible, it's not useful*. 36 | Even if you don't plan on sharing your code, imagine 37 | coming back to a project after a few weeks, or having 38 | to install it on a new machine. You'll be all the more thankful to your 39 | past self if you have a clear way to install and run your code. 40 | 41 | This repository is a collection of tips and tricks for developing stable 42 | and reproducible Python code. There is a slight focus on scientific 43 | computing, but the general principles can apply to most Python projects. 44 | If you're reading this from `GitHub `_, please check out the 45 | `documentation `_ for a 46 | more in-depth explanation of the topics covered. 47 | 48 | The intended audience is myself (as I often find myself going to past 49 | projects to find how I did something!), but also for students and 50 | anyone who is interested in learning some new tricks or even 51 | sharing their own! I try to follow the principles laid out here on 52 | development and reproducibility, so feel free to point out any lapses 53 | or suggest improvements, either by opening an issue or pull request. 54 | 55 | As is typical in open source, there are many ways to do the same thing. 56 | But hopefully this gives you a starting point. Feel free to pick and 57 | choose the features that you like. This flexibility is one of the best 58 | (and worst parts) of open source. Some of the things we cover: 59 | 60 | * Packaging and distribution with `Poetry `_. 61 | * Virtual environments with `Conda `_ and `virtualenv `_. 62 | * Version control with Git/GitHub. 63 | * Reproducible examples with `Hydra `_. 64 | * Documentation with `Sphinx `_. 65 | * Code formatting with Black, Flake8, and isort. 66 | * Unit tests and continuous integration with `pytest `_ and `GitHub Actions `_. 67 | * Remote development with `VS Code ` and `SSH `_. 68 | 69 | The accompanying 70 | `slides `__ 71 | and `video `__ 72 | are from a tutorial given at LauzHack's `Deep Learning Bootcamp `__. 73 | Feel free to modify and use it for your own purposes. 74 | 75 | .. note:: 76 | 77 | A good amount of this documentation and code is written with `GitHub 78 | Copilot `_, which I highly recommend for development. If you don't like 79 | writing documentation, it is a great way to get started as it is able to 80 | understand the functionality of your code and produce meaningful text to describe it. 81 | It should be used be used with caution, *but it can be a great tool for getting started* 82 | and you often you need to make a few tweaks (*like the previous repetition*). 83 | But it's a huge time-saver! 84 | 85 | Installation 86 | ============ 87 | 88 | This "dummy" package can be installed with pip: 89 | 90 | .. code:: bash 91 | 92 | pip install pydevtips 93 | 94 | Or from source. Firstly, Poetry must be installed: https://python-poetry.org/docs/#installation 95 | 96 | .. code:: bash 97 | 98 | # -- recommend using pipx 99 | pipx install poetry 100 | 101 | # -- or with official installer 102 | # curl -sSL https://install.python-poetry.org | python3 - 103 | 104 | # I recommend creating virtual environment in the project folder 105 | poetry config virtualenvs.in-project true 106 | 107 | 108 | Then the following commands can be run: 109 | 110 | .. code:: bash 111 | 112 | # get source code 113 | git clone git@github.com:ebezzam/python-dev-tips.git 114 | cd python-dev-tips 115 | 116 | # create new environment, press enter to accept 117 | # -- important to set python version, otherwise `python` executable may not exist 118 | # -- (would be `python3` instead) 119 | conda create -n project_env python=3.11 120 | 121 | # view available environments 122 | conda info --envs 123 | 124 | # activate environment 125 | conda activate project_env 126 | 127 | # install package locally 128 | (project_env) poetry install --with dev 129 | 130 | # install pre-commit hooks 131 | (project_env) pre-commit install 132 | 133 | # run tests 134 | (project_env) pytest 135 | 136 | # deactivate environment 137 | (project_env) conda deactivate 138 | 139 | Or without ``conda`` and just Poetry: 140 | 141 | .. code:: bash 142 | 143 | # get source code 144 | git clone git@github.com:ebezzam/python-dev-tips.git 145 | cd python-dev-tips 146 | 147 | # install package locally 148 | poetry install --with dev 149 | # -- `poetry env info` to see where the virtual environment is located 150 | 151 | # install pre-commit hooks 152 | poetry run pre-commit install 153 | 154 | # run tests 155 | poetry run pytest 156 | 157 | ## NOTE that Python related commands need to be run 158 | ## with `poetry run` or by activating the virtual environment 159 | ## https://python-poetry.org/docs/basic-usage/#activating-the-virtual-environment 160 | 161 | Examples 162 | ======== 163 | 164 | Examples can be found in the ``examples`` and ``notebooks`` folders. 165 | Scripts from the ``examples`` folder should be run from the root of the 166 | repository, e.g.: 167 | 168 | .. code:: bash 169 | 170 | python examples/real_convolve.py 171 | 172 | Parameter setting is done with `hydra `_. More on that 173 | in the `Reproducible examples `_ 174 | section of the documentation. 175 | 176 | 177 | TODO 178 | ==== 179 | 180 | - switch to ruff for code formatting: https://docs.astral.sh/ruff/ 181 | - numba: https://numba.pydata.org/ 182 | - picking a license 183 | - change documentation links to main branch 184 | - github page 185 | - point out features in scripts: object-oriented, asserts, tqdm, type hints 186 | - manifest file to not include file in package 187 | - GitHub actions for releasing to PyPi when changes to version 188 | - pytorch compatible 189 | - Cython / C++ 190 | -------------------------------------------------------------------------------- /docs/source/documentation.rst: -------------------------------------------------------------------------------- 1 | Documentation 2 | ============= 3 | 4 | *"Documentation is a love letter that you write to your future self."* 5 | 6 | *-- Damian Conway* 7 | 8 | Documentation is a critical part of any software project. It is the first 9 | point of contact for new users, and it is the first thing developers look 10 | for when they need to understand how a piece of code works. It is 11 | also a great way to share your knowledge with the community. 12 | 13 | *As a minimum*, it's good to have a README file that describes: 14 | 15 | * What the project is about. 16 | * How to install the project. 17 | * How to use the project. 18 | 19 | Inside your code, comments can be very useful to describe what a function or 20 | script does. Even better, is to add `docstrings `_ 21 | to your functions and classes. Docstrings are long-form 22 | comments beneath class and functions declaration that typically describe: 23 | 24 | * What the function or class does. 25 | * Its inputs and their data types. 26 | * Its outputs and their data types. 27 | 28 | For example: 29 | 30 | .. code:: Python 31 | 32 | def add(a, b): 33 | """ 34 | Add two integers. 35 | 36 | Parameters 37 | ---------- 38 | a : int 39 | First integer. 40 | b : int 41 | Second integer. 42 | 43 | Returns 44 | ------- 45 | result : int 46 | Sum of inputs. 47 | 48 | """ 49 | 50 | assert isinstance(a, int) 51 | assert isinstance(b, int) 52 | return a + b 53 | 54 | A documentation generator, such as `Sphinx `__, 55 | can then be used to render your docstrings into a static website, as shown at 56 | :func:`pydevtips.utils.add` for the above example. 57 | 58 | 59 | This website can be hosted *for free* on `ReadTheDocs `__ (like this!) 60 | or on GitHub pages. It is also possible to host the documentation on your own 61 | server. 62 | 63 | Below we describe how to setup your project to generate documentation with Sphinx, 64 | and how to host it on ReadTheDocs. 65 | 66 | Initial setup 67 | ------------- 68 | 69 | So you've diligently written your docstrings (or used GitHub Copilot!), and you 70 | want to generate your documentation. 71 | 72 | Here are some recommended steps to get started with Sphinx: 73 | 74 | #. Create a lightweight virtual environment for building the documentation. This can save a lot of time when building and publishing documentation remotely, as we'll show with `ReadTheDocs `__. 75 | 76 | .. code:: bash 77 | 78 | # create new environment, press enter to accept 79 | conda create -n docs_env python=3.9 80 | 81 | # activate environment 82 | conda activate docs_env 83 | 84 | # install requirements stored in `docs` 85 | # (may differ from our file depending on your needs) 86 | (docs_env) cd docs 87 | (docs_env) pip install -r requirements.txt 88 | 89 | #. Inside your virtual environment, run ``sphinx-quickstart`` to creates a lot of the boilerplate configurations. This will guide you through a set of questions, such as your project details. We recommend creating separate ``source`` and ``build`` directories. 90 | #. Build the documentation, e.g. with ``make html`` (from inside the ``docs`` folder). 91 | #. Open ``docs/build/html/index.html`` in a browser to see your initial documentation! 92 | 93 | Editing your documentation 94 | -------------------------- 95 | 96 | The ``index.rst`` file will serve as the "homepage" for your documentation (built into ``index.html``). 97 | Typically, people have the same content as their README. Before you copy-and-paste 98 | the contents (!), you can directly insert the contents with the following line. 99 | 100 | .. code:: rst 101 | 102 | .. include:: ../../README.rst 103 | 104 | .. note:: 105 | 106 | `reStructuredText `__ 107 | is the default plaintext markup language used by Sphinx. At this point, you may be thinking: 108 | *"But my README is a Markdown file (.md)..."*. While there are `tools `__ 109 | to make Sphinx compatible with Markdown, I think you will save yourself more headaches to simply 110 | switch to reStructuredText. There are also `online tools `__ 111 | to help you with that. 112 | 113 | 114 | Adding new pages to your documentation amount to: 115 | 116 | #. Creating new RST files. 117 | #. Including them in your ``index.rst`` file. 118 | #. Rebuilding the documentation, e.g. with ``make html`` (from inside the ``docs`` folder). 119 | 120 | You may also need to edit the ``conf.py`` file to use different features. 121 | Check out our `index.rst `__ 122 | and `conf.py `__ 123 | files for example configurations. 124 | 125 | 126 | You can do a clean build of your documentation with the following commands: 127 | 128 | .. code:: bash 129 | 130 | # inside `docs` folder 131 | make clean 132 | make html 133 | 134 | 135 | Pro-tips 136 | -------- 137 | 138 | * Changing to the ReadTheDocs theme inside `conf.py `__. 139 | * `Intersphinx `__ for linking to other documentations. 140 | In the ``conf.py`` file: `add `__ 141 | the Sphinx extension, and `link `__ 142 | to the other documentation. Inside your documentation you can link to the other library, e.g. 143 | for data types: 144 | 145 | .. code:: Python 146 | 147 | ... 148 | 149 | """ 150 | Parameters 151 | ---------- 152 | filter : :py:class:`~numpy.ndarray` 153 | """ 154 | 155 | ... 156 | 157 | which renders as in :func:`pydevtips.fftconvolve.RFFTConvolve.__init__` 158 | with a clickable link to NumPy's documentation. 159 | * `Mock modules `__ to keep your documentation virtual environment light. 160 | * `Add the path `__ 161 | to your package, so that it doesn't have to be installed (again keeping your documentation environment light!). 162 | * `Automate year `__. 163 | * You can reference other sections in your documentation by their title, e.g. :ref:`like this ` with ``:ref:`like this ``. 164 | 165 | Publishing 166 | ---------- 167 | 168 | With a set of HTML files, there are many ways to publish your documentation online. 169 | We present one approach through `ReadTheDocs `__ (RTD), which is 170 | free and very popular among Python developers. Another popular free options is through 171 | `GitHub Pages `__. I prefer RTD to not have the GitHub username or 172 | organization in the documentation URL. 173 | 174 | To publish on RTD: 175 | 176 | #. Make an account: https://readthedocs.org/accounts/signup/ 177 | #. Import a project from the `dashboard `__. There are two ways to do this: (1) linking your GitHub account and selecting one of your **public** repositories, or (2) importing the project manually. When linking to GitHub, the documentation is re-built whenever there are changes to the selected branch. 178 | #. Check your project page for the build status. 179 | 180 | You can (optionally) define a `.readthedocs.yaml `__ 181 | file to ensure a build environment as close as possible to your local machine. 182 | -------------------------------------------------------------------------------- /docs/source/packaging.rst: -------------------------------------------------------------------------------- 1 | Packaging 2 | ========= 3 | 4 | Packaging your code is the process of organizing your project files 5 | so that it can be built and installed by other users. As well as 6 | your source code, there are a few files that are needed so that 7 | your project "pip-installable", i.e. can be installed with 8 | `pip `_ and uploaded to 9 | `PyPI `_. 10 | 11 | While the `Python Packaging Authority (PyPA) `_ 12 | provides a comprehensive `guide `_ 13 | on packaging Python projects, this project instead uses `Poetry `_. 14 | Poetry is one of many tools that can be used to manage Python packages and environments. 15 | It simplifies/automates a lot of small aspects than if we were to use 16 | approach described by PyPA. 17 | 18 | `uv `_ is another popular tool for Python packaging, 19 | written in Rust. 20 | 21 | 22 | Poetry setup 23 | ------------ 24 | 25 | As a first-time setup, Poetry needs to be installed. The process is described on 26 | `this page `_. I recommend using the 27 | ``pipx`` method: 28 | 29 | 1. Install ``pipx``: https://pipx.pypa.io/stable/installation/ 30 | 2. Install Poetry: ``pipx install poetry`` 31 | 32 | 33 | Project structure 34 | ----------------- 35 | 36 | A typical (minimal) file structure for a Python project to be 37 | "pip-installable" looks like so: 38 | 39 | .. code:: 40 | 41 | project/ 42 | |-- __init__.py 43 | |-- code.py 44 | |-- # other files 45 | README.rst # or .md, .txt, etc 46 | pyproject.toml # or setup.cfg or setup.py 47 | 48 | where: 49 | 50 | * The folder ``project`` contains your source code files and 51 | an ``__init__.py``, which could be empty. 52 | * ``README.rst`` which is not necessary for building the Python, but 53 | is often the first file that new users will look into to learn 54 | about the project, how to install it, and how to use it. On GitHub 55 | and PyPI, it will be rendered as the "homepage" for your project. 56 | * ``pyproject.toml`` is a configuration file for building the project. 57 | More on that :ref:`below `. In the past, Python projects 58 | used ``setup.py`` for this purpose, but ``pyproject.toml`` is becoming 59 | more and more the norm. As this project used to use ``setup.py``, 60 | more on that can be found :ref:`below `. 61 | 62 | 63 | Project Management and Packaging with Poetry 64 | --------------------------------------------- 65 | 66 | `This page `_ provides an 67 | overview of how to use Poetry: 68 | 69 | * For a `new project `_, 70 | you can run ``poetry new poetry-demo``. This will create a new folder with 71 | the project structure described above, including an empty ``tests`` folder. 72 | * For an `existing project `_, 73 | you can run ``poetry init``. Through the command line, you will be asked 74 | a series of questions to fill in the ``pyproject.toml`` file. 75 | 76 | The project can then be built (locally) with the command below: 77 | 78 | .. code:: bash 79 | 80 | (project_env) poetry install 81 | 82 | The project can be imported in your Python script as: 83 | 84 | .. code:: Python 85 | 86 | import project 87 | 88 | .. note:: 89 | One useful feature of Poetry is that with a **single command** you can 90 | install new dependencies and update the ``pyproject.toml`` file. 91 | For example, to install ``pandas`` and add to ``pyproject.toml``, 92 | you can run: 93 | 94 | .. code:: bash 95 | 96 | (project_env) poetry add pandas 97 | 98 | You can also create groups of dependencies, e.g. for development 99 | dependencies, by running: 100 | 101 | .. code:: bash 102 | 103 | (project_env) poetry add --group dev sphinx 104 | 105 | So that during installation, you can specify which group of 106 | dependencies: 107 | 108 | .. code:: bash 109 | 110 | (project_env) poetry install --with dev 111 | 112 | More on managing dependencies can be found `here `_. 113 | 114 | 115 | pyproject.toml 116 | -------------- 117 | 118 | Using a ``pyproject.toml`` file is not unique to Poetry, but is becoming 119 | more and more the norm for Python projects. Below is the ``pyproject.toml`` 120 | file for this project. 121 | 122 | .. literalinclude:: ../../pyproject.toml 123 | :caption: pyproject.toml 124 | :linenos: 125 | 126 | Everything except the sections labeled with "manually added" was 127 | automatically generated by Poetry. 128 | 129 | More info on configuring the ``pyproject.toml`` file can be found at the links below: 130 | 131 | * General configuration (`short `_ 132 | and `in-depth `_ guides). 133 | * Poetry configuration: https://python-poetry.org/docs/pyproject/ 134 | 135 | 136 | Publishing to PyPI with Poetry 137 | ------------------------------ 138 | 139 | Poetry also makes building and deploying to PyPI easy. 140 | `This article `_ 141 | sums up the process well: 142 | 143 | 1. (First-time) setup: 144 | 145 | * Create an account on PyPI: https://pypi.org/account/register/ 146 | * Create a token: https://pypi.org/manage/account/ 147 | * Add the token to Poetry: ``poetry config pypi-token.pypi `` 148 | 2. Update the version number in the ``pyproject.toml`` file. See `Semantic Versioning `__ for recommendations on picking version numbers. 149 | 3. Build the package: ``poetry build`` 150 | 4. Upload to PyPI: ``poetry publish``. Check https://pypi.org/project/pydevtips/ 🎉 151 | 152 | If there are issues in publishing to PyPI, you can check the logs with: 153 | 154 | .. code:: bash 155 | 156 | poetry publish --dry-run 157 | 158 | Or if there are issues with rendering the README file, you can check the logs with (``twine`` is required): 159 | 160 | .. code:: bash 161 | 162 | twine check dist/pydevtips-X.X.X.tar.gz # replace X.X.X with version number 163 | 164 | 165 | Creating a new release on GitHub 166 | -------------------------------- 167 | 168 | For a new release, you need to create a new tag on GitHub. This can be done with the commands below: 169 | 170 | .. code:: bash 171 | 172 | git tag -a X.X.X -m "version X.X.X" 173 | git push origin X.X.X 174 | 175 | You can create a new release with the following steps: 176 | 177 | #. From the `tags page `_, click (the rightmost) "..." dropdown menu. 178 | #. Select "Create release". 179 | #. At the bottom, press "Publish release". 180 | 181 | 182 | setup.py (old way) 183 | ------------------ 184 | 185 | The use of ``setup.py`` is an older way to Python projects, e.g, with ``setuptools``. 186 | Below is a previously-used ``setup.py`` file for this project. 187 | 188 | .. literalinclude:: ../../OLD_setup.py 189 | :caption: setup.py 190 | :linenos: 191 | 192 | * Lines 3-4: use the contents of ``README.rst`` to be rendered for the homepage 193 | on PyPI. *Be sure to set the correct file extension and set Line 13 194 | accordingly*. 195 | * Line 7: specifies the name of the package / in which folder the source code 196 | is located, such that the package can be installed with ``pip install pydevtips`` 197 | if on PyPI and imported as ``import pydevtips``. 198 | * Line 8: sets the package version, *which should be (typically) modified before 199 | uploading a new version to PyPI (below)*. 200 | * Line 9-10: for your name and contact info. 201 | * Line 20-26: specifies the Python version and package dependencies. 202 | 203 | The project can then be built (locally) with the command below: 204 | 205 | .. code:: bash 206 | 207 | (project_env) pip install -e . 208 | 209 | The project can be imported in your Python script as: 210 | 211 | .. code:: Python 212 | 213 | import project 214 | 215 | For a more in-depth description, check out `this article `_. 216 | 217 | Uploading your project to PyPI is traditionally done with the `twine `__ 218 | library. 219 | 220 | .. code:: bash 221 | 222 | # inside virtual environment 223 | (project_env) pip install twine 224 | 225 | In the steps below, replace "X.X.X" with the appropriate version number, *matching the one 226 | in your* ``setup.py`` *file*. See `Semantic Versioning `__ for 227 | recommendations on picking version numbers. 228 | 229 | .. code:: bash 230 | 231 | # edit version in `setup.py` 232 | # build package 233 | (project_env) python setup.py sdist bdist_wheel 234 | # -- creates zip in dist folder 235 | 236 | # upload to pypi 237 | (project_env) python -m twine upload dist/pydevtips-X.X.X.tar.gz 238 | # -- X.X.X is the version number in setup.py 239 | # -- enter username and password 240 | # -- check https://pypi.org/project/pydevtips/X.X.X/ 241 | 242 | # new tag on GitHub 243 | git tag -a X.X.X -m "version X.X.X" 244 | git push origin X.X.X 245 | 246 | If you project is hosted on GitHub, you can create a new release by: 247 | 248 | #. Clicking (the rightmost) "..." dropdown menu (from the `tags page `_). 249 | #. Selecting "Create release". 250 | #. At the bottom pressing "Publish release". 251 | --------------------------------------------------------------------------------