├── .editorconfig ├── .github ├── ISSUE_TEMPLATE.md └── workflows │ ├── docs.yml │ ├── latest-dependencies.yml │ └── tests.yml ├── .gitignore ├── AUTHORS.rst ├── CONTRIBUTING.md ├── DEVELOPMENT.md ├── HISTORY.md ├── LICENSE ├── MANIFEST.in ├── Makefile ├── PERPRIMITIVEDOCS.md ├── PRIMITIVES.md ├── README.md ├── USAGE.md ├── docs ├── Makefile ├── authors.rst ├── conf.py ├── contributing.rst ├── history.rst ├── images │ ├── dai-logo-white-200.png │ └── dai-logo-white.ico ├── index.rst ├── make.bat ├── readme.rst └── usage.rst ├── setup.cfg ├── setup.py ├── sigpro ├── __init__.py ├── aggregations │ ├── __init__.py │ ├── amplitude │ │ ├── __init__.py │ │ └── statistical.py │ ├── frequency │ │ ├── __init__.py │ │ └── band.py │ └── frequency_time │ │ └── __init__.py ├── basic_primitives.py ├── contributing.py ├── contributing_primitive.py ├── core.py ├── data │ └── demo_timeseries.csv ├── demo.py ├── pipeline.py ├── primitive.py ├── primitives │ └── sigpro │ │ ├── SigPro.json │ │ ├── aggregations │ │ ├── amplitude │ │ │ └── statistical │ │ │ │ ├── crest_factor.json │ │ │ │ ├── kurtosis.json │ │ │ │ ├── mean.json │ │ │ │ ├── rms.json │ │ │ │ ├── skew.json │ │ │ │ ├── std.json │ │ │ │ └── var.json │ │ └── frequency │ │ │ └── band │ │ │ ├── band_mean.json │ │ │ └── band_rms.json │ │ └── transformations │ │ ├── amplitude │ │ ├── identity │ │ │ └── identity.json │ │ └── spectrum │ │ │ └── power_spectrum.json │ │ ├── frequency │ │ ├── band │ │ │ └── frequency_band.json │ │ ├── fft │ │ │ ├── fft.json │ │ │ └── fft_real.json │ │ └── fftfreq │ │ │ └── fft_freq.json │ │ └── frequency_time │ │ └── stft │ │ ├── stft.json │ │ └── stft_real.json └── transformations │ ├── __init__.py │ ├── amplitude │ ├── __init__.py │ ├── identity.py │ └── spectrum.py │ ├── frequency │ ├── __init__.py │ ├── band.py │ ├── fft.py │ └── fftfreq.py │ └── frequency_time │ ├── __init__.py │ └── stft.py ├── tasks.py ├── tests ├── __init__.py ├── integration │ ├── __init__.py │ ├── test_contributing.py │ ├── test_contributing_primitive.py │ ├── test_core.py │ ├── test_demo.py │ ├── test_pipeline.py │ └── test_primitive.py ├── requirement_files │ └── latest_requirements.txt └── unit │ ├── __init__.py │ ├── aggregations │ ├── __init__.py │ ├── amplitude │ │ ├── __init__.py │ │ └── test_statistical.py │ └── frequency │ │ ├── __init__.py │ │ └── test_band.py │ ├── test___init__.py │ └── transformations │ ├── amplitude │ ├── __init__.py │ └── identity.py │ ├── frequency │ ├── __init__.py │ ├── test_fft.py │ └── test_fftfreq.py │ └── frequency_time │ ├── __init__.py │ └── test_stft.py ├── tox.ini └── tutorials ├── primitives_pipelines_tutorial.ipynb ├── primitives_pipelines_tutorial.md ├── sigpro_pipeline_demo.ipynb └── sigpro_pipeline_demo.md /.editorconfig: -------------------------------------------------------------------------------- 1 | # http://editorconfig.org 2 | 3 | root = true 4 | 5 | [*] 6 | indent_style = space 7 | indent_size = 4 8 | trim_trailing_whitespace = true 9 | insert_final_newline = true 10 | charset = utf-8 11 | end_of_line = lf 12 | 13 | [*.py] 14 | max_line_length = 99 15 | 16 | [*.bat] 17 | indent_style = tab 18 | end_of_line = crlf 19 | 20 | [LICENSE] 21 | insert_final_newline = false 22 | 23 | [Makefile] 24 | indent_style = tab 25 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | * SigPro version: 2 | * Python version: 3 | * Operating System: 4 | 5 | ### Description 6 | 7 | Describe what you were trying to get done. 8 | Tell us what happened, what went wrong, and what you expected to happen. 9 | 10 | ### What I Did 11 | 12 | ``` 13 | Paste the command(s) you ran and the output. 14 | If there was a crash, please include the traceback here. 15 | ``` 16 | -------------------------------------------------------------------------------- /.github/workflows/docs.yml: -------------------------------------------------------------------------------- 1 | name: Generate Docs 2 | 3 | on: 4 | push: 5 | branches: [ stable ] 6 | 7 | jobs: 8 | 9 | docs: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - uses: actions/checkout@v2 13 | 14 | - name: Python 15 | uses: actions/setup-python@v1 16 | with: 17 | python-version: 3.9 18 | 19 | - name: Build 20 | run: | 21 | python -m pip install --upgrade pip 22 | pip install -e .[dev] 23 | make docs 24 | - name: Deploy 25 | uses: peaceiris/actions-gh-pages@v3 26 | with: 27 | github_token: ${{secrets.GITHUB_TOKEN}} 28 | publish_dir: docs/_build/html 29 | -------------------------------------------------------------------------------- /.github/workflows/latest-dependencies.yml: -------------------------------------------------------------------------------- 1 | # name: Latest Dependency Checker 2 | # on: 3 | # schedule: 4 | # - cron: '0 * * * *' 5 | # workflow_dispatch: 6 | # jobs: 7 | # build: 8 | # runs-on: ubuntu-latest 9 | # steps: 10 | # - uses: actions/checkout@v2 11 | # - name: Set up Python 3.9 12 | # uses: actions/setup-python@v2 13 | # with: 14 | # python-version: 3.9 15 | # - name: Update dependencies 16 | # run: | 17 | # python -m pip install --upgrade pip 18 | # python -m pip install -e .[test] 19 | # make checkdeps OUTPUT_PATH=tests/requirement_files/latest_requirements.txt 20 | # - name: Create pull request 21 | # uses: peter-evans/create-pull-request@v3 22 | # with: 23 | # token: ${{ secrets.REPO_SCOPED_TOKEN }} 24 | # commit-message: Update latest dependencies 25 | # title: Automated Latest Dependency Updates 26 | # body: "This is an auto-generated PR with **latest** dependency updates." 27 | # branch: latest-dep-update 28 | # branch-suffix: short-commit-hash 29 | # base: master 30 | # team-reviewers: core 31 | -------------------------------------------------------------------------------- /.github/workflows/tests.yml: -------------------------------------------------------------------------------- 1 | name: Run Tests 2 | 3 | on: 4 | push: 5 | branches: [ '*' ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | lint: 11 | runs-on: ${{ matrix.os }} 12 | strategy: 13 | matrix: 14 | python-version: ['3.9'] 15 | os: [ubuntu-latest, macos-latest, windows-latest] 16 | steps: 17 | - uses: actions/checkout@v1 18 | - name: Set up Python ${{ matrix.python-version }} 19 | uses: actions/setup-python@v2 20 | with: 21 | python-version: ${{ matrix.python-version }} 22 | - name: Install package 23 | run: pip install invoke .[dev] 24 | - name: invoke lint 25 | run: invoke lint 26 | 27 | 28 | docs: 29 | runs-on: ${{ matrix.os }} 30 | strategy: 31 | matrix: 32 | python-version: ['3.9'] 33 | os: [ubuntu-latest] 34 | steps: 35 | - uses: actions/checkout@v1 36 | - name: Set up Python ${{ matrix.python-version }} 37 | uses: actions/setup-python@v2 38 | with: 39 | python-version: ${{ matrix.python-version }} 40 | - name: Install package 41 | run: pip install .[dev] 42 | - name: make docs 43 | run: make docs 44 | 45 | 46 | unit: 47 | runs-on: ${{ matrix.os }} 48 | strategy: 49 | matrix: 50 | python-version: ['3.9', '3.10', '3.11', '3.12'] 51 | os: [ubuntu-latest, macos-latest, windows-latest] 52 | steps: 53 | - uses: actions/checkout@v1 54 | - name: Set up Python ${{ matrix.python-version }} 55 | uses: actions/setup-python@v2 56 | with: 57 | python-version: ${{ matrix.python-version }} 58 | - if: matrix.os == 'windows-latest' && matrix.python-version == 3.6 59 | name: Install dependencies - Windows with Python 3.6 60 | run: python -m pip install pywinpty==2.0.1 61 | - name: Install package and dependencies 62 | run: pip install invoke .[test] 63 | - name: invoke pytest 64 | run: invoke pytest 65 | 66 | 67 | minimum: 68 | runs-on: ${{ matrix.os }} 69 | strategy: 70 | matrix: 71 | python-version: ['3.9', '3.10', '3.11', '3.12'] 72 | os: [ubuntu-latest, macos-latest] 73 | steps: 74 | - uses: actions/checkout@v1 75 | - name: Set up Python ${{ matrix.python-version }} 76 | uses: actions/setup-python@v2 77 | with: 78 | python-version: ${{ matrix.python-version }} 79 | - name: Install package and dependencies 80 | run: pip install invoke .[test] 81 | - name: invoke minimum 82 | run: invoke minimum 83 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | 58 | # Flask stuff: 59 | instance/ 60 | .webassets-cache 61 | 62 | # Scrapy stuff: 63 | .scrapy 64 | 65 | # Sphinx documentation 66 | docs/_build/ 67 | docs/api/ 68 | docs/sigpro.rst 69 | docs/sigpro.*.rst 70 | docs/modules.rst 71 | 72 | # PyBuilder 73 | target/ 74 | 75 | # Jupyter Notebook 76 | .ipynb_checkpoints 77 | 78 | # pyenv 79 | .python-version 80 | 81 | # celery beat schedule file 82 | celerybeat-schedule 83 | 84 | # SageMath parsed files 85 | *.sage.py 86 | 87 | # dotenv 88 | .env 89 | 90 | # virtualenv 91 | .venv 92 | venv/ 93 | ENV/ 94 | 95 | # Spyder project settings 96 | .spyderproject 97 | .spyproject 98 | 99 | # Rope project settings 100 | .ropeproject 101 | 102 | # mkdocs documentation 103 | /site 104 | 105 | # mypy 106 | .mypy_cache/ 107 | 108 | # other 109 | .DS_Store 110 | 111 | # Vim 112 | .*.swp 113 | -------------------------------------------------------------------------------- /AUTHORS.rst: -------------------------------------------------------------------------------- 1 | ======= 2 | Credits 3 | ======= 4 | 5 | * Plamen Valentinov 6 | * Carles Sala 7 | * Kalyan Veeramachaneni 8 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to SigPro 2 | 3 | Here you will find a series of steps that describe how to contribute a new implementation, fix or 4 | update to `SigPro` library. 5 | 6 | ## 1. Clone the repository 7 | 8 | The first step would be to clone the `SigPro` repository. In order to do so 9 | make sure that you have access to the repository by accessing it direcly 10 | [https://github.com/sintel-dev/SigPro/]( 11 | https://github.com/sintel-dev/SigPro/). 12 | 13 | If you have access to the repository and you have your `ssh` keys configured 14 | in your github account, you can clone it by using the following command 15 | 16 | ```bash 17 | git clone git@github.com:sintel-dev/SigPro.git 18 | ``` 19 | 20 | If you don't have your `ssh` keys configured you can clone the repository 21 | using your login name and password running the following command: 22 | 23 | ```bash 24 | git clone https://github.com/sintel-dev/SigPro 25 | ``` 26 | 27 | Next, you can enter your repository folder, create a virtualenv and install 28 | the project and the development dependencies. 29 | **Note**: You need to have virtualenv and virtualenvwrapper installed for 30 | these steps to work 31 | 32 | ```bash 33 | cd SigPro 34 | mkvirtualenv sigpro 35 | make install-develop 36 | ``` 37 | 38 | ## 2. Prepare your branch 39 | 40 | Before going any further, create the new `git` branch to which you will 41 | be pushing your development. 42 | 43 | To do so, type the following command with the desired name for your branch: 44 | 45 | ```bash 46 | git checkout -b 47 | ``` 48 | 49 | Try to use the naming scheme of prefixing your branch with issue-X where X is 50 | the associated issue, such as issue-3-fix-foo-bug. And if you are not 51 | developing on your own fork, further prefix the branch with your GitHub 52 | username, like githubusername/issue-3-fix-foo-bug. 53 | 54 | ## 3. Code your changes 55 | 56 | Once you have your branch ready and set, you can start coding your changes. 57 | 58 | When doing so, ensure that the code is placed in its corresponding python 59 | file, and if you think that this doesn't exist, create it in the location 60 | that you find more suitable. However, when it comes to `aggregations` and 61 | `tranformations`, those are bound to their type and subtype. 62 | 63 | Also, while hacking your changes, make sure to follow the [PEP8]( 64 | https://www.python.org/dev/peps/pep-0008/) style guide for python code 65 | and our `setup.cfg` (the main change is that we allow 99 characters per 66 | line). 67 | 68 | 69 | ## 4. Test your changes 70 | 71 | Now that you have implemented your changes, make sure to cover all your 72 | developments with the required unit tests, and that none of the old tests 73 | fail as consequence of your changes. For this, make sure to run the tests 74 | suite and check the code coverage by using the following commands: 75 | 76 | ```bash 77 | $ make lint # Check code styling 78 | $ make test # Run the tests 79 | $ make coverage # Get the coverage report 80 | ``` 81 | 82 | Once you have finished coding your changes, make sure to run the following 83 | command to ensure that your changes pass all the styling checks and tests 84 | including other Python supported versions using: 85 | 86 | ```bash 87 | $ make test-all 88 | ``` 89 | 90 | ## 5. Create a pull request 91 | 92 | Once you have created and tested your primitive, you can create a pull 93 | request by doing the following steps: 94 | 95 | 0. (You did this previoulsy, but make sure) Create a new branch or ensure you are in the correct branch. Run the command `git branch` to see at which branch you are pointing. If you are in the desired follow the next step. 96 | 1. Add the new files and the updated ones. By running `git status` you will see the modified and `new/untracked` files. Use `git add` to `add` the files that involve your implementation, such as the new primitive `json` file, the new module with the new transformation or aggregation and other changes that you may have done to existing files (such as `setup.py` if you updated or introduce a new dependency). 97 | 2. Commit your changes using `git commit -m "Implement my new transformation"`. 98 | 3. Push your branch: `git push --set-upstream origin `. 99 | 4. Go to [https://github.com/sintel-dev/SigPro/](https://github.com/sintel-dev/SigPro/) and create a pull request from this branch to the master branch. 100 | -------------------------------------------------------------------------------- /HISTORY.md: -------------------------------------------------------------------------------- 1 | # History 2 | 3 | ## 0.3.0 - 2025-02-17 4 | 5 | ### Features 6 | * Add Support for Python 3.12 & Remove 3.8 - [Issue #63](https://github.com/sintel-dev/SigPro/pull/63) by @sarahmish 7 | * Add fft frequency transformation - [Issue #62](https://github.com/sintel-dev/SigPro/pull/62) by @SaraPido 8 | * Band rms - [Issue #61](https://github.com/sintel-dev/SigPro/pull/61) by @SaraPido 9 | 10 | 11 | ## 0.2.1 - 2024-04-24 12 | 13 | ### Features 14 | 15 | * Frequency primitive - [Issue #60](https://github.com/sintel-dev/SigPro/pull/60) by @SaraPido 16 | 17 | 18 | ## 0.2.0 - 2024-02-02 19 | 20 | ### Features 21 | * Demo Notebooks for Pipeline usage - [Issue #55](https://github.com/sintel-dev/SigPro/issues/55) by @andyx13 22 | * Added `contributing_primitive` and `basic_primitives` module to assist with new primitive creation/usage - [Issue #54](https://github.com/sintel-dev/SigPro/issues/54) by @andyx13 23 | * Incorrect classification for stft.json and stft_real.json - [Issue #53](https://github.com/sintel-dev/SigPro/issues/53) by @andyx13 24 | * Support for more complex pipeline architectures - [Issue #52](https://github.com/sintel-dev/SigPro/issues/52) by @andyx13 25 | * Update primitive interfaces - [Issue #51](https://github.com/sintel-dev/SigPro/issues/51) by @andyx13 26 | * Syntax for pipeline creation - [Issue #41](https://github.com/sintel-dev/SigPro/issues/41) by @andyx13 27 | * Load demo dataset at random index - [Issue #35](https://github.com/sintel-dev/SigPro/issues/35) by @andyx13 28 | 29 | 30 | ## 0.1.2 - 2023-12-11 31 | 32 | ### Features 33 | * Python version update - [Issue #44](https://github.com/sintel-dev/SigPro/issues/44) by @andyx13 34 | * Add demo notebook and per-primitive documentation - [Issue #47](https://github.com/sintel-dev/SigPro/issues/47) by @andyx13 35 | 36 | 37 | ## 0.1.1 - 2023-04-06 38 | 39 | ### Features 40 | * Accepting single value data frame format - [Issue #36](https://github.com/sintel-dev/SigPro/issues/36) by @frances-h @sarahmish 41 | * Update demos - [Issue #26](https://github.com/sintel-dev/SigPro/pull/26) by @frances-h 42 | 43 | 44 | ## 0.1.0 - 2021-11-14 45 | 46 | ### Features 47 | * Rework SigPro to be class based 48 | 49 | 50 | ## 0.0.3 - 2021-09-27 51 | 52 | ### Features 53 | * Add `process_signals` function to take a collection of primitives and create features for the given data. 54 | 55 | 56 | ## 0.0.2 - 2021-02-05 57 | 58 | ### Bug Fixes 59 | 60 | * `MANIFEST.in`: copy the json files of the primitives with the package installation. 61 | 62 | 63 | ## 0.0.1 - 2021-01-26 64 | 65 | First release to PyPI. 66 | 67 | This release comes with the first version of the `contributing` module, which makes it easier 68 | to create new primitives and to test those with the demo data included in this package. 69 | 70 | This release also includes the following User Guides: 71 | 72 | * [PRIMITIVES.md](https://github.com/sintel-dev/SigPro/blob/master/PRIMITIVES.md): Information 73 | about the primitive families, their expected input and output. 74 | * [USAGE.md](https://github.com/sintel-dev/SigPro/blob/master/USAGE.md): Instructions about how 75 | to usee the three main functionalities of `SigPro`. 76 | * [DEVELOPMENT.md](https://github.com/sintel-dev/SigPro/blob/master/DEVELOPMENT.md): Step by step 77 | guide about how to write a valid `SigPro` primitive and contribute it to either `SigPro` or 78 | your own library. 79 | 80 | ### Features 81 | 82 | * Demo data: Available demo data to test primitives. 83 | * First primitives: The following list of primitives were added: 84 | * `sigpro.aggregations.amplitude.statistical.crest_factor` 85 | * `sigpro.aggregations.amplitude.statistical.kurtosis` 86 | * `sigpro.aggregations.amplitude.statistical.mean` 87 | * `sigpro.aggregations.amplitude.statistical.rms` 88 | * `sigpro.aggregations.amplitude.statistical.skew` 89 | * `sigpro.aggregations.amplitude.statistical.std` 90 | * `sigpro.aggregations.amplitude.statistical.var` 91 | * `sigpro.transformations.amplitude.identity.identity` 92 | * `sigpro.transformations.frequency.fft.fft` 93 | * `sigpro.transformations.frequency.fft.fft_real` 94 | * `sigpro.transformations.frequency_time.stft.stft` 95 | * `sigpro.transformations.frequency_time.stft.stft_real` 96 | * Contributing module. 97 | * Documentation on how to contribute new primitives and how to run those. 98 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020, MIT Data To AI Lab 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include AUTHORS.rst 2 | include CONTRIBUTING.md 3 | include HISTORY.md 4 | include LICENSE 5 | include README.md 6 | include sigpro/data/demo_timeseries.csv 7 | 8 | recursive-include sigpro *json 9 | 10 | recursive-include tests * 11 | recursive-exclude * __pycache__ 12 | recursive-exclude * *.py[co] 13 | 14 | recursive-include docs *.md *.rst conf.py Makefile make.bat *.jpg *.png *.gif 15 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .DEFAULT_GOAL := help 2 | 3 | define BROWSER_PYSCRIPT 4 | import os, webbrowser, sys 5 | 6 | try: 7 | from urllib import pathname2url 8 | except: 9 | from urllib.request import pathname2url 10 | 11 | webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1]))) 12 | endef 13 | export BROWSER_PYSCRIPT 14 | 15 | define PRINT_HELP_PYSCRIPT 16 | import re, sys 17 | 18 | for line in sys.stdin: 19 | match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line) 20 | if match: 21 | target, help = match.groups() 22 | print("%-20s %s" % (target, help)) 23 | endef 24 | export PRINT_HELP_PYSCRIPT 25 | 26 | BROWSER := python -c "$$BROWSER_PYSCRIPT" 27 | 28 | .PHONY: help 29 | help: 30 | @python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST) 31 | 32 | 33 | # CLEAN TARGETS 34 | 35 | .PHONY: clean-build 36 | clean-build: ## remove build artifacts 37 | rm -fr build/ 38 | rm -fr dist/ 39 | rm -fr .eggs/ 40 | find . -name '*.egg-info' -exec rm -fr {} + 41 | find . -name '*.egg' -exec rm -f {} + 42 | 43 | .PHONY: clean-pyc 44 | clean-pyc: ## remove Python file artifacts 45 | find . -name '*.pyc' -exec rm -f {} + 46 | find . -name '*.pyo' -exec rm -f {} + 47 | find . -name '*~' -exec rm -f {} + 48 | find . -name '__pycache__' -exec rm -fr {} + 49 | 50 | .PHONY: clean-docs 51 | clean-docs: ## remove previously built docs 52 | rm -f docs/api/*.rst 53 | -$(MAKE) -C docs clean 2>/dev/null # this fails if sphinx is not yet installed 54 | 55 | .PHONY: clean-coverage 56 | clean-coverage: ## remove coverage artifacts 57 | rm -f .coverage 58 | rm -f .coverage.* 59 | rm -fr htmlcov/ 60 | 61 | .PHONY: clean-test 62 | clean-test: ## remove test artifacts 63 | rm -fr .tox/ 64 | rm -fr .pytest_cache 65 | 66 | .PHONY: clean 67 | clean: clean-build clean-pyc clean-test clean-coverage clean-docs ## remove all build, test, coverage, docs and Python artifacts 68 | 69 | 70 | # INSTALL TARGETS 71 | 72 | .PHONY: install 73 | install: clean-build clean-pyc ## install the package to the active Python's site-packages 74 | pip install . 75 | 76 | .PHONY: install-test 77 | install-test: clean-build clean-pyc ## install the package and test dependencies 78 | pip install .[test] 79 | 80 | .PHONY: install-develop 81 | install-develop: clean-build clean-pyc ## install the package in editable mode and dependencies for development 82 | pip install -e .[dev] 83 | 84 | 85 | # LINT TARGETS 86 | 87 | .PHONY: lint 88 | lint: ## check style with flake8 and isort 89 | invoke lint 90 | 91 | .PHONY: fix-lint 92 | fix-lint: ## fix lint issues using autoflake, autopep8, and isort 93 | find sigpro -name '*.py' | xargs autoflake --in-place --remove-all-unused-imports --remove-unused-variables 94 | autopep8 --in-place --recursive --aggressive sigpro 95 | isort --apply --atomic --recursive sigpro 96 | 97 | find tests -name '*.py' | xargs autoflake --in-place --remove-all-unused-imports --remove-unused-variables 98 | autopep8 --in-place --recursive --aggressive tests 99 | isort --apply --atomic --recursive tests 100 | 101 | # TEST TARGETS 102 | 103 | .PHONY: test-unit 104 | test-unit: ## run tests quickly with the default Python 105 | invoke pytest 106 | 107 | .PHONY: test-readme 108 | test-readme: ## run the readme snippets 109 | invoke readme 110 | 111 | 112 | .PHONY: test-tutorials 113 | test-tutorials: ## run the tutorial notebooks 114 | invoke tutorials 115 | 116 | 117 | .PHONY: test 118 | test: test-unit test-readme test-tutorials ## test everything that needs test dependencies 119 | 120 | .PHONY: check-dependencies 121 | check-dependencies: ## test if there are any broken dependencies 122 | pip check 123 | 124 | .PHONY: test-devel 125 | test-devel: check-dependencies lint docs ## test everything that needs development dependencies 126 | 127 | .PHONY: test-all 128 | test-all: 129 | tox -r 130 | 131 | .PHONY: coverage 132 | coverage: ## check code coverage quickly with the default Python 133 | coverage run --source sigpro -m pytest 134 | coverage report -m 135 | coverage html 136 | $(BROWSER) htmlcov/index.html 137 | 138 | # DOCS TARGETS 139 | 140 | .PHONY: docs 141 | docs: clean-docs ## generate Sphinx HTML documentation, including API docs 142 | sphinx-apidoc --module-first --separate -T -o docs/api/ sigpro 143 | $(MAKE) -C docs html 144 | 145 | .PHONY: view-docs 146 | view-docs: docs ## view docs in browser 147 | $(BROWSER) docs/_build/html/index.html 148 | 149 | .PHONY: serve-docs 150 | serve-docs: view-docs ## compile the docs watching for changes 151 | watchmedo shell-command -W -R -D -p '*.rst;*.md' -c '$(MAKE) -C docs html' . 152 | 153 | 154 | # RELEASE TARGETS 155 | 156 | .PHONY: dist 157 | dist: clean ## builds source and wheel package 158 | python setup.py sdist 159 | python setup.py bdist_wheel 160 | ls -l dist 161 | 162 | .PHONY: publish-confirm 163 | publish-confirm: 164 | @echo "WARNING: This will irreversibly upload a new version to PyPI!" 165 | @echo -n "Please type 'confirm' to proceed: " \ 166 | && read answer \ 167 | && [ "$${answer}" = "confirm" ] 168 | 169 | .PHONY: publish-test 170 | publish-test: dist publish-confirm ## package and upload a release on TestPyPI 171 | twine upload --repository-url https://test.pypi.org/legacy/ dist/* 172 | 173 | .PHONY: publish 174 | publish: dist publish-confirm ## package and upload a release 175 | twine upload dist/* 176 | 177 | .PHONY: bumpversion-release 178 | bumpversion-release: ## Merge master to stable and bumpversion release 179 | git checkout stable || git checkout -b stable 180 | git merge --no-ff master -m"make release-tag: Merge branch 'master' into stable" 181 | bumpversion release 182 | git push --tags origin stable 183 | 184 | .PHONY: bumpversion-release-test 185 | bumpversion-release-test: ## Merge master to stable and bumpversion release 186 | git checkout stable || git checkout -b stable 187 | git merge --no-ff master -m"make release-tag: Merge branch 'master' into stable" 188 | bumpversion release --no-tag 189 | @echo git push --tags origin stable 190 | 191 | .PHONY: bumpversion-patch 192 | bumpversion-patch: ## Merge stable to master and bumpversion patch 193 | git checkout master 194 | git merge stable 195 | bumpversion --no-tag patch 196 | git push 197 | 198 | .PHONY: bumpversion-candidate 199 | bumpversion-candidate: ## Bump the version to the next candidate 200 | bumpversion candidate --no-tag 201 | 202 | .PHONY: bumpversion-minor 203 | bumpversion-minor: ## Bump the version the next minor skipping the release 204 | bumpversion --no-tag minor 205 | 206 | .PHONY: bumpversion-major 207 | bumpversion-major: ## Bump the version the next major skipping the release 208 | bumpversion --no-tag major 209 | 210 | .PHONY: bumpversion-revert 211 | bumpversion-revert: ## Undo a previous bumpversion-release 212 | git checkout master 213 | git branch -D stable 214 | 215 | CLEAN_DIR := $(shell git status --short | grep -v ??) 216 | CURRENT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD 2>/dev/null) 217 | CHANGELOG_LINES := $(shell git diff HEAD..origin/stable HISTORY.md 2>&1 | wc -l) 218 | 219 | .PHONY: check-clean 220 | check-clean: ## Check if the directory has uncommitted changes 221 | ifneq ($(CLEAN_DIR),) 222 | $(error There are uncommitted changes) 223 | endif 224 | 225 | .PHONY: check-master 226 | check-master: ## Check if we are in master branch 227 | ifneq ($(CURRENT_BRANCH),master) 228 | $(error Please make the release from master branch\n) 229 | endif 230 | 231 | .PHONY: check-history 232 | check-history: ## Check if HISTORY.md has been modified 233 | ifeq ($(CHANGELOG_LINES),0) 234 | $(error Please insert the release notes in HISTORY.md before releasing) 235 | endif 236 | 237 | .PHONY: check-release 238 | check-release: check-clean check-master check-history ## Check if the release can be made 239 | @echo "A new release can be made" 240 | 241 | .PHONY: release 242 | release: check-release bumpversion-release publish bumpversion-patch 243 | 244 | .PHONY: release-test 245 | release-test: check-release bumpversion-release-test publish-test bumpversion-revert 246 | 247 | .PHONY: release-candidate 248 | release-candidate: check-master publish bumpversion-candidate 249 | 250 | .PHONY: release-candidate-test 251 | release-candidate-test: check-clean check-master publish-test 252 | 253 | .PHONY: release-minor 254 | release-minor: check-release bumpversion-minor release 255 | 256 | .PHONY: release-major 257 | release-major: check-release bumpversion-major release 258 | 259 | .PHONY: checkdeps 260 | checkdeps: # Save the currently installed versions of the dependencies as the latest versions 261 | $(eval allow_list='mlblocks|pandas|numpy|psutil') 262 | pip freeze | grep -v "sintel-dev/SigPro.git" | grep -E $(allow_list) > $(OUTPUT_PATH) 263 | -------------------------------------------------------------------------------- /PERPRIMITIVEDOCS.md: -------------------------------------------------------------------------------- 1 | # Per-Primitive Docs 2 | 3 | Currently, the available primitives are as follows: 4 | 5 | ```python 6 | 7 | ['sigpro.SigPro', 8 | 'sigpro.aggregations.amplitude.statistical.crest_factor', 9 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 10 | 'sigpro.aggregations.amplitude.statistical.mean', 11 | 'sigpro.aggregations.amplitude.statistical.rms', 12 | 'sigpro.aggregations.amplitude.statistical.skew', 13 | 'sigpro.aggregations.amplitude.statistical.std', 14 | 'sigpro.aggregations.amplitude.statistical.var', 15 | 'sigpro.aggregations.frequency.band.band_mean', 16 | 'sigpro.transformations.amplitude.identity.identity', 17 | 'sigpro.transformations.amplitude.spectrum.power_spectrum', 18 | 'sigpro.transformations.frequency.band.frequency_band', 19 | 'sigpro.transformations.frequency.fft.fft', 20 | 'sigpro.transformations.frequency.fft.fft_real', 21 | 'sigpro.transformations.frequency_time.stft.stft', 22 | 'sigpro.transformations.frequency_time.stft.stft_real'] 23 | ``` 24 | 25 | ## sigpro.SigPro 26 | 27 | **path**: `sigpro.SigPro` 28 | 29 | **description** : Please see the SigPro page for more detailed documentation. 30 | 31 | | argument | type | description | 32 | | --- | --- | --- | 33 | | parameters | | | 34 | | See pipeline documentation | | | 35 | | hyperparameters | | | 36 | | keep_columns | bool, list | If bool, whether to keep non-feature columns. If list, keep the columns in the list. | 37 | | values_column_name | str | Name of signal values column in input. | 38 | | transformations | list | List of transformations to apply sequentially before applying aggregations in parallel. | 39 | | aggregations | list | List of aggregations to apply in parallel after all transformations are applied (in sequence). | 40 | | input_is_dataframe | bool | Whether the input is a pandas DataFrame. Defaults to True. | 41 | | output | | | 42 | | See Documentation | | | 43 | 44 | ## sigpro.transformations.amplitude.identity.identity 45 | 46 | **path**: `sigpro.transformations.amplitude.identity.identity` 47 | 48 | **description** : This primitive simply returns the input amplitude values as its output. 49 | 50 | | argument | type | description | 51 | | --- | --- | --- | 52 | | parameters | | | 53 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 54 | | hyperparameters | | | 55 | | N/A | | | 56 | | output | | | 57 | | amplitude_values | numpy.ndarray | Output (identity) signal amplitude values | 58 | 59 | ```python 60 | import numpy as np 61 | from sigpro.contributing import run_primitive 62 | 63 | data = np.array([[1,2,3,4,5]]) 64 | transformed_data = run_primitive( 65 | 'sigpro.transformations.amplitude.identity.identity', 66 | amplitude_values= data 67 | ) 68 | transformed_data 69 | ``` 70 | 71 | ## sigpro.transformations.amplitude.spectrum.power_spectrum 72 | 73 | **path**: `sigpro.transformations.amplitude.spectrum.power_spectrum` 74 | 75 | **description** : This primitive applies an RFFT on the amplitude values and return the real components. 76 | 77 | | argument | type | description | 78 | | --- | --- | --- | 79 | | parameters | | | 80 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 81 | | sampling_frequency | int, float | Sampling frequency value passed in Hz. | 82 | | hyperparameters | | | 83 | | N/A | | | 84 | | output | | | 85 | | amplitude_values | numpy.ndarray | Output real components | 86 | | frequency_values | numpy.ndarray | Frequency values | 87 | 88 | ```python 89 | import numpy as np 90 | from sigpro.contributing import run_primitive 91 | 92 | data = np.array([[1,2,3,4,5]]) 93 | frequency = 1000 94 | transformed_data = run_primitive( 95 | 'sigpro.transformations.amplitude.spectrum.power_spectrum', 96 | amplitude_values= data, 97 | sampling_frequency = frequency 98 | ) 99 | transformed_data, freq_values 100 | ``` 101 | 102 | ## sigpro.transformations.frequency.band.frequency_band 103 | 104 | **path**: `sigpro.transformations.frequency.band.frequency_band` 105 | 106 | **description** : This primitive filters between a high and low band frequency and return the amplitude values and frequency values for those. 107 | 108 | | argument | type | description | 109 | | --- | --- | --- | 110 | | parameters | | | 111 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 112 | | frequency_values | numpy.ndarray | Input frequency values passed in Hz. | 113 | | hyperparameters | | | 114 | | low | int | Lower band frequency | 115 | | high | int | Higher band frequency | 116 | | output | | | 117 | | amplitude_values | numpy.ndarray | Output real components | 118 | | frequency_values | numpy.ndarray | Frequency values | 119 | 120 | ```python 121 | import numpy as np 122 | from sigpro.contributing import run_primitive 123 | 124 | data = np.array([[1,2,3,4,5]]) 125 | frequency_values = np.array([[70,140,210, 280, 350]]) 126 | transformed_data, freq_values = run_primitive( 127 | 'sigpro.transformations.frequency.band.frequency_band', 128 | amplitude_values= data, 129 | frequency_values = frequency_values, 130 | low = 100, 131 | high = 300 132 | ) 133 | transformed_data, freq_values, 134 | ``` 135 | 136 | ## sigpro.transformations.frequency.fft.fft 137 | 138 | **path**: `sigpro.transformations.frequency.fft.fft` 139 | 140 | **description** : This primitive applies an FFT on the amplitude values using the discrete Fourier transform in `numpy`. 141 | 142 | | argument | type | description | 143 | | --- | --- | --- | 144 | | parameters | | | 145 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 146 | | sampling_frequency | int, float | Sampling frequency value passed in Hz. | 147 | | hyperparameters | | | 148 | | N/A | | | 149 | | output | | | 150 | | amplitude_values | numpy.ndarray | Output all components | 151 | | frequency_values | numpy.ndarray | Frequency values | 152 | 153 | ```python 154 | import numpy as np 155 | from sigpro.contributing import run_primitive 156 | 157 | data = np.array([[1,2,3,4,5]]) 158 | frequency = 1000 159 | transformed_data, freq_values = run_primitive( 160 | 'sigpro.transformations.frequency.fft.fft', 161 | amplitude_values= data, 162 | sampling_frequency = frequency 163 | 164 | ) 165 | transformed_data, freq_values 166 | ``` 167 | 168 | ## sigpro.transformations.frequency.fft.fft_real 169 | 170 | **path**: `sigpro.transformations.frequency.fft.fft_real` 171 | 172 | **description** : This primitive applies an FFT on the amplitude values using the discrete Fourier transform in `numpy` and returns the real components. 173 | 174 | | argument | type | description | 175 | | --- | --- | --- | 176 | | parameters | | | 177 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 178 | | sampling_frequency | int, float | Sampling frequency value passed in Hz. | 179 | | hyperparameters | | | 180 | | N/A | | | 181 | | output | | | 182 | | amplitude_values | numpy.ndarray | Output real components of FFT | 183 | | frequency_values | numpy.ndarray | Frequency values | 184 | 185 | ```python 186 | import numpy as np 187 | from sigpro.contributing import run_primitive 188 | 189 | data = np.array([[1,2,3,4,5]]) 190 | frequency = 1000 191 | transformed_data, freq_values = run_primitive( 192 | 'sigpro.transformations.frequency.fft.fft_real', 193 | amplitude_values= data, 194 | sampling_frequency = frequency 195 | 196 | ) 197 | transformed_data, freq_values 198 | ``` 199 | 200 | ## sigpro.transformations.frequency_time.stft.stft 201 | 202 | **path**: `sigpro.transformations.frequency.stft.stft` 203 | 204 | **description** : This primitive computes and returns the short time Fourier transform. 205 | 206 | | argument | type | description | 207 | | --- | --- | --- | 208 | | parameters | | | 209 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 210 | | sampling_frequency | int, float | Sampling frequency value passed in Hz. | 211 | | hyperparameters | | | 212 | | N/A | | | 213 | | output | | | 214 | | amplitude_values | numpy.ndarray | Output all components of STFT | 215 | | frequency_values | numpy.ndarray | Frequency values | 216 | | time_values | numpy.ndarray | Time values | 217 | 218 | ```python 219 | import numpy as np 220 | from sigpro.contributing import run_primitive 221 | 222 | data = np.array([[1,2,3,4,5]]) 223 | frequency = 1000 224 | transformed_data, freq_values, time_values = run_primitive( 225 | 'sigpro.transformations.frequency_time.stft.stft', #note: this is inconsistent 226 | amplitude_values= data, 227 | sampling_frequency = frequency 228 | 229 | ) 230 | transformed_data, freq_values, time_values 231 | ``` 232 | 233 | ## sigpro.transformations.frequency_time.stft.stft_real 234 | 235 | **path**: `sigpro.transformations.frequency.stft.stft_real` 236 | 237 | **description** : This primitive computes and returns the real part of the short time Fourier transform. 238 | 239 | | argument | type | description | 240 | | --- | --- | --- | 241 | | parameters | | | 242 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 243 | | sampling_frequency | int, float | Sampling frequency value passed in Hz. | 244 | | hyperparameters | | | 245 | | N/A | | | 246 | | output | | | 247 | | amplitude_values | numpy.ndarray | Output real components of STFT | 248 | | frequency_values | numpy.ndarray | Frequency values | 249 | | time_values | numpy.ndarray | Time values | 250 | 251 | ```python 252 | import numpy as np 253 | from sigpro.contributing import run_primitive 254 | 255 | data = np.array([[1,2,3,4,5]]) 256 | frequency = 1000 257 | transformed_data, freq_values, time_values = run_primitive( 258 | 'sigpro.transformations.frequency_time.stft.stft_real', #note: this is inconsistent 259 | amplitude_values= data, 260 | sampling_frequency = frequency 261 | 262 | ) 263 | transformed_data, freq_values, time_values 264 | ``` 265 | 266 | ## sigpro.aggregation.amplitude.statistical.mean 267 | 268 | **path**: `sigpro.aggregation.amplitude.statistical.mean` 269 | 270 | **description** : This primitive computes and returns the arithmetic mean of the input values. 271 | 272 | | argument | type | description | 273 | | --- | --- | --- | 274 | | parameters | | | 275 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 276 | | hyperparameters | | | 277 | | N/A | | | 278 | | output | | | 279 | | mean_value | float | Output mean of amplitude values | 280 | 281 | ```python 282 | import numpy as np 283 | from sigpro.contributing import run_primitive 284 | 285 | data = np.array([1,2,3,4,5]) 286 | output = run_primitive( 287 | 'sigpro.aggregations.amplitude.statistical.mean', 288 | amplitude_values= data, 289 | ) 290 | output 291 | ``` 292 | 293 | ## sigpro.aggregation.amplitude.statistical.std 294 | 295 | **path**: `sigpro.aggregation.amplitude.statistical.std` 296 | 297 | **description** : This primitive computes and returns the standard deviation of the input values. 298 | 299 | | argument | type | description | 300 | | --- | --- | --- | 301 | | parameters | | | 302 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 303 | | hyperparameters | | | 304 | | N/A | | | 305 | | output | | | 306 | | std_value | float | Output standard deviation of amplitude values | 307 | 308 | ```python 309 | import numpy as np 310 | from sigpro.contributing import run_primitive 311 | 312 | data = np.array([1,2,3,4,5]) 313 | output = run_primitive( 314 | 'sigpro.aggregations.amplitude.statistical.std', 315 | amplitude_values= data, 316 | ) 317 | output 318 | ``` 319 | 320 | ## sigpro.aggregation.amplitude.statistical.kurtosis 321 | 322 | **path**: `sigpro.aggregation.amplitude.statistical.std` 323 | 324 | **description** : This primitive computes and returns the kurtosis of the input values. 325 | 326 | | argument | type | description | 327 | | --- | --- | --- | 328 | | parameters | | | 329 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 330 | | hyperparameters | | | 331 | | fisher | bool | If True (default), use Fisher definition (normal 0.0). If False, use Pearson definition (normal 3.0). | 332 | | bias | bool | If False, correct calculations for statistical bias. Defaults to True. | 333 | | output | | | 334 | | kurtosis_value | float | Output kurtosis of amplitude values | 335 | 336 | ```python 337 | import numpy as np 338 | from sigpro.contributing import run_primitive 339 | 340 | data = np.array([1,2,3,4,5]) 341 | output = run_primitive( 342 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 343 | amplitude_values= data, 344 | fisher = True, 345 | bias = True 346 | ) 347 | output 348 | ``` 349 | 350 | ## sigpro.aggregation.amplitude.statistical.var 351 | 352 | **path**: `sigpro.aggregation.amplitude.statistical.var` 353 | 354 | **description** : This primitive computes and returns the variance of the input values. 355 | 356 | | argument | type | description | 357 | | --- | --- | --- | 358 | | parameters | | | 359 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 360 | | hyperparameters | | | 361 | | N/A | | | 362 | | output | | | 363 | | var_value | float | Output variance of amplitude values | 364 | 365 | ```python 366 | import numpy as np 367 | from sigpro.contributing import run_primitive 368 | 369 | data = np.array([1,2,3,4,5]) 370 | output = run_primitive( 371 | 'sigpro.aggregations.amplitude.statistical.var', 372 | amplitude_values= data, 373 | ) 374 | output 375 | ``` 376 | 377 | ## sigpro.aggregation.amplitude.statistical.rms 378 | 379 | **path**: `sigpro.aggregation.amplitude.statistical.rms` 380 | 381 | **description** : This primitive computes and returns the root mean square (RMS) of the input values. 382 | 383 | | argument | type | description | 384 | | --- | --- | --- | 385 | | parameters | | | 386 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 387 | | hyperparameters | | | 388 | | N/A | | | 389 | | output | | | 390 | | rms_value | float | Output RMS of amplitude values | 391 | 392 | ```python 393 | import numpy as np 394 | from sigpro.contributing import run_primitive 395 | 396 | data = np.array([1,2,3,4,5]) 397 | output = run_primitive( 398 | 'sigpro.aggregations.amplitude.statistical.rms', 399 | amplitude_values= data, 400 | ) 401 | output 402 | ``` 403 | 404 | ## sigpro.aggregation.amplitude.statistical.skew 405 | 406 | **path**: `sigpro.aggregation.amplitude.statistical.skew` 407 | 408 | **description** : This primitive computes and returns the skew of the input values. 409 | 410 | | argument | type | description | 411 | | --- | --- | --- | 412 | | parameters | | | 413 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 414 | | hyperparameters | | | 415 | | N/A | | | 416 | | output | | | 417 | | skew_value | float | Output skew of amplitude values | 418 | 419 | ```python 420 | import numpy as np 421 | from sigpro.contributing import run_primitive 422 | 423 | data = np.array([1,2,3,4,5]) 424 | output = run_primitive( 425 | 'sigpro.aggregations.amplitude.statistical.skew', 426 | amplitude_values= data, 427 | ) 428 | output 429 | ``` 430 | 431 | ## sigpro.aggregation.amplitude.statistical.crest_factor 432 | 433 | **path**: `sigpro.aggregation.amplitude.statistical.crest_factor` 434 | 435 | **description** : This primitive computes and returns the crest factor (ratio of peak to RMS) of the input values. 436 | 437 | | argument | type | description | 438 | | --- | --- | --- | 439 | | parameters | | | 440 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 441 | | hyperparameters | | | 442 | | N/A | | | 443 | | output | | | 444 | | crest_factor_value | float | Output crest factor of amplitude values | 445 | 446 | ```python 447 | import numpy as np 448 | from sigpro.contributing import run_primitive 449 | 450 | data = np.array([1,2,3,4,5]) 451 | output = run_primitive( 452 | 'sigpro.aggregations.amplitude.statistical.crest_factor', 453 | amplitude_values= data, 454 | ) 455 | output 456 | ``` 457 | 458 | ## sigpro.aggregations.frequency.band.band_mean 459 | 460 | **path**: `sigpro.aggregations.frequency.band.band_mean` 461 | 462 | **description** : This primitive filters between a high and low band and computes the mean value for this specific band. 463 | 464 | | argument | type | description | 465 | | --- | --- | --- | 466 | | parameters | | | 467 | | amplitude_values | numpy.ndarray | Input signal amplitude values | 468 | | frequency_values | numpy.ndarray | Input frequency values passed in Hz. | 469 | | hyperparameters | | | 470 | | min_frequency | float | Lower band threshold. | 471 | | max_frequency | float | Upper band threshold. | 472 | | output | | | 473 | | value | float | Output mean of amplitude values within frequency band. | 474 | 475 | ```python 476 | import numpy as np 477 | from sigpro.contributing import run_primitive 478 | 479 | data = np.array([[1,2,3,4,5]]) 480 | frequency_values = np.array([[70,140,210, 280, 350]]) 481 | output = run_primitive( 482 | 'sigpro.aggregations.frequency.band.band_mean', 483 | amplitude_values= data, 484 | frequency_values = frequency_values, 485 | min_frequency = 100, 486 | max_frequency = 300 487 | ) 488 | output 489 | ``` 490 | -------------------------------------------------------------------------------- /PRIMITIVES.md: -------------------------------------------------------------------------------- 1 | # SigPro Primitives 2 | 3 | In this document you will find information about what kind of primitives `SigPro` contains and 4 | the type and subtypes of those, with a detailed information about their input and output 5 | considering the type and subtype of the primitive. 6 | 7 | ## Overview 8 | 9 | The goal of SigPro is to be able to process *time series* data passed as 10 | time stamped *time series segments* and extract *feature time series* from 11 | them by applying multiple types of *transformations* and *aggregations*. 12 | 13 | ## Transformations and Aggregations 14 | 15 | All the timeseries processing in SigPro happens in two families of 16 | functions: 17 | 18 | * Transformations: They convert time series segment vectors into new 19 | vectors that represent the same information in a different format. 20 | * Aggregations: They convert time series segment vectors or transformed 21 | representations of them into individual features. 22 | 23 | ### Transformations 24 | 25 | Transformations are simple Python functions that transform 1d arrays of 26 | time series values into other arrays of values of 1, 2 or 3 dimensions. 27 | These transforms can be in time domain or frequency domain and/or joint 28 | time frequency domain. 29 | 30 | #### Input 31 | 32 | The input format for the transformation functions is always at least a 1d 33 | numpy array. Optionally, functions can accept additional variables as 34 | input, such as the sampling frequency, contextual values or 35 | hyperparameters. 36 | 37 | #### Types of transformations 38 | 39 | ##### Amplitude transformations 40 | 41 | These transformations take as input a 1d array of values and return another 42 | 1d array of values. The returned array does not necessarily have the same 43 | length as the inputted one. 44 | 45 | * Input: 46 | * `amplitude_values`: 1d array of values. 47 | * `**context_values`: additional variables from the context. 48 | * `**hyperparameters`: additional arguments to control the behavior. 49 | * Output 50 | * `transformed`: 1d array of values. 51 | 52 | ```python 53 | def amplitude_transform( 54 | amplitude_values: np.ndarray, 55 | **context_values, 56 | **hyperparameter_values, 57 | ) -> np.ndarray: 58 | ``` 59 | 60 | ###### Examples 61 | 62 | The simplest example of this type of transformation is the `identity`, 63 | which receives a time series segment as input and returns it unmodified: 64 | 65 | ```python 66 | def identity_transform(amplitude_values): 67 | return amplitude_values 68 | ``` 69 | 70 | A more complex example which would use contextual information as well as 71 | hyperparameters would look like this: 72 | 73 | 74 | ```python 75 | def complex_transform(amplitude_values, a_context_value, a_hyperparam): 76 | # here we would manipulate the values based on `a_context_value` 77 | # and a_hyperparam value. 78 | ... 79 | return transformed_values 80 | ``` 81 | 82 | ##### Frequency transformations 83 | 84 | These transformations take as input a 1d array of time series values and a 85 | sampling frequency, and perform a frequency based transformations. 86 | 87 | The outputs are 2 1d arrays, the first one being the frequency amplitudes 88 | and the other one being the associated frequencies. 89 | 90 | * Input: 91 | * `amplitude_values`: 1d array of values. 92 | * `sampling_frequency`: sampling frequency. 93 | * `**context_values`: additional variables from the context. 94 | * `**hyperparameters`: additional arguments to control the behavior. 95 | * Output 96 | * `transformed`: 1d array of values. 97 | * `frequencies`: 1d array of frequency values. 98 | 99 | ```python 100 | def spectrum_analysis_transform( 101 | amplitude_values: np.ndarray, 102 | sampling_frequency: float, 103 | **context_values, 104 | **hyperparameter_values, 105 | ) -> tuple[np.ndarray]: 106 | ``` 107 | 108 | ###### Examples 109 | 110 | An example of this is the `fft`, which receives a time segment and a 111 | sampling frequency as input and returns the fft amplitudes and frequencies 112 | as output. 113 | 114 | ```python 115 | import numpy as np 116 | 117 | def fft_transform(amplitude_values, sampling_frequency): 118 | amplitude_values = np.fft.fft(amplitude_values) 119 | frequency_values = np.fft.fftfreq(len(amplitude_values), sampling_frequency) 120 | return amplitude_values, frequency_values 121 | ``` 122 | 123 | ##### Frequency Time (2D Spectrum analysis) transformations 124 | 125 | These transformations take as input a 1d array of values and a sampling 126 | frequency, and perform multiple frequency based transformations over 127 | multiple points in time within the sequence. 128 | 129 | The outputs are TBD 130 | 131 | * Input: 132 | * `values`: 1d array of values. 133 | * `sampling_frequency`: sampling frequency. 134 | * `**context_values`: additional variables from the context. 135 | * `**hyperparameters`: additional arguments to control the behavior. 136 | * Output 137 | * `dataframe` 138 | 139 | ```python 140 | def spectrum_2d_analysis_transform( 141 | amplitude_values: np.ndarray, 142 | sampling_frequency: float, 143 | **context_values, 144 | **hyperparameter_values, 145 | ) -> (np.ndarray, np.ndarray, np.ndarray) 146 | ``` 147 | 148 | An example of this is the `stft`, which receives a time segment and a 149 | sampling frequency as input and returns a vector with multiple fft 150 | transformations applied in different points in time over the input segment: 151 | 152 | ```python 153 | import scipy.signal 154 | 155 | def stft(amplitude_values, sampling_frequency): 156 | frequency_values, time_values, amplitude_values = scipy.signal.stft( 157 | amplitude_values, 158 | fs=sampling_frequency 159 | ) 160 | return amplitude_values, frequency_values, time_values 161 | ``` 162 | 163 | ### Aggregations 164 | 165 | Transforms simply convert the signal into an alternate representation. They 166 | don't directly result in `features` for machine learning or even a 167 | feature time series. In general, further aggregations need to be applied to 168 | the output from the transforms. In general these aggregations are of three 169 | kinds: 170 | 171 | * `Statistical aggregations`: This just aggregate using different 172 | statistical functions - rms value, mean value, and several others. 173 | * `Spectral aggregations`: These are very domain specific and this is where 174 | an expert can put their intelligence. These could be something like total 175 | energy in the frequency bin 10-100Mhz. As a result, the input to these 176 | aggregations is both the ```amplitude_values``` output, but also the 177 | ```frequencies```. And similarly, if the transform is joint time-frequency 178 | transform, then the data frame is provided. 179 | * ```Comparitive aggregations```: These are special aggregations, where along 180 | side the information from the transformation, a reference is also provided. 181 | This reference can be from physics based simulation, or transform of a 182 | neighboring window, or data from a known problematic scenario. 183 | 184 | #### Input 185 | 186 | The input format for these aggregations is the output from the different 187 | transformations + optional additional context values or hyperparameters. 188 | 189 | The outputs are always one float value or a tuple with multiple float 190 | values. 191 | 192 | #### Aggregation Types 193 | 194 | ##### Amplitude aggregations 195 | 196 | These aggregations take as input a 1d array of values. 197 | 198 | * Input: 199 | * `amplitude_values`: 1d array of values. 200 | * `**context_values`: additional variables from the context. 201 | * `**hyperparameters`: additional arguments to control the behavior. 202 | * Output 203 | * `feature` or `features`: single float value, or tuple with multiple 204 | float values. 205 | 206 | ```python 207 | def statistical_aggregation( 208 | amplitude_values: np.ndarray, 209 | **context_values, 210 | **hyperparameter_values, 211 | ) -> float or tuple[float]: 212 | ``` 213 | 214 | An example of such an aggregation is a simple mean: 215 | 216 | ```python 217 | import numpy as np 218 | 219 | def mean(values): 220 | return np.mean(values) 221 | ``` 222 | 223 | ##### Frequency aggregations (Spectrum analysis) 224 | 225 | These transformations take as input a 2 1d arrays, one containing 226 | amplitude values and the other one containing frequency values. 227 | 228 | * Input: 229 | * `amplitude_values`: 1d array of values. 230 | * `frequency_values`: 1d array of frequency values. 231 | * `**context_values`: additional variables from the context. 232 | * `**hyperparameters`: additional arguments to control the behavior. 233 | * Output 234 | * `feature` or `features`: single float value, or tuple with multiple 235 | float values. 236 | 237 | ```python 238 | def spectrum_analysis_aggregation( 239 | amplitude_values: np.ndarray, 240 | frequency_values: np.ndarray, 241 | **context_values, 242 | **hyperparameter_values, 243 | ) -> float or tuple[float]: 244 | ``` 245 | 246 | 247 | ##### Frequency time aggregations (2D Spectrum analysis) 248 | 249 | These transformations take as input the output of the frequency time 250 | transforms. 251 | 252 | * Input: 253 | * `amplitude_values`: transformed amplitude values from transformation 254 | output. 255 | * `frequency_values`: frequency values from transformation output. 256 | * `time_values`: time values from transformation output. 257 | * `**context_values`: additional variables from the context. 258 | * `**hyperparameters`: additional arguments to control the behavior. 259 | * Output 260 | * `feature` or `features`: single float value, or tuple with multiple 261 | float values. 262 | 263 | ```python 264 | def spectrum_2d_analysis_aggregation( 265 | dataframe: pd.DataFrame, 266 | **context_values, 267 | **hyperparameter_values, 268 | ) -> float or tuple[float]: 269 | ``` 270 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | DAI-Lab 3 | An open source project from Data to AI Lab at MIT. 4 |

5 | 6 | [![Development Status](https://img.shields.io/badge/Development%20Status-2%20--%20Pre--Alpha-yellow)](https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha) 7 | [![PyPi Shield](https://img.shields.io/pypi/v/SigPro.svg)](https://pypi.python.org/pypi/SigPro) 8 | [![Tests](https://github.com/sintel-dev/SigPro/workflows/Run%20Tests/badge.svg)](https://github.com/sintel-dev/SigPro/actions?query=workflow%3A%22Run+Tests%22+branch%3Amaster) 9 | [![Downloads](https://pepy.tech/badge/sigpro)](https://pepy.tech/project/sigpro) 10 | 11 | 12 | # SigPro: Signal Processing Tools for Machine Learning 13 | 14 | * License: [MIT](https://github.com/sintel-dev/SigPro/blob/master/LICENSE) 15 | * Development Status: [Pre-Alpha](https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha) 16 | * Homepage: https://github.com/sintel-dev/SigPro 17 | 18 | ## Overview 19 | 20 | SigPro offers an end-to-end solution to efficiently apply multiple *signal processing techniques* 21 | to convert *raw time series* into *feature time series* that encode the knowledge of domain experts 22 | in order to solve time series machine learning problems. 23 | 24 | # Install 25 | 26 | ## Requirements 27 | 28 | **SigPro** has been developed and tested on [Python 3.9, 3.10, 3.11 and 3.12](https://www.python.org/downloads/) 29 | on GNU/Linux and macOS systems. 30 | 31 | Also, although it is not strictly required, the usage of a [virtualenv]( 32 | https://virtualenv.pypa.io/en/latest/) is highly recommended in order to avoid 33 | interfering with other software installed in the system where **SigPro** is run. 34 | 35 | ## Install with pip 36 | 37 | The easiest and recommended way to install **SigPro** is using [pip]( 38 | https://pip.pypa.io/en/stable/): 39 | 40 | ```bash 41 | pip install sigpro 42 | ``` 43 | 44 | This will pull and install the latest stable release from [PyPi](https://pypi.org/). 45 | 46 | If you want to install from source or contribute to the project please read the 47 | [Contributing Guide](CONTRIBUTING.md). 48 | 49 | 50 | # User Guides 51 | 52 | `SigPro` comes with the following user guides: 53 | 54 | * [PRIMITIVES.md](PRIMITIVES.md): Information about the primitive families, their expected input 55 | and output. 56 | * [USAGE.md](USAGE.md): Instructions about how to usee the three main functionalities of `SigPro`. 57 | * [DEVELOPMENT.md](DEVELOPMENT.md): Step by step guide about how to write a valid `SigPro` 58 | primitive and contribute it to either `SigPro` or your own library. 59 | -------------------------------------------------------------------------------- /USAGE.md: -------------------------------------------------------------------------------- 1 | # SigPro Usage 2 | 3 | In this document you will find instructions about how to use the three 4 | main functionalities of the SigPro library: 5 | 6 | * Find Primitives: How to find the primitive that you need for your task. 7 | * Run a Primitive: How to run a primitive as a standalone function. 8 | * Create a primitive JSON: How to create a primitive JSON for a python 9 | function. 10 | 11 | ## Find primitives 12 | 13 | Before you start using any SigPro primitive you will to find the 14 | ones that are suitable for your task. 15 | 16 | This can be done with the `sigpro.get_primitives` function. 17 | 18 | In order to get the complete list of primitives available you can 19 | call it without any arguments: 20 | 21 | ```python 22 | from sigpro import get_primitives 23 | 24 | get_primitives() 25 | ``` 26 | 27 | The output will be a list with the names of all the primitives 28 | that SigPro is able to find: 29 | 30 | ``` 31 | ['sigpro.aggregations.amplitude.statistical.crest_factor', 32 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 33 | 'sigpro.aggregations.amplitude.statistical.mean', 34 | 'sigpro.aggregations.amplitude.statistical.rms', 35 | 'sigpro.aggregations.amplitude.statistical.skew', 36 | 'sigpro.aggregations.amplitude.statistical.std', 37 | 'sigpro.aggregations.amplitude.statistical.var', 38 | 'sigpro.transformations.amplitude.identity.identity', 39 | 'sigpro.transformations.frequency.fft.fft', 40 | 'sigpro.transformations.frequency.fft.fft_real', 41 | 'sigpro.transformations.frequency_time.stft.stft', 42 | 'sigpro.transformations.frequency_time.stft.stft_real'] 43 | ``` 44 | 45 | ### Filter primitives by Type and Subtype 46 | 47 | In most cases you will want to search for a primitive of a specific 48 | type or subtype. 49 | 50 | In order to do this, you can pass the `type` and `subtype` arguments 51 | to the `get_primitives` function. 52 | 53 | For example, if you only want to obtain the list of *frequency transformation* primitives you can call `get_primitives` like this: 54 | 55 | ```python 56 | get_primitives(primitive_type='transformation', primitive_subtype='frequency') 57 | ``` 58 | 59 | ### Filter primitives by Name 60 | 61 | If you want to narrow the search bases on the name of the primitives 62 | you can pass a *string pattern*, and `get_primitives` will return only 63 | the primitives that match the indicated pattern. 64 | 65 | For example, if we are interested only in *frequency transformation* 66 | primitives based on *fft* we can call `get_primitives` as follows: 67 | 68 | ```python 69 | get_primitives('fft', primitive_type='transformation', primitive_subtype='frequency') 70 | ``` 71 | 72 | ## Run a Primitive 73 | 74 | SigPro primitives can be run in a standalone mode, which allows exploring 75 | their behavior for learning or debugging purposes. 76 | 77 | In order to do this you can use the `run_primitive` function passing 78 | the name of the primitive that you want to run. 79 | 80 | For example, if you want to run the 81 | `sigpro.aggregations.amplitude.statistical.mean` primitive you can use: 82 | 83 | ```python 84 | from sigpro import run_primitive 85 | 86 | run_primitive('sigpro.aggregations.amplitude.statistical.mean') 87 | ``` 88 | 89 | This will do the following steps: 90 | 91 | * Load the indicate primitive. 92 | * Load demo data of the corresponding type and subtype. In this case, this 93 | will load the Amplitude Aggregation demo data using the 94 | `get_amplitude_demo`. 95 | * Call the indicated primitive passing a randomly selected row from the 96 | loaded demo data. 97 | * Return the primitive output value (or values). 98 | 99 | ### Selecting the data which the primitive runs on 100 | 101 | In some cases you will want to run the primitive multiple times on the 102 | same values instead of randomly selecting one row from the demo data. 103 | 104 | In order to restrict the execution to particular demo row, you can 105 | pass the `demo_row_index` value indicating the row that you want to use. 106 | 107 | ```python 108 | run_primitive( 109 | 'sigpro.aggregations.amplitude.statistical.mean', 110 | demo_row_index=1 111 | ) 112 | ``` 113 | 114 | In this case, the row used by the `run_primitive` function will be the 115 | one returned by the `get_amplitude_demo` when passing the same row index 116 | as input: 117 | 118 | ```python 119 | from sigpro.demo import get_amplitude_demo 120 | 121 | row = get_amplitude_demo(index=1) 122 | ``` 123 | 124 | ### Running the primitive with your own data 125 | 126 | In some cases you will want to run the primitive with your own data. 127 | 128 | In order to do so, you should pass the expected data by the primitive as 129 | additional arguments to the `run_primitive` function. Those additional 130 | arguments depend on the type and subtype of the primitive, and must be 131 | passed to their respective fields: 132 | 133 | * `amplitude_values (numpy.ndarray or None)`: 134 | Array of floats representing signal values or ``None``. 135 | * `sampling_frequency (float, int or None)`: 136 | Sampling frequency value passed in Hz or ``None``. 137 | * `frequency_values (numpy.ndarray or None)`: 138 | Array of floats representing frequency values for the given amplitude values 139 | or ``None``. 140 | * `time_values (numpy.ndarray or None)`: 141 | Array of floats representing time values or ``None``. 142 | 143 | Here is an example where you pass your own `amplitude_values` for the `mean` 144 | primitive (bear in mind that this primitive only requires amplitude values 145 | if more data was needed then it has to be passed as well): 146 | 147 | ```python 148 | import numpy as np 149 | 150 | data = np.array([1, 2, 3]) 151 | 152 | run_primitive( 153 | 'sigpro.aggregations.amplitude.statistical.mean', 154 | amplitude_values=data 155 | ) 156 | ``` 157 | 158 | 159 | ### Passing hyperparameter values or additional context values 160 | 161 | When a primitive accepts hyperparameters or requires additional context 162 | values, those can be specified at the end of the `run_primitive` function. 163 | 164 | The primitive `sigpro.aggregations.amplitude.statistical.kurtosis` 165 | can take as input `fisher` and `bias`, both boolean values, and in order 166 | to specify them we use `run_primitive` like this: 167 | 168 | ```python 169 | run_primitive( 170 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 171 | fisher=True, 172 | bias=False 173 | ) 174 | ``` 175 | 176 | This way we specified those two additional arguments for the primitive. 177 | 178 | ### Full `run_primitive` arguments list 179 | 180 | The complete list of arguments that this function takes are as follow: 181 | 182 | * `primitive (str)`: 183 | Path or name of the primitive to be used. 184 | * `primitive_type (str)`: 185 | Type to which the primitive belongs to. 186 | * `primitive_subtype (str)`: 187 | Subtype to which the primitive belongs to. 188 | * `amplitude_values (numpy.ndarray or None)`: 189 | Array of floats representing signal values or ``None``. 190 | * `sampling_frequency (float, int or None)`: 191 | Sampling frequency value passed in Hz or ``None``. 192 | * `frequency_values (numpy.ndarray or None)`: 193 | Array of floats representing frequency values for the given amplitude values 194 | or ``None``. 195 | * `time_values (numpy.ndarray or None)`: 196 | Array of floats representing time values or ``None``. 197 | * `demo_row_index (int or None)`: 198 | If `int`, return the value at that index if `None` return a random index. This is used 199 | if no amplitude values are provided. 200 | * `context (optional)`: 201 | Additional context arguments required to run the primitive. 202 | * `hyperparameters (optional)`: 203 | Additional hyperparameters or tunable hyperparameters arguments. 204 | 205 | ## Create a primitive JSON 206 | 207 | If you need to create a primitive JSON for a new Python function that you 208 | implemented you can use the `sigpro.contributin.make_primitive` function, 209 | which helps you in the process. 210 | 211 | > For complete instructions about how to write a primitive from scratch 212 | > and contribute it to the project please read the [Contributing Guide]( 213 | > CONTRIBUTING.md) 214 | 215 | For example, if you want to create the primitive for the implemented code 216 | in `sigpro.aggregations.amplitude.statistical.py` for the method `mean`, you 217 | can use: 218 | 219 | ```python 220 | from sigpro.contributing import make_primitive 221 | 222 | make_primitive( 223 | 'sigpro.aggregations.amplitude.statistical.mean' 224 | primitive_type='aggregation', 225 | primitive_subtype='amplitude', 226 | ) 227 | ``` 228 | 229 | This function will do the following steps: 230 | 231 | * Validate that the inputs that the primitive expects are correct for its 232 | type and subtype. 233 | * Validate that the function also expects the specified context arguments 234 | and hyperparameters. 235 | * Validate that there are no additional arguments that have not been 236 | specified. 237 | * If everything is correct, create a JSON file for your primitive in the 238 | `primitives` folder. 239 | 240 | 241 | And once it finishes it will return you the absolute path where the primitive 242 | has been stored: 243 | 244 | ``` 245 | /path/to/sigpro/primitives/aggregations/amplitude/statistical/mean.json 246 | ``` 247 | 248 | If you explore this file, you will see the primitive `json` annotation: 249 | 250 | ```json 251 | { 252 | "name": "sigpro.aggregations.amplitude.statistical.mean", 253 | "primitive": "sigpro.aggregations.amplitude.statistical.mean", 254 | "classifiers": { 255 | "type": "aggregation", 256 | "subtype": "amplitude" 257 | }, 258 | "produce": { 259 | "args": [ 260 | { 261 | "name": "amplitude_values", 262 | "type": "numpy.ndarray" 263 | } 264 | ], 265 | "output": [ 266 | { 267 | "name": "value", 268 | "type": "float" 269 | } 270 | ] 271 | }, 272 | "hyperparameters": { 273 | "fixed": {}, 274 | "tunable": {} 275 | } 276 | } 277 | ``` 278 | 279 | ### Using your primitive path 280 | 281 | By default, `make_primitive` will store the primitives inside the folder 282 | `sigpro/primitives` which is located in the project. If you want to store 283 | the primitives in a different directory you can do so by using the argument 284 | `primitives_path`: 285 | 286 | ```python 287 | make_primitive( 288 | 'sigpro.aggregations.amplitude.statistical.mean', 289 | primitive_type='aggregation', 290 | primitive_subtype='amplitude', 291 | primitives_path='my_primitives' 292 | ) 293 | ``` 294 | 295 | The output path now would be: 296 | 297 | ``` 298 | /path/to/my_primitives/aggregations/amplitude/statistical/mean.json 299 | ``` 300 | 301 | 302 | ### Storing flat json file 303 | 304 | If you would like to store your primtives without the path tree, you can do 305 | so by stting the argument `primitives_subfolders` to `False`: 306 | 307 | ```python 308 | make_primitive( 309 | 'sigpro.aggregations.amplitude.statistical.mean', 310 | primitive_type='aggregation', 311 | primitive_subtype='amplitude', 312 | primitives_path='my_primitives', 313 | primitives_subfolders=False, 314 | ) 315 | ``` 316 | 317 | This will create a file named `sigpro.aggregations.statistical.mean.json` 318 | inside of the folder `my_primitives`, the contents of this `json` file 319 | are exactly the same as the one created previously, but the nomenclature of 320 | the file it's different, we don't have the subfolder structure. 321 | 322 | ### Creating a primitive with tunable hyperparameters 323 | 324 | In some cases our primitives have tunable hyperparameters or fixed ones. 325 | In order to do so you have to create a dictionary of hyperparameters with 326 | a dictionary that contains their default value and their type (in string): 327 | 328 | In this example you see how the `kurtosis` primitive is being created with 329 | the tunable hyperparameters `fisher` and `bias` both being `booleans` 330 | 331 | ```python 332 | tunable_hyperparameters = { 333 | 'fisher': { 334 | 'default': True, 335 | 'type': 'bool' 336 | }, 337 | 'bias': { 338 | 'default': True, 339 | 'type': 'bool' 340 | } 341 | } 342 | 343 | make_primitive( 344 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 345 | primitive_type='aggregation', 346 | primitive_subtype='amplitude', 347 | tunable_hyperparameters=tunable_hyperparameters 348 | ) 349 | ``` 350 | 351 | ### Create a primitive with context arguments 352 | 353 | When a primitive requires of a context argument, we have to use the 354 | argument `context_arguments` and pass it to `make_primitive`. This 355 | argument has to be a python `list` that contains python `dictionaries` 356 | whith the `name` and`type` of the `context_argument`. 357 | 358 | *This is just an example of how you would pass the context argument, this 359 | primitive does not exist.* 360 | 361 | ```python 362 | context_arguments = [ 363 | { 364 | 'name': 'test_arg1', 365 | 'type': 'float', 366 | }, 367 | { 368 | 'name': 'test_arg2', 369 | 'type': 'numpy.ndarray', 370 | }, 371 | ] 372 | 373 | make_primitive( 374 | 'sigpro.aggregation.amplitude.context.contextual_method', 375 | context_arguments=context_arguments 376 | ) 377 | ``` 378 | 379 | ### Full `make_primitive` arguments list 380 | 381 | The complete list of arguments that this function takes are as follow: 382 | 383 | * `primitive (str)`: 384 | The name of the primitive, the python path including the name of the 385 | module and the name of the function. 386 | * `primitive_type (str)`: 387 | Type of primitive. 388 | * `primitive_subtype (str)`: 389 | Subtype of the primitive. 390 | * `context_arguments (list or None)`: 391 | A list with dictionaries containing the name and type of the context 392 | arguments. 393 | * `fixed_hyperparameters (dict or None)`: 394 | A dictionary containing as key the name of the hyperparameter and as 395 | value a dictionary containing the type and the default value that it 396 | should take. 397 | * `tunable_hyperparameters (dict or None)`: 398 | A dictionary containing as key the name of the hyperparameter and as 399 | value a dictionary containing the type and the default value and the 400 | range of values that it can take. 401 | * `primitive_outputs (list or None)`: 402 | A list with dictionaries containing the name and type of the output 403 | values. If ``None`` default values for those will be used. 404 | * `primitives_path (str)`: 405 | Path to the root of the primitives folder, in which the primitives JSON 406 | will be stored. Defaults to `sigpro/primitives`. 407 | * `primitives_subfolders (bool)`: 408 | Whether to store the primitive JSON in a subfolder tree (``True``) or to 409 | use a flat primitive name (``False``). Defaults to ``True``. 410 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = python -msphinx 7 | SPHINXPROJ = sigpro 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /docs/authors.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../AUTHORS.rst 2 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # SigPro documentation build configuration file, created by 5 | # sphinx-quickstart on Fri Jun 9 13:47:02 2017. 6 | # 7 | # This file is execfile()d with the current directory set to its 8 | # containing dir. 9 | # 10 | # Note that not all possible configuration values are present in this 11 | # autogenerated file. 12 | # 13 | # All configuration values have a default; values that are commented out 14 | # serve to show the default. 15 | 16 | # If extensions (or modules to document with autodoc) are in another 17 | # directory, add these directories to sys.path here. If the directory is 18 | # relative to the documentation root, use os.path.abspath to make it 19 | # absolute, like shown here. 20 | 21 | import sphinx_rtd_theme # For read the docs theme 22 | 23 | import sigpro 24 | 25 | # -- General configuration --------------------------------------------- 26 | 27 | # If your documentation needs a minimal Sphinx version, state it here. 28 | # 29 | # needs_sphinx = '1.0' 30 | 31 | # Add any Sphinx extension module names here, as strings. They can be 32 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 33 | extensions = [ 34 | 'm2r', 35 | 'sphinx.ext.autodoc', 36 | 'sphinx.ext.githubpages', 37 | 'sphinx.ext.viewcode', 38 | 'sphinx.ext.napoleon', 39 | 'autodocsumm', 40 | ] 41 | 42 | autodoc_default_options = { 43 | 'autosummary': True, 44 | } 45 | 46 | # Add any paths that contain templates here, relative to this directory. 47 | templates_path = ['_templates'] 48 | 49 | # The suffix(es) of source filenames. 50 | # You can specify multiple suffix as a list of string: 51 | source_suffix = ['.rst', '.md'] 52 | 53 | # The master toctree document. 54 | master_doc = 'index' 55 | 56 | # General information about the project. 57 | project = 'SigPro' 58 | slug = 'sigpro' 59 | title = project + ' Documentation', 60 | copyright = '2020, MIT Data To AI Lab' 61 | author = 'MIT Data To AI Lab' 62 | description = 'Signal Processing Tools for Machine Learning' 63 | user = 'sintel-dev' 64 | 65 | # The version info for the project you're documenting, acts as replacement 66 | # for |version| and |release|, also used in various other places throughout 67 | # the built documents. 68 | # 69 | # The short X.Y version. 70 | version = sigpro.__version__ 71 | # The full version, including alpha/beta/rc tags. 72 | release = sigpro.__version__ 73 | 74 | # The language for content autogenerated by Sphinx. Refer to documentation 75 | # for a list of supported languages. 76 | # 77 | # This is also used if you do content translation via gettext catalogs. 78 | # Usually you set "language" from the command line for these cases. 79 | language = None 80 | 81 | # List of patterns, relative to source directory, that match files and 82 | # directories to ignore when looking for source files. 83 | # This patterns also effect to html_static_path and html_extra_path 84 | exclude_patterns = ['.py', '_build', 'Thumbs.db', '.DS_Store', '**.ipynb_checkpoints'] 85 | 86 | # The name of the Pygments (syntax highlighting) style to use. 87 | pygments_style = 'sphinx' 88 | 89 | # If true, `todo` and `todoList` produce output, else they produce nothing. 90 | todo_include_todos = False 91 | 92 | # -- Options for HTML output ------------------------------------------- 93 | 94 | # The theme to use for HTML and HTML Help pages. See the documentation for 95 | # a list of builtin themes. 96 | # 97 | html_theme = 'sphinx_rtd_theme' 98 | html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] 99 | 100 | # Readthedocs additions 101 | html_context = { 102 | 'display_github': True, 103 | 'github_user': user, 104 | 'github_repo': project, 105 | 'github_version': 'master', 106 | 'conf_py_path': '/docs/', 107 | } 108 | 109 | # Theme options are theme-specific and customize the look and feel of a 110 | # theme further. For a list of options available for each theme, see the 111 | # documentation. 112 | html_theme_options = { 113 | 'collapse_navigation': False, 114 | 'display_version': False, 115 | } 116 | 117 | # Add any paths that contain custom static files (such as style sheets) here, 118 | # relative to this directory. They are copied after the builtin static files, 119 | # so a file named "default.css" will overwrite the builtin "default.css". 120 | # html_static_path = ['_static'] 121 | 122 | # The name of an image file (relative to this directory) to use as a favicon of 123 | # the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 124 | # pixels large. 125 | html_favicon = 'images/dai-logo-white.ico' 126 | 127 | # If given, this must be the name of an image file (path relative to the 128 | # configuration directory) that is the logo of the docs. It is placed at 129 | # the top of the sidebar; its width should therefore not exceed 200 pixels. 130 | html_logo = 'images/dai-logo-white-200.png' 131 | 132 | # -- Options for HTMLHelp output --------------------------------------- 133 | 134 | # Output file base name for HTML help builder. 135 | htmlhelp_basename = slug + 'doc' 136 | 137 | 138 | # -- Options for LaTeX output ------------------------------------------ 139 | 140 | latex_elements = { 141 | # The paper size ('letterpaper' or 'a4paper'). 142 | # 143 | # 'papersize': 'letterpaper', 144 | 145 | # The font size ('10pt', '11pt' or '12pt'). 146 | # 147 | # 'pointsize': '10pt', 148 | 149 | # Additional stuff for the LaTeX preamble. 150 | # 151 | # 'preamble': '', 152 | 153 | # Latex figure (float) alignment 154 | # 155 | # 'figure_align': 'htbp', 156 | } 157 | 158 | # Grouping the document tree into LaTeX files. List of tuples 159 | # (source start file, target name, title, author, documentclass 160 | # [howto, manual, or own class]). 161 | latex_documents = [( 162 | master_doc, 163 | slug + '.tex', 164 | title, 165 | author, 166 | 'manual' 167 | )] 168 | 169 | 170 | # -- Options for manual page output ------------------------------------ 171 | 172 | # One entry per manual page. List of tuples 173 | # (source start file, name, description, authors, manual section). 174 | man_pages = [( 175 | master_doc, 176 | slug, 177 | title, 178 | [author], 179 | 1 180 | )] 181 | 182 | 183 | # -- Options for Texinfo output ---------------------------------------- 184 | 185 | # Grouping the document tree into Texinfo files. List of tuples 186 | # (source start file, target name, title, author, 187 | # dir menu entry, description, category) 188 | texinfo_documents = [( 189 | master_doc, 190 | slug, 191 | title, 192 | author, 193 | slug, 194 | description, 195 | 'Miscellaneous' 196 | )] 197 | -------------------------------------------------------------------------------- /docs/contributing.rst: -------------------------------------------------------------------------------- 1 | .. mdinclude:: ../CONTRIBUTING.md 2 | -------------------------------------------------------------------------------- /docs/history.rst: -------------------------------------------------------------------------------- 1 | .. mdinclude:: ../HISTORY.md 2 | -------------------------------------------------------------------------------- /docs/images/dai-logo-white-200.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sintel-dev/SigPro/36798ad91a70137c1688b56d8abbc21628724db8/docs/images/dai-logo-white-200.png -------------------------------------------------------------------------------- /docs/images/dai-logo-white.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sintel-dev/SigPro/36798ad91a70137c1688b56d8abbc21628724db8/docs/images/dai-logo-white.ico -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: readme.rst 2 | 3 | .. toctree:: 4 | :hidden: 5 | :maxdepth: 2 6 | 7 | Overview 8 | 9 | .. toctree:: 10 | :caption: Resources 11 | :hidden: 12 | 13 | API Reference 14 | usage 15 | contributing 16 | authors 17 | history 18 | 19 | Indices and tables 20 | ================== 21 | * :ref:`genindex` 22 | * :ref:`modindex` 23 | * :ref:`search` 24 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=python -msphinx 9 | ) 10 | set SOURCEDIR=. 11 | set BUILDDIR=_build 12 | set SPHINXPROJ=sigpro 13 | 14 | if "%1" == "" goto help 15 | 16 | %SPHINXBUILD% >NUL 2>NUL 17 | if errorlevel 9009 ( 18 | echo. 19 | echo.The Sphinx module was not found. Make sure you have Sphinx installed, 20 | echo.then set the SPHINXBUILD environment variable to point to the full 21 | echo.path of the 'sphinx-build' executable. Alternatively you may add the 22 | echo.Sphinx directory to PATH. 23 | echo. 24 | echo.If you don't have Sphinx installed, grab it from 25 | echo.http://sphinx-doc.org/ 26 | exit /b 1 27 | ) 28 | 29 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 30 | goto end 31 | 32 | :help 33 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 34 | 35 | :end 36 | popd 37 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. mdinclude:: ../README.md 2 | -------------------------------------------------------------------------------- /docs/usage.rst: -------------------------------------------------------------------------------- 1 | .. mdinclude:: ../USAGE.md 2 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bumpversion] 2 | current_version = 0.3.1.dev0 3 | commit = True 4 | tag = True 5 | parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? 6 | serialize = 7 | {major}.{minor}.{patch}.{release}{candidate} 8 | {major}.{minor}.{patch} 9 | 10 | [bumpversion:part:release] 11 | optional_value = release 12 | first_value = dev 13 | values = 14 | dev 15 | release 16 | 17 | [bumpversion:part:candidate] 18 | 19 | [bumpversion:file:setup.py] 20 | search = version='{current_version}' 21 | replace = version='{new_version}' 22 | 23 | [bumpversion:file:sigpro/__init__.py] 24 | search = __version__ = '{current_version}' 25 | replace = __version__ = '{new_version}' 26 | 27 | [bdist_wheel] 28 | universal = 1 29 | 30 | [flake8] 31 | max-line-length = 99 32 | exclude = docs, .tox, .git, __pycache__, .ipynb_checkpoints 33 | ignore = D107, SFS3 # SFS2 can be changed to SFS3 if supporting >=3.6 34 | 35 | [isort] 36 | include_trailing_comment = True 37 | line_length = 99 38 | lines_between_types = 0 39 | multi_line_output = 4 40 | not_skip = __init__.py 41 | use_parentheses = True 42 | 43 | [aliases] 44 | test = pytest 45 | 46 | [tool:pytest] 47 | collect_ignore = ['setup.py'] 48 | 49 | [pydocstyle] 50 | ignore = D107, D407 51 | 52 | [pylint] 53 | extension-pkg-whitelist = numpy 54 | min-similarity-lines = 5 55 | ignore-comments = yes 56 | ignore-docstrings = yes 57 | ignore-imports = yes 58 | max-args = 9 59 | good-names = df 60 | disable = unspecified-encoding 61 | 62 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | """The setup script.""" 5 | 6 | from setuptools import setup, find_packages 7 | 8 | with open('README.md', encoding='utf-8') as readme_file: 9 | readme = readme_file.read() 10 | 11 | with open('HISTORY.md', encoding='utf-8') as history_file: 12 | history = history_file.read() 13 | 14 | install_requires = [ 15 | 'mlblocks>=0.6.2', 16 | 'pandas>=1.5.3', 17 | 'numpy>=1.26.0', 18 | 'scipy>=1.11.3', 19 | ] 20 | 21 | setup_requires = [ 22 | 'pytest-runner>=2.11.1', 23 | ] 24 | 25 | tests_require = [ 26 | 'pytest>=7.2.2', 27 | 'pytest-cov>=4.1.0', 28 | 'jupyter>=1.0.0,<2', 29 | 'rundoc>=0.4.3,<0.5', 30 | ] 31 | 32 | development_requires = [ 33 | # general 34 | 'bumpversion>=0.5.3,<0.6', 35 | 'pip>=9.0.1', 36 | 'watchdog>=0.8.3,<0.11', 37 | 38 | # docs 39 | 'm2r>=0.2.0,<0.3', 40 | 'Sphinx>=1.7.1,<3', 41 | 'sphinx_rtd_theme>=0.2.4,<0.5', 42 | 'autodocsumm>=0.1.10', 43 | 'markupsafe<2.1.0', 44 | 'Jinja2>=2,<3.1', 45 | 46 | # fails on Sphinx < v3.4 47 | 'alabaster<=0.7.12', 48 | # fails on Sphins < v5.0 49 | 'sphinxcontrib-applehelp<1.0.8', 50 | 'sphinxcontrib-devhelp<1.0.6', 51 | 'sphinxcontrib-htmlhelp<2.0.5', 52 | 'sphinxcontrib-serializinghtml<1.1.10', 53 | 'sphinxcontrib-qthelp<1.0.7', 54 | 55 | # style check 56 | 'flake8>=3.7.7,<4', 57 | 'flake8-absolute-import>=1.0,<2', 58 | 'flake8-docstrings>=1.5.0,<2', 59 | 'flake8-sfs>=0.0.3,<0.1', 60 | 'isort>=4.3.4,<5', 61 | 'pylint>=2.5.3,<3', 62 | 63 | # fix style issues 64 | 'autoflake>=1.1,<2', 65 | 'autopep8>=1.4.3,<2', 66 | 67 | # distribute on PyPI 68 | 'twine>=1.10.0,<4', 69 | 'wheel>=0.30.0', 70 | 71 | # Advanced testing 72 | 'coverage>=4.5.1,<6', 73 | 'tox>=2.9.1,<4', 74 | 'importlib-metadata<5', 75 | 'mistune<2', 76 | 'invoke', 77 | ] 78 | 79 | setup( 80 | author='MIT Data To AI Lab', 81 | author_email='dailabmit@gmail.com', 82 | classifiers=[ 83 | 'Development Status :: 2 - Pre-Alpha', 84 | 'Intended Audience :: Developers', 85 | 'License :: OSI Approved :: MIT License', 86 | 'Natural Language :: English', 87 | 'Programming Language :: Python :: 3', 88 | 'Programming Language :: Python :: 3.9', 89 | 'Programming Language :: Python :: 3.10', 90 | 'Programming Language :: Python :: 3.11', 91 | 'Programming Language :: Python :: 3.12', 92 | ], 93 | description='Signal Processing Tools for Machine Learning', 94 | entry_points={ 95 | 'mlblocks': [ 96 | 'primitives=sigpro:MLBLOCKS_PRIMITIVES', 97 | ], 98 | }, 99 | extras_require={ 100 | 'test': tests_require, 101 | 'dev': development_requires + tests_require, 102 | }, 103 | include_package_data=True, 104 | install_requires=install_requires, 105 | keywords='sigpro signal processing tools machine learning', 106 | license='MIT license', 107 | long_description=readme + '\n\n' + history, 108 | long_description_content_type='text/markdown', 109 | name='sigpro', 110 | packages=find_packages(include=['sigpro', 'sigpro.*']), 111 | python_requires='>=3.9,<3.13', 112 | setup_requires=setup_requires, 113 | test_suite='tests', 114 | tests_require=tests_require, 115 | url='https://github.com/sintel-dev/SigPro', 116 | version='0.3.1.dev0', 117 | zip_safe=False, 118 | ) 119 | -------------------------------------------------------------------------------- /sigpro/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """Top-level package for SigPro.""" 4 | 5 | __author__ = """MIT Data To AI Lab""" 6 | __email__ = 'dailabmit@gmail.com' 7 | __version__ = '0.3.1.dev0' 8 | 9 | import os 10 | 11 | from mlblocks import discovery 12 | 13 | from sigpro.core import SigPro 14 | 15 | _BASE_PATH = os.path.abspath(os.path.dirname(__file__)) 16 | MLBLOCKS_PRIMITIVES = os.path.join(_BASE_PATH, 'primitives') 17 | 18 | 19 | def get_primitives(name=None, primitive_type=None, primitive_subtype=None): 20 | """Get a list of the available primitives. 21 | 22 | Optionally filter by primitive type: ``transformation`` or ``aggregation``. 23 | 24 | Args: 25 | primitive_type (str): 26 | Filter by primitive type. ``transformation`` or ``aggregation``. 27 | 28 | Returns: 29 | list: 30 | List of the names of the available primitives. 31 | """ 32 | filters = {} 33 | if primitive_type: 34 | if primitive_type not in ('transformation', 'aggregation'): 35 | raise ValueError('primitive_type must be `transformation` or `aggregation`.') 36 | 37 | filters['classifiers.type'] = primitive_type 38 | 39 | if primitive_subtype: 40 | if primitive_subtype not in ('amplitude', 'frequency', 'frequency_time'): 41 | raise ValueError( 42 | 'primitive_subtype must be `amplitude`, `frequency` or `frequency_time`.') 43 | 44 | filters['classifiers.subtype'] = primitive_subtype 45 | 46 | return discovery.find_primitives(name or 'sigpro', filters) 47 | 48 | 49 | __all__ = ('SigPro', ) 50 | -------------------------------------------------------------------------------- /sigpro/aggregations/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Aggregations module.""" 2 | -------------------------------------------------------------------------------- /sigpro/aggregations/amplitude/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Aggregations Ampltiude module.""" 2 | -------------------------------------------------------------------------------- /sigpro/aggregations/amplitude/statistical.py: -------------------------------------------------------------------------------- 1 | """Amplitude statistical module.""" 2 | 3 | import numpy as np 4 | import scipy.stats 5 | 6 | 7 | def mean(amplitude_values): 8 | """Calculate the mean value of the values. 9 | 10 | Args: 11 | amplitude_values (numpy.ndarray): 12 | Array of floats representing signal values. 13 | 14 | Returns: 15 | float: 16 | `mean` value of the input array. 17 | """ 18 | return np.mean(amplitude_values) 19 | 20 | 21 | def std(amplitude_values): 22 | """Compute the arithmetic mean value of the values. 23 | 24 | Args: 25 | amplitude_values (numpy.ndarray): 26 | Array of floats representing signal values. 27 | 28 | Returns: 29 | float: 30 | `std` value of the input array. 31 | """ 32 | return np.std(amplitude_values) 33 | 34 | 35 | def var(amplitude_values): 36 | """Compute the variance value of the values. 37 | 38 | Args: 39 | amplitude_values (numpy.ndarray): 40 | Array of floats representing signal values. 41 | 42 | Returns: 43 | float: 44 | `std` value of the input array. 45 | """ 46 | return np.var(amplitude_values) 47 | 48 | 49 | def rms(amplitude_values): 50 | """Compute the RMS (Root Mean Square) of the values. 51 | 52 | Args: 53 | amplitude_values (numpy.ndarray): 54 | Array of floats representing signal values. 55 | 56 | Returns: 57 | float: 58 | RMS of the input array. 59 | """ 60 | return np.sqrt((np.array(amplitude_values) ** 2).mean()) 61 | 62 | 63 | def crest_factor(amplitude_values): 64 | """Compute the ratio of the peak to the RMS. 65 | 66 | Used for estimating the amount of impact wear in a bearing. 67 | 68 | Args: 69 | amplitude_values (numpy.ndarray): 70 | Array of floats representing signal values. 71 | 72 | Returns: 73 | float: 74 | The crest factor of the inputted values. 75 | """ 76 | peak = max(np.abs(amplitude_values)) 77 | return peak / rms(amplitude_values) 78 | 79 | 80 | def skew(amplitude_values): 81 | """Compute the sample skewness of an array of values. 82 | 83 | Args: 84 | amplitude_values (numpy.ndarray): 85 | Array of floats representing signal values. 86 | 87 | Returns: 88 | float: 89 | The skewness value of the input array. 90 | """ 91 | return scipy.stats.skew(amplitude_values) 92 | 93 | 94 | def kurtosis(amplitude_values, fisher=True, bias=True): 95 | """Compute the kurtosis ,Fisher or Pearson, of an array of values. 96 | 97 | Args: 98 | amplitude_values (numpy.ndarray): 99 | Array of floats representing signal values. 100 | fisher (bool): 101 | If ``True``, Fisher’s definition is used (normal ==> 0.0). If ``False``, 102 | Pearson’s definition is used (normal ==> 3.0). Defaults to ``True``. 103 | bias (bool): 104 | If ``False``, then the calculations are corrected for statistical bias. 105 | Defaults to ``True``. 106 | 107 | Returns: 108 | float: 109 | The kurtosis value of the input array. If all values are equal, return 110 | `-3` for Fisher's definition and `0` for Pearson's definition. 111 | """ 112 | return scipy.stats.kurtosis(amplitude_values, fisher=fisher, bias=bias) 113 | -------------------------------------------------------------------------------- /sigpro/aggregations/frequency/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Amplitude Frequency module.""" 2 | -------------------------------------------------------------------------------- /sigpro/aggregations/frequency/band.py: -------------------------------------------------------------------------------- 1 | """Band Module.""" 2 | 3 | import numpy as np 4 | 5 | 6 | def band_mean(amplitude_values, frequency_values, min_frequency, max_frequency): 7 | """Compute the mean values for a specific band. 8 | 9 | Filter between a high and low band and compute the mean value for this specific band. 10 | 11 | Args: 12 | amplitude_values (np.ndarray): 13 | A numpy array with the signal values. 14 | frequency_values (np.ndarray): 15 | A numpy array with the frequency values. 16 | min_frequency (int or float): 17 | Band minimum. 18 | max_frequency (int or float): 19 | Band maximum. 20 | 21 | Returns: 22 | float: 23 | Mean value for the given band. 24 | """ 25 | lower_frequency_than = frequency_values <= max_frequency 26 | higher_frequency_than = frequency_values >= min_frequency 27 | selected_idx = np.where(higher_frequency_than & lower_frequency_than) 28 | selected_values = amplitude_values[selected_idx] 29 | 30 | return np.mean(selected_values) 31 | 32 | 33 | def band_rms(amplitude_values, frequency_values, min_frequency, max_frequency): 34 | """ 35 | Compute the rms values for a specific band. 36 | 37 | Filter between a high and low band (inclusive) and compute the rms value for this 38 | specific band. 39 | 40 | Args: 41 | amplitude_values (np.ndarray): 42 | A numpy array with the signal values. 43 | frequency_values (np.ndarray): 44 | A numpy array with the frequency values. 45 | min_frequency (int or float): 46 | Band minimum. 47 | max_frequency (int or float): 48 | Band maximum. 49 | 50 | Returns: 51 | float: 52 | rms value for the given band. 53 | """ 54 | lower_frequency_than = frequency_values <= max_frequency 55 | higher_frequency_than = frequency_values >= min_frequency 56 | 57 | selected_idx = np.ravel(np.where(higher_frequency_than & lower_frequency_than)) 58 | selected_values = amplitude_values[selected_idx] 59 | 60 | return np.sqrt(np.mean(np.square(selected_values))) 61 | -------------------------------------------------------------------------------- /sigpro/aggregations/frequency_time/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Aggregation Frequency Time module.""" 2 | -------------------------------------------------------------------------------- /sigpro/basic_primitives.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """Reference class implementations of existing primitives.""" 3 | from sigpro import contributing, primitive 4 | 5 | # Transformations 6 | 7 | 8 | class Identity(primitive.AmplitudeTransformation): 9 | """Identity primitive class.""" 10 | 11 | def __init__(self): 12 | super().__init__('sigpro.transformations.amplitude.identity.identity') 13 | 14 | 15 | class PowerSpectrum(primitive.AmplitudeTransformation): 16 | """PowerSpectrum primitive class.""" 17 | 18 | def __init__(self): 19 | super().__init__('sigpro.transformations.amplitude.spectrum.power_spectrum') 20 | primitive_spec = contributing._get_primitive_spec('transformation', 'frequency') 21 | self.set_primitive_inputs(primitive_spec['args']) 22 | self.set_primitive_outputs(primitive_spec['output']) 23 | 24 | 25 | class FFT(primitive.FrequencyTransformation): 26 | """FFT primitive class.""" 27 | 28 | def __init__(self): 29 | super().__init__("sigpro.transformations.frequency.fft.fft") 30 | 31 | 32 | class FFTFreq(primitive.FrequencyTransformation): 33 | """FFT Freq primitive class.""" 34 | 35 | def __init__(self): 36 | super().__init__("sigpro.transformations.frequency.fftfreq.fft_freq") 37 | 38 | 39 | class FFTReal(primitive.FrequencyTransformation): 40 | """FFTReal primitive class.""" 41 | 42 | def __init__(self): 43 | super().__init__("sigpro.transformations.frequency.fft.fft_real") 44 | 45 | 46 | class FrequencyBand(primitive.FrequencyTransformation): 47 | """ 48 | FrequencyBand primitive class. 49 | 50 | Filter between a high and low band frequency and return the amplitude values and 51 | frequency values for those. 52 | 53 | Args: 54 | low (int): Lower band frequency of filter. 55 | high (int): Higher band frequency of filter. 56 | """ 57 | 58 | def __init__(self, low, high): 59 | super().__init__("sigpro.transformations.frequency.band.frequency_band", 60 | init_params={'low': low, 'high': high}) 61 | self.set_primitive_inputs([{"name": "amplitude_values", "type": "numpy.ndarray"}, 62 | {"name": "frequency_values", "type": "numpy.ndarray"}]) 63 | self.set_primitive_outputs([{'name': 'amplitude_values', 'type': "numpy.ndarray"}, 64 | {'name': 'frequency_values', 'type': "numpy.ndarray"}]) 65 | self.set_fixed_hyperparameters({'low': {'type': 'int'}, 'high': {'type': 'int'}}) 66 | 67 | 68 | class STFT(primitive.FrequencyTimeTransformation): 69 | """STFT primitive class.""" 70 | 71 | def __init__(self): 72 | super().__init__('sigpro.transformations.frequency_time.stft.stft') 73 | self.set_primitive_outputs([{"name": "amplitude_values", "type": "numpy.ndarray"}, 74 | {"name": "frequency_values", "type": "numpy.ndarray"}, 75 | {"name": "time_values", "type": "numpy.ndarray"}]) 76 | 77 | 78 | class STFTReal(primitive.FrequencyTimeTransformation): 79 | """STFTReal primitive class.""" 80 | 81 | def __init__(self): 82 | super().__init__('sigpro.transformations.frequency_time.stft.stft_real') 83 | self.set_primitive_outputs([{"name": "real_amplitude_values", "type": "numpy.ndarray"}, 84 | {"name": "frequency_values", "type": "numpy.ndarray"}, 85 | {"name": "time_values", "type": "numpy.ndarray"}]) 86 | 87 | # Aggregations 88 | 89 | 90 | class CrestFactor(primitive.AmplitudeAggregation): 91 | """CrestFactor primitive class.""" 92 | 93 | def __init__(self): 94 | super().__init__('sigpro.aggregations.amplitude.statistical.crest_factor') 95 | self.set_primitive_outputs([{'name': 'crest_factor_value', 'type': "float"}]) 96 | 97 | 98 | class Kurtosis(primitive.AmplitudeAggregation): 99 | """ 100 | Kurtosis primitive class. 101 | 102 | Computes the kurtosis value of the input array. If all values are equal, return 103 | `-3` for Fisher's definition and `0` for Pearson's definition. 104 | 105 | Args: 106 | fisher (bool): 107 | If ``True``, Fisher’s definition is used (normal ==> 0.0). If ``False``, 108 | Pearson’s definition is used (normal ==> 3.0). Defaults to ``True``. 109 | bias (bool): 110 | If ``False``, then the calculations are corrected for statistical bias. 111 | Defaults to ``True``. 112 | """ 113 | 114 | def __init__(self, fisher=True, bias=True): 115 | super().__init__('sigpro.aggregations.amplitude.statistical.kurtosis', 116 | init_params={'fisher': fisher, 'bias': bias}) 117 | self.set_primitive_outputs([{'name': 'kurtosis_value', 'type': "float"}]) 118 | self.set_fixed_hyperparameters({'fisher': {'type': 'bool', 'default': True}, 119 | 'bias': {'type': 'bool', 'default': True}}) 120 | 121 | 122 | class Mean(primitive.AmplitudeAggregation): 123 | """Mean primitive class.""" 124 | 125 | def __init__(self): 126 | super().__init__('sigpro.aggregations.amplitude.statistical.mean') 127 | self.set_primitive_outputs([{'name': 'mean_value', 'type': "float"}]) 128 | 129 | 130 | class RMS(primitive.AmplitudeAggregation): 131 | """RMS primitive class.""" 132 | 133 | def __init__(self): 134 | super().__init__('sigpro.aggregations.amplitude.statistical.rms') 135 | self.set_primitive_outputs([{'name': 'rms_value', 'type': "float"}]) 136 | 137 | 138 | class Skew(primitive.AmplitudeAggregation): 139 | """Skew primitive class.""" 140 | 141 | def __init__(self): 142 | super().__init__('sigpro.aggregations.amplitude.statistical.skew') 143 | self.set_primitive_outputs([{'name': 'skew_value', 'type': "float"}]) 144 | 145 | 146 | class Std(primitive.AmplitudeAggregation): 147 | """Std primitive class.""" 148 | 149 | def __init__(self): 150 | super().__init__('sigpro.aggregations.amplitude.statistical.std') 151 | self.set_primitive_outputs([{'name': 'std_value', 'type': "float"}]) 152 | 153 | 154 | class Var(primitive.AmplitudeAggregation): 155 | """Var primitive class.""" 156 | 157 | def __init__(self): 158 | super().__init__('sigpro.aggregations.amplitude.statistical.var') 159 | self.set_primitive_outputs([{'name': 'var_value', 'type': "float"}]) 160 | 161 | 162 | class BandMean(primitive.FrequencyAggregation): 163 | """ 164 | BandMean primitive class. 165 | 166 | Filters between a high and low band and compute the mean value for this specific band. 167 | 168 | Args: 169 | min_frequency (int or float): 170 | Band minimum. 171 | max_frequency (int or float): 172 | Band maximum. 173 | """ 174 | 175 | def __init__(self, min_frequency, max_frequency): 176 | super().__init__('sigpro.aggregations.frequency.band.band_mean', init_params={ 177 | 'min_frequency': min_frequency, 'max_frequency': max_frequency}) 178 | self.set_fixed_hyperparameters({'min_frequency': {'type': 'float'}, 179 | 'max_frequency': {'type': 'float'}}) 180 | 181 | 182 | class BandRMS(primitive.FrequencyAggregation): 183 | """ 184 | BandRMS primitive class. 185 | 186 | Filter between a high and low band (inclusive) and compute the rms value for this 187 | specific band. 188 | 189 | Args: 190 | min_frequency (int or float): 191 | Band minimum. 192 | max_frequency (int or float): 193 | Band maximum. 194 | """ 195 | 196 | def __init__(self, min_frequency, max_frequency): 197 | super().__init__('sigpro.aggregations.frequency.band.band_rms', init_params={ 198 | 'min_frequency': min_frequency, 'max_frequency': max_frequency}) 199 | self.set_fixed_hyperparameters({'min_frequency': {'type': 'float'}, 200 | 'max_frequency': {'type': 'float'}}) 201 | -------------------------------------------------------------------------------- /sigpro/contributing_primitive.py: -------------------------------------------------------------------------------- 1 | """Contributing primitive classes.""" 2 | import copy 3 | 4 | from sigpro.contributing import make_primitive 5 | from sigpro.primitive import ( 6 | AmplitudeAggregation, AmplitudeTransformation, FrequencyAggregation, FrequencyTimeAggregation, 7 | FrequencyTimeTransformation, FrequencyTransformation) 8 | 9 | TAXONOMY = { 10 | 'transformation': { 11 | 'frequency': FrequencyTransformation, 12 | 'amplitude': AmplitudeTransformation, 13 | 'frequency_time': FrequencyTimeTransformation, 14 | }, 'aggregation': { 15 | 'frequency': FrequencyAggregation, 16 | 'amplitude': AmplitudeAggregation, 17 | 'frequency_time': FrequencyTimeAggregation, 18 | } 19 | } 20 | 21 | 22 | def get_primitive_class(primitive, primitive_type, primitive_subtype, 23 | context_arguments=None, fixed_hyperparameters=None, 24 | tunable_hyperparameters=None, primitive_inputs=None, 25 | primitive_outputs=None): 26 | """ 27 | Get a dynamically generated primitive class. 28 | 29 | Args: 30 | primitive (str): 31 | The name of the primitive, the python path including the name of the 32 | module and the name of the function. 33 | primitive_type (str): 34 | Type of primitive. 35 | primitive_subtype (str): 36 | Subtype of the primitive. 37 | context_arguments (list or None): 38 | A list with dictionaries containing the name and type of the context arguments. 39 | fixed_hyperparameters (dict or None): 40 | A dictionary containing as key the name of the hyperparameter and as 41 | value a dictionary containing the type and the default value that it 42 | should take. 43 | tunable_hyperparameters (dict or None): 44 | A dictionary containing as key the name of the hyperparameter and as 45 | value a dictionary containing the type and the default value and the 46 | range of values that it can take. 47 | primitive_inputs (list or None): 48 | A list with dictionaries containing the name and type of the input values. If 49 | ``None`` default values for those will be used. 50 | primitive_outputs (list or None): 51 | A list with dictionaries containing the name and type of the output values. If 52 | ``None`` default values for those will be used. 53 | 54 | Raises: 55 | ValueError: 56 | If the primitive specification arguments are not valid. 57 | 58 | Returns: 59 | type: 60 | Dynamically-generated custom Primitive type. 61 | """ 62 | primitive_type_class = TAXONOMY[primitive_type][primitive_subtype] 63 | 64 | class UserPrimitive(primitive_type_class): # pylint: disable=too-few-public-methods 65 | """User-defined Dynamic Primitive Class.""" 66 | 67 | def __init__(self, **kwargs): 68 | init_params = {} 69 | if fixed_hyperparameters is not None: 70 | init_params = {param: kwargs[param] for param in fixed_hyperparameters} 71 | super().__init__(primitive, init_params=init_params) 72 | if fixed_hyperparameters is not None: 73 | self.set_fixed_hyperparameters(copy.deepcopy(fixed_hyperparameters)) 74 | if tunable_hyperparameters is not None: 75 | self.set_tunable_hyperparameters(copy.deepcopy(tunable_hyperparameters)) 76 | if primitive_inputs is not None: 77 | self.set_primitive_inputs(copy.deepcopy(primitive_inputs)) 78 | if primitive_outputs is not None: 79 | self.set_primitive_outputs(copy.deepcopy(primitive_outputs)) 80 | if context_arguments is not None: 81 | self.set_context_arguments(copy.deepcopy(context_arguments)) 82 | 83 | type_name = f'Custom_{primitive}' 84 | 85 | return type(type_name, (UserPrimitive, ), {}) 86 | 87 | # pylint: disable = too-many-arguments 88 | 89 | 90 | def make_primitive_class(primitive, primitive_type, primitive_subtype, 91 | context_arguments=None, fixed_hyperparameters=None, 92 | tunable_hyperparameters=None, primitive_inputs=None, 93 | primitive_outputs=None, primitives_path='sigpro/primitives', 94 | primitives_subfolders=True): 95 | """ 96 | Get a dynamically generated primitive class and make the primitive JSON. 97 | 98 | Args: 99 | primitive (str): 100 | The name of the primitive, the python path including the name of the 101 | module and the name of the function. 102 | primitive_type (str): 103 | Type of primitive. 104 | primitive_subtype (str): 105 | Subtype of the primitive. 106 | context_arguments (list or None): 107 | A list with dictionaries containing the name and type of the context arguments. 108 | fixed_hyperparameters (dict or None): 109 | A dictionary containing as key the name of the hyperparameter and as 110 | value a dictionary containing the type and the default value that it 111 | should take. 112 | tunable_hyperparameters (dict or None): 113 | A dictionary containing as key the name of the hyperparameter and as 114 | value a dictionary containing the type and the default value and the 115 | range of values that it can take. 116 | primitive_inputs (list or None): 117 | A list with dictionaries containing the name and type of the input values. If 118 | ``None`` default values for those will be used. 119 | primitive_outputs (list or None): 120 | A list with dictionaries containing the name and type of the output values. If 121 | ``None`` default values for those will be used. 122 | primitives_path (str): 123 | Path to the root of the primitives folder, in which the primitives JSON will be stored. 124 | Defaults to `sigpro/primitives`. 125 | primitives_subfolders (bool): 126 | Whether to store the primitive JSON in a subfolder tree (``True``) or to use a flat 127 | primitive name (``False``). Defaults to ``True``. 128 | 129 | Raises: 130 | ValueError: 131 | If the primitive specification arguments are not valid. 132 | 133 | Returns: 134 | type: 135 | Dynamically-generated custom Primitive type. 136 | str: 137 | Path of the generated JSON file. 138 | """ 139 | primitive_path = make_primitive(primitive, primitive_type, primitive_subtype, 140 | context_arguments, fixed_hyperparameters, 141 | tunable_hyperparameters, primitive_inputs, 142 | primitive_outputs, primitives_path, 143 | primitives_subfolders) 144 | return get_primitive_class(primitive, primitive_type, primitive_subtype, 145 | context_arguments, fixed_hyperparameters, 146 | tunable_hyperparameters, primitive_inputs, 147 | primitive_outputs), primitive_path 148 | -------------------------------------------------------------------------------- /sigpro/core.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """Process Signals core functionality.""" 3 | 4 | from collections import Counter 5 | from copy import deepcopy 6 | 7 | import pandas as pd 8 | from mlblocks import MLPipeline, load_primitive 9 | 10 | DEFAULT_INPUT = [ 11 | { 12 | 'name': 'readings', 13 | 'keyword': 'data', 14 | 'type': 'pandas.DataFrame' 15 | }, 16 | { 17 | 'name': 'feature_columns', 18 | 'default': None, 19 | 'type': 'list' 20 | } 21 | ] 22 | 23 | DEFAULT_OUTPUT = [ 24 | { 25 | 'name': 'readings', 26 | 'type': 'pandas.DataFrame' 27 | }, 28 | { 29 | 'name': 'feature_columns', 30 | 'type': 'list' 31 | } 32 | ] 33 | 34 | 35 | class SigPro: 36 | """SigPro class applies multiple transformation and aggregation primitives. 37 | 38 | The Process Signals is responsible for applying a collection of primitives specified by the 39 | user in order to create features for the given data. 40 | 41 | Given a list of transformations and aggregations which are composed 42 | by dictionaries with the following specification: 43 | 44 | * ``Name``: 45 | Name of the transformation / aggregation. 46 | * ``primitive``: 47 | Name of the primitive to apply. 48 | * ``init_params``: 49 | Dictionary containing the initializing parameters for the primitive. 50 | 51 | The process signals will build an ``mlblocks.MLPipeline`` and will generate the features 52 | by previously applying the transformations and then compute the aggregations. 53 | 54 | Args: 55 | data (pandas.DataFrame): 56 | Dataframe with a column that contains signal values. 57 | transformations (list): 58 | List of dictionaries containing the transformation primitives. 59 | aggregations (list): 60 | List of dictionaries containing the aggregation primitives. 61 | values_column_name (str): 62 | The name of the column that contains the signal values. Defaults to ``values``. 63 | keep_columns (Union[bool, list]): 64 | Whether to keep non-feature columns in the output DataFrame or not. 65 | If a list of column names are passed, those columns are kept. 66 | """ 67 | 68 | def _build_pipeline(self): 69 | """Build Pipeline function. 70 | 71 | Given a list of transformations and aggregations build a pipeline 72 | with the output of the aggregations, which take as name the specified 73 | name of the transformations and the aggregation. This lists are composed 74 | by dictionaries with the following specification: 75 | 76 | * ``Name``: 77 | Name of the transformation / aggregation. 78 | * ``primitive``: 79 | Name of the primitive to apply. 80 | * ``init_params``: 81 | Dictionary containing the initializing parameters for the primitive. 82 | 83 | Args: 84 | transformations (list): 85 | List of dictionaries containing the transformation primitives. 86 | aggregations (list): 87 | List of dictionaries containing the aggregation primitives. 88 | 89 | Returns: 90 | mlblocks.MLPipeline: 91 | An ``MLPipeline`` object that first applies all the transformations 92 | and then produces as output the aggregations specified. 93 | """ 94 | primitives = [] 95 | init_params = {} 96 | prefix = [] 97 | outputs = [] 98 | counter = Counter() 99 | 100 | for transformation in self.transformations: 101 | name = transformation.get('name') 102 | if name is None: 103 | name = transformation['primitive'].split('.')[-1] 104 | 105 | prefix.append(name) 106 | primitive = transformation['primitive'] 107 | counter[primitive] += 1 108 | primitive_name = f'{primitive}#{counter[primitive]}' 109 | primitives.append(primitive) 110 | params = transformation.get('init_params') 111 | if params: 112 | init_params[primitive_name] = params 113 | 114 | prefix = '.'.join(prefix) if prefix else '' 115 | 116 | for aggregation in self.aggregations: 117 | name = aggregation.get('name') 118 | if name is None: 119 | name = aggregation['primitive'].split('.')[-1] 120 | 121 | aggregation_name = f'{prefix}.{name}' if prefix else name 122 | 123 | primitive = aggregation['primitive'] 124 | counter[primitive] += 1 125 | primitive_name = f'{primitive}#{counter[primitive]}' 126 | primitives.append(primitive) 127 | 128 | primitive = load_primitive(primitive) 129 | primitive_outputs = primitive['produce']['output'] 130 | 131 | params = aggregation.get('init_params') 132 | if params: 133 | init_params[primitive_name] = params 134 | 135 | if name.lower() == 'sigpro': 136 | primitive = MLPipeline([primitive], init_params={'sigpro.SigPro#1': params}) 137 | primitive_outputs = primitive.get_outputs() 138 | 139 | # primitive_outputs = getattr(self, primitive_outputs)() 140 | if not isinstance(primitive_outputs, str): 141 | for output in primitive_outputs: 142 | output = output['name'] 143 | outputs.append({ 144 | 'name': f'{aggregation_name}.{output}', 145 | 'variable': f'{primitive_name}.{output}' 146 | }) 147 | 148 | outputs = {'default': outputs} if outputs else None 149 | 150 | return MLPipeline( 151 | primitives, 152 | init_params=init_params, 153 | outputs=outputs 154 | ) 155 | 156 | def __init__(self, transformations, aggregations, values_column_name='values', 157 | keep_columns=False, input_is_dataframe=True): 158 | 159 | self.transformations = transformations 160 | self.aggregations = aggregations 161 | self.values_column_name = values_column_name 162 | self.keep_columns = keep_columns 163 | self.input_is_dataframe = input_is_dataframe 164 | self.pipeline = self._build_pipeline() 165 | 166 | def _apply_pipeline(self, window, is_series=False): 167 | """Apply a ``mlblocks.MLPipeline`` to a row. 168 | 169 | Apply a ``MLPipeline`` to a window of a ``pd.DataFrame``, this function can 170 | be combined with the ``pd.DataFrame.apply`` method to be applied to the 171 | entire data frame. 172 | 173 | Args: 174 | window (pd.Series): 175 | Row or multiple rows (window) used to apply the pipeline to. 176 | is_series (bool): 177 | Indicator whether window is formated as a series or dataframe. 178 | """ 179 | if is_series: 180 | context = window.to_dict() 181 | amplitude_values = context.pop(self.values_column_name) 182 | else: 183 | context = {} if window.empty else { 184 | k: v for k, v in window.iloc[0].to_dict().items() if k != self.values_column_name 185 | } 186 | amplitude_values = list(window[self.values_column_name]) 187 | 188 | output = self.pipeline.predict( 189 | amplitude_values=amplitude_values, 190 | **context, 191 | ) 192 | output_names = self.pipeline.get_output_names() 193 | 194 | # ensure that we can iterate over output 195 | output = output if isinstance(output, tuple) else (output, ) 196 | 197 | return pd.Series(dict(zip(output_names, output))) 198 | 199 | def process_signal(self, data=None, window=None, time_index=None, groupby_index=None, 200 | feature_columns=None, **kwargs): 201 | """Apply multiple transformation and aggregation primitives. 202 | 203 | Args: 204 | data (pandas.DataFrame): 205 | Dataframe with a column that contains signal values. 206 | window (str): 207 | Duration of window size, e.g. ('1h'). 208 | time_index (str): 209 | Column in ``data`` that represents the time index. 210 | groupby_index (str or list[str]): 211 | Column(s) to group together and take the window over. 212 | feature_columns (list): 213 | List of column names from the input data frame that must be considered as 214 | features and should not be dropped. 215 | 216 | Returns: 217 | tuple: 218 | pandas.DataFrame: 219 | A data frame with new feature columns by applying the previous primitives. If 220 | ``keep_values`` is ``True`` the original signal values will be conserved in the 221 | data frame, otherwise the original signal values will be deleted. 222 | list: 223 | A list with the feature names generated. 224 | """ 225 | if data is None: 226 | window = pd.Series(kwargs) 227 | values = self._apply_pipeline(window, is_series=True).values 228 | return values if len(values) > 1 else values[0] 229 | 230 | data = data.copy() 231 | if window is not None and groupby_index is not None: 232 | features = data.set_index(time_index).groupby(groupby_index).resample( 233 | rule=window, **kwargs).apply( 234 | self._apply_pipeline 235 | ).reset_index() 236 | data = features 237 | 238 | else: 239 | features = data.apply( 240 | self._apply_pipeline, 241 | axis=1, 242 | is_series=True 243 | ) 244 | data = pd.concat([data, features], axis=1) 245 | 246 | if feature_columns: 247 | feature_columns = feature_columns + list(features.columns) 248 | else: 249 | feature_columns = list(features.columns) 250 | 251 | if isinstance(self.keep_columns, list): 252 | data = data[self.keep_columns + feature_columns] 253 | elif not self.keep_columns: 254 | data = data[feature_columns] 255 | 256 | return data, feature_columns 257 | 258 | def get_input_args(self): 259 | """Return the pipeline input args.""" 260 | if self.input_is_dataframe: 261 | return deepcopy(DEFAULT_INPUT) 262 | 263 | return self.pipeline.get_predict_args() 264 | 265 | def get_output_args(self): 266 | """Return the pipeline output args.""" 267 | if self.input_is_dataframe: 268 | return deepcopy(DEFAULT_OUTPUT) 269 | 270 | return self.pipeline.get_outputs() 271 | -------------------------------------------------------------------------------- /sigpro/demo.py: -------------------------------------------------------------------------------- 1 | """Functions to load demo data.""" 2 | 3 | import json 4 | import os 5 | import random 6 | 7 | import numpy as np 8 | import pandas as pd 9 | from scipy.signal import stft 10 | 11 | DEMO_PATH = os.path.join(os.path.dirname(__file__), 'data') 12 | 13 | 14 | def _load_demo(nrows=None): 15 | demo_path = os.path.join(DEMO_PATH, 'demo_timeseries.csv') 16 | df = pd.read_csv(demo_path, parse_dates=['timestamp'], nrows=nrows) 17 | df['sampling_frequency'] = 1000 18 | df["values"] = df["values"].apply(json.loads).apply(list) 19 | 20 | return df 21 | 22 | 23 | def get_demo_data(nrows=None): 24 | """Get a demo ``pandas.DataFrame`` containing the accepted data format. 25 | 26 | Args: 27 | nrows (int): 28 | Number of rows to load from the demo datasets. 29 | 30 | Returns: 31 | A ``pd.DataFrame`` containing as ``values`` the signal values. 32 | """ 33 | df = _load_demo(nrows) 34 | df = df.explode('values').reset_index(drop=True) 35 | 36 | time_delta = pd.to_timedelta(list(range(400)) * 750, 's') 37 | df['timestamp'] = df['timestamp'] + time_delta 38 | return df 39 | 40 | 41 | def get_demo_primitives(): 42 | """Get a dict of demo transformation and aggregation primitives. 43 | 44 | Returns: 45 | A tuple containing the list of transformation primitives and 46 | the list aggregation primitives 47 | """ 48 | transformations = [ 49 | { 50 | "name": "fft", 51 | "primitive": "sigpro.transformations.frequency.fft.fft" 52 | } 53 | ] 54 | aggregations = [ 55 | { 56 | "name": "mean", 57 | "primitive": "sigpro.aggregations.amplitude.statistical.mean" 58 | }, 59 | { 60 | "name": "std", 61 | "primitive": "sigpro.aggregations.amplitude.statistical.std" 62 | } 63 | ] 64 | 65 | return transformations, aggregations 66 | 67 | 68 | def get_amplitude_demo(index=None): 69 | """Get amplitude values and sampling frequency used. 70 | 71 | The amplitude demo data is meant to be used for the any ``transformation`` functions 72 | that recieve as an input ``amplitude_values`` and ``sampling_frequency``. 73 | 74 | This amplitude values are loaded from the demo data without any transformations 75 | being applied to those values. There are `750` different signals. You can specify 76 | the desired index in order to retrive the same signal over and over, otherwise it 77 | will return a random signal. 78 | 79 | Args: 80 | index (int or None): 81 | If `int`, return the value at that index if `None` return a random index. 82 | 83 | Returns: 84 | tuple: 85 | A tuple with a `np.array` containing amplitude values and as second element the 86 | sampling frequency used. 87 | """ 88 | df = _load_demo() 89 | if index is None: 90 | index = random.randint(0, len(df) - 1) 91 | 92 | return np.array(df.iloc[index]['values']), 10000 93 | 94 | 95 | def get_frequency_demo(index=None, real=True): 96 | """Get amplitude values and the corresponding frequency values. 97 | 98 | The frequency demo data is meant to be used for the ``frequency aggregations`` 99 | functions that recieve as an input ``amplitude_values`` and ``frequency_values``. 100 | 101 | This amplitude values are loaded from the demo data with ``fft`` transformations 102 | being applied to those values. There are `750` different signals. You can specify 103 | the desired index in order to retrive the same signal over and over, otherwise it 104 | will return a random signal. 105 | 106 | Args: 107 | index (int or None): 108 | If `int`, return the value at that index if `None` return a random index. 109 | real (bool): 110 | If ``True``, return the real values for the computed ``fft`` transformations, 111 | if it's set to ``False`` it will return a complex ndarray. Defaults to ``True``. 112 | 113 | Returns: 114 | tuple: 115 | A tuple two `np.array` containing amplitude values and frequency values. 116 | """ 117 | amplitude_values, sampling_frequency = get_amplitude_demo(index) 118 | fft_values = np.fft.fft(amplitude_values) 119 | length = len(fft_values) 120 | frequencies = np.fft.fftfreq(len(fft_values), 1 / sampling_frequency) 121 | if real: 122 | fft_values = np.real(fft_values) 123 | 124 | return fft_values[0:length // 2], frequencies[0:length // 2] 125 | 126 | 127 | def get_frequency_time_demo(index=None, real=True): 128 | """Get amplitude values, frequency values and time values. 129 | 130 | The frequency time demo data is meant to be used for the ``frequency time aggregations`` 131 | functions that recieve as an input ``amplitude_values``, ``frequency_values`` and 132 | ``time_values``. 133 | 134 | This amplitude values are loaded from the demo data with ``fft`` transformations 135 | being applied to those values then a ``stft`` is being computed. There are `750` 136 | different signals. You can specify the desired index in order to retrive the same 137 | signal over and over, otherwise it will return a random signal. 138 | 139 | Args: 140 | index (int or None): 141 | If `int`, return the value at that index if `None` return a random index. 142 | real (bool): 143 | If ``True``, return the real values for the computed ``stft`` transformations, 144 | if it's set to ``False`` it will return a complex ndarray. Defaults to ``True``. 145 | 146 | Returns: 147 | tuple: 148 | A tuple two `np.array` containing amplitude values and frequency values. 149 | """ 150 | amplitude_values, sampling_frequency = get_amplitude_demo(index) 151 | sample_frequencies, time_values, amplitude_values = stft( 152 | amplitude_values, 153 | fs=sampling_frequency 154 | ) 155 | 156 | if real: 157 | amplitude_values = np.real(amplitude_values) 158 | 159 | return amplitude_values, sample_frequencies, time_values 160 | -------------------------------------------------------------------------------- /sigpro/primitive.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """SigPro Primitive class.""" 3 | 4 | import copy 5 | 6 | from mlblocks.mlblock import import_object 7 | 8 | from sigpro.contributing import ( 9 | _check_primitive_type_and_subtype, _get_primitive_args, _get_primitive_spec, 10 | _make_primitive_dict, _write_primitive) 11 | 12 | 13 | class Primitive(): # pylint: disable=too-many-instance-attributes 14 | """ 15 | Represents a SigPro primitive. 16 | 17 | Each primitive object represents a specific transformation or aggregation. Moreover, 18 | a Primitive maintains all the information in its JSON annotation as well as its 19 | hyperparameter values. 20 | 21 | Args: 22 | primitive (str): 23 | The name of the primitive, the python path including the name of the 24 | module and the name of the function. 25 | primitive_type (str): 26 | Type of primitive. 27 | primitive_subtype (str): 28 | Subtype of the primitive. 29 | init_params (dict): 30 | Initial (fixed) hyperparameter values of the primitive in 31 | {hyperparam_name: hyperparam_value} format. 32 | """ 33 | 34 | def __init__(self, primitive, primitive_type, primitive_subtype, init_params=None): 35 | 36 | self.primitive = primitive 37 | self.tag = primitive.split('.')[-1] 38 | self.primitive_type = primitive_type 39 | self.primitive_subtype = primitive_subtype 40 | self.tunable_hyperparameters = {} 41 | self.fixed_hyperparameters = {} 42 | self.context_arguments = [] 43 | primitive_spec = _get_primitive_spec(primitive_type, primitive_subtype) 44 | self.primitive_inputs = primitive_spec['args'] 45 | self.primitive_outputs = primitive_spec['output'] 46 | 47 | _check_primitive_type_and_subtype(primitive_type, primitive_subtype) 48 | 49 | self.primitive_function = import_object(primitive) 50 | if init_params is None: 51 | init_params = {} 52 | self.hyperparameter_values = init_params 53 | 54 | def get_name(self): 55 | """Get the name of the primitive.""" 56 | return self.primitive 57 | 58 | def get_tag(self): 59 | """Get the tag of the primitive.""" 60 | return self.tag 61 | 62 | def get_inputs(self): 63 | """Get the inputs of the primitive.""" 64 | return copy.deepcopy(self.primitive_inputs) 65 | 66 | def get_outputs(self): 67 | """Get the outputs of the primitive.""" 68 | return copy.deepcopy(self.primitive_outputs) 69 | 70 | def get_type_subtype(self): 71 | """Get the type and subtype of the primitive.""" 72 | return self.primitive_type, self.primitive_subtype 73 | 74 | def get_context_arguments(self): 75 | """Get the context arguments of the primitive.""" 76 | return copy.deepcopy(self.context_arguments) 77 | 78 | def _validate_primitive_spec(self): # check compatibility of given parameters. 79 | _get_primitive_args( 80 | self.primitive_function, 81 | self.primitive_inputs, 82 | self.context_arguments, 83 | self.fixed_hyperparameters, 84 | self.tunable_hyperparameters) 85 | 86 | def get_hyperparam_dict(self): 87 | """Return the dictionary of fixed hyperparameters for use in Pipelines.""" 88 | return {'name': self.get_tag(), 'primitive': self.get_name(), 89 | 'init_params': copy.deepcopy(self.hyperparameter_values)} 90 | 91 | def set_tag(self, tag): 92 | """Set the tag of a primitive.""" 93 | self.tag = tag 94 | return self 95 | 96 | def set_primitive_inputs(self, primitive_inputs): 97 | """Set primitive inputs.""" 98 | self.primitive_inputs = primitive_inputs 99 | 100 | def set_primitive_outputs(self, primitive_outputs): 101 | """Set primitive outputs.""" 102 | self.primitive_outputs = primitive_outputs 103 | 104 | def _set_primitive_type(self, primitive_type): 105 | self.primitive_type = primitive_type 106 | 107 | def _set_primitive_subtype(self, primitive_subtype): 108 | self.primitive_subtype = primitive_subtype 109 | 110 | def set_context_arguments(self, context_arguments): 111 | """Set context_arguments of a primitive.""" 112 | self.context_arguments = context_arguments 113 | 114 | def set_tunable_hyperparameters(self, tunable_hyperparameters): 115 | """Set tunable hyperparameters of a primitive.""" 116 | self.tunable_hyperparameters = tunable_hyperparameters 117 | 118 | def set_fixed_hyperparameters(self, fixed_hyperparameters): 119 | """Set fixed hyperparameters of a primitive.""" 120 | self.fixed_hyperparameters = fixed_hyperparameters 121 | 122 | def make_primitive_json(self): 123 | """ 124 | View the primitive json produced by a Primitive object. 125 | 126 | Raises: 127 | ValueError: 128 | If the primitive specification arguments are not valid (as in sigpro.contributing). 129 | 130 | Returns: 131 | dict: 132 | Dictionary containing the JSON annotation for the primitive. 133 | """ 134 | self._validate_primitive_spec() 135 | return _make_primitive_dict(self.primitive, self.primitive_type, 136 | self.primitive_subtype, self.context_arguments, 137 | self.fixed_hyperparameters, self.tunable_hyperparameters, 138 | self.primitive_inputs, self.primitive_outputs) 139 | 140 | def write_primitive_json(self, primitives_path='sigpro/primitives', 141 | primitives_subfolders=True): 142 | """ 143 | Write the primitive json produced by a Primitive object and return the path. 144 | 145 | Args: 146 | primitives_path (str): 147 | Path to the root of the primitives folder, in which the primitives JSON will be 148 | stored. Defaults to `sigpro/primitives`. 149 | primitives_subfolders (bool): 150 | Whether to store the primitive JSON in a subfolder tree (``True``) or to use a flat 151 | primitive name (``False``). Defaults to ``True``. 152 | 153 | Returns: 154 | str: 155 | Path of the generated JSON file. 156 | """ 157 | return _write_primitive(self.make_primitive_json(), self.primitive, 158 | primitives_path, primitives_subfolders) 159 | 160 | # Primitive inheritance subclasses 161 | # pylint: disable=super-init-not-called 162 | # Transformations 163 | 164 | 165 | class TransformationPrimitive(Primitive): 166 | """Generic transformation primitive.""" 167 | 168 | def __init__(self, primitive, primitive_subtype, init_params=None): 169 | super().__init__(primitive, 'transformation', primitive_subtype, init_params=init_params) 170 | 171 | 172 | class AmplitudeTransformation(TransformationPrimitive): 173 | """Generic amplitude transformation primitive.""" 174 | 175 | def __init__(self, primitive, init_params=None): 176 | super().__init__(primitive, 'amplitude', init_params=init_params) 177 | 178 | 179 | class FrequencyTransformation(TransformationPrimitive): 180 | """Generic frequency transformation primitive.""" 181 | 182 | def __init__(self, primitive, init_params=None): 183 | super().__init__(primitive, 'frequency', init_params=init_params) 184 | 185 | 186 | class FrequencyTimeTransformation(TransformationPrimitive): 187 | """Generic frequency-time transformation primitive.""" 188 | 189 | def __init__(self, primitive, init_params=None): 190 | super().__init__(primitive, 'frequency_time', init_params=init_params) 191 | 192 | 193 | class ComparativeTransformation(TransformationPrimitive): 194 | """Generic comparative transformation primitive.""" 195 | 196 | def __init__(self, primitive, init_params=None): 197 | raise NotImplementedError 198 | # Aggregations 199 | 200 | 201 | class AggregationPrimitive(Primitive): 202 | """Generic aggregation primitive.""" 203 | 204 | def __init__(self, primitive, primitive_subtype, init_params=None): 205 | super().__init__(primitive, 'aggregation', primitive_subtype, init_params=init_params) 206 | 207 | 208 | class AmplitudeAggregation(AggregationPrimitive): 209 | """Generic amplitude aggregation primitive.""" 210 | 211 | def __init__(self, primitive, init_params=None): 212 | super().__init__(primitive, 'amplitude', init_params=init_params) 213 | 214 | 215 | class FrequencyAggregation(AggregationPrimitive): 216 | """Generic frequency aggregation primitive.""" 217 | 218 | def __init__(self, primitive, init_params=None): 219 | super().__init__(primitive, 'frequency', init_params=init_params) 220 | 221 | 222 | class FrequencyTimeAggregation(AggregationPrimitive): 223 | """Generic frequency-time aggregation primitive.""" 224 | 225 | def __init__(self, primitive, init_params=None): 226 | super().__init__(primitive, 'frequency_time', init_params=init_params) 227 | 228 | 229 | class ComparativeAggregation(AggregationPrimitive): 230 | """Generic comparative aggregation primitive.""" 231 | 232 | def __init__(self, primitive, init_params=None): 233 | raise NotImplementedError 234 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/SigPro.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.SigPro", 3 | "primitive": "sigpro.SigPro", 4 | "classifiers": { 5 | "type": "preprocessor", 6 | "subtype": "feature_extractor" 7 | }, 8 | "produce": { 9 | "method": "process_signal", 10 | "args": "get_input_args", 11 | "output": "get_output_args" 12 | }, 13 | "hyperparameters": { 14 | "fixed": { 15 | "keep_columns": { 16 | "type": "bool or list", 17 | "default": false 18 | }, 19 | "values_column_name": { 20 | "type": "str", 21 | "default": "values" 22 | }, 23 | "transformations": { 24 | "type": "list" 25 | }, 26 | "aggregations": { 27 | "type": "list" 28 | }, 29 | "input_is_dataframe": { 30 | "type": "bool", 31 | "default": true 32 | } 33 | } 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/crest_factor.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.rms", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.rms", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "crest_factor_value", 18 | "type": "float" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/kurtosis.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.kurtosis", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.kurtosis", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "kurtosis_value", 18 | "type": "float" 19 | } 20 | ] 21 | }, 22 | "hyperparameters": { 23 | "fixed": { 24 | "fisher": { 25 | "type": "bool", 26 | "default": true 27 | }, 28 | "bias": { 29 | "type": "bool", 30 | "default": true 31 | } 32 | } 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/mean.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.mean", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.mean", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "mean_value", 18 | "type": "float" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/rms.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.rms", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.rms", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "rms_value", 18 | "type": "float" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/skew.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.skew", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.skew", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "skew_value", 18 | "type": "float" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/std.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.std", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.std", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "std_value", 18 | "type": "float" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/amplitude/statistical/var.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.amplitude.statistical.var", 3 | "primitive": "sigpro.aggregations.amplitude.statistical.var", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "var_value", 18 | "type": "float" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/frequency/band/band_mean.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.frequency.band.band_mean", 3 | "primitive": "sigpro.aggregations.frequency.band.band_mean", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "frequency" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "frequency_values", 16 | "type": "numpy.ndarray" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "value", 22 | "type": "float" 23 | } 24 | ] 25 | }, 26 | "hyperparameters": { 27 | "fixed": { 28 | "min_frequency": { 29 | "type": "float" 30 | }, 31 | "max_frequency": { 32 | "type": "float" 33 | } 34 | }, 35 | "tunable": {} 36 | } 37 | } -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/aggregations/frequency/band/band_rms.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.aggregations.frequency.band.band_rms", 3 | "primitive": "sigpro.aggregations.frequency.band.band_rms", 4 | "classifiers": { 5 | "type": "aggregation", 6 | "subtype": "frequency" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "frequency_values", 16 | "type": "numpy.ndarray" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "value", 22 | "type": "float" 23 | } 24 | ] 25 | }, 26 | "hyperparameters": { 27 | "fixed": { 28 | "min_frequency": { 29 | "type": "float" 30 | }, 31 | "max_frequency": { 32 | "type": "float" 33 | } 34 | }, 35 | "tunable": {} 36 | } 37 | } -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/amplitude/identity/identity.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.amplitude.identity.identity", 3 | "primitive": "sigpro.transformations.amplitude.identity.identity", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | } 14 | ], 15 | "output": [ 16 | { 17 | "name": "amplitude_values", 18 | "type": "numpy.ndarray" 19 | } 20 | ] 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/amplitude/spectrum/power_spectrum.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.amplitude.spectrum.power_spectrum", 3 | "primitive": "sigpro.transformations.amplitude.spectrum.power_spectrum", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "amplitude" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "sampling_frequency", 16 | "type": "float" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | } 28 | ] 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/frequency/band/frequency_band.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.frequency.band.frequency_band", 3 | "primitive": "sigpro.transformations.frequency.band.frequency_band", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "frequency" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "frequency_values", 16 | "type": "numpy.ndarray" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | } 28 | ] 29 | }, 30 | "hyperparameters": { 31 | "fixed": { 32 | "low": { 33 | "type": "int" 34 | }, 35 | "high": { 36 | "type": "int" 37 | } 38 | }, 39 | "tunable": {} 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/frequency/fft/fft.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.frequency.fft.fft", 3 | "primitive": "sigpro.transformations.frequency.fft.fft", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "frequency" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "sampling_frequency", 16 | "type": "float" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | } 28 | ] 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/frequency/fft/fft_real.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.frequency.fft.fft_real", 3 | "primitive": "sigpro.transformations.frequency.fft.fft_real", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "frequency" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "sampling_frequency", 16 | "type": "float" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | } 28 | ] 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/frequency/fftfreq/fft_freq.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.frequency.fftfreq.fft_freq", 3 | "primitive": "sigpro.transformations.frequency.fftfreq.fft_freq", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "frequency" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "sampling_frequency", 16 | "type": "float" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | } 28 | ] 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/frequency_time/stft/stft.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.frequency_time.stft.stft", 3 | "primitive": "sigpro.transformations.frequency_time.stft.stft", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "frequency_time" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "sampling_frequency", 16 | "type": "float" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | }, 28 | { 29 | "name": "time_values", 30 | "type": "numpy.ndarray" 31 | } 32 | ] 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /sigpro/primitives/sigpro/transformations/frequency_time/stft/stft_real.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "sigpro.transformations.frequency_time.stft.stft_real", 3 | "primitive": "sigpro.transformations.frequency_time.stft.stft_real", 4 | "classifiers": { 5 | "type": "transformation", 6 | "subtype": "frequency_time" 7 | }, 8 | "produce": { 9 | "args": [ 10 | { 11 | "name": "amplitude_values", 12 | "type": "numpy.ndarray" 13 | }, 14 | { 15 | "name": "sampling_frequency", 16 | "type": "float" 17 | } 18 | ], 19 | "output": [ 20 | { 21 | "name": "real_amplitude_values", 22 | "type": "numpy.ndarray" 23 | }, 24 | { 25 | "name": "frequency_values", 26 | "type": "numpy.ndarray" 27 | }, 28 | { 29 | "name": "time_values", 30 | "type": "numpy.ndarray" 31 | } 32 | ] 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /sigpro/transformations/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations module.""" 2 | -------------------------------------------------------------------------------- /sigpro/transformations/amplitude/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Amplitude module.""" 2 | -------------------------------------------------------------------------------- /sigpro/transformations/amplitude/identity.py: -------------------------------------------------------------------------------- 1 | """Transformations Amplitude Identity module.""" 2 | 3 | 4 | def identity(amplitude_values): 5 | """Return the identity of the amplitude values.""" 6 | return amplitude_values 7 | -------------------------------------------------------------------------------- /sigpro/transformations/amplitude/spectrum.py: -------------------------------------------------------------------------------- 1 | """Build power spectrum from amplitude values.""" 2 | 3 | 4 | import numpy as np 5 | 6 | 7 | def power_spectrum(amplitude_values, sampling_frequency): 8 | """Apply an RFFT on the amplitude values and return the real components. 9 | 10 | This computes the discrete Fourier Transform using the `rfft` function 11 | from `numpy.fft` module and compute the frequency values using the 12 | `rfftfreq` from the same module. 13 | 14 | Args: 15 | amplitude_values (np.ndarray): 16 | A numpy array with the signal values. 17 | sampling_frequency (int or float): 18 | Sampling frequency value passed in Hz. 19 | 20 | Returns: 21 | tuple: 22 | * `amplitude_values (numpy.ndarray)` 23 | * `frequency_values (numpy.ndarray)` 24 | """ 25 | frequency_values = np.fft.rfftfreq(len(amplitude_values), 1 / sampling_frequency) 26 | amplitude_values = np.abs(np.fft.rfft(amplitude_values)) ** 2 27 | 28 | return amplitude_values, frequency_values 29 | -------------------------------------------------------------------------------- /sigpro/transformations/frequency/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Frequency module.""" 2 | -------------------------------------------------------------------------------- /sigpro/transformations/frequency/band.py: -------------------------------------------------------------------------------- 1 | """SigPro Frequency Band module.""" 2 | 3 | 4 | def frequency_band(amplitude_values, frequency_values, low, high): 5 | """Extract a specific band. 6 | 7 | Filter between a high and low band frequency and return the amplitude values and frequency 8 | values for those. 9 | 10 | Args: 11 | amplitude_values (np.ndarray): 12 | A numpy array with the signal values. 13 | frequency_values (np.ndarray): 14 | A numpy array with the frequency values. 15 | Returns: 16 | tuple: 17 | * `amplitude_values (numpy.ndarray)` for the selected frequency values. 18 | * `frequency_values (numpy.ndarray)` for the selected frequency values. 19 | """ 20 | mask = (frequency_values > low) & (frequency_values < high) 21 | return amplitude_values[mask], frequency_values[mask] 22 | -------------------------------------------------------------------------------- /sigpro/transformations/frequency/fft.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Frequency FFT module.""" 2 | 3 | import numpy as np 4 | 5 | 6 | def fft(amplitude_values, sampling_frequency): 7 | """Apply an FFT on the amplitude values and return the real components. 8 | 9 | This computes the discrete Fourier Transform using the `fft` function 10 | from `numpy.fft` module and compute the frequency values using the 11 | `fftfreq` from the same module. 12 | 13 | Args: 14 | amplitude_values (np.ndarray): 15 | A numpy array with the signal values. 16 | sampling_frequency (int or float): 17 | Sampling frequency value passed in Hz. 18 | 19 | Returns: 20 | tuple: 21 | * `amplitude_values (numpy.ndarray)` 22 | * `frequency_values (numpy.ndarray)` 23 | """ 24 | amplitude_values = np.fft.fft(amplitude_values) 25 | frequency_values = np.fft.fftfreq(len(amplitude_values), 1 / sampling_frequency) 26 | 27 | return amplitude_values, frequency_values 28 | 29 | 30 | def fft_real(amplitude_values, sampling_frequency): 31 | """Apply an FFT on the amplitude values and return the real components. 32 | 33 | This computes the discrete Fourier Transform using the `fft` function 34 | from `numpy.fft` module and then extracting the real part of the 35 | returned vector to discard the complex components that represent 36 | the phase values. 37 | Also compute the frequency values using the `fftfreq` from the 38 | same module. 39 | 40 | Args: 41 | amplitude_values (np.ndarray): 42 | A numpy array with the signal values. 43 | sampling_frequency (int or float): 44 | Sampling frequency value passed in Hz. 45 | 46 | Returns: 47 | tuple: 48 | * `amplitude_values (numpy.ndarray)` 49 | * `frequency_values (numpy.ndarray)` 50 | """ 51 | amplitude_values, frequency_values = fft(amplitude_values, sampling_frequency) 52 | 53 | return np.real(amplitude_values), np.real(frequency_values) 54 | -------------------------------------------------------------------------------- /sigpro/transformations/frequency/fftfreq.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Frequency module.""" 2 | 3 | import numpy as np 4 | 5 | 6 | def fft_freq(amplitude_values, sampling_frequency): 7 | """Compute the Frequency having FFT values. 8 | 9 | Args: 10 | amplitude_values (np.ndarray): 11 | A numpy array with the fft values. 12 | sampling_frequency (int or float): 13 | Sampling frequency value passed in Hz. 14 | Returns: 15 | tuple: 16 | * `amplitude_values (numpy.ndarray)` 17 | * `frequency_values (numpy.ndarray)` 18 | """ 19 | # frequency_values = np.fft.fftfreq(len(amplitude_values), 1 / sampling_frequency) 20 | frequency_values = np.arange(0, len(amplitude_values)) * sampling_frequency 21 | 22 | return np.array(amplitude_values), np.array(frequency_values) 23 | -------------------------------------------------------------------------------- /sigpro/transformations/frequency_time/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Frequency Time module.""" 2 | -------------------------------------------------------------------------------- /sigpro/transformations/frequency_time/stft.py: -------------------------------------------------------------------------------- 1 | """Transformations Frequency Time - Short Time Fourier Transform module.""" 2 | 3 | import numpy as np 4 | import scipy.signal 5 | 6 | 7 | def stft(amplitude_values, sampling_frequency): 8 | """Compute the Short Time Fourier Transform. 9 | 10 | Args: 11 | amplitude_values (np.ndarray): 12 | A numpy array with the signal values. 13 | sampling_frequency (int or float): 14 | Sampling frequency value passed in Hz. 15 | Returns: 16 | tuple: 17 | * `amplitude_values (numpy.ndarray)` 18 | * `frequency_values (numpy.ndarray)` 19 | * `time_values (numpy.ndarray)` 20 | """ 21 | frequency_values, time_values, amplitude_values = scipy.signal.stft( 22 | amplitude_values, 23 | fs=sampling_frequency 24 | ) 25 | return amplitude_values, frequency_values, time_values 26 | 27 | 28 | def stft_real(amplitude_values, sampling_frequency): 29 | """Compute the Short Time Fourier Transform and it's real values. 30 | 31 | Args: 32 | amplitude_values (np.ndarray): 33 | A numpy array with the signal values. 34 | sampling_frequency (int or float): 35 | Sampling frequency value passed in Hz. 36 | Returns: 37 | tuple: 38 | * `amplitude_values (numpy.ndarray)` 39 | * `frequency_values (numpy.ndarray)` 40 | * `time_values (numpy.ndarray)` 41 | """ 42 | frequency_values, time_values, amplitude_values = scipy.signal.stft( 43 | amplitude_values, 44 | fs=sampling_frequency 45 | ) 46 | return np.real(amplitude_values), frequency_values, time_values 47 | -------------------------------------------------------------------------------- /tasks.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import os 3 | import re 4 | import shutil 5 | import stat 6 | from pathlib import Path 7 | 8 | from invoke import task 9 | 10 | 11 | @task 12 | def pytest(c): 13 | c.run('python -m pytest --cov=sigpro') 14 | 15 | 16 | @task 17 | def install_minimum(c): 18 | with open('setup.py', 'r') as setup_py: 19 | lines = setup_py.read().splitlines() 20 | 21 | versions = [] 22 | started = False 23 | for line in lines: 24 | if started: 25 | if line == ']': 26 | break 27 | 28 | line = line.strip() 29 | line = re.sub(r',?<=?[\d.]*,?', '', line) 30 | line = re.sub(r'>=?', '==', line) 31 | line = re.sub(r"""['",]""", '', line) 32 | versions.append(line) 33 | 34 | elif line.startswith('install_requires = ['): 35 | started = True 36 | 37 | c.run(f'python -m pip install {" ".join(versions)}') 38 | 39 | 40 | @task 41 | def minimum(c): 42 | install_minimum(c) 43 | c.run('python -m pip check') 44 | c.run('python -m pytest') 45 | 46 | 47 | @task 48 | def readme(c): 49 | test_path = Path('tests/readme_test') 50 | if test_path.exists() and test_path.is_dir(): 51 | shutil.rmtree(test_path) 52 | 53 | cwd = os.getcwd() 54 | os.makedirs(test_path, exist_ok=True) 55 | shutil.copy('README.md', test_path / 'README.md') 56 | os.chdir(test_path) 57 | c.run('rundoc run --single-session python3 -t python3 README.md') 58 | os.chdir(cwd) 59 | shutil.rmtree(test_path) 60 | 61 | 62 | @task 63 | def tutorials(c): 64 | for ipynb_file in glob.glob('tutorials/*.ipynb') + glob.glob('tutorials/**/*.ipynb'): 65 | if '.ipynb_checkpoints' not in ipynb_file: 66 | c.run(( 67 | 'jupyter nbconvert --execute --ExecutePreprocessor.timeout=3600 ' 68 | f'--to=html --stdout {ipynb_file}' 69 | ), hide='out') 70 | 71 | 72 | @task 73 | def lint(c): 74 | c.run('flake8 sigpro') 75 | c.run('flake8 tests --ignore=D,SFS2') 76 | c.run('isort -c --recursive sigpro tests') 77 | # c.run('pydocstyle sigpro') 78 | c.run('pylint sigpro --rcfile=setup.cfg') 79 | 80 | 81 | def remove_readonly(func, path, _): 82 | "Clear the readonly bit and reattempt the removal" 83 | os.chmod(path, stat.S_IWRITE) 84 | func(path) 85 | 86 | 87 | @task 88 | def rmdir(c, path): 89 | try: 90 | shutil.rmtree(path, onerror=remove_readonly) 91 | except PermissionError: 92 | pass 93 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | def safe_compare_dataframes(first, second): 2 | """Compare two dataframes even if they have NaN values. 3 | 4 | Args: 5 | first (pandas.DataFrame): DataFrame to compare 6 | second (pandas.DataFrame): DataFrame to compare 7 | 8 | Returns: 9 | bool 10 | """ 11 | 12 | if first.isnull().all().all(): 13 | return first.equals(second) 14 | 15 | else: 16 | nulls = (first.isnull() == second.isnull()).all().all() 17 | values = (first[~first.isnull()] == second[~second.isnull()]).all().all() 18 | return nulls and values 19 | -------------------------------------------------------------------------------- /tests/integration/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sintel-dev/SigPro/36798ad91a70137c1688b56d8abbc21628724db8/tests/integration/__init__.py -------------------------------------------------------------------------------- /tests/integration/test_contributing.py: -------------------------------------------------------------------------------- 1 | """Test module for SigPro contributing module.""" 2 | import json 3 | import os 4 | import tempfile 5 | 6 | import pytest 7 | 8 | from sigpro.contributing import make_primitive, run_primitive 9 | 10 | EXPECTED_PRIMITIVE_DICT = { 11 | "name": "sigpro.aggregations.amplitude.statistical.mean", 12 | "primitive": "sigpro.aggregations.amplitude.statistical.mean", 13 | "classifiers": { 14 | "type": "aggregation", 15 | "subtype": "amplitude" 16 | }, 17 | "produce": { 18 | "args": [ 19 | { 20 | "name": "amplitude_values", 21 | "type": "numpy.ndarray" 22 | } 23 | ], 24 | "output": [ 25 | { 26 | "name": "value", 27 | "type": "float" 28 | } 29 | ] 30 | }, 31 | 'hyperparameters': { 32 | 'fixed': {}, 33 | 'tunable': {} 34 | } 35 | } 36 | 37 | 38 | def test_make_primitive_invalid_type(): 39 | with pytest.raises(ValueError): 40 | make_primitive('invalid', 'invalid', 'invalid') 41 | 42 | 43 | def test_make_primitive_invalid_subtype(): 44 | with pytest.raises(ValueError): 45 | make_primitive('invalid', 'aggregation', 'invalid') 46 | 47 | 48 | def test_make_primitive_invalid_name(): 49 | with pytest.raises(ImportError): 50 | make_primitive('invalid', 'aggregation', 'amplitude') 51 | 52 | 53 | def test_make_primitive_missing_additional_hyperparameters(): 54 | with pytest.raises(ValueError): 55 | make_primitive( 56 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 57 | 'aggregation', 58 | 'amplitude' 59 | ) 60 | 61 | 62 | def test_make_primitive_primitives_subfolders_true(): 63 | with tempfile.TemporaryDirectory('sigpro') as tmp_dir: 64 | expected_result = ['sigpro', 'aggregations', 'amplitude', 'statistical', 'mean.json'] 65 | expected_result = os.path.join(tmp_dir, *expected_result) 66 | result = make_primitive( 67 | 'sigpro.aggregations.amplitude.statistical.mean', 68 | 'aggregation', 69 | 'amplitude', 70 | primitives_path=tmp_dir, 71 | ) 72 | assert result == expected_result 73 | with open(result, 'rb') as created_primitive: 74 | primitive_dict = json.load(created_primitive) 75 | assert primitive_dict == EXPECTED_PRIMITIVE_DICT 76 | 77 | 78 | def test_make_primitive_primitives_subfolders_false(): 79 | with tempfile.TemporaryDirectory('sigpro') as tmp_dir: 80 | expected_result = 'sigpro.aggregations.amplitude.statistical.mean.json' 81 | expected_result = os.path.join(tmp_dir, expected_result) 82 | result = make_primitive( 83 | 'sigpro.aggregations.amplitude.statistical.mean', 84 | 'aggregation', 85 | 'amplitude', 86 | primitives_path=tmp_dir, 87 | primitives_subfolders=False 88 | ) 89 | assert result == expected_result 90 | with open(result, 'rb') as created_primitive: 91 | primitive_dict = json.load(created_primitive) 92 | assert primitive_dict == EXPECTED_PRIMITIVE_DICT 93 | 94 | 95 | def test_run_primitive_aggregation_no_hyperparameters(): 96 | result = run_primitive('sigpro.aggregations.amplitude.statistical.mean', demo_row_index=0) 97 | assert round(result, 6) == 0.021361 98 | 99 | 100 | def test_run_primitive_aggregation_hyperparameters(): 101 | result_false = run_primitive( 102 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 103 | demo_row_index=0, 104 | fisher=False 105 | ) 106 | 107 | result_default = run_primitive( 108 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 109 | demo_row_index=0, 110 | ) 111 | 112 | assert result_false != result_default 113 | assert round(result_false, 6) == 2.280983 114 | assert round(result_default, 6) == -0.719017 115 | 116 | 117 | def test_run_primitive_transformation(): 118 | result = run_primitive('sigpro.transformations.amplitude.spectrum.power_spectrum') 119 | assert len(result[0]) == 201 120 | assert len(result[1]) == 201 121 | -------------------------------------------------------------------------------- /tests/integration/test_contributing_primitive.py: -------------------------------------------------------------------------------- 1 | """Test module for SigPro contributing_primitive module.""" 2 | 3 | import json 4 | import os 5 | import tempfile 6 | 7 | from sigpro.basic_primitives import Mean 8 | from sigpro.contributing_primitive import make_primitive_class 9 | 10 | EXPECTED_PRIMITIVE_DICT = { 11 | "name": "sigpro.aggregations.amplitude.statistical.mean", 12 | "primitive": "sigpro.aggregations.amplitude.statistical.mean", 13 | "classifiers": { 14 | "type": "aggregation", 15 | "subtype": "amplitude" 16 | }, 17 | "produce": { 18 | "args": [ 19 | { 20 | "name": "amplitude_values", 21 | "type": "numpy.ndarray" 22 | } 23 | ], 24 | "output": [ 25 | { 26 | "name": "mean_value", 27 | "type": "float" 28 | } 29 | ] 30 | }, 31 | 'hyperparameters': { 32 | 'fixed': {}, 33 | 'tunable': {} 34 | } 35 | } 36 | 37 | 38 | def test_make_primitive_class_primitives_subfolders_true(): 39 | with tempfile.TemporaryDirectory('sigpro') as tmp_dir: 40 | expected_result = ['sigpro', 'aggregations', 'amplitude', 'statistical', 'mean.json'] 41 | expected_result = os.path.join(tmp_dir, *expected_result) 42 | primitive_outputs = [{'name': 'mean_value', 'type': 'float'}] 43 | Mean_dynamic, result = make_primitive_class( 44 | 'sigpro.aggregations.amplitude.statistical.mean', 45 | 'aggregation', 46 | 'amplitude', 47 | primitives_path=tmp_dir, 48 | primitive_outputs=primitive_outputs 49 | ) 50 | assert result == expected_result 51 | with open(result, 'rb') as created_primitive: 52 | primitive_dict = json.load(created_primitive) 53 | assert primitive_dict == EXPECTED_PRIMITIVE_DICT 54 | 55 | mean_instance = Mean_dynamic() 56 | mean_default = Mean() 57 | assert mean_instance.make_primitive_json() == mean_default.make_primitive_json() 58 | 59 | 60 | def test_make_primitive_class_primitives_subfolders_false(): 61 | with tempfile.TemporaryDirectory('sigpro') as tmp_dir: 62 | expected_result = 'sigpro.aggregations.amplitude.statistical.mean.json' 63 | expected_result = os.path.join(tmp_dir, expected_result) 64 | primitive_outputs = [{'name': 'mean_value', 'type': 'float'}] 65 | Mean_dynamic, result = make_primitive_class( 66 | 'sigpro.aggregations.amplitude.statistical.mean', 67 | 'aggregation', 68 | 'amplitude', 69 | primitives_path=tmp_dir, 70 | primitive_outputs=primitive_outputs, 71 | primitives_subfolders=False 72 | ) 73 | assert result == expected_result 74 | with open(result, 'rb') as created_primitive: 75 | primitive_dict = json.load(created_primitive) 76 | assert primitive_dict == EXPECTED_PRIMITIVE_DICT 77 | 78 | mean_instance = Mean_dynamic() 79 | mean_default = Mean() 80 | assert mean_instance.make_primitive_json() == mean_default.make_primitive_json() 81 | -------------------------------------------------------------------------------- /tests/integration/test_core.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | from mlblocks import MLPipeline 3 | 4 | 5 | def test_SigPro(): 6 | # setup 7 | pipeline = MLPipeline({ 8 | 'primitives': [ 9 | 'sigpro.SigPro', 10 | 'sigpro.SigPro', 11 | ], 12 | 'init_params': { 13 | 'sigpro.SigPro#1': { 14 | 'values_column_name': 'signal_values', 15 | 'keep_columns': True, 16 | 'transformations': [ 17 | { 18 | 'name': 'identity', 19 | 'primitive': 'sigpro.transformations.amplitude.identity.identity', 20 | }, 21 | { 22 | 'name': 'fft', 23 | 'primitive': 'sigpro.transformations.frequency.fft.fft_real', 24 | }, 25 | ], 26 | 'aggregations': [ 27 | { 28 | 'name': 'mean', 29 | 'primitive': 'sigpro.aggregations.amplitude.statistical.mean', 30 | }, 31 | { 32 | 'name': 'rms', 33 | 'primitive': 'sigpro.aggregations.amplitude.statistical.rms', 34 | }, 35 | ] 36 | }, 37 | 'sigpro.SigPro#2': { 38 | 'values_column_name': 'signal_values', 39 | 'keep_columns': ['dummy'], 40 | 'transformations': [ 41 | { 42 | 'name': 'identity', 43 | 'primitive': 'sigpro.transformations.amplitude.identity.identity', 44 | }, 45 | ], 46 | 'aggregations': [ 47 | { 48 | 'name': 'std', 49 | 'primitive': 'sigpro.aggregations.amplitude.statistical.std', 50 | }, 51 | ] 52 | } 53 | } 54 | }) 55 | 56 | data = pd.DataFrame({ 57 | 'timestamp': pd.to_datetime(['2020-01-01 00:00:00']), 58 | 'signal_values': [[1, 2, 3, 4]], 59 | 'sampling_frequency': [1000], 60 | 'dummy': [1], 61 | }) 62 | 63 | # run 64 | output = pipeline.predict(readings=data) 65 | outputs = dict(zip(pipeline.get_output_names(), output)) 66 | 67 | # assert 68 | expected_features = [ 69 | 'identity.fft.mean.mean_value', 70 | 'identity.fft.rms.rms_value', 71 | 'identity.std.std_value' 72 | ] 73 | assert outputs['feature_columns'] == expected_features 74 | 75 | expected_readings = pd.DataFrame({ 76 | 'dummy': [1], 77 | 'identity.fft.mean.mean_value': [1.0], 78 | 'identity.fft.rms.rms_value': [5.291503], 79 | 'identity.std.std_value': [1.118034], 80 | }) 81 | pd.testing.assert_frame_equal(expected_readings, outputs['readings']) 82 | 83 | 84 | def test_SigPro_nested_pipeline(): 85 | """Test nested sigpro primitive.""" 86 | # setup 87 | aggregations = [ 88 | { 89 | 'primitive': 'sigpro.SigPro', 90 | 'init_params': { 91 | 'keep_columns': True, 92 | 'input_is_dataframe': False, 93 | 'values_column_name': 'amplitude_values', 94 | 'transformations': [{ 95 | 'primitive': 'sigpro.transformations.frequency.band.frequency_band', 96 | 'init_params': { 97 | 'low': 100, 98 | 'high': 200 99 | } 100 | }], 101 | 'aggregations': [{ 102 | 'primitive': 'sigpro.aggregations.amplitude.statistical.mean' 103 | }] 104 | } 105 | }, 106 | { 107 | 'primitive': 'sigpro.SigPro', 108 | 'init_params': { 109 | 'input_is_dataframe': False, 110 | 'values_column_name': 'amplitude_values', 111 | 'transformations': [{ 112 | 'primitive': 'sigpro.transformations.frequency.band.frequency_band', 113 | 'init_params': { 114 | 'low': 3000, 115 | 'high': 4000 116 | } 117 | }], 118 | 'aggregations': [{ 119 | 'name': 'band_3k_4k_mean', 120 | 'primitive': 'sigpro.aggregations.amplitude.statistical.mean' 121 | }], 122 | } 123 | }, 124 | { 125 | 'primitive': 'sigpro.aggregations.amplitude.statistical.mean' 126 | } 127 | ] 128 | 129 | pipeline = MLPipeline({ 130 | 'primitives': ['sigpro.SigPro'], 131 | 'init_params': { 132 | 'sigpro.SigPro#1': { 133 | 'transformations': [{ 134 | 'primitive': 'sigpro.transformations.frequency.fft.fft_real' 135 | }], 136 | 'aggregations': aggregations 137 | } 138 | } 139 | }) 140 | 141 | data = pd.DataFrame({ 142 | 'timestamp': pd.to_datetime(['2020-01-01 00:00:00']), 143 | 'values': [[1, 2, 3, 4, 5, 6]], 144 | 'sampling_frequency': [10000], 145 | 'dummy': [1], 146 | }) 147 | 148 | # run 149 | output = pipeline.predict(readings=data) 150 | outputs = dict(zip(pipeline.get_output_names(), output)) 151 | 152 | # assert 153 | expected_features = [ 154 | 'fft_real.SigPro.frequency_band.mean.mean_value', 155 | 'fft_real.SigPro.frequency_band.band_3k_4k_mean.mean_value', 156 | 'fft_real.mean.mean_value' 157 | ] 158 | 159 | assert outputs['feature_columns'] == expected_features 160 | expected_readings = pd.DataFrame({ 161 | 'fft_real.SigPro.frequency_band.mean.mean_value': [float('nan')], 162 | 'fft_real.SigPro.frequency_band.band_3k_4k_mean.mean_value': [-3.0], 163 | 'fft_real.mean.mean_value': [1.0], 164 | }) 165 | 166 | pd.testing.assert_frame_equal(expected_readings, outputs['readings']) 167 | -------------------------------------------------------------------------------- /tests/integration/test_demo.py: -------------------------------------------------------------------------------- 1 | """Test module for SigPro demo.""" 2 | import numpy as np 3 | 4 | from sigpro.demo import ( 5 | get_amplitude_demo, get_demo_data, get_demo_primitives, get_frequency_demo, 6 | get_frequency_time_demo) 7 | 8 | EXPECTED_SHAPE = (300000, 5) 9 | EXPECTED_COLUMNS = ['turbine_id', 'signal_id', 'timestamp', 'values', 'sampling_frequency'] 10 | EXPECTED_SAMPLING_FREQUENCY = 10000 11 | EXPECTED_VALUES_LENGTH = 400 12 | EXPECTED_FREQUENCY_LENGTH = 400 13 | 14 | 15 | def test_get_demo_data(): 16 | df = get_demo_data() 17 | assert EXPECTED_SHAPE == df.shape 18 | assert EXPECTED_COLUMNS == list(df.columns) 19 | 20 | 21 | def test_get_demo_primitives(): 22 | trans, aggs = get_demo_primitives() 23 | assert len(trans) == 1 24 | assert len(aggs) == 2 25 | for primitive in [*trans, *aggs]: 26 | assert 'name' in primitive 27 | assert 'primitive' in primitive 28 | 29 | 30 | def test_get_amplitude_demo_without_index(): 31 | values, sampling_frequency = get_amplitude_demo() 32 | assert EXPECTED_VALUES_LENGTH == len(values) 33 | assert EXPECTED_SAMPLING_FREQUENCY == sampling_frequency 34 | 35 | 36 | def test_get_amplitude_demo_indexed(): 37 | values, sampling_frequency = get_amplitude_demo(index=1) 38 | assert EXPECTED_VALUES_LENGTH == len(values) 39 | assert EXPECTED_SAMPLING_FREQUENCY == sampling_frequency 40 | 41 | 42 | def test_get_frequency_demo_without_index(): 43 | values, frequency_values = get_frequency_demo() 44 | assert EXPECTED_VALUES_LENGTH // 2 == len(values) 45 | assert EXPECTED_FREQUENCY_LENGTH // 2 == len(frequency_values) 46 | 47 | 48 | def test_get_frequency_demo_indexed(): 49 | values, frequency_values = get_frequency_demo(index=1) 50 | assert EXPECTED_VALUES_LENGTH // 2 == len(values) 51 | assert EXPECTED_FREQUENCY_LENGTH // 2 == len(frequency_values) 52 | 53 | 54 | def test_get_frequency_demo_complex(): 55 | values, frequency_values = get_frequency_demo(real=False) 56 | value = values[0] 57 | assert isinstance(value, np.complex128) 58 | assert EXPECTED_VALUES_LENGTH // 2 == len(values) 59 | assert EXPECTED_FREQUENCY_LENGTH // 2 == len(frequency_values) 60 | 61 | 62 | def test_get_frequency_time_demo_without_index(): 63 | values, frequencies, time_values = get_frequency_time_demo() 64 | assert 129 == len(values) 65 | assert 129 == len(frequencies) 66 | assert 5 == len(time_values) 67 | 68 | 69 | def test_get_frequency_time_demo_indexed(): 70 | values, frequencies, time_values = get_frequency_time_demo(index=1) 71 | assert 129 == len(values) 72 | assert 129 == len(frequencies) 73 | assert 5 == len(time_values) 74 | 75 | 76 | def test_get_frequency_time_demo_complex(): 77 | values, frequencies, time_values = get_frequency_time_demo(real=False) 78 | value = values[0][0] 79 | assert 129 == len(values) 80 | assert 129 == len(frequencies) 81 | assert 5 == len(time_values) 82 | assert isinstance(value, np.complex128) 83 | -------------------------------------------------------------------------------- /tests/integration/test_pipeline.py: -------------------------------------------------------------------------------- 1 | """Test module for SigPro pipeline module.""" 2 | import pandas as pd 3 | import pytest 4 | 5 | from sigpro import pipeline, primitive 6 | from sigpro.basic_primitives import FFT, BandMean, FFTReal, Identity, Kurtosis, Mean 7 | 8 | TEST_INPUT = pd.DataFrame({'timestamp': pd.to_datetime(['2020-01-01 00:00:00']), 9 | 'values': [[1, 2, 3, 4, 5, 6]], 10 | 'sampling_frequency': [10000], 11 | 'dummy': [1], }) 12 | 13 | TEST_OUTPUT = pd.DataFrame({'fftr.id1.bm.value': [(-3 + 0j)], 14 | 'fftr.id1.mean.mean_value': [(1 + 0j)], 15 | 'fftr.mean.mean_value': [(1 + 0j)], 16 | 'fftr.id1.kurtosis.kurtosis_value': [(4.2 + 0j)], 17 | 'fftr.id2.bm.value': [(-3 + 0j)], 18 | 'fftr.id2.mean.mean_value': [(1 + 0j)], 19 | 'fftr.id2.kurtosis.kurtosis_value': [(4.2 + 0j)], 20 | 'fft.id1.bm.value': [(-3 + 3.4641016151377544j)], 21 | 'fft.id1.mean.mean_value': [(1 + 0j)], 22 | 'fft.id1.kurtosis.kurtosis_value': [(5.34 + 3.866899242231838e-18j)], 23 | 'fft.id2.bm.value': [(-3 + 3.4641016151377544j)], 24 | 'fft.id2.mean.mean_value': [(1 + 0j)], 25 | 'fft.id2.kurtosis.kurtosis_value': [(5.34 + 3.866899242231838e-18j)]}) 26 | 27 | 28 | def _verify_pipeline_outputs(sigpro_pipeline, input_data, output_data, columns_to_check=None): 29 | """Verify that a pipeline produces the output data on a set of dataframe inputs.""" 30 | assert isinstance(sigpro_pipeline, pipeline.Pipeline) 31 | assert input_data is not None 32 | assert output_data is not None 33 | assert columns_to_check != [] 34 | 35 | processed_signal, feature_list = sigpro_pipeline.process_signal(input_data) 36 | if columns_to_check is None: 37 | columns_to_check = feature_list[:] 38 | for column in columns_to_check: 39 | assert column in feature_list 40 | assert column in processed_signal.columns 41 | cols_reduced = [i for i in columns_to_check if i in output_data.columns] 42 | pd.testing.assert_frame_equal( 43 | processed_signal[cols_reduced], 44 | output_data[cols_reduced], 45 | rtol=1e-2 46 | ) 47 | 48 | 49 | def test_linear_pipeline(): 50 | """build_linear_pipeline test.""" 51 | 52 | transformations = [Identity().set_tag('id1'), FFTReal().set_tag('fftr')] 53 | aggregations = [Mean(), Kurtosis(fisher=False)] 54 | 55 | assert isinstance(transformations[0], primitive.Primitive) 56 | 57 | sample_pipeline = pipeline.build_linear_pipeline(transformations, aggregations) # incomplete 58 | 59 | assert isinstance(sample_pipeline, pipeline.LinearPipeline) 60 | 61 | _verify_pipeline_outputs(sample_pipeline, TEST_INPUT, TEST_OUTPUT) 62 | 63 | 64 | def test_tree_pipeline(): 65 | """build_tree_pipeline test.""" 66 | 67 | t_layer1 = [FFTReal().set_tag('fftr'), FFT()] 68 | t_layer2 = [Identity().set_tag('id1'), Identity().set_tag('id2')] 69 | 70 | a_layer = [BandMean(200, 50000).set_tag('bm'), Mean(), Kurtosis(fisher=False)] 71 | 72 | sample_pipeline = pipeline.build_tree_pipeline([t_layer1, t_layer2], a_layer) 73 | 74 | assert isinstance(sample_pipeline, pipeline.LayerPipeline) 75 | assert len(set(sample_pipeline.get_output_features())) == 12 # 2 * 2 * 3 76 | 77 | _verify_pipeline_outputs(sample_pipeline, TEST_INPUT, TEST_OUTPUT) 78 | 79 | 80 | def test_layer_pipeline(): 81 | """build_layer_pipeline test.""" 82 | 83 | p1, p2 = FFTReal().set_tag('fftr'), FFT() 84 | p3, p4 = Identity().set_tag('id1'), Identity().set_tag('id2') 85 | p5, p6, p7 = BandMean(200, 50000).set_tag('bm'), Mean(), Kurtosis(fisher=False) 86 | p8 = Identity().set_tag('id3') # unused primitive 87 | 88 | all_primitives = [p1, p2, p3, p4, p5, p6, p7, p8] 89 | 90 | features = [(p1, p3, p5), (p1, p3, p6), (p2, p3, p6), (p2, p4, p6), (p2, p4, p7), (p1, p6)] 91 | 92 | sample_pipeline = pipeline.build_layer_pipeline(all_primitives, features) 93 | 94 | assert isinstance(sample_pipeline, pipeline.LayerPipeline) 95 | 96 | out_features = sample_pipeline.get_output_combinations() 97 | for feature in features: 98 | assert feature in out_features 99 | 100 | _verify_pipeline_outputs(sample_pipeline, TEST_INPUT, TEST_OUTPUT) 101 | 102 | 103 | def test_merge_pipelines(): 104 | """merge_pipelines test.""" 105 | p1, p2 = FFTReal().set_tag('fftr'), FFT() 106 | p3, p4 = Identity().set_tag('id1'), Identity().set_tag('id2') 107 | p5, p6, p7 = BandMean(200, 50000).set_tag('bm'), Mean(), Kurtosis(fisher=False) 108 | p8 = Identity().set_tag('id3') # unused primitive 109 | 110 | all_primitives = [p1, p2, p3, p4, p5, p6, p7, p8] 111 | 112 | features1 = [(p1, p3, p5), (p1, p3, p6), (p2, p3, p6), (p2, p4, p6), (p2, p4, p7)] 113 | 114 | sample_pipeline1 = pipeline.build_layer_pipeline(all_primitives, features1) 115 | 116 | sample_pipeline2 = pipeline.build_tree_pipeline([[p1, p2], [p3]], [p5]) 117 | 118 | sample_pipeline3 = pipeline.build_linear_pipeline([p1, p4], [p6]) 119 | 120 | features = features1 + [(p2, p3, p5), (p1, p4, p6)] 121 | sample_pipeline = pipeline.merge_pipelines([sample_pipeline1, 122 | sample_pipeline2, 123 | sample_pipeline3]) 124 | 125 | assert isinstance(sample_pipeline, pipeline.LayerPipeline) 126 | 127 | out_features = sample_pipeline.get_output_combinations() 128 | for feature in features: 129 | assert feature in out_features 130 | 131 | 132 | def test_invalid_tree_pipelines(): 133 | """Test invalid tree pipelines.""" 134 | 135 | p1, p2, p2_duplicate = FFTReal().set_tag('fftr'), FFT(), FFT() 136 | p3, p4 = Identity().set_tag('id1'), Identity().set_tag('id2') 137 | p5, p6, p7 = BandMean(200, 50000).set_tag('bm'), Mean(), Kurtosis(fisher=False) 138 | 139 | t_layer1 = [p1, p2] 140 | t_layer2 = [p3, p4] 141 | 142 | a_layer = [p5, p6, p7] 143 | 144 | # Empty Cartesian product 145 | with pytest.raises(ValueError): 146 | pipeline.build_tree_pipeline([t_layer1, []], a_layer) 147 | with pytest.raises(ValueError): 148 | pipeline.build_tree_pipeline([[], t_layer2], a_layer) 149 | with pytest.raises(ValueError): 150 | pipeline.build_tree_pipeline([t_layer1, t_layer2], []) 151 | 152 | # Duplicate tags 153 | with pytest.raises(ValueError): 154 | pipeline.build_tree_pipeline([t_layer1 + [p2_duplicate], t_layer2], a_layer) 155 | 156 | # Incorrect primitive order 157 | with pytest.raises(ValueError): 158 | pipeline.build_tree_pipeline([t_layer1, a_layer], t_layer2) 159 | 160 | 161 | def test_invalid_layer_pipelines(): 162 | """Test invalid pipeline formation.""" 163 | 164 | p1, p2, p2_duplicate = FFTReal().set_tag('fftr'), FFT(), FFT() 165 | p3, p4 = Identity().set_tag('id1'), Identity().set_tag('id2') 166 | p5, p6, p7 = BandMean(200, 50000).set_tag('bm'), Mean(), Kurtosis(fisher=False) 167 | p8 = Identity().set_tag('id3') # unused primitive 168 | 169 | all_primitives = [p1, p2, p3, p4, p5, p6, p7, p8] 170 | 171 | all_primitives_duplicate = all_primitives + [p2_duplicate] 172 | 173 | features = [(p1, p3, p5), (p1, p3, p6), (p2, p3, p6), (p2, p4, p6), (p2, p4, p7)] 174 | 175 | no_agg_end = (p1, p3, p4) 176 | intermediate_agg = (p1, p6, p7) 177 | 178 | blank_features = [tuple(), tuple()] 179 | 180 | # Primitive in combination not contained in primitives 181 | with pytest.raises(ValueError): 182 | pipeline.build_layer_pipeline([p1, p2, p3, p5, p6, p7, p8], features) 183 | 184 | # Duplicate primitive 185 | with pytest.raises(ValueError): 186 | pipeline.build_layer_pipeline(all_primitives_duplicate, features) 187 | 188 | # No nontrivial features 189 | with pytest.raises(ValueError): 190 | pipeline.build_layer_pipeline([p1, p3, p5], blank_features) 191 | 192 | with pytest.raises(ValueError): 193 | pipeline.build_layer_pipeline([p1, p3, p5], []) 194 | 195 | # At least one feature in incorrect format 196 | with pytest.raises(ValueError): 197 | pipeline.build_layer_pipeline(all_primitives, features + [no_agg_end]) 198 | 199 | with pytest.raises(ValueError): 200 | pipeline.build_layer_pipeline(all_primitives, features + [intermediate_agg]) 201 | -------------------------------------------------------------------------------- /tests/integration/test_primitive.py: -------------------------------------------------------------------------------- 1 | """Test module for SigPro primitive and basic_primitives modules.""" 2 | 3 | from sigpro import basic_primitives, primitive 4 | 5 | 6 | def test_basic_primitives(): 7 | """Test basic_primitives module.""" 8 | 9 | identity = basic_primitives.Identity() 10 | power_spectrum = basic_primitives.PowerSpectrum() 11 | assert isinstance(identity, primitive.Primitive) 12 | assert isinstance(power_spectrum, primitive.Primitive) 13 | assert identity.get_type_subtype() == ('transformation', 'amplitude') 14 | assert power_spectrum.get_type_subtype() == ('transformation', 'amplitude') 15 | identity.make_primitive_json() 16 | power_spectrum.make_primitive_json() 17 | 18 | fft = basic_primitives.FFT() 19 | fft_real = basic_primitives.FFTReal() 20 | 21 | assert isinstance(fft, primitive.Primitive) 22 | assert isinstance(fft_real, primitive.Primitive) 23 | assert fft.get_type_subtype() == ('transformation', 'frequency') 24 | assert fft_real.get_type_subtype() == ('transformation', 'frequency') 25 | fft.make_primitive_json() 26 | fft_real.make_primitive_json() 27 | 28 | frequency_band = basic_primitives.FrequencyBand(low=10, high=20) 29 | assert isinstance(frequency_band, primitive.Primitive) 30 | assert frequency_band.get_type_subtype() == ('transformation', 'frequency') 31 | frequency_band.make_primitive_json() 32 | 33 | stft = basic_primitives.STFT() 34 | stft_real = basic_primitives.STFTReal() 35 | assert isinstance(stft, primitive.Primitive) 36 | assert isinstance(stft_real, primitive.Primitive) 37 | assert stft.get_type_subtype() == ('transformation', 'frequency_time') 38 | assert stft_real.get_type_subtype() == ('transformation', 'frequency_time') 39 | stft.make_primitive_json() 40 | stft_real.make_primitive_json() 41 | 42 | std = basic_primitives.Std() 43 | var = basic_primitives.Var() 44 | rms = basic_primitives.RMS() 45 | cf = basic_primitives.CrestFactor() 46 | skew = basic_primitives.Skew() 47 | assert isinstance(std, primitive.Primitive) 48 | assert isinstance(var, primitive.Primitive) 49 | assert isinstance(rms, primitive.Primitive) 50 | assert isinstance(cf, primitive.Primitive) 51 | assert isinstance(skew, primitive.Primitive) 52 | assert std.get_type_subtype() == ('aggregation', 'amplitude') 53 | assert var.get_type_subtype() == ('aggregation', 'amplitude') 54 | assert rms.get_type_subtype() == ('aggregation', 'amplitude') 55 | assert cf.get_type_subtype() == ('aggregation', 'amplitude') 56 | assert skew.get_type_subtype() == ('aggregation', 'amplitude') 57 | std.make_primitive_json() 58 | var.make_primitive_json() 59 | rms.make_primitive_json() 60 | cf.make_primitive_json() 61 | skew.make_primitive_json() 62 | 63 | band_mean = basic_primitives.BandMean(min_frequency=0, max_frequency=100) 64 | assert isinstance(band_mean, primitive.Primitive) 65 | assert band_mean.get_type_subtype() == ('aggregation', 'frequency') 66 | band_mean.make_primitive_json() 67 | 68 | 69 | def test_primitives(): 70 | """Test primitives module.""" 71 | 72 | kurtosis = basic_primitives.Kurtosis(bias=False) 73 | kurtosis.set_tag('kurtosis_test') 74 | primitive_str = 'sigpro.aggregations.amplitude.statistical.kurtosis' 75 | init_params = {'fisher': True, 'bias': False} 76 | assert kurtosis.get_hyperparam_dict() == {'name': 'kurtosis_test', 77 | 'primitive': primitive_str, 78 | 'init_params': init_params} 79 | kurtosis.make_primitive_json() 80 | frequency_band = basic_primitives.FrequencyBand(low=10, high=50) 81 | frequency_band.set_tag('frequency_band_test') 82 | primitive_str = 'sigpro.transformations.frequency.band.frequency_band' 83 | init_params = {'low': 10, 'high': 50} 84 | assert frequency_band.get_hyperparam_dict() == {'name': 'frequency_band_test', 85 | 'primitive': primitive_str, 86 | 'init_params': init_params} 87 | -------------------------------------------------------------------------------- /tests/requirement_files/latest_requirements.txt: -------------------------------------------------------------------------------- 1 | mlblocks==0.5.0 2 | numpy==1.24.2 3 | pandas==1.5.3 4 | psutil==5.9.4 5 | -------------------------------------------------------------------------------- /tests/unit/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sintel-dev/SigPro/36798ad91a70137c1688b56d8abbc21628724db8/tests/unit/__init__.py -------------------------------------------------------------------------------- /tests/unit/aggregations/__init__.py: -------------------------------------------------------------------------------- 1 | """Aggregations test module.""" 2 | -------------------------------------------------------------------------------- /tests/unit/aggregations/amplitude/__init__.py: -------------------------------------------------------------------------------- 1 | """Test module for Aggregations Amplitude""" 2 | -------------------------------------------------------------------------------- /tests/unit/aggregations/amplitude/test_statistical.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """Tests for sigpro.aggregations.amplitude.statistical package.""" 4 | 5 | from sigpro.aggregations.amplitude.statistical import ( 6 | crest_factor, kurtosis, mean, rms, skew, std, var) 7 | 8 | VALUES = list(range(20)) 9 | 10 | 11 | def test_crest_factor(): 12 | expected = 19. / 11.113055 13 | result = crest_factor(VALUES) 14 | assert round(result, 4) == round(expected, 4) 15 | 16 | 17 | def test_rms(): 18 | result = rms(VALUES) 19 | assert round(result, 6) == 11.113055 20 | 21 | 22 | def test_mean(): 23 | result = mean(VALUES) 24 | assert result == 9.5 25 | 26 | 27 | def test_std(): 28 | result = std(VALUES) 29 | assert round(result, 6) == 5.766281 30 | 31 | 32 | def test_var(): 33 | result = var(VALUES) 34 | assert result == 33.25 35 | 36 | 37 | def test_skew(): 38 | result = skew(VALUES) 39 | assert result == 0.0 40 | 41 | 42 | def test_kurtosis_fisher(): 43 | result = kurtosis(VALUES) 44 | assert round(result, 6) == -1.206015 45 | 46 | 47 | def test_kurtosis_fisher_bias_false(): 48 | result = kurtosis(VALUES, bias=False) 49 | assert result == -1.2 50 | 51 | 52 | def test_kurtosis_pearson(): 53 | result = kurtosis(VALUES, fisher=False) 54 | assert round(result, 6) == 1.793985 55 | 56 | 57 | def test_kurtosis_pearson_bias_false(): 58 | result = kurtosis(VALUES, fisher=False, bias=False) 59 | assert result == 1.8 60 | -------------------------------------------------------------------------------- /tests/unit/aggregations/frequency/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sintel-dev/SigPro/36798ad91a70137c1688b56d8abbc21628724db8/tests/unit/aggregations/frequency/__init__.py -------------------------------------------------------------------------------- /tests/unit/aggregations/frequency/test_band.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """Tests for sigpro.aggregations.frequency.band package.""" 4 | 5 | import numpy as np 6 | 7 | from sigpro.aggregations.frequency.band import band_mean 8 | 9 | AMPLITUDE_VALUES = np.arange(200) 10 | FREQUENCY_VALUES = np.arange(200) 11 | 12 | 13 | def test_band_mean(): 14 | # setup 15 | expected = 15 # we computed previoulsy 16 | 17 | # run 18 | result = band_mean(AMPLITUDE_VALUES, FREQUENCY_VALUES, min_frequency=10, max_frequency=20) 19 | 20 | # assert 21 | assert result == expected 22 | -------------------------------------------------------------------------------- /tests/unit/test___init__.py: -------------------------------------------------------------------------------- 1 | def test(): 2 | pass 3 | -------------------------------------------------------------------------------- /tests/unit/transformations/amplitude/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Amplitude Test module.""" 2 | -------------------------------------------------------------------------------- /tests/unit/transformations/amplitude/identity.py: -------------------------------------------------------------------------------- 1 | """Test identity module.""" 2 | 3 | from sigpro.transformations.amplitude.identity import identity 4 | 5 | 6 | def test_identity(): 7 | values = list(range(20)) 8 | result = identity(values.copy()) 9 | assert values == result 10 | -------------------------------------------------------------------------------- /tests/unit/transformations/frequency/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Frequency Test module.""" 2 | -------------------------------------------------------------------------------- /tests/unit/transformations/frequency/test_fft.py: -------------------------------------------------------------------------------- 1 | """Tests for sigpro.transformations.frequency.fft module.""" 2 | import numpy as np 3 | 4 | from sigpro.transformations.frequency.fft import fft, fft_real 5 | 6 | 7 | def test_fft(): 8 | # setup 9 | values = [1, 1, 1, 1, 1] 10 | 11 | # run 12 | amplitude_values, frequency_values = fft(values, 10) 13 | 14 | # assert 15 | expected_amplitude_values = [5. + 0.j, 0. + 0.j, 0. + 0.j, 0. + 0.j, 0. + 0.j] 16 | expected_frequency_values = [0., 2., 4., -4., -2.] 17 | np.testing.assert_array_almost_equal(amplitude_values, expected_amplitude_values) 18 | np.testing.assert_array_almost_equal(frequency_values, expected_frequency_values) 19 | 20 | 21 | def test_fft_real(): 22 | # setup 23 | values = [1, 1, 0, 1, 1] 24 | 25 | # run 26 | amplitude_values, frequency_values = fft_real(values, 10) 27 | 28 | # assert 29 | expected_amplitude_values = [4.0, 0.80901699, -0.309017, -0.309017, 0.809017] 30 | expected_frequency_values = [0., 2., 4., -4., -2.] 31 | np.testing.assert_array_almost_equal(amplitude_values, expected_amplitude_values) 32 | np.testing.assert_array_almost_equal(frequency_values, expected_frequency_values) 33 | -------------------------------------------------------------------------------- /tests/unit/transformations/frequency/test_fftfreq.py: -------------------------------------------------------------------------------- 1 | """Tests for sigpro.transformations.frequency module.""" 2 | import numpy as np 3 | 4 | from sigpro.transformations.frequency.fftfreq import fft_freq 5 | 6 | 7 | def test_fft_freq(): 8 | # setup 9 | amplitude_values = [1.5, -0.5, 2.0, 0.5, -1.0] 10 | sampling_frequency = 2 11 | 12 | # run 13 | amplitude_result, frequency_result = fft_freq(amplitude_values, sampling_frequency) 14 | 15 | # assert 16 | expected_amplitude_values = np.array([1.5, -0.5, 2.0, 0.5, -1.0]) 17 | expected_frequency_values = np.array([0., 2., 4., 6., 8.]) 18 | np.testing.assert_array_almost_equal(amplitude_result, expected_amplitude_values) 19 | np.testing.assert_array_almost_equal(frequency_result, expected_frequency_values) 20 | -------------------------------------------------------------------------------- /tests/unit/transformations/frequency_time/__init__.py: -------------------------------------------------------------------------------- 1 | """SigPro Transformations Frequency Time Test module.""" 2 | -------------------------------------------------------------------------------- /tests/unit/transformations/frequency_time/test_stft.py: -------------------------------------------------------------------------------- 1 | """Tests for sigpro.transformations.frequency_time.stft module.""" 2 | 3 | import numpy as np 4 | 5 | from sigpro.transformations.frequency_time.stft import stft, stft_real 6 | 7 | 8 | def test_stft(): 9 | # setup 10 | values = list(range(256)) 11 | frequency = 10 12 | 13 | # run 14 | amplitude_values, frequency_values, time_values = stft(values, frequency) 15 | 16 | # assert 17 | expected_amplitude_values_len = 129 18 | expected_frequency_values_len = 129 19 | expected_time_values_len = 3 20 | value = amplitude_values[0][0] 21 | 22 | assert isinstance(value, np.complex128) 23 | assert len(amplitude_values) == expected_amplitude_values_len 24 | assert len(frequency_values) == expected_frequency_values_len 25 | assert len(time_values) == expected_time_values_len 26 | 27 | 28 | def test_stft_real(): 29 | # setup 30 | values = list(range(256)) 31 | frequency = 10 32 | 33 | # run 34 | amplitude_values, frequency_values, time_values = stft_real(values, frequency) 35 | 36 | # assert 37 | expected_amplitude_values_len = 129 38 | expected_frequency_values_len = 129 39 | expected_time_values_len = 3 40 | value = amplitude_values[0][0] 41 | 42 | assert isinstance(value, np.float64) 43 | assert len(amplitude_values) == expected_amplitude_values_len 44 | assert len(frequency_values) == expected_frequency_values_len 45 | assert len(time_values) == expected_time_values_len 46 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py3{8,9,10,11}-{lint,readme,pytest,minimum} 3 | 4 | [travis] 5 | python = 6 | 3.11: py311-lint, py311-readme, py311-pytest, py311-minimum, py311-tutorials 7 | 3.10: py310-lint, py310-readme, py310-pytest, py310-minimum, py310-tutorials 8 | 3.9: py39-lint, py39-readme, py39-pytest, py39-minimum, py39-tutorials 9 | 3.8: py38-lint, py38-readme, py38-pytest, py38-minimum, py38-tutorials 10 | 11 | [gh-actions] 12 | python = 13 | 3.11: py311-lint, py311-readme, py311-pytest, py311-minimum, py311-tutorials 14 | 3.10: py310-lint, py310-readme, py310-pytest, py310-minimum, py310-tutorials 15 | 3.9: py39-lint, py39-readme, py39-pytest, py39-minimum, py39-tutorials 16 | 3.8: py38-lint, py38-readme, py38-pytest, py38-minimum, py38-tutorials 17 | 18 | [testenv] 19 | passenv = CI TRAVIS TRAVIS_* 20 | skipsdist = false 21 | skip_install = false 22 | deps = 23 | invoke 24 | readme: rundoc 25 | tutorials: jupyter 26 | extras = 27 | lint: dev 28 | pytest: test 29 | minimum: test 30 | tutorials: ctgan 31 | commands = 32 | lint: invoke lint 33 | readme: invoke readme 34 | pytest: invoke pytest 35 | minimum: invoke minimum 36 | tutorials: invoke tutorials 37 | invoke rmdir --path {envdir} 38 | -------------------------------------------------------------------------------- /tutorials/sigpro_pipeline_demo.md: -------------------------------------------------------------------------------- 1 | # Processing Signals with Pipelines 2 | 3 | Now that we have identified and/or generated several primitives for our signal feature generation, we would like to define a reusable *pipeline* for doing so. 4 | 5 | First, let's import the required libraries and functions. 6 | 7 | 8 | 9 | ```python 10 | import sigpro 11 | import numpy as np 12 | import pandas as pd 13 | from matplotlib import pyplot as plt 14 | from sigpro.demo import _load_demo as get_demo 15 | ``` 16 | 17 | 18 | ## Defining Primitives 19 | 20 | Recall that we can obtain the list of available primitives with the `get_primitives` method: 21 | 22 | 23 | 24 | ```python 25 | from sigpro import get_primitives 26 | 27 | get_primitives() 28 | ``` 29 | 30 | 31 | 32 | 33 | ['sigpro.SigPro', 34 | 'sigpro.aggregations.amplitude.statistical.crest_factor', 35 | 'sigpro.aggregations.amplitude.statistical.kurtosis', 36 | 'sigpro.aggregations.amplitude.statistical.mean', 37 | 'sigpro.aggregations.amplitude.statistical.rms', 38 | 'sigpro.aggregations.amplitude.statistical.skew', 39 | 'sigpro.aggregations.amplitude.statistical.std', 40 | 'sigpro.aggregations.amplitude.statistical.var', 41 | 'sigpro.aggregations.frequency.band.band_mean', 42 | 'sigpro.transformations.amplitude.identity.identity', 43 | 'sigpro.transformations.amplitude.spectrum.power_spectrum', 44 | 'sigpro.transformations.frequency.band.frequency_band', 45 | 'sigpro.transformations.frequency.fft.fft', 46 | 'sigpro.transformations.frequency.fft.fft_real', 47 | 'sigpro.transformations.frequency_time.stft.stft', 48 | 'sigpro.transformations.frequency_time.stft.stft_real'] 49 | 50 | 51 | 52 | In addition, we can also define our own custom primitives. 53 | 54 | ## Building a Pipeline 55 | 56 | Let’s go ahead and define a feature processing pipeline that sequentially applies the `identity`and `fft` transformations before applying the `std` aggregation. To pass these primitives into the signal processor, we must write each primitive as a dictionary with the following fields: 57 | 58 | - `name`: Name of the transformation / aggregation. 59 | - `primitive`: Name of the primitive to apply. 60 | - `init_params`: Dictionary containing the initializing parameters for the primitive. * 61 | 62 | Since we choose not to specify any initial parameters, we do not set `init_params` in these dictionaries. 63 | 64 | 65 | ```python 66 | identity_transform = {'name': 'identity1', 67 | 'primitive': 'sigpro.transformations.amplitude.identity.identity'} 68 | 69 | fft_transform = {'name': 'fft1', 70 | 'primitive': 'sigpro.transformations.frequency.fft.fft'} 71 | 72 | std_agg = {'name': 'std1', 73 | 'primitive': "sigpro.aggregations.amplitude.statistical.std"} 74 | ``` 75 | 76 | 77 | We now define a new pipeline containing the primitives we would like to apply. At minimum, we will need to pass in a list of transformations and a list of aggregations; the full list of available arguments is given below. 78 | 79 | - Inputs: 80 | - `transformations (list)` : List of dictionaries containing the transformation primitives. 81 | - `aggregations (list)`: List of dictionaries containing the aggregation primitives. 82 | - `values_column_name (str)`(optional):The name of the column that contains the signal values. Defaults to `'values'`. 83 | - `keep_columns (Union[bool, list])` (optional): Whether to keep non-feature columns in the output DataFrame or not. If a list of column names are passed, those columns are kept. Defaults to `False`. 84 | - `input_is_dataframe (bool)` (optional): Whether the input is a pandas Dataframe. Defaults to `True`. 85 | 86 | Returning to the example: 87 | 88 | 89 | ```python 90 | transformations = [identity_transform, fft_transform] 91 | 92 | aggregations = [std_agg] 93 | 94 | mypipeline = sigpro.SigPro(transformations, aggregations, values_column_name = 'yvalues', keep_columns = True) 95 | ``` 96 | 97 | 98 | SigPro will proceed to build an `MLPipeline` that can be reused to build features. 99 | 100 | To check that `mypipeline` was defined correctly, we can check the input and output arguments with the `get_input_args` and `get_output_args` methods. 101 | 102 | 103 | ```python 104 | input_args = mypipeline.get_input_args() 105 | output_args = mypipeline.get_output_args() 106 | 107 | print(input_args) 108 | print(output_args) 109 | ``` 110 | 111 | [{'name': 'readings', 'keyword': 'data', 'type': 'pandas.DataFrame'}, {'name': 'feature_columns', 'default': None, 'type': 'list'}] 112 | [{'name': 'readings', 'type': 'pandas.DataFrame'}, {'name': 'feature_columns', 'type': 'list'}] 113 | 114 | 115 | ## Applying a Pipeline with `process_signal` 116 | 117 | Once our pipeline is correctly defined, we apply the `process_signal` method to a demo dataset. Recall that `process_signal` is defined as follows: 118 | 119 | 120 | ```python 121 | def process_signal(self, data=None, window=None, time_index=None, groupby_index=None, 122 | feature_columns=None, **kwargs): 123 | 124 | ... 125 | return data, feature_columns 126 | ``` 127 | 128 | `process_signal` accepts as input the following arguments: 129 | 130 | - `data (pd.Dataframe)` : Dataframe with a column containing signal values. 131 | - `window (str)`: Duration of window size, e.g. ('1h'). 132 | - `time_index (str)`: Name of column in `data` that represents the time index. 133 | - `groupby_index (str or list[str])`: List of column names to group together and take the window over. 134 | - `feature_columns (list)`: List of columns from the input data that should be considered as features (and not dropped). 135 | 136 | `process_signal` outputs the following: 137 | 138 | - `data (pd.Dataframe)`: Dataframe containing output feature values as constructed from the signal 139 | - `feature_columns (list)`: list of (generated) feature names. 140 | 141 | We now apply our pipeline to a toy dataset. We define our toy dataset as follows: 142 | 143 | 144 | ```python 145 | demo_dataset = get_demo() 146 | demo_dataset.columns = ['turbine_id', 'signal_id', 'xvalues', 'yvalues', 'sampling_frequency'] 147 | demo_dataset.head() 148 | ``` 149 | 150 | 151 | 152 | 153 |
154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 |
turbine_idsignal_idxvaluesyvaluessampling_frequency
0T001Sensor1_signal12020-01-01 00:00:00[0.43616983763682876, -0.17662312586241055, 0....1000
1T001Sensor1_signal12020-01-01 01:00:00[0.8023828754411122, -0.14122063493312714, -0....1000
2T001Sensor1_signal12020-01-01 02:00:00[-1.3143142430046044, -1.1055740033788437, -0....1000
3T001Sensor1_signal12020-01-01 03:00:00[-0.45981995520032104, -0.3255426061995603, -0...1000
4T001Sensor1_signal12020-01-01 04:00:00[-0.6380405111460377, -0.11924167777027689, 0....1000
209 |
210 | 211 | 212 | 213 | Finally, we apply the `process_signal` method of our previously defined pipeline: 214 | 215 | 216 | ```python 217 | processed_data, feature_columns = mypipeline.process_signal(demo_dataset, time_index = 'xvalues') 218 | 219 | processed_data.head() 220 | 221 | ``` 222 | 223 | 224 | 225 | 226 |
227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 |
turbine_idsignal_idxvaluesyvaluessampling_frequencyidentity1.fft1.std1.std_value
0T001Sensor1_signal12020-01-01 00:00:00[0.43616983763682876, -0.17662312586241055, 0....100014.444991
1T001Sensor1_signal12020-01-01 01:00:00[0.8023828754411122, -0.14122063493312714, -0....100012.326223
2T001Sensor1_signal12020-01-01 02:00:00[-1.3143142430046044, -1.1055740033788437, -0....100012.051415
3T001Sensor1_signal12020-01-01 03:00:00[-0.45981995520032104, -0.3255426061995603, -0...100010.657243
4T001Sensor1_signal12020-01-01 04:00:00[-0.6380405111460377, -0.11924167777027689, 0....100012.640728
288 |
289 | 290 | 291 | 292 | 293 | Success! We have managed to apply the primitives to generate features on the input dataset. 294 | 295 | 296 | 297 | ```python 298 | 299 | ``` 300 | --------------------------------------------------------------------------------