├── .github └── workflows │ └── rust.yml ├── .gitignore ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Cargo.toml ├── LICENSE ├── README.md ├── SECURITY.md ├── benches └── benchmarks.rs └── src ├── activations.rs ├── average_pooling.rs ├── batch_normalization.rs ├── conv1d.rs ├── dense.rs ├── dropout.rs ├── embedding.rs ├── lib.rs ├── max_pooling.rs └── padding.rs /.github/workflows/rust.yml: -------------------------------------------------------------------------------- 1 | name: Rust 2 | 3 | on: 4 | push: 5 | branches: [ "main" ] 6 | pull_request: 7 | branches: [ "main" ] 8 | 9 | env: 10 | CARGO_TERM_COLOR: always 11 | 12 | jobs: 13 | formatting: 14 | name: Cargo fmt 15 | 16 | runs-on: ubuntu-latest 17 | 18 | steps: 19 | - uses: actions/checkout@v3 20 | - uses: actions-rust-lang/setup-rust-toolchain@v1 21 | with: 22 | components: rustfmt 23 | - name: Rustfmt Check 24 | uses: actions-rust-lang/rustfmt@v1 25 | 26 | build: 27 | 28 | runs-on: ubuntu-latest 29 | 30 | steps: 31 | - uses: actions/checkout@v3 32 | - name: Build 33 | run: cargo build --verbose 34 | - name: Run tests 35 | run: cargo test --verbose 36 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | **/target 2 | **/target-linux 3 | Cargo.lock 4 | docker-build 5 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this project will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), 6 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 7 | 8 | ## [0.4.0] 2023-09-12 9 | 10 | ### Changed 11 | 12 | - Correctly update minor version 13 | 14 | ### Fixed 15 | 16 | - Comparison against `0.0` in Dropout impl 17 | 18 | ## [0.3.3] 2023-09-08 19 | 20 | ### Changed 21 | 22 | - Add change log. 23 | - Set Rust MSRV 1.60.0 24 | - Fix spelling. 25 | - Update dependencies. 26 | - Update compiler to rustc 1.69 and fix clippy warnings. 27 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # `tf-layers` Community Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | We as members, contributors, and leaders pledge to make participation in our 6 | community a harassment-free experience for everyone, regardless of age, body 7 | size, visible or invisible disability, ethnicity, sex characteristics, gender 8 | identity and expression, level of experience, education, socio-economic status, 9 | nationality, personal appearance, race, religion, or sexual identity 10 | and orientation. 11 | 12 | We pledge to act and interact in ways that contribute to an open, welcoming, 13 | diverse, inclusive, and healthy community. 14 | 15 | ## Our Standards 16 | 17 | Examples of behavior that contributes to a positive environment for our 18 | community include: 19 | 20 | * Demonstrating empathy and kindness toward other people 21 | * Being respectful of differing opinions, viewpoints, and experiences 22 | * Giving and gracefully accepting constructive feedback 23 | * Accepting responsibility and apologizing to those affected by our mistakes, 24 | and learning from the experience 25 | * Focusing on what is best not just for us as individuals, but for the 26 | overall community 27 | 28 | Examples of unacceptable behavior include: 29 | 30 | * The use of sexualized language or imagery, and sexual attention or 31 | advances of any kind 32 | * Trolling, insulting or derogatory comments, and personal or political attacks 33 | * Public or private harassment 34 | * Publishing others' private information, such as a physical or email 35 | address, without their explicit permission 36 | * Other conduct which could reasonably be considered inappropriate in a 37 | professional setting 38 | 39 | ## Enforcement Responsibilities 40 | 41 | Community leaders are responsible for clarifying and enforcing our standards of 42 | acceptable behavior and will take appropriate and fair corrective action in 43 | response to any behavior that they deem inappropriate, threatening, offensive, 44 | or harmful. 45 | 46 | Community leaders have the right and responsibility to remove, edit, or reject 47 | comments, commits, code, wiki edits, issues, and other contributions that are 48 | not aligned to this Code of Conduct, and will communicate reasons for moderation 49 | decisions when appropriate. 50 | 51 | ## Scope 52 | 53 | This Code of Conduct applies within all community spaces, and also applies when 54 | an individual is officially representing the community in public spaces. 55 | Examples of representing our community include using an official e-mail address, 56 | posting via an official social media account, or acting as an appointed 57 | representative at an online or offline event. 58 | 59 | ## Enforcement 60 | 61 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 62 | reported to the community leaders responsible for enforcement at 63 | dsci-oss@crowdstrike.com or https://crowdstrike.ethicspoint.com/. 64 | 65 | All complaints will be reviewed and investigated promptly and fairly. 66 | 67 | All community leaders are obligated to respect the privacy and security of the 68 | reporter of any incident. 69 | 70 | ## Enforcement Guidelines 71 | 72 | Community leaders will follow these Community Impact Guidelines in determining 73 | the consequences for any action they deem in violation of this Code of Conduct: 74 | 75 | ### 1. Correction 76 | 77 | **Community Impact**: Use of inappropriate language or other behavior deemed 78 | unprofessional or unwelcome in the community. 79 | 80 | **Consequence**: A private, written warning from community leaders, providing 81 | clarity around the nature of the violation and an explanation of why the 82 | behavior was inappropriate. A public apology may be requested. 83 | 84 | ### 2. Warning 85 | 86 | **Community Impact**: A violation through a single incident or series 87 | of actions. 88 | 89 | **Consequence**: A warning with consequences for continued behavior. No 90 | interaction with the people involved, including unsolicited interaction with 91 | those enforcing the Code of Conduct, for a specified period of time. This 92 | includes avoiding interactions in community spaces as well as external channels 93 | like social media. Violating these terms may lead to a temporary or 94 | permanent ban. 95 | 96 | ### 3. Temporary Ban 97 | 98 | **Community Impact**: A serious violation of community standards, including 99 | sustained inappropriate behavior. 100 | 101 | **Consequence**: A temporary ban from any sort of interaction or public 102 | communication with the community for a specified period of time. No public or 103 | private interaction with the people involved, including unsolicited interaction 104 | with those enforcing the Code of Conduct, is allowed during this period. 105 | Violating these terms may lead to a permanent ban. 106 | 107 | ### 4. Permanent Ban 108 | 109 | **Community Impact**: Demonstrating a pattern of violation of community 110 | standards, including sustained inappropriate behavior, harassment of an 111 | individual, or aggression toward or disparagement of classes of individuals. 112 | 113 | **Consequence**: A permanent ban from any sort of public interaction within 114 | the community. 115 | 116 | ## Attribution 117 | 118 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 119 | version 2.0, available at 120 | https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. 121 | 122 | Community Impact Guidelines were inspired by [Mozilla's code of conduct 123 | enforcement ladder](https://github.com/mozilla/diversity). 124 | 125 | [homepage]: https://www.contributor-covenant.org 126 | 127 | For answers to common questions about this code of conduct, see the FAQ at 128 | https://www.contributor-covenant.org/faq. Translations are available at 129 | https://www.contributor-covenant.org/translations. 130 | 131 | [![Twitter URL](https://img.shields.io/twitter/url?label=Follow%20%40CrowdStrike&style=social&url=https%3A%2F%2Ftwitter.com%2FCrowdStrike)](https://twitter.com/CrowdStrike) 132 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to this repository 2 | 3 | ## Getting started 4 | 5 | _Welcome!_ We're excited you want to take part in the `tf-layers` community! 6 | 7 | Please review this document for details regarding getting started with your first contribution, packages you'll need to install as a developer, and our Pull Request process. If you have any questions, please let us know by 8 | posting your question as an [issue](https://github.com/CrowdStrike/tf-layers/issues/new). 9 | 10 | ### Before you begin 11 | 12 | * Have you read the [Code of Conduct](CODE_OF_CONDUCT.md)? The Code of Conduct helps us establish community norms and how they'll be enforced. 13 | 14 | ### Table of Contents 15 | 16 | * [How you can contribute](#how-you-can-contribute) 17 | * [Bug reporting](#bug-reporting-and-questions-are-handled-using-githubs-issues) 18 | * [Pull Requests](#pull-requests) 19 | * [Contributor dependencies](#additional-contributor-package-requirements) 20 | * [Unit testing](#unit-testing--code-coverage) 21 | * [Linting](#linting) 22 | * [Breaking changes](#breaking-changes) 23 | * [Branch targeting](#branch-targeting) 24 | * [Suggestions](#suggestions) 25 | 26 | ## How you can contribute 27 | 28 | * See something? Say something! Submit a [bug report](https://github.com/CrowdStrike/tf-layers/issues) to let the community know what you've experienced or found. Bonus points if you suggest possible fixes or what you feel may resolve the issue. For example: "_Attempted to use the XZY API class but it errored out. Could a more descriptive error code be returned?_" 29 | * Submit a [Pull Request](#pull-requests) 30 | 31 | ### Bug reporting and questions are handled using GitHub's issues 32 | 33 | We use GitHub issues to track bugs. Report a bug by opening a [new issue](https://github.com/CrowdStrike/tf-layers/issues). 34 | 35 | ## Pull Requests 36 | 37 | ### All contributions will be submitted under the MIT license 38 | 39 | When you submit code changes, your submissions are understood to be under the same MIT [license](LICENSE) that covers the project. 40 | If this is a concern, contact the maintainers before contributing. 41 | 42 | ### Breaking changes 43 | 44 | In an effort to maintain backwards compatibility, we thoroughly unit test every Pull Request for any issues. These unit tests are intended to catch general programmatic errors, possible vulnerabilities (via bandit) and _potential breaking changes_. 45 | 46 | > If you have to adjust a unit test locally in order to produce passing results, there is a possibility you are working with a potential breaking change. 47 | 48 | Please fully document changes to unit tests within your Pull Request. If you did not specify "Breaking Change" on the punch list in the description, and the change is identified as possibly breaking, this may delay or prevent approval of your PR. 49 | 50 | ### Versioning 51 | 52 | We use [SemVer](https://semver.org/) as our versioning scheme. (Example: _2.1.4_) 53 | 54 | ### Pull Request template 55 | 56 | Please use the pull request template provided, making sure the following details are included in your request: 57 | 58 | * Is this a breaking change? 59 | * Are all new or changed code paths covered by unit testing? 60 | * A complete listing of issues addressed or closed with this change. 61 | * A complete listing of any enhancements provided by this change. 62 | * Any usage details developers may need to make use of this new functionality. 63 | * Does additional documentation need to be developed beyond what is listed in your Pull Request? 64 | * Any other salient points of interest. 65 | 66 | ### Approval / Merging 67 | 68 | All Pull Requests must be approved by at least one maintainer. Once approved, a maintainer will perform the merge and execute any backend 69 | processes related to package deployment. At this time, contributors _do not_ have the ability to merge to the `main` branch. 70 | 71 | ## Suggestions 72 | 73 | If you have suggestions on how this process could be improved, please let us know by [posting an issue](https://github.com/CrowdStrike/tf-layers/issues). 74 | -------------------------------------------------------------------------------- /Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "tensorflow-layers" 3 | version = "0.4.0" 4 | authors = ["Crowdstrike DSCI "] 5 | edition = "2021" 6 | rust = "1.60.0" 7 | description = "Pure Rust implementation of layers used in Tensorflow models" 8 | license = "MIT" 9 | include = ["Cargo.toml", "README.md", "benches/*", "src/*"] 10 | 11 | [dependencies] 12 | ndarray = { version = "0.15.5", features = ["serde-1"] } 13 | ndarray-rand = "0.14.0" 14 | num-traits = "0.2.14" 15 | serde = { version = "1.0.188", features = ["derive"] } 16 | 17 | [dev-dependencies] 18 | criterion = "0.5.1" 19 | lazy_static = "1.4.0" 20 | 21 | [[bench]] 22 | name = "benchmarks" 23 | harness = false 24 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | # MIT License 2 | 3 | Copyright (c) 2022 Crowdstrike 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tensorflow Layers 2 | 3 | This is an auxiliary crate for tf2rust which provides Rust native implementations for some of the Tensorflow models layers. 4 | 5 | ## Building 6 | 7 | ## Testing 8 | 9 | ## Getting Help 10 | 11 | `tf-layers` is an open source project, not a CrowdStrike product. As such it carries no formal support, expressed or implied. 12 | 13 | If you encounter any issues while using `tf-layers`, you can create an issue on our Github repo for bugs, enhancements, or other requests. 14 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | [![Twitter URL](https://img.shields.io/twitter/url?label=Follow%20%40CrowdStrike&style=social&url=https%3A%2F%2Ftwitter.com%2FCrowdStrike)](https://twitter.com/CrowdStrike) 2 | 3 | ![GitHub top language](https://img.shields.io/github/languages/top/crowdstrike/falconpy?logo=python&logoColor=white) 4 | ![GitHub issues](https://img.shields.io/github/issues-raw/crowdstrike/falconpy?logo=github) 5 | ![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/crowdstrike/falconpy?color=green&logo=github) 6 | 7 | # Security Policy 8 | 9 | This document outlines security policy and procedures for the CrowdStrike `tf-layers` project. 10 | 11 | * [Supported versions](#supported-versions) 12 | * [Reporting a potential security vulnerability](#reporting-a-potential-security-vulnerability) 13 | * [Disclosure and Mitigation Process](#disclosure-and-mitigation-process) 14 | 15 | ## Supported versions 16 | 17 | When discovered, we release security vulnerability patches for the most recent release at an accelerated cadence. 18 | 19 | ## Reporting a potential security vulnerability 20 | 21 | We have multiple avenues to receive security-related vulnerability reports. 22 | 23 | Please report suspected security vulnerabilities by: 24 | 25 | * Submitting a [bug](https://github.com/CrowdStrike/tf-layers/issues/new?assignees=&labels=bug+%3Abug%3A&template=bug_report.md&title=%5B+BUG+%5D+...). 26 | * Starting a new [discussion](https://github.com/CrowdStrike/tf-layers/discussions). 27 | * Submitting a [pull request](https://github.com/CrowdStrike/tf-layers/pulls) to potentially resolve the issue. (New contributors: please review the content located [here](https://github.com/CrowdStrike/tf-layers/blob/main/CONTRIBUTING.md).) 28 | * Sending an email to __dsci-oss@crowdstrike.com__. 29 | 30 | ## Disclosure and mitigation process 31 | 32 | Upon receiving a security bug report, the issue will be assigned to one of the project maintainers. This person will coordinate the related fix and release 33 | process, involving the following steps: 34 | 35 | * Communicate with you to confirm we have received the report and provide you with a status update. 36 | 37 | * You should receive this message within 48 - 72 business hours. 38 | 39 | * Confirmation of the issue and a determination of affected versions. 40 | * An audit of the codebase to find any potentially similar problems. 41 | * Preparation of patches for all releases still under maintenance. 42 | 43 | * These patches will be submitted as a separate pull request and contain a version update. 44 | * This pull request will be flagged as a security fix. 45 | * Once merged, and after post-merge unit testing has been completed, the patch will be immediately published to both PyPI repositories. 46 | 47 | ## Comments 48 | 49 | If you have suggestions on how this process could be improved, please let us know by [starting a new discussion](https://github.com/CrowdStrike/tf-layers/discussions). 50 | -------------------------------------------------------------------------------- /benches/benchmarks.rs: -------------------------------------------------------------------------------- 1 | #![allow(clippy::cast_precision_loss)] 2 | #![allow(clippy::wildcard_imports)] 3 | 4 | #[macro_use] 5 | extern crate lazy_static; 6 | 7 | use criterion::{black_box, criterion_group, criterion_main, Criterion}; 8 | use ndarray::*; 9 | use tensorflow_layers::*; 10 | 11 | lazy_static! { 12 | static ref INPUT3D: Array3 = 13 | Array::from_shape_fn((10, 25, 4), |(x, y, z)| (x + y + z) as f32); 14 | static ref INPUT3D_USIZE: Array3 = 15 | Array::from_shape_fn((10, 25, 4), |(x, y, z)| x + y + z); 16 | static ref INPUT2D: Array2 = Array::from_shape_fn((250, 4), |(x, y)| (x + y) as f32); 17 | static ref INPUT2D_USIZE: Array2 = Array::from_shape_fn((250, 4), |(x, y)| x + y); 18 | static ref INPUT1D: Array1 = Array::from_shape_fn(1000, |i| i as f32); 19 | static ref INPUT1D_USIZE: Array1 = Array::from_shape_fn(1000, |i| i); 20 | } 21 | 22 | fn bench_2d_dense_layer(c: &mut Criterion) { 23 | let weights = Array2::from_elem([4, 19], 2.7); 24 | let bias = Array1::from_elem([19], 2.7); 25 | let activation = Activation::Linear; 26 | let dense_layer = DenseLayer::new(weights, bias, activation); 27 | 28 | c.bench_function("2d_dense_layer", |b| { 29 | b.iter(|| dense_layer.apply2d(black_box(&INPUT2D))); 30 | }); 31 | } 32 | 33 | fn bench_3d_dense_layer(c: &mut Criterion) { 34 | let weights = Array2::from_elem([4, 19], 2.7); 35 | let bias = Array1::from_elem([19], 2.7); 36 | let activation = Activation::Linear; 37 | let dense_layer = DenseLayer::new(weights, bias, activation); 38 | 39 | c.bench_function("3d_dense_layer", |b| { 40 | b.iter(|| dense_layer.apply3d(black_box(&INPUT3D))); 41 | }); 42 | } 43 | 44 | fn bench_conv1d_layer(c: &mut Criterion) { 45 | let conv1d_layer = Conv1DLayer::new( 46 | Array3::from_elem([10, 3, 4], 2.7), 47 | Array1::from_elem([10], -1.5), 48 | 1, 49 | vec![(1, 2), (2, 3), (0, 0)], 50 | 0, 51 | 0, 52 | Activation::Linear, 53 | ); 54 | 55 | c.bench_function("conv1d_layer", |b| { 56 | b.iter(|| conv1d_layer.apply(black_box(&INPUT3D))); 57 | }); 58 | } 59 | 60 | fn bench_dropout_layer(c: &mut Criterion) { 61 | let dropout_layer = Dropout::new(0.2); 62 | 63 | c.bench_function("dropout_layer", |b| { 64 | b.iter(|| dropout_layer.apply(black_box(&INPUT2D))); 65 | }); 66 | } 67 | 68 | fn bench_1d_embedding_layer(c: &mut Criterion) { 69 | let weights: Array2 = Array::linspace(1., 1000., 1000) 70 | .into_shape([1000, 1]) 71 | .unwrap(); 72 | let embedding_layer = EmbeddingLayer::new(weights); 73 | 74 | c.bench_function("1d_embedding_layer", |b| { 75 | b.iter(|| embedding_layer.apply(black_box(&INPUT1D_USIZE))); 76 | }); 77 | } 78 | 79 | fn bench_2d_embedding_layer(c: &mut Criterion) { 80 | let weights: Array2 = Array::linspace(1., 1000., 1000) 81 | .into_shape([500, 2]) 82 | .unwrap(); 83 | let embedding_layer = EmbeddingLayer::new(weights); 84 | 85 | c.bench_function("2d_embedding_layer", |b| { 86 | b.iter(|| embedding_layer.apply(black_box(&INPUT2D_USIZE))); 87 | }); 88 | } 89 | 90 | fn bench_3d_embedding_layer(c: &mut Criterion) { 91 | let weights: Array2 = Array::linspace(1., 1000., 1000) 92 | .into_shape([500, 2]) 93 | .unwrap(); 94 | let embedding_layer = EmbeddingLayer::new(weights); 95 | 96 | c.bench_function("3d_embedding_layer", |b| { 97 | b.iter(|| embedding_layer.apply(black_box(&INPUT3D_USIZE))); 98 | }); 99 | } 100 | 101 | fn bench_1d_padding_layer(c: &mut Criterion) { 102 | c.bench_function("1d_padding_layer", |b| { 103 | b.iter(|| padding(black_box(&INPUT1D_USIZE), &[(3, 2)])); 104 | }); 105 | } 106 | 107 | fn bench_2d_padding_layer(c: &mut Criterion) { 108 | c.bench_function("2d_padding_layer", |b| { 109 | b.iter(|| padding(black_box(&INPUT2D_USIZE), &[(2, 3), (2, 3)])); 110 | }); 111 | } 112 | 113 | fn bench_3d_padding_layer(c: &mut Criterion) { 114 | c.bench_function("3d_padding_layer", |b| { 115 | b.iter(|| padding(black_box(&INPUT3D_USIZE), &[(2, 1), (2, 3), (0, 1)])); 116 | }); 117 | } 118 | 119 | fn bench_1d_max_pooling1d(c: &mut Criterion) { 120 | let pooling_layer = MaxPooling1DLayer::new(3, 2, vec![(3, 2)]); 121 | c.bench_function("1d_max_pooling1d", |b| { 122 | b.iter(|| pooling_layer.apply(black_box(&INPUT1D))); 123 | }); 124 | } 125 | 126 | fn bench_2d_max_pooling1d(c: &mut Criterion) { 127 | let pooling_layer = MaxPooling1DLayer::new(3, 2, vec![(2, 3), (2, 3)]); 128 | c.bench_function("2d_max_pooling1d", |b| { 129 | b.iter(|| pooling_layer.apply(black_box(&INPUT2D))); 130 | }); 131 | } 132 | 133 | fn bench_3d_max_pooling1d(c: &mut Criterion) { 134 | let pooling_layer = MaxPooling1DLayer::new(3, 2, vec![(2, 1), (2, 3), (0, 1)]); 135 | c.bench_function("3d_max_pooling1d", |b| { 136 | b.iter(|| pooling_layer.apply(black_box(&INPUT3D))); 137 | }); 138 | } 139 | 140 | fn bench_1d_avg_pooling1d(c: &mut Criterion) { 141 | let pooling_layer = AveragePooling1DLayer::new(3, 2, vec![(2, 5)]); 142 | c.bench_function("1d_avg_pooling1d", |b| { 143 | b.iter(|| pooling_layer.apply(black_box(&INPUT1D))); 144 | }); 145 | } 146 | 147 | fn bench_2d_avg_pooling1d(c: &mut Criterion) { 148 | let pooling_layer = AveragePooling1DLayer::new(3, 2, vec![(2, 3), (2, 3)]); 149 | c.bench_function("2d_avg_pooling1d", |b| { 150 | b.iter(|| pooling_layer.apply(black_box(&INPUT2D))); 151 | }); 152 | } 153 | 154 | fn bench_3d_avg_pooling1d(c: &mut Criterion) { 155 | let pooling_layer = AveragePooling1DLayer::new(3, 2, vec![(2, 1), (2, 3), (0, 1)]); 156 | c.bench_function("3d_avg_pooling1d", |b| { 157 | b.iter(|| pooling_layer.apply(black_box(&INPUT3D))); 158 | }); 159 | } 160 | 161 | fn bench_1d_batch_normalization(c: &mut Criterion) { 162 | let gamma = Array::from_shape_vec(2, vec![2.25, 2.25]).unwrap(); 163 | let epsilon = 0.0001; 164 | let batch_normalization_layer = BatchNormalization::new( 165 | gamma, 166 | INPUT1D.clone(), 167 | INPUT1D.clone(), 168 | INPUT1D.clone(), 169 | epsilon, 170 | ); 171 | 172 | c.bench_function("1d_batch_normalization", |b| { 173 | b.iter(|| batch_normalization_layer.apply(black_box(&INPUT1D))); 174 | }); 175 | } 176 | 177 | fn bench_2d_batch_normalization(c: &mut Criterion) { 178 | let gamma = Array::from_shape_vec(2, vec![2.25, 2.25]).unwrap(); 179 | let beta = Array::from_shape_vec(4, vec![3.0, 4.0, 2.5, 3.5]).unwrap(); 180 | let moving_mean = Array::from_shape_vec(4, vec![1.0, 1.5, -1.0, -1.5]).unwrap(); 181 | let moving_variance = Array::from_shape_vec(4, vec![0.5, 0.7, 0.2, 0.7]).unwrap(); 182 | let epsilon = 0.0001; 183 | 184 | let batch_normalization_layer = 185 | BatchNormalization::new(gamma, beta, moving_mean, moving_variance, epsilon); 186 | 187 | c.bench_function("2d_batch_normalization", |b| { 188 | b.iter(|| batch_normalization_layer.apply(black_box(&INPUT2D))); 189 | }); 190 | } 191 | 192 | fn bench_3d_batch_normalization(c: &mut Criterion) { 193 | let gamma = Array::from_shape_vec(2, vec![2.25, 2.25]).unwrap(); 194 | let beta = Array::from_shape_vec(4, vec![3.0, 4.0, 2.5, 3.5]).unwrap(); 195 | let moving_mean = Array::from_shape_vec(4, vec![1.0, 1.5, -1.0, -1.5]).unwrap(); 196 | let moving_variance = Array::from_shape_vec(4, vec![0.5, 0.7, 0.2, 0.7]).unwrap(); 197 | let epsilon = 0.0001; 198 | 199 | let batch_normalization_layer = 200 | BatchNormalization::new(gamma, beta, moving_mean, moving_variance, epsilon); 201 | 202 | c.bench_function("3d_batch_normalization", |b| { 203 | b.iter(|| batch_normalization_layer.apply(black_box(&INPUT3D))); 204 | }); 205 | } 206 | 207 | fn bench_1d_softmax_activation(c: &mut Criterion) { 208 | c.bench_function("1d_softmax_activation", |b| { 209 | b.iter(|| Activation::Softmax.activation(black_box(&INPUT1D))); 210 | }); 211 | } 212 | fn bench_2d_softmax_activation(c: &mut Criterion) { 213 | c.bench_function("2d_softmax_activation", |b| { 214 | b.iter(|| Activation::Softmax.activation(black_box(&INPUT2D))); 215 | }); 216 | } 217 | fn bench_3d_softmax_activation(c: &mut Criterion) { 218 | c.bench_function("3d_softmax_activation", |b| { 219 | b.iter(|| Activation::Softmax.activation(black_box(&INPUT3D))); 220 | }); 221 | } 222 | 223 | criterion_group!( 224 | benches, 225 | bench_2d_dense_layer, 226 | bench_3d_dense_layer, 227 | bench_conv1d_layer, 228 | bench_dropout_layer, 229 | bench_1d_embedding_layer, 230 | bench_2d_embedding_layer, 231 | bench_3d_embedding_layer, 232 | bench_1d_padding_layer, 233 | bench_2d_padding_layer, 234 | bench_3d_padding_layer, 235 | bench_1d_max_pooling1d, 236 | bench_2d_max_pooling1d, 237 | bench_3d_max_pooling1d, 238 | bench_1d_avg_pooling1d, 239 | bench_2d_avg_pooling1d, 240 | bench_3d_avg_pooling1d, 241 | bench_1d_batch_normalization, 242 | bench_2d_batch_normalization, 243 | bench_3d_batch_normalization, 244 | bench_1d_softmax_activation, 245 | bench_2d_softmax_activation, 246 | bench_3d_softmax_activation, 247 | ); 248 | 249 | criterion_main!(benches); 250 | -------------------------------------------------------------------------------- /src/activations.rs: -------------------------------------------------------------------------------- 1 | use ndarray::{Array, Axis, Dimension}; 2 | use serde::{Deserialize, Serialize}; 3 | use std::cmp::PartialEq; 4 | 5 | /// Activation functions supported 6 | #[derive(Serialize, Deserialize, PartialEq, Debug, Clone, Copy)] 7 | pub enum Activation { 8 | /// The linear unit activation function, `linear(x) = x` 9 | Linear, 10 | /// The rectified linear unit activation function, `relu(x) = max(x, 0)`. 11 | Relu, 12 | /// The thresholded rectified linear unit function, defined as: 13 | /// f(x) = x, for x > theta 14 | /// f(x) = 0 otherwise` 15 | ThresholdedRelu(f32), 16 | /// The scaled exponential linear unit activation, defined as: 17 | /// if x > 0: return scale * x 18 | /// if x < 0: return scale * alpha * (exp(x) - 1) 19 | Selu, 20 | /// Exponential activation function 21 | Exp, 22 | /// The hard sigmoid activation function, defined as: 23 | /// f(x) = 0, for x < -2.5 24 | /// f(x) = 1, for x > 2.5 25 | /// f(x) = 0.2*x + 0,5, otherwise 26 | HardSigmoid, 27 | /// The sigmoid activation function, `sigmoid(x) = 1 / (1 + exp(-x))`. 28 | Sigmoid, 29 | /// The softmax activation function. Softmax converts a real vector to a 30 | /// vector of categorical probabilities. The elements of the output vector 31 | /// are in range (0, 1) and sum to 1. 32 | Softmax, 33 | /// Softplus activation function, `softplus(x) = ln(exp(x) + 1)` 34 | Softplus, 35 | /// Softsign activation function, `softsign(x) = x / (abs(x) + 1)` 36 | Softsign, 37 | /// Swish activation function, `swish(x) = x * sigmoid(x)` 38 | Swish, 39 | /// Hyperbolic tangent activation function. 40 | Tanh, 41 | /// No-Op 42 | None, 43 | } 44 | 45 | /// An implementation of multiple activations. 46 | impl Activation { 47 | /// Applies the specified activation onto an array 48 | /// Returns a newly allocated array 49 | #[must_use] 50 | pub fn activation(self, data: &Array) -> Array { 51 | let mut result = data.clone(); 52 | self.activation_mut(&mut result); 53 | 54 | result 55 | } 56 | 57 | /// Applies the specified activation onto an array in place 58 | pub fn activation_mut(self, data: &mut Array) { 59 | match self { 60 | // Softmax makes use of a per-row computation process 61 | // Converts from [a,b,c] to [e^a / (e^a + e^b + e^c), e^b / (e^a + e^b + e^c), e^c / (e^a + e^b + e^c)] (in place) 62 | Self::Softmax => { 63 | let axis = data.ndim() - 1; 64 | 65 | // Since Softmax([a_1, a_2, ... , a_n]) <=> Softmax([a_1 - x, a_2 - x, ..., a_n - x]), we will choose x = max(row) 66 | // and rewrite the data in order to avoid overflow (computing exp^1000 will generate this for instance). 67 | for mut row in data.lanes_mut(Axis(axis)) { 68 | // find the maximum element from the array 69 | let maximum_elem: f32 = row.fold(f32::NEG_INFINITY, |a, b| f32::max(a, *b)); 70 | // subtract the maximum from each element of the array and compute the exponential function. 71 | row.mapv_inplace(|elem| f32::exp(elem - maximum_elem)); 72 | // get the sum all of the exponentials 73 | let sum_of_row_exponentials = row.sum(); 74 | // normalize 75 | row /= sum_of_row_exponentials; 76 | } 77 | } 78 | 79 | // The other activations use a per-element function 80 | // Convert from [a,b,c] to [activation(a), activation(b), activation(c)] (inplace) 81 | Self::Relu => data.mapv_inplace(|elem| f32::max(0.0, elem)), 82 | 83 | Self::Exp => data.mapv_inplace(f32::exp), 84 | 85 | Self::ThresholdedRelu(x) => data.mapv_inplace(|elem| f32::max(x, elem)), 86 | 87 | Self::Selu => { 88 | // constants taken from: https://www.tensorflow.org/api_docs/python/tf/keras/activations/selu 89 | #[allow(clippy::excessive_precision)] 90 | const SCALE: f32 = 1.050_700_98; 91 | #[allow(clippy::excessive_precision)] 92 | const ALPHA: f32 = 1.673_263_24; 93 | 94 | data.mapv_inplace(|elem| { 95 | if elem < 0.0 { 96 | SCALE * ALPHA * (elem.exp() - 1.0) 97 | } else { 98 | SCALE * elem 99 | } 100 | }); 101 | } 102 | 103 | Self::HardSigmoid => data.mapv_inplace(|elem| { 104 | if elem < -2.5 { 105 | 0.0 106 | } else if elem > 2.5 { 107 | 1.0 108 | } else { 109 | 0.2 * elem + 0.5 110 | } 111 | }), 112 | 113 | Self::Sigmoid => data.mapv_inplace(|elem| 1.0 / (1.0 + (-elem).exp())), 114 | 115 | Self::Softplus => data.mapv_inplace(|elem| (1.0 + elem.exp()).ln()), 116 | 117 | Self::Softsign => data.mapv_inplace(|elem| elem / (1.0 + elem.abs())), 118 | 119 | Self::Swish => data.mapv_inplace(|elem| elem * (1.0 / (1.0 + (-elem).exp()))), 120 | 121 | Self::Tanh => data.mapv_inplace(f32::tanh), 122 | 123 | Self::Linear | Self::None => {} // no-op 124 | }; 125 | } 126 | } 127 | 128 | #[cfg(test)] 129 | #[allow(clippy::excessive_precision)] 130 | #[allow(clippy::approx_constant)] 131 | #[allow(clippy::unreadable_literal)] 132 | mod tests { 133 | use super::*; 134 | use ndarray::{array, Array1, Array2, Array3}; 135 | 136 | #[test] 137 | fn test_relu_1d() { 138 | let data: Array1 = Array1::from_shape_vec([4], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 139 | 140 | let result: Array1 = Activation::Relu.activation(&data); 141 | let expected: Array1 = array![0.0, 2.0, 4.0, 0.0]; 142 | 143 | assert_eq!(expected, result); 144 | } 145 | 146 | #[test] 147 | fn test_softmax_1d() { 148 | let data: Array1 = Array1::from_shape_vec([4], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 149 | 150 | let result: Array1 = Activation::Softmax.activation(&data); 151 | let expected: Array1 = array![0.0021784895, 0.118941486, 0.87886536, 0.000014678546]; 152 | 153 | assert_eq!(expected, result); 154 | } 155 | 156 | #[test] 157 | fn test_softmax_1d_large_exponents() { 158 | let data: Array1 = 159 | Array1::from_shape_vec([4], vec![-250.0, -250.0, -250.0, 250.0]).unwrap(); 160 | 161 | let result: Array1 = Activation::Softmax.activation(&data); 162 | let expected: Array1 = array![0.0, 0.0, 0.0, 1.0]; 163 | 164 | assert_eq!(expected, result); 165 | } 166 | 167 | #[test] 168 | fn test_relu_2d() { 169 | let data: Array2 = Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 170 | 171 | let result: Array2 = Activation::Relu.activation(&data); 172 | let expected: Array2 = array![[0.0, 2.0], [4.0, 0.0]]; 173 | 174 | assert_eq!(expected, result); 175 | } 176 | 177 | #[test] 178 | fn test_selu_2d() { 179 | let data: Array2 = Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 180 | 181 | let result: Array2 = Activation::Selu.activation(&data); 182 | let expected: Array2 = array![[-1.5201665, 2.101402], [4.202804, -1.7564961]]; 183 | 184 | assert_eq!(expected, result); 185 | } 186 | 187 | #[test] 188 | fn test_hardsigmoid_1d() { 189 | let data: Array1 = Array1::from_shape_vec(5, vec![-3.0, -1.0, 0.0, 1.0, 3.0]).unwrap(); 190 | 191 | let result: Array1 = Activation::HardSigmoid.activation(&data); 192 | let expected = array![0.0, 0.3, 0.5, 0.7, 1.0]; 193 | 194 | assert_eq!(expected, result); 195 | } 196 | 197 | #[test] 198 | fn test_hardsigmoid_2d() { 199 | let data: Array2 = Array2::from_shape_vec((2, 2), vec![-3.0, -1.0, 0.0, 1.0]).unwrap(); 200 | 201 | let result: Array2 = Activation::HardSigmoid.activation(&data); 202 | let expected = array![[0.0, 0.3], [0.5, 0.7]]; 203 | 204 | assert_eq!(expected, result); 205 | } 206 | 207 | #[test] 208 | fn test_hardsigmoid_3d() { 209 | let data: Array3 = array![ 210 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 211 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 212 | ]; 213 | 214 | let result: Array3 = Activation::HardSigmoid.activation(&data); 215 | let expected = array![ 216 | [[0.099999994, 0.9, 0.61], [1.0, 0.0, 0.06999996]], 217 | [[0.0, 0.4, 1.0], [0.0, 0.5, 1.0]] 218 | ]; 219 | 220 | assert_eq!(expected, result); 221 | } 222 | 223 | #[test] 224 | fn test_sigmoid_2d() { 225 | let data: Array2 = Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 226 | 227 | let result: Array2 = Activation::Sigmoid.activation(&data); 228 | let expected: Array2 = array![[0.11920292, 0.880797], [0.98201376, 0.0009110512]]; 229 | 230 | assert_eq!(expected, result); 231 | } 232 | 233 | #[test] 234 | fn test_softmax_2d() { 235 | let data: Array2 = Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 236 | 237 | let result: Array2 = Activation::Softmax.activation(&data); 238 | let expected: Array2 = array![[0.01798621, 0.98201376], [0.9999833, 0.000016701422]]; 239 | 240 | assert_eq!(expected, result); 241 | } 242 | 243 | #[test] 244 | fn test_no_activation_2d() { 245 | let data: Array2 = Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 246 | 247 | let result: Array2 = Activation::None.activation(&data); 248 | let expected = data; 249 | 250 | assert_eq!(expected, result); 251 | } 252 | 253 | #[test] 254 | fn test_relu_3d() { 255 | let data: Array3 = array![ 256 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 257 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 258 | ]; 259 | 260 | let result: Array3 = Activation::Relu.activation(&data); 261 | let expected: Array3 = array![ 262 | [[0., 2., 0.55], [4., 0., 0.]], 263 | [[0., 0., 3.33], [0., 0., 2.75]] 264 | ]; 265 | 266 | assert_eq!(expected, result); 267 | } 268 | 269 | #[test] 270 | fn test_selu_3d() { 271 | let data: Array3 = array![ 272 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 273 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 274 | ]; 275 | 276 | let result: Array3 = Activation::Selu.activation(&data); 277 | let expected: Array3 = array![ 278 | [ 279 | [-1.5201665, 2.101402, 0.57788557], 280 | [4.202804, -1.7564961, -1.5533086] 281 | ], 282 | [ 283 | [-1.6705687, -0.69175816, 3.4988344], 284 | [-1.712859, 0., 2.889428] 285 | ] 286 | ]; 287 | 288 | assert_eq!(expected, result); 289 | } 290 | 291 | #[test] 292 | fn test_sigmoid_3d() { 293 | let data: Array3 = array![ 294 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 295 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 296 | ]; 297 | 298 | let result: Array3 = Activation::Sigmoid.activation(&data); 299 | let expected: Array3 = array![ 300 | [ 301 | [0.11920292, 0.880797, 0.6341356], 302 | [0.98201376, 0.0009110512, 0.10433122] 303 | ], 304 | [ 305 | [0.047425874, 0.37754068, 0.9654438], 306 | [0.02508696, 0.5, 0.93991333] 307 | ] 308 | ]; 309 | 310 | assert_eq!(expected, result); 311 | } 312 | 313 | #[test] 314 | fn test_softmax_3d() { 315 | let data: Array3 = array![ 316 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 317 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 318 | ]; 319 | 320 | let result: Array3 = Activation::Softmax.activation(&data); 321 | let expected: Array3 = array![ 322 | [ 323 | [0.01461876, 0.7981573, 0.18722397], 324 | [0.9978544, 0.000016665866, 0.0021289042] 325 | ], 326 | [ 327 | [0.0017411319, 0.021211328, 0.97704756], 328 | [0.0015437938, 0.05999389, 0.9384623] 329 | ] 330 | ]; 331 | 332 | assert_eq!(expected, result); 333 | } 334 | 335 | #[test] 336 | fn test_exp_1d() { 337 | let data: Array1 = Array1::from_shape_vec(5, vec![-3.0, -1.0, 0.0, 1.0, 3.0]).unwrap(); 338 | 339 | let result = Activation::Exp.activation(&data); 340 | let expected: Array1 = 341 | Array1::from_shape_vec(5, vec![0.049787067, 0.36787945, 1., 2.7182817, 20.085537]) 342 | .unwrap(); 343 | 344 | assert_eq!(expected, result); 345 | } 346 | 347 | #[test] 348 | fn test_exp_2d() { 349 | let data: Array2 = Array2::from_shape_vec((2, 2), vec![-3.0, -1.0, 0.0, 1.0]).unwrap(); 350 | 351 | let result = Activation::Exp.activation(&data); 352 | let expected = array![[0.049787067, 0.36787945], [1., 2.7182817]]; 353 | 354 | assert_eq!(expected, result); 355 | } 356 | 357 | #[test] 358 | fn test_softplus_1d() { 359 | let input: Array1 = 360 | Array1::from_shape_vec(5, vec![-20.0, -1.0, 0.0, 1.0, 20.0]).unwrap(); 361 | let expected = 362 | Array1::from_shape_vec(5, vec![0.0, 0.31326166, 0.69314718, 1.3132616, 20.0]).unwrap(); 363 | 364 | assert_eq!(Activation::Softplus.activation(&input), expected); 365 | } 366 | 367 | #[test] 368 | fn test_softplus_2d_mut() { 369 | let mut input: Array2 = 370 | Array2::from_shape_vec((2, 3), vec![-20.0, -1.0, 0.0, 1.0, 20.0, 10.0]).unwrap(); 371 | let expected = Array2::from_shape_vec( 372 | (2, 3), 373 | vec![0.0, 0.31326166, 0.69314718, 1.3132616, 20.0, 10.000046], 374 | ) 375 | .unwrap(); 376 | 377 | Activation::Softplus.activation_mut(&mut input); 378 | 379 | assert_eq!(input, expected); 380 | } 381 | 382 | #[test] 383 | fn test_softsign_1d() { 384 | let input: Array1 = Array1::from_shape_vec(3, vec![-1.0, 0.0, 1.0]).unwrap(); 385 | let expected = Array1::from_shape_vec(3, vec![-0.5, 0.0, 0.5]).unwrap(); 386 | 387 | assert_eq!(Activation::Softsign.activation(&input), expected); 388 | } 389 | 390 | #[test] 391 | fn test_softsign_2d_mut() { 392 | let mut input: Array2 = 393 | Array2::from_shape_vec((2, 2), vec![-1.0, 0.0, f32::MIN, 1.0]).unwrap(); 394 | let expected = Array2::from_shape_vec((2, 2), vec![-0.5, 0.0, -1.0, 0.5]).unwrap(); 395 | 396 | Activation::Softsign.activation_mut(&mut input); 397 | 398 | assert_eq!(input, expected); 399 | } 400 | 401 | #[test] 402 | fn test_swish_1d() { 403 | let data: Array1 = Array1::from_shape_vec(4, vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 404 | 405 | let result: Array1 = Activation::Swish.activation(&data); 406 | let expected: Array1 = array![-0.23840584, 1.761594, 3.92805516, -0.006377358]; 407 | 408 | assert_eq!(expected, result); 409 | } 410 | 411 | #[test] 412 | fn test_swish_2d() { 413 | let data: Array2 = Array2::from_shape_vec((2, 2), vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 414 | 415 | let result: Array2 = Activation::Swish.activation(&data); 416 | let expected: Array2 = array![[-0.23840584, 1.761594], [3.92805516, -0.006377358]]; 417 | 418 | assert_eq!(expected, result); 419 | } 420 | 421 | #[test] 422 | fn test_tanh_1d() { 423 | let input: Array1 = 424 | Array1::from_shape_vec(5, vec![-3.0, -1.0, 0.0, 1.0, 3.0]).unwrap(); 425 | let expected = 426 | Array1::from_shape_vec(5, vec![-0.9950548, -0.7615942, 0.0, 0.7615942, 0.9950548]) 427 | .unwrap(); 428 | 429 | assert_eq!(Activation::Tanh.activation(&input), expected); 430 | } 431 | 432 | #[test] 433 | fn test_tanh_2d_mut() { 434 | let mut input: Array2 = 435 | Array2::from_shape_vec((2, 3), vec![-3.0, -1.0, 0.0, 1.0, 3.0, f32::MIN]).unwrap(); 436 | let expected = Array2::from_shape_vec( 437 | (2, 3), 438 | vec![-0.9950548, -0.7615942, 0.0, 0.7615942, 0.9950548, -1.0], 439 | ) 440 | .unwrap(); 441 | 442 | Activation::Tanh.activation_mut(&mut input); 443 | 444 | assert_eq!(input, expected); 445 | } 446 | 447 | #[test] 448 | fn test_no_activation_3d() { 449 | let data: Array3 = array![ 450 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 451 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 452 | ]; 453 | 454 | let result: Array3 = Activation::None.activation(&data); 455 | let expected: Array3 = data; 456 | 457 | assert_eq!(expected, result); 458 | } 459 | 460 | #[test] 461 | fn test_relu_1d_mut() { 462 | let mut data: Array1 = 463 | Array1::from_shape_vec([4], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 464 | 465 | Activation::Relu.activation_mut(&mut data); 466 | let expected: Array1 = array![0.0, 2.0, 4.0, 0.0]; 467 | 468 | assert_eq!(expected, data); 469 | } 470 | 471 | #[test] 472 | fn test_softmax_1d_mut() { 473 | let mut data: Array1 = 474 | Array1::from_shape_vec([4], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 475 | 476 | Activation::Softmax.activation_mut(&mut data); 477 | let expected: Array1 = array![0.0021784895, 0.118941486, 0.87886536, 0.000014678546]; 478 | 479 | assert_eq!(expected, data); 480 | } 481 | 482 | #[test] 483 | fn test_relu_2d_mut() { 484 | let mut data: Array2 = 485 | Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 486 | 487 | Activation::Relu.activation_mut(&mut data); 488 | let expected: Array2 = array![[0.0, 2.0], [4.0, 0.0]]; 489 | 490 | assert_eq!(expected, data); 491 | } 492 | 493 | #[test] 494 | fn test_softmax_2d_mut() { 495 | let mut data: Array2 = 496 | Array2::from_shape_vec([2, 2], vec![-2.0, 2.0, 4.0, -7.0]).unwrap(); 497 | 498 | Activation::Softmax.activation_mut(&mut data); 499 | let expected: Array2 = array![[0.01798621, 0.98201376], [0.9999833, 0.000016701422]]; 500 | 501 | assert_eq!(expected, data); 502 | } 503 | 504 | #[test] 505 | fn test_relu_3d_mut() { 506 | let mut data: Array3 = array![ 507 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 508 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 509 | ]; 510 | 511 | Activation::Relu.activation_mut(&mut data); 512 | let expected: Array3 = array![ 513 | [[0., 2., 0.55], [4., 0., 0.]], 514 | [[0., 0., 3.33], [0., 0., 2.75]] 515 | ]; 516 | 517 | assert_eq!(expected, data); 518 | } 519 | 520 | #[test] 521 | fn test_softmax_3d_mut() { 522 | let mut data: Array3 = array![ 523 | [[-2.0, 2.0, 0.55], [4.0, -7.0, -2.15]], 524 | [[-3.0, -0.5, 3.33], [-3.66, 0.0, 2.75]] 525 | ]; 526 | 527 | Activation::Softmax.activation_mut(&mut data); 528 | let expected: Array3 = array![ 529 | [ 530 | [0.01461876, 0.7981573, 0.18722397], 531 | [0.9978544, 0.000016665866, 0.0021289042] 532 | ], 533 | [ 534 | [0.0017411319, 0.021211328, 0.97704756], 535 | [0.0015437938, 0.05999389, 0.9384623] 536 | ] 537 | ]; 538 | 539 | assert_eq!(expected, data); 540 | } 541 | } 542 | -------------------------------------------------------------------------------- /src/average_pooling.rs: -------------------------------------------------------------------------------- 1 | // Copyright (c) 2020-2023, CrowdStrike, Inc. All rights reserved. 2 | // Authors: marian.radu@crowdstrike.com 3 | 4 | use crate::padding::padding; 5 | use ndarray::{Array, Axis, Dimension}; 6 | use serde::{Deserialize, Serialize}; 7 | use std::cmp::min; 8 | 9 | /// Defines a 1D average pooling layer type. 10 | /// Downsamples the input representation by taking the average value 11 | /// over the window defined by `pool_size`. The window is shifted by `strides`. 12 | #[derive(Serialize, Deserialize, Debug, Clone)] 13 | pub struct AveragePooling1DLayer { 14 | /// Size of the pooling windows. 15 | pool_size: usize, 16 | /// Factor by which to downscale. E.g. 2 will halve the input. 17 | strides: usize, 18 | /// Vector of (prefix, suffix) tuples, one for every input dimension, 19 | /// used for resizing of input, by adding padding as prefix and suffix. 20 | padding: Vec<(usize, usize)>, 21 | } 22 | 23 | impl AveragePooling1DLayer { 24 | /// Returns a new [`AveragePooling1DLayer`] from predefined parameters. 25 | /// 26 | /// Will panic if `pool_size` or `strides` are 0 or if the padding is empty. 27 | /// 28 | /// # Panics 29 | /// Will panic if `strides` or `pool_size` equal 0. 30 | /// Will panic if `padding` is empty. 31 | #[must_use] 32 | pub fn new( 33 | pool_size: usize, 34 | strides: usize, 35 | padding: Vec<(usize, usize)>, 36 | ) -> AveragePooling1DLayer { 37 | assert!( 38 | strides > 0 && pool_size > 0, 39 | "Strides and pool_size should be non-zero!" 40 | ); 41 | assert!(!padding.is_empty(), "Padding vector should not be empty!"); 42 | 43 | AveragePooling1DLayer { 44 | pool_size, 45 | strides, 46 | padding, 47 | } 48 | } 49 | 50 | /// Apply average pooling on the input data. 51 | /// Note: The pooling shape is `(self.pool_size,)` for 1d arrays and `(self.pool_size, 1, 1, ...)` for Nd arrays. 52 | /// Similarly, the stride is `(self.strides,)` for 1d arrays and `(self.strides, 1, 1, ...)` for Nd arrays 53 | /// 54 | /// # Panics 55 | /// Pooling cannot be larger than the data! 56 | #[must_use] 57 | pub fn apply(&self, data: &Array) -> Array { 58 | // Data must be padded before applying the pooling layer. 59 | // padding will fail if data.ndim() != padding.len() \ 60 | let data = padding(data, &self.padding); 61 | 62 | // Compute the output shape 63 | let mut out_shape = data.raw_dim(); 64 | // the second axis will be the different one, unless this is an Array1 65 | let axis = min(1, data.ndim() - 1); 66 | // check bounds 67 | assert!( 68 | data.shape()[axis] >= self.pool_size, 69 | "Pooling size({}) cannot be larger than the data size({})!", 70 | data.shape()[axis], 71 | self.pool_size 72 | ); 73 | // the number of sliding windows of `pool_size` size, with a slide size of 74 | // `strides`, that fit into `(data.shape()[axis]` 75 | out_shape[axis] = (data.shape()[axis] - self.pool_size) / self.strides + 1; 76 | 77 | let mut result = Array::zeros(out_shape); 78 | 79 | for (mut out_lane, lane) in result 80 | .lanes_mut(Axis(axis)) 81 | .into_iter() 82 | .zip(data.lanes(Axis(axis))) 83 | { 84 | let pool_lane = lane 85 | .windows(self.pool_size) 86 | .into_iter() 87 | .step_by(self.strides); 88 | 89 | for (elem, window_matrix) in out_lane.iter_mut().zip(pool_lane) { 90 | *elem = window_matrix.mean().unwrap(); 91 | } 92 | } 93 | 94 | result 95 | } 96 | } 97 | 98 | #[cfg(test)] 99 | #[allow(clippy::cast_precision_loss)] 100 | #[allow(clippy::excessive_precision)] 101 | #[allow(clippy::unreadable_literal)] 102 | mod tests { 103 | use super::*; 104 | use ndarray::{array, Array1, Array2, Array3}; 105 | 106 | #[test] 107 | fn test_averagepooling1d() { 108 | let data: Array1 = array![4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464]; 109 | 110 | let averagepooling_layer = AveragePooling1DLayer::new(3, 2, vec![(0, 0)]); 111 | let result = averagepooling_layer.apply(&data); 112 | let expected: Array1 = array![4.9976077, 4.868549]; 113 | 114 | assert_eq!(expected, result); 115 | } 116 | 117 | #[test] 118 | fn test_averagepooling1d_identity() { 119 | let data: Array1 = array![4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464]; 120 | 121 | let averagepooling_layer = AveragePooling1DLayer::new(1, 1, vec![(0, 0)]); 122 | let result = averagepooling_layer.apply(&data); 123 | // no change should happen 124 | assert_eq!(data, result); 125 | } 126 | 127 | #[test] 128 | fn test_averagepooling2d() { 129 | let data: Array2 = array![ 130 | [4., 3., 1., 5.], 131 | [1., 3., 4., 8.], 132 | [4., 5., 4., 3.], 133 | [6., 5., 9., 4.] 134 | ]; 135 | 136 | let averagepooling_layer = AveragePooling1DLayer::new(2, 2, vec![(0, 0), (0, 0)]); 137 | let result = averagepooling_layer.apply(&data); 138 | let expected: Array2 = array![[3.5, 3.0], [2.0, 6.0], [4.5, 3.5], [5.5, 6.5]]; 139 | 140 | assert_eq!(expected, result); 141 | } 142 | 143 | #[test] 144 | fn test_averagepooling3d() { 145 | let data: Array3 = array![ 146 | [ 147 | [4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464], 148 | [7.30367663, 9.564133, 4.76055381, -0.07668671, 1.63573266], 149 | [-0.96895455, -0.38939883, 4.20417899, 2.02164234, 4.17297862], 150 | [-0.65123754, 1.9421113, 0.08885265, 7.81152724, 5.85272977] 151 | ], 152 | [ 153 | [5.97592671, -1.39181533, 5.77478317, 4.33229714, 3.36414305], 154 | [1.71159761, 3.1096064, 2.43456038, 2.94875466, 1.45737179], 155 | [4.9765289, 5.64986778, 2.21295668, 1.3772863, 4.30951371], 156 | [-0.99992831, 1.10193819, 1.15754957, 0.05423748, -1.58379326] 157 | ] 158 | ]; 159 | 160 | let averagepooling_layer = AveragePooling1DLayer::new(3, 2, vec![(0, 0), (0, 0), (0, 0)]); 161 | let result = averagepooling_layer.apply(&data); 162 | let expected: Array3 = array![ 163 | [[3.5003555, 4.2849374, 5.370378, 2.5491219, 2.5218484]], 164 | [[4.221351, 2.4558864, 3.4740999, 2.8861125, 3.0436764]] 165 | ]; 166 | 167 | assert_eq!(expected, result); 168 | } 169 | 170 | #[test] 171 | fn test_averagepooling3d_with_padding() { 172 | let data: Array3 = array![ 173 | [ 174 | [4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464], 175 | [7.30367663, 9.564133, 4.76055381, -0.07668671, 1.63573266], 176 | [-0.96895455, -0.38939883, 4.20417899, 2.02164234, 4.17297862], 177 | [-0.65123754, 1.9421113, 0.08885265, 7.81152724, 5.85272977] 178 | ], 179 | [ 180 | [5.97592671, -1.39181533, 5.77478317, 4.33229714, 3.36414305], 181 | [1.71159761, 3.1096064, 2.43456038, 2.94875466, 1.45737179], 182 | [4.9765289, 5.64986778, 2.21295668, 1.3772863, 4.30951371], 183 | [-0.99992831, 1.10193819, 1.15754957, 0.05423748, -1.58379326] 184 | ] 185 | ]; 186 | 187 | let averagepooling_layer = AveragePooling1DLayer::new(2, 2, vec![(1, 0), (0, 1), (1, 1)]); 188 | let result = averagepooling_layer.apply(&data); 189 | let expected: Array3 = array![ 190 | [ 191 | [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 192 | [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] 193 | ], 194 | [ 195 | [0.0, 5.73501, 6.6221056, 5.9534774, 2.8128617, 1.6962836, 0.0], 196 | [0.0, -0.810096, 0.7763562, 2.1465158, 4.916585, 5.012854, 0.0] 197 | ], 198 | [ 199 | [0.0, 3.8437622, 0.8588956, 4.1046715, 3.6405258, 2.4107575, 0.0], 200 | [0.0, 1.9883004, 3.3759031, 1.6852531, 0.7157619, 1.3628602, 0.0] 201 | ] 202 | ]; 203 | 204 | assert_eq!(expected, result); 205 | } 206 | 207 | #[test] 208 | #[should_panic] 209 | fn test_averagepooling_panic_higher_pool_size() { 210 | let data: Array3 = Array3::from_shape_fn([2, 3, 3], |(i, j, k)| { 211 | if i % 2 == 0 { 212 | (i + j + k) as f32 213 | } else { 214 | -((i + j + k) as f32) 215 | } 216 | }); 217 | // panics because pool_size=7 is greater than 3 (rows) + 2 + 1 (padding) = 6 218 | let averagepooling_layer = AveragePooling1DLayer::new(7, 2, vec![(0, 0), (2, 1), (0, 0)]); 219 | _ = averagepooling_layer.apply(&data); 220 | } 221 | } 222 | -------------------------------------------------------------------------------- /src/batch_normalization.rs: -------------------------------------------------------------------------------- 1 | use ndarray::{Array, Array1, Axis, Dimension, Zip}; 2 | use serde::{Deserialize, Serialize}; 3 | 4 | /// Defines a batch normalization layer 5 | #[derive(Serialize, Deserialize, Debug, Clone)] 6 | pub struct BatchNormalization { 7 | gamma: Array1, 8 | beta: Array1, 9 | moving_mean: Array1, 10 | moving_variance: Array1, 11 | epsilon: f32, 12 | } 13 | 14 | impl BatchNormalization { 15 | /// Construct a new [`BatchNormalization`] from predefined parameters. 16 | /// 17 | /// # Panics 18 | /// Will panic if `beta`, `moving_mean` and `moving_variance` don't have identical shapes. 19 | #[must_use] 20 | pub fn new( 21 | gamma: Array1, 22 | beta: Array1, 23 | moving_mean: Array1, 24 | moving_variance: Array1, 25 | epsilon: f32, 26 | ) -> BatchNormalization { 27 | assert!( 28 | gamma.shape() == beta.shape() 29 | && beta.shape() == moving_mean.shape() 30 | && moving_mean.shape() == moving_variance.shape(), 31 | "gamma, beta, moving_mean and moving_variance must all have the same shape!" 32 | ); 33 | BatchNormalization { 34 | gamma, 35 | beta, 36 | moving_mean, 37 | moving_variance, 38 | epsilon, 39 | } 40 | } 41 | 42 | /// Apply batch normalization to be used at inference time 43 | /// Returns a normalized array 44 | #[must_use] 45 | pub fn apply(&self, data: &Array) -> Array { 46 | let mut output = data.clone(); 47 | self.apply_mut(&mut output); 48 | 49 | output 50 | } 51 | 52 | /// Apply batch normalization inplace, to be used at inference time 53 | pub fn apply_mut(&self, data: &mut Array) { 54 | let axis = data.ndim() - 1; 55 | 56 | // the input data's last axis shape must match the shape of one of self.[beta, moving_average, moving_variance] 57 | assert!( 58 | data.shape()[axis] == self.beta.shape()[0], 59 | "Input data's last axis's shape must match beta/moving_mean/moving_variance" 60 | ); 61 | 62 | for mut lane in data.lanes_mut(Axis(axis)) { 63 | Zip::from(&mut lane) 64 | .and(&self.gamma) 65 | .and(&self.beta) 66 | .and(&self.moving_mean) 67 | .and(&self.moving_variance) 68 | .for_each(|elem, g, b, m, v| { 69 | *elem = (*elem - m) / (v + self.epsilon).sqrt() * g + b; 70 | }); 71 | } 72 | } 73 | } 74 | 75 | #[cfg(test)] 76 | #[allow(clippy::unreadable_literal)] 77 | mod tests { 78 | use super::*; 79 | use ndarray::{arr2, arr3, Array2, Array3}; 80 | 81 | #[test] 82 | fn test_batchnormalization_1d() { 83 | let data = Array::from_shape_vec(2, vec![1.0, 2.0]).unwrap(); 84 | 85 | let gamma = Array::from_shape_vec(2, vec![2.25, 2.25]).unwrap(); 86 | let beta = Array::from_shape_vec(2, vec![3.0, 4.0]).unwrap(); 87 | let moving_mean = Array::from_shape_vec(2, vec![1.0, 1.5]).unwrap(); 88 | let moving_variance = Array::from_shape_vec(2, vec![0.5, 0.7]).unwrap(); 89 | let epsilon = 0.0001; 90 | 91 | let batch_normalization_layer = 92 | BatchNormalization::new(gamma, beta, moving_mean, moving_variance, epsilon); 93 | 94 | let result = batch_normalization_layer.apply(&data); 95 | let expected = Array::from_shape_vec(2, vec![3.0, 5.344536]).unwrap(); 96 | 97 | assert_eq!(expected, result); 98 | } 99 | 100 | #[test] 101 | fn test_batchnormalization_2d() { 102 | let data: Array2 = arr2(&[ 103 | [1.0, 2.0], 104 | [3.0, 4.0], 105 | [5.0, 6.0], 106 | [7.0, 8.0], 107 | [9.0, 10.0], 108 | [11.0, 12.0], 109 | ]); 110 | 111 | let gamma = Array::from_shape_vec(2, vec![2.25, 2.25]).unwrap(); 112 | let beta = Array::from_shape_vec(2, vec![3.0, 4.0]).unwrap(); 113 | let moving_mean = Array::from_shape_vec(2, vec![1.0, 1.5]).unwrap(); 114 | let moving_variance = Array::from_shape_vec(2, vec![0.5, 0.7]).unwrap(); 115 | let epsilon = 0.0001; 116 | 117 | let batch_normalization_layer = 118 | BatchNormalization::new(gamma, beta, moving_mean, moving_variance, epsilon); 119 | 120 | let result = batch_normalization_layer.apply(&data); 121 | let expected: Array2 = arr2(&[ 122 | [3.0, 5.344536], 123 | [9.363325, 10.722681], 124 | [15.726649, 16.100824], 125 | [22.089973, 21.47897], 126 | [28.453299, 26.857113], 127 | [34.816624, 32.23526], 128 | ]); 129 | 130 | assert_eq!(expected, result); 131 | } 132 | 133 | #[test] 134 | fn test_batchnormalization_3d() { 135 | let data: Array3 = arr3(&[ 136 | [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], 137 | [[7.0, 8.0], [9.0, 10.0], [11.0, 12.0]], 138 | ]); 139 | 140 | let gamma = Array::from_shape_vec(2, vec![2.25, 2.25]).unwrap(); 141 | let beta = Array::from_shape_vec(2, vec![3.0, 4.0]).unwrap(); 142 | let moving_mean = Array::from_shape_vec(2, vec![1.0, 1.5]).unwrap(); 143 | let moving_variance = Array::from_shape_vec(2, vec![0.5, 0.7]).unwrap(); 144 | let epsilon = 0.0001; 145 | 146 | let batch_normalization_layer = 147 | BatchNormalization::new(gamma, beta, moving_mean, moving_variance, epsilon); 148 | 149 | let result = batch_normalization_layer.apply(&data); 150 | let expected: Array3 = arr3(&[ 151 | [ 152 | [3.0, 5.344536], 153 | [9.363325, 10.722681], 154 | [15.726649, 16.100824], 155 | ], 156 | [ 157 | [22.089973, 21.47897], 158 | [28.453299, 26.857113], 159 | [34.816624, 32.23526], 160 | ], 161 | ]); 162 | 163 | assert_eq!(expected, result); 164 | } 165 | } 166 | -------------------------------------------------------------------------------- /src/conv1d.rs: -------------------------------------------------------------------------------- 1 | use crate::activations::Activation; 2 | use crate::padding::padding; 3 | use ndarray::{Array1, Array2, Array3, Axis}; 4 | use serde::{Deserialize, Serialize}; 5 | 6 | /// Defines a one dimensional convolutional layer kernel 7 | #[derive(Serialize, Deserialize, Debug, Clone)] 8 | pub struct Conv1DLayer { 9 | weights: Array2, 10 | bias: Array1, 11 | kernel_size: usize, 12 | nb_filters: usize, 13 | no_columns: usize, 14 | strides: usize, 15 | padding: Vec<(usize, usize)>, 16 | dilation_rate: usize, 17 | groups: usize, 18 | activation: Activation, 19 | } 20 | 21 | impl Conv1DLayer { 22 | /// Returns a new [`Conv1DLayer`] from predefined parameters. 23 | /// 24 | /// # Panics 25 | /// If `weights` cannot be converted to the output shape. 26 | #[must_use] 27 | pub fn new( 28 | weights: Array3, 29 | bias: Array1, 30 | strides: usize, 31 | padding: Vec<(usize, usize)>, 32 | dilation_rate: usize, 33 | groups: usize, 34 | activation: Activation, 35 | ) -> Conv1DLayer { 36 | let nb_filters = weights.len_of(Axis(0)); 37 | let kernel_size = weights.len_of(Axis(1)); 38 | let no_columns = weights.len_of(Axis(2)); 39 | 40 | // Reformat the weights shape (3D -> 2D) such that each kernel becomes a column 41 | // in the new 2D matrix. 42 | let weights: Array2 = { 43 | // Convert 3D weights into 2D weights by flattening over Axis(0) and transpose the 44 | // result as it will be called as input.dot(weights) 45 | let output_shape = [ 46 | weights.raw_dim()[0], 47 | weights.raw_dim()[1] * weights.raw_dim()[2], 48 | ]; 49 | weights.into_shape(output_shape).unwrap().reversed_axes() 50 | }; 51 | 52 | Conv1DLayer { 53 | weights, 54 | bias, 55 | kernel_size, 56 | nb_filters, 57 | no_columns, 58 | strides, 59 | padding, 60 | dilation_rate, 61 | groups, 62 | activation, 63 | } 64 | } 65 | 66 | /// Returns a convolution of this kernel with the input data 67 | /// 68 | /// # Panics 69 | /// Input data and `Conv1DLayer`'s weights must have the same number of columns. 70 | /// 71 | /// Kernel cannot be larger than the data (this includes padding). 72 | #[must_use] 73 | pub fn apply(&self, data: &Array3) -> Array3 { 74 | // Data must be padded before applying the pooling layer. 75 | // E.g. A padding [[0,0], [1,1], [0, 0]] over a 3d array (assume the 2d matrix showed below is a sample 76 | // from the entire 3d array when iterating over Axis(0)) means: 77 | // 0 0 0 78 | // 1 1 1 1 1 1 79 | // 1 1 1 => 1 1 1 80 | // 1 1 1 1 1 1 81 | // 0 0 0 82 | 83 | // First, apply padding to the input. 84 | let data: Array3 = padding(data, &self.padding); 85 | 86 | // Assert data (after padding) and weights have the same number of columns 87 | assert_eq!( 88 | data.len_of(Axis(2)), 89 | self.no_columns, 90 | "Input data and Conv1DLayer's weights must have the same number of columns!", 91 | ); 92 | 93 | // Check bounds for the rows (Axis(1)) 94 | assert!( 95 | data.shape()[1] >= self.kernel_size, 96 | "Kernel size({}) cannot be larger than the data size({}) (this includes padding)!", 97 | data.shape()[1], 98 | self.kernel_size 99 | ); 100 | 101 | // The number of sliding windows of `kernel_size` size, with a slide size of 102 | // `strides`, that fit into `(data.shape()[axis]` 103 | let no_sliding_windows = (data.shape()[1] - self.kernel_size) / self.strides + 1; 104 | 105 | // Transform 3D data into 2D data, such that each row will be the flattened image of each 106 | // window (used in convolution) + one new element for the bias of each kernel 107 | let intermediate_shape: [usize; 2] = [ 108 | data.shape()[0] * no_sliding_windows, 109 | self.kernel_size * self.no_columns, 110 | ]; 111 | 112 | // Intermediate 1D vector to store all the flattened windows. 113 | let mut vector: Vec = 114 | Vec::with_capacity(intermediate_shape[0] * intermediate_shape[1]); 115 | 116 | // Iterate over each sample (since data comes in batch of multiple samples), 117 | // flatten their corresponding windows and add their elements to the vector. 118 | for data_matrix in data.axis_iter(Axis(0)) { 119 | // Create an iterator that gives windows of the same sizes as the kernels. 120 | // Strides represents the steps to skip. 121 | // E.g. pool_size=3, strides=2, result_matrix.shape() == [8, 10] => 122 | // [0-2, :], [2-4, :], [4-6, :] (ranges are inclusive, so 0-2 means 0,1,2) 123 | // Obs: since [6-8, :] is out of range, it will not be included. 124 | for window in data_matrix 125 | .windows([self.kernel_size, data_matrix.len_of(Axis(1))]) 126 | .into_iter() 127 | .step_by(self.strides) 128 | { 129 | // Push the last flattened window 130 | vector.extend_from_slice(window.as_slice().unwrap()); 131 | } 132 | } 133 | 134 | // Reshape the 3D input into 2D (to be used in convolution operation). 135 | let data_intermediate: Array2 = 136 | Array2::from_shape_vec(intermediate_shape, vector).unwrap(); 137 | 138 | // Output shape 139 | let mut out_shape = data.raw_dim(); 140 | // the number of sliding windows of `kernel_size` size, with a slide size of 141 | // `strides`, that fit into `(data.shape()[axis]` 142 | out_shape[1] = no_sliding_windows; 143 | out_shape[2] = self.nb_filters; 144 | 145 | // Here all the computation happens. Reshape is done to ensure the data 146 | // is reconstructed back from 2D to 3D. 147 | let mut result = data_intermediate 148 | .dot(&self.weights) 149 | .into_shape(out_shape) 150 | .unwrap(); 151 | 152 | // Apply the bias 153 | result += &self.bias; 154 | 155 | // Apply in-place activation on the result matrix. 156 | self.activation.activation_mut(&mut result); 157 | result 158 | } 159 | } 160 | 161 | #[cfg(test)] 162 | #[allow(clippy::unreadable_literal)] 163 | mod tests { 164 | use super::*; 165 | use ndarray::{arr3, Array, Array1, Array3}; 166 | 167 | #[test] 168 | fn test_conv1d_simple() { 169 | let data = Array3::from_elem([5, 11, 17], 1.0); 170 | 171 | let conv1d_layer = Conv1DLayer::new( 172 | Array3::from_elem([29, 3, 17], 1.0), 173 | Array1::from_elem([29], 1.0), 174 | 1, 175 | vec![(0, 0), (0, 0), (0, 0)], 176 | 0, 177 | 0, 178 | Activation::Linear, 179 | ); 180 | 181 | let result = conv1d_layer.apply(&data); 182 | let expected = Array3::from_elem([5, 9, 29], 52.0); 183 | 184 | assert_eq!(expected, result); 185 | } 186 | 187 | #[test] 188 | fn test_conv1d_with_padding() { 189 | let data = Array3::from_elem([1, 4, 5], 1.0); 190 | 191 | let conv1d_layer = Conv1DLayer::new( 192 | Array3::from_elem([2, 3, 5], 1.0), 193 | Array1::from_elem([2], 1.0), 194 | 1, 195 | vec![(0, 0), (2, 1), (0, 0)], 196 | 0, 197 | 0, 198 | Activation::Linear, 199 | ); 200 | 201 | let result = conv1d_layer.apply(&data); 202 | 203 | let expected: Array3 = Array3::from_shape_vec( 204 | [1, 5, 2], 205 | vec![6.0, 6.0, 11.0, 11.0, 16.0, 16.0, 16.0, 16.0, 11.0, 11.0], 206 | ) 207 | .unwrap(); 208 | 209 | assert_eq!(expected, result); 210 | } 211 | 212 | #[test] 213 | fn test_conv1d_complex() { 214 | let data: Array3 = Array::linspace(1.0, 189.0, 189) 215 | .into_shape([3, 7, 9]) 216 | .unwrap(); 217 | 218 | let weights: Array3 = Array::linspace(1.0, 360.0, 360) 219 | .into_shape([10, 4, 9]) 220 | .unwrap(); 221 | let bias: Array1 = Array::linspace(1.0, 10.0, 10).into_shape([10]).unwrap(); 222 | 223 | let conv1d_layer = Conv1DLayer::new( 224 | weights, 225 | bias, 226 | 2, 227 | vec![(0, 0), (2, 3), (0, 0)], 228 | 0, 229 | 0, 230 | Activation::Linear, 231 | ); 232 | 233 | let result = conv1d_layer.apply(&data); 234 | let expected: Array3 = arr3(&[ 235 | [ 236 | [ 237 | 5188.0, 11345.0, 17502.0, 23659.0, 29816.0, 35973.0, 42130.0, 48287.0, 54444.0, 238 | 60601.0, 239 | ], 240 | [ 241 | 16207.0, 40184.0, 64161.0, 88138.0, 112115.0, 136092.0, 160069.0, 184046.0, 242 | 208023.0, 232000.0, 243 | ], 244 | [ 245 | 28195.0, 75500.0, 122805.0, 170110.0, 217415.0, 264720.0, 312025.0, 359330.0, 246 | 406635.0, 453940.0, 247 | ], 248 | [ 249 | 20539.0, 69140.0, 117741.0, 166342.0, 214943.0, 263544.0, 312145.0, 360746.0, 250 | 409347.0, 457948.0, 251 | ], 252 | [ 253 | 2716.0, 21833.0, 40950.0, 60067.0, 79184.0, 98301.0, 117418.0, 136535.0, 254 | 155652.0, 174769.0, 255 | ], 256 | ], 257 | [ 258 | [ 259 | 36373.0, 83354.0, 130335.0, 177316.0, 224297.0, 271278.0, 318259.0, 365240.0, 260 | 412221.0, 459202.0, 261 | ], 262 | [ 263 | 58165.0, 163790.0, 269415.0, 375040.0, 480665.0, 586290.0, 691915.0, 797540.0, 264 | 903165.0, 1008790.0, 265 | ], 266 | [ 267 | 70153.0, 199106.0, 328059.0, 457012.0, 585965.0, 714918.0, 843871.0, 972824.0, 268 | 1101777.0, 1230730.0, 269 | ], 270 | [ 271 | 44353.0, 154190.0, 264027.0, 373864.0, 483701.0, 593538.0, 703375.0, 813212.0, 272 | 923049.0, 1032886.0, 273 | ], 274 | [ 275 | 5551.0, 45080.0, 84609.0, 124138.0, 163667.0, 203196.0, 242725.0, 282254.0, 276 | 321783.0, 361312.0, 277 | ], 278 | ], 279 | [ 280 | [ 281 | 67558.0, 155363.0, 243168.0, 330973.0, 418778.0, 506583.0, 594388.0, 682193.0, 282 | 769998.0, 857803.0, 283 | ], 284 | [ 285 | 100123.0, 287396.0, 474669.0, 661942.0, 849215.0, 1036488.0, 1223761.0, 286 | 1411034.0, 1598307.0, 1785580.0, 287 | ], 288 | [ 289 | 112111.0, 322712.0, 533313.0, 743914.0, 954515.0, 1165116.0, 1375717.0, 290 | 1586318.0, 1796919.0, 2007520.0, 291 | ], 292 | [ 293 | 68167.0, 239240.0, 410313.0, 581386.0, 752459.0, 923532.0, 1094605.0, 294 | 1265678.0, 1436751.0, 1607824.0, 295 | ], 296 | [ 297 | 8386.0, 68327.0, 128268.0, 188209.0, 248150.0, 308091.0, 368032.0, 427973.0, 298 | 487914.0, 547855.0, 299 | ], 300 | ], 301 | ]); 302 | 303 | assert_eq!(expected, result); 304 | } 305 | 306 | #[test] 307 | #[should_panic] 308 | fn test_conv1d_panic_wrong_bias_dimension() { 309 | let data = Array3::from_elem([17, 10, 5], 1.0); 310 | 311 | let conv1d_layer = Conv1DLayer::new( 312 | Array3::from_elem([2, 5, 5], 1.0), 313 | Array1::from_elem([3], 1.0), // instead of 3 it should be a 2 314 | 1, 315 | vec![(0, 0), (2, 1), (0, 0)], 316 | 0, 317 | 0, 318 | Activation::Linear, 319 | ); 320 | 321 | _ = conv1d_layer.apply(&data); 322 | } 323 | 324 | #[test] 325 | #[should_panic] 326 | fn test_conv1d_panic_inconsistent_shape_data_shape_weight() { 327 | let data = Array3::from_elem([17, 10, 5], 1.0); 328 | 329 | let conv1d_layer = Conv1DLayer::new( 330 | Array3::from_elem([2, 5, 8], 1.0), // instead of 8 it should be a 5 331 | Array1::from_elem([2], 1.0), 332 | 1, 333 | vec![(0, 0), (2, 1), (0, 0)], 334 | 0, 335 | 0, 336 | Activation::Linear, 337 | ); 338 | 339 | _ = conv1d_layer.apply(&data); 340 | } 341 | 342 | #[test] 343 | #[should_panic] 344 | fn test_conv1d_panic_higher_kernel() { 345 | let data = Array3::from_elem([17, 10, 5], 1.0); 346 | 347 | let conv1d_layer = Conv1DLayer::new( 348 | // the kernel size (15) must be <= than 10 (data rows per sample) + the 2 paddings (2 + 1) = 13 349 | Array3::from_elem([2, 15, 5], 1.0), 350 | Array1::from_elem([2], 1.0), 351 | 1, 352 | vec![(0, 0), (2, 1), (0, 0)], 353 | 0, 354 | 0, 355 | Activation::Linear, 356 | ); 357 | 358 | _ = conv1d_layer.apply(&data); 359 | } 360 | } 361 | -------------------------------------------------------------------------------- /src/dense.rs: -------------------------------------------------------------------------------- 1 | use crate::activations::Activation; 2 | use ndarray::{Array1, Array2, Array3, Axis}; 3 | use serde::{Deserialize, Serialize}; 4 | 5 | /// Defines a regular densely-connected Neural Network layer. 6 | /// Dense implements the operation: 7 | /// output = activation(dot(input, kernel) + bias) 8 | #[derive(Serialize, Deserialize, Debug, Clone)] 9 | pub struct DenseLayer { 10 | /// kernel weights matrix 11 | weights: Array2, 12 | /// bias vector 13 | bias: Array1, 14 | /// activation function type 15 | activation: Activation, 16 | } 17 | 18 | impl DenseLayer { 19 | /// Returns a new [`DenseLayer`] from predefined parameters. 20 | #[must_use] 21 | pub fn new(weights: Array2, bias: Array1, activation: Activation) -> DenseLayer { 22 | DenseLayer { 23 | weights, 24 | bias, 25 | activation, 26 | } 27 | } 28 | 29 | /// Returns the result of the dense layer operation on the input 2D data 30 | /// 31 | /// # Panics 32 | /// Will panic when `data` axes are not equal lengths. 33 | #[must_use] 34 | pub fn apply2d(&self, data: &Array2) -> Array2 { 35 | // Since we need to compute data * self.weights + self.bias 36 | // we must assert that the multiplication step can take place 37 | // self.weights shape: (features_in, features_out) 38 | // data shape: (batch_size, features_in) 39 | assert_eq!(self.weights.len_of(Axis(0)), data.len_of(Axis(1))); 40 | 41 | // result shape: (batch_size, features_out) 42 | let mut result: Array2 = data.dot(&self.weights); 43 | result += &self.bias; 44 | 45 | // Apply in-place activation on the result matrix. 46 | self.activation.activation_mut(&mut result); 47 | 48 | result 49 | } 50 | 51 | /// Returns the result of the dense layer operation on the input 3D data 52 | /// 53 | /// # Panics 54 | /// `weights` has to be the same shape as data. 55 | #[must_use] 56 | pub fn apply3d(&self, data: &Array3) -> Array3 { 57 | // Since we need to compute data * self.weights + self.bias 58 | // we must assert that the multiplication step can take place 59 | // self.weights shape: (features_in, features_out) 60 | // data shape: (batch_size, _, features_in) 61 | assert_eq!(self.weights.len_of(Axis(0)), data.len_of(Axis(2))); 62 | 63 | // result shape: (batch_size, _, features_out) 64 | let mut result = Array3::zeros((data.shape()[0], data.shape()[1], self.weights.shape()[1])); 65 | for (mut out2d, arr2d) in result.axis_iter_mut(Axis(0)).zip(data.axis_iter(Axis(0))) { 66 | let mut tmp2d = arr2d.dot(&self.weights); 67 | tmp2d += &self.bias; 68 | out2d.assign(&tmp2d); 69 | } 70 | 71 | // Apply in-place activation on the result matrix. 72 | self.activation.activation_mut(&mut result); 73 | 74 | result 75 | } 76 | } 77 | 78 | #[cfg(test)] 79 | mod tests { 80 | use super::*; 81 | use ndarray::{array, Array, Array1, Array2, Array3}; 82 | 83 | #[test] 84 | fn test_dense_2d_simple() { 85 | let data = Array2::from_elem([11, 7], 1.0); 86 | 87 | let weights = Array2::from_elem([7, 19], 1.0); 88 | let bias = Array1::from_elem([19], 1.0); 89 | let activation = Activation::Linear; 90 | let dense_layer = DenseLayer::new(weights, bias, activation); 91 | 92 | let result = dense_layer.apply2d(&data); 93 | let expected = Array2::from_elem([11, 19], 8.0); 94 | 95 | assert_eq!(expected, result); 96 | } 97 | 98 | #[test] 99 | fn test_dense_3d_simple() { 100 | let data = Array3::from_elem([10, 11, 7], 1.0); 101 | 102 | let weights = Array2::from_elem([7, 19], 1.0); 103 | let bias = Array1::from_elem([19], 1.0); 104 | let activation = Activation::Linear; 105 | let dense_layer = DenseLayer::new(weights, bias, activation); 106 | 107 | let result = dense_layer.apply3d(&data); 108 | let expected = Array3::from_elem([10, 11, 19], 8.0); 109 | 110 | assert_eq!(expected, result); 111 | } 112 | 113 | #[test] 114 | fn test_dense_2d_complex() { 115 | let data: Array2 = Array::linspace(1., 33., 33).into_shape([3, 11]).unwrap(); 116 | 117 | let weights: Array2 = Array::linspace(1., 88., 88).into_shape([11, 8]).unwrap(); 118 | let bias: Array1 = Array::linspace(1., 8., 8).into_shape([8]).unwrap(); 119 | let activation = Activation::Linear; 120 | let dense_layer = DenseLayer::new(weights, bias, activation); 121 | 122 | let result = dense_layer.apply2d(&data); 123 | let expected = array![ 124 | [3587.0, 3654.0, 3721.0, 3788.0, 3855.0, 3922.0, 3989.0, 4056.0], 125 | [8548.0, 8736.0, 8924.0, 9112.0, 9300.0, 9488.0, 9676.0, 9864.0], 126 | [13509.0, 13818.0, 14127.0, 14436.0, 14745.0, 15054.0, 15363.0, 15672.0] 127 | ]; 128 | 129 | assert_eq!(expected, result); 130 | } 131 | 132 | #[test] 133 | fn test_dense_3d_complex() { 134 | let data: Array3 = Array::linspace(1., 66., 66).into_shape([2, 3, 11]).unwrap(); 135 | 136 | let weights: Array2 = Array::linspace(1., 88., 88).into_shape([11, 8]).unwrap(); 137 | let bias: Array1 = Array::linspace(1., 8., 8).into_shape([8]).unwrap(); 138 | let activation = Activation::Linear; 139 | let dense_layer = DenseLayer::new(weights, bias, activation); 140 | 141 | let result = dense_layer.apply3d(&data); 142 | let expected = array![ 143 | [ 144 | [3587.0, 3654.0, 3721.0, 3788.0, 3855.0, 3922.0, 3989.0, 4056.0], 145 | [8548.0, 8736.0, 8924.0, 9112.0, 9300.0, 9488.0, 9676.0, 9864.0], 146 | [13509.0, 13818.0, 14127.0, 14436.0, 14745.0, 15054.0, 15363.0, 15672.0] 147 | ], 148 | [ 149 | [18470.0, 18900.0, 19330.0, 19760.0, 20190.0, 20620.0, 21050.0, 21480.0], 150 | [23431.0, 23982.0, 24533.0, 25084.0, 25635.0, 26186.0, 26737.0, 27288.0], 151 | [28392.0, 29064.0, 29736.0, 30408.0, 31080.0, 31752.0, 32424.0, 33096.0] 152 | ] 153 | ]; 154 | 155 | assert_eq!(expected, result); 156 | } 157 | 158 | #[test] 159 | #[should_panic] 160 | fn test_dense_2d_panic_wrong_in_features() { 161 | let data: Array2 = Array::linspace(1., 33., 33).into_shape([3, 11]).unwrap(); 162 | 163 | // instead of 10 it should be 11 (features_in) 164 | let weights: Array2 = Array::linspace(1., 80., 80).into_shape([10, 8]).unwrap(); 165 | let bias: Array1 = Array::linspace(1., 8., 8).into_shape([8]).unwrap(); 166 | let activation = Activation::Linear; 167 | let dense_layer = DenseLayer::new(weights, bias, activation); 168 | 169 | _ = dense_layer.apply2d(&data); 170 | } 171 | 172 | #[test] 173 | #[should_panic] 174 | fn test_dense_2d_panic_bias() { 175 | let data: Array2 = Array::linspace(1., 33., 33).into_shape([3, 11]).unwrap(); 176 | 177 | let weights: Array2 = Array::linspace(1., 88., 88).into_shape([11, 8]).unwrap(); 178 | // instead of 7 it should be 8 (features_out) 179 | let bias: Array1 = Array::linspace(1., 7., 7).into_shape([7]).unwrap(); 180 | let activation = Activation::Linear; 181 | let dense_layer = DenseLayer::new(weights, bias, activation); 182 | 183 | _ = dense_layer.apply2d(&data); 184 | } 185 | } 186 | -------------------------------------------------------------------------------- /src/dropout.rs: -------------------------------------------------------------------------------- 1 | use ndarray::{Array2, Zip}; 2 | use ndarray_rand::{rand, rand_distr::Bernoulli, RandomExt}; 3 | use serde::{Deserialize, Serialize}; 4 | 5 | /// The implementation of Dropout layer 6 | #[derive(Serialize, Deserialize, Debug, Clone, Copy)] 7 | pub struct Dropout { 8 | /// Between 0 an 1 9 | rate: f32, 10 | } 11 | 12 | impl Dropout { 13 | /// This creates a new Dropout layer with the rate set 14 | /// 15 | /// # Panics 16 | /// If `rate` is not in 0.0 to 1.0 range 17 | #[must_use] 18 | pub fn new(rate: f32) -> Dropout { 19 | assert!((0.0..=1.0).contains(&rate)); 20 | Dropout { rate } 21 | } 22 | 23 | /// Randomly sets the input units to 0 with a frequency of rate 24 | /// Input not set to 0 are scaled up by 1/(1-rate) such that the expected value is unchanged 25 | /// 26 | /// # Panics 27 | /// It could panic if `rate` is out of range, but the check is performed beforehand. 28 | #[must_use] 29 | pub fn apply(&self, data: &Array2) -> Array2 { 30 | if (self.rate - 1.0).abs() < 0.000_001 { 31 | return Array2::zeros(data.dim()); 32 | } 33 | 34 | if self.rate.abs() < 0.000_001 { 35 | return data.clone(); 36 | } 37 | 38 | let mut res: Array2 = data.clone(); 39 | let mut rng = rand::thread_rng(); 40 | 41 | let mask2 = Array2::random_using( 42 | data.dim(), 43 | // This unwrap is safe as the code asserts range before. 44 | Bernoulli::new(f64::from(1.0 - self.rate)).unwrap(), 45 | &mut rng, 46 | ); 47 | 48 | Zip::from(&mut res).and(&mask2).for_each(|x, mask| { 49 | if *mask { 50 | *x /= 1.0 - self.rate; 51 | } else { 52 | *x = 0_f32; 53 | } 54 | }); 55 | 56 | res 57 | } 58 | } 59 | 60 | #[cfg(test)] 61 | mod tests { 62 | use super::*; 63 | #[test] 64 | fn dropout_simple() { 65 | let dims = (5, 2); 66 | let data = 67 | Array2::from_shape_vec(dims, vec![1.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]) 68 | .unwrap(); 69 | let dropout = Dropout::new(0.2); 70 | let result = dropout.apply(&data); 71 | assert_ne!(data, result); 72 | } 73 | 74 | #[test] 75 | fn dropout_one_rate() { 76 | let dims = (5, 2); 77 | let data = 78 | Array2::from_shape_vec(dims, vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]) 79 | .unwrap(); 80 | let dropout = Dropout::new(1.0); 81 | 82 | let result = dropout.apply(&data); 83 | assert_eq!(result, Array2::zeros(dims)); 84 | } 85 | 86 | #[test] 87 | fn dropout_zero_rate() { 88 | let dims = (5, 2); 89 | let data = 90 | Array2::from_shape_vec(dims, vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]) 91 | .unwrap(); 92 | let dropout = Dropout::new(0.0); 93 | 94 | let result = dropout.apply(&data); 95 | assert_eq!(result, data); 96 | } 97 | } 98 | -------------------------------------------------------------------------------- /src/embedding.rs: -------------------------------------------------------------------------------- 1 | use ndarray::{Array, Array2, Axis, Dimension}; 2 | use serde::{Deserialize, Serialize}; 3 | 4 | /// Defines a neural network embedding layer for turning indexes 5 | /// into dense vectors of fixed size. 6 | #[derive(Serialize, Deserialize, Debug, Clone)] 7 | pub struct EmbeddingLayer { 8 | /// The embedding matrix 9 | embedding: Array2, 10 | } 11 | 12 | impl EmbeddingLayer { 13 | /// Returns a new [`EmbeddingLayer`] from an embedding matrix 14 | #[must_use] 15 | pub fn new(embedding: Array2) -> EmbeddingLayer { 16 | EmbeddingLayer { embedding } 17 | } 18 | 19 | /// Converts each element of the input data to a 1D array of `f32`. 20 | /// (the corresponding row in `self.embedding`) 21 | /// 22 | /// # Examples 23 | /// 24 | /// ```txt 25 | /// [[1,2,3], [1, 5, 5]] => [[embedding.index_axis(Axis(0), 1), embedding.index_axis(Axis(0), 2), embedding.index_axis(Axis(0), 3)], 26 | /// [embedding.index_axis(Axis(0), 1), embedding.index_axis(Axis(0), 5), embedding.index_axis(Axis(0), 5)]] 27 | /// 28 | /// [[[1],[2],[3]], [[1], [5], [5]]] => [[[embedding.index_axis(Axis(0), 1)], [embedding.index_axis(Axis(0), 2)], [embedding.index_axis(Axis(0), 3)]], 29 | /// [[embedding.index_axis(Axis(0), 1)], [embedding.index_axis(Axis(0), 5)], [embedding.index_axis(Axis(0), 5)]]] 30 | /// ``` 31 | /// 32 | /// As a direct consequence, the result matrix has a new dimension added. 33 | #[must_use] 34 | pub fn apply + Copy, D: Dimension>( 35 | &self, 36 | data: &Array, 37 | ) -> Array { 38 | // add a new axis the size of embedding's column count 39 | let mut dim = data.raw_dim().insert_axis(Axis(data.ndim())); 40 | dim[data.ndim()] = self.embedding.ncols(); 41 | 42 | let mut result: Array = Array::zeros(dim); 43 | 44 | // Loop over each row in the new `result` array zipped with the associated 45 | // `elem` from `data`, and populate the row with the corresponding embedding 46 | for (mut out, &elem) in result.lanes_mut(Axis(data.ndim())).into_iter().zip(data) { 47 | out.assign(&self.embedding.row(elem.into())); 48 | } 49 | 50 | result 51 | } 52 | } 53 | 54 | #[cfg(test)] 55 | mod tests { 56 | use super::*; 57 | use ndarray::{arr3, array, Array, Array2, Array3, Array4}; 58 | 59 | #[test] 60 | fn test_embedding_2d() { 61 | let data: Array2 = array![[1, 2, 3], [4, 5, 6], [7, 8, 9], [1, 4, 7]]; 62 | 63 | let weights: Array2 = Array::linspace(1., 1000., 1000) 64 | .into_shape([100, 10]) 65 | .unwrap(); 66 | let embedding_layer = EmbeddingLayer::new(weights); 67 | 68 | let result: Array3 = embedding_layer.apply(&data); 69 | let expected: Array3 = arr3(&[ 70 | [ 71 | [11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0], 72 | [21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0], 73 | [31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0], 74 | ], 75 | [ 76 | [41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0], 77 | [51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0, 60.0], 78 | [61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0, 70.0], 79 | ], 80 | [ 81 | [71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0], 82 | [81.0, 82.0, 83.0, 84.0, 85.0, 86.0, 87.0, 88.0, 89.0, 90.0], 83 | [91.0, 92.0, 93.0, 94.0, 95.0, 96.0, 97.0, 98.0, 99.0, 100.0], 84 | ], 85 | [ 86 | [11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0], 87 | [41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0], 88 | [71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0], 89 | ], 90 | ]); 91 | assert_eq!(expected, result); 92 | } 93 | 94 | #[test] 95 | fn test_embedding_2d_u16_input() { 96 | let data: Array2 = array![[1, 2, 3], [4, 5, 6], [7, 8, 9], [1, 4, 7]]; 97 | 98 | let weights: Array2 = Array::linspace(1., 1000., 1000) 99 | .into_shape([100, 10]) 100 | .unwrap(); 101 | let embedding_layer = EmbeddingLayer::new(weights); 102 | 103 | let result: Array3 = embedding_layer.apply(&data); 104 | let expected: Array3 = arr3(&[ 105 | [ 106 | [11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0], 107 | [21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0], 108 | [31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0], 109 | ], 110 | [ 111 | [41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0], 112 | [51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0, 60.0], 113 | [61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0, 70.0], 114 | ], 115 | [ 116 | [71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0], 117 | [81.0, 82.0, 83.0, 84.0, 85.0, 86.0, 87.0, 88.0, 89.0, 90.0], 118 | [91.0, 92.0, 93.0, 94.0, 95.0, 96.0, 97.0, 98.0, 99.0, 100.0], 119 | ], 120 | [ 121 | [11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0], 122 | [41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0], 123 | [71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0], 124 | ], 125 | ]); 126 | assert_eq!(expected, result); 127 | } 128 | 129 | #[test] 130 | fn test_embedding_3d() { 131 | let data: Array3 = array![[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [1, 4, 7]]]; 132 | 133 | let weights: Array2 = Array::linspace(1., 1000., 1000) 134 | .into_shape([100, 10]) 135 | .unwrap(); 136 | let embedding_layer = EmbeddingLayer::new(weights); 137 | 138 | let result: Array4 = embedding_layer.apply(&data); 139 | let expected: Array4 = Array4::from_shape_vec( 140 | [2, 2, 3, 10], 141 | vec![ 142 | 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 143 | 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 144 | 39.0, 40.0, 41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 52.0, 145 | 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0, 60.0, 61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 146 | 67.0, 68.0, 69.0, 70.0, 71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0, 147 | 81.0, 82.0, 83.0, 84.0, 85.0, 86.0, 87.0, 88.0, 89.0, 90.0, 91.0, 92.0, 93.0, 94.0, 148 | 95.0, 96.0, 97.0, 98.0, 99.0, 100.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 149 | 18.0, 19.0, 20.0, 41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 71.0, 150 | 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0, 151 | ], 152 | ) 153 | .unwrap(); 154 | assert_eq!(expected, result); 155 | } 156 | 157 | #[test] 158 | #[should_panic] 159 | fn test_embedding_2d_panic() { 160 | let data: Array2 = array![[1, 2, 3], [4, 5, 100], [7, 8, 9], [1, 4, 7]]; 161 | 162 | let weights: Array2 = Array::linspace(1., 1000., 1000) 163 | .into_shape([100, 10]) 164 | .unwrap(); 165 | let embedding_layer = EmbeddingLayer::new(weights); 166 | 167 | _ = embedding_layer.apply(&data); 168 | } 169 | 170 | #[test] 171 | #[should_panic] 172 | fn test_embedding_3d_panic() { 173 | let data: Array3 = array![[[1, 2, 3], [4, 5, 100]], [[7, 8, 9], [1, 4, 7]]]; 174 | 175 | let weights: Array2 = Array::linspace(1., 1000., 1000) 176 | .into_shape([100, 10]) 177 | .unwrap(); 178 | let embedding_layer = EmbeddingLayer::new(weights); 179 | 180 | _ = embedding_layer.apply(&data); 181 | } 182 | } 183 | -------------------------------------------------------------------------------- /src/lib.rs: -------------------------------------------------------------------------------- 1 | //! Crate for Neural Network Layer Operations 2 | 3 | #![deny(missing_docs)] 4 | #![warn( 5 | deprecated_in_future, 6 | elided_lifetimes_in_paths, 7 | keyword_idents, 8 | missing_copy_implementations, 9 | missing_debug_implementations, 10 | non_ascii_idents, 11 | rustdoc::private_doc_tests, 12 | single_use_lifetimes, 13 | trivial_casts, 14 | trivial_numeric_casts, 15 | unreachable_pub, 16 | unused_extern_crates 17 | )] 18 | // clippy doesn't like that all our types below are named after the modules 19 | // they're in, but those repeated names are not part of our public API (because 20 | // the modules are private and we re-export the types from here). disable this 21 | // lint. 22 | #![allow(clippy::module_name_repetitions)] 23 | 24 | mod activations; 25 | mod average_pooling; 26 | mod batch_normalization; 27 | mod conv1d; 28 | mod dense; 29 | mod dropout; 30 | mod embedding; 31 | mod max_pooling; 32 | mod padding; 33 | 34 | pub use activations::Activation; 35 | pub use average_pooling::AveragePooling1DLayer; 36 | pub use batch_normalization::BatchNormalization; 37 | pub use conv1d::Conv1DLayer; 38 | pub use dense::DenseLayer; 39 | pub use dropout::Dropout; 40 | pub use embedding::EmbeddingLayer; 41 | pub use max_pooling::MaxPooling1DLayer; 42 | pub use padding::padding; 43 | -------------------------------------------------------------------------------- /src/max_pooling.rs: -------------------------------------------------------------------------------- 1 | use crate::padding::padding; 2 | use ndarray::{Array, Axis, Dimension}; 3 | use serde::{Deserialize, Serialize}; 4 | use std::cmp::min; 5 | 6 | /// Defines a 1D maximum pooling layer type 7 | /// Downsamples the input representation by taking the maximum value 8 | /// over the window defined by `pool_size`. The window is shifted by `strides`. 9 | #[derive(Serialize, Deserialize, Debug, Clone)] 10 | pub struct MaxPooling1DLayer { 11 | /// Size of the pooling windows. 12 | pool_size: usize, 13 | /// Factor by which to downscale. E.g. 2 will halve the input. 14 | strides: usize, 15 | /// Vector of (prefix, suffix) tuples, one for every input dimension, 16 | /// used for resizing of input, by adding padding as prefix and suffix. 17 | padding: Vec<(usize, usize)>, 18 | } 19 | 20 | impl MaxPooling1DLayer { 21 | /// Returns a new [`MaxPooling1DLayer`] from predefined parameters. 22 | /// 23 | /// # Panics 24 | /// Will panic if `pool_size` or `strides` are 0 or if the padding is empty. 25 | #[must_use] 26 | pub fn new( 27 | pool_size: usize, 28 | strides: usize, 29 | padding: Vec<(usize, usize)>, 30 | ) -> MaxPooling1DLayer { 31 | assert!( 32 | strides > 0 && pool_size > 0, 33 | "Strides and pool_size should be non-zero!" 34 | ); 35 | assert!(!padding.is_empty(), "Padding vector should not be empty!"); 36 | 37 | MaxPooling1DLayer { 38 | pool_size, 39 | strides, 40 | padding, 41 | } 42 | } 43 | 44 | /// Apply max pooling on the input data. 45 | /// Note: The pooling shape is `(self.pool_size,)` for 1d arrays and `(self.pool_size, 1, 1, ...)` for Nd arrays. 46 | /// Similarly, the stride is `(self.strides,)` for 1d arrays and `(self.strides, 1, 1, ...)` for Nd arrays 47 | /// 48 | /// # Panics 49 | /// Pooling cannot be larger than the data. 50 | #[must_use] 51 | pub fn apply(&self, data: &Array) -> Array { 52 | // Data must be padded before applying the pooling layer. 53 | // padding will fail if data.ndim() != padding.len() \ 54 | let data = padding(data, &self.padding); 55 | 56 | // Compute the output shape 57 | let mut out_shape = data.raw_dim(); 58 | // the second axis will be the different one, unless this is an Array1 59 | let axis = min(1, data.ndim() - 1); 60 | // check bounds 61 | assert!( 62 | data.shape()[axis] >= self.pool_size, 63 | "Pooling size({}) cannot be larger than the data size({})!", 64 | data.shape()[axis], 65 | self.pool_size 66 | ); 67 | // the number of sliding windows of `pool_size` size, with a slide size of 68 | // `strides`, that fit into `(data.shape()[axis]` 69 | out_shape[axis] = (data.shape()[axis] - self.pool_size) / self.strides + 1; 70 | 71 | let mut result = Array::zeros(out_shape); 72 | 73 | for (mut out_lane, lane) in result 74 | .lanes_mut(Axis(axis)) 75 | .into_iter() 76 | .zip(data.lanes(Axis(axis))) 77 | { 78 | let pool_lane = lane 79 | .windows(self.pool_size) 80 | .into_iter() 81 | .step_by(self.strides); 82 | 83 | for (elem, window) in out_lane.iter_mut().zip(pool_lane) { 84 | *elem = window.fold(f32::NEG_INFINITY, |prev, &curr| f32::max(prev, curr)); 85 | } 86 | } 87 | 88 | result 89 | } 90 | } 91 | 92 | #[cfg(test)] 93 | #[allow(clippy::cast_precision_loss)] 94 | #[allow(clippy::excessive_precision)] 95 | #[allow(clippy::unreadable_literal)] 96 | mod tests { 97 | use super::*; 98 | use ndarray::{array, Array1, Array2, Array3}; 99 | 100 | #[test] 101 | fn test_maxpooling1d() { 102 | let data: Array1 = array![4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464]; 103 | 104 | let pooling_layer = MaxPooling1DLayer::new(3, 2, vec![(0, 0)]); 105 | let result = pooling_layer.apply(&data); 106 | let expected: Array1 = array![7.146401, 7.146401]; 107 | 108 | assert_eq!(expected, result); 109 | } 110 | 111 | #[test] 112 | fn test_maxpooling1d_identity() { 113 | let data: Array1 = array![4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464]; 114 | 115 | let pooling_layer = MaxPooling1DLayer::new(1, 1, vec![(0, 0)]); 116 | let result = pooling_layer.apply(&data); 117 | // no change should happen 118 | assert_eq!(data, result); 119 | } 120 | 121 | #[test] 122 | fn test_maxpooling2d() { 123 | let data: Array2 = array![ 124 | [4., 3., 1., 5.], 125 | [1., 3., 4., 8.], 126 | [4., 5., 4., 3.], 127 | [6., 5., 9., 4.] 128 | ]; 129 | 130 | let pooling_layer = MaxPooling1DLayer::new(2, 2, vec![(0, 0), (0, 0)]); 131 | let result = pooling_layer.apply(&data); 132 | let expected: Array2 = array![[4.0, 5.0], [3.0, 8.0], [5.0, 4.0], [6.0, 9.0]]; 133 | 134 | assert_eq!(expected, result); 135 | } 136 | 137 | #[test] 138 | fn test_maxpooling() { 139 | let data: Array3 = Array3::from_shape_fn([2, 5, 5], |(i, j, k)| { 140 | if i % 2 == 0 { 141 | (i + j + k) as f32 142 | } else { 143 | -((i + j + k) as f32) 144 | } 145 | }); 146 | let maxpooling_layer = MaxPooling1DLayer::new(3, 2, vec![(0, 0), (2, 3), (0, 0)]); 147 | let result = maxpooling_layer.apply(&data); 148 | let expected: Array3 = array![ 149 | [ 150 | [0.0, 1.0, 2.0, 3.0, 4.0], 151 | [2.0, 3.0, 4.0, 5.0, 6.0], 152 | [4.0, 5.0, 6.0, 7.0, 8.0], 153 | [4.0, 5.0, 6.0, 7.0, 8.0], 154 | ], 155 | [ 156 | [0.0, 0.0, 0.0, 0.0, 0.0], 157 | [-1.0, -2.0, -3.0, -4.0, -5.0], 158 | [-3.0, -4.0, -5.0, -6.0, -7.0], 159 | [0.0, 0.0, 0.0, 0.0, 0.0], 160 | ], 161 | ]; 162 | 163 | assert_eq!(expected, result); 164 | } 165 | 166 | #[test] 167 | fn test_maxpooling3d_with_padding() { 168 | let data: Array3 = array![ 169 | [ 170 | [4.16634429, 3.6800784, 7.14640084, 5.70240999, 1.75683464], 171 | [7.30367663, 9.564133, 4.76055381, -0.07668671, 1.63573266], 172 | [-0.96895455, -0.38939883, 4.20417899, 2.02164234, 4.17297862], 173 | [-0.65123754, 1.9421113, 0.08885265, 7.81152724, 5.85272977] 174 | ], 175 | [ 176 | [5.97592671, -1.39181533, 5.77478317, 4.33229714, 3.36414305], 177 | [1.71159761, 3.1096064, 2.43456038, 2.94875466, 1.45737179], 178 | [4.9765289, 5.64986778, 2.21295668, 1.3772863, 4.30951371], 179 | [-0.99992831, 1.10193819, 1.15754957, 0.05423748, -1.58379326] 180 | ] 181 | ]; 182 | 183 | let pooling_layer = MaxPooling1DLayer::new(2, 2, vec![(1, 0), (0, 1), (1, 1)]); 184 | let result = pooling_layer.apply(&data); 185 | let expected: Array3 = array![ 186 | [ 187 | [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 188 | [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] 189 | ], 190 | [ 191 | [0.0, 7.3036766, 9.564133, 7.146401, 5.70241, 1.7568346, 0.0], 192 | [ 193 | 0.0, 194 | -0.65123755, 195 | 1.9421113, 196 | 4.204179, 197 | 7.8115273, 198 | 5.85273, 199 | 0.0 200 | ] 201 | ], 202 | [ 203 | [0.0, 5.975927, 3.1096065, 5.774783, 4.3322973, 3.3641431, 0.0], 204 | [0.0, 4.976529, 5.649868, 2.2129567, 1.3772863, 4.3095136, 0.0] 205 | ] 206 | ]; 207 | 208 | assert_eq!(expected, result); 209 | } 210 | 211 | #[test] 212 | #[should_panic] 213 | fn test_maxpooling_panic_higher_pool_size() { 214 | let data: Array3 = Array3::from_shape_fn([2, 3, 3], |(i, j, k)| { 215 | if i % 2 == 0 { 216 | (i + j + k) as f32 217 | } else { 218 | -((i + j + k) as f32) 219 | } 220 | }); 221 | let maxpooling_layer = MaxPooling1DLayer::new(7, 2, [(0, 0), (2, 1), (0, 0)].to_vec()); 222 | _ = maxpooling_layer.apply(&data); 223 | } 224 | } 225 | -------------------------------------------------------------------------------- /src/padding.rs: -------------------------------------------------------------------------------- 1 | use ndarray::{Array, ArrayBase, Data, Dimension, Slice}; 2 | use num_traits::Zero; 3 | 4 | use ndarray::Axis; 5 | 6 | /// Add zero-padding before and/or after data in each dimension. 7 | /// 8 | /// The length of `pad_width` must match the number of dimensions in `data`. For 9 | /// each dimension of `data`, the corresponding entry in `pad_width` is a pair 10 | /// of numbers `[i, j]`. `i` is the amount of zero-padding to insert before the 11 | /// data on the corresponding Axis, while `j` is the padding after. 12 | /// 13 | /// E.g., a padding of `[[1,1], [0, 0]]` over a 2d array means to insert 1 14 | /// row of zeroes before and after (axis 0), and to insert 0 columns of zeroes 15 | /// before and after (axis 1): 16 | /// 17 | /// ```txt 18 | /// 0 0 0 19 | /// 1 1 1 1 1 1 20 | /// 1 1 1 => 1 1 1 21 | /// 1 1 1 1 1 1 22 | /// 0 0 0 23 | /// ``` 24 | /// 25 | /// # Panics 26 | /// Will panic if `ndim` of `data` is not the same length as `pad_with`. 27 | #[must_use] 28 | pub fn padding(data: &ArrayBase, pad_width: &[(usize, usize)]) -> Array 29 | where 30 | A: Clone + Zero, 31 | S: Data, 32 | D: Dimension, 33 | { 34 | // For each data's dimension there is a list of two usize numbers [i, j] as padding 35 | // i is the padding before the corresponding Axis, while j is the padding after. 36 | assert_eq!( 37 | data.ndim(), 38 | pad_width.len(), 39 | "Ndims of data must match the length of pad_with." 40 | ); 41 | 42 | // Compute the output shape 43 | let mut padded_shape = data.raw_dim(); 44 | for (axis, &(pad_lo, pad_hi)) in pad_width.iter().enumerate() { 45 | padded_shape[axis] += pad_lo + pad_hi; 46 | } 47 | 48 | // Create an array full of zeros with this new shape. 49 | let mut padded = Array::zeros(padded_shape); 50 | 51 | // Transfer data to the padded matrix, taking into consideration the place where this data will be inserted. 52 | let mut orig_portion = padded.view_mut(); 53 | for (axis, (&axis_size, &(pad_lo, _))) in data.shape().iter().zip(pad_width).enumerate() { 54 | orig_portion.slice_axis_inplace(Axis(axis), Slice::from(pad_lo..(pad_lo + axis_size))); 55 | } 56 | orig_portion.assign(data); 57 | 58 | // Return the new array containing the padding 59 | padded 60 | } 61 | 62 | #[cfg(test)] 63 | mod tests { 64 | use super::*; 65 | use ndarray::{Array1, Array2, Array3}; 66 | 67 | #[test] 68 | fn test_padding_1d() { 69 | let in_arr = Array1::from(vec![1, 2, 3, 4]); 70 | let expected = Array1::from(vec![0, 0, 1, 2, 3, 4, 0]); 71 | 72 | assert!(padding(&in_arr, &[(2, 1)]) == expected); 73 | } 74 | 75 | #[test] 76 | fn test_padding_2d() { 77 | let in_arr = Array2::from_shape_vec((2, 2), vec![1, 2, 3, 4]).unwrap(); 78 | let expected = Array2::from_shape_vec( 79 | (5, 7), 80 | vec![ 81 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 3, 4, 0, 0, 0, 82 | 0, 0, 0, 0, 0, 0, 0, 83 | ], 84 | ) 85 | .unwrap(); 86 | 87 | assert!(padding(&in_arr, &[(2, 1), (2, 3)]) == expected); 88 | } 89 | 90 | #[test] 91 | fn test_padding_3d() { 92 | let in_arr = Array3::from_shape_vec((2, 2, 2), vec![1, 2, 3, 4, 5, 6, 7, 8]).unwrap(); 93 | let expected = Array3::from_shape_vec( 94 | (4, 3, 5), 95 | vec![ 96 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 3, 4, 0, 0, 0, 97 | 0, 0, 0, 0, 0, 5, 6, 0, 0, 0, 7, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 98 | 0, 0, 0, 0, 99 | ], 100 | ) 101 | .unwrap(); 102 | 103 | assert!(padding(&in_arr, &[(1, 1), (0, 1), (3, 0)]) == expected); 104 | } 105 | } 106 | --------------------------------------------------------------------------------