├── .gitignore ├── .gitmodules ├── BuildInstructions.md ├── LICENSE ├── ReadMe.md ├── assets ├── dataset1.jpg ├── dataset2.jpg ├── macbuild.png ├── overview.gif ├── play.gif ├── plot.jpg ├── save.jpg ├── screenshot_1.png ├── terrain_training_cppn_s.gif ├── terrain_training_points_s.gif ├── training.jpg └── trajectory.jpg └── src ├── CMakeLists.txt ├── backend ├── CMakeLists.txt ├── backend.cpp ├── backend.h ├── parsing_utils.cpp └── parsing_utils.h ├── frontend └── maxmsp │ ├── nn.terrain.encode │ ├── CMakeLists.txt │ └── nn.terrain.encode.cpp │ ├── nn.terrain.gui │ ├── CMakeLists.txt │ ├── nn.terrain.gui.cpp │ ├── pen.h │ └── terrain.h │ ├── nn.terrain.record │ ├── CMakeLists.txt │ └── nn.terrain.record.cpp │ ├── nn.terrain_tilde │ ├── CMakeLists.txt │ └── nn.terrain_tilde.cpp │ └── shared │ ├── circular_buffer.h │ ├── min_dictionary.h │ ├── min_path.h │ └── utils.h └── package-info.json.in /.gitignore: -------------------------------------------------------------------------------- 1 | __* 2 | sysbuild 3 | *.sdf 4 | *.suo 5 | *.sln 6 | *.opensdf 7 | log.txt 8 | externals 9 | extensions 10 | support 11 | build/ 12 | build-*/ 13 | tests 14 | *.o 15 | *.dylib 16 | tmp 17 | .DS_Store 18 | .vs 19 | .vscode 20 | 21 | package-info.json 22 | 23 | pretrained_models 24 | 25 | source/min-api 26 | source/min-lib 27 | models 28 | libtorch 29 | docs 30 | out 31 | help 32 | logs 33 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "min-api-360af42"] 2 | path = src/frontend/maxmsp/min-api 3 | url = https://github.com/jasper-zheng/min-api 4 | -------------------------------------------------------------------------------- /BuildInstructions.md: -------------------------------------------------------------------------------- 1 | # Build Instructions 2 | 3 | If the externals have trouble opening in Max, or doesn't work correctly with nn_tilde, you might want to build the externals yourself: 4 | 5 | ## Prerequisites 6 | 7 | MacOS (arm64): 8 | - Xcode 11 or 12 (you can get from the App Store for free). 9 | - Download arm64 LibTorch [here](https://pytorch.org/get-started/locally/) and unzip it to a known directory. LibTorch's torch version should be the same as nn_tilde. 10 | - Install a recent version of [CMake](https://cmake.org/download/) (version 3.19 or higher). 11 | 12 | Windows: 13 | - Download LibTorch [here](https://pytorch.org/get-started/locally/) and unzip it to a known directory. LibTorch's torch version should be the same as nn_tilde. 14 | - If you would like to enable GPU training/inference, you'll need to select the CUDA version of LibTorch, and have the corresponding [CUDA ToolKit](https://developer.nvidia.com/cuda-toolkit). 15 | - Install a recent version of [CMake](https://cmake.org/download/) (version 3.19 or higher). 16 | 17 | ## Build on MacOS 18 | 19 | Recursively clone this repository into **Max's Packages folder**. Terminal command: 20 | 21 | ``` 22 | git clone https://github.com/jasper-zheng/nn_terrain.git --recursive 23 | ``` 24 | 25 | In Terminal, cd into the `nn_terrain` folder you cloned, and make a new folder named `build`. and cd into that folder: 26 | 27 | ``` 28 | cd nn_terrain 29 | mkdir build 30 | cd build 31 | ``` 32 | 33 | Run the command below to generate an Xcode project, replace `path/to/libtorch` to the libtorch folder you've downloaded: 34 | 35 | ``` 36 | cmake ../src/ -G Xcode -DCMAKE_PREFIX_PATH=/path/to/libtorch -DCMAKE_BUILD_TYPE=Release -DCMAKE_OSX_ARCHITECTURES=arm64 37 | ``` 38 | 39 | An Xcode project will be created in this `build` folder, you can either open the Xcode project and build it from there, or build by running the command below: 40 | 41 | ``` 42 | cmake --build . 43 | ``` 44 | 45 | The `.mxo` files will be created in the `src/externals` folder, move them `~/Documents/Max 9/Packages/nn_terrain/externals/` 46 | 47 | 48 | Additionally, taken from [min-devkit](https://github.com/Cycling74/min-devkit/tree/main): 49 | 50 | > If you are running on a Mac with Apple Silicon, you might see an error cannot be loaded due to system security policy when loading your externals in Max. To resolve this, you can ad-hoc codesign your external with codesign `--force --deep -s - myobject.mxo`. 51 | 52 | ## Build on Windows 53 | 54 | Recursively clone this repository into **Max's Packages folder**. Terminal command: 55 | 56 | ``` 57 | git clone https://github.com/jasper-zheng/nn_terrain.git --recurse-submodules 58 | ``` 59 | 60 | In Terminal, cd into the `nn_terrain` folder you cloned, and make a new folder named `build`. and cd into that folder: 61 | 62 | ``` 63 | cd nn_terrain 64 | mkdir build 65 | cd build 66 | ``` 67 | 68 | Then run the command below to generate a project buildsystem, replace `path\to\libtorch` to the libtorch folder you've downloaded, and make sure `Visual Studio 17 2022` is set to your build system generator (run `cmake --help` to get a list of available options). 69 | 70 | ``` 71 | cmake ..\src -A x64 -DCMAKE_PREFIX_PATH="path\to\libtorch" -G "Visual Studio 17 2022" 72 | ``` 73 | 74 | Having generated the projects, now you can build by opening the .sln file in Visual Studio, or build on the command line using: 75 | 76 | ``` 77 | cmake --build . --config Release 78 | ``` 79 | 80 | The externals will be created in the `src/externals` folder, move them `~/Documents/Max 9/Packages/nn_terrain/externals/` 81 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /ReadMe.md: -------------------------------------------------------------------------------- 1 | # Latent Terrain: Coordinates-to-Latents Generator for Neural Audio Autoencoders 2 | 3 | > Instruction Max patches can be found in the `.maxhelp` help file. 4 | > More demo/example patches will be available soon. 5 | 6 | 7 | 8 | *Latent terrain* is a coordinates-to-latents mapping model for neural audio autoencoders, can be used to build a mountainous and steep surface map for the autoencoder's latent space. A terrain produces continuous latent vectors in real-time, taking coordinates in the control space as inputs. 9 | 10 | #### Table of Contents 11 | 12 | - [What's a Neural Audio Autoencoder and why Latent Terrain?](#whats-a-neural-audio-autoencoder-and-why-latent-terrain) 13 | - [Demos](#demos) 14 | - [Compatibility & Installation](#compatibility-and-installation) 15 | - [Usage](#usage) 16 | - [Building a customised terrain](#building-a-customised-terrain) 17 | 21 | - [Visualising a terrain](#visualising-a-terrain) 22 | - [Programming trajectory playback](#programming-trajectory-playback) 23 | - [Stylus mode](#stylus-mode) 24 | - [Point-by-Point Steering](#point-by-point-steering) 25 | - [TODOs](#todos) 26 | - [Build Instructions](BuildInstructions.md) 27 | - [Acknowledgements](#acknowledgements) 28 | 29 | ## Need help! 30 | 31 | Hi, this is Shuoyang (Jasper). `nn.terrain` is part of my ongoing PhD work on **Discovering Musical Affordances in Neural Audio Synthesis** with Anna Xambó Sedó and Nick Bryan-Kinns, and part of the work has been (will be) on putting AI audio generators into the hands of composers/musicians. 32 | 33 | Therefore, I would love to have you involved in it - if you have any feedback, a features request, a demo / a device / or a ^@#*$- made with nn.terrain, I would love to hear. If you would like to collaborate on anything, please leave a message in this form: https://forms.office.com/e/EJ4WHfru1A 34 | 35 | 36 | 37 | ## What's a Neural Audio Autoencoder and why Latent Terrain? 38 | 39 | A neural audio autoencoder (such as [RAVE](https://github.com/acids-ircam/RAVE)) is an AI audio generation tool, it has two components: an encoder and a decoder. 40 | - The `encoder` compresses a piece of audio signal into a sequence of latent vectors (a **latent trajectory**). This compression happens in the time domain, so that the sampling rate goes from 44100Hz (audio sampling rate) to 21.5Hz (latent space sampling rate). 41 | ![img](assets/trajectory.jpg) 42 | 43 | - The `decoder` takes the latent trajectory to produce a piece of audio signal. The decoder can also be used as a parametric synthesiser by manually navigating the latent space (i.e., **latent space walk**). 44 | 45 | *Latent terrain* allows you to navigate latent space of the generative AI like walking on a terrain surface. It tailors the latent space to a low-dimensional (e.g., a 2D plane) control space. And this terrain surface is **nonlinear** (i.e., able to produce complex sequential patterns), **continuous** (i.e., allows for smooth interpolations), and **tailorable** (i.e., DIY your own materials with interactive machine learning). 46 | 47 | This repository is a set of Max externals to build, visualise, and program latent terrain: 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 65 | 66 | 67 | 68 | 72 | 73 | 74 | 75 | 79 | 80 | 81 | 82 | 88 | 89 | 90 | 91 |
ObjectDescription
nn.terrain~ 60 | 61 | - Load, build, train, and save a terrain. 62 | 63 | - Perform the coordinates-to-latents mapping. 64 |
nn.terrain.encode 69 | 70 | - Encode audio buffers into latent trajectories using a pre-trained audio autoencoder, to be used as training data for `nn.terrain~`. 71 |
nn.terrain.record 76 | 77 | - Record latent trajectories to be used as training data for `nn.terrain~`. 78 |
nn.terrain.gui 83 | 84 | - Edit spatial trajectories to be used as training data for `nn.terrain~`. 85 | - Visualise the terrain. 86 | - Create and program trajectory playbacks. 87 |
92 | 93 | ## Demos 94 | 95 | The projection from a latent space to a latent terrain is done by pairing latent trajectories and spatial trajectories on a 2D plane (or any low-dimensional space). After providing examples of inputs (spatial trajectories) and their corresponding outputs (latent trajectories), the terrain can be trained very quickly (~15s) using supervised machine learning. 96 | 97 | https://github.com/user-attachments/assets/17a306d2-791a-4322-9ec6-aa788713cbac 98 | 99 | Sound synthesising with latent terrain is similar to wave terrain synthesis, operating in the latent space of an audio autoencoder. An audio fragment can be synthesised by pathing through the terrain surface. 100 | 101 | https://github.com/user-attachments/assets/2da6380f-272a-40ff-a595-07f7d4008e3b 102 | 103 | 108 | 109 | 114 | 115 | 116 | 117 | A presentation at the IRCAM Forum Workshops 2025 can be found in [this article](https://forum.ircam.fr/article/detail/latent-terrain-dissecting-the-latent-space-of-neural-audio-autoencoder-by-shuoyang-jasper-zheng/). 118 | 119 | 120 | 122 | 123 | 124 | 125 | ## Compatibility and Installation 126 | 127 | This external works with [nn_tilde v1.5.6 (torch v2.0.0/2.0.1)](https://github.com/acids-ircam/nn_tilde/releases/tag/v1.5.6). If you have a `nn~` built from another torch version, you might have to build this yourself. See the [Build Instructions](BuildInstructions.md) documentation. 128 | 129 | We only have MaxMSP version at the moment, sorry. Windows and arm64 macOS supported. 130 | 131 | ### macOS 132 | 133 | Uncompress the `.tar.gz` file in the `Package` folder of your Max installation, which is usually in `~/Documents/Max 9/Packages/`. 134 | 135 | Reopen Max and you can find all nn.terrain objects. You might get a quarantine warning, proceed will disable this warning. If they still have trouble opening, or doesn't work correctly with nn_tilde, you might want to build the externals yourself, see [Build Instructions](BuildInstructions.md). 136 | 137 | ### Windows 138 | 139 | Uncompress the `.tar.gz` file in the `Package` folder of your Max installation, which is usually in `~/Documents/Max 9/Packages/`. 140 | 141 | Copy all `.dll` files in the package next to the ˋMax.exeˋ executable (if you have already done this for nn_tilde, you don't need to do this again). 142 | 143 | ## Usage 144 | Here we briefly walk through the features/functionalities, while detailed walkthroughs can be found in the `.maxhelp` help file for each object. 145 | 146 | 147 | ### Building a customised terrain 148 | 149 | A terrain is built by pairing latent trajectories and coordinate trajectories. And then using a supervised machine learning algorithm to learn this pairing. Here are the steps: 150 | 151 | #### Terrain parameters 152 | 153 | First, we'll define the dimensionality of the latent space and control space. This can be set by the first two arguments of the object. For instance, `nn.terrain~ 2 8` will create a terrain that takes 2 input signals and produces a 8-dimensional latent vector. 154 | 155 | Arguments of `nn.terrain~`: 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 |
ObjectArgumentsDescription
nn.terrain~control_dimNumber of input channels
latent_dimNumber of output channels, this should usually be the dimensionality of the autoencoder's latent space.
gauss_scale 178 | (optional) 179 | 180 | - Gaussian scale of the Fourier feature mapping. A higher Gaussian scale leads to a noisy terrain, lower leads to smoother less-mountainous terrain. 181 | - A float value between 0.05 - 0.3 is suggested. 182 | - If the Gaussian scale is 0, the Fourier feature mapping layer will be removed, resulting in a very smooth (low-frequency) terrain, useful for point-by-point mode.
network_channel(optional) The number of neurons in each hidden layer. Defined by the network_channel argument. By default 128
feature_size(optional) The size of the random Fourier feature mapping, a higher feature size is suggested when using a high-dimensional control space. By default 256
buffer_size(optional) The inference of the terrain will happen once per buffer size, keeping it the same with the buffer size of nn~ is suggested.
199 | 200 | 201 | 202 | 203 | 204 | #### Training examples preparation 205 | 206 | - Gathering **latent trajectories**: 207 | - Create a `nn.terrain.encode` and put audio sample(s) in `buffer~` or `polybuffer~` 208 | - Use `append` message to add buffers to the encoder, message `encode` to convert them to latent trajectory in a dictionary file. 209 | - Additionally, see `nn.terrain.encode`'s help file for how to encode samples from a `playlist~` object. 210 | ![img](assets/dataset1.jpg) 211 | 212 | - Gathering **spatial trajectories**: 213 | - Create a `nn.terrain.gui` and set the `UI Task (task)` attribute to `Dataset` 214 | - Define the target length of trajectories (in ms) using a list message or an `append` message. 215 | - Draw lines as trajectories. 216 | ![img](assets/dataset2.jpg) 217 | 218 | #### Training 219 | - Terrain training can be done within the `nn.terrain~` object: 220 | - Send the data dictionaries we got in previous step to `nn.terrain~` 221 | - The terrain will be trained for 10 epochs once a `train` message is received (this number can be changed by the `training_epoch` attribute). 222 | - `route` the last outlet to `loss` and `epoch` to inspect the training loss. 223 | ![img](assets/training.jpg) 224 | - 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 |
ObjectTraining AttributesDescription
nn.terrain~training_batchsizeBatch size used to train the neural network in the terrain (i.e., how many coordinate-latent pairs). Just a matter of memory consumption, won't affect the result too much.
training_epochHow many epochs will be trained once a train message is received.
training_lrLearning rate used when training.
training_workerThe same effect as PyTorch dataloader's num_workers, won't affect the result too much.
253 | - After you feel good about the training loss, the terrain is good to go. 254 | 255 | #### Saving (Checkpoints) 256 | 257 | - Use the `checkpoint` message to save the tarrain to a `.pt` file. Saving name and path can be set in attributes. 258 | - Load a terrain `.pt` file by giving its file name as an argument. 259 | ![img](assets/save.jpg) 260 | 261 | ### Visualising a terrain 262 | 263 | Since the control space is 2D, the latent space can be visualised by sampling the control space across a closed interval (i.e., width and height in this example). Use the `plot_interval` message to do this: 264 | - `plot_interval` for 2D plane takes 6 arguments: 265 | - lower and upper bound values of the x and y axes in the control space (these usually are the same as the `values_bound` attribute of `nn.terrain.gui`) 266 | - resolution of the x and y axes (these usually are the same as the width and height of `nn.terrain.gui`) 267 | ![img](assets/plot.jpg) 268 | ### Programming trajectory playback 269 | 270 | You can create trajectories to navigate the terrain. This trajectory playback can be controled be a signal input. 271 | - Set the 'UI Tasks' (`task`) attribute of nn.terrain.gui to 'play'. 272 | - This behaviour is similar to the `play~` object in Max. 273 | - 274 | 275 | 276 | ### Stylus mode 277 | 278 | Set the 'UI Tasks' (`task`) attribute of nn.terrain.gui to 'stylus' to use it as a trackpad. If you are using a tablet/stylus, it also supports the pen pressure. 279 | 280 | 281 | https://github.com/user-attachments/assets/2dd7edea-583d-410b-8b09-7aa1eec09bfa 282 | 283 | ### Point-by-Point Steering 284 | 285 | [todo] It also supports the [point-by-point steering approach](https://vigliensoni.com/portfolio/42-vigliensoni23steering/) proposed by Vigliensoni and Fiebrink (2023). 286 | 287 | ## TODOs 288 | 289 | - [✕︎] Load and inference scripted mapping model exported bt torchscript. 290 | - [✔︎] Display terrain visualisation. 291 | - [✔︎] Greyscale (one-channel) 292 | - [✔︎] Multi-channel (yes but no documentation atm) 293 | - [✔︎] Interactive training of terrain models in Max MSP. 294 | - [✔︎] Customised configuration of Fourier-CPPNs (Tancik et al., 2020). 295 | - [✕︎] Example patches, tutorials... 296 | - [✕︎] PureData 297 | 298 | ## Build Instructions 299 | 300 | If the externals have trouble opening in Max, or doesn't work correctly with nn_tilde, you might want to build the externals yourself, see the [Build Instructions](BuildInstructions.md) documentation. 301 | 302 | ## Acknowledgements 303 | 304 | - Shuoyang Zheng, the author of this work, is supported by UK Research and Innovation [EP/S022694/1]. 305 | 306 | - This is built on top of acids-ircam's [nn_tilde](https://github.com/acids-ircam/nn_tilde), with a lot of reused code including the cmakelists templates, `backend.cpp`, `circular_buffer.h`, and the model performing loop in `nn.terrain_tilde.cpp`. 307 | - Caillon, A., Esling, P., 2022. Streamable Neural Audio Synthesis With Non-Causal Convolutions. https://doi.org/10.48550/arXiv.2204.07064 308 | - Tancik, M., Srinivasan, P.P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J.T., Ng, R., 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS. 309 | - Vigliensoni, G., Fiebrink, R., 2023. Steering latent audio models through interactive machine learning, in: In Proceedings of the 14th International Conference on Computational Creativity. 310 | 311 | -------------------------------------------------------------------------------- /assets/dataset1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/dataset1.jpg -------------------------------------------------------------------------------- /assets/dataset2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/dataset2.jpg -------------------------------------------------------------------------------- /assets/macbuild.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/macbuild.png -------------------------------------------------------------------------------- /assets/overview.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/overview.gif -------------------------------------------------------------------------------- /assets/play.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/play.gif -------------------------------------------------------------------------------- /assets/plot.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/plot.jpg -------------------------------------------------------------------------------- /assets/save.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/save.jpg -------------------------------------------------------------------------------- /assets/screenshot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/screenshot_1.png -------------------------------------------------------------------------------- /assets/terrain_training_cppn_s.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/terrain_training_cppn_s.gif -------------------------------------------------------------------------------- /assets/terrain_training_points_s.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/terrain_training_points_s.gif -------------------------------------------------------------------------------- /assets/training.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/training.jpg -------------------------------------------------------------------------------- /assets/trajectory.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasper-zheng/nn_terrain/0e39608730b21de879b0cd4920e3babc20c818ea/assets/trajectory.jpg -------------------------------------------------------------------------------- /src/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.0 FATAL_ERROR) 2 | if (APPLE) 3 | set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0") 4 | endif() 5 | 6 | project(nn_terrain) 7 | 8 | set(CMAKE_POSITION_INDEPENDENT_CODE ON) 9 | 10 | if (APPLE) 11 | set(CMAKE_CXX_FLAGS "-faligned-allocation") 12 | if (CMAKE_OSX_ARCHITECTURES STREQUAL "") 13 | set(CMAKE_OSX_ARCHITECTURES ${CMAKE_HOST_SYSTEM_PROCESSOR}) 14 | endif() 15 | message("CMAKE_OSX_ARCHITECTURES: ${CMAKE_OSX_ARCHITECTURES}") 16 | endif() 17 | 18 | add_subdirectory(backend) # DEEP LEARNING BACKEND 19 | 20 | # if ((NOT ("${PUREDATA_INCLUDE_DIR}" STREQUAL "")) OR (UNIX AND NOT APPLE)) 21 | # add_subdirectory(frontend/puredata/nn_terrain) # PURE DATA EXTERNAL 22 | # endif() 23 | 24 | if(APPLE OR MSVC) 25 | add_subdirectory(frontend/maxmsp/nn.terrain_tilde) # MAX MSP EXTERNAL 26 | add_subdirectory(frontend/maxmsp/nn.terrain.encode) # MAX MSP EXTERNAL 27 | add_subdirectory(frontend/maxmsp/nn.terrain.record) # MAX MSP EXTERNAL 28 | add_subdirectory(frontend/maxmsp/nn.terrain.gui) # MAX MSP EXTERNAL 29 | endif() -------------------------------------------------------------------------------- /src/backend/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.0 FATAL_ERROR) 2 | if (APPLE) 3 | set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum OS X deployment version" FORCE) 4 | endif() 5 | 6 | project(backend) 7 | 8 | find_package(Torch REQUIRED) 9 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}") 10 | 11 | add_library(backend STATIC parsing_utils.cpp backend.cpp backend.h) 12 | target_link_libraries(backend "${TORCH_LIBRARIES}") 13 | set_property(TARGET backend PROPERTY CXX_STANDARD 17) 14 | 15 | if(MSVC) 16 | set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /MT") 17 | set(CMAKE_CXX_FLAGS_MINSIZEREL "${CMAKE_CXX_FLAGS_MINSIZEREL} /MT") 18 | set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} /MT") 19 | set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MT") 20 | endif() -------------------------------------------------------------------------------- /src/backend/backend.cpp: -------------------------------------------------------------------------------- 1 | // Adapted from https://github.com/acids-ircam/nn_tilde/blob/master/src/backend/backend.cpp 2 | 3 | #include "backend.h" 4 | #include "parsing_utils.h" 5 | #include 6 | #include 7 | #include 8 | 9 | #define CPU torch::kCPU 10 | #define CUDA torch::kCUDA 11 | #define MPS torch::kMPS 12 | 13 | //#ifdef WIN_VERSION 14 | #include 15 | //#endif 16 | 17 | FCPPN::FCPPN() : m_loaded(0), m_device(CPU), m_use_gpu(false) { 18 | at::init_num_threads(); 19 | } 20 | 21 | Backend::Backend() : m_loaded(0), m_device(CPU), m_use_gpu(false) { 22 | at::init_num_threads(); 23 | } 24 | 25 | void Backend::perform(std::vector in_buffer, 26 | std::vector out_buffer, 27 | int n_vec, // m_buffer_size 28 | std::string method, 29 | int n_batches) { 30 | c10::InferenceMode guard; 31 | 32 | auto params = get_method_params(method); 33 | // std::cout << "in_buffer length : " << in_buffer.size() << std::endl; 34 | // std::cout << "out_buffer length : " << out_buffer.size() << std::endl; 35 | 36 | if (!params.size()) 37 | return; 38 | 39 | auto in_dim = params[0]; 40 | auto in_ratio = params[1]; 41 | auto out_dim = params[2]; 42 | auto out_ratio = params[3]; 43 | 44 | if (!m_loaded) 45 | return; 46 | 47 | // COPY BUFFER INTO A TENSOR 48 | std::vector tensor_in; 49 | // for (auto buf : in_buffer) 50 | for (int i(0); i < in_buffer.size(); i++) { 51 | tensor_in.push_back(torch::from_blob(in_buffer[i], {1, 1, n_vec})); 52 | } 53 | 54 | auto cat_tensor_in = torch::cat(tensor_in, 1); 55 | cat_tensor_in = cat_tensor_in.reshape({in_dim, n_batches, -1, in_ratio}); 56 | cat_tensor_in = cat_tensor_in.select(-1, -1); // this line gets rid of the last dimension 57 | cat_tensor_in = cat_tensor_in.permute({1, 0, 2}); // -> {n_batches, in_dim, n latent per buffer} 58 | 59 | 60 | // SEND TENSOR TO DEVICE 61 | std::unique_lock model_lock(m_model_mutex); 62 | cat_tensor_in = cat_tensor_in.to(m_device); 63 | std::vector inputs = {cat_tensor_in}; 64 | 65 | // PROCESS TENSOR 66 | at::Tensor tensor_out; 67 | try { 68 | tensor_out = m_model.get_method(method)(inputs).toTensor(); 69 | tensor_out = tensor_out.repeat_interleave(out_ratio).reshape( 70 | {n_batches, out_dim, -1}); 71 | } catch (const std::exception &e) { 72 | std::cerr << e.what() << '\n'; 73 | return; 74 | } 75 | model_lock.unlock(); 76 | 77 | int out_batches(tensor_out.size(0)), out_channels(tensor_out.size(1)), 78 | out_n_vec(tensor_out.size(2)); 79 | 80 | // for (int b(0); b < out_batches; b++) { 81 | // for (int c(0); c < out_channels; c++) { 82 | // std::cout << b << ";" << c << ";" << tensor_out[b][c].min().item() << std::endl; 83 | // } 84 | // } 85 | 86 | // CHECKS ON TENSOR SHAPE 87 | if (out_batches * out_channels != out_buffer.size()) { 88 | std::cout << "bad out_buffer size, expected " << out_batches * out_channels 89 | << " buffers, got " << out_buffer.size() << "!\n"; 90 | return; 91 | } 92 | 93 | if (out_n_vec != n_vec) { 94 | std::cout << "model output size is not consistent, expected " << n_vec 95 | << " samples, got " << out_n_vec << "!\n"; 96 | return; 97 | } 98 | 99 | tensor_out = tensor_out.to(CPU); 100 | tensor_out = tensor_out.reshape({out_batches * out_channels, -1}); 101 | auto out_ptr = tensor_out.contiguous().data_ptr(); 102 | 103 | for (int i(0); i < out_buffer.size(); i++) { 104 | memcpy(out_buffer[i], out_ptr + i * n_vec, n_vec * sizeof(float)); 105 | } 106 | } 107 | 108 | int Backend::load(std::string path) { 109 | try { 110 | auto model = torch::jit::load(path); 111 | model.eval(); 112 | model.to(m_device); 113 | 114 | std::unique_lock model_lock(m_model_mutex); 115 | m_model = model; 116 | m_loaded = 1; 117 | model_lock.unlock(); 118 | 119 | m_available_methods = get_available_methods(); 120 | m_path = path; 121 | return 0; 122 | } catch (const std::exception &e) { 123 | std::cerr << e.what() << '\n'; 124 | return 1; 125 | } 126 | } 127 | 128 | int Backend::reload() { 129 | auto return_code = load(m_path); 130 | return return_code; 131 | } 132 | 133 | bool Backend::has_method(std::string method_name) { 134 | std::unique_lock model_lock(m_model_mutex); 135 | for (const auto &m : m_model.get_methods()) { 136 | if (m.name() == method_name) 137 | return true; 138 | } 139 | return false; 140 | } 141 | 142 | bool Backend::has_settable_attribute(std::string attribute) { 143 | for (const auto &a : get_settable_attributes()) { 144 | if (a == attribute) 145 | return true; 146 | } 147 | return false; 148 | } 149 | 150 | std::vector Backend::get_available_methods() { 151 | std::vector methods; 152 | try { 153 | std::vector dumb_input = {}; 154 | 155 | std::unique_lock model_lock(m_model_mutex); 156 | auto methods_from_model = 157 | m_model.get_method("get_methods")(dumb_input).toList(); 158 | model_lock.unlock(); 159 | 160 | for (int i = 0; i < methods_from_model.size(); i++) { 161 | methods.push_back(methods_from_model.get(i).toStringRef()); 162 | } 163 | } catch (...) { 164 | std::unique_lock model_lock(m_model_mutex); 165 | for (const auto &m : m_model.get_methods()) { 166 | try { 167 | auto method_params = m_model.attr(m.name() + "_params"); 168 | methods.push_back(m.name()); 169 | } catch (...) { 170 | } 171 | } 172 | model_lock.unlock(); 173 | } 174 | return methods; 175 | } 176 | 177 | std::vector Backend::get_available_attributes() { 178 | std::vector attributes; 179 | std::unique_lock model_lock(m_model_mutex); 180 | for (const auto &attribute : m_model.named_attributes()) 181 | attributes.push_back(attribute.name); 182 | return attributes; 183 | } 184 | 185 | std::vector Backend::get_settable_attributes() { 186 | std::vector attributes; 187 | try { 188 | std::vector dumb_input = {}; 189 | std::unique_lock model_lock(m_model_mutex); 190 | auto methods_from_model = 191 | m_model.get_method("get_attributes")(dumb_input).toList(); 192 | model_lock.unlock(); 193 | for (int i = 0; i < methods_from_model.size(); i++) { 194 | attributes.push_back(methods_from_model.get(i).toStringRef()); 195 | } 196 | } catch (...) { 197 | std::unique_lock model_lock(m_model_mutex); 198 | for (const auto &a : m_model.named_attributes()) { 199 | try { 200 | auto method_params = m_model.attr(a.name + "_params"); 201 | attributes.push_back(a.name); 202 | } catch (...) { 203 | } 204 | } 205 | model_lock.unlock(); 206 | } 207 | return attributes; 208 | } 209 | 210 | std::vector Backend::get_attribute(std::string attribute_name) { 211 | std::string attribute_getter_name = "get_" + attribute_name; 212 | try { 213 | std::unique_lock model_lock(m_model_mutex); 214 | auto attribute_getter = m_model.get_method(attribute_getter_name); 215 | model_lock.unlock(); 216 | } catch (...) { 217 | throw "getter for attribute " + attribute_name + " not found in model"; 218 | } 219 | std::vector getter_inputs = {}, attributes; 220 | try { 221 | try { 222 | std::unique_lock model_lock(m_model_mutex); 223 | attributes = m_model.get_method(attribute_getter_name)(getter_inputs) 224 | .toList() 225 | .vec(); 226 | model_lock.unlock(); 227 | } catch (...) { 228 | std::unique_lock model_lock(m_model_mutex); 229 | auto output_tuple = 230 | m_model.get_method(attribute_getter_name)(getter_inputs).toTuple(); 231 | attributes = (*output_tuple.get()).elements(); 232 | model_lock.unlock(); 233 | } 234 | } catch (...) { 235 | std::unique_lock model_lock(m_model_mutex); 236 | attributes.push_back( 237 | m_model.get_method(attribute_getter_name)(getter_inputs)); 238 | model_lock.unlock(); 239 | } 240 | return attributes; 241 | } 242 | 243 | std::string Backend::get_attribute_as_string(std::string attribute_name) { 244 | std::vector getter_outputs = get_attribute(attribute_name); 245 | // finstringd arguments 246 | torch::Tensor setter_params; 247 | try { 248 | std::unique_lock model_lock(m_model_mutex); 249 | setter_params = m_model.attr(attribute_name + "_params").toTensor(); 250 | model_lock.unlock(); 251 | } catch (...) { 252 | throw "parameters to set attribute " + attribute_name + 253 | " not found in model"; 254 | } 255 | std::string current_attr = ""; 256 | for (int i = 0; i < setter_params.size(0); i++) { 257 | int current_id = setter_params[i].item().toInt(); 258 | switch (current_id) { 259 | // bool case 260 | case 0: { 261 | current_attr += (getter_outputs[i].toBool()) ? "true" : "false"; 262 | break; 263 | } 264 | // int case 265 | case 1: { 266 | current_attr += std::to_string(getter_outputs[i].toInt()); 267 | break; 268 | } 269 | // float case 270 | case 2: { 271 | float result = getter_outputs[i].to(); 272 | current_attr += std::to_string(result); 273 | break; 274 | } 275 | // str case 276 | case 3: { 277 | current_attr += getter_outputs[i].toStringRef(); 278 | break; 279 | } 280 | default: { 281 | throw "bad type id : " + std::to_string(current_id) + "at index " + 282 | std::to_string(i); 283 | break; 284 | } 285 | } 286 | if (i < setter_params.size(0) - 1) 287 | current_attr += " "; 288 | } 289 | return current_attr; 290 | } 291 | 292 | void Backend::set_attribute(std::string attribute_name, 293 | std::vector attribute_args) { 294 | // find setter 295 | std::string attribute_setter_name = "set_" + attribute_name; 296 | try { 297 | std::unique_lock model_lock(m_model_mutex); 298 | auto attribute_setter = m_model.get_method(attribute_setter_name); 299 | model_lock.unlock(); 300 | } catch (...) { 301 | throw "setter for attribute " + attribute_name + " not found in model"; 302 | } 303 | // find arguments 304 | torch::Tensor setter_params; 305 | try { 306 | std::unique_lock model_lock(m_model_mutex); 307 | setter_params = m_model.attr(attribute_name + "_params").toTensor(); 308 | model_lock.unlock(); 309 | } catch (...) { 310 | throw "parameters to set attribute " + attribute_name + 311 | " not found in model"; 312 | } 313 | // process inputs 314 | std::vector setter_inputs = {}; 315 | for (int i = 0; i < setter_params.size(0); i++) { 316 | int current_id = setter_params[i].item().toInt(); 317 | switch (current_id) { 318 | // bool case 319 | case 0: 320 | setter_inputs.push_back(c10::IValue(to_bool(attribute_args[i]))); 321 | break; 322 | // int case 323 | case 1: 324 | setter_inputs.push_back(c10::IValue(to_int(attribute_args[i]))); 325 | break; 326 | // float case 327 | case 2: 328 | setter_inputs.push_back(c10::IValue(to_float(attribute_args[i]))); 329 | break; 330 | // str case 331 | case 3: 332 | setter_inputs.push_back(c10::IValue(attribute_args[i])); 333 | break; 334 | default: 335 | throw "bad type id : " + std::to_string(current_id) + "at index " + 336 | std::to_string(i); 337 | break; 338 | } 339 | } 340 | try { 341 | std::unique_lock model_lock(m_model_mutex); 342 | auto setter_out = m_model.get_method(attribute_setter_name)(setter_inputs); 343 | model_lock.unlock(); 344 | int setter_result = setter_out.toInt(); 345 | if (setter_result != 0) { 346 | throw "setter returned -1"; 347 | } 348 | } catch (...) { 349 | throw "setter for " + attribute_name + " failed"; 350 | } 351 | } 352 | 353 | std::vector Backend::get_method_params(std::string method) { 354 | std::vector params; 355 | 356 | if (std::find(m_available_methods.begin(), m_available_methods.end(), 357 | method) != m_available_methods.end()) { 358 | try { 359 | std::unique_lock model_lock(m_model_mutex); 360 | auto p = m_model.attr(method + "_params").toTensor(); 361 | model_lock.unlock(); 362 | for (int i(0); i < 4; i++) 363 | params.push_back(p[i].item().to()); 364 | } catch (...) { 365 | } 366 | } 367 | return params; 368 | } 369 | 370 | 371 | int Backend::get_higher_ratio() { 372 | int higher_ratio = 1; 373 | for (const auto &method : m_available_methods) { 374 | auto params = get_method_params(method); 375 | if (!params.size()) 376 | continue; // METHOD NOT USABLE, SKIPPING 377 | int max_ratio = std::max(params[1], params[3]); 378 | higher_ratio = std::max(higher_ratio, max_ratio); 379 | } 380 | return higher_ratio; 381 | } 382 | 383 | bool Backend::is_loaded() { return m_loaded; } 384 | 385 | void Backend::use_gpu(bool value) { 386 | std::unique_lock model_lock(m_model_mutex); 387 | if (value) { 388 | if (torch::hasCUDA()) { 389 | std::cout << "sending model to cuda" << std::endl; 390 | m_device = CUDA; 391 | } else if (torch::hasMPS()) { 392 | std::cout << "sending model to mps" << std::endl; 393 | m_device = MPS; 394 | } else { 395 | std::cout << "sending model to cpu" << std::endl; 396 | m_device = CPU; 397 | } 398 | } else { 399 | m_device = CPU; 400 | } 401 | m_model.to(m_device); 402 | } 403 | 404 | void FCPPN::use_gpu(bool value) { 405 | std::unique_lock model_lock(m_model_mutex); 406 | if (value) { 407 | if (torch::hasCUDA()) { 408 | std::cout << "sending model to cuda" << std::endl; 409 | m_device = CUDA; 410 | } else if (torch::hasMPS()) { 411 | std::cout << "sending model to mps" << std::endl; 412 | m_device = MPS; 413 | } else { 414 | std::cout << "sending model to cpu" << std::endl; 415 | m_device = CPU; 416 | } 417 | } else { 418 | m_device = CPU; 419 | } 420 | m_model->to(m_device); 421 | // model_lock.unlock(); 422 | } 423 | 424 | bool FCPPN::save(std::string save_path, std::string save_name, int m_in_dim, int m_out_dim, 425 | int m_cmax, float m_gauss_scale, int m_mapping_size) { 426 | // std::unique_lock model_lock(m_model_mutex); 427 | try { 428 | m_model->to(CPU); 429 | // TODO: this is not a save path: 430 | torch::serialize::OutputArchive archive; 431 | m_model->save(archive); 432 | // archive << value; 433 | 434 | archive.write("m_in_dim", torch::tensor(m_in_dim)); 435 | archive.write("m_out_dim", torch::tensor(m_out_dim)); 436 | archive.write("m_cmax", torch::tensor(m_cmax)); 437 | archive.write("m_gauss_scale", torch::tensor(m_gauss_scale)); 438 | archive.write("m_mapping_size", torch::tensor(m_mapping_size)); 439 | 440 | // Save the archive to the specified file path 441 | std::string src_path_str = (std::filesystem::path(save_path) / save_name).string(); 442 | archive.save_to(src_path_str); 443 | 444 | m_model->to(m_device); 445 | return true; 446 | } catch (const std::exception &e) { 447 | m_model->to(m_device); 448 | throw e; 449 | return false; 450 | } 451 | } 452 | 453 | 454 | 455 | -------------------------------------------------------------------------------- /src/backend/backend.h: -------------------------------------------------------------------------------- 1 | // Adapted from https://github.com/acids-ircam/nn_tilde/blob/master/src/backend/backend.h 2 | // 3 | 4 | 5 | #pragma once 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | 12 | 13 | struct FourierCPPNImpl : torch::nn::Module { 14 | 15 | FourierCPPNImpl(int in_dim, int out_dim, int c_max, float gauss_scale, int mapping_size) : 16 | fc_in_linear(torch::nn::Linear(in_dim, c_max)), 17 | fc_in_gau(torch::nn::Linear(mapping_size*2, c_max)), 18 | fc2(torch::nn::Linear(c_max, c_max)), 19 | fc3(torch::nn::Linear(c_max, c_max)), 20 | fc4(torch::nn::Linear(c_max, c_max)), 21 | fc_out(torch::nn::Linear(c_max, out_dim)), 22 | prelu1(torch::nn::PReLU()), 23 | prelu2(torch::nn::PReLU()), 24 | prelu3(torch::nn::PReLU()), 25 | prelu4(torch::nn::PReLU()), 26 | two_pi(torch::tensor(2 * M_PI)), 27 | B_T(torch::normal(0, gauss_scale, {mapping_size, in_dim}).transpose(0, 1)) 28 | { 29 | // Construct and register submodules. 30 | gauss = gauss_scale; 31 | c_in_dim = in_dim; 32 | c_out_dim = out_dim; 33 | 34 | if (gauss == 0.0f) { 35 | register_module("fc_in_linear", fc_in_linear); 36 | } else { 37 | register_module("fc_in_gau", fc_in_gau); 38 | register_buffer("two_pi", two_pi); 39 | register_buffer("B_T", B_T); 40 | } 41 | register_module("prelu1", prelu1); 42 | register_module("fc2", fc2); 43 | register_module("prelu2", prelu2); 44 | register_module("fc3", fc3); 45 | register_module("prelu3", prelu3); 46 | register_module("fc4", fc4); 47 | register_module("prelu4", prelu4); 48 | register_module("fc_out", fc_out); 49 | } 50 | 51 | torch::Tensor forward(torch::Tensor x) { 52 | if (gauss > 0.0f) { 53 | x = torch::matmul(two_pi * x, B_T); 54 | x = torch::cat({x.sin(), x.cos()}, 1); 55 | x = prelu1(fc_in_gau(x)); 56 | } else { 57 | x = prelu1(fc_in_linear(x)); 58 | } 59 | x = prelu2(fc2(x)); 60 | x = prelu3(fc3(x)); 61 | x = prelu4(fc4(x)); 62 | x = fc_out(x); 63 | 64 | // x = torch::cat({x, x, x, x}, 1); 65 | // x = torch::ones_like(x); 66 | 67 | return x; 68 | } 69 | 70 | float gauss; 71 | int c_in_dim, c_out_dim; 72 | // std::mutex m_model_mutex; 73 | c10::DeviceType m_device; 74 | torch::nn::Linear fc_in_linear{nullptr}, fc_in_gau{nullptr}, fc2{nullptr}, fc3{nullptr}, fc4{nullptr}, fc_out{nullptr}; 75 | torch::nn::PReLU prelu1{nullptr}, prelu2{nullptr}, prelu3{nullptr}, prelu4{nullptr}; 76 | torch::Tensor two_pi{nullptr}, B_T{nullptr}; 77 | }; 78 | TORCH_MODULE(FourierCPPN); 79 | 80 | 81 | 82 | class Backend { 83 | public: 84 | torch::jit::script::Module m_model; 85 | int m_loaded; 86 | std::string m_path; 87 | std::mutex m_model_mutex; 88 | std::vector m_available_methods; 89 | c10::DeviceType m_device; 90 | bool m_use_gpu; 91 | 92 | 93 | Backend(); 94 | void perform(std::vector in_buffer, std::vector out_buffer, 95 | int n_vec, std::string method, int n_batches); 96 | // void freeze(std::vector in_buffer, std::vector out_buffer, 97 | // int n_vec, std::string method, int n_batches); 98 | bool has_method(std::string method_name); 99 | bool has_settable_attribute(std::string attribute); 100 | std::vector get_available_methods(); 101 | std::vector get_available_attributes(); 102 | std::vector get_settable_attributes(); 103 | std::vector get_attribute(std::string attribute_name); 104 | std::string get_attribute_as_string(std::string attribute_name); 105 | void set_attribute(std::string attribute_name, 106 | std::vector attribute_args); 107 | 108 | std::vector get_method_params(std::string method); 109 | int get_higher_ratio(); 110 | int load(std::string path); 111 | int reload(); 112 | bool is_loaded(); 113 | torch::jit::script::Module get_model() { return m_model; } 114 | void use_gpu(bool value); 115 | }; 116 | 117 | class FCPPN{ 118 | public: 119 | FourierCPPN m_model{nullptr}; 120 | int m_loaded; 121 | std::mutex m_model_mutex; 122 | c10::DeviceType m_device; 123 | bool m_use_gpu; 124 | 125 | FCPPN(); 126 | 127 | int create(int in_dim, int out_dim, int c_max, float gauss_scale, int mapping_size){ 128 | try{ 129 | m_model = FourierCPPN(in_dim, out_dim, c_max, gauss_scale, mapping_size); 130 | m_loaded = 1; 131 | m_model->eval(); 132 | return 0; 133 | } catch (const std::exception &e) { 134 | std::cerr << e.what() << '\n'; 135 | return 1; 136 | } 137 | } 138 | bool is_loaded() { return m_loaded; } 139 | void use_gpu(bool value); 140 | bool save(std::string save_path, std::string save_name, int m_in_dim, int m_out_dim, int m_cmax, float m_gauss_scale, int m_mapping_size); 141 | // std::string load(std::string load_path); 142 | 143 | }; 144 | 145 | 146 | 147 | -------------------------------------------------------------------------------- /src/backend/parsing_utils.cpp: -------------------------------------------------------------------------------- 1 | #include "parsing_utils.h" 2 | 3 | bool to_bool(std::string str) { 4 | std::transform(str.begin(), str.end(), str.begin(), ::tolower); 5 | std::istringstream is(str); 6 | bool b; 7 | is >> std::boolalpha >> b; 8 | return b; 9 | } 10 | 11 | int to_int(std::string str) { return stoi(str); } 12 | 13 | float to_float(std::string str) { return stof(str); } -------------------------------------------------------------------------------- /src/backend/parsing_utils.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | bool to_bool(std::string str); 9 | int to_int(std::string str); 10 | float to_float(std::string str); -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.encode/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | # Copyright 2018 The Min-DevKit Authors. All rights reserved. 2 | # Use of this source code is governed by the MIT License found in the License.md file. 3 | 4 | cmake_minimum_required(VERSION 3.0) 5 | 6 | set ( MSVC_COMPILER_NAME "MSVC" ) 7 | if (${CMAKE_CXX_COMPILER_ID} STREQUAL ${MSVC_COMPILER_NAME}) 8 | string (SUBSTRING ${CMAKE_CXX_COMPILER_VERSION} 0 4 MSVC_VERSION_SHORT) 9 | string (SUBSTRING ${CMAKE_CXX_COMPILER_VERSION} 0 2 MSVC_VERSION_MAJOR) 10 | string (SUBSTRING ${CMAKE_CXX_COMPILER_VERSION} 3 1 MSVC_VERSION_MINOR) 11 | 12 | if (${MSVC_VERSION_MAJOR} VERSION_LESS 19 OR ${MSVC_VERSION_MAJOR} MATCHES 19 AND ${MSVC_VERSION_MINOR} VERSION_LESS 1) 13 | # message(STATUS "Visual Studio ${MSVC_VERSION_SHORT} detected. Visual Studio 17 (19.1) or greater is required for UI objects.") 14 | message(STATUS "Visual Studio 17 or greater is required for UI objects.") 15 | message(STATUS "SKIPPING!") 16 | return () 17 | endif () 18 | endif () 19 | 20 | set(C74_MIN_API_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../min-api) 21 | include(${C74_MIN_API_DIR}/script/min-pretarget.cmake) 22 | 23 | if (APPLE) 24 | set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum OS X deployment version" FORCE) 25 | endif() 26 | 27 | ############################################################# 28 | # MAX EXTERNAL 29 | ############################################################# 30 | 31 | 32 | include_directories( 33 | "${C74_INCLUDES}" 34 | ) 35 | find_package(Torch REQUIRED) 36 | 37 | set( 38 | SOURCE_FILES 39 | nn.terrain.encode.cpp 40 | "../shared/utils.h" 41 | "../shared/min_path.h" 42 | "../shared/min_dictionary.h" 43 | ) 44 | 45 | if (UNIX) 46 | execute_process( 47 | COMMAND git describe --tags 48 | WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR} 49 | OUTPUT_VARIABLE VERSION 50 | OUTPUT_STRIP_TRAILING_WHITESPACE 51 | ) 52 | message(STATUS ${VERSION}) 53 | add_definitions(-DVERSION="${VERSION}") 54 | endif() 55 | 56 | add_library( 57 | ${PROJECT_NAME} 58 | MODULE 59 | ${SOURCE_FILES} 60 | "../shared/utils.h" 61 | "../shared/min_path.h" 62 | "../shared/min_dictionary.h" 63 | ) 64 | target_link_libraries(${PROJECT_NAME} PRIVATE backend) 65 | 66 | if (APPLE) 67 | set_target_properties(${PROJECT_NAME} PROPERTIES 68 | BUILD_WITH_INSTALL_RPATH FALSE 69 | LINK_FLAGS "-Wl,-rpath,@loader_path/" 70 | ) 71 | set(AUTHOR_DOMAIN "com.jasperzheng") 72 | set(BUNDLE_IDENTIFIER "nn-terrain-encode") 73 | endif() 74 | 75 | include(${C74_MIN_API_DIR}/script/min-posttarget.cmake) 76 | 77 | if (APPLE) # COPY TORCH DYLIB IN THE LOADER FOLDER 78 | add_custom_command( 79 | TARGET ${PROJECT_NAME} 80 | POST_BUILD 81 | COMMAND cp "${TORCH_INSTALL_PREFIX}/lib/*.dylib" "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${${PROJECT_NAME}_EXTERN_OUTPUT_NAME}.mxo/Contents/MacOS/" 82 | COMMENT "Copy Torch Libraries" 83 | ) 84 | 85 | if (CMAKE_OSX_ARCHITECTURES STREQUAL "arm64") 86 | add_custom_command( 87 | TARGET ${PROJECT_NAME} 88 | POST_BUILD 89 | COMMAND "codesign" "--force" "--deep" "-s" "-" "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${${PROJECT_NAME}_EXTERN_OUTPUT_NAME}.mxo" 90 | COMMENT "Codesign external" 91 | ) 92 | endif() 93 | endif() 94 | 95 | if (MSVC) # COPY TORCH DLL IN THE LOADER FOLDER 96 | add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD 97 | COMMAND ${CMAKE_COMMAND} -E copy_directory "${TORCH_INSTALL_PREFIX}/lib/" ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/ 98 | ) 99 | endif() 100 | 101 | if (MSVC) 102 | set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 20) 103 | endif() 104 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.encode/nn.terrain.encode.cpp: -------------------------------------------------------------------------------- 1 | #include "../../../backend/backend.h" 2 | #include "c74_min.h" 3 | 4 | #include 5 | #include 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | 13 | #include "../shared/min_path.h" 14 | #include "../shared/min_dictionary.h" 15 | 16 | #include "../shared/utils.h" 17 | 18 | #include 19 | 20 | #ifndef VERSION 21 | #define VERSION "UNDEFINED" 22 | #endif 23 | 24 | #define None at::indexing::None 25 | #define Slice torch::indexing::Slice 26 | 27 | using namespace c74::min; 28 | using namespace c74::min::ui; 29 | 30 | 31 | //std::string min_devkit_path() { 32 | //#ifdef WIN_VERSION 33 | // char pathstr[4096]; 34 | // HMODULE hm = nullptr; 35 | // 36 | // if (!GetModuleHandleExA( 37 | // GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR)&min_devkit_path, &hm)) { 38 | // int ret = GetLastError(); 39 | // fprintf(stderr, "GetModuleHandle() returned %d\n", ret); 40 | // } 41 | // GetModuleFileNameA(hm, pathstr, sizeof(pathstr)); 42 | // 43 | // // path now is the path to this external's binary, including the binary filename. 44 | // auto filename = strrchr(pathstr, '\\'); 45 | // if (filename) 46 | // *filename = 0; 47 | // auto externals = strrchr(pathstr, '\\'); 48 | // if (externals) 49 | // *externals = 0; 50 | // 51 | // path p{ pathstr }; // convert to Max path 52 | // return p; 53 | //#endif // WIN_VERSION 54 | // 55 | //#ifdef MAC_VERSION 56 | // CFBundleRef this_bundle = CFBundleGetBundleWithIdentifier(CFSTR("com.jasperzheng.nn-terrain-encode")); 57 | // CFURLRef this_url = CFBundleCopyExecutableURL(this_bundle); 58 | // char this_path[4096]; 59 | // CFURLGetFileSystemRepresentation(this_url, true, reinterpret_cast(this_path), 4096); 60 | // string this_path_str{ this_path }; 61 | // CFRelease(this_url); 62 | // 63 | // // we now have a path like this: 64 | // // /Users/tim/Materials/min-devkit/externals/min.project.mxo/Contents/MacOS/min.project" 65 | // // so we need to chop off 5 slashes from the end 66 | // 67 | // auto iter = this_path_str.find("/externals/nn.terrain.encode.mxo/Contents/MacOS/nn.terrain.encode"); 68 | // this_path_str.erase(iter, strlen("/externals/nn.terrain.encode.mxo/Contents/MacOS/nn.terrain.encode")); 69 | // return this_path_str; 70 | //#endif // MAC_VERSION 71 | //} 72 | 73 | class nn_terrain : public object{ 74 | 75 | public: 76 | MIN_DESCRIPTION{"Encoding audio buffers to latent vectors with neural audio autoencoder"}; 77 | MIN_AUTHOR{"Jasper Shuoyang Zheng"}; 78 | MIN_RELATED { "nn.terrain, nn.terrain.record, nn~, nn.terrain.gui" }; 79 | 80 | inlet<> append_inlet { this, "(append) Append buffers to the buffer pool", "message"}; 81 | 82 | outlet<> dataset_output { this, "(dictionary) Latent vectors encoded from buffers, to be sent to nn.terrain", "dictionary"}; 83 | outlet<> latent_len_output { this, "(list) Number of latents along each trajectory", "list"}; 84 | outlet<> latent_len_all_output { this, "(int) Total number of latents along all trajectories", "int"}; 85 | outlet<> log_output { this, "(dictionary) List of buffers in the pool", "dictionary"}; 86 | 87 | nn_terrain(const atoms &args = {}); 88 | ~nn_terrain(); 89 | 90 | std::vector> buffers; 91 | 92 | std::unique_ptr m_model; 93 | 94 | bool m_is_backend_init = false; 95 | std::string d_method{"decode"}; 96 | std::string e_method{"encode"}; 97 | 98 | c74::min::path m_path; 99 | int m_in_dim{1}, m_in_ratio{1}, m_out_dim{8}, m_out_ratio{2048}, m_higher_ratio{2048}; 100 | 101 | int load_encoder(string path_str); 102 | 103 | argument path_arg{this, "model_path", "Path to the pretrained model (encoder).", true}; 104 | argument latent_dim_arg {this, "latent_dim", "(Optional) Dimensionality of the autoencoder's latent space."}; 105 | 106 | 107 | // ENABLE / DISABLE ATTRIBUTE 108 | attribute gpu{this, "gpu", true, 109 | description{"Enable / disable gpu usage when available"}, 110 | setter{ 111 | [this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 112 | if (m_is_backend_init) 113 | m_model->use_gpu(bool(args[0])); 114 | // TODO: t_model->use_gpu(bool(args[0])); 115 | return args; 116 | } 117 | } 118 | }; 119 | 120 | int latent_dim = 8; 121 | 122 | attribute batch_size{ this, "encoder_batch_size", 64, description{"batch size used when encoding audio buffers"}, range{{16, 256}} }; 123 | 124 | // BOOT STAMP 125 | message<> maxclass_setup{ 126 | this, "maxclass_setup", 127 | [this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 128 | cout << "torch version: " << TORCH_VERSION << endl; 129 | return {}; 130 | }}; 131 | 132 | min_dict training_data = {symbol(true)}; 133 | min_dict buffer_dict = {symbol(true)}; 134 | min_dict latent_dict = {symbol(true)}; 135 | atoms latent_lengths; 136 | numbers dataset_counts = {0,0,0}; 137 | 138 | int latent_lengths_all = 0; 139 | int buffer_count = 0; 140 | message<> m_set_bufferlist {this, "append", "Add a buffer to the buffer pool", 141 | MIN_FUNCTION { 142 | if (args.size() != 1){ 143 | cerr << "args size " << args.size() << " error, args size should be 1" << endl; 144 | return {}; 145 | } 146 | buffer_dict[buffer_count] = std::string(args[0]); 147 | buffer_dict.touch(); 148 | 149 | buffers.push_back(std::make_unique(this)); 150 | atom buffer_name = buffer_dict[buffer_count]; 151 | buffers[buffer_count]->set(buffer_name); 152 | 153 | buffer_lock b {*buffers[buffer_count]}; 154 | int latents_count = static_cast(std::ceil(b.frame_count() / static_cast(m_out_ratio))); 155 | latent_lengths.push_back(latents_count); 156 | latent_lengths_all += latents_count; 157 | 158 | buffer_count++; 159 | log_output.send("dictionary", buffer_dict.name()); 160 | latent_len_all_output.send(latent_lengths_all); 161 | latent_len_output.send(latent_lengths); 162 | return {}; 163 | } 164 | }; 165 | 166 | 167 | 168 | message<> dictionary { this, "dictionary", 169 | "Use a dictionary to gather training data for the terrain", 170 | MIN_FUNCTION { 171 | c74::max::t_symbol **keys = NULL; 172 | long numkeys = 0; 173 | 174 | min_dict d = {args[0]}; 175 | c74::max::dictionary_getkeys(d.m_instance, &numkeys, &keys); 176 | 177 | for (int i = 0; i < numkeys; i++){ 178 | string key_str = std::string(keys[i]->s_name); 179 | atom d_data = d[key_str]; 180 | min_dict d_data_dict = {d_data}; 181 | // atom count = c74::max::dictionary_getentrycount(d_data_dict.m_instance); 182 | 183 | if (key_str == "coordinates"){ 184 | cout << "please send coordinates directly to nn.terrain" << endl; 185 | } else if (key_str == "buffers"){ 186 | buffer_dict.clear(); 187 | buffer_dict.copyunique(d_data_dict); 188 | latent_lengths.clear(); 189 | latent_lengths_all = 0; 190 | buffers.erase(std::remove_if(buffers.begin(), buffers.end(), [](auto const& pi){ return *pi % 2 == 0; }), buffers.end()); 191 | buffers.clear(); 192 | 193 | int numkeys_b = static_cast(c74::max::dictionary_getentrycount(buffer_dict.m_instance)); 194 | for (int i = 1; i <= numkeys_b; i++){ 195 | buffers.push_back(std::make_unique(this)); 196 | atom buffer_name = buffer_dict[i]; 197 | buffers[i-1]->set(buffer_name); 198 | 199 | buffer_lock b {*buffers[i-1]}; 200 | int latents_count = static_cast(std::ceil(b.frame_count()/static_cast(m_out_ratio))); 201 | latent_lengths.push_back(latents_count); 202 | latent_lengths_all += latents_count; 203 | } 204 | } else { 205 | cerr << "unknown key: " << key_str << endl; 206 | } 207 | } 208 | if (keys){ 209 | c74::max::dictionary_freekeys(d.m_instance, numkeys, keys); 210 | } 211 | latent_len_all_output.send(latent_lengths_all); 212 | latent_len_output.send(latent_lengths); 213 | // traj_len_output.send(dataset_counts[0]); 214 | return {}; 215 | } 216 | }; 217 | 218 | message<> create_dataset_from_buffers {this, "encode", "Encode buffers in the pool to a sequence of latent vectors", 219 | MIN_FUNCTION { 220 | 221 | if (!m_model->is_loaded()){ 222 | cerr << "encoder not loaded" << endl; 223 | return {}; 224 | } 225 | if (buffers.empty()){ 226 | 227 | cerr << "no buffers loaded" << endl; 228 | return {}; 229 | } 230 | // ============== 231 | // return {}; 232 | std::vector tensor_in; 233 | std::vector latent_lens; 234 | try{ 235 | // TODO: need a better way to convert buffers to tensors 236 | for (int i(0); i < buffers.size(); i++) { 237 | buffer_lock b {*buffers[i]}; 238 | 239 | // vector buffer_data(b.lookup(0, 0), b.lookup(b.frame_count()-1, 0)); 240 | 241 | vector buffer_data; 242 | for (int j(0); j < b.frame_count(); j++){ 243 | float buffer_data_p = b.lookup(j, 0); 244 | buffer_data.push_back(buffer_data_p); 245 | } 246 | 247 | at::Tensor buffer_tensor = torch::from_blob(buffer_data.data(), {1, 1, static_cast(b.frame_count())}, torch::kFloat); 248 | buffer_tensor = buffer_tensor.clone(); 249 | 250 | // if buffer size is not divisible by m_out_ratio (2048), pad with zeros at the end 251 | int zeros = m_out_ratio - b.frame_count() % m_out_ratio; 252 | latent_lens.push_back( 253 | std::ceil(static_cast(b.frame_count()) / static_cast(m_out_ratio)) 254 | ); 255 | at::Tensor zero_fill = torch::zeros({1,1,zeros}); 256 | buffer_tensor = torch::cat({buffer_tensor, zero_fill}, 2); 257 | tensor_in.push_back(buffer_tensor); 258 | } 259 | } catch (const std::exception &e) { 260 | cerr << e.what() << endl; 261 | return {}; 262 | } 263 | // return {}; 264 | auto cat_tensor_in = torch::cat(tensor_in, 2); // -> [1, 1, num_samples] 265 | 266 | // batching, in case the buffers are too long: 267 | at::Tensor cat_tensor_in_trim, cat_tensor_in_left; 268 | try{ 269 | float leftover = cat_tensor_in.size(2) % (m_out_ratio*batch_size); 270 | float trim = cat_tensor_in.size(2) - leftover; 271 | 272 | cat_tensor_in_trim = cat_tensor_in.index({Slice(None), Slice(None), Slice(None, trim) }); 273 | cat_tensor_in_trim = cat_tensor_in_trim.reshape({-1,1,(m_out_ratio*batch_size)}); 274 | cat_tensor_in_left = cat_tensor_in.index({Slice(None), Slice(None), Slice(trim, None)}); 275 | } catch (const std::exception &e) { 276 | cerr << e.what() << endl; 277 | return {}; 278 | } 279 | 280 | // forward pass: 281 | std::vector tensor_out_trim; 282 | at::Tensor cat_tensor_out; 283 | // return {}; 284 | std::unique_lock model_lock(m_model->m_model_mutex); 285 | // torch::NoGradGuard no_grad; 286 | try { 287 | torch::NoGradGuard no_grad; 288 | for (int i(0); i < cat_tensor_in_trim.size(0); i++){ 289 | at::Tensor input_tensor = cat_tensor_in_trim.index({i}).unsqueeze(0).to(m_model->m_device); 290 | std::vector inputs = {input_tensor}; 291 | 292 | at::Tensor tensor_out = m_model->m_model.get_method(e_method)(inputs).toTensor(); 293 | // tensor_out = tensor_out.clone(); 294 | tensor_out = tensor_out.index({0}).permute({1,0}).to(torch::kCPU); 295 | tensor_out_trim.push_back(tensor_out); 296 | } 297 | cat_tensor_in_left = cat_tensor_in_left.to(m_model->m_device); 298 | std::vector inputs_left = {cat_tensor_in_left}; 299 | at::Tensor tensor_out_left = m_model->m_model.get_method(e_method)(inputs_left).toTensor(); 300 | tensor_out_left = tensor_out_left.index({0}).permute({1,0}).to(torch::kCPU); 301 | tensor_out_trim.push_back(tensor_out_left); 302 | 303 | cat_tensor_out = torch::cat(tensor_out_trim, 0); 304 | 305 | model_lock.unlock(); 306 | } catch (const std::exception &e) { 307 | std::string str = e.what(); 308 | if (str.size() > 1000) { 309 | str.erase(0, 1000); 310 | } 311 | cerr << str << endl; 312 | 313 | return {}; 314 | } 315 | // return {}; 316 | 317 | latent_dict.clear(); 318 | 319 | torch::Tensor cat_tensor_out_clone = cat_tensor_out.permute({1,0}).contiguous(); 320 | const float* latents_ptr = cat_tensor_out_clone.data_ptr(); 321 | 322 | int c = 0; 323 | for (int i(0); i < latent_lens.size(); i++){ 324 | min_dict traj_dict = {}; 325 | 326 | for (int z(0); z < cat_tensor_out.size(1); z++){ // latent_dim 327 | atoms result(latents_ptr + c + z*cat_tensor_out.size(0), latents_ptr + c + z*cat_tensor_out.size(0)+ latent_lens[i]); 328 | symbol skey{z}; 329 | c74::max::dictionary_appendatoms(traj_dict.m_instance, skey, result.size(), &result[0]); 330 | } 331 | c += latent_lens[i]; 332 | latent_dict[std::to_string(i+1)] = traj_dict; 333 | } 334 | 335 | training_data.touch(); 336 | dataset_output.send("dictionary", training_data.name()); 337 | return {}; 338 | } 339 | }; 340 | 341 | message<> clear_buffers {this, "clear", "Clear recorded latents", 342 | MIN_FUNCTION { 343 | buffer_dict.clear(); 344 | buffer_dict.touch(); 345 | buffer_count = 0; 346 | 347 | buffers.erase(std::remove_if(buffers.begin(), buffers.end(), [](auto const& pi){ return *pi % 2 == 0; }), buffers.end()); 348 | buffers.clear(); 349 | 350 | latent_lengths_all = 0; 351 | latent_lengths.clear(); 352 | 353 | log_output.send("dictionary", buffer_dict.name()); 354 | return {}; 355 | } 356 | }; 357 | }; 358 | 359 | 360 | int nn_terrain::load_encoder(string path_str){ 361 | 362 | if (path_str.length()>3){ 363 | if (path_str.substr(path_str.length() - 3) != ".ts"){ 364 | path_str = path_str + ".ts"; 365 | } 366 | } 367 | m_path = path(path_str); 368 | 369 | if (m_model->load(std::string(m_path))) { 370 | cerr << "error during loading" << endl; 371 | error(); 372 | return 0; 373 | } 374 | m_model->use_gpu(gpu); 375 | 376 | 377 | m_higher_ratio = m_model->get_higher_ratio(); 378 | 379 | // GET MODEL'S METHOD PARAMETERS 380 | auto params = m_model->get_method_params(e_method); 381 | 382 | 383 | if (!params.size()) { 384 | error("method " + e_method + " not found !"); 385 | } 386 | 387 | m_in_dim = params[0]; 388 | m_in_ratio = params[1]; 389 | m_out_dim = params[2]; 390 | m_out_ratio = params[3]; 391 | cout << "m_in_dim: " << m_in_dim << "\n m_in_ratio: " << m_in_ratio << "\n m_out_dim: " << m_out_dim << "\n m_out_ratio: " << m_out_ratio << endl; 392 | 393 | // Calling forward in a thread causes memory leak in windows. 394 | // See https://github.com/pytorch/pytorch/issues/24237 395 | //#ifdef _WIN32 396 | // m_use_thread = false; 397 | //#endif 398 | 399 | // if (m_use_thread) { 400 | // m_compute_thread = std::make_unique(model_perform_loop, this); 401 | // } 402 | 403 | return m_out_dim; 404 | } 405 | 406 | 407 | nn_terrain::nn_terrain(const atoms &args){ 408 | 409 | m_model = std::make_unique(); 410 | m_is_backend_init = true; 411 | 412 | training_data["latents"] = latent_dict; 413 | // CHECK ARGUMENTS 414 | if (!args.size()) { 415 | return; 416 | } else if (args.size() == 2) { // TWO ARGUMENT IS GIVEN: 417 | // a_type 1: int, a_type 2: float, a_type 3: symbol 418 | if (args[0].a_type == 1 && args[1].a_type == 3) { // (int, symbol) 419 | latent_dim = int(args[1]); 420 | int latent_available = load_encoder(std::string(args[0])); 421 | if (latent_available > latent_dim){ 422 | cout << "defined latent_dim is lower than the model's latent_dim, " << latent_available - latent_dim << " latents from model is unused" << endl; 423 | } else if (latent_available < latent_dim){ 424 | cout << "defined latent_dim is higher than the model's latent_dim, " << latent_dim - latent_available << " latents will remain empty" << endl; 425 | } 426 | } else { 427 | cerr << "error: if two arguments are given, the first argument should be int, second argument should be symbol" << endl; 428 | } 429 | } else if (args.size() == 1) { // ONE ARGUMENT (symbols) ARE GIVEN 430 | if (args[0].a_type == 3){ 431 | latent_dim = load_encoder(std::string(args[0])); 432 | } else { 433 | cerr << "error: if one argument is given, it should be symbol" << endl; 434 | } 435 | } else { // THREE ARGUMENTS ARE GIVEN 436 | cerr << "error: maximum two arguments" << endl; 437 | } 438 | 439 | // external_path = min_devkit_path(); 440 | // cout << external_path << endl; 441 | 442 | return; 443 | } 444 | 445 | nn_terrain::~nn_terrain() { 446 | 447 | } 448 | MIN_EXTERNAL(nn_terrain); 449 | 450 | 451 | 452 | 453 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.gui/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | # Copyright 2018 The Min-DevKit Authors. All rights reserved. 2 | # Use of this source code is governed by the MIT License found in the License.md file. 3 | 4 | cmake_minimum_required(VERSION 3.0) 5 | 6 | set(C74_MIN_API_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../min-api) 7 | include(${C74_MIN_API_DIR}/script/min-pretarget.cmake) 8 | 9 | if (APPLE) 10 | set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum OS X deployment version" FORCE) 11 | endif() 12 | 13 | ############################################################# 14 | # MAX EXTERNAL 15 | ############################################################# 16 | 17 | 18 | include_directories( 19 | "${C74_INCLUDES}" 20 | ) 21 | 22 | 23 | set( SOURCE_FILES 24 | ${PROJECT_NAME}.cpp 25 | "pen.h" 26 | "../shared/min_path.h" 27 | "terrain.h" 28 | "../shared/min_dictionary.h" 29 | ) 30 | 31 | 32 | add_library( 33 | ${PROJECT_NAME} 34 | MODULE 35 | ${SOURCE_FILES} 36 | "pen.h" 37 | "../shared/min_path.h" 38 | "terrain.h" 39 | "../shared/min_dictionary.h" 40 | ) 41 | if (APPLE) 42 | target_link_libraries(${PROJECT_NAME} PUBLIC "$") 43 | target_link_libraries(${PROJECT_NAME} PUBLIC "$") 44 | set_target_properties(${PROJECT_NAME} PROPERTIES 45 | LINK_FLAGS "-Wl,-F/Library/Frameworks" 46 | ) 47 | # I can't figure out how to set the Bundle Identifier nicely in for max-sdk-base, so I'm just gonna do this thing here... 48 | set(AUTHOR_DOMAIN "com.jasperzheng") 49 | set(BUNDLE_IDENTIFIER "nn-terrain-gui") 50 | endif() 51 | 52 | include(${C74_MIN_API_DIR}/script/min-posttarget.cmake) 53 | 54 | 55 | ############################################################# 56 | # UNIT TEST 57 | ############################################################# 58 | 59 | #include(${C74_MIN_API_DIR}/test/min-object-unittest.cmake) 60 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.gui/pen.h: -------------------------------------------------------------------------------- 1 | 2 | #include 3 | #include 4 | #include 5 | 6 | #include "../shared/min_path.h" 7 | 8 | #pragma once 9 | 10 | using namespace c74::min; 11 | using namespace c74::min::ui; 12 | 13 | class pen { 14 | public: 15 | pen(object_base* an_owner, const function& a_function = nullptr, const function& traj_function = nullptr, const function& play_function = nullptr) 16 | : m_width{ 600.0 } 17 | , m_height{ 150.0 } 18 | , m_ink_callback{ a_function } 19 | , m_traj_callback{ traj_function } 20 | , m_play_callback{ play_function } { 21 | } 22 | 23 | ~pen() { 24 | if (m_surface) { 25 | c74::max::jgraphics_surface_destroy(m_surface); 26 | m_surface = nullptr; 27 | } 28 | if (m_history_surface) { 29 | c74::max::jgraphics_surface_destroy(m_history_surface); 30 | m_history_surface = nullptr; 31 | } 32 | } 33 | 34 | void clear_history() { 35 | m_history_surface = c74::max::jgraphics_image_surface_create(c74::max::JGRAPHICS_FORMAT_ARGB32, m_width, m_height); 36 | draw_inks(m_width, m_height); 37 | } 38 | 39 | void draw_trajs(const int width, const int height) { 40 | auto old_surface = m_traj_surface; 41 | m_traj_surface = c74::max::jgraphics_image_surface_create(c74::max::JGRAPHICS_FORMAT_ARGB32, width, height); 42 | m_width = width; 43 | m_height = height; 44 | c74::max::t_jgraphics* ctx = jgraphics_create(m_traj_surface); 45 | atoms a{ { ctx, m_width, m_height} }; 46 | target t{ a }; 47 | 48 | auto length = m_traj_callback(a, 0)[0]; 49 | 50 | c74::max::jgraphics_destroy(ctx); 51 | if (old_surface) { 52 | c74::max::jgraphics_surface_destroy(old_surface); 53 | } 54 | } 55 | 56 | void draw_plays(const int width, const int height) { 57 | auto old_surface = m_traj_surface; 58 | m_traj_surface = c74::max::jgraphics_image_surface_create(c74::max::JGRAPHICS_FORMAT_ARGB32, width, height); 59 | m_width = width; 60 | m_height = height; 61 | c74::max::t_jgraphics* ctx = jgraphics_create(m_traj_surface); 62 | atoms a{ { ctx, m_width, m_height} }; 63 | target t{ a }; 64 | 65 | auto length = m_play_callback(a, 0)[0]; 66 | 67 | c74::max::jgraphics_destroy(ctx); 68 | if (old_surface) { 69 | c74::max::jgraphics_surface_destroy(old_surface); 70 | } 71 | } 72 | 73 | void draw_trajs_surface(ui::target& t) { 74 | c74::max::jgraphics_image_surface_draw(t, m_traj_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, m_width, m_height }); 75 | } 76 | 77 | void draw_inks(const int width, const int height) { 78 | auto old_surface = m_surface; 79 | m_surface = c74::max::jgraphics_image_surface_create(c74::max::JGRAPHICS_FORMAT_ARGB32, width, height); 80 | m_width = width; 81 | m_height = height; 82 | c74::max::t_jgraphics* ctx = jgraphics_create(m_surface); 83 | 84 | atoms a{ { ctx, m_width, m_height} }; 85 | target t{ a }; 86 | 87 | if (m_history_surface) { 88 | color c{ 1.0, 1.0, 1.0, 1.0 }; 89 | c74::max::jgraphics_set_source_jrgba(ctx, c); 90 | c74::max::jgraphics_image_surface_draw(ctx, m_history_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, m_width, m_height }); 91 | } 92 | 93 | m_history_ink = m_ink_callback(a, 0)[0]; 94 | 95 | c74::max::jgraphics_destroy(ctx); 96 | 97 | if (old_surface) { 98 | c74::max::jgraphics_surface_destroy(old_surface); 99 | } 100 | } 101 | 102 | void draw_history(ui::target& t, const double width, const double height) { 103 | if (m_history_surface) { 104 | c74::max::jgraphics_image_surface_draw(t, m_history_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, m_width, m_height }); 105 | } 106 | } 107 | 108 | 109 | void draw_surface(ui::target& t, const double width, const double height) { 110 | c74::max::jgraphics_image_surface_draw(t, m_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, width, height }); 111 | } 112 | 113 | void lock_canvas() { 114 | auto old_surface = m_history_surface; 115 | m_history_surface = c74::max::jgraphics_image_surface_create(c74::max::JGRAPHICS_FORMAT_ARGB32, m_width, m_height); 116 | c74::max::t_jgraphics* ctx = jgraphics_create(m_history_surface); 117 | 118 | color c{ 1.0, 1.0, 1.0, 1.0 }; 119 | c74::max::jgraphics_set_source_jrgba(ctx, c); 120 | c74::max::jgraphics_image_surface_draw(ctx, m_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, m_width, m_height }); 121 | c74::max::jgraphics_destroy(ctx); 122 | if (old_surface) { 123 | c74::max::jgraphics_surface_destroy(old_surface); 124 | } 125 | } 126 | 127 | void save_png(symbol file_name, min_path m_log_path, long dpi) { 128 | c74::max::jgraphics_image_surface_writepng(m_surface, file_name, m_log_path.get_path()/*c74::max::path_desktopfolder()*/, dpi); 129 | } 130 | 131 | 132 | private: 133 | double m_width; 134 | double m_height; 135 | function m_ink_callback; 136 | function m_traj_callback; 137 | function m_play_callback; 138 | 139 | c74::max::t_jsurface* m_traj_surface{ nullptr }; 140 | 141 | c74::max::t_jsurface* m_surface{ nullptr }; 142 | c74::max::t_jsurface* m_history_surface{ nullptr }; 143 | 144 | double m_history_ink{ 0.0 }; 145 | }; 146 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.gui/terrain.h: -------------------------------------------------------------------------------- 1 | 2 | #include "../shared/min_path.h" 3 | 4 | #pragma once 5 | 6 | 7 | using namespace c74::min::ui; 8 | using namespace c74::min; 9 | 10 | class terrain { 11 | public: 12 | terrain(object_base* an_owner, const double width, const double height, const function& a_function = nullptr) 13 | : m_width{ width } 14 | , m_height{ height } 15 | , m_draw_callback{ a_function } { 16 | } 17 | 18 | ~terrain() { 19 | if (m_surface) { 20 | c74::max::jgraphics_surface_destroy(m_surface); 21 | m_surface = nullptr; 22 | } 23 | } 24 | 25 | 26 | void redraw(const int width, const int height) { 27 | auto old_surface = m_surface; 28 | m_surface = c74::max::jgraphics_image_surface_create(c74::max::JGRAPHICS_FORMAT_ARGB32, width, height); 29 | m_width = width; 30 | m_height = height; 31 | c74::max::t_jgraphics* ctx = jgraphics_create(m_surface); 32 | 33 | atoms a{ { ctx, m_width, m_height} }; 34 | 35 | target t{ a }; 36 | 37 | m_draw_callback(a, 0); 38 | c74::max::jgraphics_image_surface_draw(ctx, m_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, m_width, m_height }); 39 | 40 | c74::max::jgraphics_destroy(ctx); 41 | 42 | if (old_surface) { 43 | c74::max::jgraphics_surface_destroy(old_surface); 44 | } 45 | } 46 | 47 | void save_png(symbol file_name, min_path m_log_path, long dpi) { 48 | c74::max::jgraphics_image_surface_writepng(m_surface, file_name, m_log_path.get_path(), dpi); 49 | } 50 | 51 | void draw_surface(ui::target& t, const double width, const double height) { 52 | c74::max::jgraphics_image_surface_draw(t, m_surface, { 0.0, 0.0, m_width, m_height }, { 0.0, 0.0, width, height }); 53 | } 54 | 55 | private: 56 | double m_width; 57 | double m_height; 58 | function m_draw_callback; 59 | c74::max::t_jsurface* m_surface{ nullptr }; 60 | }; 61 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.record/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | # Copyright 2018 The Min-DevKit Authors. All rights reserved. 2 | # Use of this source code is governed by the MIT License found in the License.md file. 3 | 4 | cmake_minimum_required(VERSION 3.0) 5 | 6 | set(C74_MIN_API_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../min-api) 7 | include(${C74_MIN_API_DIR}/script/min-pretarget.cmake) 8 | 9 | if (APPLE) 10 | set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum OS X deployment version" FORCE) 11 | endif() 12 | 13 | ############################################################# 14 | # MAX EXTERNAL 15 | ############################################################# 16 | 17 | 18 | include_directories( 19 | "${C74_INCLUDES}" 20 | ) 21 | 22 | 23 | 24 | set( 25 | SOURCE_FILES 26 | nn.terrain.record.cpp 27 | "../shared/circular_buffer.h" 28 | "../shared/utils.h" 29 | "../shared/min_path.h" 30 | "../shared/min_dictionary.h" 31 | ) 32 | 33 | 34 | add_library( 35 | ${PROJECT_NAME} 36 | MODULE 37 | ${SOURCE_FILES} 38 | "../shared/circular_buffer.h" 39 | "../shared/utils.h" 40 | "../shared/min_path.h" 41 | "../shared/min_dictionary.h" 42 | ) 43 | if (APPLE) 44 | set(AUTHOR_DOMAIN "com.jasperzheng") 45 | set(BUNDLE_IDENTIFIER "nn-terrain-record") 46 | endif() 47 | 48 | include(${C74_MIN_API_DIR}/script/min-posttarget.cmake) 49 | 50 | 51 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain.record/nn.terrain.record.cpp: -------------------------------------------------------------------------------- 1 | //#include "../../../backend/backend.h" 2 | #include "../shared/circular_buffer.h" 3 | 4 | #include "c74_min.h" 5 | 6 | #include 7 | #include 8 | 9 | #include 10 | #include 11 | 12 | #include "../shared/min_path.h" 13 | #include "../shared/min_dictionary.h" 14 | 15 | #include "../shared/utils.h" 16 | 17 | #include 18 | 19 | #ifndef VERSION 20 | #define VERSION "UNDEFINED" 21 | #endif 22 | 23 | using namespace c74::min; 24 | using namespace c74::min::ui; 25 | 26 | typedef struct traj_info_type { 27 | vector> traj; 28 | int length; 29 | } traj_info; 30 | 31 | class nn_terrain : public object, public vector_operator<> { 32 | 33 | public: 34 | MIN_DESCRIPTION{"Recording a multichannel signal as latent vectors, for nn.terrain."}; 35 | MIN_TAGS { "" }; 36 | MIN_AUTHOR{"Jasper Shuoyang Zheng"}; 37 | MIN_RELATED { "nn.terrain, nn.terrain.encode, nn.terrain.record, nn~" }; 38 | 39 | std::vector>> m_inlets; 40 | 41 | outlet<> dataset_output { this, "(dictionary) Recorded latent vectors, to be sent to nn.terrain", "dictionary"}; 42 | 43 | outlet<> latent_len_output { this, "(list) Number of latents along each trajectory", "list"}; 44 | outlet<> latent_len_all_output { this, "(int) Total number of latents along all trajectories", "int"}; 45 | outlet<> traj_num_output { this, "(int) Number of trajectories", "int"}; 46 | 47 | nn_terrain(const atoms &args = {}); 48 | ~nn_terrain(); 49 | 50 | 51 | // BUFFER RELATED MEMBERS 52 | int m_buffer_size{2048}; 53 | std::unique_ptr[]> m_in_buffer; 54 | std::unique_ptr[]> m_out_buffer; 55 | std::vector> m_in_model, m_out_model; 56 | 57 | vector m_trajs; 58 | 59 | void operator()(audio_bundle input, audio_bundle output); 60 | void perform(audio_bundle input, audio_bundle output); 61 | 62 | 63 | argument latent_dim_arg {this, "latent_dim", "(int) Dimensionality of the latent vector.", true}; 64 | argument buffer_arg {this, "buffer_size", "(Optional) Record the signal once per buffer, 2048 by default."}; 65 | 66 | 67 | int latent_dim = 8; 68 | 69 | bool recorder_initialised = false; 70 | // BOOT STAMP 71 | message<> maxclass_setup{ 72 | this, "maxclass_setup", 73 | [this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 74 | return {}; 75 | }}; 76 | message<> dspsetup {this, "dspsetup", 77 | MIN_FUNCTION { 78 | m_one_over_samplerate = 1.0 / samplerate(); 79 | return {}; 80 | } 81 | }; 82 | 83 | min_dict training_data = {symbol(true)}; 84 | // min_dict coord_dict = {symbol(true)}; 85 | // min_dict buffer_dict = {symbol(true)}; 86 | min_dict latent_dict = {symbol(true)}; 87 | 88 | atoms latent_lengths; 89 | 90 | int record_count_batch = 0; 91 | int record_count = 0; 92 | 93 | 94 | attribute record_latents {this, "record", false, 95 | description {"Record input signal as latents vectors, once per buffer_size."}, 96 | setter{[this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 97 | if (args[0] && !record_latents){ 98 | // start a new trajectory 99 | m_in_model.clear(); 100 | m_in_buffer = std::make_unique[]>(latent_dim); 101 | 102 | vector> traj; 103 | for (int i(0); i < latent_dim; i++){ 104 | m_in_buffer[i].initialize(m_buffer_size); //4096 105 | m_in_model.push_back(std::make_unique(m_buffer_size)); 106 | 107 | vector latents; 108 | traj.push_back(latents); 109 | } 110 | m_trajs.push_back(new traj_info{traj, 0}); 111 | } else if (!args[0] && record_latents){ 112 | if (record_count_batch > 0 && !m_trajs.empty()){ 113 | // latent_lengths.push_back(record_count_batch); 114 | record_count_batch = 0; 115 | send_recordings(); 116 | } 117 | } 118 | return args; 119 | } 120 | }}; 121 | 122 | attribute record_limit {this, "record_limit", 2048, 123 | description {"Maximum number of latents to record; once reached, recording stops."},range{{1, 16384}}}; 124 | 125 | message<> clear_latents {this, "clear", "Clear recorded latents", 126 | MIN_FUNCTION { 127 | latent_dict.clear(); 128 | m_trajs.clear(); 129 | training_data.touch(); 130 | record_count_batch = 0; 131 | 132 | // latent_lengths.clear(); 133 | 134 | send_recordings(); 135 | return {}; 136 | } 137 | }; 138 | 139 | message<> bang {this, "bang", "Resend the latents dataset", 140 | MIN_FUNCTION { 141 | send_recordings(); 142 | return {}; 143 | } 144 | }; 145 | 146 | message<> record_latent {this, "poke", "Record just one latent vector from the signal", 147 | MIN_FUNCTION { 148 | if(record_latents){ 149 | cerr << "cannot poke while recording" << endl; 150 | return {}; 151 | } 152 | vector> traj; 153 | for (int i(0); i < latent_dim; i++){ 154 | vector latents; 155 | float latent = m_in_buffer[i].get_no_pop(); 156 | latents.push_back(latent); 157 | traj.push_back(latents); 158 | } 159 | m_trajs.push_back(new traj_info{traj, 1}); 160 | 161 | training_data.touch(); 162 | 163 | send_recordings(); 164 | return {}; 165 | } 166 | }; 167 | 168 | void send_recordings() { 169 | //cout << "sending latents" << endl; 170 | 171 | if (m_trajs.empty()){ 172 | traj_num_output.send(m_trajs.size()); 173 | latent_len_all_output.send(0); 174 | return; 175 | } 176 | latent_lengths.clear(); 177 | record_count = 0; 178 | int c = 0; 179 | for (int i(0); i < m_trajs.size(); i++){ 180 | if (!m_trajs[i]->traj[0].empty()){ 181 | min_dict traj_dict = {}; 182 | for (int z(0); z < latent_dim; z++){ 183 | 184 | atoms result(m_trajs[i]->traj[z].begin(), m_trajs[i]->traj[z].end()); 185 | symbol skey{z}; 186 | c74::max::dictionary_appendatoms(traj_dict.m_instance, skey, result.size(), &result[0]); 187 | 188 | } 189 | latent_dict[std::to_string(c+1)] = traj_dict; 190 | latent_lengths.push_back(m_trajs[i]->length); 191 | record_count += m_trajs[i]->length; 192 | c+= 1; 193 | } 194 | } 195 | training_data.touch(); 196 | 197 | traj_num_output.send(m_trajs.size()); 198 | latent_len_all_output.send(record_count); 199 | latent_len_output.send(latent_lengths); 200 | 201 | dataset_output.send("dictionary", training_data.name()); 202 | } 203 | 204 | private: 205 | double m_one_over_samplerate { 1.0 }; 206 | 207 | }; 208 | 209 | void nn_terrain::operator()(audio_bundle input, audio_bundle output) { 210 | perform(input, output); 211 | } 212 | 213 | 214 | 215 | void nn_terrain::perform(audio_bundle input, audio_bundle output) { 216 | if (!recorder_initialised){ 217 | return; 218 | } 219 | 220 | auto vec_size = input.frame_count();// 128 221 | // COPY INPUT TO CIRCULAR BUFFER 222 | for (int c(0); c < input.channel_count(); c++) { // 8 223 | auto in = input.samples(c); 224 | m_in_buffer[c].put(in, vec_size); 225 | } 226 | 227 | if (m_in_buffer[0].full()) { // BUFFER IS FULL //2048 228 | // TRANSFER MEMORY BETWEEN INPUT CIRCULAR BUFFER AND MODEL BUFFER 229 | if (record_latents){ 230 | auto& traj {m_trajs.back()}; 231 | for (int c(0); c < latent_dim; c++){ 232 | auto& traj_dim {traj->traj[c]}; 233 | float latent = m_in_buffer[c].get_no_pop(); 234 | traj_dim.push_back(latent); 235 | } 236 | traj->length += 1; 237 | record_count_batch += 1; 238 | if (record_count_batch >= record_limit){ 239 | record_count_batch = 0; 240 | record_latents = false; 241 | send_recordings(); 242 | cwarn << "recording latents stopped because maximum length reached" << endl; 243 | } 244 | } 245 | for (int c(0); c < input.channel_count(); c++) { // 8 246 | m_in_buffer[c].get(m_in_model[c].get(), m_buffer_size); 247 | } 248 | 249 | } 250 | } 251 | 252 | 253 | nn_terrain::nn_terrain(const atoms &args){ 254 | 255 | training_data["latents"] = latent_dict; 256 | // CHECK ARGUMENTS 257 | if (!args.size()) { 258 | return; 259 | } 260 | if (args.size() >= 1) { // TWO ARGUMENT IS GIVEN: 261 | // a_type 1: int, a_type 2: float, a_type 3: symbol 262 | if (args[0].a_type == 1) { // (int) 263 | if (int(args[0]) < 1 || int(args[0]) > 1024) { 264 | cerr << "error: the latent dimension should be in the range [1, 1024]" << endl; 265 | return; 266 | } 267 | latent_dim = int(args[0]); 268 | } else { 269 | cerr << "error: the argument should be int (number of latent dimensions) in the range [1, 1024]" << endl; 270 | } 271 | } 272 | if (args.size() >= 2) { // TWO ARGUMENTS ARE GIVEN: 273 | m_buffer_size = power_ceil(int(args[1])); 274 | } 275 | if (args.size() >= 3) { // THREE ARGUMENTS ARE GIVEN 276 | cerr << "error: maximum one argument" << endl; 277 | return; 278 | } 279 | 280 | m_in_buffer = std::make_unique[]>(latent_dim); 281 | for (int i(0); i < latent_dim; i++) { 282 | std::string input_label = "(signal) latent codes input " + std::to_string(i); 283 | 284 | m_inlets.push_back(std::make_unique>(this, input_label, "float")); 285 | m_in_buffer[i].initialize(m_buffer_size); //4096 286 | m_in_model.push_back(std::make_unique(m_buffer_size)); 287 | } 288 | 289 | recorder_initialised = true; 290 | return; 291 | } 292 | 293 | nn_terrain::~nn_terrain() { 294 | 295 | } 296 | MIN_EXTERNAL(nn_terrain); 297 | 298 | 299 | 300 | 301 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain_tilde/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | # Copyright 2018 The Min-DevKit Authors. All rights reserved. 2 | # Use of this source code is governed by the MIT License found in the License.md file. 3 | 4 | cmake_minimum_required(VERSION 3.0) 5 | 6 | set ( MSVC_COMPILER_NAME "MSVC" ) 7 | if (${CMAKE_CXX_COMPILER_ID} STREQUAL ${MSVC_COMPILER_NAME}) 8 | string (SUBSTRING ${CMAKE_CXX_COMPILER_VERSION} 0 4 MSVC_VERSION_SHORT) 9 | string (SUBSTRING ${CMAKE_CXX_COMPILER_VERSION} 0 2 MSVC_VERSION_MAJOR) 10 | string (SUBSTRING ${CMAKE_CXX_COMPILER_VERSION} 3 1 MSVC_VERSION_MINOR) 11 | 12 | if (${MSVC_VERSION_MAJOR} VERSION_LESS 19 OR ${MSVC_VERSION_MAJOR} MATCHES 19 AND ${MSVC_VERSION_MINOR} VERSION_LESS 1) 13 | # message(STATUS "Visual Studio ${MSVC_VERSION_SHORT} detected. Visual Studio 17 (19.1) or greater is required for UI objects.") 14 | message(STATUS "Visual Studio 17 or greater is required for UI objects.") 15 | message(STATUS "SKIPPING!") 16 | return () 17 | endif () 18 | endif () 19 | 20 | set(C74_MIN_API_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../min-api) 21 | include(${C74_MIN_API_DIR}/script/min-pretarget.cmake) 22 | 23 | if (APPLE) 24 | set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum OS X deployment version" FORCE) 25 | endif() 26 | 27 | ############################################################# 28 | # MAX EXTERNAL 29 | ############################################################# 30 | 31 | 32 | include_directories( 33 | "${C74_INCLUDES}" 34 | ) 35 | find_package(Torch REQUIRED) 36 | 37 | set( 38 | SOURCE_FILES 39 | nn.terrain_tilde.cpp 40 | "../shared/circular_buffer.h" 41 | "../shared/utils.h" 42 | "../shared/min_path.h" 43 | "../shared/min_dictionary.h" 44 | ) 45 | 46 | if (UNIX) 47 | execute_process( 48 | COMMAND git describe --tags 49 | WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR} 50 | OUTPUT_VARIABLE VERSION 51 | OUTPUT_STRIP_TRAILING_WHITESPACE 52 | ) 53 | message(STATUS ${VERSION}) 54 | add_definitions(-DVERSION="${VERSION}") 55 | endif() 56 | 57 | add_library( 58 | ${PROJECT_NAME} 59 | MODULE 60 | ${SOURCE_FILES} 61 | "../shared/circular_buffer.h" 62 | "../shared/utils.h" 63 | "../shared/min_path.h" 64 | "../shared/min_dictionary.h" 65 | ) 66 | target_link_libraries(${PROJECT_NAME} PRIVATE backend) 67 | 68 | if (APPLE) 69 | target_link_libraries(${PROJECT_NAME} PUBLIC "$") 70 | set_target_properties(${PROJECT_NAME} PROPERTIES 71 | BUILD_WITH_INSTALL_RPATH FALSE 72 | LINK_FLAGS "-Wl,-F/Library/Frameworks,-rpath,@loader_path/" 73 | ) 74 | # I can't figure out how to set the Bundle Identifier nicely in for max-sdk-base, so I'm just gonna do this thing here... 75 | set(AUTHOR_DOMAIN "com.jasperzheng") 76 | set(BUNDLE_IDENTIFIER "nn-terrain-") 77 | endif() 78 | 79 | include(${C74_MIN_API_DIR}/script/min-posttarget.cmake) 80 | 81 | if (APPLE) # COPY TORCH DYLIB IN THE LOADER FOLDER 82 | add_custom_command( 83 | TARGET ${PROJECT_NAME} 84 | POST_BUILD 85 | COMMAND cp "${TORCH_INSTALL_PREFIX}/lib/*.dylib" "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${${PROJECT_NAME}_EXTERN_OUTPUT_NAME}.mxo/Contents/MacOS/" 86 | COMMENT "Copy Torch Libraries" 87 | ) 88 | 89 | if (CMAKE_OSX_ARCHITECTURES STREQUAL "arm64") 90 | add_custom_command( 91 | TARGET ${PROJECT_NAME} 92 | POST_BUILD 93 | COMMAND "codesign" "--force" "--deep" "-s" "-" "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${${PROJECT_NAME}_EXTERN_OUTPUT_NAME}.mxo" 94 | COMMENT "Codesign external" 95 | ) 96 | endif() 97 | endif() 98 | 99 | if (MSVC) # COPY TORCH DLL IN THE LOADER FOLDER 100 | add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD 101 | COMMAND ${CMAKE_COMMAND} -E copy_directory "${TORCH_INSTALL_PREFIX}/lib/" ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/ 102 | ) 103 | endif() 104 | 105 | if (MSVC) 106 | set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 20) 107 | endif() 108 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/nn.terrain_tilde/nn.terrain_tilde.cpp: -------------------------------------------------------------------------------- 1 | #include "../../../backend/backend.h" 2 | #include "../shared/circular_buffer.h" 3 | 4 | #include "c74_min.h" 5 | 6 | #include 7 | #include 8 | #include 9 | 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | 16 | #include "../shared/min_path.h" 17 | #include "../shared/min_dictionary.h" 18 | 19 | #include "../shared/utils.h" 20 | 21 | #include 22 | 23 | #ifndef VERSION 24 | #define VERSION "UNDEFINED" 25 | #endif 26 | 27 | #define CPU torch::kCPU 28 | #define CUDA torch::kCUDA 29 | #define MPS torch::kMPS 30 | 31 | #define None at::indexing::None 32 | #define Slice torch::indexing::Slice 33 | 34 | using namespace c74::min; 35 | using namespace c74::min::ui; 36 | 37 | std::string min_devkit_path() { 38 | #ifdef WIN_VERSION 39 | char pathstr[4096]; 40 | HMODULE hm = nullptr; 41 | if (!GetModuleHandleExA( 42 | GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, (LPCSTR)&min_devkit_path, &hm)) { 43 | int ret = GetLastError(); 44 | fprintf(stderr, "GetModuleHandle() returned %d\n", ret); 45 | } 46 | GetModuleFileNameA(hm, pathstr, sizeof(pathstr)); 47 | // path now is the path to this external's binary, including the binary filename. 48 | auto filename = strrchr(pathstr, '\\'); 49 | if (filename) 50 | *filename = 0; 51 | auto externals = strrchr(pathstr, '\\'); 52 | if (externals) 53 | *externals = 0; 54 | path p{ pathstr }; // convert to Max path 55 | return p; 56 | #endif // WIN_VERSION 57 | 58 | #ifdef MAC_VERSION 59 | CFBundleRef this_bundle = CFBundleGetBundleWithIdentifier(CFSTR("com.jasperzheng.nn-terrain-")); 60 | CFURLRef this_url = CFBundleCopyExecutableURL(this_bundle); 61 | char this_path[4096]; 62 | CFURLGetFileSystemRepresentation(this_url, true, reinterpret_cast(this_path), 4096); 63 | string this_path_str{ this_path }; 64 | CFRelease(this_url); 65 | // we now have a path like this: 66 | // /Users/tim/Materials/min-devkit/externals/min.project.mxo/Contents/MacOS/min.project" 67 | // so we need to chop off 5 slashes from the end 68 | auto iter = this_path_str.find("/externals/nn.terrain~.mxo/Contents/MacOS/nn.terrain~"); 69 | this_path_str.erase(iter, strlen("/externals/nn.terrain~.mxo/Contents/MacOS/nn.terrain~")); 70 | return this_path_str; 71 | #endif // MAC_VERSION 72 | } 73 | 74 | 75 | class CustomDataset : public torch::data::datasets::Dataset { 76 | private: 77 | torch::Tensor data, targets; 78 | 79 | public: 80 | // Constructor 81 | CustomDataset(torch::Tensor data, torch::Tensor targets) 82 | : data(data), targets(targets) {} 83 | 84 | // Get size of dataset 85 | torch::optional size() const override { 86 | return data.size(0); 87 | } 88 | 89 | // Get a single data sample 90 | torch::data::Example<> get(size_t index) override { 91 | return {data[index], targets[index]}; 92 | } 93 | }; 94 | 95 | class nn_terrain : public object, public vector_operator<> { 96 | 97 | public: 98 | MIN_DESCRIPTION{"Latent terrain priors for neural audio autoencoders, for generating latent vectors"}; 99 | MIN_TAGS { "" }; 100 | MIN_AUTHOR{"Jasper Shuoyang Zheng"}; 101 | MIN_RELATED { "nn~, nn.terrain.gui, nn.terrain.encode, nn.terrain.record" }; 102 | 103 | 104 | // INLETS OUTLETS 105 | std::vector>> m_outlets; 106 | std::vector>> m_inlets; 107 | 108 | nn_terrain(const atoms &args = {}); 109 | ~nn_terrain(); 110 | 111 | std::vector> buffers; 112 | 113 | 114 | // BACKEND 115 | std::unique_ptr cppn_model; 116 | 117 | bool cppn_init = false; 118 | c74::min::path t_path; 119 | 120 | std::string d_method{"decode"}; 121 | std::string e_method{"encode"}; 122 | 123 | // std::vector settable_attributes; 124 | // bool has_settable_attribute(std::string attribute); 125 | 126 | // c74::min::path m_path; 127 | int m_in_dim{2}, m_in_ratio{2048}, m_out_dim{8}, m_out_ratio{1}, m_cmax{128}, m_mapping_size{256}; 128 | float m_gauss_scale{0.2}; 129 | 130 | 131 | // BUFFER RELATED MEMBERS 132 | int m_buffer_size{2048}; 133 | std::unique_ptr[]> m_in_buffer; 134 | std::unique_ptr[]> m_out_buffer; 135 | std::vector> m_in_model, m_out_model; 136 | std::vector> m_latent_out; 137 | 138 | // AUDIO PERFORM 139 | bool m_use_thread{true}, m_should_stop_perform_thread{false}; 140 | std::unique_ptr m_compute_thread{nullptr}; 141 | std::binary_semaphore m_data_available_lock{0}; 142 | std::binary_semaphore m_result_available_lock{1}; 143 | 144 | // void display_args(); 145 | atoms create_dataloader(); 146 | void operator()(audio_bundle input, audio_bundle output); 147 | void perform(audio_bundle input, audio_bundle output); 148 | void cppn_infer(float *, std::vector); 149 | // atoms freeze_terrain(int width_x, int height_y, string terrain_name, int stride); 150 | 151 | at::Tensor sampled_tensor_in; 152 | min_dict sampled_dict{symbol(true)}; 153 | min_dict terrain_dict{symbol(true)}; 154 | string sampled_str; 155 | float x_lo_prev{0.0}, x_hi_prev{0.0}, y_lo_prev{0.0}, y_hi_prev{0.0}, stride_prev{0.0}; 156 | int x_res_prev{0}, y_res_prev{0}, plot_resolution_prev{0}, c_prev{0}; 157 | void sample_interval(float x_lo, float x_hi, int x_res, 158 | float y_lo, float y_hi, int y_res, int c); 159 | void create_sampled_tensor_in(); 160 | 161 | // void freeze_and_compile_terrain(int width_x, int height_y, string terrain_name); 162 | 163 | // float x{0.0f}, y{0.0f}, p{0.0f}; 164 | 165 | //caution 166 | // std::unique_ptr latent_codes = std::make_unique(8); 167 | 168 | float twoPi = 4 * acos(0.0); 169 | 170 | int inputPhase{0}; 171 | int lastIdx{0}; 172 | 173 | bool time_to_refresh = false; 174 | 175 | 176 | argument control_dim_arg {this, "control_dim", "Dimensionality of the input space.", true}; 177 | argument latent_dim_arg {this, "latent_dim", "Dimensionality of the autoencoder's latent space.", true}; 178 | argument t_gauss_scale_arg {this, "gauss_scale", "(Optional) Gaussian scale of the neural network. A float value between 0-1 is suggested, by default is 0.2. A higher Gaussian scale leads to a noisy terrain. If the Gaussian scale is 0, the Fourier feature mapping layer will be removed, resulting in a smooth (low frequency) terrain."}; 179 | argument t_cmax_arg {this, "network_channel", "(Optional) Number of channels in the neural network's fully connected layers, by default is 128"}; 180 | argument mapping_size_arg {this, "feature_size", "(Optional) Size of the random Fourier feature mapping, higher feature size is suggested when using high dimensional control space, by default is 256."}; 181 | argument buffer_arg {this, "buffer_size", "(Optional) Size of the internal buffer (can't be lower than the decoder method's ratio)."}; 182 | 183 | argument model_path_arg {this, "checkpoint_path", "If only one argument is given, it should be the path to a model checkpoint."}; 184 | 185 | attribute enable_cppn {this, "enable_terrain", false, title{": Enable Terrain"}, 186 | description{"Enable / disable cppn computation"}, 187 | setter{ 188 | [this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 189 | if (cppn_init && args[0]){ 190 | return {true}; 191 | } else { 192 | return {false}; 193 | } 194 | return args; 195 | } 196 | } 197 | }; 198 | 199 | message<> m_sample_interval {this, "plot_interval", "Sample the terrain across a closed interval to plot into the GUI, only 2D interval dimension is supported at the moment.

Args:
1D [not implemented]: lo, hi, resolution;
2D: x_lo, x_hi, x_resolution, y_lo, y_hi, y_resolution, color_channel (optional, default 1)", 200 | MIN_FUNCTION { 201 | //args: 202 | // x_lo, x_hi, y_lo, y_hi, stride, latent_clamp_min, latent_clamp_max 203 | // lo, hi, stride, latent_clamp_min, latent_clamp_max 204 | if (args.size() == 3) { 205 | // cout << args[0] << " " << args[1] << " " << args[2] << " " << args[3] << " " << args[4] << endl; 206 | cerr << "plot_interval: 1D sampling not implemented" << endl; 207 | return {}; 208 | } else if (args.size() == 6) { 209 | sample_interval(args[0], args[1], args[2], args[3], args[4], args[5], 1); 210 | return {}; 211 | } else if (args.size() == 7) { 212 | sample_interval(args[0], args[1], args[2], args[3], args[4], args[5], args[6]); 213 | return {}; 214 | } 215 | cerr << "plot_interval: wrong number of arguments" << endl; 216 | return {}; 217 | } 218 | }; 219 | 220 | attribute plot_resolution {this, "plot_resolution", 1, title{": Plot Resolution"}, description {"(int) 1~16"},range{{1, 16}}}; 221 | 222 | attribute lr{ this, "training_lr", 0.001, title{": Training Learning Rate"}, description{"learning rate when training the nn"}, range{{0.0001, 0.1}} }; 223 | attribute batch_size{ this, "training_batchsize", 32, title{": Training Batch Size"}, description{"batch size when training the nn"}, range{{4, 128}} }; 224 | attribute worker{ this, "training_worker", 2, title{": Training Workers"}, description{"number of worker in dataloader"}, range{{0, 4}} }; 225 | 226 | attribute epoch_per_cycle {this, "training_epoch", 10, title{": Training Epoch"}, description {"(int) train the model for this many epochs when one train message is received"}}; 227 | 228 | message<> maxclass_setup{ 229 | this, "maxclass_setup", [this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 230 | cout << "torch version: " << TORCH_VERSION << endl; 231 | return {}; 232 | } 233 | }; 234 | 235 | message<> dspsetup {this, "dspsetup", 236 | MIN_FUNCTION { 237 | m_one_over_samplerate = 1.0 / samplerate(); 238 | return {}; 239 | } 240 | }; 241 | 242 | at::Tensor training_coords; 243 | at::Tensor training_latents; 244 | 245 | min_dict training_data = {symbol(true)}; 246 | min_dict coord_dict = {symbol(true)}; 247 | min_dict latent_dict = {symbol(true)}; 248 | int coord_len = 0; 249 | int latent_len = 0; 250 | 251 | bool dataset_updated = true; 252 | bool dataloader_ready = false; 253 | 254 | message<> dictionary { this, "dictionary", 255 | "Use a dictionary to gather training data for the terrain", 256 | MIN_FUNCTION { 257 | c74::max::t_symbol **keys = NULL; 258 | long numkeys = 0; 259 | 260 | min_dict d = {args[0]}; 261 | c74::max::dictionary_getkeys(d.m_instance, &numkeys, &keys); 262 | 263 | for (int i = 0; i < numkeys; i++){ 264 | string key_str = std::string(keys[i]->s_name); 265 | atom d_data = d[key_str]; 266 | min_dict d_data_dict = {d_data}; 267 | atom count = c74::max::dictionary_getentrycount(d_data_dict.m_instance); 268 | 269 | if (key_str == "coordinates"){ 270 | coord_len = count; 271 | if (coord_len == 0) { 272 | coord_dict.clear(); 273 | } else { 274 | coord_dict = d_data_dict; 275 | } 276 | dataset_updated = true; 277 | } else if (key_str == "latents"){ 278 | latent_len = count; 279 | if (latent_len==0){ 280 | latent_dict.clear(); 281 | } else { 282 | latent_dict = d_data_dict; 283 | } 284 | dataset_updated = true; 285 | } else { 286 | cerr << "unknown key, or some dictionaries are empty " << key_str << endl; 287 | } 288 | } 289 | // freeing memory for keys 290 | if (keys){ 291 | c74::max::dictionary_freekeys(d.m_instance, numkeys, keys); 292 | } 293 | training_data.touch(); 294 | m_outlets[m_outlets.size()-1]->send("dataset","dictionary", training_data.name()); 295 | return {}; 296 | } 297 | }; 298 | 299 | 300 | std::unique_ptr>>, torch::data::samplers::RandomSampler>> data_loader; 301 | 302 | std::unique_ptr optimizer; 303 | 304 | message<> model_summary {this, "model_summary", "Model summary", 305 | MIN_FUNCTION { 306 | min_dict net_args_dict {symbol(true)}; 307 | 308 | net_args_dict["input_dimension"] = m_in_dim; 309 | net_args_dict["output_dimension"] = m_out_dim; 310 | net_args_dict["gaussian_scale"] = m_gauss_scale; 311 | net_args_dict["number_of_channels"] = m_cmax; 312 | net_args_dict["features_size"] = m_mapping_size; 313 | net_args_dict["number_of_parameters"] = static_cast(cppn_model->m_model->parameters().size()); 314 | // net_args_dict["saving_path"] = (std::filesystem::path(std::to_string(external_path)) / (std::to_string(saving_name)+".pt")).string(); 315 | 316 | m_outlets[m_outlets.size()-1]->send("summary", "dictionary", net_args_dict.name()); 317 | return {}; 318 | } 319 | }; 320 | 321 | min_dict dataset_info_dict {symbol(true)}; 322 | min_dict dataset_co_dict {symbol(true)}; 323 | min_dict dataset_lt_dict {symbol(true)}; 324 | atoms dataset_summary; 325 | message<> dataset_summary_m {this, "dataset_summary", "Training pairs summary", 326 | MIN_FUNCTION { 327 | try{ 328 | if (dataset_updated){ 329 | dataset_summary = create_dataloader(); 330 | } 331 | 332 | if (dataset_summary[0] == 0){ 333 | cerr << "no data loaded" << endl; 334 | return {}; 335 | } 336 | symbol shape_key{"shape"}; 337 | symbol range_key{"range"}; 338 | 339 | dataset_info_dict["dataset_len"] = static_cast(dataset_summary[0]); 340 | 341 | atoms shape_co = {dataset_summary[3], dataset_summary[4]}; 342 | c74::max::dictionary_appendatoms(dataset_co_dict.m_instance, shape_key, shape_co.size(), &shape_co[0]); 343 | atoms range_co = {dataset_summary[5], dataset_summary[6]}; 344 | c74::max::dictionary_appendatoms(dataset_co_dict.m_instance, range_key, range_co.size(), &range_co[0]); 345 | 346 | atoms shape_lt = {dataset_summary[7], dataset_summary[8]}; 347 | c74::max::dictionary_appendatoms(dataset_lt_dict.m_instance, shape_key, shape_lt.size(), &shape_lt[0]); 348 | atoms range_lt = {dataset_summary[9], dataset_summary[10]}; 349 | c74::max::dictionary_appendatoms(dataset_lt_dict.m_instance, range_key, range_lt.size(), &range_lt[0]); 350 | 351 | dataset_info_dict.touch(); 352 | m_outlets[m_outlets.size()-1]->send("summary", "dictionary", dataset_info_dict.name()); 353 | } catch (const std::exception &e) { 354 | cerr << e.what() << endl; 355 | } 356 | return {}; 357 | } 358 | }; 359 | 360 | attribute cppn_gpu{this, "gpu", false, 361 | description{"Enable / disable gpu usage when available"}, 362 | setter{ 363 | [this](const c74::min::atoms &args, const int inlet) -> c74::min::atoms { 364 | if (cppn_init){ 365 | if (cppn_model->is_loaded()){ 366 | cppn_model->use_gpu(bool(args[0])); 367 | optimizer = std::make_unique(cppn_model->m_model->parameters(), static_cast(lr)); 368 | 369 | if (total_epochs>0){ 370 | cwarn << "Changing device after any training will re-initialise the optimizer, be careful." << endl; 371 | } 372 | if (bool(args[0])){ 373 | cwarn << "The terrain model is suggested to be operated on the CPU because it's a very small neural network without any convolutional layer." << endl; 374 | } 375 | // cout << "cppn gpu: " << bool(args[0]) << endl; 376 | } 377 | } 378 | return args; 379 | } 380 | } 381 | }; 382 | 383 | 384 | 385 | int total_epochs = 0; 386 | message<> train_cycle {this, "train", "Train the terrain model with one batch", 387 | MIN_FUNCTION { 388 | if (dataset_updated){ 389 | try{ 390 | dataset_summary = create_dataloader(); 391 | if (dataset_summary[0]==0){ 392 | cerr << "no data loaded" << endl; 393 | return {}; 394 | } 395 | } catch (const std::exception &e) { 396 | cerr << e.what() << endl; 397 | return {}; 398 | } 399 | } 400 | // return {}; 401 | // cout << "Training the terrain model with " << epoch_per_cycle <<" batch" << endl; 402 | 403 | cppn_model->m_model->train(); 404 | 405 | size_t batch_count = 0; 406 | float loss_log = 0.0; 407 | try { 408 | for (int i(0); i < epoch_per_cycle; i++){ 409 | for (auto& batch : *data_loader) { 410 | 411 | optimizer->zero_grad(); 412 | torch::Tensor batch_data = batch.data.to(cppn_model->m_device); 413 | torch::Tensor prediction = cppn_model->m_model->forward(batch_data); 414 | 415 | torch::Tensor batch_target = batch.target.to(cppn_model->m_device); 416 | 417 | torch::Tensor loss = torch::mse_loss(prediction, batch_target).to(cppn_model->m_device); 418 | loss.backward(); 419 | 420 | optimizer->step(); 421 | 422 | loss_log+=loss.to(torch::kCPU).item(); 423 | 424 | batch_count++; 425 | } 426 | } 427 | } catch (const std::exception &e) { 428 | cerr << e.what() << endl; 429 | } 430 | if (batch_count == 0){ 431 | cerr << "error" << endl; 432 | cppn_model->m_model->eval(); 433 | return {}; 434 | } 435 | total_epochs += epoch_per_cycle; 436 | loss_log /= batch_count; 437 | cppn_model->m_model->eval(); 438 | 439 | m_outlets[m_outlets.size()-1]->send("loss", loss_log); 440 | m_outlets[m_outlets.size()-1]->send("epoch", total_epochs); 441 | return {}; 442 | } 443 | }; 444 | attribute external_path{ this, "saving_path", "none", title{": Saving Path"}, setter{ MIN_FUNCTION{ 445 | if (args[0] == "none") { 446 | symbol min_path_this = min_devkit_path(); 447 | return { min_path_this }; 448 | } else { 449 | try { 450 | min_path min_path_this = min_path(static_cast(args[0])); 451 | return { args[0] }; 452 | } catch (...) { 453 | symbol min_path_this = min_devkit_path(); 454 | cwarn << "Error reading folder, set to default" << endl; 455 | return { min_path_this }; 456 | } 457 | } 458 | } } 459 | }; 460 | // std::string external_path; 461 | 462 | attribute saving_name{ this, "saving_name", "UntitledTerrain", title{": Saving Name"}, setter{ MIN_FUNCTION{ 463 | std::string str = std::string(args[0]); 464 | str.erase(std::remove(str.begin(), str.end(), ' '), str.end()); // remove spaces 465 | if (str.empty()) { 466 | return { "UntitledTerrain" }; 467 | } else { 468 | return { args[0] }; 469 | } 470 | } } 471 | }; 472 | 473 | message<> save_cppn {this, "checkpoint", "Checkpoint and save the terrain model", 474 | MIN_FUNCTION { 475 | if (!cppn_init){ 476 | return {}; 477 | } 478 | std::string saved_name = std::to_string(saving_name); 479 | if (saved_name.substr(saved_name.length() - 3) != ".pt"){ 480 | saved_name = saved_name + ".pt"; 481 | } 482 | try { 483 | if(cppn_model->save(std::to_string(external_path), saved_name, m_in_dim, m_out_dim, m_cmax, m_gauss_scale, m_mapping_size)){ 484 | cout << saved_name << " saved to: " << std::to_string(external_path) << endl; 485 | } 486 | } catch (const std::exception &e) { 487 | cerr << "error saving the model:" << e.what() << endl; 488 | } 489 | return {}; 490 | } 491 | }; 492 | 493 | private: 494 | // double m_playback_position { 0.0 }; // normalized range 495 | // size_t m_record_position { 0 }; // native range 496 | double m_one_over_samplerate { 1.0 }; 497 | 498 | }; 499 | 500 | atoms nn_terrain::create_dataloader() { 501 | int coord_count = static_cast(c74::max::dictionary_getentrycount(coord_dict.m_instance)); 502 | int latent_count = static_cast(c74::max::dictionary_getentrycount(latent_dict.m_instance)); 503 | 504 | if (coord_count==0 || latent_count==0){ 505 | cerr << "coordinates or latents not set" << endl; 506 | return {{0}}; 507 | } 508 | 509 | int dataset_len = 0; 510 | int traj_len = 0; 511 | 512 | if (latent_count > coord_count){ 513 | cwarn << "latent trejectories "< audio trejectories "< latent trejectories"< coord_trim; 523 | std::vector latent_trim; 524 | 525 | for (int i(0); i < traj_len; i++){ 526 | 527 | std::vector coords; 528 | std::vector latents; 529 | 530 | int point_count = 0; 531 | atom c_data = coord_dict[std::to_string(i+1)]; 532 | min_dict c_seg_dict = {c_data}; 533 | atoms seg_c_atoms_tp = c_seg_dict["0"]; 534 | 535 | atom l_data = latent_dict[std::to_string(i+1)]; 536 | min_dict l_seg_dict = {l_data}; 537 | atoms seg_l_atoms_tp = l_seg_dict["0"]; 538 | 539 | point_count = seg_c_atoms_tp.size() > seg_l_atoms_tp.size() ? seg_l_atoms_tp.size() : seg_c_atoms_tp.size(); 540 | 541 | int coord_dim = static_cast(c74::max::dictionary_getentrycount(c_seg_dict.m_instance)); 542 | int latent_dim_here = static_cast(c74::max::dictionary_getentrycount(l_seg_dict.m_instance)); 543 | 544 | if (m_out_dim != latent_dim_here){ 545 | cerr << "latents dimension " << latent_dim_here << " doesn't match latent_dim" << endl; 546 | return {{0}}; 547 | } 548 | for (int k(0); k < m_in_dim; k++){ 549 | atoms seg_c_atoms; 550 | try { 551 | seg_c_atoms = c_seg_dict[std::to_string(k)]; 552 | } catch (const std::exception &e) { 553 | cerr << "error getting coordinates from dictionary: " << e.what() << endl; 554 | return {{0}}; 555 | } 556 | for (int j(0); j < point_count; j++){ 557 | coords.push_back(static_cast(seg_c_atoms[j])); 558 | } 559 | } 560 | for (int k(0); k < m_out_dim; k++){ 561 | atoms seg_l_atoms; 562 | try { 563 | seg_l_atoms = l_seg_dict[std::to_string(k)]; 564 | } catch (const std::exception &e) { 565 | cerr << "error getting latents from dictionary: " << e.what() << endl; 566 | return {{0}}; 567 | } 568 | for (int j(0); j < point_count; j++){ 569 | latents.push_back(static_cast(seg_l_atoms[j])); 570 | } 571 | } 572 | coord_trim.push_back(torch::from_blob(coords.data(), {m_in_dim, point_count}, torch::kFloat).clone()); 573 | latent_trim.push_back(torch::from_blob(latents.data(), {m_out_dim, point_count}, torch::kFloat).clone()); 574 | dataset_len += point_count; 575 | } 576 | 577 | 578 | if (coord_trim.size() == 0 || latent_trim.size() == 0){ 579 | cerr << "error converting to tensor" << endl; 580 | return {{0}}; 581 | } 582 | training_coords = torch::cat(coord_trim, 1).permute({1,0}); 583 | training_latents = torch::cat(latent_trim, 1).permute({1,0}); 584 | 585 | // for (int i(0); i < training_coords.ndimension(); i++){ 586 | // cout << "coords dim " << i << ": " << training_coords.size(i) << endl; 587 | // } 588 | // for (int i(0); i < training_latents.ndimension(); i++){ 589 | // cout << "latents dim " << i << ": " << training_latents.size(i) << endl; 590 | // } 591 | 592 | 593 | std::vector traj_info = { 594 | static_cast(training_coords.size(0)), 595 | static_cast(training_coords.size(1)) 596 | }; 597 | std::vector traj_range_info = { 598 | training_coords.min().item(), 599 | training_coords.max().item() 600 | }; 601 | std::vector lat_info = { 602 | static_cast(training_latents.size(0)), 603 | static_cast(training_latents.size(1)) 604 | }; 605 | std::vector lat_range_info = { 606 | training_latents.min().item(), 607 | training_latents.max().item() 608 | }; 609 | 610 | auto dataset = CustomDataset(training_coords,training_latents).map(torch::data::transforms::Stack<>()); 611 | 612 | m_outlets[m_outlets.size()-1]->send("dataset_length", static_cast(dataset.size().value())); 613 | data_loader = torch::data::make_data_loader(std::move(dataset), 614 | torch::data::DataLoaderOptions().batch_size(batch_size).workers(worker).drop_last(false)); 615 | dataloader_ready = true; 616 | dataset_updated = false; 617 | 618 | return {{ 619 | dataset_len, 620 | coord_count, 621 | latent_count, 622 | traj_info[0], // trajs.size(0) 623 | traj_info[1], // trajs.size(1) 624 | traj_range_info[0], // traj_range_info.min() 625 | traj_range_info[1], // traj_range_info.max() 626 | lat_info[0], // latents.size(0) 627 | lat_info[1], // latents.size(1) 628 | lat_range_info[0], // lat_range_info.min() 629 | lat_range_info[1] // lat_range_info.max() 630 | }}; 631 | } 632 | 633 | void model_perform_loop(nn_terrain *nn_instance) { 634 | std::vector in_model, out_model; 635 | 636 | for (auto &ptr : nn_instance->m_in_model) 637 | in_model.push_back(ptr.get()); 638 | 639 | for (auto &ptr : nn_instance->m_out_model) 640 | out_model.push_back(ptr.get()); 641 | 642 | while (!nn_instance->m_should_stop_perform_thread) { 643 | if (nn_instance->m_data_available_lock.try_acquire_for(std::chrono::milliseconds(200))) { 644 | // nn_instance->m_model->perform(in_model, out_model, 645 | // nn_instance->m_buffer_size, 646 | // nn_instance->d_method, 1); 647 | nn_instance->m_result_available_lock.release(); 648 | } 649 | } 650 | } 651 | 652 | 653 | void fill_with_zero(audio_bundle output) { 654 | for (int c(0); c < output.channel_count(); c++) { 655 | auto out = output.samples(c); 656 | for (int i(0); i < output.frame_count(); i++) { 657 | out[i] = 0.; 658 | } 659 | } 660 | } 661 | 662 | void nn_terrain::operator()(audio_bundle input, audio_bundle output) { 663 | // CHECK IF MODEL IS LOADED AND ENABLED 664 | if (!cppn_init || !enable_cppn) { 665 | fill_with_zero(output); 666 | return; 667 | } 668 | perform(input, output); 669 | } 670 | 671 | void nn_terrain::sample_interval(float x_lo, float x_hi, int x_res, float y_lo, float y_hi, int y_res, int c){ 672 | if (x_lo == x_lo_prev && x_hi == x_hi_prev && y_lo == y_lo_prev && y_hi == y_hi_prev && y_res == y_res_prev && x_res == x_res_prev && plot_resolution == plot_resolution_prev && c == c_prev){ 673 | // do nothing 674 | } else { 675 | x_lo_prev = x_lo; 676 | x_hi_prev = x_hi; 677 | y_lo_prev = y_lo; 678 | y_hi_prev = y_hi; 679 | x_res_prev = x_res; 680 | y_res_prev = y_res; 681 | plot_resolution_prev = plot_resolution; 682 | c_prev = c; 683 | create_sampled_tensor_in(); 684 | } 685 | if (sampled_tensor_in.numel() == 0){ 686 | return; 687 | } 688 | 689 | terrain_dict.clear(); 690 | std::unique_lock model_lock(cppn_model->m_model_mutex); 691 | if (c == 1){ 692 | try { 693 | for (int i(0); i < sampled_tensor_in.size(0); i++){ // y 694 | at::Tensor tensor_in = sampled_tensor_in.index({i}).to(cppn_model->m_device); 695 | at::Tensor tensor_out = cppn_model->m_model->forward(tensor_in).to(torch::kFloat).to(torch::kCPU); 696 | tensor_out = tensor_out.index({Slice(None), 0}); 697 | auto ten_out_ptr = tensor_out.contiguous().data_ptr(); 698 | atoms result(ten_out_ptr, ten_out_ptr + sampled_tensor_in.size(1)); 699 | 700 | symbol skey{i}; 701 | c74::max::dictionary_appendatoms(terrain_dict.m_instance, skey, result.size(), &result[0]); 702 | } 703 | } catch (const std::exception &e) { 704 | cerr << e.what() << endl; 705 | } 706 | } else if (c == 4) { 707 | try { 708 | 709 | for (int i(0); i < sampled_tensor_in.size(0); i++){ // y 710 | min_dict row_dict = {}; 711 | at::Tensor tensor_in = sampled_tensor_in.index({i}).to(cppn_model->m_device); 712 | at::Tensor tensor_out = cppn_model->m_model->forward(tensor_in).to(torch::kFloat).to(torch::kCPU); 713 | 714 | for (int j(0); j < 4; j++){ 715 | 716 | at::Tensor a_out = tensor_out.index({Slice(None), j}).clone().contiguous(); 717 | 718 | auto ten_out_ptr = a_out.data_ptr(); 719 | atoms result(ten_out_ptr, ten_out_ptr + sampled_tensor_in.size(1)); 720 | 721 | symbol skey{j}; 722 | c74::max::dictionary_appendatoms(row_dict.m_instance, skey, result.size(), &result[0]); 723 | } 724 | terrain_dict[std::to_string(i)] = row_dict; 725 | } 726 | } catch (const std::exception &e) { 727 | cerr << e.what() << endl; 728 | } 729 | } 730 | model_lock.unlock(); 731 | 732 | atom count = c74::max::dictionary_getentrycount(terrain_dict.m_instance); 733 | if (count == 0){ 734 | return; 735 | } 736 | sampled_dict.touch(); 737 | m_outlets[m_outlets.size()-2]->send("dictionary", sampled_dict.name()); 738 | } 739 | 740 | void nn_terrain::create_sampled_tensor_in(){ 741 | vector tensor_in_data; 742 | 743 | float x_stride = (x_hi_prev-x_lo_prev)/x_res_prev; 744 | float y_stride = (y_hi_prev-y_lo_prev)/y_res_prev; 745 | 746 | for (int i(0); i < y_res_prev; i+=plot_resolution){ 747 | for (int j(0); j < x_res_prev; j+=plot_resolution){ 748 | tensor_in_data.push_back(x_lo_prev + j * x_stride); 749 | tensor_in_data.push_back(y_lo_prev + i * y_stride); 750 | } 751 | } 752 | int canvas_width = static_cast(std::ceil(x_res_prev / static_cast(plot_resolution))); 753 | int canvas_height = static_cast(std::ceil(y_res_prev / static_cast(plot_resolution))); 754 | int canvas_size = canvas_width * canvas_height * 2; 755 | 756 | if (tensor_in_data.size() != canvas_size){ 757 | cerr << "tensor_in_data size mismatch: " << tensor_in_data.size() << ", " << canvas_size << endl; 758 | return; 759 | } 760 | sampled_tensor_in = torch::from_blob(tensor_in_data.data(), {canvas_height, canvas_width, 2}, torch::kFloat); 761 | sampled_tensor_in = sampled_tensor_in.clone(); 762 | } 763 | 764 | void nn_terrain::perform(audio_bundle input, audio_bundle output) { 765 | auto vec_size = input.frame_count();// 128 766 | // COPY INPUT TO CIRCULAR BUFFER 767 | for (int c(0); c < input.channel_count(); c++) { // 2 768 | auto in = input.samples(c); 769 | m_in_buffer[c].put(in, vec_size); 770 | } 771 | 772 | if (m_in_buffer[0].full()) { // BUFFER IS FULL // 2048 773 | 774 | std::vector out_model; 775 | for (int c(0); c < m_out_dim; c++){ 776 | out_model.push_back(m_latent_out[c].get()); 777 | } 778 | vector float_in; 779 | for (int c(0); c < input.channel_count(); c++){ 780 | float_in.push_back(*input.samples(c)); 781 | m_in_buffer[c].reset(); 782 | } 783 | 784 | cppn_infer(float_in.data(), out_model); 785 | 786 | for (int c(0); c < m_out_dim; c++){ 787 | m_out_buffer[c].put(m_latent_out[c].get(), m_buffer_size); 788 | } 789 | } 790 | 791 | // COPY CIRCULAR BUFFER TO OUTPUT 792 | for (int c(0); c < output.channel_count(); c++) { // 8 793 | auto out = output.samples(c); 794 | if (enable_cppn){ 795 | m_out_buffer[c].get(out, vec_size); 796 | // memcpy(out, m_latent_out[c].get(), vec_size * sizeof(float)); 797 | } 798 | } 799 | } 800 | 801 | void nn_terrain::cppn_infer(float* float_in, std::vector out_buffer){ 802 | if (!cppn_init){ 803 | return; 804 | } 805 | at::Tensor tensor_in = torch::from_blob(float_in, {1, 2}, torch::kFloat); 806 | 807 | std::unique_lock model_lock(cppn_model->m_model_mutex); 808 | tensor_in = tensor_in.to(cppn_model->m_device); 809 | at::Tensor tensor_out; 810 | try { 811 | 812 | tensor_out = cppn_model->m_model->forward(tensor_in); 813 | 814 | tensor_out = tensor_out.clamp_min({-100.0f}).clamp_max({100.0f}); 815 | 816 | // ----------------------------- 817 | 818 | tensor_out = tensor_out.repeat_interleave(m_buffer_size).reshape({1, m_out_dim, -1}); //2048 is buffersize 819 | 820 | // ----------------------------- 821 | 822 | } catch (const std::exception &e) { 823 | cout << e.what() << endl; 824 | } 825 | model_lock.unlock(); 826 | 827 | tensor_out = tensor_out.to(torch::kCPU); 828 | tensor_out = tensor_out.reshape({1 * m_out_dim, -1}); // -> [8, 2048] 829 | auto out_ptr = tensor_out.contiguous().data_ptr(); 830 | 831 | for (int i(0); i < out_buffer.size(); i++) { 832 | memcpy(out_buffer[i], out_ptr + i * m_buffer_size, m_buffer_size * sizeof(float)); 833 | } 834 | } 835 | 836 | 837 | 838 | nn_terrain::nn_terrain(const atoms &args){ 839 | 840 | // t_model = std::make_unique(); 841 | 842 | // arguments: 843 | // (path) 844 | // (in_dim, out_dim) 845 | // (in_dim, out_dim, gauss_scale) 846 | // (in_dim, out_dim, gauss_scale, c_max) 847 | // (in_dim, out_dim, gauss_scale, c_max, buffer_size) 848 | // (in_dim, out_dim, gauss_scale, c_max, mapping_size, buffer_size) 849 | torch::serialize::InputArchive archive; 850 | 851 | // CHECK ARGUMENTS 852 | if (!args.size()) { 853 | return; 854 | } 855 | if (args.size() == 1) { // ONE ARGUMENT IS GIVEN 856 | if (args[0].a_type == 3){ 857 | std::string model_path = std::string(args[0]); 858 | if (model_path.substr(model_path.length() - 3) != ".pt") 859 | model_path = model_path + ".pt"; 860 | min_path m_path = min_path(model_path); 861 | 862 | try { 863 | // Load the archive from the specified file path 864 | archive.load_from(std::string(m_path)); 865 | 866 | // Load initialization arguments 867 | torch::Tensor in_dim_tensor, out_dim_tensor, c_max_tensor, gauss_scale_tensor, mapping_size_tensor; 868 | archive.read("m_in_dim", in_dim_tensor); 869 | archive.read("m_out_dim", out_dim_tensor); 870 | archive.read("m_cmax", c_max_tensor); 871 | archive.read("m_gauss_scale", gauss_scale_tensor); 872 | archive.read("m_mapping_size", mapping_size_tensor); 873 | 874 | m_in_dim = in_dim_tensor.item(); 875 | m_out_dim = out_dim_tensor.item(); 876 | m_cmax = c_max_tensor.item(); 877 | m_gauss_scale = gauss_scale_tensor.item(); 878 | m_mapping_size = mapping_size_tensor.item(); 879 | } catch (const std::exception& e) { 880 | cerr << "Error loading from checkpoint: " << e.what() << endl; 881 | return; 882 | } 883 | } else { 884 | cerr << "argument should be a path" << endl; 885 | return; 886 | } 887 | } 888 | if (args.size() >= 2) { // TWO ARGUMENTS ARE GIVEN 889 | if (args[0].a_type == 1 && args[1].a_type == 1) { 890 | if (int(args[0]) > 0 && int(args[1]) > 0 && int(args[0]) <= 6 && int(args[1]) <= 1024) { 891 | m_in_dim = int(args[0]); 892 | m_out_dim = int(args[1]); 893 | } else { 894 | cerr << "control_dim should be positive integers <= 6, latent_dim should be positive integers <= 1024" << endl; 895 | return; 896 | } 897 | } else { 898 | cerr << "first and second args should be integers" << endl; 899 | return; 900 | } 901 | } 902 | if (args.size() >= 3) { // THREE ARGUMENTS ARE GIVEN 903 | if (args[2].a_type == 2) { 904 | if (float(args[2]) >= 0 && float(args[2]) <= 10) { 905 | m_gauss_scale = float(args[2]); 906 | } else { 907 | cerr << "third arg should be non-negative float <= 10" << endl; 908 | return; 909 | } 910 | } else { 911 | cerr << "third arg should be a float" << endl; 912 | return; 913 | } 914 | } 915 | if (args.size() >= 4) { // Four ARGUMENTS ARE GIVEN 916 | if (args[3].a_type == 1) { 917 | if (int(args[3]) <= 1024) { 918 | m_cmax = int(args[3]); 919 | } else { 920 | cerr << "fourth arg should be positive integers <= 1024" << endl; 921 | return; 922 | } 923 | } else { 924 | cerr << "fourth arg be an integer" << endl; 925 | return; 926 | } 927 | } 928 | if (args.size() >= 5) { // Five ARGUMENTS ARE GIVEN 929 | if (args[4].a_type == 1) { 930 | if (int(args[4]) <= 2048 && int(args[4]) >= 16) { 931 | m_cmax = int(args[4]); 932 | } else { 933 | cerr << "fifth arg should be positive integers between [16, 2048]" << endl; 934 | return; 935 | } 936 | } else { 937 | cerr << "fifth arg be an integer" << endl; 938 | return; 939 | } 940 | } 941 | if (args.size() >= 6) { 942 | m_buffer_size = power_ceil(int(args[5])); 943 | } 944 | if (args.size() >= 7) { 945 | cerr << "too many arguments" << endl; 946 | return; 947 | } 948 | 949 | cppn_model = std::make_unique(); 950 | 951 | if (cppn_model->create(m_in_dim, m_out_dim, m_cmax, m_gauss_scale, m_mapping_size)) { 952 | cerr << "error during creating model" << endl; 953 | error(); 954 | return; 955 | } 956 | 957 | if (args.size() == 1) { // if loading from a checkpoint 958 | try{ 959 | cppn_model->m_model->load(archive); 960 | } catch (const std::exception& e) { 961 | cerr << "Error loading weights: " << e.what() << endl; 962 | return; 963 | } 964 | } 965 | 966 | // TODO: do this: 967 | cppn_model->m_model->eval(); 968 | 969 | torch::Tensor test_inputs = torch::ones({1, m_in_dim}, torch::kFloat); 970 | 971 | 972 | auto output_tensor = cppn_model->m_model->forward(test_inputs); 973 | 974 | optimizer = std::make_unique(cppn_model->m_model->parameters(), static_cast(lr)); 975 | 976 | cppn_init = true; 977 | 978 | m_use_thread = false; 979 | 980 | // CREATE INLETS, OUTLETS and BUFFERS 981 | m_in_buffer = std::make_unique[]>(m_in_dim); 982 | for (int i(0); i < m_in_dim; i++) { 983 | std::string input_label = "(signal) input from control space dimension: " + std::to_string(i); 984 | m_inlets.push_back(std::make_unique>(this, input_label, "float")); 985 | m_in_buffer[i].initialize(m_buffer_size); //2048 986 | m_in_model.push_back(std::make_unique(m_buffer_size)); 987 | } 988 | 989 | m_out_buffer = std::make_unique[]>(m_out_dim); 990 | for (int i(0); i < m_out_dim; i++) { 991 | std::string output_label = "(signal) output value at latent space dimension: " + std::to_string(i); 992 | m_outlets.push_back(std::make_unique>(this, output_label, "signal")); 993 | m_out_buffer[i].initialize(m_buffer_size); 994 | m_out_model.push_back(std::make_unique(m_buffer_size)); 995 | m_latent_out.push_back(std::make_unique(m_buffer_size)); 996 | } 997 | m_outlets.push_back(std::make_unique>(this, "(dictionary) sampled interval", "dictionary")); 998 | m_outlets.push_back(std::make_unique>(this, "(message) logging information, route option: summary, dataset, dataset_length, epoch, loss", "message")); 999 | 1000 | 1001 | external_path = min_devkit_path(); 1002 | // external_path = std::string(min_path(min_path::system::max)); 1003 | // cout << "path " << external_path << endl; 1004 | 1005 | training_data["coordinates"] = coord_dict; 1006 | training_data["latents"] = latent_dict; 1007 | 1008 | sampled_dict["terrain"] = terrain_dict; 1009 | 1010 | dataset_info_dict["coordinates"] = dataset_co_dict; 1011 | dataset_info_dict["latents"] = dataset_lt_dict; 1012 | 1013 | cout << "terrain setup finished" << endl; 1014 | } 1015 | 1016 | nn_terrain::~nn_terrain() { 1017 | 1018 | terrain_dict.clear(); 1019 | latent_dict.clear(); 1020 | coord_dict.clear(); 1021 | 1022 | m_should_stop_perform_thread = true; 1023 | if (m_compute_thread) 1024 | m_compute_thread->join(); 1025 | } 1026 | MIN_EXTERNAL(nn_terrain); 1027 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/shared/circular_buffer.h: -------------------------------------------------------------------------------- 1 | // 2 | // circular_buffer.h 3 | // https://github.com/acids-ircam/nn_tilde/blob/master/src/frontend/maxmsp/shared/circular_buffer.h 4 | // 5 | // Created by Jasper Zheng on 29/11/2024. 6 | // 7 | 8 | #pragma once 9 | #include 10 | 11 | template class circular_buffer { 12 | public: 13 | circular_buffer(); 14 | void initialize(size_t size); 15 | bool empty(); 16 | bool full(); 17 | void put(in_type *input_array, int N); 18 | void get(out_type *output_array, int N); 19 | out_type get_no_pop(); 20 | void reset(); 21 | 22 | protected: 23 | std::unique_ptr _buffer; 24 | size_t _max_size; 25 | 26 | int _head = 0; 27 | int _tail = 0; 28 | int _count = 0; 29 | bool _full = false; 30 | }; 31 | 32 | template 33 | circular_buffer::circular_buffer() {} 34 | 35 | template 36 | void circular_buffer::initialize(size_t size) { 37 | _buffer = std::make_unique(size); 38 | _max_size = size; 39 | } 40 | 41 | template 42 | bool circular_buffer::empty() { 43 | return (!_full && _head == _tail); 44 | } 45 | 46 | template 47 | bool circular_buffer::full() { 48 | return _full; 49 | } 50 | 51 | template 52 | void circular_buffer::put(in_type *input_array, int N) { 53 | if (!_max_size) 54 | return; 55 | 56 | while (N--) { 57 | _buffer[_head] = out_type(*(input_array++)); 58 | _head = (_head + 1) % _max_size; 59 | if (_full) 60 | _tail = (_tail + 1) % _max_size; 61 | _full = _head == _tail; 62 | } 63 | } 64 | 65 | template 66 | void circular_buffer::get(out_type *output_array, int N) { 67 | if (!_max_size) 68 | return; 69 | 70 | while (N--) { 71 | if (empty()) { 72 | *(output_array++) = out_type(); 73 | } else { 74 | *(output_array++) = _buffer[_tail]; 75 | _tail = (_tail + 1) % _max_size; 76 | _full = false; 77 | } 78 | } 79 | } 80 | 81 | template 82 | out_type circular_buffer::get_no_pop() { 83 | return _buffer[_tail]; 84 | } 85 | 86 | template 87 | void circular_buffer::reset() { 88 | _head = _tail; 89 | _count = 0; 90 | _full = false; 91 | } 92 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/shared/min_dictionary.h: -------------------------------------------------------------------------------- 1 | /// @file 2 | /// @ingroup minapi 3 | /// @copyright Copyright 2018 The Min-API Authors. All rights reserved. 4 | /// @license Use of this source code is governed by the MIT License found in the License.md file. 5 | 6 | #pragma once 7 | 8 | namespace c74::min { 9 | 10 | class min_dict { 11 | public: 12 | /// Create (or reference an existing) dictionary by name 13 | min_dict(const symbol name) { 14 | auto d = max::dictobj_findregistered_retain(name); 15 | 16 | if (!d) { // didn't find a dictionary with that name, so create it 17 | d = max::dictionary_new(); 18 | max::t_symbol* s = name; 19 | m_instance = max::dictobj_register(d, &s); 20 | } 21 | else { 22 | m_instance = d; 23 | } 24 | } 25 | 26 | /// Create an unregistered dictionary from dict-syntax 27 | min_dict(const atoms& content) { 28 | max::dictobj_dictionaryfromatoms(&m_instance, static_cast(content.size()), &content[0]); 29 | } 30 | 31 | /// Create an unregistered dictionary 32 | /// @param d optionally getting a handle to an old-school t_dictionary 33 | /// @param take_ownership defaults to true, change to false only in exceptional cases 34 | min_dict(max::t_dictionary* d = nullptr, const bool take_ownership = true) { 35 | if (d == nullptr) 36 | m_instance = max::dictionary_new(); 37 | else { 38 | if (take_ownership) 39 | max::object_retain((max::t_object*)d); 40 | else 41 | m_has_ownership = false; 42 | m_instance = d; 43 | } 44 | } 45 | 46 | 47 | min_dict(const atom an_atom_containing_a_dict) { 48 | auto a = static_cast(&an_atom_containing_a_dict); 49 | m_instance = static_cast(max::atom_getobj(a)); 50 | if (!m_instance) 51 | error("no dictionary in atom"); 52 | auto err = max::object_retain((max::t_object*)m_instance); 53 | error(err, "failed to retain dictionary instance"); 54 | } 55 | 56 | 57 | ~min_dict() { 58 | if (m_has_ownership) 59 | object_free(m_instance); 60 | } 61 | 62 | 63 | min_dict& operator=(const min_dict& value) { 64 | max::dictionary_clone_to_existing(value.m_instance, m_instance); 65 | return *this; 66 | } 67 | 68 | 69 | min_dict& operator=(const atom& value) { 70 | auto a = static_cast(&value); 71 | if (max::atomisdictionary((max::t_atom *)a)) 72 | m_instance = static_cast(max::atom_getobj(a)); 73 | return *this; 74 | } 75 | 76 | 77 | /// Cast the dictionary to a Max t_object so that, e.g., it can be used with the C Max API. 78 | /// Be exceedingly careful with this! 79 | /// When you pass the pointer out are you pointing the instance without incrementing the reference count? 80 | 81 | // TODO: We need copy (retain) and move (don't retain) semantics ???? 82 | // TODO: we don't have a copy constructor! 83 | 84 | operator max::t_object*() const { 85 | max::object_retain((max::t_object*)m_instance); 86 | return (max::t_object*)m_instance; 87 | } 88 | 89 | 90 | // bounds check: if key doesn't exist, throw 91 | atom_reference at(const symbol key) { 92 | long argc = 0; 93 | max::t_atom* argv = nullptr; 94 | auto err = max::dictionary_getatoms(m_instance, key, &argc, &argv); 95 | 96 | error(err, "could not get key from dictionary"); 97 | return atom_reference(argc, argv); 98 | } 99 | 100 | 101 | // bounds check: if key doesn't exist, create it 102 | atom_reference operator[](symbol key) { 103 | if (!max::dictionary_hasentry(m_instance, key)) 104 | max::dictionary_appendatom(m_instance, key, &atoms{0}[0]); 105 | return at(key); 106 | }; 107 | 108 | atom_reference operator[](int key) { 109 | symbol skey{key}; 110 | return (*this)[skey]; 111 | }; 112 | 113 | 114 | symbol name() const { 115 | return dictobj_namefromptr(m_instance); 116 | } 117 | 118 | 119 | bool valid() const { 120 | return m_instance != nullptr; 121 | } 122 | 123 | 124 | void clear() { 125 | dictionary_clear(m_instance); 126 | } 127 | 128 | 129 | void copyunique(const min_dict& source) { 130 | dictionary_copyunique(m_instance, source.m_instance); 131 | } 132 | 133 | 134 | void touch() { 135 | object_notify(m_instance, k_sym_modified, nullptr); 136 | } 137 | 138 | 139 | /// Register an existing dictionary 140 | /// @param name name to register the dictionary under 141 | void register_as(const symbol name) { 142 | if (m_instance != nullptr) { 143 | max::t_symbol* s = name; 144 | m_instance = max::dictobj_register(m_instance, &s); 145 | } 146 | } 147 | 148 | 149 | // private: 150 | max::t_dictionary* m_instance { nullptr }; 151 | bool m_has_ownership { true }; 152 | }; 153 | 154 | } // namespace c74::min 155 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/shared/min_path.h: -------------------------------------------------------------------------------- 1 | /// @file 2 | /// @ingroup minapi 3 | /// @copyright Copyright 2018 The Min-API Authors. All rights reserved. 4 | /// @license Use of this source code is governed by the MIT License found in the License.md file. 5 | 6 | #pragma once 7 | #include 8 | #include 9 | #include 10 | 11 | namespace c74::min { 12 | 13 | 14 | class min_path { 15 | public: 16 | enum class system { undefined = 0, application, desktop, preferences, temp, max }; 17 | 18 | enum class filetype { any = 0, folder, external, patcher, audio }; 19 | 20 | using filedate = max::t_ptr_uint; 21 | 22 | // uninitialized path 23 | min_path() {} 24 | 25 | // path initialized to a system directory 26 | min_path(const system initial) 27 | : m_directory{ true } { 28 | switch (initial) { 29 | case system::application: 30 | m_path = max::path_getapppath(); 31 | break; 32 | case system::desktop: 33 | m_path = max::path_desktopfolder(); 34 | break; 35 | case system::preferences: 36 | max::preferences_path(nullptr, true, &m_path); 37 | break; 38 | case system::temp: 39 | m_path = max::path_tempfolder(); 40 | break; 41 | case system::max: 42 | m_path = max::path_usermaxfolder(); 43 | break; 44 | default: 45 | m_path = 0; 46 | break; 47 | } 48 | } 49 | 50 | 51 | // path initialized to a user-supplied path id (discouraged, but might be provided by legacy Max API) 52 | min_path(const short path_id) 53 | : m_path{ path_id } 54 | , m_directory{ true } 55 | {} 56 | 57 | // path initialized by name 58 | min_path(const std::string& name, const filetype type = filetype::any, const bool create = false) { 59 | strncpy(m_filename, name.c_str(), max::MAX_PATH_CHARS); 60 | 61 | auto types = typelist(type); 62 | max::t_fourcc* first_type{ nullptr }; 63 | if (types.size()) 64 | first_type = &types[0]; 65 | 66 | if (type == filetype::folder) 67 | m_directory = true; 68 | 69 | auto err = max::locatefile_extended(m_filename, &m_path, &m_type, first_type, static_cast(types.size())); 70 | if (err) { 71 | if (create) { 72 | if (type == filetype::folder) { 73 | char fullpath[max::MAX_PATH_CHARS]; 74 | char filename[max::MAX_FILENAME_CHARS]; 75 | max::path_nameconform(name.c_str(), fullpath, max::PATH_STYLE_MAX, max::PATH_TYPE_ABSOLUTE); 76 | 77 | char* foldername = strrchr(fullpath, '/'); 78 | short parent_folder_path = 0; 79 | 80 | if (foldername) { 81 | *foldername = 0; 82 | foldername++; 83 | 84 | err = max::path_frompathname(fullpath, &parent_folder_path, filename); 85 | if (!err) 86 | err = max::path_createfolder(parent_folder_path, foldername, &m_path); 87 | if (err) 88 | error("error trying to create folder"); 89 | } 90 | else 91 | error("no folder name provided"); 92 | 93 | m_filename[0] = 0; 94 | } 95 | else { 96 | error("can only create folders, not files"); 97 | } 98 | } 99 | else { 100 | error("file not found"); 101 | } 102 | } 103 | else { 104 | if (m_type == 'fold') 105 | m_directory = true; 106 | } 107 | 108 | 109 | if (m_directory) { 110 | auto err = max::path_getpath(m_path, m_filename, &m_path); 111 | if (err) 112 | assert(false); 113 | } 114 | } 115 | 116 | 117 | min_path(const atoms& optional_name, const filetype type = filetype::any) { 118 | if (!optional_name.empty()) 119 | *this = min_path(static_cast(optional_name[0])); 120 | else { 121 | auto types = typelist(type); 122 | 123 | if (max::open_dialog(m_filename, &m_path, &m_type, &types[0], static_cast(types.size()))) 124 | error("file not chosen"); 125 | } 126 | } 127 | 128 | 129 | std::vector typelist(const filetype type) { 130 | std::vector list; 131 | max::t_fourcc types[max::TYPELIST_SIZE]; 132 | short type_count = 0; 133 | 134 | // if (type == filetype::any) we don't need to do anything at all 135 | 136 | if (type == filetype::external) 137 | max::typelist_make(types, max::TYPELIST_EXTERNS, &type_count); 138 | else if (type == filetype::audio) { 139 | types[type_count++] = 'WAVE'; 140 | types[type_count++] = 'AIFF'; 141 | types[type_count++] = 'FLAC'; 142 | types[type_count++] = 'Mp3 '; 143 | types[type_count++] = 'M4a '; 144 | types[type_count++] = 'CAF '; 145 | } 146 | else if (type == filetype::folder) { 147 | types[0] = 'fold'; 148 | type_count = 1; 149 | } 150 | else if (type == filetype::patcher) { 151 | max::typelist_make(types, max::TYPELIST_MAXFILES, &type_count); 152 | } 153 | 154 | for (auto i = 0; i < type_count; ++i) 155 | list.push_back(types[i]); 156 | 157 | return list; // TODO: std::move ? 158 | } 159 | 160 | 161 | operator string() const { 162 | char pathname[max::MAX_PATH_CHARS]; 163 | 164 | max::path_toabsolutesystempath(m_path, m_filename, pathname); 165 | std::string s = pathname; 166 | return s; 167 | } 168 | 169 | 170 | // is this a valid path? 171 | operator bool() const { 172 | return m_path != 0; 173 | } 174 | 175 | 176 | filedate date_modified() const { 177 | int err; 178 | filedate date; 179 | 180 | if (m_directory) 181 | err = max::path_getmoddate(m_path, &date); 182 | else 183 | err = max::path_getfilemoddate(m_filename, m_path, &date); 184 | 185 | if (err || date == 0) { 186 | error("failed to get date modified"); 187 | } 188 | return date; 189 | } 190 | 191 | 192 | using enumerate_function = std::function; 193 | 194 | void enumerate(const filetype a_type, const enumerate_function a_callback) { 195 | if (!m_directory) 196 | return; 197 | if (!m_path) 198 | return; 199 | 200 | char fullpath_to_this_folder[max::MAX_PATH_CHARS]; 201 | max::path_topathname(m_path, m_filename, fullpath_to_this_folder); 202 | // TODO: error checking 203 | 204 | auto fold = max::path_openfolder(m_path); 205 | if (!fold) 206 | return; 207 | 208 | auto types = typelist(a_type); 209 | max::t_fourcc type; 210 | char name[256]; 211 | 212 | while (max::path_foldernextfile(fold, &type, name, false)) { 213 | bool match{ false }; 214 | 215 | if (types.empty()) 216 | match = true; 217 | else { 218 | for (auto& i_type : types) { 219 | if (type == i_type) { 220 | match = true; 221 | break; 222 | } 223 | } 224 | } 225 | if (match) { 226 | a_callback(name); 227 | } 228 | } 229 | max::path_closefolder(fold); 230 | } 231 | 232 | 233 | string name() const { 234 | if (m_directory) { 235 | char pathname[max::MAX_PATH_CHARS]; 236 | max::path_toabsolutesystempath(m_path, m_filename, pathname); 237 | char* last = strrchr(pathname, '/') + 1; 238 | return last; 239 | } 240 | else 241 | return m_filename; 242 | } 243 | 244 | /// Copy the file/folder represented by this path to a specified destination. 245 | /// @param destination_folder The folder will be the folder containing the copy of this path 246 | 247 | void copy(const min_path& destination_folder, const string& destination_name) { 248 | short newpath{}; 249 | if (m_directory) 250 | c74::max::path_copyfolder(m_path, destination_folder.m_path, (char*)destination_name.c_str(), true, &newpath); 251 | else 252 | c74::max::path_copyfile(m_path, m_filename, destination_folder.m_path, (char*)destination_name.c_str()); 253 | } 254 | 255 | short get_path() const { 256 | return m_path; 257 | } 258 | 259 | private: 260 | short m_path{}; 261 | char m_filename[max::MAX_PATH_CHARS]{}; 262 | max::t_fourcc m_type{}; 263 | bool m_directory{}; 264 | }; 265 | 266 | atoms create_log_and_save(const std::string& save_name, const std::string& base_path, const std::string& input) { 267 | 268 | char buffer[256]; 269 | struct tm * datetime; 270 | std::time_t timestamp; 271 | 272 | std::time(×tamp); 273 | datetime = localtime(×tamp); 274 | 275 | std::strftime(buffer, sizeof(buffer), "%Y%m%d%H%M%S", datetime); 276 | 277 | string file_name_str = save_name + buffer; 278 | string src_path_str = (std::filesystem::path(base_path) / (file_name_str+".txt")).string(); 279 | 280 | 281 | // string src_content; 282 | // { 283 | // std::ifstream in{ src_path_str }; 284 | // src_content = string{ std::istreambuf_iterator(in), std::istreambuf_iterator() }; 285 | // } 286 | // { 287 | // std::ofstream out{ src_path_str }; 288 | // out << input; 289 | // out.close(); 290 | // } 291 | 292 | return { {src_path_str, file_name_str} }; 293 | } 294 | 295 | 296 | } // namespace c74::min 297 | 298 | 299 | -------------------------------------------------------------------------------- /src/frontend/maxmsp/shared/utils.h: -------------------------------------------------------------------------------- 1 | 2 | #include 3 | #include 4 | #include 5 | 6 | #include "c74_min_api.h" 7 | 8 | 9 | using namespace c74::min; 10 | using namespace c74::min::ui; 11 | 12 | unsigned power_ceil(unsigned x) { 13 | if (x <= 1) 14 | return 1; 15 | int power = 2; 16 | x--; 17 | while (x >>= 1) 18 | power <<= 1; 19 | return power; 20 | } 21 | -------------------------------------------------------------------------------- /src/package-info.json.in: -------------------------------------------------------------------------------- 1 | { 2 | "author" : "Shuoyang Jasper Zheng", 3 | "description" : "Coordinates-to-Latents Mapping for AI Audio Generation Models", 4 | "homepatcher" : "Latent Terrain Synth 2D.maxpat", 5 | "max_version_min" : "8.0", "max_version_max" : "none", 6 | "name" : "@C74_PACKAGE_NAME@", 7 | "os" : { 8 | "macintosh" : { 9 | "platform" : [ "x64", "aarch64" ], 10 | "min_version" : "none" 11 | }, 12 | "windows" : { 13 | "platform" : [ "x64" ], 14 | "min_version" : "none" 15 | } 16 | }, 17 | "tags" : [ ], 18 | "package_extra" : { 19 | "reverse_domain" : "com.jasperzheng" 20 | }, 21 | "version" : "@GIT_VERSION_MAJ@.@GIT_VERSION_MIN@.@GIT_VERSION_SUB@", 22 | "website" : "https://github.com/jasper-zheng/nn_terrain" 23 | } 24 | --------------------------------------------------------------------------------