├── .gitignore ├── Dockerfile ├── README.md ├── TCA-extended.ipynb ├── TCA.ipynb ├── data ├── latent.npy ├── neuron_factor.npy ├── observed.npy ├── time_factor.npy └── trial_factor.npy ├── figures ├── d-way.png ├── factor0.png ├── factor1.png ├── factor2.png ├── groundtruth-estimate-rs.png ├── groundtruth-estimate.png ├── matrix-decomposition.png ├── metric-1-rs.png ├── metric-1.png ├── model.png ├── neuron-time.png ├── output_sample.png ├── tensor-decomposition.png └── time-trial.png ├── jupyter_notebook_config.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.pyd 3 | *~ 4 | .idea 5 | .cache 6 | .ipynb_checkpoints 7 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM jupyter/datascience-notebook:r-4.1.2 2 | 3 | USER root 4 | 5 | # Install essential Python packages 6 | RUN python3 -m pip --no-cache-dir install \ 7 | "tensorly==0.4.3" \ 8 | git+https://github.com/ahwillia/tensortools 9 | 10 | EXPOSE 8888 11 | ADD ./jupyter_notebook_config.py /root/.jupyter/ 12 | WORKDIR /project 13 | ENTRYPOINT ["jupyter", "lab", "--allow-root"] 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tensor Decomposition in Python 2 | 3 | This repository gives a quick tutorial on implementing Canonical Polyadic tensor decomposition in Python, including a brief introduction to Tensor decomposition. However, the main purpose of this notebook is to focus on the implementation of tensor decomposition in Python. In line with these objectives, we will implement tensor decomposition using two libraries available in Python ([TensorLy](http://tensorly.org/stable/index.html) and [tensortools](https://tensortools-docs.readthedocs.io/en/latest/)) and a simple implementation of Tensor Decomposition with Numpy (via alternating optimization). Furthermore, the result of these three approaches are compared in terms of reconstruction error and execution time. 4 | 5 | ## Content 6 | 1. **Tensor Decomposition** 7 | 2. **Implementation** 8 | - TensorLy 9 | - tensortools 10 | - Numpy 11 | 3. **Results and Comparison** 12 | 13 | --- 14 | 15 | ### Tensor Decomposition 16 |

17 | Fig1 18 |

19 | 20 | ### Example Scenario (with a neuroscientific theme) 21 | 22 | An attempt to partially reproducing the result of [this paper](https://doi.org/10.1016/j.neuron.2018.05.015)'s figure 2. 23 | 24 |

25 | Fig1 26 | Fig1 27 |

28 | 29 | ### Example Results and Comparison 30 |

31 | Fig1 32 | Fig1 33 |

34 | 35 | For the complete tutorial click [here](https://medium.com/@mohammadbashiri93/tensor-decomposition-in-python-f1aa2f9adbf4), and for the extended version click [here](https://github.com/mohammadbashiri/tensor-decomposition-in-python/blob/master/TCA-extended.ipynb). 36 | 37 | 38 | --- 39 | ## Dependencies 40 | - Numpy 41 | - Matplotlib 42 | - Seaborn 43 | - TensorLy == 0.4.3 44 | - tensortools 45 | --- 46 | ## Running the notebooks inside docker container 47 | Of course you can simply install the dependencies and run the code in your own Python environment. But another option to run the notebook in a docker container: 48 | 49 | 1. Clone the repository 50 | ``` 51 | git clone https://github.com/mohammadbashiri/tensor-decomposition-in-python.git 52 | ``` 53 | 2. Navigate to the project directory 54 | ``` 55 | cd tensor-decomposition-in-python 56 | ``` 57 | 3. Build the docker image 58 | ``` 59 | docker build -t ds . 60 | ``` 61 | 4. Create a container from the created image 62 | ``` 63 | docker run -p 7890:8888 -v $(pwd):/project --name ds_container -d --rm ds 64 | ``` 65 | 5. By entering `localhost:7890` in your browser you can open the current directory inside jupyterlab and run the notebooks 66 | --- 67 | ## Acknowledgement 68 | I would like to thank [Annika Thierfelder](https://github.com/athierfelder) for her constructive feedback on the content. 69 | 70 | --- 71 | ## References 72 | - Tuncer, Yalcin, Murat M. Tanik, and David B. Allison. "An overview of statistical decomposition techniques applied to complex systems." Computational statistics & data analysis 52.5 (2008): 2292–2310. 73 | - Cichocki, Andrzej, et al. "Tensor decompositions for signal processing applications: From two-way to multiway component analysis." IEEE Signal Processing Magazine 32.2 (2015): 145–163. 74 | - Williams, Alex H., et al. "Unsupervised Discovery of Demixed, Low-Dimensional Neural Dynamics across Multiple Timescales through Tensor Component Analysis." Neuron (2018). 75 | - [Talk](https://www.youtube.com/watch?v=L8uT6hgMt00&t=1302s) by Tamara Kolda 76 | - Tutorial by Alex Williams: [part 1](https://www.youtube.com/watch?v=hmmnRF66hOA), [part 2](https://www.youtube.com/watch?v=O-YTsSuEFiM&t=5s) 77 | -------------------------------------------------------------------------------- /TCA.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Tensor Decomposition in Python" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This notebook gives a quick introduction to Tensor decomposition. However, the main purpose of this notebook is to focus on the implementation of tensor decomposition in Python. In line with these objectives, we will implement tensor decomposition using two libraries available in Python ([TensorLy](http://tensorly.org/stable/index.html) and [tensortools](https://tensortools-docs.readthedocs.io/en/latest/)) and a simple implementation of Tensor Decomposition with Numpy (via alternating optimization). Furthermore, the result of these three approaches are compared in terms of reconstruction error and execution time.\n", 15 | "\n", 16 | "For an extended version of this notebook, click [here](https://github.com/mohammadbashiri/tensor-decomposition-in-python/blob/master/TCA-extended.ipynb)" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## Content\n", 24 | "1. **Tensor Decomposition**\n", 25 | "2. **Implementation**\n", 26 | " - TensorLy\n", 27 | " - tensortools\n", 28 | " - Numpy\n", 29 | "3. **Results and Comparison**" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "---" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "## Tensor Decomposition" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": {}, 49 | "source": [ 50 | "Let's simply start by defining each term in the title." 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "**Tensor**: A tensor is a **multidimensional array**. Also known as **d-way array**, in which \"d\" stands for \"dimensions\". Hence, pretty much all of the geometrical data structures we work with are tensors. Up until $d=2$, these tensors have specific names:\n", 58 | "- zero-way tensor: **scalar**\n", 59 | "- one-way tensor: **vector**\n", 60 | "- two-way tensor: **matrix**\n", 61 | "\n", 62 | "One thing to keep in mind, that might be helpful to know for the following sections, is that as the dimensions of the data structure increases we need more values to locate a single element of our data structure. For instance, we can locate any element of a matrix using two values $(i, j)$ and any element of a tensor via three values $(i, j, k)$. Here is a visual representation (up until three-way tensors)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "
\n", 70 | " \"d-way\" \n", 71 | "
" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "**Decomposition**: Decomposition is a process of breaking up into constituent elements. In mathematical analysis, it means factorization of a d-way tensor. In systems science, it consists of finding an optimal partition of a system in terms of its subsystems. In general, decompositions are motivated by a need to obtain a much simpler body of constituents that can best represent a given system (or data) [[1]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2735056/pdf/nihms39318.pdf)." 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "**Tensor Decomposition**: Data can be organized as a d-way tensor. Consequently, the decoposition of such data is called a d-way (tensor) decomposition." 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "### Matrix Decomposition (aka Two-way Decomposition)" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "Before diving into three-way decomposition, let's quickly refresh ourselves with some of the two-way decompositions (i.e., matrix decomposition). Approaches to two-way decomposition are well established, and include Principal Component Analysis (PCA), Independent Component Analysis (ICA), Nonnegative Matrix Factorization (NMF) and Sparse Component Analysis (SCA). These techniques have become standard tools for e.g., blind source separation (BSS), feature extraction, or classification [[2]](https://ieeexplore.ieee.org/abstract/document/7038247). Here is the general idea: Given $X \\in \\mathbb{R}^{I \\times J}$, we would like to have a model $M$ which approximates $X$ via $\\textbf{a} \\in \\mathbb{R}^I $ and $\\textbf{b} \\in \\mathbb{R}^J$, such that:" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "$$ X \\approx M = \\sum_{r=1}^{R} a_r \\cdot b_r^{T} = a_r \\circ b_r = A \\cdot B^T $$\n", 107 | "\n", 108 | "$$ X \\in \\mathbb{R}^{I \\times J}, \\textbf{a} \\in \\mathbb{R}^I, \\textbf{b} \\in \\mathbb{R}^J$$" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "Where R is the new (reduced) dimension of our data, often referred to as **rank**. This operation is simply the summation of outer products of each column of $A$ and $B$, where the column index is specified by $r$, as depicted below:" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "
\n", 123 | " \"matrix-decomposition\"\n", 124 | "
" 125 | ] 126 | }, 127 | { 128 | "cell_type": "markdown", 129 | "metadata": {}, 130 | "source": [ 131 | "Such decomposition is known as *Factor Analysis*. The formulation above suffers from a problem called the **Rotation Problem**. That is, we can insert any nonsingular rotation matrix, $Z$, in the formulation above , and still end up with the same approximation of $X$ (given that columns of Z have an amplitude of 1)." 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": {}, 137 | "source": [ 138 | "$$ X \\approx M = \\sum_{r=1}^{R} a_r \\circ z_r^T \\circ z_r^{-1} \\circ b_r^{T} = A \\cdot Z^T \\cdot Z^{-1} \\cdot B^T $$" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "metadata": {}, 144 | "source": [ 145 | "Hence, if the above formulation is unconstrained, it results in infinitely many combinations of $A$ and $B$. Standard matrix factorizations in linear algebra, such as the QR-factorization, Eigenvalue Decomposition (EVD), and Singular Value Decomposition (SVD), are only special cases of the above formulation, and owe their uniqueness to hard and restrictive constraints such as triangularity and orthogonality. [[2]](fhttps://ieeexplore.ieee.org/abstract/document/7038247)" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "### Three-way Tensor Decomposition" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "Three-way decomposition is merely the extension of the two-way decomposition. However, although in the two-way case explicit constraints must be imposed on the problem to result in a unique solution, the high dimensionality of the tensor format comes with blessings — these include possibilities to obtain compact representations, uniqueness of decompositions, flexibility in the choice of constraints, and generality of components that can be identified [[2]](https://ieeexplore.ieee.org/abstract/document/7038247). In case of three-way decomposition, we have a three-way tensor and we would like to have a model $M$ which approximates $X \\in \\mathbb{R}^{I \\times J \\times K}$, via $\\textbf{a} \\in \\mathbb{R}^I $, $\\textbf{b} \\in \\mathbb{R}^J$, and $\\textbf{c} \\in \\mathbb{R}^K$ such that:" 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "$$ X \\approx M = \\sum_{r=1}^{R} a_r \\circ b_r \\circ c_r$$\n", 167 | "\n", 168 | "$$X \\in \\mathbb{R}^{I \\times J \\times K}, \\textbf{a} \\in \\mathbb{R}^I, \\textbf{b} \\in \\mathbb{R}^J, \\textbf{c} \\in \\mathbb{R}^K$$" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "metadata": {}, 174 | "source": [ 175 | "Where R is the new (reduced) dimension of our data, often referred to as **rank**. As a result of such decomposition, we will have three Matrices $A \\in \\mathbb{R}^{I \\times R}$, $B \\in \\mathbb{R}^{J \\times R}$, and $C \\in \\mathbb{R}^{K \\times R}$. This operation is simply the summation of outer product of each column of $A$, $B$, and $C$ where the column index is specified by $r$, as depicted below:" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "
\n", 183 | " \"tensor-decomposition\"\n", 184 | "
" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "## How to find A, B, and C?" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "In this section we will focus on the implementation of three-way tensor decomposition using two Python libraries: [TensorLy](http://tensorly.org/stable/index.html) and [tensortools](https://tensortools-docs.readthedocs.io/en/latest/). Furthermore, we will also implement a very simple three-way tensor decomposer using Numpy and an alternating optimization algorithm. The example used here is inspired by Figure 2 (as depicted below) of [this paper](https://doi.org/10.1016/j.neuron.2018.05.015) which introduced tensor decomposition, mainly to the neuroscience research." 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "
\n", 206 | "

\n", 207 | " \n", 208 | " \n", 209 | "

\n", 210 | "
" 211 | ] 212 | }, 213 | { 214 | "cell_type": "markdown", 215 | "metadata": {}, 216 | "source": [ 217 | "Let's start by importing the libraries and functions that we are going to need:" 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": 3, 223 | "metadata": {}, 224 | "outputs": [], 225 | "source": [ 226 | "import numpy as np\n", 227 | "import matplotlib.pyplot as plt\n", 228 | "import seaborn as sns\n", 229 | "\n", 230 | "from sklearn.decomposition import FactorAnalysis, PCA\n", 231 | "import tensortools as tt\n", 232 | "from tensortools.operations import unfold as tt_unfold, khatri_rao\n", 233 | "import tensorly as tl\n", 234 | "from tensorly import unfold as tl_unfold\n", 235 | "from tensorly.decomposition import parafac\n", 236 | "\n", 237 | "from utils import *" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": {}, 243 | "source": [ 244 | "### Load Tensor" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "Let's load the data - I have already created the data, shown in previous figure." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": 5, 257 | "metadata": {}, 258 | "outputs": [], 259 | "source": [ 260 | "time_factor = np.load(\"data/time_factor.npy\")\n", 261 | "neuron_factor = np.load(\"data/neuron_factor.npy\")\n", 262 | "trial_factor = np.load(\"data/trial_factor.npy\")\n", 263 | "latent = np.load(\"data/latent.npy\")\n", 264 | "observed = np.load(\"data/observed.npy\")" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "Below we can see how the amplitude of each one of the latent factors (neurons) change over time and trials:" 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "metadata": {}, 277 | "source": [ 278 | "
\n", 279 | "

\n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | "

\n", 284 | "
" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": {}, 290 | "source": [ 291 | "## Implementation of Tensor Decomposition (using the libraries)" 292 | ] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "metadata": {}, 297 | "source": [ 298 | "Concerning the libraries, the implementation is very similar: Simply call the decomposition function and pass the tensor and the rank (number of factors). In **TensorLy**, the function is called `parafac` which implements Canocical Polyadic (CP) decomposition via alternating least squares (ALS). There is a history of how this name came about, but it is also known as Canonical Decomposition (CANDECOMP), as well as Parallel Factorization (PARAFAC). In **tensortools** it is called `cp_als` which implements the same thing." 299 | ] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "metadata": {}, 304 | "source": [ 305 | "Below is the a minimal code to implement CP tensor decomposition in both libraries:\n", 306 | "- Specify the tensor, and the rank (number of factors)\n", 307 | "- Use the function to decompose the tensor\n", 308 | "- Furthermore, we can reconstruct the estimate $M$ using the `reconstruct` function provided in the script." 309 | ] 310 | }, 311 | { 312 | "cell_type": "markdown", 313 | "metadata": {}, 314 | "source": [ 315 | "``` python\n", 316 | "# Specify the tensor and the rank\n", 317 | "X, rank = observed, 3\n", 318 | "\n", 319 | "# Perform CP decompositon using TensorLy\n", 320 | "factors_tl = parafac(X, rank=rank)\n", 321 | "\n", 322 | "# Perform CP decomposition using tensortools\n", 323 | "U = tt.cp_als(X, rank=rank, verbose=False)\n", 324 | "factors_tt = U.factors.factors\n", 325 | "\n", 326 | "# Reconstruct M, with the result of each library\n", 327 | "M_tl = reconstruct(factors_tl)\n", 328 | "M_tt = reconstruct(factors_tt)\n", 329 | "\n", 330 | "# plot the decomposed factors\n", 331 | "plot_factors(factors_tl)\n", 332 | "plot_factors(factors_tt)\n", 333 | "```\n", 334 | "
" 335 | ] 336 | }, 337 | { 338 | "cell_type": "markdown", 339 | "metadata": {}, 340 | "source": [ 341 | "In our case, the resulting factors plot is shown below (only for one of the results):" 342 | ] 343 | }, 344 | { 345 | "cell_type": "markdown", 346 | "metadata": {}, 347 | "source": [ 348 | "
\n", 349 | " \n", 350 | "
" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "### Tensor Decomposition using Numpy" 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "metadata": {}, 363 | "source": [ 364 | "#### The optimization problem" 365 | ] 366 | }, 367 | { 368 | "cell_type": "markdown", 369 | "metadata": {}, 370 | "source": [ 371 | "Ultimately we would like to minimize the difference between $X$ (ground truth) and $M$ (model estimate, which is the approximation of $X$). Hence, we can formulate our loss function as the Squared Error between $X$ and $M$:\n" 372 | ] 373 | }, 374 | { 375 | "cell_type": "markdown", 376 | "metadata": {}, 377 | "source": [ 378 | "$$\\underset{A, B, C}{\\operatorname{min}} \\sum_{i, j, k} (x_{ijk} - m_{ijk})^2 = \\underset{A, B, C}{\\operatorname{min}} \\sum_{i, j, k} (x_{ijk} - \\sum_{r=1}^{R} a_{ir} b_{jr} c_{kr})^2$$" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "metadata": {}, 384 | "source": [ 385 | "$M$ is being computed using three Matrices $A$, $B$ and $C$, all of which we do not know. One way to find them is to optimize for one, while fixing the other two. Once we optimize one, we use it as a fixed matrix while optimizing for another. And we alternate between optimizing $A$, $B$ and $C$ until convergence, or a stoppage criterion. Hence, we are going to write our loss function for each Matrix, as follows:" 386 | ] 387 | }, 388 | { 389 | "cell_type": "markdown", 390 | "metadata": {}, 391 | "source": [ 392 | "$$\\underset{\\color{red}{A}}{\\operatorname{min}} \\sum_{i, j, k} (x_{ijk} - \\sum_{r=1}^{R} \\color{red}{a_{ir}} b_{jr} c_{kr})^2 = \n", 393 | "\\underset{\\color{red}{A}}{\\operatorname{min}} || X_{(0)} - \\color{red}{A}(B \\odot C)^T ||_F^2$$" 394 | ] 395 | }, 396 | { 397 | "cell_type": "markdown", 398 | "metadata": {}, 399 | "source": [ 400 | "$$\\underset{\\color{red}{B}}{\\operatorname{min}} \\sum_{i, j, k} (x_{ijk} - \\sum_{r=1}^{R} a_{ir} \\color{red}{b_{jr}} c_{kr})^2 = \n", 401 | "\\underset{\\color{red}{B}}{\\operatorname{min}} || X_{(1)} - \\color{red}{B}(A \\odot C)^T ||_F^2$$" 402 | ] 403 | }, 404 | { 405 | "cell_type": "markdown", 406 | "metadata": {}, 407 | "source": [ 408 | "$$\\underset{\\color{red}{C}}{\\operatorname{min}} \\sum_{i, j, k} (x_{ijk} - \\sum_{r=1}^{R} a_{ir} b_{jr} \\color{red}{c_{kr}})^2 = \n", 409 | "\\underset{\\color{red}{C}}{\\operatorname{min}} || X_{(2)} - \\color{red}{C}(A \\odot B)^T ||_F^2$$" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "metadata": {}, 415 | "source": [ 416 | "Where $X_{(0)}$ denotes the mode-0 unfolding of tensor X into a matrix, and so on. and $(B \\odot C)^T$ denotes the *Khatri-Rao* product which combines B and C into a single matrix. In general, this is a non-convex problem; however, when we optimize for one matrix at the time, that is a convex problem. If you'd like to know more about unfolding a tensor into a matrix, click on this [link](http://jeankossaifi.com/blog/unfolding.html)." 417 | ] 418 | }, 419 | { 420 | "cell_type": "markdown", 421 | "metadata": {}, 422 | "source": [ 423 | "``` python\n", 424 | "def decompose_three_way(tensor, rank, max_iter=501, verbose=False):\n", 425 | "\n", 426 | " # a = np.random.random((rank, tensor.shape[0]))\n", 427 | " b = np.random.random((rank, tensor.shape[1]))\n", 428 | " c = np.random.random((rank, tensor.shape[2]))\n", 429 | "\n", 430 | " for epoch in range(max_iter):\n", 431 | " # optimize a\n", 432 | " input_a = khatri_rao([b.T, c.T])\n", 433 | " target_a = tl.unfold(tensor, mode=0).T\n", 434 | " a = np.linalg.solve(input_a.T.dot(input_a), input_a.T.dot(target_a))\n", 435 | "\n", 436 | " # optimize b\n", 437 | " input_b = khatri_rao([a.T, c.T])\n", 438 | " target_b = tl.unfold(tensor, mode=1).T\n", 439 | " b = np.linalg.solve(input_b.T.dot(input_b), input_b.T.dot(target_b))\n", 440 | " \n", 441 | " # optimize c\n", 442 | " input_c = khatri_rao([a.T, b.T])\n", 443 | " target_c = tl.unfold(tensor, mode=2).T\n", 444 | " c = np.linalg.solve(input_c.T.dot(input_c), input_c.T.dot(target_c))\n", 445 | "\n", 446 | " if verbose and epoch % int(max_iter * .2) == 0:\n", 447 | " res_a = np.square(input_a.dot(a) - target_a)\n", 448 | " res_b = np.square(input_b.dot(b) - target_b)\n", 449 | " res_c = np.square(input_c.dot(c) - target_c)\n", 450 | " print(\"Epoch:\", epoch, \"| Loss (C):\", res_a.mean(), \"| Loss (B):\", res_b.mean(), \"| Loss (C):\", res_c.mean())\n", 451 | " \n", 452 | " return a.T, b.T, c.T\n", 453 | "```\n", 454 | "
" 455 | ] 456 | }, 457 | { 458 | "cell_type": "markdown", 459 | "metadata": {}, 460 | "source": [ 461 | "Now, similar to the libraries, we can decompose out tensor, given the rank (number of factors):" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": {}, 467 | "source": [ 468 | "``` python\n", 469 | "factors_np = decompose_three_way(X, rank)\n", 470 | "```\n", 471 | "
" 472 | ] 473 | }, 474 | { 475 | "cell_type": "markdown", 476 | "metadata": {}, 477 | "source": [ 478 | "### Resulting Factors from Numpy and Comparing the three approaches" 479 | ] 480 | }, 481 | { 482 | "cell_type": "markdown", 483 | "metadata": {}, 484 | "source": [ 485 | "
\n", 486 | "

\n", 487 | " \n", 488 | " \n", 489 | "

\n", 490 | "
" 491 | ] 492 | }, 493 | { 494 | "cell_type": "markdown", 495 | "metadata": {}, 496 | "source": [ 497 | "---" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "metadata": {}, 503 | "source": [ 504 | "## Acknowledgement\n", 505 | "I would like to thank David Greenberg and Annika Thierfelder for their constructive feedback on the content of this notebook." 506 | ] 507 | }, 508 | { 509 | "cell_type": "markdown", 510 | "metadata": {}, 511 | "source": [ 512 | "## References\n", 513 | "- Tuncer, Yalcin, Murat M. Tanik, and David B. Allison. \"An overview of statistical decomposition techniques applied to complex systems.\" Computational statistics & data analysis 52.5 (2008): 2292–2310.\n", 514 | "- Cichocki, Andrzej, et al. \"Tensor decompositions for signal processing applications: From two-way to multiway component analysis.\" IEEE Signal Processing Magazine 32.2 (2015): 145–163.\n", 515 | "- Williams, Alex H., et al. \"Unsupervised Discovery of Demixed, Low-Dimensional Neural Dynamics across Multiple Timescales through Tensor Component Analysis.\" Neuron (2018).\n", 516 | "- [Talk](https://www.youtube.com/watch?v=L8uT6hgMt00&t=1302s) by Tamara Kolda\n", 517 | "- Tutorial by Alex Williams: [part 1](https://www.youtube.com/watch?v=hmmnRF66hOA), [part 2](https://www.youtube.com/watch?v=O-YTsSuEFiM&t=5s)" 518 | ] 519 | } 520 | ], 521 | "metadata": { 522 | "kernelspec": { 523 | "display_name": "Python 3", 524 | "language": "python", 525 | "name": "python3" 526 | }, 527 | "language_info": { 528 | "codemirror_mode": { 529 | "name": "ipython", 530 | "version": 3 531 | }, 532 | "file_extension": ".py", 533 | "mimetype": "text/x-python", 534 | "name": "python", 535 | "nbconvert_exporter": "python", 536 | "pygments_lexer": "ipython3", 537 | "version": "3.7.1" 538 | } 539 | }, 540 | "nbformat": 4, 541 | "nbformat_minor": 2 542 | } 543 | -------------------------------------------------------------------------------- /data/latent.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/data/latent.npy -------------------------------------------------------------------------------- /data/neuron_factor.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/data/neuron_factor.npy -------------------------------------------------------------------------------- /data/observed.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/data/observed.npy -------------------------------------------------------------------------------- /data/time_factor.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/data/time_factor.npy -------------------------------------------------------------------------------- /data/trial_factor.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/data/trial_factor.npy -------------------------------------------------------------------------------- /figures/d-way.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/d-way.png -------------------------------------------------------------------------------- /figures/factor0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/factor0.png -------------------------------------------------------------------------------- /figures/factor1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/factor1.png -------------------------------------------------------------------------------- /figures/factor2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/factor2.png -------------------------------------------------------------------------------- /figures/groundtruth-estimate-rs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/groundtruth-estimate-rs.png -------------------------------------------------------------------------------- /figures/groundtruth-estimate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/groundtruth-estimate.png -------------------------------------------------------------------------------- /figures/matrix-decomposition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/matrix-decomposition.png -------------------------------------------------------------------------------- /figures/metric-1-rs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/metric-1-rs.png -------------------------------------------------------------------------------- /figures/metric-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/metric-1.png -------------------------------------------------------------------------------- /figures/model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/model.png -------------------------------------------------------------------------------- /figures/neuron-time.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/neuron-time.png -------------------------------------------------------------------------------- /figures/output_sample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/output_sample.png -------------------------------------------------------------------------------- /figures/tensor-decomposition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/tensor-decomposition.png -------------------------------------------------------------------------------- /figures/time-trial.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mohammadbashiri/tensor-decomposition-in-python/46a00e7b8ded5d5cda07b59059e3304a532dfd1b/figures/time-trial.png -------------------------------------------------------------------------------- /jupyter_notebook_config.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | # Accept all incoming requests 4 | c.NotebookApp.ip = "0.0.0.0" 5 | c.NotebookApp.port = 8888 6 | c.NotebookApp.open_browser = False 7 | c.NotebookApp.token = "" # os.environ.get("JUPYTER_PASSWORD", "my-secret-password") 8 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib.pyplot as plt 3 | import seaborn as sns 4 | 5 | 6 | def rank_one_tensor(a, b, c): 7 | """Returns a rank 1 tensor, given three vectors 8 | """ 9 | a = a.reshape(-1, 1).astype(np.float32) 10 | b = b.reshape(-1, 1).astype(np.float32) 11 | c = c.reshape(-1, 1).astype(np.float32) 12 | return np.tensordot(a * b.T, c, axes=0)[:, :, :, 0] 13 | 14 | 15 | def normalize(x, lower=0, upper=1, axis=0): 16 | return (x - x.min(axis=axis)) / (x.max(axis=axis) - x.min(axis=axis)) 17 | 18 | 19 | def reconstruct(factors, rank=None): 20 | a, b, c = factors 21 | rank = rank if rank else a.shape[1] 22 | R1s = np.zeros((a.shape[0], b.shape[0], c.shape[0], rank)) 23 | for i in range(rank): 24 | R1s[:, :, :, i] = rank_one_tensor(a[:, i], b[:, i], c[:, i]) 25 | return R1s.sum(axis=3) 26 | 27 | def plot_factors(factors, d=3): 28 | a, b, c = factors 29 | rank = a.shape[1] 30 | fig, axes = plt.subplots(rank, d, figsize=(8, int(rank * 1.2 + 1))) 31 | factors_name = ["Time", "Features", "Time"] if d==3 else ["Time", "Features"] 32 | for ind, (factor, axs) in enumerate(zip(factors[:d], axes.T)): 33 | axs[-1].set_xlabel(factors_name[ind]) 34 | for i, (f, ax) in enumerate(zip(factor.T, axs)): 35 | sns.despine(top=True, ax=ax) 36 | ax.plot(f) 37 | axes[i, 0].set_ylabel("Factor " + str(i+1)) 38 | fig.tight_layout() 39 | 40 | 41 | def compare_factors(factors, factors_actual, factors_ind=[0, 1, 2], fig=None): 42 | 43 | a_actual, b_actual, c_actual = factors_actual 44 | a, b, c = factors 45 | rank = a.shape[1] 46 | 47 | fig, axes = fig, np.array(fig.axes).reshape(rank, -1) if fig else plt.subplots(rank, 3, figsize=(8, int(rank * 1.2 + 1))) 48 | sns.despine(top=True) 49 | 50 | f_ind = factors_ind 51 | 52 | for ind, ax in enumerate(axes): 53 | ax1, ax2, ax3 = ax 54 | label, label_actual = ("Estimate", "Ground truth") if ind==0 else (None, None) 55 | ax1.plot(a_actual[:, ind], lw=5, c='b', alpha=.8, label=label_actual); # a 56 | ax1.plot(a[:, f_ind[ind]], lw=2, c='red', label=label); # a 57 | ax2.plot(b_actual[:, ind], lw=5, c='b', alpha=.8); # b 58 | ax2.plot(b[:, f_ind[ind]], lw=2, c='red'); # a 59 | ax3.plot(c_actual[:, ind], lw=5, c='b', alpha=.8); # c 60 | ax3.plot(c[:, f_ind[ind]], lw=2, c='red'); # a 61 | 62 | ax2.set_yticklabels([]) 63 | ax2.set_yticks([]) 64 | ax3.set_yticklabels([]) 65 | ax3.set_yticks([]) 66 | ax1.set_ylabel("Factor {}".format(ind+1), fontsize=15) 67 | 68 | if ind != 2: 69 | ax1.set_xticks([]) 70 | ax1.set_xticklabels([]) 71 | ax2.set_xticks([]) 72 | ax2.set_xticklabels([]) 73 | ax3.set_xticks([]) 74 | ax3.set_xticklabels([]) 75 | else: 76 | ax1.set_xlabel("Time", fontsize=15) 77 | ax2.set_xlabel("Neuron", fontsize=15) 78 | ax3.set_xlabel("Trial", fontsize=15) 79 | 80 | fig.tight_layout() 81 | fig.legend(loc='lower left', bbox_to_anchor= (0.08, -0.02), ncol=2, 82 | borderaxespad=0, fontsize=15, frameon=False) 83 | 84 | return fig, axes --------------------------------------------------------------------------------