├── LICENSE
├── README.md
├── examples
    ├── datasets
    │   ├── MG.mat
    │   └── pianomidi.mat
    ├── main_MackeyGlass.py
    └── main_MultivariatePolyphonic.py
├── pyproject.toml
├── setup.py
└── src
    └── deepesn
        ├── __init__.py
        ├── configurations.py
        ├── deepesn.py
        ├── metrics.py
        └── task.py


/LICENSE:
--------------------------------------------------------------------------------
 1 | BSD 3-Clause License
 2 | 
 3 | Copyright (c) 2018, Luca Pedrelli
 4 | All rights reserved.
 5 | 
 6 | Redistribution and use in source and binary forms, with or without
 7 | modification, are permitted provided that the following conditions are met:
 8 | 
 9 | * Redistributions of source code must retain the above copyright notice, this
10 |   list of conditions and the following disclaimer.
11 | 
12 | * Redistributions in binary form must reproduce the above copyright notice,
13 |   this list of conditions and the following disclaimer in the documentation
14 |   and/or other materials provided with the distribution.
15 | 
16 | * Neither the name of the copyright holder nor the names of its
17 |   contributors may be used to endorse or promote products derived from
18 |   this software without specific prior written permission.
19 | 
20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Deep Echo State Network Python library (DeepESNpy), 2018
 2 | 
 3 | Deep Echo State Network (DeepESN) is an extension of the ESN model towards the deep learning paradigm (Deep Reservoir Computing). A DeepESN is a deep Recurrent Neural Network composed by a hierarchy of recurrent layers intrinsically able to develop hierarchical and distributed temporal features. Such characteristics make DeepESN suitable for time-series and sequences processing.
 4 | 
 5 | All details about DeepESN model are described in the reference paper (**CITATION REQUEST**): <br />
 6 | C. Gallicchio, A. Micheli, L. Pedrelli, <br />
 7 | ["Deep Reservoir Computing: A Critical Experimental Analysis", Neurocomputing, 2017, vol. 268, pp. 87-99](https://www.sciencedirect.com/science/article/pii/S0925231217307567)
 8 | 
 9 | The design of DeepESN model in multivariate time-series prediction tasks is described in the following paper: <br />
10 | C. Gallicchio, A. Micheli, L. Pedrelli, <br />
11 | ["Design of deep echo state networks", Neural Networks, 2018, vol. 108, pp. 33-47](https://www.sciencedirect.com/science/article/pii/S0893608018302223)
12 | 
13 | 
14 | **AUTHOR INFORMATION**
15 | 
16 | Luca Pedrelli <br />
17 | luca.pedrelli@di.unipi.it <br />
18 | lucapedrelli@gmail.com
19 | 
20 | Department of Computer Science - University of Pisa (Italy) <br />
21 | Computational Intelligence & Machine Learning (CIML) Group <br />
22 | http://www.di.unipi.it/groups/ciml/
23 | 
24 | ## Installation
25 | 
26 | To install DeepESN, run the following command:
27 | 
28 | ```bash
29 | pip install DeepESN
30 | 


--------------------------------------------------------------------------------
/examples/datasets/MG.mat:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/lucapedrelli/DeepESN/67ecfd7e1abf83db4853f47a307b639ac44b89de/examples/datasets/MG.mat


--------------------------------------------------------------------------------
/examples/datasets/pianomidi.mat:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/lucapedrelli/DeepESN/67ecfd7e1abf83db4853f47a307b639ac44b89de/examples/datasets/pianomidi.mat


--------------------------------------------------------------------------------
/examples/main_MackeyGlass.py:
--------------------------------------------------------------------------------
 1 | '''
 2 | This is an example of the application of DeepESN model for next-step prediction on Mackey Glass time-series. 
 3 | 
 4 | Reference paper for DeepESN model:
 5 | C. Gallicchio, A. Micheli, L. Pedrelli, "Deep Reservoir Computing: A Critical Experimental Analysis", 
 6 | Neurocomputing, 2017, vol. 268, pp. 87-99
 7 | 
 8 | Reference paper for the design of DeepESN model in multivariate time-series prediction tasks:
 9 | C. Gallicchio, A. Micheli, L. Pedrelli, "Design of deep echo state networks",
10 | Neural Networks, 2018, vol. 108, pp. 33-47 
11 |     
12 | ----
13 | 
14 | This file is a part of the DeepESN Python Library (DeepESNpy)
15 | 
16 | Luca Pedrelli
17 | luca.pedrelli@di.unipi.it
18 | lucapedrelli@gmail.com
19 | 
20 | Department of Computer Science - University of Pisa (Italy)
21 | Computational Intelligence & Machine Learning (CIML) Group
22 | http://www.di.unipi.it/groups/ciml/
23 | 
24 | ----
25 | '''
26 | 
27 | import numpy as np
28 | import random
29 | from deepesn import DeepESN
30 | from deepesn.metrics import MSE 
31 | from deepesn.configurations import config_MG
32 | from deepesn.task import load_MG, select_indexes
33 | class Struct(object): pass
34 | 
35 | # sistemare indici per IP in config_pianomidi, mettere da un'altra parte
36 | # sistema selezione indici con transiente messi all'interno della rete
37 | def main():
38 |     
39 |     # fix a seed for the reproducibility of results
40 |     np.random.seed(7)
41 |    
42 |     # dataset path 
43 |     path = 'datasets'
44 |     dataset, Nu, error_function, optimization_problem, TR_indexes, VL_indexes, TS_indexes = load_MG(path, MSE)
45 | 
46 |     # load configuration for pianomidi task
47 |     configs = config_MG(list(TR_indexes) + list(VL_indexes))
48 |     
49 |     # Be careful with memory usage
50 |     Nr = 100 # number of recurrent units
51 |     Nl = 5 # number of recurrent layers
52 |     reg = 0.0;
53 |     transient = 100
54 |     
55 |     deepESN = DeepESN(Nu, Nr, Nl, configs)
56 |     states = deepESN.computeState(dataset.inputs, deepESN.IPconf.DeepIP)
57 |     
58 |     train_states = select_indexes(states, list(TR_indexes) + list(VL_indexes), transient)
59 |     train_targets = select_indexes(dataset.targets, list(TR_indexes) + list(VL_indexes), transient)
60 |     test_states = select_indexes(states, TS_indexes)
61 |     test_targets = select_indexes(dataset.targets, TS_indexes)
62 |     
63 |     deepESN.trainReadout(train_states, train_targets, reg)
64 |     
65 |     train_outputs = deepESN.computeOutput(train_states)
66 |     train_error = error_function(train_outputs, train_targets)
67 |     print('Training ACC: ', np.mean(train_error), '\n')
68 |     
69 |     test_outputs = deepESN.computeOutput(test_states)
70 |     test_error = error_function(test_outputs, test_targets)
71 |     print('Test ACC: ', np.mean(test_error), '\n')
72 |  
73 |  
74 | if __name__ == "__main__":
75 |     main()
76 |     
77 | 


--------------------------------------------------------------------------------
/examples/main_MultivariatePolyphonic.py:
--------------------------------------------------------------------------------
 1 | '''
 2 | This is an example of the application of DeepESN model for multivariate time-series prediction task 
 3 | on Piano-midi.de (see http://www-etud.iro.umontreal.ca/~boulanni/icml2012) dataset.
 4 | The dataset is a polyphonic music task characterized by 88-dimensional sequences representing musical compositions.
 5 | Starting from played notes at time t, the aim is to predict the played notes at time t+1.
 6 | 
 7 | Reference paper for DeepESN model:
 8 | C. Gallicchio, A. Micheli, L. Pedrelli, "Deep Reservoir Computing: A Critical Experimental Analysis", 
 9 | Neurocomputing, 2017, vol. 268, pp. 87-99
10 | 
11 | In this Example we consider the hyper-parameters designed in the following paper:
12 | C. Gallicchio, A. Micheli, L. Pedrelli, "Design of deep echo state networks",
13 | Neural Networks, 2018, vol. 108, pp. 33-47
14 | 
15 | However, for the ease of calculation, in this example
16 | we consider only Nr = 100 units and Nl = 5 layers instead of Nr = 200 and Nl = 35.
17 | 
18 | ----
19 | 
20 | This file is a part of the DeepESN Python Library (DeepESNpy)
21 | 
22 | Luca Pedrelli
23 | luca.pedrelli@di.unipi.it
24 | lucapedrelli@gmail.com
25 | 
26 | Department of Computer Science - University of Pisa (Italy)
27 | Computational Intelligence & Machine Learning (CIML) Group
28 | http://www.di.unipi.it/groups/ciml/
29 | 
30 | ----
31 | '''
32 | 
33 | import numpy as np
34 | import random
35 | from DeepESN import DeepESN
36 | from deepesn.metrics import computeMusicAccuracy 
37 | from deepesn.configurations import config_pianomidi
38 | from deepesn.task import load_pianomidi, select_indexes
39 | class Struct(object): pass
40 | 
41 | # sistemare indici per IP in config_pianomidi, mettere da un'altra parte
42 | # sistema selezione indici con transiente messi all'interno della rete
43 | def main():
44 |     
45 |     # fix a seed for the reproducibility of results
46 |     np.random.seed(7)
47 |    
48 |     # dataset path 
49 |     path = 'datasets'
50 |     dataset, Nu, error_function, optimization_problem, TR_indexes, VL_indexes, TS_indexes = load_pianomidi(path, computeMusicAccuracy)
51 | 
52 |     # load configuration for pianomidi task
53 |     configs = config_pianomidi(list(TR_indexes) + list(VL_indexes))   
54 |     
55 |     # Be careful with memory usage
56 |     Nr = 100 # number of recurrent units
57 |     Nl = 5 # number of recurrent layers
58 |     reg = 10.0**-2;
59 |     transient = 5
60 |     
61 |     deepESN = DeepESN(Nu, Nr, Nl, configs)
62 |     states = deepESN.computeState(dataset.inputs, deepESN.IPconf.DeepIP)
63 |     
64 |     train_states = select_indexes(states, list(TR_indexes) + list(VL_indexes), transient)
65 |     train_targets = select_indexes(dataset.targets, list(TR_indexes) + list(VL_indexes), transient)
66 |     test_states = select_indexes(states, TS_indexes)
67 |     test_targets = select_indexes(dataset.targets, TS_indexes)
68 |     
69 |     deepESN.trainReadout(train_states, train_targets, reg)
70 |     
71 |     train_outputs = deepESN.computeOutput(train_states)
72 |     train_error = error_function(train_outputs, train_targets)
73 |     print('Training ACC: ', np.mean(train_error), '\n')
74 |     
75 |     test_outputs = deepESN.computeOutput(test_states)
76 |     test_error = error_function(test_outputs, test_targets)
77 |     print('Test ACC: ', np.mean(test_error), '\n')
78 |  
79 |  
80 | if __name__ == "__main__":
81 |     main()
82 |     
83 | 


--------------------------------------------------------------------------------
/pyproject.toml:
--------------------------------------------------------------------------------
1 | [build-system]
2 | requires = [
3 |     "setuptools>=42",
4 |     "wheel"
5 | ]
6 | build-backend = "setuptools.build_meta"


--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
 1 | from setuptools import setup, find_packages
 2 | 
 3 | setup(
 4 |     name='DeepESN',
 5 |     version='1.1.5',
 6 |     packages=find_packages(where='src'),
 7 |     package_dir={'': 'src'},
 8 |     install_requires=[
 9 |         'scipy',
10 |         'numpy'
11 |     ],
12 |     description="Deep Echo State Network Python library (DeepESNpy), 2018",
13 |     author='Luca Pedrelli',
14 |     author_email='lucapedrelli@gmail.com',
15 |     license='BSD 3-Clause',
16 |     url="https://github.com/lucapedrelli/DeepESN"
17 | )
18 | 


--------------------------------------------------------------------------------
/src/deepesn/__init__.py:
--------------------------------------------------------------------------------
1 | from .deepesn import DeepESN
2 | 


--------------------------------------------------------------------------------
/src/deepesn/configurations.py:
--------------------------------------------------------------------------------
 1 | '''
 2 | Model configuration file
 3 | 
 4 | Reference paper for DeepESN model:
 5 | C. Gallicchio, A. Micheli, L. Pedrelli, "Deep Reservoir Computing: A Critical Experimental Analysis", 
 6 | Neurocomputing, 2017, vol. 268, pp. 87-99
 7 | 
 8 | Reference paper for the design of DeepESN model in multivariate time-series prediction tasks:
 9 | C. Gallicchio, A. Micheli, L. Pedrelli, "Design of deep echo state networks",
10 | Neural Networks, 2018, vol. 108, pp. 33-47 
11 |     
12 | ----
13 | 
14 | This file is a part of the DeepESN Python Library (DeepESNpy)
15 | 
16 | Luca Pedrelli
17 | luca.pedrelli@di.unipi.it
18 | lucapedrelli@gmail.com
19 | 
20 | Department of Computer Science - University of Pisa (Italy)
21 | Computational Intelligence & Machine Learning (CIML) Group
22 | http://www.di.unipi.it/groups/ciml/
23 | 
24 | ----
25 | '''
26 | 
27 | import numpy as np
28 | 
29 | class Struct(object): pass
30 | 
31 | def config_pianomidi(IP_indexes):
32 | 
33 |     configs = Struct()
34 |     
35 |     configs.rhos = 0.1 # set spectral radius 0.1 for all recurrent layers
36 |     configs.lis = 0.7 # set li 0.7 for all recurrent layers
37 |     configs.iss = 0.1 # set insput scale 0.1 for all recurrent layers
38 |     
39 |     configs.IPconf = Struct()
40 |     configs.IPconf.DeepIP = 1 # activate pre-train
41 |     configs.IPconf.threshold = 0.1 # threshold for gradient descent in pre-train algorithm
42 |     configs.IPconf.eta = 10**-5 # learning rate for IP rule
43 |     configs.IPconf.mu = 0 # mean of target gaussian function
44 |     configs.IPconf.sigma = 0.1 # std of target gaussian function
45 |     configs.IPconf.Nepochs = 10 # maximum number of epochs
46 |     configs.IPconf.indexes = IP_indexes # perform the pre-train on these indexes
47 | 
48 |     configs.reservoirConf = Struct()
49 |     configs.reservoirConf.connectivity = 1 # connectivity of recurrent matrix
50 |     
51 |     configs.readout = Struct()
52 |     configs.readout.trainMethod = 'NormalEquations' # train with normal equations (faster)
53 |     configs.readout.regularizations = 10.0**np.array(range(-4,-1,1))
54 |     
55 |     return configs
56 | 
57 | 
58 | def config_MG(IP_indexes):
59 | 
60 |     configs = Struct()
61 |     
62 |     configs.rhos = 0.9 # set spectral radius 0.9 for all recurrent layers
63 |     configs.lis = 1.0 # set li 1.0 for all recurrent layers
64 |     configs.iss = 0.1 # set insput scale 0.1 for all recurrent layers
65 |     
66 |     configs.IPconf = Struct()
67 |     configs.IPconf.DeepIP = 0 # deactivate pre-train
68 | 
69 |     configs.reservoirConf = Struct()
70 |     configs.reservoirConf.connectivity = 1 # connectivity of recurrent matrix
71 |     
72 |     configs.readout = Struct()
73 |     configs.readout.trainMethod = 'SVD' # train with singular value decomposition (more accurate)
74 |     configs.readout.regularizations = 10.0**np.array(range(-16,-1,1))
75 |     
76 |     return configs


--------------------------------------------------------------------------------
/src/deepesn/deepesn.py:
--------------------------------------------------------------------------------
  1 | import numpy as np
  2 | import numpy.matlib as npm
  3 | import scipy as sc
  4 | import random
  5 | import sys
  6 | 
  7 | class DeepESN():
  8 |     
  9 |     '''
 10 |     Deep Echo State Network (DeepESN) class:
 11 |     this class implement the DeepESN model suitable for 
 12 |     time-serie prediction and sequence classification.
 13 | 
 14 |     Reference paper for DeepESN model:
 15 |     C. Gallicchio, A. Micheli, L. Pedrelli, "Deep Reservoir Computing: A
 16 |     Critical Experimental Analysis", Neurocomputing, 2017, vol. 268, pp. 87-99
 17 |     
 18 |     Reference paper for the design of DeepESN model in multivariate time-series prediction tasks:
 19 |     C. Gallicchio, A. Micheli, L. Pedrelli, "Design of deep echo state networks",
 20 |     Neural Networks, 2018, vol. 108, pp. 33-47 
 21 |     
 22 |     ----
 23 | 
 24 |     This file is a part of the DeepESN Python Library (DeepESNpy)
 25 | 
 26 |     Luca Pedrelli
 27 |     luca.pedrelli@di.unipi.it
 28 |     lucapedrelli@gmail.com
 29 | 
 30 |     Department of Computer Science - University of Pisa (Italy)
 31 |     Computational Intelligence & Machine Learning (CIML) Group
 32 |     http://www.di.unipi.it/groups/ciml/
 33 | 
 34 |     ----
 35 |     '''
 36 |     
 37 |     def __init__(self, Nu,Nr,Nl, configs, verbose=0):
 38 |         # initialize the DeepESN model
 39 |         
 40 |         if verbose:
 41 |             sys.stdout.write('init DeepESN...')
 42 |             sys.stdout.flush()
 43 |         
 44 |         rhos = np.array(configs.rhos) # spectral radius (maximum absolute eigenvalue)
 45 |         lis = np.array(configs.lis) # leaky rate
 46 |         iss = np.array(configs.iss) # input scale
 47 |         IPconf = configs.IPconf # configuration for Deep Intrinsic Plasticity
 48 |         reservoirConf = configs.reservoirConf # reservoir configurations
 49 |         
 50 |         if len(rhos.shape) == 0:
 51 |             rhos = npm.repmat(rhos, 1,Nl)[0]
 52 | 
 53 |         if len(lis.shape) == 0:
 54 |             lis = npm.repmat(lis, 1,Nl)[0]
 55 | 
 56 |         if len(iss.shape) == 0:
 57 |             iss = npm.repmat(iss, 1,Nl)[0]
 58 |             
 59 |         self.W = {} # recurrent weights
 60 |         self.Win = {} # recurrent weights
 61 |         self.Gain = {} # activation function gain
 62 |         self.Bias = {} # activation function bias
 63 | 
 64 |         self.Nu = Nu # number of inputs
 65 |         self.Nr = Nr # number of units per layer
 66 |         self.Nl = Nl # number of layers
 67 |         self.rhos = rhos.tolist() # list of spectral radius
 68 |         self.lis = lis # list of leaky rate
 69 |         self.iss = iss # list of input scale
 70 | 
 71 |         self.IPconf = IPconf   
 72 |         
 73 |         self.readout = configs.readout
 74 |                
 75 |         # sparse recurrent weights init
 76 |         if reservoirConf.connectivity < 1:
 77 |             for layer in range(Nl):
 78 |                 self.W[layer] = np.zeros((Nr,Nr))
 79 |                 for row in range(Nr):
 80 |                     number_row_elements = round(reservoirConf.connectivity * Nr)
 81 |                     row_elements = random.sample(range(Nr), number_row_elements)
 82 |                     self.W[layer][row,row_elements] = np.random.uniform(-1,+1, size = (1,number_row_elements))
 83 |                     
 84 |         # full-connected recurrent weights init      
 85 |         else:
 86 |             for layer in range(Nl):
 87 |                 self.W[layer] = np.random.uniform(-1,+1, size = (Nr,Nr))
 88 |         
 89 |         # layers init
 90 |         for layer in range(Nl):
 91 | 
 92 |             target_li = lis[layer]
 93 |             target_rho = rhos[layer]
 94 |             input_scale = iss[layer]
 95 | 
 96 |             if layer==0:
 97 |                 self.Win[layer] = np.random.uniform(-input_scale, input_scale, size=(Nr,Nu+1))
 98 |             else:
 99 |                 self.Win[layer] = np.random.uniform(-input_scale, input_scale, size=(Nr,Nr+1))
100 | 
101 |             Ws = (1-target_li) * np.eye(self.W[layer].shape[0], self.W[layer].shape[1]) + target_li * self.W[layer]
102 |             eig_value,eig_vector = np.linalg.eig(Ws)  
103 |             actual_rho = np.max(np.absolute(eig_value))
104 | 
105 |             Ws = (Ws *target_rho)/actual_rho
106 |             self.W[layer] = (target_li**-1) * (Ws - (1.-target_li) * np.eye(self.W[layer].shape[0], self.W[layer].shape[1]))
107 |             
108 |             self.Gain[layer] = np.ones((Nr,1))
109 |             self.Bias[layer] = np.zeros((Nr,1)) 
110 |          
111 |         if verbose:
112 |             print('done.')
113 |             sys.stdout.flush()
114 |     
115 |     def computeLayerState(self, input, layer, initialStatesLayer = None, DeepIP = 0):  
116 |         # compute the state of a layer with pre-training if DeepIP == 1                    
117 |         
118 |         state = np.zeros((self.Nr, input.shape[1]))   
119 |         
120 |         if initialStatesLayer is None:
121 |             initialStatesLayer = np.zeros(state[:,0:1].shape)
122 |         
123 |         input = self.Win[layer][:,0:-1].dot(input) + np.expand_dims(self.Win[layer][:,-1],1)   
124 |         
125 |         if DeepIP:
126 |             state_net = np.zeros((self.Nr, input.shape[1]))
127 |             state_net[:,0:1] = input[:,0:1]
128 |             state[:,0:1] = self.lis[layer] * np.tanh(np.multiply(self.Gain[layer], state_net[:,0:1]) + self.Bias[layer])
129 |         else:
130 |             #state[:,0:1] = self.lis[layer] * np.tanh(np.multiply(self.Gain[layer], input[:,0:1]) + self.Bias[layer])        
131 |             state[:,0:1] = (1-self.lis[layer]) * initialStatesLayer + self.lis[layer] * np.tanh( np.multiply(self.Gain[layer], self.W[layer].dot(initialStatesLayer) + input[:,0:1]) + self.Bias[layer])        
132 |  
133 |         for t in range(1,state.shape[1]):
134 |             if DeepIP:
135 |                 state_net[:,t:t+1] = self.W[layer].dot(state[:,t-1:t]) + input[:,t:t+1]
136 |                 state[:,t:t+1] = (1-self.lis[layer]) * state[:,t-1:t] + self.lis[layer] * np.tanh(np.multiply(self.Gain[layer], state_net[:,t:t+1]) + self.Bias[layer])
137 |                 
138 |                 eta = self.IPconf.eta
139 |                 mu = self.IPconf.mu
140 |                 sigma2 = self.IPconf.sigma**2
141 |             
142 |                 # IP learning rule
143 |                 deltaBias = -eta*((-mu/sigma2)+ np.multiply(state[:,t:t+1], (2*sigma2+1-(state[:,t:t+1]**2)+mu*state[:,t:t+1])/sigma2))
144 |                 deltaGain = eta / npm.repmat(self.Gain[layer],1,state_net[:,t:t+1].shape[1]) + deltaBias * state_net[:,t:t+1]
145 |                 
146 |                 # update gain and bias of activation function
147 |                 self.Gain[layer] = self.Gain[layer] + deltaGain
148 |                 self.Bias[layer] = self.Bias[layer] + deltaBias
149 |                 
150 |             else:
151 |                 state[:,t:t+1] = (1-self.lis[layer]) * state[:,t-1:t] + self.lis[layer] * np.tanh( np.multiply(self.Gain[layer], self.W[layer].dot(state[:,t-1:t]) + input[:,t:t+1]) + self.Bias[layer])
152 |                 
153 |         return state
154 | 
155 |     def computeDeepIntrinsicPlasticity(self, inputs):
156 |         # we incrementally perform the pre-training (deep intrinsic plasticity) over layers
157 |         
158 |         len_inputs = range(len(inputs))
159 |         states = []
160 |         
161 |         for i in len_inputs:
162 |             states.append(np.zeros((self.Nr*self.Nl, inputs[i].shape[1])))
163 |         
164 |         for layer in range(self.Nl):
165 | 
166 |             for epoch in range(self.IPconf.Nepochs):
167 |                 Gain_epoch = self.Gain[layer]
168 |                 Bias_epoch = self.Bias[layer]
169 | 
170 | 
171 |                 if len(inputs) == 1:
172 |                     self.computeLayerState(inputs[0][:,self.IPconf.indexes], layer, DeepIP = 1)
173 |                 else:
174 |                     for i in self.IPconf.indexes:
175 |                         self.computeLayerState(inputs[i], layer, DeepIP = 1)
176 |                        
177 |                 
178 |                 if (np.linalg.norm(self.Gain[layer]-Gain_epoch,2) < self.IPconf.threshold) and (np.linalg.norm(self.Bias[layer]-Bias_epoch,2)< self.IPconf.threshold):
179 |                     sys.stdout.write(str(epoch+1))
180 |                     sys.stdout.write('.')
181 |                     sys.stdout.flush()
182 |                     break
183 |                 
184 |                 if epoch+1 == self.IPconf.Nepochs:
185 |                     sys.stdout.write(str(epoch+1))
186 |                     sys.stdout.write('.')
187 |                     sys.stdout.flush()
188 |             
189 |             inputs2 = []
190 |             for i in range(len(inputs)):
191 |                 inputs2.append(self.computeLayerState(inputs[i], layer))
192 |             
193 |             for i in range(len(inputs)):
194 |                 states[i][(layer)*self.Nr: (layer+1)*self.Nr,:] = inputs2[i]
195 | 
196 |             inputs = inputs2                   
197 |             
198 |         return states
199 |     
200 |     def computeState(self,inputs, DeepIP = 0, initialStates = None, verbose=0):
201 |         # compute the global state of DeepESN with pre-training if DeepIP == 1         
202 |         
203 |         if self.IPconf.DeepIP and DeepIP:
204 |             if verbose:
205 |                 sys.stdout.write('compute state with DeepIP...')
206 |                 sys.stdout.flush()
207 |             states = self.computeDeepIntrinsicPlasticity(inputs)
208 |         else:      
209 |             if verbose:
210 |                 sys.stdout.write('compute state...')
211 |                 sys.stdout.flush()
212 |             states = []
213 | 
214 |             for i_seq in range(len(inputs)):
215 |                 states.append(self.computeGlobalState(inputs[i_seq], initialStates))
216 |                 
217 |         if verbose:        
218 |             print('done.')
219 |             sys.stdout.flush()
220 |         
221 |         return states
222 |     
223 |     def computeGlobalState(self,input, initialStates):
224 |         # compute the global state of DeepESN
225 | 
226 |         state = np.zeros((self.Nl*self.Nr,input.shape[1]))
227 |         
228 |         initialStatesLayer = None
229 | 
230 | 
231 |         for layer in range(self.Nl):
232 |             if initialStates is not None:
233 |                 initialStatesLayer = initialStates[(layer)*self.Nr: (layer+1)*self.Nr,:]            
234 |             state[(layer)*self.Nr: (layer+1)*self.Nr,:] = self.computeLayerState(input, layer, initialStatesLayer, 0)    
235 |             input = state[(layer)*self.Nr: (layer+1)*self.Nr,:]   
236 | 
237 |         return state
238 |         
239 |     def trainReadout(self,trainStates,trainTargets,lb, verbose=0):
240 |         # train the readout of DeepESN
241 | 
242 |         trainStates = np.concatenate(trainStates,1)
243 |         trainTargets = np.concatenate(trainTargets,1)
244 |         
245 |         # add bias
246 |         X = np.ones((trainStates.shape[0]+1, trainStates.shape[1]))
247 |         X[:-1,:] = trainStates    
248 |         trainStates = X  
249 |         
250 |         if verbose:
251 |             sys.stdout.write('train readout...')
252 |             sys.stdout.flush()
253 |         
254 |         if self.readout.trainMethod == 'SVD': # SVD, accurate method
255 |             U, s, V = np.linalg.svd(trainStates, full_matrices=False);  
256 |             s = s/(s**2 + lb)
257 |                       
258 |             self.Wout = trainTargets.dot(np.multiply(V.T, np.expand_dims(s,0)).dot(U.T));
259 |             
260 |         else:  # NormalEquation, fast method
261 |             B = trainTargets.dot(trainStates.T)
262 |             A = trainStates.dot(trainStates.T)
263 | 
264 |             self.Wout = np.linalg.solve((A + np.eye(A.shape[0], A.shape[1]) * lb), B.T).T
265 | 
266 |         if verbose:
267 |             print('done.')
268 |             sys.stdout.flush()
269 |         
270 |     def computeOutput(self,state):
271 |         # compute a linear combination between the global state and the output weights  
272 |         state = np.concatenate(state,1)
273 |         return self.Wout[:,0:-1].dot(state) + np.expand_dims(self.Wout[:,-1],1) # Wout product + add bias
274 | 


--------------------------------------------------------------------------------
/src/deepesn/metrics.py:
--------------------------------------------------------------------------------
 1 | '''
 2 | Metrics functions
 3 | 
 4 | ----
 5 | 
 6 | This file is a part of the DeepESN Python Library (DeepESNpy)
 7 | 
 8 | Luca Pedrelli
 9 | luca.pedrelli@di.unipi.it
10 | lucapedrelli@gmail.com
11 | 
12 | Department of Computer Science - University of Pisa (Italy)
13 | Computational Intelligence & Machine Learning (CIML) Group
14 | http://www.di.unipi.it/groups/ciml/
15 | 
16 | ----
17 | '''
18 | 
19 | import numpy as np
20 | 
21 | # Accuracy function used to evaluate the prediction in polyphonic music tasks: true positive/(true positive + false positive + false negative)
22 | def computeMusicAccuracy(threshold,X,Y):
23 |     Y = np.concatenate(Y,1)
24 |     Nsys = np.sum(X>threshold, axis=0)
25 |     Nref = np.sum(Y>threshold, axis=0)
26 |     Ncorr = np.sum((X>threshold) * (Y>threshold), axis=0)
27 |         
28 |     TP = np.sum(Ncorr)
29 |     FP = np.sum(Nsys-Ncorr)
30 |     FN = np.sum(Nref-Ncorr)
31 |     ACCURACY = TP/float(TP + FP + FN)
32 |     return ACCURACY
33 | 
34 | # Mean Absolute Percentage Error
35 | def MAPE(X, Y): 
36 |     return np.mean(np.abs((Y - X) / Y))
37 | 
38 | # Mean Squared Error
39 | def MSE(X,Y):
40 |     return  np.mean((X-Y)**2)
41 | 
42 | # Normalized Mean Squared Error
43 | def NRMSE(X,Y):
44 |     return  (np.mean((X-Y)**2) / (np.std(Y)**2))**0.5
45 | 
46 | # Mean Absolute Error
47 | def MAE(X,Y):
48 |     return  np.mean(np.abs(X-Y))
49 | 


--------------------------------------------------------------------------------
/src/deepesn/task.py:
--------------------------------------------------------------------------------
 1 | '''
 2 | Task configuration file
 3 |     
 4 | ----
 5 | 
 6 | This file is a part of the DeepESN Python Library (DeepESNpy)
 7 | 
 8 | Luca Pedrelli
 9 | luca.pedrelli@di.unipi.it
10 | lucapedrelli@gmail.com
11 | 
12 | Department of Computer Science - University of Pisa (Italy)
13 | Computational Intelligence & Machine Learning (CIML) Group
14 | http://www.di.unipi.it/groups/ciml/
15 | 
16 | ----
17 | '''
18 | 
19 | import functools
20 | import os
21 | from scipy.io import loadmat
22 | import numpy as np
23 | 
24 | class Struct(object): pass
25 | 
26 | def select_indexes(data, indexes, transient=0):
27 | 
28 |     if len(data) == 1:
29 |         return [data[0][:,indexes][:,transient:]]
30 |     
31 |     return [data[i][:,transient:] for i in indexes]
32 | 
33 | def load_pianomidi(path, metric_function):
34 |     
35 |     data = loadmat(os.path.join(path, 'pianomidi.mat')) # load dataset
36 | 
37 |     dataset = Struct()
38 |     dataset.name = data['dataset'][0][0][0][0]
39 |     dataset.inputs = data['dataset'][0][0][1][0].tolist()
40 |     dataset.targets = data['dataset'][0][0][2][0].tolist()
41 | 
42 |     # input dimension
43 |     Nu = dataset.inputs[0].shape[0]
44 | 
45 |     # function used for model evaluation
46 |     error_function = functools.partial(metric_function, 0.5)     
47 |     
48 |     # select the model that achieves the maximum accuracy on validation set
49 |     optimization_problem = np.argmax    
50 |     
51 |     
52 |     TR_indexes = range(87) # indexes for training, validation and test set in Piano-midi.de task
53 |     VL_indexes = range(87,99)
54 |     TS_indexes = range(99,124)
55 |     
56 |     return dataset, Nu, error_function, optimization_problem, TR_indexes, VL_indexes, TS_indexes
57 | 
58 | def load_MG(path, metric_function):
59 |     
60 |     data = loadmat(os.path.join(path, 'MG.mat')) # load dataset
61 | 
62 |     dataset = Struct()
63 |     dataset.name = data['dataset'][0][0][0][0]
64 |     dataset.inputs = data['dataset'][0][0][1][0]
65 |     dataset.targets = data['dataset'][0][0][2][0]
66 | 
67 |     # input dimension
68 |     Nu = dataset.inputs[0].shape[0]
69 | 
70 |     # function used for model evaluation
71 |     error_function = metric_function    
72 |     
73 |     # select the model that achieves the maximum accuracy on validation set
74 |     optimization_problem = np.argmin   
75 |     
76 |     
77 |     TR_indexes = range(4000) # indexes for training, validation and test set in Piano-midi.de task
78 |     VL_indexes = range(4000,5000)
79 |     TS_indexes = range(5000,9999)
80 |     
81 |     return dataset, Nu, error_function, optimization_problem, TR_indexes, VL_indexes, TS_indexes
82 |   


--------------------------------------------------------------------------------