├── README.md ├── Supplementary Files ├── data_loader ├── augmentations.py └── dataset_loader.py ├── data_preprocess ├── data_preprocess_HAR.py ├── data_preprocess_SHAR.py ├── data_preprocess_UEA.py └── data_preprocess_wisdm.py ├── main.py ├── models ├── BYOL │ ├── mlp_head.py │ ├── resnet_base_network.py │ ├── train_BYOL.py │ ├── trainer.py │ └── utils.py ├── CPC │ ├── config.py │ ├── feature_loader.py │ ├── model.py │ ├── train_CPC.py │ └── transformers.py ├── IDFD │ ├── config_files │ │ ├── EigenWorms_Configs.py │ │ ├── FingerMovements_Configs.py │ │ ├── HAR_Configs.py │ │ ├── PenDigits_Configs.py │ │ ├── SHAR_Configs.py │ │ ├── epilepsy_Configs.py │ │ └── wisdm_Configs.py │ └── train_IDFD.py ├── MHCCL │ ├── framework.py │ └── train_MHCCL.py ├── PCL │ ├── builder.py │ └── train_PCL.py ├── SimCLR │ ├── loss.py │ ├── models.py │ ├── train_SimCLR.py │ └── utils.py ├── SwAV │ ├── hubconf.py │ ├── logger.py │ ├── resnet50.py │ ├── train_SwAV.py │ └── utils.py ├── TLoss │ ├── scikit_wrappers.py │ └── train_TLoss.py ├── TS2Vec │ ├── datautils.py │ ├── train_TS2Vec.py │ ├── ts2vec.py │ └── utils.py ├── TST │ ├── loss.py │ ├── optimizers.py │ ├── running.py │ ├── ts_transformer.py │ └── utils.py ├── TSTCC │ ├── TC.py │ ├── loss.py │ ├── model.py │ ├── train_TSTCC.py │ └── utils.py ├── TimeNet │ ├── framework.py │ └── train_TimeNet.py └── notes.txt ├── organization.png └── taxonomy.png /README.md: -------------------------------------------------------------------------------- 1 | # ULTS 2 | A unified and standardized library of unsupervised representation learning approaches for time series 3 | 4 | 5 | ## Description of this library: 6 | ULTS is a unified and standardized library under the PyTorch framework to enable quick and convenient evaluations on unsupervised representation learning approaches for time series. ULTS integrates 17 representative models covering 2 deep clustering, 2 reconstruction-based and 13 self-supervised learning methods including 2 adversarial, 2 predictive and 9 contrastive ones. For more information, please refer to our paper: [Unsupervised Representation Learning for Time Series: A Review](https://arxiv.org/abs/2308.01578). 7 | 8 | 9 | ## Abstract 10 | Unsupervised representation learning approaches aim to learn discriminative feature representations from unlabeled data, without the requirement of annotating every sample. Enabling unsupervised representation learning is extremely crucial for time series data, due to its unique annotation bottleneck caused by its complex characteristics and lack of visual cues compared with other data modalities. In recent years, unsupervised representation learning techniques have advanced rapidly in various domains. However, there is a lack of systematic analysis of unsupervised representation learning approaches for time series. To fill the gap, we conduct a comprehensive literature review of existing rapidly evolving unsupervised representation learning approaches for time series. Moreover, we also develop a unified and standardized library, named ULTS ({i.e., Unsupervised Learning for Time Series), to facilitate fast implementations and unified evaluations on various models. With ULTS, we empirically evaluate state-of-the-art approaches, especially the rapidly evolving contrastive learning methods, on 9 diverse real-world datasets. We further discuss practical considerations as well as open research challenges on unsupervised representation learning for time series to facilitate future research in this field. 11 | 12 | ## Taxonomy: 13 | ![image](https://github.com/mqwfrog/ULTS/blob/main/taxonomy.png) 14 | 15 | 16 | ## Organization: 17 | ![image](https://github.com/mqwfrog/ULTS/blob/main/organization.png) 18 | 19 | ## Models Implemented in ULTS: 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 |
1st Category2nd Category3rd CategoryModel (Official Implementations)
Deep Clustering Methods--DeepCluster https://github.com/facebookresearch/deepcluster
--IDFD https://github.com/TTN-YKK/Clustering_friendly_representation_learning
Reconstruction-based Methods--TimeNet https://github.com/paudan/TimeNet
--Deconv https://github.com/cauchyturing/Deconv_SAX
Self-supervised Learning MethodsAdversarial-TimeGAN https://github.com/jsyoon0823/TimeGAN
TimeVAE https://github.com/abudesai/timeVAE
Predictive-EEG-SSL https://github.com/mlberkeley/eeg-ssl
TST https://github.com/gzerveas/mvts_transformer
ContrastiveInstance-LevelSimCLR https://github.com/google-research/simclr
BYOL https://github.com/deepmind/deepmind-research/tree/master/byol
CPC https://github.com/facebookresearch/CPC_audio
Prototype-LevelSwAV https://github.com/facebookresearch/swav
PCL https://github.com/salesforce/PCL
MHCCL https://github.com/mqwfrog/MHCCL
Temporal-LevelTS2Vec https://github.com/yuezhihan/ts2vec
TS-TCC https://github.com/emadeldeen24/TS-TCC
T-Loss https://github.com/White-Link/UnsupervisedScalableRepresentationLearningTimeSeries
74 | 75 | ## Requirements for this library: 76 | - Python ≥ 3.6 77 | - PyTorch ≥ 1.4 78 | 79 | ## Required packages for this library: 80 | - numpy 81 | - sklearn 82 | - openpyxl 83 | - torchvision 84 | - random 85 | - copy 86 | - pandas 87 | - matplotlib 88 | - time 89 | - collections 90 | - scipy 91 | - pynndescent 92 | - builtins 93 | - math 94 | - shutil 95 | - os 96 | - sys 97 | - warnings 98 | - tqdm 99 | - argparse 100 | - tensorboard_logger 101 | 102 | 103 | ## Data: 104 | - The [UCI](https://archive.ics.uci.edu/datasets) archive includes 85 multivariate time series datasets for classification tasks. These datasets covers various application fields including audio spectra classification, business, ECG/EEG classification, human activity recognition, gas detection, motion classification, etc. 105 | - The [UEA](http://www.timeseriesclassification.com/dataset.php) archive includes 30 multivariate time series datasets, covers the application fields of audio spectra classification, ECG/EEG/MEG classification, human activity recognition, motion classification, etc. 106 | - The [MTS](http://www.mustafabaydogan.com/multivariate-time-series-discretization-for-classification.html) archive, also known as Baydogan's archive, includes 13 multivariate time series datasets, covers the application fields of audio spectra classification, ECG classification, human activity recognition, motion classification, etc. 107 | 108 | 109 | ## Codes: 110 | The codes in ULTS library are organized as follows: 111 | - The [main.py](https://github.com/mqwfrog/ULTS/blob/main/main.py) includes the training method for all models. 112 | - The [models](https://github.com/mqwfrog/ULTS/tree/main/models) folder contain all 17 unsupervised learning models. 113 | - The [data_preprocess](https://github.com/mqwfrog/ULTS/tree/main/data_preprocess) folder contain the codes to preprocess data from different archives. 114 | - The [data_loader](https://github.com/mqwfrog/ULTS/tree/main/data_loader) folder contains the codes to perform augmentation transformations and to load the dataset. 115 | 116 | 117 | ## Running: 118 | 
119 | python main.py \
120 | --dataset_name wisdm \
121 | --uid SimCLR
122 | --lr 0.03 \
123 | --batch_size 128 \
124 | --feature_size 128
125 |
126 | 127 | 128 | 129 | ## Results: 130 | - The experimental results will be saved in "experiment_{args.model}_{args.dataset}" directory by default. 131 | 132 | 133 | ## Citation: 134 | If you find any of the codes helpful, kindly cite our paper. 135 | 136 | 
137 | @misc{meng2023unsupervised, 
138 |       title={Unsupervised Representation Learning for Time Series: A Review}, 
139 |       author={Qianwen Meng and Hangwei Qian and Yong Liu and Yonghui Xu and Zhiqi Shen and Lizhen Cui},   
140 |       year={2023},
141 |       eprint={2308.01578},
142 |       archivePrefix={arXiv},
143 |       primaryClass={cs.LG}
144 | }
145 |
146 | 147 | ## References: 148 | Part of the codes are referenced from 149 | https://github.com/PatrickHua/SimSiam 150 | https://github.com/facebookresearch/deepcluster 151 | https://github.com/TTN-YKK/Clustering_friendly_representation_learning 152 | https://github.com/paudan/TimeNet 153 | https://github.com/cauchyturing/Deconv_SAX 154 | https://github.com/jsyoon0823/TimeGAN 155 | https://github.com/abudesai/timeVAE 156 | https://github.com/joergsimon/SSL-ECG-Paper-Reimplementaton 157 | https://github.com/mlberkeley/eeg-ssl 158 | https://github.com/gzerveas/mvts_transformer 159 | https://github.com/google-research/simclr 160 | https://github.com/deepmind/deepmind-research/tree/master/byol 161 | https://github.com/lucidrains/byol-pytorch 162 | https://github.com/facebookresearch/CPC_audio 163 | https://github.com/abhinavagarwalla/swav-cifar10 164 | https://github.com/facebookresearch/swav 165 | https://github.com/salesforce/PCL 166 | https://github.com/mqwfrog/MHCCL 167 | https://github.com/yuezhihan/ts2vec 168 | https://github.com/emadeldeen24/TS-TCC 169 | https://github.com/White-Link/UnsupervisedScalableRepresentationLearningTimeSeries 170 | -------------------------------------------------------------------------------- /Supplementary Files: -------------------------------------------------------------------------------- 1 | Some supplementary files 2 | -------------------------------------------------------------------------------- /data_loader/augmentations.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | from scipy.interpolate import CubicSpline 4 | 5 | def DataTransform(sample): 6 | 7 | aug_1 = scaling(sample, sigma=1.1) 8 | aug_2 = jitter(permutation(sample, max_segments=5, seg_mode="random"), sigma=0.8) 9 | 10 | # You can choose any augmentation transformations or combinations listed as follows: 11 | # x_jitter = jitter(sample, sigma=0.8) #sigma: [0.01, 0.5] 12 | # x_scaling = scaling(sample, sigma=1.1) #sigma: [0.1, 2.] 13 | # x_permutation = permutation(sample, max_segments=5, seg_mode="random") #max_segments: [3., 6.] 14 | # x_rotation = rotation(sample) 15 | # x_magnitude_warp = magnitude_warp(sample, sigma=0.2, knot=4) #sigma: [0.1, 2.], knot: [3, 5] 16 | # x_time_warp = time_warp(sample, sigma=0.2, knot=4) #sigma: [0.01, 0.5], knot: [3, 5] 17 | # x_window_slice = window_slice(sample, reduce_ratio=0.9) #reduce_ratio: [0.95, 0.6] 18 | 19 | return aug_1, aug_2 20 | 21 | def jitter(x, sigma=0.8): 22 | x_new = x + np.random.normal(loc=0., scale=sigma, size=x.shape) 23 | return x_new 24 | 25 | 26 | def scaling(x, sigma = 0.1): 27 | scaler = np.random.normal(loc=1.0, scale = sigma, size = (1, x.shape[1])) 28 | noise = np.matmul(np.ones((x.shape[2], 1)), scaler) 29 | x = x.permute(0,2,1) 30 | x_new = x.clone() 31 | for i in range(0, x.shape[0]): 32 | x_new[i] = x[i] * noise 33 | return x_new.permute(0,2,1) 34 | 35 | 36 | def permutation(x, max_segments=5, seg_mode="random"): 37 | orig_steps = np.arange(x.shape[2]) 38 | num_segs = np.random.randint(1, max_segments, size=(x.shape[0])) 39 | x_new = np.zeros_like(x) 40 | for i, pat in enumerate(x): 41 | if num_segs[i] > 1: 42 | if seg_mode == "random": 43 | split_points = np.random.choice(x.shape[2] - 2, num_segs[i] - 1, replace=False) 44 | split_points.sort() 45 | splits = np.split(orig_steps, split_points) 46 | else: 47 | splits = np.array_split(orig_steps, num_segs[i]) 48 | warp = np.concatenate(np.random.permutation(splits)).ravel() 49 | x_new[i] = pat[0,warp] 50 | else: 51 | x_new[i] = pat 52 | return torch.from_numpy(x_new) 53 | 54 | 55 | def rotation(x): 56 | flip = np.random.choice([-1, 1], size=(x.shape[0],x.shape[2])) 57 | rotate_axis = np.arange(x.shape[2]) 58 | np.random.shuffle(rotate_axis) 59 | x_new = torch.from_numpy(flip)[:,np.newaxis,:] * x[:,:,rotate_axis] 60 | return x_new 61 | 62 | 63 | def magnitude_warp(x, sigma=0.2, knot=4): 64 | orig_steps = np.arange(x.shape[1]) 65 | random_warps = np.random.normal(loc=1.0, scale=sigma, size=(x.shape[0], knot + 2, x.shape[2])) 66 | warp_steps = (np.ones((x.shape[2], 1)) * (np.linspace(0, x.shape[1] - 1., num=knot + 2))).T 67 | x_new = np.zeros_like(x) 68 | for i, pat in enumerate(x): 69 | li = [] 70 | for dim in range(x.shape[2]): 71 | li.append(CubicSpline(warp_steps[:, dim], random_warps[i, :, dim])(orig_steps)) 72 | warper = np.array(li).T 73 | x_new[i] = pat * warper 74 | return torch.from_numpy(x_new) 75 | 76 | 77 | def time_warp(x, sigma=0.2, knot=4): 78 | orig_steps = np.arange(x.shape[1]) 79 | random_warps = np.random.normal(loc=1.0, scale=sigma, size=(x.shape[0], knot + 2, x.shape[2])) 80 | warp_steps = (np.ones((x.shape[2], 1)) * (np.linspace(0, x.shape[1] - 1., num=knot + 2))).T 81 | x_new = np.zeros_like(x) 82 | for i, pat in enumerate(x): 83 | for dim in range(x.shape[2]): 84 | time_warp = CubicSpline(warp_steps[:, dim], warp_steps[:, dim] * random_warps[i, :, dim])(orig_steps) 85 | scale = (x.shape[1] - 1) / time_warp[-1] 86 | x_new[i, :, dim] = np.interp(orig_steps, np.clip(scale * time_warp, 0, x.shape[1] - 1), pat[:, dim]).T 87 | return torch.from_numpy(x_new) 88 | 89 | 90 | def window_slice(x, reduce_ratio=0.9): 91 | target_len = np.ceil(reduce_ratio*x.shape[1]).astype(int) 92 | if target_len >= x.shape[1]: 93 | return x 94 | starts = np.random.randint(low=0, high=x.shape[1]-target_len, size=(x.shape[0])).astype(int) 95 | ends = (target_len + starts).astype(int) 96 | x_new = np.zeros_like(x) 97 | for i, pat in enumerate(x): 98 | for dim in range(x.shape[2]): 99 | x_new[i,:,dim] = np.interp(np.linspace(0, target_len, num=x.shape[1]), np.arange(target_len), pat[starts[i]:ends[i],dim]).T 100 | return x_new 101 | 102 | -------------------------------------------------------------------------------- /data_loader/dataset_loader.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.utils.data import DataLoader 3 | from torch.utils.data import Dataset 4 | import os 5 | import numpy as np 6 | from .augmentations import DataTransform 7 | 8 | 9 | class Load_Dataset(Dataset): 10 | # Initialize your data, download, etc. 11 | def __init__(self, dataset, config, training_mode): 12 | super(Load_Dataset, self).__init__() 13 | self.training_mode = training_mode 14 | 15 | X_train = dataset["samples"] 16 | y_train = dataset["labels"] 17 | 18 | if len(X_train.shape) < 3: 19 | X_train = X_train.unsqueeze(2) 20 | 21 | if X_train.shape.index(min(X_train.shape)) != 1: # make sure the Channels in second dim 22 | X_train = X_train.permute(0, 2, 1) 23 | 24 | if isinstance(X_train, np.ndarray): 25 | self.x_data = torch.from_numpy(X_train) 26 | self.y_data = torch.from_numpy(y_train).long() 27 | else: 28 | self.x_data = X_train 29 | self.y_data = y_train 30 | 31 | self.len = X_train.shape[0] 32 | if training_mode == "self_supervised": # no need to apply Augmentations in other modes 33 | self.aug1, self.aug2 = DataTransform(self.x_data, config) 34 | 35 | def __getitem__(self, index): 36 | if self.training_mode == "self_supervised": 37 | return self.x_data[index], self.y_data[index], self.aug1[index], self.aug2[index], index 38 | else: 39 | return self.x_data[index], self.y_data[index], self.x_data[index], self.x_data[index], index 40 | 41 | def __len__(self): 42 | return self.len 43 | 44 | 45 | def data_generator_all(data_path, configs, training_mode): 46 | 47 | train_dataset = torch.load(os.path.join(data_path, "train.pt")) 48 | valid_dataset = torch.load(os.path.join(data_path, "val.pt")) 49 | test_dataset = torch.load(os.path.join(data_path, "test.pt")) 50 | 51 | 52 | train_dataset = Load_Dataset(train_dataset, configs, training_mode) 53 | valid_dataset = Load_Dataset(valid_dataset, configs, training_mode) 54 | test_dataset = Load_Dataset(test_dataset, configs, training_mode) 55 | 56 | # print(len(train_dataset)) # HAR: 7352 , wisdm: 2617 57 | # print(len(valid_dataset)) # HAR: 1471, wisdm: 655 58 | # print(len(test_dataset)) # HAR: 2947, wisdm: 819 59 | 60 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=configs.batch_size, 61 | shuffle=True, drop_last=configs.drop_last, 62 | num_workers=0) 63 | 64 | valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=configs.batch_size, 65 | shuffle=False, drop_last=configs.drop_last, 66 | num_workers=0) 67 | 68 | test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=configs.batch_size, 69 | shuffle=False, drop_last=False, 70 | num_workers=0) 71 | 72 | return train_loader, valid_loader, test_loader 73 | 74 | 75 | def data_generator(data_path, configs, training_mode): 76 | 77 | train_dataset = torch.load(os.path.join(data_path, "train.pt")) 78 | test_dataset = torch.load(os.path.join(data_path, "test.pt")) 79 | 80 | 81 | train_dataset = Load_Dataset(train_dataset, configs, training_mode) 82 | test_dataset = Load_Dataset(test_dataset, configs, training_mode) 83 | 84 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=configs.batch_size, 85 | shuffle=True, drop_last=configs.drop_last, 86 | num_workers=0) 87 | 88 | test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=configs.batch_size, 89 | shuffle=False, drop_last=False, 90 | num_workers=0) 91 | 92 | return train_loader, test_loader 93 | -------------------------------------------------------------------------------- /data_preprocess/data_preprocess_HAR.py: -------------------------------------------------------------------------------- 1 | import random 2 | import torch 3 | import os 4 | import numpy as np 5 | from sklearn.model_selection import train_test_split 6 | 7 | data_dir = 'UCI HAR Dataset' 8 | output_dir = '../../data/HAR' 9 | 10 | subject_data = np.loadtxt(f'{data_dir}/train/subject_train.txt') 11 | # Samples 12 | train_acc_x = np.loadtxt(f'{data_dir}/train/Inertial Signals/body_acc_x_train.txt') 13 | train_acc_y = np.loadtxt(f'{data_dir}/train/Inertial Signals/body_acc_y_train.txt') 14 | train_acc_z = np.loadtxt(f'{data_dir}/train/Inertial Signals/body_acc_z_train.txt') 15 | train_gyro_x = np.loadtxt(f'{data_dir}/train/Inertial Signals/body_gyro_x_train.txt') 16 | train_gyro_y = np.loadtxt(f'{data_dir}/train/Inertial Signals/body_gyro_y_train.txt') 17 | train_gyro_z = np.loadtxt(f'{data_dir}/train/Inertial Signals/body_gyro_z_train.txt') 18 | train_tot_acc_x = np.loadtxt(f'{data_dir}/train/Inertial Signals/total_acc_x_train.txt') 19 | train_tot_acc_y = np.loadtxt(f'{data_dir}/train/Inertial Signals/total_acc_y_train.txt') 20 | train_tot_acc_z = np.loadtxt(f'{data_dir}/train/Inertial Signals/total_acc_z_train.txt') 21 | 22 | test_acc_x = np.loadtxt(f'{data_dir}/test/Inertial Signals/body_acc_x_test.txt') 23 | test_acc_y = np.loadtxt(f'{data_dir}/test/Inertial Signals/body_acc_y_test.txt') 24 | test_acc_z = np.loadtxt(f'{data_dir}/test/Inertial Signals/body_acc_z_test.txt') 25 | test_gyro_x = np.loadtxt(f'{data_dir}/test/Inertial Signals/body_gyro_x_test.txt') 26 | test_gyro_y = np.loadtxt(f'{data_dir}/test/Inertial Signals/body_gyro_y_test.txt') 27 | test_gyro_z = np.loadtxt(f'{data_dir}/test/Inertial Signals/body_gyro_z_test.txt') 28 | test_tot_acc_x = np.loadtxt(f'{data_dir}/test/Inertial Signals/total_acc_x_test.txt') 29 | test_tot_acc_y = np.loadtxt(f'{data_dir}/test/Inertial Signals/total_acc_y_test.txt') 30 | test_tot_acc_z = np.loadtxt(f'{data_dir}/test/Inertial Signals/total_acc_z_test.txt') 31 | 32 | # Stacking channels together data 33 | train_data = np.stack((train_acc_x, train_acc_y, train_acc_z, 34 | train_gyro_x, train_gyro_y, train_gyro_z, 35 | train_tot_acc_x, train_tot_acc_y, train_tot_acc_z), axis=1) 36 | X_test = np.stack((test_acc_x, test_acc_y, test_acc_z, 37 | test_gyro_x, test_gyro_y, test_gyro_z, 38 | test_tot_acc_x, test_tot_acc_y, test_tot_acc_z), axis=1) 39 | # labels 40 | train_labels = np.loadtxt(f'{data_dir}/train/y_train.txt') 41 | train_labels -= np.min(train_labels) 42 | y_test = np.loadtxt(f'{data_dir}/test/y_test.txt') 43 | y_test -= np.min(y_test) 44 | 45 | 46 | X_train, X_val, y_train, y_val = train_test_split(train_data, train_labels, test_size=0.2, random_state=42) 47 | 48 | dat_dict = dict() 49 | dat_dict["samples"] = torch.from_numpy(X_train) 50 | dat_dict["labels"] = torch.from_numpy(y_train) 51 | torch.save(dat_dict, os.path.join(output_dir, "train.pt")) 52 | 53 | dat_dict = dict() 54 | dat_dict["samples"] = torch.from_numpy(X_val) 55 | dat_dict["labels"] = torch.from_numpy(y_val) 56 | torch.save(dat_dict, os.path.join(output_dir, "val.pt")) 57 | 58 | dat_dict = dict() 59 | dat_dict["samples"] = torch.from_numpy(X_test) 60 | dat_dict["labels"] = torch.from_numpy(y_test) 61 | torch.save(dat_dict, os.path.join(output_dir, "test.pt")) 62 | -------------------------------------------------------------------------------- /data_preprocess/data_preprocess_SHAR.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import pandas as pd 4 | import torch 5 | from scipy.io import loadmat 6 | from pathlib import Path 7 | from typing import List, Tuple, Union, Optional 8 | from sklearn.model_selection import train_test_split 9 | from data_preprocessing.base import BaseDataset, check_path 10 | 11 | __all__ = ['UniMib', 'load', 'load_raw'] 12 | 13 | # Meta Info 14 | SUBJECTS = tuple(range(1, 30+1)) 15 | ACTIVITIES = tuple(['StandingUpFS', 'StandingUpFL', 'Walking', 'Running', 'GoingUpS', 'Jumping', 'GoingDownS', 'LyingDownFS', 'SittingDown', 'FallingForw', 'FallingRight', 'FallingBack', 'HittingObstacle', 'FallingWithPS', 'FallingBackSC', 'Syncope', 'FallingLeft']) 16 | GENDER = {'M': 0, 'F': 1} 17 | Sampling_Rate = 50 # Hz 18 | 19 | class UniMib(BaseDataset): 20 | def __init__(self, path:Path): 21 | super().__init__(path) 22 | 23 | def load(self, data_type:str, window_size:Optional[int]=None, stride:Optional[int]=None, ftrim_sec:int=3, btrim_sec:int=3, subjects:Optional[list]=None) -> Tuple[np.ndarray, np.ndarray]: 24 | if data_type != 'raw': 25 | data, meta = load(path=self.path, data_type=data_type) 26 | segments = {'acceleration': data, 'activity': meta['activity'], 'subject': meta['subject']} 27 | x = np.stack(segments['acceleration']).transpose(0, 2, 1) 28 | y = np.stack(segments['activity']) 29 | x_frames = x 30 | y -= np.min(y) 31 | y_frames = y 32 | 33 | # subject filtering 34 | if subjects is not None: 35 | flags = np.zeros(len(x_frames), dtype=bool) 36 | for sub in subjects: 37 | flags = np.logical_or(flags, y_frames[:, 1] == sub) 38 | x_frames = x_frames[flags] 39 | y_frames = y_frames[flags] 40 | return x_frames, y_frames 41 | 42 | 43 | def load(path:Union[Path,str], data_type:str='full') -> Tuple[List[pd.DataFrame], pd.DataFrame]: 44 | path = check_path(path) 45 | raw = load_raw(path, data_type) 46 | data, meta = reformat(raw) 47 | return data, meta 48 | 49 | 50 | def load_raw(path:Path, data_type:str = 'full') -> Union[Tuple[np.ndarray, pd.DataFrame], Tuple[List[pd.DataFrame], pd.DataFrame]]: 51 | if not isinstance(data_type, str): 52 | raise TypeError('expected type of "type" argument is str, but {}'.format(type(data_type))) 53 | if data_type not in ['full', 'adl', 'fall']: 54 | raise ValueError('unknown data type, {}'.format(data_type)) 55 | if data_type == 'full': 56 | prefix = 'acc' 57 | elif data_type == 'adl': 58 | prefix = 'adl' 59 | elif data_type == 'fall': 60 | prefix = 'fall' 61 | if data_type != 'raw': 62 | data = loadmat(str(path / f'{prefix}_data.mat'))[f'{prefix}_data'].reshape([-1, 3, 151]) 63 | labels = loadmat(str(path / f'{prefix}_labels.mat'))[f'{prefix}_labels'] 64 | meta = labels 65 | meta = pd.DataFrame(meta, columns=['activity', 'subject', 'trial_id']) 66 | meta = meta.astype({'activity': np.int8, 'subject': np.int8, 'trial_id': np.int8}) 67 | return data, meta 68 | 69 | 70 | def reformat(raw) -> Tuple[List[pd.DataFrame], pd.DataFrame]: 71 | data, meta = raw 72 | data = list(map(lambda x: pd.DataFrame(x.T, columns=['x', 'y', 'z']), data)) 73 | return data, meta 74 | 75 | 76 | if __name__ == '__main__': 77 | 78 | output_dir = r'../../data/SHAR' 79 | unimib_path = Path('./data') 80 | unimib = UniMib(unimib_path) 81 | x, y = unimib.load(data_type='full') 82 | 83 | X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42) 84 | X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42) 85 | 86 | dat_dict = dict() 87 | dat_dict["samples"] = torch.from_numpy(X_train) 88 | dat_dict["labels"] = torch.from_numpy(y_train) 89 | torch.save(dat_dict, os.path.join(output_dir, "train.pt")) 90 | 91 | dat_dict = dict() 92 | dat_dict["samples"] = torch.from_numpy(X_val) 93 | dat_dict["labels"] = torch.from_numpy(y_val) 94 | torch.save(dat_dict, os.path.join(output_dir, "val.pt")) 95 | 96 | dat_dict = dict() 97 | dat_dict["samples"] = torch.from_numpy(X_test) 98 | dat_dict["labels"] = torch.from_numpy(y_test) 99 | torch.save(dat_dict, os.path.join(output_dir, "test.pt")) 100 | -------------------------------------------------------------------------------- /data_preprocess/data_preprocess_UEA.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | import numpy as np 4 | import torch 5 | from tqdm import tqdm 6 | from sktime.utils.data_io import load_from_arff_to_dataframe 7 | from sklearn.preprocessing import LabelEncoder 8 | 9 | # original UEA(0,1,2) [instances, length, features/channels] 10 | # UEA(0,1,2) --> later will be permuted in dataloader-->get UEA(0,2,1) [instances, features/channels, length] 11 | 12 | DATA_DIR = '../data' 13 | def mkdir_if_not_exists(loc, file=False): 14 | loc_ = os.path.dirname(loc) if file else loc 15 | if not os.path.exists(loc): 16 | os.makedirs(loc_, exist_ok=True) 17 | 18 | def create_torch_data(train_file, test_file): 19 | # Get arff format 20 | train_data, train_labels = load_from_arff_to_dataframe(train_file) 21 | test_data, test_labels = load_from_arff_to_dataframe(test_file) 22 | 23 | def convert_data(data): 24 | # Expand the series to numpy 25 | data_expand = data.applymap(lambda x: x.values).values 26 | # Single array, then to tensor 27 | data_numpy = np.stack([np.vstack(x).T for x in data_expand]) 28 | tensor_data = torch.Tensor(data_numpy) 29 | return tensor_data 30 | 31 | train_data, test_data = convert_data(train_data), convert_data(test_data) 32 | 33 | # Encode labels as often given as strings 34 | encoder = LabelEncoder().fit(train_labels) 35 | train_labels, test_labels = encoder.transform(train_labels), encoder.transform(test_labels) 36 | train_labels, test_labels = torch.Tensor(train_labels), torch.Tensor(test_labels) 37 | 38 | return train_data, test_data, train_labels, test_labels 39 | def save_pickle(obj, filename, protocol=4, create_folder=True): 40 | if create_folder: 41 | mkdir_if_not_exists(filename, file=True) 42 | 43 | # Save 44 | with open(filename, 'wb') as file: 45 | pickle.dump(obj, file, protocol=protocol) 46 | 47 | def convert_all_files(dataset='uea'): 48 | """ Convert all files from a given /raw/{subfolder} into torch data to be stored in /interim. """ 49 | assert dataset in ['uea', 'ucr'] 50 | folder = 'UEA' 51 | arff_folder = DATA_DIR + '/raw/{}/Multivariate_arff'.format(folder) 52 | 53 | # Time for a big for loop 54 | for ds_name in tqdm([x for x in os.listdir(arff_folder) if os.path.isdir(arff_folder + '/' + x)]): 55 | # File locations 56 | print(f'ds_name:{ds_name}') 57 | train_file = arff_folder + '/{}/{}_TRAIN.arff'.format(ds_name, ds_name) 58 | test_file = arff_folder + '/{}/{}_TEST.arff'.format(ds_name, ds_name) 59 | 60 | # Ready save dir 61 | save_dir = DATA_DIR + '/processed/{}/{}'.format(folder, ds_name) 62 | print(f'save_dir:{save_dir}') 63 | # If files don't exist, skip. 64 | if any([x.split('/')[-1] not in os.listdir(arff_folder + '/{}'.format(ds_name)) for x in (train_file, test_file)]): 65 | if ds_name not in ['Images', 'Descriptions']: 66 | print('No files found for folder: {}'.format(ds_name)) 67 | continue 68 | # elif os.path.isdir(save_dir): 69 | # print('Files already exist for: {}'.format(ds_name)) 70 | # continue 71 | else: 72 | train_data, test_data, train_labels, test_labels = create_torch_data(train_file, test_file) 73 | 74 | dat_dict = dict() 75 | dat_dict["samples"] = train_data 76 | dat_dict["labels"] = train_labels 77 | torch.save(dat_dict, ds_name+"_train.pt") 78 | 79 | dat_dict = dict() 80 | dat_dict["samples"] = test_data 81 | dat_dict["labels"] = test_labels 82 | torch.save(dat_dict, ds_name+"_test.pt") 83 | 84 | # # Compile train and test data together 85 | # data = torch.cat([train_data, test_data]) 86 | # labels = torch.cat([train_labels, test_labels]) 87 | # 88 | # # Save original train test indexes in case we wish to use original splits 89 | # original_idxs = (np.arange(0, train_data.size(0)), np.arange(train_data.size(0), data.size(0))) 90 | 91 | # # Save data 92 | # save_pickle(data, save_dir + '/data.pkl') 93 | # save_pickle(labels, save_dir + '/labels.pkl') 94 | # save_pickle(original_idxs, save_dir + '/original_idxs.pkl') 95 | 96 | 97 | if __name__ == '__main__': 98 | dataset = 'uea' 99 | convert_all_files(dataset) 100 | -------------------------------------------------------------------------------- /data_preprocess/data_preprocess_wisdm.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | import re 4 | import torch 5 | import numpy as np 6 | import pandas as pd 7 | import pickle 8 | import scipy 9 | import scipy.interpolate 10 | import pathlib 11 | import typing 12 | import errno 13 | from pathlib import Path 14 | from typing import Optional, Union, List, Tuple 15 | from sklearn.model_selection import train_test_split 16 | 17 | __all__ = ['WISDM', 'load', 'load_raw'] 18 | 19 | # Meta Info 20 | SUBJECTS = tuple(range(1, 36+1)) 21 | ACTIVITIES = tuple(['Walking', 'Jogging', 'Sitting', 'Standing', 'Upstairs', 'Downstairs']) 22 | Sampling_Rate = 20 # Hz 23 | 24 | class BaseDataset(object): 25 | def __init__(self, path: Path): 26 | self.path = path 27 | def load(self, *args): 28 | raise NotImplementedError 29 | 30 | class WISDM(BaseDataset): 31 | def __init__(self, path:Path): 32 | super().__init__(path) 33 | 34 | def load(self, window_size:int, stride:int, ftrim_sec:int=3, btrim_sec:int=3, subjects:Optional[list]=None) -> Tuple[np.ndarray, np.ndarray]: 35 | segments, meta = load(path=self.path) 36 | segments = [m.join(seg) for seg, m in zip(segments, meta)] 37 | 38 | x_frames, y_frames = [], [] 39 | for seg in segments: 40 | fs = split_using_sliding_window( 41 | np.array(seg), window_size=window_size, stride=stride, 42 | ftrim=Sampling_Rate*ftrim_sec, btrim=Sampling_Rate*btrim_sec, 43 | return_error_value=None) 44 | if fs is not None: 45 | x_frames += [fs[:, :, 3:]] 46 | y_frames += [np.uint8(fs[:, 0, 1:2][..., ::-1])] 47 | else: 48 | pass 49 | x_frames = np.concatenate(x_frames).transpose([0, 2, 1]) 50 | y_frames = np.concatenate(y_frames) 51 | y_frames -= np.min(y_frames) 52 | y_frames = y_frames.squeeze(1) 53 | 54 | # subject filtering 55 | if subjects is not None: 56 | flags = np.zeros(len(x_frames), dtype=bool) 57 | for sub in subjects: 58 | flags = np.logical_or(flags, y_frames[:, 1] == sub) 59 | x_frames = x_frames[flags] 60 | y_frames = y_frames[flags] 61 | 62 | return x_frames, y_frames 63 | 64 | def check_path(path: Union[Path, str]) -> Path: 65 | if isinstance(path, str): 66 | path = Path(path) 67 | elif not isinstance(path, Path): 68 | raise TypeError('expected type of "path" is Path or str, but {}'.format(type(path))) 69 | if not path.exists(): 70 | raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), str(path)) 71 | return path 72 | 73 | def to_frames_using_reshape(src: np.ndarray, window_size: int) -> np.ndarray: 74 | num_frames = (src.shape[0] - window_size) // window_size + 1 75 | ret = src[:(num_frames * window_size)] 76 | return ret.reshape(-1, window_size, *src.shape[1:]) 77 | 78 | def to_frames(src: np.ndarray, window_size: int, stride: int, stride_mode: str = 'index') -> np.ndarray: 79 | assert stride > 0, 'stride={}'.format(stride) 80 | assert stride_mode in ['index', 'nptrick'], "stride_mode is 'index' or 'nptrick'. stride_mode={}".format( 81 | stride_mode) 82 | if stride == window_size: 83 | return to_frames_using_reshape(src, window_size) 84 | elif stride_mode == 'index': 85 | return to_frames_using_index(src, window_size, stride) 86 | else: 87 | return to_frames_using_nptricks(src, window_size, stride) 88 | 89 | def to_frames_using_index(src: np.ndarray, window_size: int, stride: int) -> np.ndarray: 90 | assert stride > 0, 'stride={}'.format(stride) 91 | num_frames = (len(src) - window_size) // stride + 1 92 | idx = np.arange(window_size).reshape(-1, window_size).repeat(num_frames, axis=0) + np.arange( 93 | num_frames).reshape(num_frames, 1) * stride 94 | return src[idx] 95 | 96 | def to_frames_using_nptricks(src: np.ndarray, window_size: int, stride: int) -> np.ndarray: 97 | 98 | assert stride > 0, 'stride={}'.format(stride) 99 | num_frames = (src.shape[0] - window_size) // stride + 1 100 | ret_shape = (num_frames, window_size, *src.shape[1:]) 101 | strides = (stride * src.strides[0], *src.strides) 102 | return np.lib.stride_tricks.as_strided(src, shape=ret_shape, strides=strides) 103 | 104 | def split_using_sliding_window(segment: np.ndarray, **options) -> np.ndarray: 105 | assert len(segment.shape) == 2, "Segment's shape is (segment_size, ch). This segment shape is {}".format( 106 | segment.shape) 107 | window_size = options.pop('window_size', 512) 108 | stride = options.pop('stride', None) 109 | ftrim = options.pop('ftrim', 5) 110 | btrim = options.pop('btrim', 5) 111 | return_error_value = options.pop('return_error_value', None) 112 | assert not bool(options), "args error: key {} is not exist.".format(list(options.keys())) 113 | assert type(window_size) is int, "type(window_size) is int: {}".format(type(window_size)) 114 | assert ftrim >= 0 and btrim >= 0, "ftrim >= 0 and btrim >= 0: ftrim={}, btrim={}".format(ftrim, btrim) 115 | if type(segment) is not np.ndarray: 116 | return return_error_value 117 | if len(segment) < ftrim + btrim: 118 | return return_error_value 119 | if btrim == 0: 120 | seg = segment[ftrim:].copy() 121 | else: 122 | seg = segment[ftrim: -btrim].copy() 123 | if len(seg) < window_size: 124 | return return_error_value 125 | if stride is None: 126 | stride = window_size 127 | return to_frames(seg, window_size, stride, stride_mode='index') 128 | 129 | def split_using_target(src: np.ndarray, target: np.ndarray) -> typing.Dict[int, typing.List[np.ndarray]]: 130 | from collections import defaultdict 131 | rshifted = np.roll(target, 1) 132 | diff = target - rshifted 133 | diff[0] = 1 134 | idxes = np.where(diff != 0)[0] 135 | 136 | ret = defaultdict(list) 137 | for i in range(1, len(idxes)): 138 | ret[target[idxes[i - 1]]].append(src[idxes[i - 1]:idxes[i]].copy()) 139 | ret[target[idxes[-1]]].append(src[idxes[-1]:].copy()) 140 | return dict(ret) 141 | 142 | def interpolate(src: np.ndarray, rate: int, kind: str = 'linear', axis: int = -1) -> np.ndarray: 143 | N = src.shape[axis] 144 | x_low = np.linspace(0, 1, N) 145 | x_target = np.linspace(0, 1, N + (N - 1) * (rate - 1)) 146 | f = scipy.interpolate.interp1d(x_low, src, kind=kind, axis=axis) 147 | return f(x_target) 148 | 149 | def pickle_dump(obj: typing.Any, path: typing.Union[str, pathlib.Path]) -> None: 150 | with open(path, mode='wb') as f: 151 | pickle.dump(obj, f) 152 | return 153 | 154 | def pickle_load(path: pathlib.Path) -> typing.Any: 155 | with open(path, mode='rb') as f: 156 | data = pickle.load(f) 157 | return data 158 | 159 | def load(path:Union[Path,str]) -> Tuple[List[pd.DataFrame], List[pd.DataFrame]]: 160 | path = check_path(path) 161 | raw = load_raw(path) 162 | data, meta = reformat(raw) 163 | return data, meta 164 | 165 | def load_raw(path:Path) -> pd.DataFrame: 166 | path = path / 'WISDM_ar_v1.1_raw.txt' 167 | with path.open('r') as fp: 168 | whole_str = fp.read() 169 | 170 | whole_str = whole_str.replace(',;', ';') 171 | semi_separated = re.split('[;\n]', whole_str) 172 | semi_separated = list(filter(lambda x: x != '', semi_separated)) 173 | comma_separated = [r.strip().split(',') for r in semi_separated] 174 | 175 | # debug 176 | for s in comma_separated: 177 | if len(s) != 6: 178 | print('[miss format?]: {}'.format(s)) 179 | 180 | raw_data = pd.DataFrame(comma_separated) 181 | raw_data.columns = ['user', 'activity', 'timestamp', 'x-acceleration', 'y-acceleration', 'z-acceleration'] 182 | raw_data['z-acceleration'] = raw_data['z-acceleration'].replace('', np.nan) 183 | 184 | # convert activity name to activity id 185 | raw_data = raw_data.replace(list(ACTIVITIES), list(range(len(ACTIVITIES)))) 186 | 187 | raw_data = raw_data.astype({'user': 'uint8', 'activity': 'uint8', 'timestamp': 'uint64', 'x-acceleration': 'float64', 'y-acceleration': 'float64', 'z-acceleration': 'float64'}) 188 | raw_data[['x-acceleration', 'y-acceleration', 'z-acceleration']] = raw_data[['x-acceleration', 'y-acceleration', 'z-acceleration']].fillna(method='ffill') 189 | 190 | return raw_data 191 | 192 | 193 | def reformat(raw) -> Tuple[List[pd.DataFrame], List[pd.DataFrame]]: 194 | raw_array = raw.to_numpy() 195 | 196 | # segment (by user and activity) 197 | sdata_splited_by_subjects = split_using_target(src=raw_array, target=raw_array[:, 0]) 198 | segments = [] 199 | for sub_id in sdata_splited_by_subjects.keys(): 200 | for src in sdata_splited_by_subjects[sub_id]: 201 | splited = split_using_target(src=src, target=src[:, 1]) 202 | for act_id in splited.keys(): 203 | segments += splited[act_id] 204 | 205 | segments = list(map(lambda seg: pd.DataFrame(seg, columns=raw.columns).astype(raw.dtypes.to_dict()), segments)) 206 | data = list(map(lambda seg: pd.DataFrame(seg.iloc[:, 3:], columns=raw.columns[3:]), segments)) 207 | meta = list(map(lambda seg: pd.DataFrame(seg.iloc[:, :3], columns=raw.columns[:3]), segments)) 208 | 209 | return data, meta 210 | 211 | if __name__ == '__main__': 212 | 213 | output_dir = r'../../data/wisdm' 214 | 215 | wisdm_path = Path('./') 216 | wisdm = WISDM(wisdm_path) 217 | 218 | x, y = wisdm.load(window_size=256, stride=256, ftrim_sec=0, btrim_sec=0) 219 | 220 | X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42) 221 | X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42) 222 | 223 | train_sample_num = int(1.0 * len(X_train)) # select x% data 224 | train_sample_list = [i for i in range(len(X_train))] 225 | train_sample_list = random.sample(train_sample_list, train_sample_num) 226 | X_train = X_train[train_sample_list, :] 227 | y_train = y_train[train_sample_list] 228 | 229 | test_sample_num = int(1.0 * len(X_test)) # select x% data 230 | test_sample_list = [i for i in range(len(X_test))] 231 | test_sample_list = random.sample(test_sample_list, test_sample_num) 232 | X_test = X_test[test_sample_list, :] 233 | y_test = y_test[test_sample_list] 234 | 235 | dat_dict = dict() 236 | dat_dict["samples"] = torch.from_numpy(X_train) 237 | dat_dict["labels"] = torch.from_numpy(y_train) 238 | torch.save(dat_dict, os.path.join(output_dir, "train.pt")) 239 | 240 | dat_dict = dict() 241 | dat_dict["samples"] = torch.from_numpy(X_val) 242 | dat_dict["labels"] = torch.from_numpy(y_val) 243 | torch.save(dat_dict, os.path.join(output_dir, "val.pt")) 244 | 245 | dat_dict = dict() 246 | dat_dict["samples"] = torch.from_numpy(X_test) 247 | dat_dict["labels"] = torch.from_numpy(y_test) 248 | torch.save(dat_dict, os.path.join(output_dir, "test.pt")) 249 | 250 | 251 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import torch 3 | 4 | from models.SimCLR.train_SimCLR import * 5 | from models.BYOL.train_BYOL import * 6 | from models.CPC.train_CPC import * 7 | from models.SwAV.train_SwAV import * 8 | from models.PCL.train_PCL import * 9 | from models.MHCCL.train_MHCCL import * 10 | from models.TS2Vec.train_TS2Vec import * 11 | from models.TSTCC.train_TSTCC import * 12 | from models.TLoss.train_TLoss import * 13 | from models.TST.train_TST import * 14 | from models.train_DeepCluster import * 15 | from models.train_IDFD import * 16 | 17 | 18 | parser = argparse.ArgumentParser() 19 | 20 | # models 21 | parser.add_argument('--uid', type=str, default='SimCLR', 22 | help='Staging identifier (default: SimCLR)') 23 | 24 | # datasets 25 | parser.add_argument('--dataset_name', default='wisdm', type=str, 26 | help='Choice of Datasets: wisdm, epilepsy, HAR, SHAR, etc.') 27 | parser.add_argument('--feature_size', type=int, default=128, 28 | help='Feature output size (default: 128') 29 | parser.add_argument('--batch_size', type=int, default=64, metavar='N', 30 | help='input training batch-size') 31 | 32 | # hyperparameters 33 | parser.add_argument('--accumulation_steps', type=int, default=4, metavar='N', 34 | help='Gradient accumulation steps (default: 4') 35 | parser.add_argument('--epochs', type=int, default=5, metavar='N', 36 | help='number of training epochs (default: 150)') 37 | parser.add_argument('--seed', default=0, type=int, 38 | help='seed value') 39 | parser.add_argument('--lr', type=float, default=1e-3, 40 | help='learning rate (default: 1e-3') 41 | parser.add_argument("--decay_lr", default=1e-6, action="store", type=float, 42 | help='Learning rate decay (default: 1e-6') 43 | parser.add_argument('--tau', default=0.5, type=float, 44 | help='Tau temperature smoothing (default 0.5)') 45 | 46 | # gpu 47 | parser.add_argument('--no_cuda', action='store_true', default=False, 48 | help='disables cuda (default: False') 49 | parser.add_argument('--device_id', type=int, default=0, 50 | help='GPU device id (default: 0') 51 | 52 | # directions 53 | parser.add_argument('--data_dir', type=str, default='data', 54 | help='Path to dataset (default: data') 55 | parser.add_argument('--exp_dir', default='exp1', type=str, 56 | help='Logs and checkpoints of experiments') 57 | 58 | # modes 59 | parser.add_argument('--training_mode', default='self_supervised', type=str, 60 | help='Modes of choice: random_init, supervised, self_supervised, fine_tune, train_linear') 61 | parser.add_argument('--load_model', type=str, default=None, 62 | help='Load model to resume training for (default None)') 63 | 64 | # SwAV 65 | parser.add_argument("--epsilon", default=0.05, type=float, 66 | help="regularization parameter for Sinkhorn-Knopp algorithm") 67 | parser.add_argument("--sinkhorn_iterations", default=3, type=int, 68 | help="number of iterations in Sinkhorn-Knopp algorithm") 69 | parser.add_argument("--nmb_prototypes", default=1000, type=int, 70 | help="number of prototypes") 71 | parser.add_argument("--queue_length", type=int, default=0, 72 | help="length of the queue (0 for no queue)") 73 | parser.add_argument("--epoch_queue_starts", type=int, default=15, 74 | help="from this epoch, we start using a queue") 75 | parser.add_argument("--final_lr", type=float, default=0.0006, help="final learning rate") 76 | parser.add_argument("--freeze_prototypes_niters", default=900, type=int, 77 | help="freeze the prototypes during this many iterations from the start") 78 | parser.add_argument("--warmup_epochs", default=10, type=int, help="number of warmup epochs to only train with InfoNCE loss") 79 | parser.add_argument("--start_warmup", default=0, type=float, 80 | help="initial warmup learning rate") 81 | parser.add_argument("--dist_url", default="env://", type=str, help="""url used to set up distributed 82 | training; see https://pytorch.org/docs/stable/distributed.html""") 83 | parser.add_argument("--world_size", default=-1, type=int, help=""" 84 | number of processes: it is set automatically and 85 | should not be passed as argument""") 86 | parser.add_argument("--rank", default=0, type=int, help="""rank of this process: 87 | it is set automatically and should not be passed as argument""") 88 | parser.add_argument("--local_rank", default=0, type=int, 89 | help="this argument is not used and should be ignored") 90 | parser.add_argument("--arch", default="resnet50", type=str, help="convnet architecture") 91 | parser.add_argument("--hidden_mlp", default=64, type=int, 92 | help="hidden layer dimension in projection head") #2048->64 93 | parser.add_argument("--workers", default=10, type=int, 94 | help="number of data loading workers") 95 | parser.add_argument("--checkpoint_freq", type=int, default=25, 96 | help="Save the model periodically") 97 | parser.add_argument("--use_fp16", type=bool_flag, default=True, 98 | help="whether to train with mixed precision or not") 99 | parser.add_argument("--sync_bn", type=str, default="pytorch", help="synchronize bn") 100 | 101 | # PCL 102 | parser.add_argument('--master_port', type=str, default='29501', 103 | help='avoid address already in use') 104 | parser.add_argument('--schedule', default=[120, 160], nargs='*', type=int, 105 | help='learning rate schedule (when to drop lr by 10x)') 106 | parser.add_argument('--print_freq', default=100, type=int, 107 | metavar='N', help='print frequency (default: 10)') 108 | parser.add_argument('--multiprocessing_distributed', action='store_true', 109 | help='Use multi-processing distributed training to launch ' 110 | 'N processes per node, which has N GPUs. This is the ' 111 | 'fastest way to use PyTorch for either single node or ' 112 | 'multi node data parallel training') 113 | parser.add_argument('--dist_backend', default='nccl', type=str, 114 | help='distributed backend') 115 | parser.add_argument('--pcl_r', default=4, type=int, 116 | help='queue size; number of negative pairs; needs to be smaller than num_cluster (default: 16384)') 117 | parser.add_argument('--momentum', default=0.9, type=float, metavar='M', 118 | help='momentum of SGD solver') 119 | parser.add_argument('--moco_m', default=0.999, type=float, 120 | help='moco momentum of updating key encoder (default: 0.999)') 121 | parser.add_argument('--mlp', action='store_true', 122 | help='use mlp head') 123 | parser.add_argument('--cos', action='store_true', 124 | help='use cosine lr schedule') 125 | parser.add_argument('--num_cluster', default='6,12,24', type=str, 126 | help='number of clusters') 127 | 128 | 129 | # MHCCL 130 | parser.add_argument('--posi', default=3, type=int, 131 | help='number of positive instance pairs (default: 3)') 132 | parser.add_argument('--negi', default=4, type=int, 133 | help='number of negative instance pairs(default: 4)') 134 | parser.add_argument('--posp', default=3, type=int, 135 | help='number of positive prototype pairs (default: 3)') 136 | parser.add_argument('--negp', default=4, type=int, 137 | help='number of negative prototype pairs(default: 4)') 138 | parser.add_argument('--tempi', default=0.2, type=float, 139 | help='softmax temperature for instances') 140 | parser.add_argument('--tempp', default=0.3, type=float, 141 | help='softmax temperature for prototypes') 142 | parser.add_argument('--layers', default=3, type=int, 143 | help='save the results of bottom # layers (default 3 for wisdm)') 144 | parser.add_argument('--req_clust', default=None, type=int, 145 | help='specify the number of clusters ') 146 | parser.add_argument('--protoNCE_only', action="store_true", 147 | help='use protoNCE loss only ') 148 | parser.add_argument('--mask_layer0', action="store_true", 149 | help='mask points and recompute centroids at the bottom layer 0') 150 | parser.add_argument('--mask_others', action="store_true", 151 | help='mask points and recompute centroids at all top layers') 152 | parser.add_argument('--replace_centroids', action="store_true", 153 | help='replace computed prototypes with raw data') 154 | parser.add_argument('--usetemp', action="store_true", 155 | help='adopt temperature in loss') 156 | parser.add_argument('--mask_mode', default='mask_farthest', type=str, choices=['mask_farthest', 'mask_threshold', 'mask_proportion'], 157 | help='select the mask mode (default: mask_farthest, other values:' 158 | 'mask_threshold(if use, specify the dist_threshold), ' 159 | 'mask_proportion(if use, specify the proportion') 160 | parser.add_argument('--dist_threshold', default=0.3, type=float, 161 | help='specify the distance threshold beyond which points will be masked ' 162 | 'when select the mask_threshold mode') 163 | parser.add_argument('--proportion', default=0.5, type=float, 164 | help='specify the proportion of how much points far from the centroids will be masked ' 165 | 'when select the mask_proportion mode') 166 | 167 | 168 | # TS2Vec 169 | parser.add_argument('--archive', type=str, required=True, help='The archive name that the dataset belongs to. This can be set to UCR, UEA, forecast_csv, or forecast_csv_univar') 170 | parser.add_argument('--max_train_length', type=int, default=3000, help='For sequence with a length greater than , it would be cropped into some sequences, each of which has a length less than (defaults to 3000)') 171 | parser.add_argument('--iters', type=int, default=None, help='The number of iterations') 172 | parser.add_argument('--save_every', type=int, default=None, help='Save the checkpoint every iterations/epochs') 173 | parser.add_argument('--max_threads', type=int, default=None, help='The maximum allowed number of threads used by this process') 174 | parser.add_argument('--eval', action="store_true", help='Whether to perform evaluation after training') 175 | parser.add_argument('--irregular', type=float, default=0, help='The ratio of missing observations (defaults to 0)') 176 | 177 | 178 | # TLoss 179 | parser.add_argument('--hyper', type=str, metavar='FILE', required=True, 180 | help='path of the file of hyperparameters to use ' + 181 | 'for training; must be a JSON file') 182 | parser.add_argument('--load', action='store_true', default=False, 183 | help='activate to load the estimator instead of ' + 184 | 'training it') 185 | parser.add_argument('--fit_classifier', action='store_true', default=False, 186 | help='if not supervised, activate to load the ' + 187 | 'model and retrain the classifier') 188 | 189 | if __name__ == '__main__': 190 | args=parser.parse_args() 191 | 192 | if args.uid == 'SimCLR': 193 | train_SimCLR(parser).excute() 194 | 195 | elif args.uid == 'BYOL': 196 | train_BYOL(parser).excute() 197 | 198 | elif args.uid == 'CPC': 199 | train_CPC(parser).excute() 200 | 201 | elif args.uid == 'SwAV': 202 | train_SwAV(parser).excute() 203 | 204 | elif args.uid == 'PCL': 205 | train_PCL(parser).excute() 206 | 207 | elif args.uid == 'MHCCL': 208 | train_MHCCL(parser).excute() 209 | 210 | elif args.uid == 'TS2Vec': 211 | train_TS2Vec(parser).excute() 212 | 213 | elif args.uid == 'TSTCC': 214 | train_TSTCC(parser).excute() 215 | 216 | elif args.uid == 'TLoss': 217 | train_TLoss(parser).excute() 218 | 219 | elif args.uid == 'TST': 220 | train_TST(parser).excute() 221 | 222 | elif args.uid == 'DeepCluster': 223 | train_DeepCluster(parser).excute() 224 | 225 | elif args.uid == 'IDFD': 226 | train_IDFD(parser).excute() 227 | 228 | 229 | 230 | 231 | -------------------------------------------------------------------------------- /models/BYOL/mlp_head.py: -------------------------------------------------------------------------------- 1 | from torch import nn 2 | 3 | 4 | class MLPHead(nn.Module): 5 | def __init__(self, in_channels, mlp_hidden_size, projection_size): 6 | super(MLPHead, self).__init__() 7 | 8 | self.net = nn.Sequential( 9 | nn.Linear(in_channels, mlp_hidden_size), 10 | nn.BatchNorm1d(mlp_hidden_size), 11 | nn.ReLU(inplace=True), 12 | nn.Linear(mlp_hidden_size, projection_size) 13 | ) 14 | 15 | def forward(self, x): 16 | return self.net(x) 17 | -------------------------------------------------------------------------------- /models/BYOL/resnet_base_network.py: -------------------------------------------------------------------------------- 1 | import torchvision.models as models 2 | import torch 3 | from models.mlp_head import MLPHead 4 | from torch import nn 5 | from config_files.epilepsy_Configs import Config as Configs 6 | 7 | class ResNet18(torch.nn.Module): 8 | def __init__(self, *args, **kwargs): 9 | super(ResNet18, self).__init__() 10 | if kwargs['name'] == 'resnet18': 11 | resnet = models.resnet18(pretrained=False) 12 | # resnet.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, 13 | # bias=False) 14 | resnet.conv1 = nn.Conv2d(1, 64, kernel_size=8, stride=1, padding=43, 15 | bias=False) #epilepsy 16 | # resnet.conv1 = nn.Conv2d(9, 64, kernel_size=8, stride=1, padding=4, 17 | # bias=False) #HAR 18 | # resnet.conv1 = nn.Conv2d(3, 64, kernel_size=8, stride=1, padding=4, 19 | # bias=False) # SHAR 20 | # resnet.conv1 = nn.Conv2d(3, 64, kernel_size=8, stride=1, padding=4, 21 | # bias=False) # wisdm 22 | print(resnet) 23 | elif kwargs['name'] == 'resnet50': 24 | resnet = models.resnet50(pretrained=False) 25 | 26 | self.encoder = torch.nn.Sequential(*list(resnet.children())[:-1]) 27 | self.projetion = MLPHead(in_channels=resnet.fc.in_features, **kwargs['projection_head']) 28 | 29 | def forward(self, x): 30 | h = self.encoder(x) 31 | h = h.view(h.shape[0], h.shape[1]) 32 | return self.projetion(h) 33 | -------------------------------------------------------------------------------- /models/BYOL/train_BYOL.py: -------------------------------------------------------------------------------- 1 | import os 2 | import torch 3 | import yaml 4 | import argparse 5 | from mlp_head import MLPHead 6 | from resnet_base_network import ResNet18 7 | from trainer import BYOLTrainer 8 | from ..data_loader.dataset_loader import data_generator 9 | 10 | torch.manual_seed(0) 11 | 12 | class train_BYOL(): 13 | 14 | def __init__(self,parser): 15 | self.args = parser.parse_args() 16 | self.dataset_name = self.args.dataset_name 17 | self.method = 'BYOL' 18 | self.training_mode = self.args.training_mode 19 | self.data_dir = self.args.data_dir 20 | 21 | 22 | def excute(self): 23 | config = yaml.load(open("./config/config.yaml", "r"), Loader=yaml.FullLoader) 24 | 25 | device = 'cuda' if torch.cuda.is_available() else 'cpu' 26 | print(f"Training with: {device}") 27 | 28 | from config_files.wisdm_Configs import Config as Configs 29 | configs = Configs() 30 | train_dl, valid_dl, test_dl = data_generator(os.path.join(self.data_dir, self.dataset_name), configs, self.training_mode) # train_linear # change-2 31 | 32 | # online network 33 | online_network = ResNet18(**config['network']).to(device) 34 | pretrained_folder = config['network']['fine_tune_from'] 35 | 36 | # load pre-trained model if defined 37 | if pretrained_folder: 38 | try: 39 | checkpoints_folder = os.path.join('./runs', pretrained_folder, 'checkpoints') 40 | 41 | # load pre-trained parameters 42 | load_params = torch.load(os.path.join(os.path.join(checkpoints_folder, 'model.pth')), 43 | map_location=torch.device(torch.device(device))) 44 | 45 | online_network.load_state_dict(load_params['online_network_state_dict']) 46 | 47 | except FileNotFoundError: 48 | print("Pre-trained weights not found. Training from scratch.") 49 | 50 | # predictor network 51 | predictor = MLPHead(in_channels=online_network.projetion.net[-1].out_features, 52 | **config['network']['projection_head']).to(device) 53 | 54 | # target encoder 55 | target_network = ResNet18(**config['network']).to(device) 56 | # target_network = ResNet18(configs, **config['network']).to(device) 57 | 58 | optimizer = torch.optim.SGD(list(online_network.parameters()) + list(predictor.parameters()), 59 | **config['optimizer']['params']) 60 | 61 | trainer = BYOLTrainer(online_network=online_network, 62 | target_network=target_network, 63 | optimizer=optimizer, 64 | predictor=predictor, 65 | device=device, 66 | **config['trainer']) 67 | 68 | trainer.train(train_dl) 69 | -------------------------------------------------------------------------------- /models/BYOL/trainer.py: -------------------------------------------------------------------------------- 1 | import os 2 | import torch 3 | import torch.nn.functional as F 4 | import torchvision 5 | from torch.utils.data.dataloader import DataLoader 6 | from torch.utils.tensorboard import SummaryWriter 7 | 8 | from utils import _create_model_training_folder 9 | 10 | 11 | class BYOLTrainer: 12 | def __init__(self, online_network, target_network, predictor, optimizer, device, **params): 13 | self.online_network = online_network 14 | self.target_network = target_network 15 | self.optimizer = optimizer 16 | self.device = device 17 | self.predictor = predictor 18 | self.max_epochs = params['max_epochs'] 19 | self.writer = SummaryWriter() 20 | self.m = params['m'] 21 | self.batch_size = params['batch_size'] 22 | self.num_workers = params['num_workers'] 23 | self.checkpoint_interval = params['checkpoint_interval'] 24 | _create_model_training_folder(self.writer, files_to_same=["./config/config.yaml", "main.py", 'trainer.py']) 25 | 26 | @torch.no_grad() 27 | def _update_target_network_parameters(self): 28 | """ 29 | Momentum update of the key encoder 30 | """ 31 | for param_q, param_k in zip(self.online_network.parameters(), self.target_network.parameters()): 32 | param_k.data = param_k.data * self.m + param_q.data * (1. - self.m) 33 | 34 | @staticmethod 35 | def regression_loss(x, y): 36 | x = F.normalize(x, dim=1) 37 | y = F.normalize(y, dim=1) 38 | return 2 - 2 * (x * y).sum(dim=-1) 39 | 40 | def initializes_target_network(self): 41 | # init momentum network as encoder net 42 | for param_q, param_k in zip(self.online_network.parameters(), self.target_network.parameters()): 43 | param_k.data.copy_(param_q.data) # initialize 44 | param_k.requires_grad = False # not update by gradient 45 | 46 | def train(self, train_dataset): 47 | train_loader = train_dataset 48 | 49 | niter = 0 50 | model_checkpoints_folder = os.path.join(self.writer.log_dir, 'checkpoints') 51 | 52 | self.initializes_target_network() 53 | 54 | for batch_idx, (data, target, batch_view_1, batch_view_2) in enumerate(train_loader): 55 | print(f"train_loader_len:{len(train_loader)}") 56 | data, target = data.float().to(self.device), target.long().to(self.device) 57 | batch_view_1, batch_view_2 = batch_view_1.float().to(self.device), batch_view_2.float().to(self.device) 58 | batch_view_1= batch_view_1.unsqueeze(3) 59 | batch_view_2 = batch_view_2.unsqueeze(3) 60 | # if niter == 0: 61 | # grid = torchvision.utils.make_grid(batch_view_1[:32]) 62 | # self.writer.add_image('views_1', grid, global_step=niter) 63 | 64 | # grid = torchvision.utils.make_grid(batch_view_2[:32]) 65 | # self.writer.add_image('views_2', grid, global_step=niter) 66 | print(batch_view_2.shape) #128,1,178,1 67 | print(batch_view_1.shape) # 128,1,178,1 68 | loss = self.update(batch_view_1, batch_view_2) 69 | self.writer.add_scalar('loss', loss, global_step=niter) 70 | 71 | self.optimizer.zero_grad() 72 | loss.backward() 73 | self.optimizer.step() 74 | 75 | self._update_target_network_parameters() # update the key encoder 76 | 77 | # save checkpoints 78 | self.save_model(os.path.join(model_checkpoints_folder, 'model.pth')) 79 | 80 | def update(self, batch_view_1, batch_view_2): 81 | # compute query feature 82 | predictions_from_view_1 = self.predictor(self.online_network(batch_view_1)) 83 | predictions_from_view_2 = self.predictor(self.online_network(batch_view_2)) 84 | 85 | # compute key features 86 | with torch.no_grad(): 87 | targets_to_view_2 = self.target_network(batch_view_1) 88 | targets_to_view_1 = self.target_network(batch_view_2) 89 | 90 | loss = self.regression_loss(predictions_from_view_1, targets_to_view_1) 91 | loss += self.regression_loss(predictions_from_view_2, targets_to_view_2) 92 | return loss.mean() 93 | 94 | def save_model(self, PATH): 95 | 96 | torch.save({ 97 | 'online_network_state_dict': self.online_network.state_dict(), 98 | 'target_network_state_dict': self.target_network.state_dict(), 99 | 'optimizer_state_dict': self.optimizer.state_dict(), 100 | }, PATH) 101 | -------------------------------------------------------------------------------- /models/BYOL/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | from shutil import copyfile 3 | 4 | import torch 5 | import numpy as np 6 | import pandas as pd 7 | from sklearn.metrics import classification_report, cohen_kappa_score, confusion_matrix, accuracy_score 8 | 9 | def _create_model_training_folder(writer, files_to_same): 10 | model_checkpoints_folder = os.path.join(writer.log_dir, 'checkpoints') 11 | if not os.path.exists(model_checkpoints_folder): 12 | os.makedirs(model_checkpoints_folder) 13 | for file in files_to_same: 14 | copyfile(file, os.path.join(model_checkpoints_folder, os.path.basename(file))) 15 | 16 | -------------------------------------------------------------------------------- /models/CPC/config.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | def configs(): 4 | parser = set_default(argparse.ArgumentParser()) 5 | return parser.parse_args([]) 6 | 7 | 8 | def set_default(parser): 9 | group = parser.add_argument_group('Architecture configuration', 10 | description="The arguments defining the " 11 | "model's architecture.") 12 | group.add_argument('--hiddenEncoder', type=int, default=256, 13 | help='Hidden dimension of the encoder network.') 14 | group.add_argument('--hiddenGar', type=int, default=256, 15 | help='Hidden dimension of the auto-regressive network') 16 | group.add_argument('--nPredicts', type=int, default=12, 17 | help='Number of steps to predict.') 18 | group.add_argument('--negativeSamplingExt', type=int, default=128, 19 | help='Number of negative samples to take.') 20 | group.add_argument('--learningRate', type=float, default=2e-4) 21 | group.add_argument('--schedulerStep', type=int, default=-1, 22 | help='Step of the learning rate scheduler: at each ' 23 | 'step the learning rate is divided by 2. Default: ' 24 | 'no scheduler.') 25 | group.add_argument('--schedulerRamp', type=int, default=None, 26 | help='Enable a warm up phase for the learning rate: ' 27 | 'adds a linear ramp of the given size.') 28 | group.add_argument('--beta1', type=float, default=0.9, 29 | help='Value of beta1 for the Adam optimizer') 30 | group.add_argument('--beta2', type=float, default=0.999, 31 | help='Value of beta2 for the Adam optimizer') 32 | group.add_argument('--epsilon', type=float, default=1e-08, 33 | help='Value of epsilon for the Adam optimizer') 34 | group.add_argument('--sizeWindow', type=int, default=20480, 35 | help='Number of frames to consider at each batch.') 36 | group.add_argument('--nEpoch', type=int, default=200, 37 | help='Number of epoch to run') 38 | group.add_argument('--samplingType', type=str, default='samespeaker', 39 | choices=['samespeaker', 'uniform', 40 | 'samesequence', 'sequential'], 41 | help='How to sample the negative examples in the ' 42 | 'CPC loss.') 43 | group.add_argument('--nLevelsPhone', type=int, default=1, 44 | help='(Supervised mode only). Number of layers in ' 45 | 'the phone classification network.') 46 | group.add_argument('--cpc_mode', type=str, default=None, 47 | choices=['reverse', 'none'], 48 | help='Some variations on CPC.') 49 | group.add_argument('--encoder_type', type=str, 50 | choices=['cpc', 'mfcc', 'lfb'], 51 | default='cpc', 52 | help='Replace the encoder network by mfcc features ' 53 | 'or learned filter banks') 54 | group.add_argument('--normMode', type=str, default='layerNorm', 55 | choices=['instanceNorm', 'ID', 'layerNorm', 56 | 'batchNorm'], 57 | help="Type of normalization to use in the encoder " 58 | "network (default is layerNorm).") 59 | group.add_argument('--onEncoder', action='store_true', 60 | help="(Supervised mode only) Perform the " 61 | "classification on the encoder's output.") 62 | group.add_argument('--random_seed', type=int, default=None, 63 | help="Set a specific random seed.") 64 | group.add_argument('--speakerEmbedding', type=int, default=0, 65 | help="(Depreciated) Feed the prediction network with " 66 | "speaker embeddings along with the usual sequence.") 67 | group.add_argument('--arMode', default='LSTM', 68 | choices=['GRU', 'LSTM', 'RNN', 'no_ar', 'transformer'], 69 | help="Architecture to use for the auto-regressive " 70 | "network (default is lstm).") 71 | group.add_argument('--nLevelsGRU', type=int, default=1, 72 | help='Number of layers in the autoregressive network.') 73 | group.add_argument('--rnnMode', type=str, default='transformer', 74 | choices=['transformer', 'RNN', 'LSTM', 'linear', 75 | 'ffd', 'conv4', 'conv8', 'conv12'], 76 | help="Architecture to use for the prediction network") 77 | group.add_argument('--dropout', action='store_true', 78 | help="Add a dropout layer at the output of the " 79 | "prediction network.") 80 | group.add_argument('--abspos', action='store_true', 81 | help='If the prediction network is a transformer, ' 82 | 'active to use absolute coordinates.') 83 | 84 | return parser 85 | -------------------------------------------------------------------------------- /models/CPC/feature_loader.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torchaudio 3 | import os 4 | import json 5 | import argparse 6 | from config import configs 7 | from model import CPCModel, ConcatenatedModel 8 | 9 | 10 | class FeatureModule(torch.nn.Module): 11 | 12 | def __init__(self, featureMaker, get_encoded, collapse=False): 13 | super(FeatureModule, self).__init__() 14 | self.get_encoded = get_encoded 15 | self.featureMaker = featureMaker 16 | self.collapse = collapse 17 | 18 | def getDownsamplingFactor(self): 19 | return self.featureMaker.gEncoder.DOWNSAMPLING 20 | 21 | def forward(self, data): 22 | 23 | batchAudio, label = data 24 | cFeature, encoded, _ = self.featureMaker(batchAudio.cuda(), label) 25 | if self.get_encoded: 26 | cFeature = encoded 27 | if self.collapse: 28 | cFeature = cFeature.contiguous().view(-1, cFeature.size(2)) 29 | return cFeature 30 | 31 | 32 | class ModelPhoneCombined(torch.nn.Module): 33 | 34 | def __init__(self, model, criterion, oneHot): 35 | super(ModelPhoneCombined, self).__init__() 36 | self.model = model 37 | self.criterion = criterion 38 | self.oneHot = oneHot 39 | 40 | def getDownsamplingFactor(self): 41 | return self.model.getDownsamplingFactor() 42 | 43 | def forward(self, data): 44 | c_feature = self.model(data) 45 | pred = self.criterion.getPrediction(c_feature) 46 | P = pred.size(2) 47 | 48 | if self.oneHot: 49 | pred = pred.argmax(dim=2) 50 | pred = toOneHot(pred, P) 51 | else: 52 | pred = torch.nn.functional.softmax(pred, dim=2) 53 | return pred 54 | 55 | 56 | def loadArgs(args, locArgs, forbiddenAttr=None): 57 | for k, v in vars(locArgs).items(): 58 | if forbiddenAttr is not None: 59 | if k not in forbiddenAttr: 60 | setattr(args, k, v) 61 | else: 62 | setattr(args, k, v) 63 | 64 | 65 | def getCheckpointData(pathDir): 66 | if not os.path.isdir(pathDir): 67 | return None 68 | checkpoints = [x for x in os.listdir(pathDir) 69 | if os.path.splitext(x)[1] == '.pt' 70 | and os.path.splitext(x[11:])[0].isdigit()] 71 | if len(checkpoints) == 0: 72 | print("No checkpoints found at " + pathDir) 73 | return None 74 | checkpoints.sort(key=lambda x: int(os.path.splitext(x[11:])[0])) 75 | data = os.path.join(pathDir, checkpoints[-1]) 76 | with open(os.path.join(pathDir, 'checkpoint_logs.json'), 'rb') as file: 77 | logs = json.load(file) 78 | 79 | with open(os.path.join(pathDir, 'checkpoint_args.json'), 'rb') as file: 80 | args = json.load(file) 81 | 82 | args = argparse.Namespace(**args) 83 | defaultArgs = configs() 84 | loadArgs(defaultArgs, args) 85 | 86 | return os.path.abspath(data), logs, defaultArgs 87 | 88 | 89 | def getEncoder(args): 90 | 91 | if args.encoder_type == 'mfcc': 92 | from .model import MFCCEncoder 93 | return MFCCEncoder(args.hiddenEncoder) 94 | elif args.encoder_type == 'lfb': 95 | from .model import LFBEnconder 96 | return LFBEnconder(args.hiddenEncoder) 97 | else: 98 | from .model import CPCEncoder 99 | return CPCEncoder(args.hiddenEncoder, args.normMode) 100 | 101 | 102 | def getAR(args): 103 | if args.arMode == 'transformer': 104 | from .transformers import buildTransformerAR 105 | arNet = buildTransformerAR(args.hiddenEncoder, 1, 106 | args.sizeWindow // 160, args.abspos) 107 | args.hiddenGar = args.hiddenEncoder 108 | elif args.arMode == 'no_ar': 109 | from .model import NoAr 110 | arNet = NoAr() 111 | else: 112 | from .model import CPCAR 113 | arNet = CPCAR(args.hiddenEncoder, args.hiddenGar, 114 | args.samplingType == "sequential", 115 | args.nLevelsGRU, 116 | mode=args.arMode, 117 | reverse=args.cpc_mode == "reverse") 118 | return arNet 119 | 120 | 121 | def loadModel(pathCheckpoints, loadStateDict=True): 122 | models = [] 123 | hiddenGar, hiddenEncoder = 0, 0 124 | for path in pathCheckpoints: 125 | print(f"Loading checkpoint {path}") 126 | _, _, locArgs = getCheckpointData(os.path.dirname(path)) 127 | 128 | doLoad = locArgs.load is not None and \ 129 | (len(locArgs.load) > 1 or 130 | os.path.dirname(locArgs.load[0]) != os.path.dirname(path)) 131 | 132 | if doLoad: 133 | m_, hg, he = loadModel(locArgs.load, loadStateDict=False) 134 | hiddenGar += hg 135 | hiddenEncoder += he 136 | else: 137 | encoderNet = getEncoder(locArgs) 138 | 139 | arNet = getAR(locArgs) 140 | m_ = CPCModel(encoderNet, arNet) 141 | 142 | if loadStateDict: 143 | print(f"Loading the state dict at {path}") 144 | state_dict = torch.load(path, 'cpu') 145 | m_.load_state_dict(state_dict["gEncoder"], strict=False) 146 | if not doLoad: 147 | hiddenGar += locArgs.hiddenGar 148 | hiddenEncoder += locArgs.hiddenEncoder 149 | 150 | models.append(m_) 151 | 152 | if len(models) == 1: 153 | return models[0], hiddenGar, hiddenEncoder 154 | 155 | return ConcatenatedModel(models), hiddenGar, hiddenEncoder 156 | 157 | 158 | def get_module(i_module): 159 | if isinstance(i_module, torch.nn.DataParallel): 160 | return get_module(i_module.module) 161 | if isinstance(i_module, FeatureModule): 162 | return get_module(i_module.module) 163 | return i_module 164 | 165 | 166 | def save_checkpoint(model_state, criterion_state, optimizer_state, best_state, 167 | path_checkpoint): 168 | 169 | state_dict = {"gEncoder": model_state, 170 | "cpcCriterion": criterion_state, 171 | "optimizer": optimizer_state, 172 | "best": best_state} 173 | 174 | torch.save(state_dict, path_checkpoint) 175 | 176 | 177 | def toOneHot(inputVector, nItems): 178 | 179 | batchSize, seqSize = inputVector.size() 180 | out = torch.zeros((batchSize, seqSize, nItems), 181 | device=inputVector.device, dtype=torch.long) 182 | out.scatter_(2, inputVector.view(batchSize, seqSize, 1), 1) 183 | return out 184 | 185 | 186 | def seqNormalization(out): 187 | mean = out.mean(dim=1, keepdim=True) 188 | var = out.var(dim=1, keepdim=True) 189 | return (out - mean) / torch.sqrt(var + 1e-08) 190 | 191 | 192 | def buildFeature(featureMaker, seqPath, strict=False, 193 | maxSizeSeq=64000, seqNorm=False): 194 | 195 | seq = torchaudio.load(seqPath)[0] 196 | sizeSeq = seq.size(1) 197 | start = 0 198 | out = [] 199 | while start < sizeSeq: 200 | if strict and start + maxSizeSeq > sizeSeq: 201 | break 202 | end = min(sizeSeq, start + maxSizeSeq) 203 | subseq = (seq[:, start:end]).view(1, 1, -1).cuda(device=0) 204 | with torch.no_grad(): 205 | features = featureMaker((subseq, None)) 206 | if seqNorm: 207 | features = seqNormalization(features) 208 | out.append(features.detach().cpu()) 209 | start += maxSizeSeq 210 | 211 | if strict and start < sizeSeq: 212 | subseq = (seq[:, -maxSizeSeq:]).view(1, 1, -1).cuda(device=0) 213 | with torch.no_grad(): 214 | features = featureMaker((subseq, None)) 215 | if seqNorm: 216 | features = seqNormalization(features) 217 | delta = (sizeSeq - start) // featureMaker.getDownsamplingFactor() 218 | out.append(features[:, -delta:].detach().cpu()) 219 | 220 | out = torch.cat(out, dim=1) 221 | return out 222 | -------------------------------------------------------------------------------- /models/CPC/model.py: -------------------------------------------------------------------------------- 1 | 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | import torchaudio 5 | import torch 6 | 7 | 8 | class IDModule(nn.Module): 9 | 10 | def __init__(self, *args, **kwargs): 11 | super(IDModule, self).__init__() 12 | 13 | def forward(self, x): 14 | return x 15 | 16 | 17 | class ChannelNorm(nn.Module): 18 | 19 | def __init__(self, 20 | numFeatures, 21 | epsilon=1e-05, 22 | affine=True): 23 | 24 | super(ChannelNorm, self).__init__() 25 | if affine: 26 | self.weight = nn.parameter.Parameter(torch.Tensor(1, 27 | numFeatures, 1)) 28 | self.bias = nn.parameter.Parameter(torch.Tensor(1, numFeatures, 1)) 29 | else: 30 | self.weight = None 31 | self.bias = None 32 | self.epsilon = epsilon 33 | self.p = 0 34 | self.affine = affine 35 | self.reset_parameters() 36 | 37 | def reset_parameters(self): 38 | if self.affine: 39 | torch.nn.init.ones_(self.weight) 40 | torch.nn.init.zeros_(self.bias) 41 | 42 | def forward(self, x): 43 | 44 | cumMean = x.mean(dim=1, keepdim=True) 45 | cumVar = x.var(dim=1, keepdim=True) 46 | x = (x - cumMean)*torch.rsqrt(cumVar + self.epsilon) 47 | 48 | if self.weight is not None: 49 | x = x * self.weight + self.bias 50 | return x 51 | 52 | 53 | class CPCEncoder(nn.Module): 54 | 55 | def __init__(self, 56 | sizeHidden=512, 57 | normMode="layerNorm"): 58 | 59 | super(CPCEncoder, self).__init__() 60 | 61 | validModes = ["batchNorm", "instanceNorm", "ID", "layerNorm"] 62 | if normMode not in validModes: 63 | raise ValueError(f"Norm mode must be in {validModes}") 64 | 65 | if normMode == "instanceNorm": 66 | def normLayer(x): return nn.InstanceNorm1d(x, affine=True) 67 | elif normMode == "ID": 68 | normLayer = IDModule 69 | elif normMode == "layerNorm": 70 | normLayer = ChannelNorm 71 | else: 72 | normLayer = nn.BatchNorm1d 73 | 74 | self.dimEncoded = sizeHidden 75 | self.conv0 = nn.Conv1d(1, sizeHidden, 10, stride=5, padding=3) 76 | self.batchNorm0 = normLayer(sizeHidden) 77 | self.conv1 = nn.Conv1d(sizeHidden, sizeHidden, 8, stride=4, padding=2) 78 | self.batchNorm1 = normLayer(sizeHidden) 79 | self.conv2 = nn.Conv1d(sizeHidden, sizeHidden, 4, 80 | stride=2, padding=1) 81 | self.batchNorm2 = normLayer(sizeHidden) 82 | self.conv3 = nn.Conv1d(sizeHidden, sizeHidden, 4, stride=2, padding=1) 83 | self.batchNorm3 = normLayer(sizeHidden) 84 | self.conv4 = nn.Conv1d(sizeHidden, sizeHidden, 4, stride=2, padding=1) 85 | self.batchNorm4 = normLayer(sizeHidden) 86 | self.DOWNSAMPLING = 160 87 | 88 | def getDimOutput(self): 89 | return self.conv4.out_channels 90 | 91 | def forward(self, x): 92 | x = F.relu(self.batchNorm0(self.conv0(x))) 93 | x = F.relu(self.batchNorm1(self.conv1(x))) 94 | x = F.relu(self.batchNorm2(self.conv2(x))) 95 | x = F.relu(self.batchNorm3(self.conv3(x))) 96 | x = F.relu(self.batchNorm4(self.conv4(x))) 97 | return x 98 | 99 | 100 | class MFCCEncoder(nn.Module): 101 | 102 | def __init__(self, 103 | dimEncoded): 104 | 105 | super(MFCCEncoder, self).__init__() 106 | melkwargs = {"n_mels": max(128, dimEncoded), "n_fft": 321} 107 | self.dimEncoded = dimEncoded 108 | self.MFCC = torchaudio.transforms.MFCC(n_mfcc=dimEncoded, 109 | melkwargs=melkwargs) 110 | 111 | def forward(self, x): 112 | x = x.view(x.size(0), -1) 113 | x = self.MFCC(x) 114 | return x.permute(0, 2, 1) 115 | 116 | 117 | class LFBEnconder(nn.Module): 118 | 119 | def __init__(self, dimEncoded, normalize=True): 120 | 121 | super(LFBEnconder, self).__init__() 122 | self.dimEncoded = dimEncoded 123 | self.conv = nn.Conv1d(1, 2 * dimEncoded, 124 | 400, stride=1) 125 | self.register_buffer('han', torch.hann_window(400).view(1, 1, 400)) 126 | self.instancenorm = nn.InstanceNorm1d(dimEncoded, momentum=1) \ 127 | if normalize else None 128 | 129 | def forward(self, x): 130 | 131 | N, C, L = x.size() 132 | x = self.conv(x) 133 | x = x.view(N, self.dimEncoded, 2, -1) 134 | x = x[:, :, 0, :]**2 + x[:, :, 1, :]**2 135 | x = x.view(N * self.dimEncoded, 1, -1) 136 | x = torch.nn.functional.conv1d(x, self.han, bias=None, 137 | stride=160, padding=350) 138 | x = x.view(N, self.dimEncoded, -1) 139 | x = torch.log(1 + torch.abs(x)) 140 | 141 | # Normalization 142 | if self.instancenorm is not None: 143 | x = self.instancenorm(x) 144 | return x 145 | 146 | 147 | class CPCAR(nn.Module): 148 | 149 | def __init__(self, 150 | dimEncoded, 151 | dimOutput, 152 | keepHidden, 153 | nLevelsGRU, 154 | mode="GRU", 155 | reverse=False): 156 | 157 | super(CPCAR, self).__init__() 158 | self.RESIDUAL_STD = 0.1 159 | 160 | if mode == "LSTM": 161 | self.baseNet = nn.LSTM(dimEncoded, dimOutput, 162 | num_layers=nLevelsGRU, batch_first=True) 163 | elif mode == "RNN": 164 | self.baseNet = nn.RNN(dimEncoded, dimOutput, 165 | num_layers=nLevelsGRU, batch_first=True) 166 | else: 167 | self.baseNet = nn.GRU(dimEncoded, dimOutput, 168 | num_layers=nLevelsGRU, batch_first=True) 169 | 170 | self.hidden = None 171 | self.keepHidden = keepHidden 172 | self.reverse = reverse 173 | 174 | def getDimOutput(self): 175 | return self.baseNet.hidden_size 176 | 177 | def forward(self, x): 178 | 179 | if self.reverse: 180 | x = torch.flip(x, [1]) 181 | try: 182 | self.baseNet.flatten_parameters() 183 | except RuntimeError: 184 | pass 185 | x, h = self.baseNet(x, self.hidden) 186 | if self.keepHidden: 187 | if isinstance(h, tuple): 188 | self.hidden = tuple(x.detach() for x in h) 189 | else: 190 | self.hidden = h.detach() 191 | 192 | if self.reverse: 193 | x = torch.flip(x, [1]) 194 | return x 195 | 196 | 197 | class NoAr(nn.Module): 198 | 199 | def __init__(self, *args): 200 | super(NoAr, self).__init__() 201 | 202 | def forward(self, x): 203 | return x 204 | 205 | 206 | class BiDIRARTangled(nn.Module): 207 | 208 | def __init__(self, 209 | dimEncoded, 210 | dimOutput, 211 | nLevelsGRU): 212 | 213 | super(BiDIRARTangled, self).__init__() 214 | assert(dimOutput % 2 == 0) 215 | 216 | self.ARNet = nn.GRU(dimEncoded, dimOutput // 2, 217 | num_layers=nLevelsGRU, batch_first=True, 218 | bidirectional=True) 219 | 220 | def getDimOutput(self): 221 | return self.ARNet.hidden_size * 2 222 | 223 | def forward(self, x): 224 | 225 | self.ARNet.flatten_parameters() 226 | xf, _ = self.ARNet(x) 227 | return xf 228 | 229 | 230 | class BiDIRAR(nn.Module): 231 | 232 | def __init__(self, 233 | dimEncoded, 234 | dimOutput, 235 | nLevelsGRU): 236 | 237 | super(BiDIRAR, self).__init__() 238 | assert(dimOutput % 2 == 0) 239 | 240 | self.netForward = nn.GRU(dimEncoded, dimOutput // 2, 241 | num_layers=nLevelsGRU, batch_first=True) 242 | self.netBackward = nn.GRU(dimEncoded, dimOutput // 2, 243 | num_layers=nLevelsGRU, batch_first=True) 244 | 245 | def getDimOutput(self): 246 | return self.netForward.hidden_size * 2 247 | 248 | def forward(self, x): 249 | 250 | self.netForward.flatten_parameters() 251 | self.netBackward.flatten_parameters() 252 | xf, _ = self.netForward(x) 253 | xb, _ = self.netBackward(torch.flip(x, [1])) 254 | return torch.cat([xf, torch.flip(xb, [1])], dim=2) 255 | 256 | 257 | class CPCModel(nn.Module): 258 | 259 | def __init__(self, 260 | encoder, 261 | AR): 262 | 263 | super(CPCModel, self).__init__() 264 | self.gEncoder = encoder 265 | self.gAR = AR 266 | 267 | def forward(self, batchData, label): 268 | encodedData = self.gEncoder(batchData).permute(0, 2, 1) 269 | cFeature = self.gAR(encodedData) 270 | return cFeature, encodedData, label 271 | 272 | 273 | class ConcatenatedModel(nn.Module): 274 | 275 | def __init__(self, model_list): 276 | 277 | super(ConcatenatedModel, self).__init__() 278 | self.models = torch.nn.ModuleList(model_list) 279 | 280 | def forward(self, batchData, label): 281 | 282 | outFeatures = [] 283 | outEncoded = [] 284 | for model in self.models: 285 | cFeature, encodedData, label = model(batchData, label) 286 | outFeatures.append(cFeature) 287 | outEncoded.append(encodedData) 288 | return torch.cat(outFeatures, dim=2), \ 289 | torch.cat(outEncoded, dim=2), label 290 | -------------------------------------------------------------------------------- /models/CPC/train_CPC.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import torch 3 | from model import CPCModel as cpcmodel 4 | from config import configs 5 | from feature_loader import getEncoder, getAR, loadArgs 6 | from ..data_loader.dataset_loader import data_generator 7 | 8 | class train_CPC(): 9 | 10 | def __init__(self,parser): 11 | self.args = parser.parse_args() 12 | self.dataset_name = self.args.dataset_name 13 | self.method = 'CPC' 14 | self.training_mode = self.args.training_mode 15 | self.data_dir = self.args.data_dir 16 | 17 | def excute(self): 18 | locArgs = configs() 19 | loadArgs(locArgs, self.args) 20 | encoderNet = getEncoder(locArgs) 21 | arNet = getAR(locArgs) 22 | model = cpcmodel(encoderNet, arNet) 23 | train_dl, valid_dl, test_dl = data_generator(os.path.join(self.data_dir, self.dataset_name), locArgs, self.training_mode) 24 | 25 | model.train(train_dl) 26 | -------------------------------------------------------------------------------- /models/CPC/transformers.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import math 4 | 5 | 6 | class ScaledDotProductAttention(nn.Module): 7 | def __init__(self, 8 | sizeSeq, # Size of the input sequence 9 | dk, # Dimension of the input sequence 10 | dropout, # Dropout parameter 11 | relpos=False): # Do we retrieve positional information ? 12 | super(ScaledDotProductAttention, self).__init__() 13 | 14 | self.drop = nn.Dropout(dropout) 15 | self.softmax = nn.Softmax(dim=2) 16 | self.relpos = relpos 17 | self.sizeSeq = sizeSeq 18 | 19 | if relpos: 20 | self.Krelpos = nn.Parameter(torch.Tensor(dk, sizeSeq)) 21 | self.initmat_(self.Krelpos) 22 | self.register_buffer('z', torch.zeros(1, sizeSeq, 1)) 23 | 24 | # A mask is set so that a node never queries data in the future 25 | mask = torch.tril(torch.ones(sizeSeq, sizeSeq), diagonal=0) 26 | mask = 1 - mask 27 | mask[mask == 1] = -float('inf') 28 | self.register_buffer('mask', mask.unsqueeze(0)) 29 | 30 | def initmat_(self, mat, dim=0): 31 | stdv = 1. / math.sqrt(mat.size(dim)) 32 | mat.data.uniform_(-stdv, stdv) 33 | 34 | def forward(self, Q, K, V): 35 | # Input dim : N x sizeSeq x dk 36 | QK = torch.bmm(Q, K.transpose(-2, -1)) 37 | 38 | if self.relpos: 39 | bsz = Q.size(0) 40 | QP = Q.matmul(self.Krelpos) 41 | # This trick with z fills QP's diagonal with zeros 42 | QP = torch.cat((self.z.expand(bsz, -1, -1), QP), 2) 43 | QK += QP.view(bsz, self.sizeSeq + 1, self.sizeSeq)[:, 1:, :] 44 | A = self.softmax(QK / math.sqrt(K.size(-1)) + self.mask) 45 | return torch.bmm(self.drop(A), V) 46 | 47 | 48 | class MultiHeadAttention(nn.Module): 49 | def __init__(self, 50 | sizeSeq, # Size of a sequence 51 | dropout, # Dropout parameter 52 | dmodel, # Model's dimension 53 | nheads, # Number of heads in the model 54 | abspos): # Is positional information encoded in the input ? 55 | super(MultiHeadAttention, self).__init__() 56 | self.Wo = nn.Linear(dmodel, dmodel, bias=False) 57 | self.Wk = nn.Linear(dmodel, dmodel, bias=False) 58 | self.Wq = nn.Linear(dmodel, dmodel, bias=False) 59 | self.Wv = nn.Linear(dmodel, dmodel, bias=False) 60 | self.nheads = nheads 61 | self.dk = dmodel // nheads 62 | self.Att = ScaledDotProductAttention(sizeSeq, self.dk, 63 | dropout, not abspos) 64 | 65 | def trans_(self, x): 66 | bsz, bptt, h, dk = x.size(0), x.size(1), self.nheads, self.dk 67 | return x.view(bsz, bptt, h, dk).transpose(1, 2).contiguous().view(bsz * h, bptt, dk) 68 | 69 | def reverse_trans_(self, x): 70 | bsz, bptt, h, dk = x.size( 71 | 0) // self.nheads, x.size(1), self.nheads, self.dk 72 | return x.view(bsz, h, bptt, dk).transpose(1, 2).contiguous().view(bsz, bptt, h * dk) 73 | 74 | def forward(self, Q, K, V): 75 | q = self.trans_(self.Wq(Q)) 76 | k = self.trans_(self.Wk(K)) 77 | v = self.trans_(self.Wv(V)) 78 | y = self.reverse_trans_(self.Att(q, k, v)) 79 | return self.Wo(y) 80 | 81 | 82 | class FFNetwork(nn.Module): 83 | def __init__(self, din, dout, dff, dropout): 84 | super(FFNetwork, self).__init__() 85 | self.lin1 = nn.Linear(din, dff, bias=True) 86 | self.lin2 = nn.Linear(dff, dout, bias=True) 87 | self.relu = nn.ReLU() 88 | self.drop = nn.Dropout(dropout) 89 | 90 | def forward(self, x): 91 | return self.lin2(self.drop(self.relu(self.lin1(x)))) 92 | 93 | 94 | class TransformerLayer(nn.Module): 95 | def __init__(self, sizeSeq=32, dmodel=512, dff=2048, 96 | dropout=0.1, nheads=8, 97 | abspos=False): 98 | super(TransformerLayer, self).__init__() 99 | self.multihead = MultiHeadAttention(sizeSeq, dropout, 100 | dmodel, nheads, abspos) 101 | self.ln_multihead = nn.LayerNorm(dmodel) 102 | self.ffnetwork = FFNetwork(dmodel, dmodel, dff, dropout) 103 | self.ln_ffnetwork = nn.LayerNorm(dmodel) 104 | 105 | def forward(self, x): 106 | y = self.ln_multihead(x + self.multihead(Q=x, K=x, V=x)) 107 | return self.ln_ffnetwork(y + self.ffnetwork(y)) 108 | 109 | 110 | class StaticPositionEmbedding(nn.Module): 111 | def __init__(self, seqlen, dmodel): 112 | super(StaticPositionEmbedding, self).__init__() 113 | pos = torch.arange(0., seqlen).unsqueeze(1).repeat(1, dmodel) 114 | dim = torch.arange(0., dmodel).unsqueeze(0).repeat(seqlen, 1) 115 | div = torch.exp(- math.log(10000) * (2*(dim//2)/dmodel)) 116 | pos *= div 117 | pos[:, 0::2] = torch.sin(pos[:, 0::2]) 118 | pos[:, 1::2] = torch.cos(pos[:, 1::2]) 119 | self.register_buffer('pe', pos.unsqueeze(0)) 120 | 121 | def forward(self, x): 122 | return x + self.pe[:, :x.size(1), :] 123 | 124 | 125 | def buildTransformerAR(dimEncoded, # Output dimension of the encoder 126 | nLayers, # Number of transformer layers 127 | sizeSeq, # Expected size of the input sequence 128 | abspos): 129 | layerSequence = [] 130 | if abspos: 131 | layerSequence += [StaticPositionEmbedding(sizeSeq, dimEncoded)] 132 | layerSequence += [TransformerLayer(sizeSeq=sizeSeq, 133 | dmodel=dimEncoded, abspos=abspos) 134 | for i in range(nLayers)] 135 | return nn.Sequential(*layerSequence) 136 | -------------------------------------------------------------------------------- /models/IDFD/config_files/EigenWorms_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 6 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 5 10 | self.dropout = 0.35 11 | self.features_len = 17984 12 | 13 | # training configs 14 | self.num_epoch = 40 15 | 16 | # optimizer parameters 17 | self.beta1 = 0.9 18 | self.beta2 = 0.99 19 | self.lr = 3e-4 20 | 21 | # data parameters 22 | self.drop_last = True 23 | self.batch_size = 32 24 | 25 | self.Context_Cont = Context_Cont_configs() 26 | self.TC = TC() 27 | self.augmentation = augmentations() 28 | 29 | 30 | class augmentations(object): 31 | def __init__(self): 32 | self.jitter_scale_ratio = 1.1 33 | self.jitter_ratio = 0.8 34 | self.max_seg = 8 35 | 36 | 37 | class Context_Cont_configs(object): 38 | def __init__(self): 39 | self.temperature = 0.2 40 | self.use_cosine_similarity = True 41 | 42 | 43 | class TC(object): 44 | def __init__(self): 45 | self.hidden_dim = 100 46 | self.timesteps = 6 47 | -------------------------------------------------------------------------------- /models/IDFD/config_files/FingerMovements_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 28 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 2 10 | self.dropout = 0.35 11 | self.features_len = 50 12 | 13 | # training configs 14 | self.num_epoch = 40 15 | 16 | # optimizer parameters 17 | self.beta1 = 0.9 18 | self.beta2 = 0.99 19 | self.lr = 3e-4 20 | 21 | # data parameters 22 | self.drop_last = True 23 | self.batch_size = 128 24 | 25 | self.Context_Cont = Context_Cont_configs() 26 | self.TC = TC() 27 | self.augmentation = augmentations() 28 | 29 | 30 | class augmentations(object): 31 | def __init__(self): 32 | self.jitter_scale_ratio = 1.1 33 | self.jitter_ratio = 0.8 34 | self.max_seg = 8 35 | 36 | 37 | class Context_Cont_configs(object): 38 | def __init__(self): 39 | self.temperature = 0.2 40 | self.use_cosine_similarity = True 41 | 42 | 43 | class TC(object): 44 | def __init__(self): 45 | self.hidden_dim = 100 46 | self.timesteps = 6 47 | -------------------------------------------------------------------------------- /models/IDFD/config_files/HAR_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 9 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 6 10 | self.dropout = 0.35 11 | self.features_len = 18 12 | 13 | # training configs 14 | self.num_epoch = 40 15 | 16 | # optimizer parameters 17 | self.beta1 = 0.9 18 | self.beta2 = 0.99 19 | self.lr = 3e-4 20 | 21 | # data parameters 22 | self.drop_last = True 23 | self.batch_size = 4 24 | 25 | self.Context_Cont = Context_Cont_configs() 26 | self.TC = TC() 27 | self.augmentation = augmentations() 28 | 29 | 30 | class augmentations(object): 31 | def __init__(self): 32 | self.jitter_scale_ratio = 1.1 33 | self.jitter_ratio = 0.8 34 | self.max_seg = 8 35 | 36 | 37 | class Context_Cont_configs(object): 38 | def __init__(self): 39 | self.temperature = 0.2 40 | self.use_cosine_similarity = True 41 | 42 | 43 | class TC(object): 44 | def __init__(self): 45 | self.hidden_dim = 100 46 | self.timesteps = 6 47 | -------------------------------------------------------------------------------- /models/IDFD/config_files/PenDigits_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 2 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 10 10 | self.dropout = 0.35 11 | self.features_len = 8 12 | 13 | # training configs 14 | self.num_epoch = 40 15 | 16 | # optimizer parameters 17 | self.beta1 = 0.9 18 | self.beta2 = 0.99 19 | self.lr = 3e-4 20 | 21 | # data parameters 22 | self.drop_last = True 23 | self.batch_size = 128 24 | 25 | self.Context_Cont = Context_Cont_configs() 26 | self.TC = TC() 27 | self.augmentation = augmentations() 28 | 29 | 30 | class augmentations(object): 31 | def __init__(self): 32 | self.jitter_scale_ratio = 1.1 33 | self.jitter_ratio = 0.8 34 | self.max_seg = 8 35 | 36 | 37 | class Context_Cont_configs(object): 38 | def __init__(self): 39 | self.temperature = 0.2 40 | self.use_cosine_similarity = True 41 | 42 | 43 | class TC(object): 44 | def __init__(self): 45 | self.hidden_dim = 100 46 | self.timesteps = 6 47 | -------------------------------------------------------------------------------- /models/IDFD/config_files/SHAR_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 3 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 17 10 | self.dropout = 0.35 11 | self.features_len = 21 12 | 13 | # training configs 14 | self.num_epoch = 40 15 | 16 | # optimizer parameters 17 | self.beta1 = 0.9 18 | self.beta2 = 0.99 19 | self.lr = 3e-4 20 | 21 | # data parameters 22 | self.drop_last = True 23 | self.batch_size = 128 24 | 25 | self.Context_Cont = Context_Cont_configs() 26 | self.TC = TC() 27 | self.augmentation = augmentations() 28 | 29 | 30 | class augmentations(object): 31 | def __init__(self): 32 | self.jitter_scale_ratio = 1.1 33 | self.jitter_ratio = 0.8 34 | self.max_seg = 8 35 | 36 | 37 | class Context_Cont_configs(object): 38 | def __init__(self): 39 | self.temperature = 0.2 40 | self.use_cosine_similarity = True 41 | 42 | 43 | class TC(object): 44 | def __init__(self): 45 | self.hidden_dim = 100 46 | self.timesteps = 6 -------------------------------------------------------------------------------- /models/IDFD/config_files/epilepsy_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 1 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 2 10 | self.dropout = 0.35 11 | self.features_len = 24 12 | 13 | # training configs 14 | self.num_epoch = 10 15 | 16 | 17 | # optimizer parameters 18 | self.beta1 = 0.9 19 | self.beta2 = 0.99 20 | self.lr = 3e-4 21 | 22 | # data parameters 23 | self.drop_last = True 24 | self.batch_size = 128 25 | 26 | self.Context_Cont = Context_Cont_configs() 27 | self.TC = TC() 28 | self.augmentation = augmentations() 29 | 30 | 31 | class augmentations(object): 32 | def __init__(self): 33 | self.jitter_scale_ratio = 0.001 34 | self.jitter_ratio = 0.001 35 | self.max_seg = 5 36 | 37 | 38 | class Context_Cont_configs(object): 39 | def __init__(self): 40 | self.temperature = 0.2 41 | self.use_cosine_similarity = True 42 | 43 | 44 | class TC(object): 45 | def __init__(self): 46 | self.hidden_dim = 100 47 | self.timesteps = 10 48 | -------------------------------------------------------------------------------- /models/IDFD/config_files/wisdm_Configs.py: -------------------------------------------------------------------------------- 1 | class Config(object): 2 | def __init__(self): 3 | # model configs 4 | self.input_channels = 3 5 | self.kernel_size = 8 6 | self.stride = 1 7 | self.final_out_channels = 128 8 | 9 | self.num_classes = 6 10 | self.dropout = 0.35 11 | self.features_len = 34 12 | 13 | # training configs 14 | self.num_epoch = 40 15 | 16 | # optimizer parameters 17 | self.beta1 = 0.9 18 | self.beta2 = 0.99 19 | self.lr = 3e-4 20 | 21 | # data parameters 22 | self.drop_last = True 23 | self.batch_size = 128 24 | 25 | self.Context_Cont = Context_Cont_configs() 26 | self.TC = TC() 27 | self.augmentation = augmentations() 28 | 29 | 30 | class augmentations(object): 31 | def __init__(self): 32 | self.jitter_scale_ratio = 1.1 33 | self.jitter_ratio = 0.8 34 | self.max_seg = 8 35 | 36 | 37 | class Context_Cont_configs(object): 38 | def __init__(self): 39 | self.temperature = 0.2 40 | self.use_cosine_similarity = True 41 | 42 | 43 | class TC(object): 44 | def __init__(self): 45 | self.hidden_dim = 100 46 | self.timesteps = 256 47 | -------------------------------------------------------------------------------- /models/IDFD/train_IDFD.py: -------------------------------------------------------------------------------- 1 | import os 2 | import argparse 3 | 4 | import numpy as np 5 | from sklearn.cluster import KMeans 6 | from sklearn.metrics import normalized_mutual_info_score, adjusted_rand_score 7 | from scipy.optimize import linear_sum_assignment 8 | import tqdm.autonotebook as tqdm 9 | 10 | import torch 11 | import torch.nn as nn 12 | from torch.autograd import Function 13 | import torch.nn.functional as F 14 | from torchvision import datasets, transforms 15 | from torchvision.models import resnet 16 | 17 | from ..data_loader.dataset_loader import data_generator 18 | 19 | 20 | 21 | class train_IDFD(): 22 | 23 | def __init__(self, parser): 24 | self.args = parser.parse_args() 25 | self.method = 'IDFD' 26 | self.dataset_name = self.args.dataset_name 27 | self.epochs = self.args.epochs 28 | self.gpus = self.args.device_id 29 | self.num_workers = self.args.workers 30 | 31 | 32 | def excute(self): 33 | 34 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 35 | os.environ["CUDA_VISIBLE_DEVICES"] = self.gpus 36 | 37 | if self.dataset_name == 'HAR': 38 | from config_files.HAR_Configs import Config as Configs 39 | configs = Configs() 40 | in_channels = 9 41 | train_loader, test_loader, len_train, len_test = data_generator('data/HAR', configs, 'self_supervised') 42 | elif self.dataset_name == 'wisdm': 43 | from config_files.wisdm_Configs import Config as Configs 44 | configs = Configs() 45 | in_channels = 3 46 | train_loader, test_loader, len_train, len_test = data_generator('data/wisdm', configs, 'self_supervised') 47 | elif self.dataset_name == 'epilepsy': 48 | from config_files.epilepsy_Configs import Config as Configs 49 | configs = Configs() 50 | in_channels = 1 51 | train_loader, test_loader, len_train, len_test = data_generator('data/epilepsy', configs, 'self_supervised') 52 | elif self.dataset_name == 'SHAR': 53 | from config_files.SHAR_Configs import Config as Configs 54 | configs = Configs() 55 | in_channels = 3 56 | train_loader, test_loader, len_train, len_test = data_generator('data/SHAR', configs, 'self_supervised') 57 | elif self.dataset_name == 'PenDigits': 58 | from config_files.PenDigits_Configs import Config as Configs 59 | configs = Configs() 60 | in_channels = 2 61 | train_loader, test_loader, len_train, len_test = data_generator('data/PenDigits', configs, 'self_supervised') 62 | elif self.dataset_name == 'EigenWorms': 63 | from config_files.EigenWorms_Configs import Config as Configs 64 | configs = Configs() 65 | in_channels = 6 66 | train_loader, test_loader, len_train, len_test = data_generator('data/EigenWorms', configs, 'self_supervised') 67 | elif self.dataset_name == 'FingerMovements': 68 | from config_files.FingerMovements_Configs import Config as Configs 69 | configs = Configs() 70 | in_channels = 28 71 | train_loader, test_loader, len_train, len_test = data_generator('data/FingerMovements', configs, 'self_supervised') 72 | elif self.dataset_name == 'StandWalkJump': 73 | from config_files.StandWalkJump_Configs import Config as Configs 74 | configs = Configs() 75 | in_channels = 4 76 | train_loader, test_loader, len_train, len_test = data_generator('data/StandWalkJump', configs, 'self_supervised') 77 | elif self.dataset_name == 'PhonemeSpectra': 78 | from config_files.PhonemeSpectra_Configs import Config as Configs 79 | configs = Configs() 80 | in_channels = 11 81 | train_loader, test_loader, len_train, len_test = data_generator('data/PhonemeSpectra', configs, 'self_supervised') 82 | elif self.dataset_name == 'DuckDuckGeese': 83 | from config_files.DuckDuckGeese_Configs import Config as Configs 84 | configs = Configs() 85 | in_channels = 1345 86 | train_loader, test_loader, len_train, len_test = data_generator('data/DuckDuckGeese', configs, 'self_supervised') 87 | elif self.dataset_name == 'InsectWingbeat': 88 | from config_files.InsectWingbeat_Configs import Config as Configs 89 | configs = Configs() 90 | in_channels = 200 91 | train_loader, test_loader, len_train, len_test = data_generator('data/InsectWingbeat', configs, 'self_supervised') 92 | elif self.dataset_name == 'CharacterTrajectories': 93 | from config_files.CharacterTrajectories_Configs import Config as Configs 94 | configs = Configs() 95 | in_channels = 3 96 | train_loader, test_loader, len_train, len_test = data_generator('data/CharacterTrajectories', configs, 'self_supervised') 97 | print(f'len(train_loader):{len(train_loader)}') 98 | print(f'len(test_loader):{len(test_loader)}') 99 | print(f'len_train:{len_train}') 100 | print(f'len_test:{len_test}') 101 | 102 | 103 | low_dim = 128 104 | net = ResNet18(low_dim=low_dim, in_channels=in_channels) 105 | norm = Normalize(2) 106 | npc = NonParametricClassifier(input_dim=low_dim, 107 | output_dim=len_train, 108 | tau=1.0, 109 | momentum=0.9)#0.5 110 | loss = Loss(tau2=2.0) 111 | net, norm = net.to(device), norm.to(device) 112 | npc, loss = npc.to(device), loss.to(device) 113 | optimizer = torch.optim.Adam(net.parameters(), 114 | lr=0.03, 115 | weight_decay=5e-4) 116 | lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, 117 | [20, 80, 120, 180], 118 | gamma=0.1)#[600, 950, 1300, 1650] 119 | 120 | if torch.cuda.is_available(): 121 | net = torch.nn.DataParallel(net, 122 | device_ids=range(len( 123 | self.gpus.split(",")))) 124 | torch.backends.cudnn.benchmark = True 125 | 126 | trackers = {n: AverageTracker() for n in ["loss", "loss_id", "loss_fd"]} 127 | 128 | 129 | with tqdm.trange(self.epochs) as epoch_bar: 130 | max_acc = 0.1 131 | max_epoch = 0 132 | for epoch in epoch_bar: 133 | net.train() 134 | # for batch_idx, (inputs, _, indexes) in enumerate(tqdm.tqdm(train_loader)): 135 | for batch_idx, (inputs, targets, aug1, aug2, indexes) in enumerate(tqdm.tqdm(train_loader)): #mqw 136 | optimizer.zero_grad() 137 | 138 | aug1 = aug1.unsqueeze(3) 139 | aug1 = aug1.to(device, dtype=torch.float32, non_blocking=True) 140 | indexes = indexes.to(device, non_blocking=True) 141 | features = norm(net(aug1)) 142 | 143 | outputs = npc(features, indexes) # 144 | loss_id, loss_fd = loss(outputs, features, indexes) # 145 | # tot_loss = loss_id + loss_fd 146 | tot_loss = loss_id 147 | tot_loss.backward() 148 | optimizer.step() 149 | # track loss 150 | trackers["loss"].add(tot_loss) 151 | trackers["loss_id"].add(loss_id) 152 | trackers["loss_fd"].add(loss_fd) 153 | lr_scheduler.step() 154 | 155 | # logging 156 | postfix = {name: t.avg() for name, t in trackers.items()} 157 | epoch_bar.set_postfix(**postfix) 158 | for t in trackers.values(): 159 | t.reset() 160 | 161 | 162 | acc, nmi, ari = check_clustering_metrics(npc, train_loader) 163 | print("Epoch:{} Kmeans ACC, NMI, ARI = {}, {}, {}".format(epoch+1, acc, nmi, ari)) 164 | 165 | 166 | if acc > max_acc: 167 | max_acc = acc 168 | max_epoch = epoch+1 169 | torch.save({'net_state_dict': net.state_dict(), 'npc_state_dict': npc.state_dict(), 170 | 'optimizer_state_dict': optimizer.state_dict(), 'lr_scheduler_state_dict': lr_scheduler.state_dict(), 171 | 'max_epoch': max_epoch, 'max_acc': max_acc 172 | }, 173 | f'{self.dataset_name}_{max_epoch}_{max_acc}_model.pth') 174 | 175 | 176 | class AverageTracker(): 177 | def __init__(self): 178 | self.step = 0 179 | self.cur_avg = 0 180 | 181 | def add(self, value): 182 | self.cur_avg *= self.step / (self.step + 1) 183 | self.cur_avg += value / (self.step + 1) 184 | self.step += 1 185 | 186 | def reset(self): 187 | self.step = 0 188 | self.cur_avg = 0 189 | 190 | def avg(self): 191 | return self.cur_avg.item() 192 | 193 | 194 | def check_clustering_metrics(npc, train_loader): 195 | # print(npc) 196 | trainFeatures = npc.memory 197 | z = trainFeatures.cpu().numpy() 198 | y = np.array(train_loader.dataset.y_data) 199 | n_clusters = len(np.unique(y)) 200 | kmeans = KMeans(n_clusters=n_clusters, n_init=20) 201 | y_pred = kmeans.fit_predict(z) 202 | return metrics.acc(y, y_pred), metrics.nmi(y, y_pred), metrics.ari(y, y_pred) 203 | 204 | 205 | class metrics: 206 | ari = adjusted_rand_score 207 | nmi = normalized_mutual_info_score 208 | 209 | @staticmethod 210 | def acc(y_true, y_pred): 211 | y_true = y_true.astype(np.int64) 212 | y_pred = y_pred.astype(np.int64) 213 | assert y_pred.size == y_true.size 214 | D = max(y_pred.max(), y_true.max()) + 1 215 | w = np.zeros((D, D), dtype=np.int64) 216 | for i in range(y_pred.size): 217 | w[y_pred[i], y_true[i]] += 1 218 | row, col = linear_sum_assignment(w.max() - w) 219 | return sum([w[i, j] for i, j in zip(row, col)]) * 1.0 / y_pred.size 220 | 221 | 222 | class NonParametricClassifierOP(Function): 223 | @staticmethod 224 | def forward(ctx, x, y, memory, params): 225 | 226 | tau = params[0].item() 227 | out = x.mm(memory.t()) 228 | out.div_(tau) 229 | ctx.save_for_backward(x, memory, y, params) 230 | return out 231 | 232 | @staticmethod 233 | def backward(ctx, grad_output): 234 | x, memory, y, params = ctx.saved_tensors 235 | tau = params[0] 236 | momentum = params[1] 237 | 238 | grad_output.div_(tau) 239 | 240 | grad_input = grad_output.mm(memory) 241 | grad_input.resize_as_(x) 242 | 243 | weight_pos = memory.index_select(0, y.view(-1)).resize_as_(x) 244 | weight_pos.mul_(momentum) 245 | weight_pos.add_(x.mul(1 - momentum)) 246 | w_norm = weight_pos.pow(2).sum(1, keepdim=True).pow(0.5) 247 | updated_weight = weight_pos.div(w_norm) 248 | memory.index_copy_(0, y, updated_weight) 249 | 250 | return grad_input, None, None, None, None 251 | 252 | 253 | class NonParametricClassifier(nn.Module): 254 | def __init__(self, input_dim, output_dim, tau=1.0, momentum=0.5): 255 | super(NonParametricClassifier, self).__init__() 256 | self.register_buffer('params', torch.tensor([tau, momentum])) 257 | stdv = 1. / np.sqrt(input_dim / 3.) 258 | self.register_buffer( 259 | 'memory', 260 | torch.rand(output_dim, input_dim).mul_(2 * stdv).add_(-stdv)) 261 | 262 | def forward(self, x, y): 263 | out = NonParametricClassifierOP.apply(x, y, self.memory, self.params) 264 | return out 265 | 266 | 267 | 268 | class Normalize(nn.Module): 269 | def __init__(self, power=2): 270 | super().__init__() 271 | self.power = power 272 | 273 | def forward(self, x): 274 | norm = x.pow(self.power).sum(1, keepdim=True).pow(1. / self.power) 275 | out = x.div(norm) 276 | return out 277 | 278 | 279 | def ResNet18(low_dim=128, in_channels=3): 280 | net = resnet.ResNet(resnet.BasicBlock, [2, 2, 2, 2], low_dim) 281 | net.conv1 = nn.Conv2d(in_channels, 64, kernel_size=3, 282 | stride=1, padding=1, bias=False) 283 | net.maxpool = nn.Identity() 284 | return net 285 | 286 | 287 | class Loss(nn.Module): 288 | def __init__(self, tau2): 289 | super().__init__() 290 | self.tau2 = tau2 291 | 292 | def forward(self, x, ff, y): 293 | 294 | L_id = F.cross_entropy(x, y) 295 | 296 | norm_ff = ff / (ff**2).sum(0, keepdim=True).sqrt() 297 | coef_mat = torch.mm(norm_ff.t(), norm_ff) 298 | coef_mat.div_(self.tau2) 299 | a = torch.arange(coef_mat.size(0), device=coef_mat.device) 300 | L_fd = F.cross_entropy(coef_mat, a) 301 | return L_id, L_fd 302 | 303 | -------------------------------------------------------------------------------- /models/PCL/builder.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from random import sample 4 | from torchvision.models import resnet 5 | 6 | def ResNet18(low_dim=128, dataset_name='wisdm'): 7 | 8 | if dataset_name == 'DuckDuckGeese': 9 | in_channels = 1345 10 | elif dataset_name == 'FingerMovements': 11 | in_channels = 28 12 | elif dataset_name == 'PenDigits': 13 | in_channels = 2 14 | elif dataset_name == 'PhonemeSpectra': 15 | in_channels = 11 16 | elif dataset_name == 'StandWalkJump': 17 | in_channels = 4 18 | elif dataset_name == 'InsectWingbeat': 19 | in_channels = 200 20 | elif dataset_name == 'EigenWorms': 21 | in_channels = 6 22 | elif dataset_name == 'HAR': 23 | in_channels = 9 24 | elif dataset_name == 'SHAR': 25 | in_channels = 3 26 | elif dataset_name == 'wisdm': 27 | in_channels = 3 28 | elif dataset_name == 'epilepsy': 29 | in_channels = 1 30 | 31 | 32 | net = resnet.ResNet(resnet.BasicBlock, [2, 2, 2, 2], low_dim) 33 | if dataset_name=='wisdm': 34 | net.conv1 = nn.Conv2d(in_channels, 64, kernel_size=8, stride=1, padding=4, bias=False) 35 | elif dataset_name=='HAR': 36 | net.conv1 = nn.Conv2d(9, 64, kernel_size=8, stride=1, padding=4, bias=False) 37 | elif dataset_name=='epilepsy': 38 | net.conv1 = nn.Conv2d(1, 64, kernel_size=8, stride=1, padding=43, bias=False) 39 | elif dataset_name == 'SHAR': 40 | net.conv1 = nn.Conv2d(3, 64, kernel_size=8, stride=1, padding=4, bias=False) 41 | else: 42 | net.conv1 = nn.Conv2d(in_channels, 64, kernel_size=7, stride=2, padding=3, bias=False) 43 | 44 | 45 | net.maxpool = nn.Identity() 46 | return net 47 | 48 | class MoCo(nn.Module): 49 | 50 | def __init__(self, base_encoder, dim=128, r=4, m=0.999, T=0.1, mlp=False, dataset_name='wisdm'): 51 | """ 52 | dim: feature dimension (default: 128) 53 | r: queue size; number of negative samples/prototypes (default: 4) 54 | m: momentum for updating key encoder (default: 0.999) 55 | T: softmax temperature 56 | mlp: whether to use mlp projection 57 | """ 58 | super(MoCo, self).__init__() 59 | 60 | self.r = r 61 | self.m = m 62 | self.T = T 63 | 64 | # create the encoders 65 | # num_classes is the output fc dimension 66 | self.encoder_q = ResNet18(low_dim=dim, dataset_name=dataset_name) 67 | self.encoder_k = ResNet18(low_dim=dim, dataset_name=dataset_name) 68 | 69 | 70 | 71 | if mlp: # hack: brute-force replacement 72 | dim_mlp = self.encoder_q.fc.weight.shape[1] 73 | self.encoder_q.fc = nn.Sequential(nn.Linear(dim_mlp, dim_mlp), nn.ReLU(), self.encoder_q.fc) 74 | self.encoder_k.fc = nn.Sequential(nn.Linear(dim_mlp, dim_mlp), nn.ReLU(), self.encoder_k.fc) 75 | 76 | for param_q, param_k in zip(self.encoder_q.parameters(), self.encoder_k.parameters()): 77 | param_k.data.copy_(param_q.data) # initialize 78 | param_k.requires_grad = False # not update by gradient 79 | 80 | # create the queue 81 | self.register_buffer("queue", torch.randn(dim, r)) 82 | self.queue = nn.functional.normalize(self.queue, dim=0) 83 | 84 | self.register_buffer("queue_ptr", torch.zeros(1, dtype=torch.long)) 85 | 86 | @torch.no_grad() 87 | def _momentum_update_key_encoder(self): 88 | """ 89 | Momentum update of the key encoder 90 | """ 91 | for param_q, param_k in zip(self.encoder_q.parameters(), self.encoder_k.parameters()): 92 | param_k.data = param_k.data * self.m + param_q.data * (1. - self.m) 93 | 94 | @torch.no_grad() 95 | def _dequeue_and_enqueue(self, keys): 96 | # gather keys before updating queue 97 | keys = concat_all_gather(keys) 98 | 99 | batch_size = keys.shape[0] 100 | 101 | ptr = int(self.queue_ptr) 102 | 103 | print(f'batch_size:{batch_size}')#384 104 | print(f'self.r:{self.r}')#896 105 | print(f'keys.T.shape:{keys.T.shape}')#[128, 384] 106 | 107 | 108 | assert self.r % batch_size == 0 # for simplicity 109 | print('&*&*&*&*&*&*1') 110 | # replace the keys at ptr (dequeue and enqueue) 111 | self.queue[:, ptr:ptr + batch_size] = keys.T 112 | print('&*&*&*&*&*&*2') 113 | ptr = (ptr + batch_size) % self.r # move pointer 114 | print('&*&*&*&*&*&*3') 115 | self.queue_ptr[0] = ptr 116 | 117 | @torch.no_grad() 118 | def _batch_shuffle_ddp(self, x): 119 | """ 120 | Batch shuffle, for making use of BatchNorm. 121 | *** Only support DistributedDataParallel (DDP) model. *** 122 | """ 123 | # gather from all gpus 124 | batch_size_this = x.shape[0] 125 | x_gather = concat_all_gather(x) 126 | batch_size_all = x_gather.shape[0] 127 | 128 | num_gpus = batch_size_all // batch_size_this 129 | 130 | # random shuffle index 131 | idx_shuffle = torch.randperm(batch_size_all).cuda() 132 | 133 | # broadcast to all gpus 134 | torch.distributed.broadcast(idx_shuffle, src=0) 135 | 136 | # index for restoring 137 | idx_unshuffle = torch.argsort(idx_shuffle) 138 | 139 | # shuffled index for this gpu 140 | gpu_idx = torch.distributed.get_rank() 141 | idx_this = idx_shuffle.view(num_gpus, -1)[gpu_idx] 142 | 143 | return x_gather[idx_this], idx_unshuffle 144 | 145 | @torch.no_grad() 146 | def _batch_unshuffle_ddp(self, x, idx_unshuffle): 147 | """ 148 | Undo batch shuffle. 149 | *** Only support DistributedDataParallel (DDP) model. *** 150 | """ 151 | # gather from all gpus 152 | batch_size_this = x.shape[0] 153 | x_gather = concat_all_gather(x) 154 | batch_size_all = x_gather.shape[0] 155 | 156 | num_gpus = batch_size_all // batch_size_this 157 | 158 | # restored index for this gpu 159 | gpu_idx = torch.distributed.get_rank() 160 | idx_this = idx_unshuffle.view(num_gpus, -1)[gpu_idx] 161 | 162 | return x_gather[idx_this] 163 | 164 | def forward(self, im_q, im_k=None, is_eval=False, cluster_result=None, index=None): 165 | """ 166 | Input: 167 | im_q: a batch of query images 168 | im_k: a batch of key images 169 | is_eval: return momentum embeddings (used for clustering) 170 | cluster_result: cluster assignments, centroids, and density 171 | index: indices for training samples 172 | Output: 173 | logits, targets, proto_logits, proto_targets 174 | """ 175 | 176 | if is_eval: 177 | # self.encoder_k.conv1 = nn.Conv2d(9, 64, kernel_size=3, stride=1, padding=1, bias=False) 178 | 179 | k = self.encoder_k(im_q) 180 | 181 | k = nn.functional.normalize(k, dim=1) 182 | return k 183 | 184 | # compute key features 185 | with torch.no_grad(): # no gradient to keys 186 | self._momentum_update_key_encoder() # update the key encoder 187 | 188 | # shuffle for making use of BN 189 | im_k, idx_unshuffle = self._batch_shuffle_ddp(im_k) 190 | 191 | k = self.encoder_k(im_k) # keys: NxC 192 | k = nn.functional.normalize(k, dim=1) 193 | 194 | # undo shuffle 195 | k = self._batch_unshuffle_ddp(k, idx_unshuffle) 196 | 197 | # compute query features 198 | q = self.encoder_q(im_q) # queries: NxC 199 | q = nn.functional.normalize(q, dim=1) 200 | 201 | # compute logits 202 | # Einstein sum is more intuitive 203 | # positive logits: Nx1 204 | l_pos = torch.einsum('nc,nc->n', [q, k]).unsqueeze(-1) 205 | # negative logits: Nxr 206 | l_neg = torch.einsum('nc,ck->nk', [q, self.queue.clone().detach()]) 207 | 208 | # logits: Nx(1+r) 209 | logits = torch.cat([l_pos, l_neg], dim=1) 210 | 211 | # apply temperature 212 | logits /= self.T 213 | 214 | # labels: positive key indicators 215 | labels = torch.zeros(logits.shape[0], dtype=torch.long).cuda() 216 | 217 | # dequeue and enqueue 218 | self._dequeue_and_enqueue(k) 219 | 220 | # prototypical contrast 221 | if cluster_result is not None: 222 | proto_labels = [] 223 | proto_logits = [] 224 | for n, (im2cluster,prototypes,density) in enumerate(zip(cluster_result['im2cluster'],cluster_result['centroids'],cluster_result['density'])): 225 | # get positive prototypes 226 | pos_proto_id = im2cluster[index] 227 | pos_prototypes = prototypes[pos_proto_id] print('e') 228 | # sample negative prototypes 229 | all_proto_id = [i for i in range(im2cluster.max())] 230 | neg_proto_id = set(all_proto_id)-set(pos_proto_id.tolist()) 231 | neg_proto_id = sample(neg_proto_id,self.r) #sample r negative prototypes 232 | neg_prototypes = prototypes[neg_proto_id] 233 | proto_selected = torch.cat([pos_prototypes,neg_prototypes],dim=0) 234 | # compute prototypical logits 235 | logits_proto = torch.mm(q,proto_selected.t()) 236 | 237 | # targets for prototype assignment 238 | labels_proto = torch.linspace(0, q.size(0)-1, steps=q.size(0)).long().cuda() 239 | # scaling temperatures for the selected prototypes 240 | temp_proto = density[torch.cat([pos_proto_id,torch.LongTensor(neg_proto_id).cuda()],dim=0)] 241 | logits_proto /= temp_proto 242 | proto_labels.append(labels_proto) 243 | proto_logits.append(logits_proto) 244 | return logits, labels, proto_logits, proto_labels 245 | else: 246 | return logits, labels, None, None 247 | 248 | 249 | # utils 250 | @torch.no_grad() 251 | def concat_all_gather(tensor): 252 | """ 253 | Performs all_gather operation on the provided tensors. 254 | *** Warning ***: torch.distributed.all_gather has no gradient. 255 | """ 256 | tensors_gather = [torch.ones_like(tensor) 257 | for _ in range(torch.distributed.get_world_size())] 258 | torch.distributed.all_gather(tensors_gather, tensor, async_op=False) 259 | 260 | output = torch.cat(tensors_gather, dim=0) 261 | return output 262 | -------------------------------------------------------------------------------- /models/SimCLR/loss.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | class contrastive_loss(nn.Module): 6 | def __init__(self, tau=1, normalize=False): 7 | super(contrastive_loss, self).__init__() 8 | self.tau = tau 9 | self.normalize = normalize 10 | 11 | def forward(self, xi, xj): 12 | 13 | x = torch.cat((xi, xj), dim=0) 14 | 15 | is_cuda = x.is_cuda 16 | sim_mat = torch.mm(x, x.T) 17 | if self.normalize: 18 | sim_mat_denom = torch.mm(torch.norm(x, dim=1).unsqueeze(1), torch.norm(x, dim=1).unsqueeze(1).T) 19 | sim_mat = sim_mat / sim_mat_denom.clamp(min=1e-16) 20 | 21 | sim_mat = torch.exp(sim_mat / self.tau) 22 | 23 | # no diag because it's not diffrentiable -> sum - exp(1 / tau) 24 | # diag_ind = torch.eye(xi.size(0) * 2).bool() 25 | # diag_ind = diag_ind.cuda() if use_cuda else diag_ind 26 | 27 | # sim_mat = sim_mat.masked_fill_(diag_ind, 0) 28 | 29 | # top 30 | if self.normalize: 31 | sim_mat_denom = torch.norm(xi, dim=1) * torch.norm(xj, dim=1) 32 | sim_match = torch.exp(torch.sum(xi * xj, dim=-1) / sim_mat_denom / self.tau) 33 | else: 34 | sim_match = torch.exp(torch.sum(xi * xj, dim=-1) / self.tau) 35 | 36 | sim_match = torch.cat((sim_match, sim_match), dim=0) 37 | 38 | norm_sum = torch.exp(torch.ones(x.size(0)) / self.tau) 39 | norm_sum = norm_sum.cuda() if is_cuda else norm_sum 40 | loss = torch.mean(-torch.log(sim_match / (torch.sum(sim_mat, dim=-1) - norm_sum))) 41 | 42 | return loss 43 | -------------------------------------------------------------------------------- /models/SimCLR/train_SimCLR.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import torch 3 | import torch.optim as optim 4 | from torch.optim.lr_scheduler import ExponentialLR 5 | import numpy as np 6 | 7 | from tensorboardX import SummaryWriter 8 | from tqdm import tqdm 9 | import os 10 | import time 11 | 12 | from models import * 13 | from utils import * 14 | from loss import * 15 | 16 | from torch.utils.data.dataloader import DataLoader 17 | from ..data_loader.dataset_loader import data_generator 18 | 19 | class train_SimCLR(): 20 | 21 | def __init__(self,parser): 22 | self.args=parser.parse_args() 23 | self.train_loader=None 24 | self.test_loader=None 25 | self.logger=None 26 | # Setup tensorboard 27 | self.use_tb = self.args.exp_dir is not None 28 | self.log_dir = self.args.exp_dir 29 | # Set cuda 30 | self.use_cuda = not self.args.no_cuda and torch.cuda.is_available() 31 | if self.use_cuda: 32 | self.dtype = torch.cuda.FloatTensor 33 | self.device = torch.device("cuda") 34 | torch.cuda.set_device(self.args.device_id) 35 | print('GPU') 36 | else: 37 | self.dtype = torch.FloatTensor 38 | self.device = torch.device("cpu") 39 | print('CPU') 40 | # Setup tensorboard 41 | self.use_tb = self.args.exp_dir is not None 42 | self.log_dir = self.args.exp_dir 43 | 44 | 45 | # train validate 46 | # def train_validate(model, loader, optimizer, is_train, epoch, use_cuda): 47 | def train_validate(self,model, optimizer, is_train, epoch, use_cuda): 48 | loss_func = contrastive_loss(tau=self.args.tau) 49 | # data_loader = loader.train_loader if is_train else loader.test_loader 50 | data_loader = self.train_loader if is_train else self.test_loader 51 | 52 | if is_train: 53 | model.train() 54 | model.zero_grad() 55 | else: 56 | model.eval() 57 | 58 | desc = 'Train' if is_train else 'Validation' 59 | 60 | total_loss = 0.0 61 | 62 | tqdm_bar = tqdm(data_loader) 63 | print(f'len(tqdm_bar):{len(data_loader)}') 64 | for i, (x_i, _, x_j, _) in enumerate(tqdm_bar): batch_idx, (data, target, batch_view_1, batch_view_2) 65 | x_i = x_i.cuda() if self.use_cuda else x_i 66 | x_j = x_j.cuda() if self.use_cuda else x_j 67 | 68 | x_i = x_i.unsqueeze(3) 69 | x_j = x_j.unsqueeze(3) 70 | 71 | x_i = x_i.float().to(self.device) 72 | x_j = x_j.float().to(self.device) 73 | 74 | _, z_i = model(x_i) 75 | _, z_j = model(x_j) 76 | 77 | loss = loss_func(z_i, z_j) 78 | loss /= self.args.accumulation_steps 79 | 80 | if is_train: 81 | loss.backward() 82 | 83 | if (i + 1) % self.args.accumulation_steps == 0 and is_train: 84 | optimizer.step() 85 | model.zero_grad() 86 | 87 | total_loss += loss.item() 88 | 89 | tqdm_bar.set_description('{} Epoch: [{}] Loss: {:.4f}'.format(desc, epoch, loss.item())) 90 | 91 | return total_loss / (len(data_loader.dataset)) 92 | 93 | def execute_graph(self,model, optimizer, scheduler, epoch, use_cuda): 94 | t_loss = self.train_validate(model, optimizer, True, epoch, use_cuda) 95 | v_loss = self.train_validate(model, optimizer, False, epoch, use_cuda) 96 | 97 | scheduler.step(v_loss) 98 | 99 | if self.use_tb: 100 | self.logger.add_scalar(self.log_dir + '/train-loss', t_loss, epoch) 101 | self.logger.add_scalar(self.log_dir + '/valid-loss', v_loss, epoch) 102 | 103 | return v_loss 104 | 105 | 106 | def excute(self): 107 | # Setup asset directories 108 | if not os.path.exists('models'): 109 | os.makedirs('models') 110 | 111 | if not os.path.exists('runs'): 112 | os.makedirs('runs') 113 | 114 | # Logger 115 | if self.use_tb: 116 | self.logger = SummaryWriter(comment='_' + self.args.uid + '_' + self.args.dataset_name) 117 | 118 | if self.args.dataset_name == 'HAR': 119 | from .config_files.HAR_Configs import Config as Configs 120 | configs = Configs() 121 | in_channels = 9 122 | # valid_loader, 123 | self.train_loader, self.test_loader = data_generator('data/HAR', configs, 'self_supervised') 124 | print(f'len(train_loader):{len(self.train_loader)}') 125 | # print(f'len(valid_loader):{len(self.valid_loader)}') 126 | print(f'len(test_loader):{len(self.test_loader)}') 127 | 128 | elif self.args.dataset_name == 'wisdm': 129 | from .config_files.wisdm_Configs import Config as Configs 130 | configs = Configs() 131 | in_channels = 3 132 | train_loader, valid_loader, test_loader = data_generator('data/wisdm', configs, 'self_supervised') 133 | print(f'len(train_loader):{len(self.train_loader)}') 134 | print(f'len(valid_loader):{len(self.valid_loader)}') 135 | print(f'len(test_loader):{len(self.test_loader)}') 136 | 137 | elif self.args.dataset_name == 'epilepsy': 138 | from .config_files.epilepsy_Configs import Config as Configs 139 | configs = Configs() 140 | in_channels = 1 141 | train_loader, valid_loader, test_loader = data_generator('data/epilepsy', configs, 'self_supervised') 142 | print(f'len(train_loader):{len(self.train_loader)}') 143 | print(f'len(valid_loader):{len(self.valid_loader)}') 144 | print(f'len(test_loader):{len(self.test_loader)}') 145 | 146 | elif self.args.dataset_name == 'SHAR': 147 | from .config_files.SHAR_Configs import Config as Configs 148 | configs = Configs() 149 | in_channels = 3 150 | train_loader, valid_loader, test_loader = data_generator('data/SHAR', configs, 'self_supervised') 151 | print(f'len(train_loader):{len(train_loader)}') 152 | print(f'len(valid_loader):{len(valid_loader)}') 153 | print(f'len(test_loader):{len(test_loader)}') 154 | 155 | 156 | model = resnet50_cifar(self.args.feature_size, self.args.dataset_name).type(self.dtype) -add dataset_name 157 | # resnet50 input_dim=2048 because-- Bottleneck:expansion=4 -- 2048=512*4 158 | # model = resnet18_cifar(args.feature_size, args.dataset_name).type(dtype)-add dataset_name # try resnet18_cifar later 159 | # resnet18 input_dim=512 because-- BasicBlock:expansion=1 -- 512=512*1 160 | 161 | optimizer = optim.Adam(model.parameters(), lr=self.args.lr, weight_decay=self.args.decay_lr) 162 | scheduler = ExponentialLR(optimizer, gamma=self.args.decay_lr) 163 | 164 | # Main training loop 165 | best_loss = np.inf 166 | 167 | # Resume training 168 | if self.args.load_model is not None: 169 | if os.path.isfile(self.args.load_model): 170 | checkpoint = torch.load(self.args.load_model) 171 | model.load_state_dict(checkpoint['model']) 172 | optimizer.load_state_dict(checkpoint['optimizer']) 173 | scheduler.load_state_dict(checkpoint['scheduler']) 174 | best_loss = checkpoint['val_loss'] 175 | epoch = checkpoint['epoch'] 176 | print('Loading model: {}. Resuming from epoch: {}'.format(self.args.load_model, epoch)) 177 | else: 178 | print('Model: {} not found'.format(self.args.load_model)) 179 | 180 | for epoch in range(self.args.epochs): 181 | # v_loss = execute_graph(model, loader, optimizer, scheduler, epoch, use_cuda) 182 | v_loss = self.execute_graph(model, optimizer, scheduler, epoch, self.use_cuda) 183 | if v_loss < best_loss: 184 | best_loss = v_loss 185 | print('Writing model checkpoint') 186 | state = { 187 | 'epoch': epoch, 188 | 'model': model.state_dict(), 189 | 'optimizer': optimizer.state_dict(), 190 | 'scheduler': scheduler.state_dict(), 191 | 'val_loss': v_loss 192 | } 193 | t = time.localtime() 194 | timestamp = time.strftime('%b-%d-%Y_%H%M', t) 195 | file_name = 'models/{}_{}_{}_{:04.4f}.pt'.format(timestamp, self.args.uid, epoch, v_loss) 196 | 197 | torch.save(state, file_name) 198 | 199 | # TensorboardX logger 200 | self.logger.close() 201 | 202 | # save model / restart training 203 | 204 | 205 | -------------------------------------------------------------------------------- /models/SimCLR/utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | def type_tdouble(use_cuda=False): 6 | return torch.cuda.DoubleTensor if use_cuda else torch.DoubleTensor 7 | 8 | 9 | def one_hot(labels, n_class, use_cuda=False): 10 | # Ensure labels are [N x 1] 11 | if len(list(labels.size())) == 1: 12 | labels = labels.unsqueeze(1) 13 | mask = type_tdouble(use_cuda)(labels.size(0), n_class).fill_(0) 14 | # scatter dimension, position indices, fill_value 15 | return mask.scatter_(1, labels, 1) 16 | 17 | 18 | def init_weights(module): 19 | for m in module.modules(): 20 | if isinstance(m, nn.Conv2d): 21 | nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu') 22 | elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)): 23 | nn.init.constant_(m.weight, 1) 24 | nn.init.constant_(m.bias, 0) 25 | -------------------------------------------------------------------------------- /models/SwAV/hubconf.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torchvision 3 | from torchvision.models.resnet import resnet50 as _resnet50 4 | 5 | dependencies = ["torch", "torchvision"] 6 | 7 | 8 | def resnet50(pretrained=True, **kwargs): 9 | """ 10 | ResNet-50 pre-trained with SwAV. 11 | 12 | Note that `fc.weight` and `fc.bias` are randomly initialized. 13 | 14 | Achieves 75.3% top-1 accuracy on ImageNet when `fc` is trained. 15 | """ 16 | model = _resnet50(pretrained=False, **kwargs) 17 | if pretrained: 18 | state_dict = torch.hub.load_state_dict_from_url( 19 | url="https://dl.fbaipublicfiles.com/deepcluster/swav_800ep_pretrain.pth.tar", 20 | map_location="cpu", 21 | ) 22 | # removes "module." 23 | state_dict = {k.replace("module.", ""): v for k, v in state_dict.items()} 24 | # load weights 25 | model.load_state_dict(state_dict, strict=False) 26 | return model 27 | -------------------------------------------------------------------------------- /models/SwAV/logger.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import time 4 | from datetime import timedelta 5 | import pandas as pd 6 | 7 | 8 | class LogFormatter: 9 | def __init__(self): 10 | self.start_time = time.time() 11 | 12 | def format(self, record): 13 | elapsed_seconds = round(record.created - self.start_time) 14 | 15 | prefix = "%s - %s - %s" % ( 16 | record.levelname, 17 | time.strftime("%x %X"), 18 | timedelta(seconds=elapsed_seconds), 19 | ) 20 | message = record.getMessage() 21 | message = message.replace("\n", "\n" + " " * (len(prefix) + 3)) 22 | return "%s - %s" % (prefix, message) if message else "" 23 | 24 | 25 | def create_logger(filepath, rank): 26 | """ 27 | Create a logger. 28 | Use a different log file for each process. 29 | """ 30 | # create log formatter 31 | log_formatter = LogFormatter() 32 | 33 | # create file handler and set level to debug 34 | if filepath is not None: 35 | if rank > 0: 36 | filepath = "%s-%i" % (filepath, rank) 37 | file_handler = logging.FileHandler(filepath, "a") 38 | file_handler.setLevel(logging.DEBUG) 39 | file_handler.setFormatter(log_formatter) 40 | 41 | # create console handler and set level to info 42 | console_handler = logging.StreamHandler() 43 | console_handler.setLevel(logging.INFO) 44 | console_handler.setFormatter(log_formatter) 45 | 46 | # create logger and set level to debug 47 | logger = logging.getLogger() 48 | logger.handlers = [] 49 | logger.setLevel(logging.DEBUG) 50 | logger.propagate = False 51 | if filepath is not None: 52 | logger.addHandler(file_handler) 53 | logger.addHandler(console_handler) 54 | 55 | # reset logger elapsed time 56 | def reset_time(): 57 | log_formatter.start_time = time.time() 58 | 59 | logger.reset_time = reset_time 60 | 61 | return logger 62 | 63 | 64 | class PD_Stats(object): 65 | """ 66 | Log stuff with pandas library 67 | """ 68 | 69 | def __init__(self, path, columns): 70 | self.path = path 71 | 72 | # reload path stats 73 | if os.path.isfile(self.path): 74 | self.stats = pd.read_pickle(self.path) 75 | print(self.stats) 76 | # print(self.stats.columns) #Index(['epoch', 'loss'], dtype='object') 77 | # print(columns) #('epoch', 'loss', 'prec1', 'prec5', 'loss_val', 'prec1_val', 'prec5_val') 78 | # check that columns are the same 79 | assert list(self.stats.columns) == list(columns) 80 | 81 | else: 82 | self.stats = pd.DataFrame(columns=columns) 83 | 84 | def update(self, row, save=True): 85 | self.stats.loc[len(self.stats.index)] = row 86 | 87 | # save the statistics 88 | if save: 89 | self.stats.to_pickle(self.path) 90 | -------------------------------------------------------------------------------- /models/SwAV/utils.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from logging import getLogger 3 | import pickle 4 | import os 5 | 6 | import numpy as np 7 | import torch 8 | import torch.nn as nn 9 | 10 | from .logger import create_logger, PD_Stats 11 | 12 | import torch.distributed as dist 13 | 14 | FALSY_STRINGS = {"off", "false", "0"} 15 | TRUTHY_STRINGS = {"on", "true", "1"} 16 | 17 | 18 | logger = getLogger() 19 | 20 | 21 | def bool_flag(s): 22 | """ 23 | Parse boolean arguments from the command line. 24 | """ 25 | if s.lower() in FALSY_STRINGS: 26 | return False 27 | elif s.lower() in TRUTHY_STRINGS: 28 | return True 29 | else: 30 | raise argparse.ArgumentTypeError("invalid value for a boolean flag") 31 | 32 | 33 | def init_distributed_mode(args): 34 | """ 35 | Initialize the following variables: 36 | - world_size 37 | - rank 38 | """ 39 | 40 | args.is_slurm_job = "SLURM_JOB_ID" in os.environ 41 | 42 | if args.is_slurm_job: 43 | args.rank = int(os.environ["SLURM_PROCID"]) 44 | args.world_size = int(os.environ["SLURM_NNODES"]) * int( 45 | os.environ["SLURM_TASKS_PER_NODE"][0] 46 | ) 47 | else: 48 | # multi-GPU job (local or multi-node) - jobs started with torch.distributed.launch 49 | # read environment variables 50 | args.rank = int(os.environ["RANK"]) 51 | args.world_size = int(os.environ["WORLD_SIZE"]) 52 | 53 | # prepare distributed 54 | dist.init_process_group( 55 | backend="nccl", 56 | init_method=args.dist_url, 57 | world_size=args.world_size, 58 | rank=args.rank, 59 | ) 60 | 61 | # set cuda device 62 | args.gpu_to_work_on = args.rank % torch.cuda.device_count() 63 | torch.cuda.set_device(args.gpu_to_work_on) 64 | return 65 | 66 | 67 | def initialize_exp(params, *args, dump_params=True): 68 | if dump_params: 69 | # print(params.dump_path) 70 | pickle.dump(params, open(os.path.join(params.dump_path, "params.pkl"), "wb")) 71 | 72 | # create repo to store checkpoints 73 | params.dump_checkpoints = os.path.join(params.dump_path, "checkpoints") 74 | if not params.rank and not os.path.isdir(params.dump_checkpoints): 75 | os.mkdir(params.dump_checkpoints) 76 | print(params.rank) 77 | # create a panda object to log loss and acc 78 | training_stats = PD_Stats( 79 | os.path.join(params.dump_path, "stats" + str(params.rank) + ".pkl"), args 80 | ) 81 | 82 | # create a logger 83 | logger = create_logger( 84 | os.path.join(params.dump_path, "train.log"), rank=params.rank 85 | ) 86 | logger.info("============ Initialized logger ============") 87 | logger.info( 88 | "\n".join("%s: %s" % (k, str(v)) for k, v in sorted(dict(vars(params)).items())) 89 | ) 90 | logger.info("The experiment will be stored in %s\n" % params.dump_path) 91 | logger.info("") 92 | return logger, training_stats 93 | 94 | 95 | def restart_from_checkpoint(ckp_paths, run_variables=None, **kwargs): 96 | """ 97 | Re-start from checkpoint 98 | """ 99 | # look for a checkpoint in exp repository 100 | if isinstance(ckp_paths, list): 101 | for ckp_path in ckp_paths: 102 | if os.path.isfile(ckp_path): 103 | break 104 | else: 105 | ckp_path = ckp_paths 106 | 107 | if not os.path.isfile(ckp_path): 108 | return 109 | 110 | logger.info("Found checkpoint at {}".format(ckp_path)) 111 | 112 | # open checkpoint file 113 | checkpoint = torch.load( 114 | ckp_path, map_location="cuda:" + str(torch.distributed.get_rank() % torch.cuda.device_count()) 115 | ) 116 | 117 | for key, value in kwargs.items(): 118 | if key in checkpoint and value is not None: 119 | try: 120 | msg = value.load_state_dict(checkpoint[key], strict=False) 121 | print(msg) 122 | except TypeError: 123 | msg = value.load_state_dict(checkpoint[key]) 124 | logger.info("=> loaded {} from checkpoint '{}'".format(key, ckp_path)) 125 | else: 126 | logger.warning( 127 | "=> failed to load {} from checkpoint '{}'".format(key, ckp_path) 128 | ) 129 | 130 | # re load variable important for the run 131 | if run_variables is not None: 132 | for var_name in run_variables: 133 | if var_name in checkpoint: 134 | run_variables[var_name] = checkpoint[var_name] 135 | 136 | 137 | def fix_random_seeds(seed=31): 138 | """ 139 | Fix random seeds. 140 | """ 141 | torch.manual_seed(seed) 142 | torch.cuda.manual_seed_all(seed) 143 | np.random.seed(seed) 144 | 145 | 146 | class AverageMeter(object): 147 | """computes and stores the average and current value""" 148 | 149 | def __init__(self): 150 | self.reset() 151 | 152 | def reset(self): 153 | self.val = 0 154 | self.avg = 0 155 | self.sum = 0 156 | self.count = 0 157 | 158 | def update(self, val, n=1): 159 | self.val = val 160 | self.sum += val * n 161 | self.count += n 162 | self.avg = self.sum / self.count 163 | 164 | 165 | def accuracy(output, target, topk=(1,)): 166 | """Computes the accuracy over the k top predictions for the specified values of k""" 167 | with torch.no_grad(): 168 | maxk = max(topk) 169 | batch_size = target.size(0) 170 | 171 | _, pred = output.topk(maxk, 1, True, True) 172 | pred = pred.t() 173 | correct = pred.eq(target.view(1, -1).expand_as(pred)) 174 | 175 | res = [] 176 | for k in topk: 177 | # correct_k = correct[:k].view(-1).float().sum(0, keepdim=True) 178 | correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True) #mqw 179 | res.append(correct_k.mul_(100.0 / batch_size)) 180 | return res 181 | -------------------------------------------------------------------------------- /models/TLoss/train_TLoss.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import math 4 | import torch 5 | import numpy 6 | import argparse 7 | import numpy as np 8 | from torch.utils.data import Dataset, DataLoader 9 | 10 | import scikit_wrappers 11 | 12 | 13 | class train_TLoss(): 14 | def __init__(self,parser): 15 | self.args=parser.parse_args() 16 | self.dataset = self.args.dataset_name 17 | self.path = self.args.data_dir 18 | self.save_path = self.args.exp_dir 19 | self.cuda = not self.args.no_cuda 20 | self.gpu = self.args.device_id 21 | self.hyper = self.args.hyper 22 | self.load = self.args.load 23 | self.fit_classifier = self.args.fit_classifier 24 | 25 | def fit_hyperparameters(file, train, train_labels, cuda, gpu, 26 | save_memory=False): 27 | classifier = scikit_wrappers.CausalCNNEncoderClassifier() 28 | 29 | # Loads a given set of hyperparameters and fits a model with those 30 | hf = open(os.path.join(file), 'r') 31 | params = json.load(hf) 32 | hf.close() 33 | # Check the number of input channels 34 | params['in_channels'] = numpy.shape(train)[1] 35 | params['cuda'] = cuda 36 | params['gpu'] = gpu 37 | classifier.set_params(**params) 38 | return classifier.fit( 39 | train, train_labels, save_memory=save_memory, verbose=True 40 | ) 41 | 42 | def excute(self): 43 | train_dataset = torch.load(os.path.join(self.path, "train.pt")) 44 | # valid_dataset = torch.load(os.path.join(self.path, "val.pt")) 45 | test_dataset = torch.load(os.path.join(self.path, "test.pt")) 46 | train = np.array(train_dataset["samples"]) 47 | train = train.astype(float) 48 | train_labels = np.array(train_dataset["labels"]) 49 | test = np.array(test_dataset["samples"]) 50 | test = test.astype(float) 51 | test_labels = np.array(test_dataset["labels"]) 52 | 53 | if self.dataset not in ('HAR','wisdm','SHAR','epilepsy'): 54 | print('UEA-----need permute') 55 | train = np.swapaxes(train, 1, 2) 56 | test = np.swapaxes(test, 1, 2) 57 | 58 | print(f'self.load:{self.load}')#False-training, True-load 59 | print(f'self.fit_classifier:{self.fit_classifier}')#True-load the model and retrain the classifier 60 | if not self.load and not self.fit_classifier: 61 | classifier = fit_hyperparameters( 62 | self.hyper, train, train_labels, self.cuda, self.gpu, 63 | save_memory=True 64 | ) 65 | else: 66 | classifier = scikit_wrappers.CausalCNNEncoderClassifier() 67 | hf = open( 68 | os.path.join( 69 | self.save_path, self.dataset + '_hyperparameters.json' 70 | ), 'r' 71 | ) 72 | hp_dict = json.load(hf) 73 | hf.close() 74 | hp_dict['cuda'] = self.cuda 75 | hp_dict['gpu'] = self.gpu 76 | classifier.set_params(**hp_dict) 77 | classifier.load(os.path.join(self.save_path, self.dataset)) 78 | 79 | if not self.load:#load=false 80 | if self.fit_classifier:#fit_classifier=true 81 | classifier.fit_classifier(classifier.encode(train), train_labels) 82 | classifier.save( 83 | os.path.join(self.save_path, self.dataset) 84 | ) 85 | with open( 86 | os.path.join( 87 | self.save_path, self.dataset + '_hyperparameters.json' 88 | ), 'w' 89 | ) as fp: 90 | json.dump(classifier.get_params(), fp) 91 | print("Test accuracy: " + str(classifier.mqw_score(test, test_labels, self.dataset)))dataset_name 92 | -------------------------------------------------------------------------------- /models/TS2Vec/datautils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import torch 4 | import pandas as pd 5 | import math 6 | import random 7 | from datetime import datetime 8 | import pickle 9 | from scipy.io.arff import loadarff 10 | from sklearn.preprocessing import StandardScaler, MinMaxScaler 11 | from ..data_loader.dataset_loader import data_generator 12 | 13 | def load_UCR(dataset): 14 | train_file = os.path.join('datasets/UCR', dataset, dataset + "_TRAIN.tsv") 15 | test_file = os.path.join('datasets/UCR', dataset, dataset + "_TEST.tsv") 16 | train_df = pd.read_csv(train_file, sep='\t', header=None) 17 | test_df = pd.read_csv(test_file, sep='\t', header=None) 18 | train_array = np.array(train_df) 19 | test_array = np.array(test_df) 20 | 21 | # Move the labels to {0, ..., L-1} 22 | labels = np.unique(train_array[:, 0]) 23 | transform = {} 24 | for i, l in enumerate(labels): 25 | transform[l] = i 26 | 27 | train = train_array[:, 1:].astype(np.float64) 28 | train_labels = np.vectorize(transform.get)(train_array[:, 0]) 29 | test = test_array[:, 1:].astype(np.float64) 30 | test_labels = np.vectorize(transform.get)(test_array[:, 0]) 31 | 32 | # Normalization for non-normalized datasets 33 | # To keep the amplitude information, we do not normalize values over 34 | # individual time series, but on the whole dataset 35 | if dataset not in [ 36 | 'AllGestureWiimoteX', 37 | 'AllGestureWiimoteY', 38 | 'AllGestureWiimoteZ', 39 | 'BME', 40 | 'Chinatown', 41 | 'Crop', 42 | 'EOGHorizontalSignal', 43 | 'EOGVerticalSignal', 44 | 'Fungi', 45 | 'GestureMidAirD1', 46 | 'GestureMidAirD2', 47 | 'GestureMidAirD3', 48 | 'GesturePebbleZ1', 49 | 'GesturePebbleZ2', 50 | 'GunPointAgeSpan', 51 | 'GunPointMaleVersusFemale', 52 | 'GunPointOldVersusYoung', 53 | 'HouseTwenty', 54 | 'InsectEPGRegularTrain', 55 | 'InsectEPGSmallTrain', 56 | 'MelbournePedestrian', 57 | 'PickupGestureWiimoteZ', 58 | 'PigAirwayPressure', 59 | 'PigArtPressure', 60 | 'PigCVP', 61 | 'PLAID', 62 | 'PowerCons', 63 | 'Rock', 64 | 'SemgHandGenderCh2', 65 | 'SemgHandMovementCh2', 66 | 'SemgHandSubjectCh2', 67 | 'ShakeGestureWiimoteZ', 68 | 'SmoothSubspace', 69 | 'UMD' 70 | ]: 71 | return train[..., np.newaxis], train_labels, test[..., np.newaxis], test_labels 72 | 73 | mean = np.nanmean(train) 74 | std = np.nanstd(train) 75 | train = (train - mean) / std 76 | test = (test - mean) / std 77 | return train[..., np.newaxis], train_labels, test[..., np.newaxis], test_labels 78 | 79 | 80 | def load_UEA_origin(dataset): 81 | train_data = loadarff(f'datasets/UEA/{dataset}/{dataset}_TRAIN.arff')[0] 82 | test_data = loadarff(f'datasets/UEA/{dataset}/{dataset}_TEST.arff')[0] 83 | 84 | def extract_data(data): 85 | res_data = [] 86 | res_labels = [] 87 | for t_data, t_label in data: 88 | t_data = np.array([ d.tolist() for d in t_data ]) 89 | t_label = t_label.decode("utf-8") 90 | res_data.append(t_data) 91 | res_labels.append(t_label) 92 | return np.array(res_data).swapaxes(1, 2), np.array(res_labels) 93 | 94 | train_X, train_y = extract_data(train_data) 95 | test_X, test_y = extract_data(test_data) 96 | 97 | scaler = StandardScaler() 98 | scaler.fit(train_X.reshape(-1, train_X.shape[-1])) 99 | train_X = scaler.transform(train_X.reshape(-1, train_X.shape[-1])).reshape(train_X.shape) 100 | test_X = scaler.transform(test_X.reshape(-1, test_X.shape[-1])).reshape(test_X.shape) 101 | 102 | labels = np.unique(train_y) 103 | transform = { k : i for i, k in enumerate(labels)} 104 | train_y = np.vectorize(transform.get)(train_y) 105 | test_y = np.vectorize(transform.get)(test_y) 106 | print(train_X.shape) #[40,100,6] [Train Size, Length, channels/variables] 107 | print(train_X) 108 | print(train_y.shape) #[40,] 109 | print(train_y) #[2 2 2 2 1 1 3 0.......] 110 | return train_X, train_y, test_X, test_y 111 | 112 | 113 | def load_UEA(dataset): 114 | print('************UEA archive************') 115 | # UEA archive 116 | print(f'dataset:{dataset}') 117 | train_data = torch.load(f'datasets/{dataset}/train.pt') 118 | test_data = torch.load(f'datasets/{dataset}/test.pt') 119 | # dat_dict = dict() 120 | train_X = train_data["samples"].numpy() 121 | train_y = train_data["labels"].numpy() 122 | test_X = test_data["samples"].numpy() 123 | test_y = test_data["labels"].numpy() 124 | 125 | print(f'train_X.shape:{train_X.shape}') # BasicMotions[40,100,6] [Train Size, Length, channels/variables] HAR[5881,128,9] 126 | print(f'test_X.shape:{test_X.shape}') 127 | # print(train_X) 128 | print(f'train_y.shape:{train_y.shape}') # BasicMotions[40,] HAR[5881,] 129 | # print(train_y) # BasicMotions[2 2 2 2 1 1 3 0.......] HAR[5. 1. 1. ... 0. 4. 0.] 130 | return train_X, train_y, test_X, test_y 131 | 132 | def load_MTS(dataset): 133 | print('************HAR / SHAR / wisdm************') 134 | # MTS: multivariate time series 135 | # datasets: HAR / SHAR / wisdm 136 | # train_data = torch.load(f'datasets/{dataset}/train.pt') 137 | # test_data = torch.load(f'datasets/{dataset}/test.pt') 138 | # from config_files.HAR_Configs import Config as Configs 139 | # configs = Configs() 140 | # train_data , valid_dl, test_data = data_generator('./datasets/HAR', configs, 'self_supervised') # train_linear 141 | print(f'dataset:{dataset}') 142 | train_data = torch.load(f'datasets/{dataset}/train.pt') 143 | test_data = torch.load(f'datasets/{dataset}/test.pt') 144 | # dat_dict = dict() 145 | train_X = train_data["samples"].numpy() 146 | train_y = train_data["labels"].numpy() 147 | test_X = test_data["samples"].numpy() 148 | test_y = test_data["labels"].numpy() 149 | 150 | 151 | print(f'train_X.shape:{train_X.shape}') # BasicMotions[40,100,6] [Train Size, Length, channels] HAR[5881,128,9] 152 | print(f'test_X.shape:{test_X.shape}') 153 | # print(train_X) 154 | print(f'train_y.shape:{train_y.shape}') # BasicMotions[40,] HAR[5881,] 155 | # print(train_y) # BasicMotions[2 2 2 2 1 1 3 0.......] HAR[5. 1. 1. ... 0. 4. 0.] 156 | return train_X, train_y, test_X, test_y 157 | 158 | def load_UTS(dataset): 159 | # UTS: Univariate time series 160 | # datasets: epilepsy / sleepEDF 161 | print(f'dataset:{dataset}') 162 | train_data = torch.load(f'datasets/{dataset}/train.pt') 163 | test_data = torch.load(f'datasets/{dataset}/test.pt') 164 | # dat_dict = dict() 165 | train_X = train_data["samples"].numpy() 166 | train_y = train_data["labels"].numpy() 167 | test_X = test_data["samples"].numpy() 168 | test_y = test_data["labels"].numpy() 169 | 170 | 171 | train = train_X[:, 1:].astype(np.float64) 172 | train_labels = np.vectorize(train_y[:, 0]) 173 | test = test_X[:, 1:].astype(np.float64) 174 | test_labels = np.vectorize(test_y[:, 0]) 175 | 176 | mean = np.nanmean(train) 177 | std = np.nanstd(train) 178 | train = (train - mean) / std 179 | test = (test - mean) / std 180 | return train[..., np.newaxis], train_labels, test[..., np.newaxis], test_labels 181 | 182 | 183 | def load_forecast_npy(name, univar=False): 184 | data = np.load(f'datasets/{name}.npy') 185 | if univar: 186 | data = data[: -1:] 187 | 188 | train_slice = slice(None, int(0.6 * len(data))) 189 | valid_slice = slice(int(0.6 * len(data)), int(0.8 * len(data))) 190 | test_slice = slice(int(0.8 * len(data)), None) 191 | 192 | scaler = StandardScaler().fit(data[train_slice]) 193 | data = scaler.transform(data) 194 | data = np.expand_dims(data, 0) 195 | 196 | pred_lens = [24, 48, 96, 288, 672] 197 | return data, train_slice, valid_slice, test_slice, scaler, pred_lens, 0 198 | 199 | def _get_time_features(dt): 200 | return np.stack([ 201 | dt.minute.to_numpy(), 202 | dt.hour.to_numpy(), 203 | dt.dayofweek.to_numpy(), 204 | dt.day.to_numpy(), 205 | dt.dayofyear.to_numpy(), 206 | dt.month.to_numpy(), 207 | dt.weekofyear.to_numpy(), 208 | ], axis=1).astype(np.float) 209 | 210 | def load_forecast_csv(name, univar=False): 211 | data = pd.read_csv(f'datasets/{name}.csv', index_col='date', parse_dates=True) 212 | dt_embed = _get_time_features(data.index) 213 | n_covariate_cols = dt_embed.shape[-1] 214 | 215 | if univar: 216 | if name in ('ETTh1', 'ETTh2', 'ETTm1', 'ETTm2'): 217 | data = data[['OT']] 218 | elif name == 'electricity': 219 | data = data[['MT_001']] 220 | else: 221 | data = data.iloc[:, -1:] 222 | 223 | data = data.to_numpy() 224 | if name == 'ETTh1' or name == 'ETTh2': 225 | train_slice = slice(None, 12*30*24) 226 | valid_slice = slice(12*30*24, 16*30*24) 227 | test_slice = slice(16*30*24, 20*30*24) 228 | elif name == 'ETTm1' or name == 'ETTm2': 229 | train_slice = slice(None, 12*30*24*4) 230 | valid_slice = slice(12*30*24*4, 16*30*24*4) 231 | test_slice = slice(16*30*24*4, 20*30*24*4) 232 | else: 233 | train_slice = slice(None, int(0.6 * len(data))) 234 | valid_slice = slice(int(0.6 * len(data)), int(0.8 * len(data))) 235 | test_slice = slice(int(0.8 * len(data)), None) 236 | 237 | scaler = StandardScaler().fit(data[train_slice]) 238 | data = scaler.transform(data) 239 | if name in ('electricity'): 240 | data = np.expand_dims(data.T, -1) # Each variable is an instance rather than a feature 241 | else: 242 | data = np.expand_dims(data, 0) 243 | 244 | if n_covariate_cols > 0: 245 | dt_scaler = StandardScaler().fit(dt_embed[train_slice]) 246 | dt_embed = np.expand_dims(dt_scaler.transform(dt_embed), 0) 247 | data = np.concatenate([np.repeat(dt_embed, data.shape[0], axis=0), data], axis=-1) 248 | 249 | if name in ('ETTh1', 'ETTh2', 'electricity'): 250 | pred_lens = [24, 48, 168, 336, 720] 251 | else: 252 | pred_lens = [24, 48, 96, 288, 672] 253 | 254 | return data, train_slice, valid_slice, test_slice, scaler, pred_lens, n_covariate_cols 255 | -------------------------------------------------------------------------------- /models/TS2Vec/train_TS2Vec.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import time 4 | import datetime 5 | from ts2vec import TS2Vec 6 | import tasks 7 | import datautils 8 | from utils import init_dl_program, name_with_datetime, pkl_save, data_dropout 9 | import matplotlib.pyplot as plt 10 | import numpy as np 11 | 12 | 13 | Class train_TS2Vec(): 14 | 15 | def __init__(self, parser): 16 | self.args = parser.parse_args() 17 | self.method = 'TS2Vec' 18 | self.dataset_name = self.args.dataset_name 19 | self.run_name = self.args.run_description 20 | self.archive = self.args.archive 21 | self.gpu = self.args.device_id 22 | self.batch_size = self.args.batch_size 23 | self.lr = self.args.lr 24 | self.repr_dims = self.args.feature_size 25 | self.max_train_length = self.args.max_train_length 26 | self.iters = self.args.iters 27 | self.epochs = self.args.epochs 28 | self.save_every = self.args.checkpoint_freq 29 | self.seed = self.args.seed 30 | self.max_threads = self.args.max_threads 31 | self.eval = self.args.eval 32 | self.irregular = self.args.irregular 33 | 34 | def excute(self): 35 | device = init_dl_program(self.gpu, seed=self.seed, max_threads=self.max_threads) 36 | 37 | if self.archive == 'UCR': 38 | task_type = 'classification' 39 | train_data, train_labels, test_data, test_labels = datautils.load_UCR(self.dataset) 40 | 41 | elif self.archive == 'UEA': 42 | print('UEA') 43 | task_type = 'classification' 44 | train_data, train_labels, test_data, test_labels = datautils.load_UEA(self.dataset) 45 | elif self.archive == 'HAR' or 'SHAR' or 'wisdm' or 'epilepsy' or 'sleepEDF': 46 | print('HAR/SHAR/wisdm/epilepsy') 47 | task_type = 'classification' 48 | train_data, train_labels, test_data, test_labels = datautils.load_MTS(self.dataset) 49 | 50 | elif self.archive == 'forecast_csv': 51 | task_type = 'forecasting' 52 | data, train_slice, valid_slice, test_slice, scaler, pred_lens, n_covariate_cols = datautils.load_forecast_csv(self.dataset) 53 | train_data = data[:, train_slice] 54 | elif self.archive == 'forecast_csv_univar': 55 | task_type = 'forecasting' 56 | data, train_slice, valid_slice, test_slice, scaler, pred_lens, n_covariate_cols = datautils.load_forecast_csv(self.dataset, univar=True) 57 | train_data = data[:, train_slice] 58 | elif self.archive == 'forecast_npy': 59 | task_type = 'forecasting' 60 | data, train_slice, valid_slice, test_slice, scaler, pred_lens, n_covariate_cols = datautils.load_forecast_npy(self.dataset) 61 | train_data = data[:, train_slice] 62 | elif self.archive == 'forecast_npy_univar': 63 | task_type = 'forecasting' 64 | data, train_slice, valid_slice, test_slice, scaler, pred_lens, n_covariate_cols = datautils.load_forecast_npy(self.dataset, univar=True) 65 | train_data = data[:, train_slice] 66 | else: 67 | raise ValueError(f"Archive type {self.archive} is not supported.") 68 | 69 | if self.irregular > 0: 70 | if task_type == 'classification': 71 | train_data = data_dropout(train_data, self.irregular) 72 | test_data = data_dropout(test_data, self.irregular) 73 | else: 74 | raise ValueError(f"Task type {task_type} is not supported when irregular is positive.") 75 | 76 | config = dict( 77 | batch_size=self.batch_size, 78 | lr=self.lr, 79 | output_dims=self.repr_dims, 80 | max_train_length=self.max_train_length 81 | ) 82 | 83 | if self.save_every is not None: 84 | unit = 'epoch' if self.epochs is not None else 'iter' 85 | config[f'after_{unit}_callback'] = save_checkpoint_callback(self.save_every, unit) 86 | 87 | run_dir = 'training/' + self.dataset + '__' + name_with_datetime(self.run_name) 88 | os.makedirs(run_dir, exist_ok=True) 89 | 90 | t = time.time() 91 | 92 | model = TS2Vec( 93 | input_dims=train_data.shape[-1], 94 | device=device, 95 | **config 96 | ) 97 | loss_log = model.fit( 98 | train_data, 99 | n_epochs=self.epochs, 100 | n_iters=self.iters, 101 | verbose=True 102 | ) 103 | 104 | 105 | train_loss = loss_log 106 | plt.figure() 107 | plt.plot(np.arange(self.epochs), train_loss, label="Train") 108 | plt.title("Loss_%s"%self.dataset) 109 | plt.legend() 110 | plt.savefig(os.path.join("./loss/",'loss_%s.png'%self.dataset)) 111 | plt.show() 112 | 113 | model.save(f'{run_dir}/model.pkl') 114 | 115 | t = time.time() - t 116 | print(f"\nTraining time: {datetime.timedelta(seconds=t)}\n") 117 | 118 | if self.eval: 119 | if task_type == 'classification': #mqw dataset=self.dataset,并更改了change-svm 120 | # out, eval_res = tasks.eval_classification(model, train_data, train_labels, test_data, test_labels, dataset=self.dataset, eval_protocol='svm') 121 | out, eval_res = tasks.eval_classification(model, train_data, train_labels, test_data, test_labels, dataset=self.dataset, eval_protocol='linear') 122 | elif task_type == 'forecasting': 123 | out, eval_res = tasks.eval_forecasting(model, data, train_slice, valid_slice, test_slice, scaler, pred_lens, n_covariate_cols) 124 | else: 125 | assert False 126 | pkl_save(f'{run_dir}/out.pkl', out) 127 | pkl_save(f'{run_dir}/eval_res.pkl', eval_res) 128 | print('Evaluation result:', eval_res) 129 | 130 | print("Finished.") 131 | 132 | def save_checkpoint_callback(save_every=1, unit='epoch'): 133 | assert unit in ('epoch', 'iter') 134 | def callback(model, loss): 135 | n = model.n_epochs if unit == 'epoch' else model.n_iters 136 | if n % save_every == 0: 137 | model.save(f'{run_dir}/model_{n}.pkl') 138 | return callback 139 | -------------------------------------------------------------------------------- /models/TS2Vec/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import pickle 4 | import torch 5 | import random 6 | from datetime import datetime 7 | 8 | def pkl_save(name, var): 9 | with open(name, 'wb') as f: 10 | pickle.dump(var, f) 11 | 12 | def pkl_load(name): 13 | with open(name, 'rb') as f: 14 | return pickle.load(f) 15 | 16 | def torch_pad_nan(arr, left=0, right=0, dim=0): 17 | if left > 0: 18 | padshape = list(arr.shape) 19 | padshape[dim] = left 20 | arr = torch.cat((torch.full(padshape, np.nan), arr), dim=dim) 21 | if right > 0: 22 | padshape = list(arr.shape) 23 | padshape[dim] = right 24 | arr = torch.cat((arr, torch.full(padshape, np.nan)), dim=dim) 25 | return arr 26 | 27 | def pad_nan_to_target(array, target_length, axis=0, both_side=False): 28 | assert array.dtype in [np.float16, np.float32, np.float64] 29 | pad_size = target_length - array.shape[axis] 30 | if pad_size <= 0: 31 | return array 32 | npad = [(0, 0)] * array.ndim 33 | if both_side: 34 | npad[axis] = (pad_size // 2, pad_size - pad_size//2) 35 | else: 36 | npad[axis] = (0, pad_size) 37 | return np.pad(array, pad_width=npad, mode='constant', constant_values=np.nan) 38 | 39 | def split_with_nan(x, sections, axis=0): 40 | assert x.dtype in [np.float16, np.float32, np.float64] 41 | arrs = np.array_split(x, sections, axis=axis) 42 | target_length = arrs[0].shape[axis] 43 | for i in range(len(arrs)): 44 | arrs[i] = pad_nan_to_target(arrs[i], target_length, axis=axis) 45 | return arrs 46 | 47 | def take_per_row(A, indx, num_elem): 48 | all_indx = indx[:,None] + np.arange(num_elem) 49 | return A[torch.arange(all_indx.shape[0])[:,None], all_indx] 50 | 51 | def centerize_vary_length_series(x): 52 | prefix_zeros = np.argmax(~np.isnan(x).all(axis=-1), axis=1) 53 | suffix_zeros = np.argmax(~np.isnan(x[:, ::-1]).all(axis=-1), axis=1) 54 | offset = (prefix_zeros + suffix_zeros) // 2 - prefix_zeros 55 | rows, column_indices = np.ogrid[:x.shape[0], :x.shape[1]] 56 | offset[offset < 0] += x.shape[1] 57 | column_indices = column_indices - offset[:, np.newaxis] 58 | return x[rows, column_indices] 59 | 60 | def data_dropout(arr, p): 61 | B, T = arr.shape[0], arr.shape[1] 62 | mask = np.full(B*T, False, dtype=np.bool) 63 | ele_sel = np.random.choice( 64 | B*T, 65 | size=int(B*T*p), 66 | replace=False 67 | ) 68 | mask[ele_sel] = True 69 | res = arr.copy() 70 | res[mask.reshape(B, T)] = np.nan 71 | return res 72 | 73 | def name_with_datetime(prefix='default'): 74 | now = datetime.now() 75 | return prefix + '_' + now.strftime("%Y%m%d_%H%M%S") 76 | 77 | def init_dl_program( 78 | device_name, 79 | seed=None, 80 | use_cudnn=True, 81 | deterministic=False, 82 | benchmark=False, 83 | use_tf32=False, 84 | max_threads=None 85 | ): 86 | import torch 87 | if max_threads is not None: 88 | torch.set_num_threads(max_threads) # intraop 89 | if torch.get_num_interop_threads() != max_threads: 90 | torch.set_num_interop_threads(max_threads) # interop 91 | try: 92 | import mkl 93 | except: 94 | pass 95 | else: 96 | mkl.set_num_threads(max_threads) 97 | 98 | if seed is not None: 99 | random.seed(seed) 100 | seed += 1 101 | np.random.seed(seed) 102 | seed += 1 103 | torch.manual_seed(seed) 104 | 105 | if isinstance(device_name, (str, int)): 106 | device_name = [device_name] 107 | 108 | devices = [] 109 | for t in reversed(device_name): 110 | t_device = torch.device(t) 111 | devices.append(t_device) 112 | if t_device.type == 'cuda': 113 | assert torch.cuda.is_available() 114 | torch.cuda.set_device(t_device) 115 | if seed is not None: 116 | seed += 1 117 | torch.cuda.manual_seed(seed) 118 | devices.reverse() 119 | torch.backends.cudnn.enabled = use_cudnn 120 | torch.backends.cudnn.deterministic = deterministic 121 | torch.backends.cudnn.benchmark = benchmark 122 | 123 | if hasattr(torch.backends.cudnn, 'allow_tf32'): 124 | torch.backends.cudnn.allow_tf32 = use_tf32 125 | torch.backends.cuda.matmul.allow_tf32 = use_tf32 126 | 127 | return devices if len(devices) > 1 else devices[0] 128 | 129 | -------------------------------------------------------------------------------- /models/TST/loss.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from torch.nn import functional as F 4 | 5 | 6 | def get_loss_module(config): 7 | 8 | task = config['task'] 9 | 10 | if (task == "imputation") or (task == "transduction"): 11 | return MaskedMSELoss(reduction='none') # outputs loss for each batch element 12 | 13 | if task == "classification": 14 | return NoFussCrossEntropyLoss(reduction='none') # outputs loss for each batch sample 15 | 16 | if task == "regression": 17 | return nn.MSELoss(reduction='none') # outputs loss for each batch sample 18 | 19 | else: 20 | raise ValueError("Loss module for task '{}' does not exist".format(task)) 21 | 22 | 23 | def l2_reg_loss(model): 24 | """Returns the squared L2 norm of output layer of given model""" 25 | 26 | for name, param in model.named_parameters(): 27 | if name == 'output_layer.weight': 28 | return torch.sum(torch.square(param)) 29 | 30 | 31 | class NoFussCrossEntropyLoss(nn.CrossEntropyLoss): 32 | """ 33 | pytorch's CrossEntropyLoss is fussy: 1) needs Long (int64) targets only, and 2) only 1D. 34 | This function satisfies these requirements 35 | """ 36 | 37 | def forward(self, inp, target): 38 | return F.cross_entropy(inp, target.long().squeeze(), weight=self.weight, 39 | ignore_index=self.ignore_index, reduction=self.reduction) 40 | 41 | 42 | class MaskedMSELoss(nn.Module): 43 | """ Masked MSE Loss 44 | """ 45 | 46 | def __init__(self, reduction: str = 'mean'): 47 | 48 | super().__init__() 49 | 50 | self.reduction = reduction 51 | self.mse_loss = nn.MSELoss(reduction=self.reduction) 52 | 53 | def forward(self, 54 | y_pred: torch.Tensor, y_true: torch.Tensor, mask: torch.BoolTensor) -> torch.Tensor: 55 | """Compute the loss between a target value and a prediction. 56 | 57 | Args: 58 | y_pred: Estimated values 59 | y_true: Target values 60 | mask: boolean tensor with 0s at places where values should be ignored and 1s where they should be considered 61 | 62 | Returns 63 | ------- 64 | if reduction == 'none': 65 | (num_active,) Loss for each active batch element as a tensor with gradient attached. 66 | if reduction == 'mean': 67 | scalar mean loss over batch as a tensor with gradient attached. 68 | """ 69 | 70 | # for this particular loss, one may also elementwise multiply y_pred and y_true with the inverted mask 71 | masked_pred = torch.masked_select(y_pred, mask) 72 | masked_true = torch.masked_select(y_true, mask) 73 | 74 | return self.mse_loss(masked_pred, masked_true) 75 | -------------------------------------------------------------------------------- /models/TST/optimizers.py: -------------------------------------------------------------------------------- 1 | import math 2 | import torch 3 | from torch.optim.optimizer import Optimizer 4 | 5 | 6 | def get_optimizer(name): 7 | 8 | if name == "Adam": 9 | return torch.optim.Adam 10 | elif name == "RAdam": 11 | return RAdam 12 | 13 | 14 | # from https://github.com/LiyuanLucasLiu/RAdam/blob/master/radam/radam.py 15 | class RAdam(Optimizer): 16 | 17 | def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, degenerated_to_sgd=True): 18 | if not 0.0 <= lr: 19 | raise ValueError("Invalid learning rate: {}".format(lr)) 20 | if not 0.0 <= eps: 21 | raise ValueError("Invalid epsilon value: {}".format(eps)) 22 | if not 0.0 <= betas[0] < 1.0: 23 | raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0])) 24 | if not 0.0 <= betas[1] < 1.0: 25 | raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1])) 26 | 27 | self.degenerated_to_sgd = degenerated_to_sgd 28 | if isinstance(params, (list, tuple)) and len(params) > 0 and isinstance(params[0], dict): 29 | for param in params: 30 | if 'betas' in param and (param['betas'][0] != betas[0] or param['betas'][1] != betas[1]): 31 | param['buffer'] = [[None, None, None] for _ in range(10)] 32 | defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, 33 | buffer=[[None, None, None] for _ in range(10)]) 34 | super(RAdam, self).__init__(params, defaults) 35 | 36 | def __setstate__(self, state): 37 | super(RAdam, self).__setstate__(state) 38 | 39 | def step(self, closure=None): 40 | 41 | loss = None 42 | if closure is not None: 43 | loss = closure() 44 | 45 | for group in self.param_groups: 46 | 47 | for p in group['params']: 48 | if p.grad is None: 49 | continue 50 | grad = p.grad.data.float() 51 | if grad.is_sparse: 52 | raise RuntimeError('RAdam does not support sparse gradients') 53 | 54 | p_data_fp32 = p.data.float() 55 | 56 | state = self.state[p] 57 | 58 | if len(state) == 0: 59 | state['step'] = 0 60 | state['exp_avg'] = torch.zeros_like(p_data_fp32) 61 | state['exp_avg_sq'] = torch.zeros_like(p_data_fp32) 62 | else: 63 | state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32) 64 | state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_data_fp32) 65 | 66 | exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq'] 67 | beta1, beta2 = group['betas'] 68 | 69 | exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) 70 | exp_avg.mul_(beta1).add_(1 - beta1, grad) 71 | 72 | state['step'] += 1 73 | buffered = group['buffer'][int(state['step'] % 10)] 74 | if state['step'] == buffered[0]: 75 | N_sma, step_size = buffered[1], buffered[2] 76 | else: 77 | buffered[0] = state['step'] 78 | beta2_t = beta2 ** state['step'] 79 | N_sma_max = 2 / (1 - beta2) - 1 80 | N_sma = N_sma_max - 2 * state['step'] * beta2_t / (1 - beta2_t) 81 | buffered[1] = N_sma 82 | 83 | # more conservative since it's an approximated value 84 | if N_sma >= 5: 85 | step_size = math.sqrt( 86 | (1 - beta2_t) * (N_sma - 4) / (N_sma_max - 4) * (N_sma - 2) / N_sma * N_sma_max / ( 87 | N_sma_max - 2)) / (1 - beta1 ** state['step']) 88 | elif self.degenerated_to_sgd: 89 | step_size = 1.0 / (1 - beta1 ** state['step']) 90 | else: 91 | step_size = -1 92 | buffered[2] = step_size 93 | 94 | # more conservative since it's an approximated value 95 | if N_sma >= 5: 96 | if group['weight_decay'] != 0: 97 | p_data_fp32.add_(-group['weight_decay'] * group['lr'], p_data_fp32) 98 | denom = exp_avg_sq.sqrt().add_(group['eps']) 99 | p_data_fp32.addcdiv_(-step_size * group['lr'], exp_avg, denom) 100 | p.data.copy_(p_data_fp32) 101 | elif step_size > 0: 102 | if group['weight_decay'] != 0: 103 | p_data_fp32.add_(-group['weight_decay'] * group['lr'], p_data_fp32) 104 | p_data_fp32.add_(-step_size * group['lr'], exp_avg) 105 | p.data.copy_(p_data_fp32) 106 | 107 | return loss 108 | 109 | 110 | class PlainRAdam(Optimizer): 111 | 112 | def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, degenerated_to_sgd=True): 113 | if not 0.0 <= lr: 114 | raise ValueError("Invalid learning rate: {}".format(lr)) 115 | if not 0.0 <= eps: 116 | raise ValueError("Invalid epsilon value: {}".format(eps)) 117 | if not 0.0 <= betas[0] < 1.0: 118 | raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0])) 119 | if not 0.0 <= betas[1] < 1.0: 120 | raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1])) 121 | 122 | self.degenerated_to_sgd = degenerated_to_sgd 123 | defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay) 124 | 125 | super(PlainRAdam, self).__init__(params, defaults) 126 | 127 | def __setstate__(self, state): 128 | super(PlainRAdam, self).__setstate__(state) 129 | 130 | def step(self, closure=None): 131 | 132 | loss = None 133 | if closure is not None: 134 | loss = closure() 135 | 136 | for group in self.param_groups: 137 | 138 | for p in group['params']: 139 | if p.grad is None: 140 | continue 141 | grad = p.grad.data.float() 142 | if grad.is_sparse: 143 | raise RuntimeError('RAdam does not support sparse gradients') 144 | 145 | p_data_fp32 = p.data.float() 146 | 147 | state = self.state[p] 148 | 149 | if len(state) == 0: 150 | state['step'] = 0 151 | state['exp_avg'] = torch.zeros_like(p_data_fp32) 152 | state['exp_avg_sq'] = torch.zeros_like(p_data_fp32) 153 | else: 154 | state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32) 155 | state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_data_fp32) 156 | 157 | exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq'] 158 | beta1, beta2 = group['betas'] 159 | 160 | exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) 161 | exp_avg.mul_(beta1).add_(1 - beta1, grad) 162 | 163 | state['step'] += 1 164 | beta2_t = beta2 ** state['step'] 165 | N_sma_max = 2 / (1 - beta2) - 1 166 | N_sma = N_sma_max - 2 * state['step'] * beta2_t / (1 - beta2_t) 167 | 168 | # more conservative since it's an approximated value 169 | if N_sma >= 5: 170 | if group['weight_decay'] != 0: 171 | p_data_fp32.add_(-group['weight_decay'] * group['lr'], p_data_fp32) 172 | step_size = group['lr'] * math.sqrt( 173 | (1 - beta2_t) * (N_sma - 4) / (N_sma_max - 4) * (N_sma - 2) / N_sma * N_sma_max / ( 174 | N_sma_max - 2)) / (1 - beta1 ** state['step']) 175 | denom = exp_avg_sq.sqrt().add_(group['eps']) 176 | p_data_fp32.addcdiv_(-step_size, exp_avg, denom) 177 | p.data.copy_(p_data_fp32) 178 | elif self.degenerated_to_sgd: 179 | if group['weight_decay'] != 0: 180 | p_data_fp32.add_(-group['weight_decay'] * group['lr'], p_data_fp32) 181 | step_size = group['lr'] / (1 - beta1 ** state['step']) 182 | p_data_fp32.add_(-step_size, exp_avg) 183 | p.data.copy_(p_data_fp32) 184 | 185 | return loss 186 | 187 | 188 | class AdamW(Optimizer): 189 | 190 | def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, warmup=0): 191 | if not 0.0 <= lr: 192 | raise ValueError("Invalid learning rate: {}".format(lr)) 193 | if not 0.0 <= eps: 194 | raise ValueError("Invalid epsilon value: {}".format(eps)) 195 | if not 0.0 <= betas[0] < 1.0: 196 | raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0])) 197 | if not 0.0 <= betas[1] < 1.0: 198 | raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1])) 199 | 200 | defaults = dict(lr=lr, betas=betas, eps=eps, 201 | weight_decay=weight_decay, warmup=warmup) 202 | super(AdamW, self).__init__(params, defaults) 203 | 204 | def __setstate__(self, state): 205 | super(AdamW, self).__setstate__(state) 206 | 207 | def step(self, closure=None): 208 | loss = None 209 | if closure is not None: 210 | loss = closure() 211 | 212 | for group in self.param_groups: 213 | 214 | for p in group['params']: 215 | if p.grad is None: 216 | continue 217 | grad = p.grad.data.float() 218 | if grad.is_sparse: 219 | raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead') 220 | 221 | p_data_fp32 = p.data.float() 222 | 223 | state = self.state[p] 224 | 225 | if len(state) == 0: 226 | state['step'] = 0 227 | state['exp_avg'] = torch.zeros_like(p_data_fp32) 228 | state['exp_avg_sq'] = torch.zeros_like(p_data_fp32) 229 | else: 230 | state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32) 231 | state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_data_fp32) 232 | 233 | exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq'] 234 | beta1, beta2 = group['betas'] 235 | 236 | state['step'] += 1 237 | 238 | exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) 239 | exp_avg.mul_(beta1).add_(1 - beta1, grad) 240 | 241 | denom = exp_avg_sq.sqrt().add_(group['eps']) 242 | bias_correction1 = 1 - beta1 ** state['step'] 243 | bias_correction2 = 1 - beta2 ** state['step'] 244 | 245 | if group['warmup'] > state['step']: 246 | scheduled_lr = 1e-8 + state['step'] * group['lr'] / group['warmup'] 247 | else: 248 | scheduled_lr = group['lr'] 249 | 250 | step_size = scheduled_lr * math.sqrt(bias_correction2) / bias_correction1 251 | 252 | if group['weight_decay'] != 0: 253 | p_data_fp32.add_(-group['weight_decay'] * scheduled_lr, p_data_fp32) 254 | 255 | p_data_fp32.addcdiv_(-step_size, exp_avg, denom) 256 | 257 | p.data.copy_(p_data_fp32) 258 | 259 | return loss 260 | -------------------------------------------------------------------------------- /models/TST/utils.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import sys 4 | import builtins 5 | import functools 6 | import time 7 | import ipdb 8 | from copy import deepcopy 9 | 10 | import numpy as np 11 | import torch 12 | import xlrd 13 | import xlwt 14 | from xlutils.copy import copy 15 | 16 | import logging 17 | logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s', level=logging.INFO) 18 | logger = logging.getLogger(__name__) 19 | 20 | 21 | def timer(func): 22 | """Print the runtime of the decorated function""" 23 | @functools.wraps(func) 24 | def wrapper_timer(*args, **kwargs): 25 | start_time = time.perf_counter() # 1 26 | value = func(*args, **kwargs) 27 | end_time = time.perf_counter() # 2 28 | run_time = end_time - start_time # 3 29 | print(f"Finished {func.__name__!r} in {run_time} secs") 30 | return value 31 | return wrapper_timer 32 | 33 | 34 | def save_model(path, epoch, model, optimizer=None): 35 | if isinstance(model, torch.nn.DataParallel): 36 | state_dict = model.module.state_dict() 37 | else: 38 | state_dict = model.state_dict() 39 | data = {'epoch': epoch, 40 | 'state_dict': state_dict} 41 | if not (optimizer is None): 42 | data['optimizer'] = optimizer.state_dict() 43 | torch.save(data, path) 44 | 45 | 46 | def load_model(model, model_path, optimizer=None, resume=False, change_output=False, 47 | lr=None, lr_step=None, lr_factor=None): 48 | start_epoch = 0 49 | checkpoint = torch.load(model_path, map_location=lambda storage, loc: storage) 50 | state_dict = deepcopy(checkpoint['state_dict']) 51 | if change_output: 52 | for key, val in checkpoint['state_dict'].items(): 53 | if key.startswith('output_layer'): 54 | state_dict.pop(key) 55 | model.load_state_dict(state_dict, strict=False) 56 | print('Loaded model from {}. Epoch: {}'.format(model_path, checkpoint['epoch'])) 57 | 58 | # resume optimizer parameters 59 | if optimizer is not None and resume: 60 | if 'optimizer' in checkpoint: 61 | optimizer.load_state_dict(checkpoint['optimizer']) 62 | start_epoch = checkpoint['epoch'] 63 | start_lr = lr 64 | for i in range(len(lr_step)): 65 | if start_epoch >= lr_step[i]: 66 | start_lr *= lr_factor[i] 67 | for param_group in optimizer.param_groups: 68 | param_group['lr'] = start_lr 69 | print('Resumed optimizer with start lr', start_lr) 70 | else: 71 | print('No optimizer parameters in checkpoint.') 72 | if optimizer is not None: 73 | return model, optimizer, start_epoch 74 | else: 75 | return model 76 | 77 | 78 | def load_config(config_filepath): 79 | """ 80 | Using a json file with the master configuration (config file for each part of the pipeline), 81 | return a dictionary containing the entire configuration settings in a hierarchical fashion. 82 | """ 83 | 84 | with open(config_filepath) as cnfg: 85 | config = json.load(cnfg) 86 | 87 | return config 88 | 89 | 90 | def create_dirs(dirs): 91 | """ 92 | Input: 93 | dirs: a list of directories to create, in case these directories are not found 94 | Returns: 95 | exit_code: 0 if success, -1 if failure 96 | """ 97 | try: 98 | for dir_ in dirs: 99 | if not os.path.exists(dir_): 100 | os.makedirs(dir_) 101 | return 0 102 | except Exception as err: 103 | print("Creating directories error: {0}".format(err)) 104 | exit(-1) 105 | 106 | 107 | def export_performance_metrics(filepath, metrics_table, header, book=None, sheet_name="metrics"): 108 | """Exports performance metrics on the validation set for all epochs to an excel file""" 109 | 110 | if book is None: 111 | book = xlwt.Workbook() # new excel work book 112 | 113 | book = write_table_to_sheet([header] + metrics_table, book, sheet_name=sheet_name) 114 | 115 | book.save(filepath) 116 | logger.info("Exported per epoch performance metrics in '{}'".format(filepath)) 117 | 118 | return book 119 | 120 | 121 | def write_row(sheet, row_ind, data_list): 122 | """Write a list to row_ind row of an excel sheet""" 123 | 124 | row = sheet.row(row_ind) 125 | for col_ind, col_value in enumerate(data_list): 126 | row.write(col_ind, col_value) 127 | return 128 | 129 | 130 | def write_table_to_sheet(table, work_book, sheet_name=None): 131 | """Writes a table implemented as a list of lists to an excel sheet in the given work book object""" 132 | 133 | sheet = work_book.add_sheet(sheet_name) 134 | 135 | for row_ind, row_list in enumerate(table): 136 | write_row(sheet, row_ind, row_list) 137 | 138 | return work_book 139 | 140 | 141 | def export_record(filepath, values): 142 | """Adds a list of values as a bottom row of a table in a given excel file""" 143 | 144 | read_book = xlrd.open_workbook(filepath, formatting_info=True) 145 | read_sheet = read_book.sheet_by_index(0) 146 | last_row = read_sheet.nrows 147 | 148 | work_book = copy(read_book) 149 | sheet = work_book.get_sheet(0) 150 | write_row(sheet, last_row, values) 151 | work_book.save(filepath) 152 | 153 | 154 | def register_record(filepath, timestamp, experiment_name, best_metrics, final_metrics=None, comment=''): 155 | """ 156 | Adds the best and final metrics of a given experiment as a record in an excel sheet with other experiment records. 157 | Creates excel sheet if it doesn't exist. 158 | Args: 159 | filepath: path of excel file keeping records 160 | timestamp: string 161 | experiment_name: string 162 | best_metrics: dict of metrics at best epoch {metric_name: metric_value}. Includes "epoch" as first key 163 | final_metrics: dict of metrics at final epoch {metric_name: metric_value}. Includes "epoch" as first key 164 | comment: optional description 165 | """ 166 | metrics_names, metrics_values = zip(*best_metrics.items()) 167 | row_values = [timestamp, experiment_name, comment] + list(metrics_values) 168 | if final_metrics is not None: 169 | final_metrics_names, final_metrics_values = zip(*final_metrics.items()) 170 | row_values += list(final_metrics_values) 171 | 172 | if not os.path.exists(filepath): # Create a records file for the first time 173 | logger.warning("Records file '{}' does not exist! Creating new file ...".format(filepath)) 174 | directory = os.path.dirname(filepath) 175 | if len(directory) and not os.path.exists(directory): 176 | os.makedirs(directory) 177 | header = ["Timestamp", "Name", "Comment"] + ["Best " + m for m in metrics_names] 178 | if final_metrics is not None: 179 | header += ["Final " + m for m in final_metrics_names] 180 | book = xlwt.Workbook() # excel work book 181 | book = write_table_to_sheet([header, row_values], book, sheet_name="records") 182 | book.save(filepath) 183 | else: 184 | try: 185 | export_record(filepath, row_values) 186 | except Exception as x: 187 | alt_path = os.path.join(os.path.dirname(filepath), "record_" + experiment_name) 188 | logger.error("Failed saving in: '{}'! Will save here instead: {}".format(filepath, alt_path)) 189 | export_record(alt_path, row_values) 190 | filepath = alt_path 191 | 192 | logger.info("Exported performance record to '{}'".format(filepath)) 193 | 194 | 195 | class Printer(object): 196 | """Class for printing output by refreshing the same line in the console, e.g. for indicating progress of a process""" 197 | 198 | def __init__(self, console=True): 199 | 200 | if console: 201 | self.print = self.dyn_print 202 | else: 203 | self.print = builtins.print 204 | 205 | @staticmethod 206 | def dyn_print(data): 207 | """Print things to stdout on one line, refreshing it dynamically""" 208 | sys.stdout.write("\r\x1b[K" + data.__str__()) 209 | sys.stdout.flush() 210 | 211 | 212 | def readable_time(time_difference): 213 | """Convert a float measuring time difference in seconds into a tuple of (hours, minutes, seconds)""" 214 | 215 | hours = time_difference // 3600 216 | minutes = (time_difference // 60) % 60 217 | seconds = time_difference % 60 218 | 219 | return hours, minutes, seconds 220 | 221 | 222 | def check_model(model, verbose=False, zero_thresh=1e-8, inf_thresh=1e6, stop_on_error=False): 223 | status_ok = True 224 | for name, param in model.named_parameters(): 225 | param_ok = check_tensor(param, verbose=verbose, zero_thresh=zero_thresh, inf_thresh=inf_thresh) 226 | if not param_ok: 227 | status_ok = False 228 | print("Parameter '{}' PROBLEM".format(name)) 229 | grad_ok = True 230 | if param.grad is not None: 231 | grad_ok = check_tensor(param.grad, verbose=verbose, zero_thresh=zero_thresh, inf_thresh=inf_thresh) 232 | if not grad_ok: 233 | status_ok = False 234 | print("Gradient of parameter '{}' PROBLEM".format(name)) 235 | if stop_on_error and not (param_ok and grad_ok): 236 | ipdb.set_trace() 237 | 238 | if status_ok: 239 | print("Model Check: OK") 240 | else: 241 | print("Model Check: PROBLEM") 242 | 243 | 244 | def check_tensor(X, verbose=True, zero_thresh=1e-8, inf_thresh=1e6): 245 | 246 | is_nan = torch.isnan(X) 247 | if is_nan.any(): 248 | print("{}/{} nan".format(torch.sum(is_nan), X.numel())) 249 | return False 250 | 251 | num_small = torch.sum(torch.abs(X) < zero_thresh) 252 | num_large = torch.sum(torch.abs(X) > inf_thresh) 253 | 254 | if verbose: 255 | print("Shape: {}, {} elements".format(X.shape, X.numel())) 256 | print("No 'nan' values") 257 | print("Min: {}".format(torch.min(X))) 258 | print("Median: {}".format(torch.median(X))) 259 | print("Max: {}".format(torch.max(X))) 260 | 261 | print("Histogram of values:") 262 | values = X.view(-1).detach().numpy() 263 | hist, binedges = np.histogram(values, bins=20) 264 | for b in range(len(binedges) - 1): 265 | print("[{}, {}): {}".format(binedges[b], binedges[b + 1], hist[b])) 266 | 267 | print("{}/{} abs. values < {}".format(num_small, X.numel(), zero_thresh)) 268 | print("{}/{} abs. values > {}".format(num_large, X.numel(), inf_thresh)) 269 | 270 | if num_large: 271 | print("{}/{} abs. values > {}".format(num_large, X.numel(), inf_thresh)) 272 | return False 273 | 274 | return True 275 | 276 | 277 | def count_parameters(model, trainable=False): 278 | if trainable: 279 | return sum(p.numel() for p in model.parameters() if p.requires_grad) 280 | else: 281 | return sum(p.numel() for p in model.parameters()) 282 | 283 | 284 | def recursively_hook(model, hook_fn): 285 | for name, module in model.named_children(): #model._modules.items(): 286 | if len(list(module.children())) > 0: # if not leaf node 287 | for submodule in module.children(): 288 | recursively_hook(submodule, hook_fn) 289 | else: 290 | module.register_forward_hook(hook_fn) 291 | 292 | 293 | def compute_loss(net: torch.nn.Module, 294 | dataloader: torch.utils.data.DataLoader, 295 | loss_function: torch.nn.Module, 296 | device: torch.device = 'cpu') -> torch.Tensor: 297 | """Compute the loss of a network on a given dataset. 298 | 299 | Does not compute gradient. 300 | 301 | Parameters 302 | ---------- 303 | net: 304 | Network to evaluate. 305 | dataloader: 306 | Iterator on the dataset. 307 | loss_function: 308 | Loss function to compute. 309 | device: 310 | Torch device, or :py:class:`str`. 311 | 312 | Returns 313 | ------- 314 | Loss as a tensor with no grad. 315 | """ 316 | running_loss = 0 317 | with torch.no_grad(): 318 | for x, y in dataloader: 319 | netout = net(x.to(device)).cpu() 320 | running_loss += loss_function(y, netout) 321 | 322 | return running_loss / len(dataloader) 323 | -------------------------------------------------------------------------------- /models/TSTCC/TC.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import numpy as np 4 | from .attention import Seq_Transformer 5 | 6 | 7 | 8 | class TC(nn.Module): 9 | def __init__(self, configs, device): 10 | super(TC, self).__init__() 11 | self.num_channels = configs.final_out_channels 12 | self.timestep = configs.TC.timesteps 13 | self.Wk = nn.ModuleList([nn.Linear(configs.TC.hidden_dim, self.num_channels) for i in range(self.timestep)]) 14 | self.lsoftmax = nn.LogSoftmax() 15 | self.device = device 16 | 17 | self.projection_head = nn.Sequential( 18 | nn.Linear(configs.TC.hidden_dim, configs.final_out_channels // 2), 19 | nn.BatchNorm1d(configs.final_out_channels // 2), 20 | nn.ReLU(inplace=True), 21 | nn.Linear(configs.final_out_channels // 2, configs.final_out_channels // 4), 22 | ) 23 | # mqw-begin 24 | self.seq_transformer = Seq_Transformer(seq_size=self.timestep, patch_size=self.num_channels, dim=configs.TC.hidden_dim, depth=4, heads=4, mlp_dim=64, emb_dropout=0.) 25 | # mqw-end 26 | def forward(self, features_aug1, features_aug2): 27 | z_aug1 = features_aug1 # features are (batch_size, #channels, seq_len) 28 | seq_len = z_aug1.shape[2] 29 | z_aug1 = z_aug1.transpose(1, 2) 30 | 31 | z_aug2 = features_aug2 32 | z_aug2 = z_aug2.transpose(1, 2) 33 | 34 | batch = z_aug1.shape[0] 35 | t_samples = torch.randint(seq_len - self.timestep, size=(1,)).long().to(self.device) # randomly pick time stamps 36 | 37 | nce = 0 # average over timestep and batch 38 | encode_samples = torch.empty((self.timestep, batch, self.num_channels)).float().to(self.device) 39 | 40 | for i in np.arange(1, self.timestep + 1): 41 | encode_samples[i - 1] = z_aug2[:, t_samples + i, :].view(batch, self.num_channels) 42 | forward_seq = z_aug1[:, :t_samples + 1, :] 43 | 44 | c_t = self.seq_transformer(forward_seq) 45 | 46 | pred = torch.empty((self.timestep, batch, self.num_channels)).float().to(self.device) 47 | for i in np.arange(0, self.timestep): 48 | linear = self.Wk[i] 49 | pred[i] = linear(c_t) 50 | for i in np.arange(0, self.timestep): 51 | total = torch.mm(encode_samples[i], torch.transpose(pred[i], 0, 1)) 52 | nce += torch.sum(torch.diag(self.lsoftmax(total))) 53 | nce /= -1. * batch * self.timestep 54 | return nce, self.projection_head(c_t) -------------------------------------------------------------------------------- /models/TSTCC/loss.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import numpy as np 3 | 4 | class NTXentLoss(torch.nn.Module): 5 | 6 | def __init__(self, device, batch_size, temperature, use_cosine_similarity): 7 | super(NTXentLoss, self).__init__() 8 | self.batch_size = batch_size 9 | self.temperature = temperature 10 | self.device = device 11 | self.softmax = torch.nn.Softmax(dim=-1) 12 | self.mask_samples_from_same_repr = self._get_correlated_mask().type(torch.bool) 13 | self.similarity_function = self._get_similarity_function(use_cosine_similarity) 14 | self.criterion = torch.nn.CrossEntropyLoss(reduction="sum") 15 | 16 | def _get_similarity_function(self, use_cosine_similarity): 17 | if use_cosine_similarity: 18 | self._cosine_similarity = torch.nn.CosineSimilarity(dim=-1) 19 | return self._cosine_simililarity 20 | else: 21 | return self._dot_simililarity 22 | 23 | def _get_correlated_mask(self): 24 | diag = np.eye(2 * self.batch_size) 25 | l1 = np.eye((2 * self.batch_size), 2 * self.batch_size, k=-self.batch_size) 26 | l2 = np.eye((2 * self.batch_size), 2 * self.batch_size, k=self.batch_size) 27 | mask = torch.from_numpy((diag + l1 + l2)) 28 | mask = (1 - mask).type(torch.bool) 29 | return mask.to(self.device) 30 | 31 | @staticmethod 32 | def _dot_simililarity(x, y): 33 | v = torch.tensordot(x.unsqueeze(1), y.T.unsqueeze(0), dims=2) 34 | # x shape: (N, 1, C) 35 | # y shape: (1, C, 2N) 36 | # v shape: (N, 2N) 37 | return v 38 | 39 | def _cosine_simililarity(self, x, y): 40 | # x shape: (N, 1, C) 41 | # y shape: (1, 2N, C) 42 | # v shape: (N, 2N) 43 | v = self._cosine_similarity(x.unsqueeze(1), y.unsqueeze(0)) 44 | return v 45 | 46 | def forward(self, zis, zjs): 47 | representations = torch.cat([zjs, zis], dim=0) 48 | 49 | similarity_matrix = self.similarity_function(representations, representations) 50 | 51 | # filter out the scores from the positive samples 52 | l_pos = torch.diag(similarity_matrix, self.batch_size) 53 | r_pos = torch.diag(similarity_matrix, -self.batch_size) 54 | positives = torch.cat([l_pos, r_pos]).view(2 * self.batch_size, 1) 55 | 56 | negatives = similarity_matrix[self.mask_samples_from_same_repr].view(2 * self.batch_size, -1) 57 | 58 | logits = torch.cat((positives, negatives), dim=1) 59 | logits /= self.temperature 60 | 61 | labels = torch.zeros(2 * self.batch_size).to(self.device).long() 62 | loss = self.criterion(logits, labels) 63 | 64 | return loss / (2 * self.batch_size) 65 | -------------------------------------------------------------------------------- /models/TSTCC/model.py: -------------------------------------------------------------------------------- 1 | from torch import nn 2 | from config_files.epilepsy_Configs import Config as Configs 3 | import torch 4 | class base_Model(nn.Module): 5 | def __init__(self, configs): 6 | super(base_Model, self).__init__() 7 | # Original-HAR 8 | self.conv_block1 = nn.Sequential( 9 | nn.Conv1d(configs.input_channels, 32, kernel_size=configs.kernel_size, 10 | stride=configs.stride, bias=False, padding=(configs.kernel_size // 2)), 11 | nn.BatchNorm1d(32), 12 | nn.ReLU(), 13 | nn.MaxPool1d(kernel_size=2, stride=2, padding=1), 14 | nn.Dropout(configs.dropout) 15 | ) 16 | 17 | self.conv_block2 = nn.Sequential( 18 | nn.Conv1d(32, 64, kernel_size=8, stride=1, bias=False, padding=4), 19 | nn.BatchNorm1d(64), 20 | nn.ReLU(), 21 | nn.MaxPool1d(kernel_size=2, stride=2, padding=1) 22 | ) 23 | 24 | self.conv_block3 = nn.Sequential( 25 | nn.Conv1d(64, configs.final_out_channels, kernel_size=8, stride=1, bias=False, padding=4), 26 | nn.BatchNorm1d(configs.final_out_channels), 27 | nn.ReLU(), 28 | nn.MaxPool1d(kernel_size=2, stride=2, padding=1), 29 | ) 30 | model_output_dim = configs.features_len 31 | self.logits = nn.Linear(model_output_dim * configs.final_out_channels, configs.num_classes) 32 | 33 | def forward(self, x_in): 34 | print(f'x_in.shape:{x_in.shape}') # har: [128, 9, 128] & shar: [128,3,151] $ wisdm: [128,3,256] 35 | x = self.conv_block1(x_in) 36 | print(x.shape) #[128, 32, 65] & [128, 32, 77] & [128, 32, 129] 37 | x = self.conv_block2(x) 38 | print(x.shape) #[128, 64, 34] & [128, 64, 40]) & [128, 64, 66] 39 | x = self.conv_block3(x) 40 | print(x.shape) #[128, 128, 18] & [128, 128, 21] & [128, 128, 34] 41 | 42 | x_flat = x.reshape(x.shape[0], -1) 43 | print(f'x_flat.shape:{x_flat.shape}') 44 | logits = self.logits(x_flat) 45 | return logits, x 46 | # logits [128-batch_size, 2-num_classes] 47 | # x [128-batch_size, 128-final_out_channels, 24-features_len] 48 | 49 | if __name__ == '__main__': 50 | finalconfigs = Configs() 51 | m = base_Model(finalconfigs) 52 | x = torch.randn(32,1,178) 53 | y = m(x)[1] 54 | print(y.shape) #[128, 128, 24] -------------------------------------------------------------------------------- /models/TSTCC/train_TSTCC.py: -------------------------------------------------------------------------------- 1 | import torch 2 | 3 | import os 4 | import numpy as np 5 | from datetime import datetime 6 | import argparse 7 | 8 | from sklearn.manifold import TSNE 9 | import matplotlib.pyplot as plt 10 | 11 | from config_files.HAR_Configs import Config 12 | from utils import _logger, set_requires_grad 13 | from ..data_loader.dataset_loader import data_generator 14 | from trainer import Trainer, model_evaluate 15 | from TC import TC 16 | from utils import _calc_metrics, copy_Files 17 | from model import base_Model 18 | 19 | class train_TSTCC(): 20 | 21 | def __init__(self, parser): 22 | self.args = parser.parse_args() 23 | # Args selections 24 | self.start_time = datetime.now() 25 | self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 26 | self.experiment_description = self.args.experiment_description 27 | self.dataset_name = self.args.dataset_name 28 | self.method = 'TS-TCC' 29 | self.training_mode = self.args.training_mode 30 | self.run_description = self.args.run_description 31 | self.logs_save_dir = self.args.logs_save_dir 32 | # ##### random seeds for reproducibility ######## 33 | self.SEED = self.args.seed 34 | 35 | def plot_embedding(self,data, label, title): 36 | x_min, x_max = np.min(data, 0), np.max(data, 0) 37 | data = (data - x_min) / (x_max - x_min) 38 | fig = plt.figure(figsize=(15, 10)) 39 | ax = plt.subplot(111) 40 | ### 2D 41 | if self.dataset_name == 'SHAR' or 'CharacterTrajectories': 42 | for i in range(data.shape[0]): 43 | plt.text(data[i, 0], data[i, 1], int(label[i]), color=plt.cm.tab20(label[i] / 20), 44 | fontdict={'weight': 'bold', 'size': 7}) 45 | plt.xticks() 46 | plt.yticks() 47 | plt.title(title, fontsize=14) 48 | elif self.dataset_name == 'PhonemeSpectra': 49 | for i in range(data.shape[0]): 50 | plt.text(data[i, 0], data[i, 1], int(label[i]), color=plt.cm.tab20(label[i] / 20), 51 | fontdict={'weight': 'bold', 'size': 7}) 52 | 53 | plt.xticks() 54 | plt.yticks() 55 | plt.title(title, fontsize=14) 56 | else: 57 | for i in range(data.shape[0]): 58 | plt.text(data[i, 0], data[i, 1], int(label[i]), color=plt.cm.tab10(label[i] / 10), 59 | fontdict={'weight': 'bold', 'size': 7}) 60 | plt.xticks() 61 | plt.yticks() 62 | plt.title(title, fontsize=14) 63 | return fig 64 | 65 | 66 | def excute(self): 67 | os.makedirs(self.logs_save_dir, exist_ok=True) 68 | 69 | exec(f'from config_files.{self.dataset_name}_Configs import Config') 70 | configs = Config() 71 | 72 | print(self.SEED) 73 | torch.manual_seed(self.SEED) 74 | torch.backends.cudnn.deterministic = False 75 | torch.backends.cudnn.benchmark = False 76 | np.random.seed(self.SEED) 77 | 78 | experiment_log_dir = os.path.join(self.logs_save_dir, self.experiment_description, self.run_description, 79 | self.training_mode + f"_seed_{self.SEED}") 80 | os.makedirs(experiment_log_dir, exist_ok=True) 81 | 82 | # loop through domains 83 | counter = 0 84 | src_counter = 0 85 | 86 | # Logging 87 | log_file_name = os.path.join(experiment_log_dir, f"logs_{datetime.now().strftime('%d_%m_%Y_%H_%M_%S')}.log") 88 | logger = _logger(log_file_name) 89 | logger.debug("=" * 45) 90 | logger.debug(f'Dataset: {self.dataset_name}') 91 | logger.debug(f'Method: {self.method}') 92 | logger.debug(f'Mode: {self.training_mode}') 93 | logger.debug("=" * 45) 94 | 95 | # Load datasets 96 | data_path = f"./data/{self.dataset_name}" 97 | # train_dl, valid_dl, test_dl = data_generator(data_path, configs, training_mode) 98 | train_dl, test_dl = data_generator(data_path, configs, self.training_mode) 99 | # print(train_dl) 100 | logger.debug("Data loaded ...") 101 | 102 | # Load Model 103 | model = base_Model(configs).to(self.device) 104 | temporal_contr_model = TC(configs, self.device).to(self.device) 105 | 106 | if self.training_mode == "fine_tune": 107 | # load saved model of this experiment 108 | load_from = os.path.join( 109 | os.path.join(self.logs_save_dir, self.experiment_description, self.run_description, f"self_supervised_seed_{self.SEED}", 110 | "saved_models")) 111 | chkpoint = torch.load(os.path.join(load_from, "ckp_last.pt"), map_location=self.device) 112 | pretrained_dict = chkpoint["model_state_dict"] 113 | model_dict = model.state_dict() 114 | del_list = ['logits'] 115 | pretrained_dict_copy = pretrained_dict.copy() 116 | for i in pretrained_dict_copy.keys(): 117 | for j in del_list: 118 | if j in i: 119 | del pretrained_dict[i] 120 | model_dict.update(pretrained_dict) 121 | model.load_state_dict(model_dict) 122 | 123 | if self.training_mode == "train_linear" or "tl" in self.training_mode: 124 | load_from = os.path.join( 125 | os.path.join(self.logs_save_dir, self.experiment_description, self.run_description, f"self_supervised_seed_{self.SEED}", 126 | "saved_models")) 127 | chkpoint = torch.load(os.path.join(load_from, "ckp_last.pt"), map_location=self.device) 128 | pretrained_dict = chkpoint["model_state_dict"] 129 | model_dict = model.state_dict() 130 | 131 | # 1. filter out unnecessary keys 132 | pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict} 133 | 134 | # delete these parameters (Ex: the linear layer at the end) 135 | del_list = ['logits'] 136 | pretrained_dict_copy = pretrained_dict.copy() 137 | for i in pretrained_dict_copy.keys(): 138 | for j in del_list: 139 | if j in i: 140 | del pretrained_dict[i] 141 | model_dict.update(pretrained_dict) 142 | model.load_state_dict(model_dict) 143 | set_requires_grad(model, pretrained_dict, requires_grad=False) # Freeze everything except last layer. 144 | 145 | if self.training_mode == "random_init": 146 | model_dict = model.state_dict() 147 | 148 | # delete all the parameters except for logits 149 | del_list = ['logits'] 150 | pretrained_dict_copy = model_dict.copy() 151 | for i in pretrained_dict_copy.keys(): 152 | for j in del_list: 153 | if j in i: 154 | del model_dict[i] 155 | set_requires_grad(model, model_dict, requires_grad=False) # Freeze everything except last layer. 156 | 157 | model_optimizer = torch.optim.Adam(model.parameters(), lr=configs.lr, betas=(configs.beta1, configs.beta2), 158 | weight_decay=3e-4) 159 | temporal_contr_optimizer = torch.optim.Adam(temporal_contr_model.parameters(), lr=configs.lr, 160 | betas=(configs.beta1, configs.beta2), weight_decay=3e-4) 161 | 162 | if self.training_mode == "self_supervised": # to do it only once 163 | copy_Files(os.path.join(self.logs_save_dir, self.experiment_description, self.run_description), self.dataset_name) 164 | 165 | # Trainer 166 | # Trainer(model, temporal_contr_model, model_optimizer, temporal_contr_optimizer, train_dl, valid_dl, test_dl, device, logger, configs, experiment_log_dir, training_mode) 167 | Trainer(model, temporal_contr_model, model_optimizer, temporal_contr_optimizer, train_dl, test_dl, self.device, logger, 168 | configs, experiment_log_dir, self.training_mode) 169 | 170 | if self.training_mode != "self_supervised": 171 | # Testing 172 | outs = model_evaluate(model, temporal_contr_model, test_dl, self.device, self.training_mode) 173 | total_loss, total_acc, pred_labels, true_labels, representations = outs # mqw 174 | _calc_metrics(pred_labels, true_labels, experiment_log_dir, self.args.home_path, self.dataset_name) 175 | 176 | ### TSNE Embeddings of representations 177 | print('Starting to compute t-SNE Embeddings...') 178 | ts = TSNE(n_components=2, init='pca', 179 | random_state=0) 180 | print(f'x.shape{representations.shape}') 181 | result = ts.fit_transform(representations) 182 | fig = self.plot_embedding(result, true_labels, 183 | f't-SNE Embeddings of Time Series Representations---Dataset: {self.dataset_name}') # 显示图像 184 | if not os.path.exists(f'./eb/eb_{self.dataset_name}'): 185 | os.makedirs(f'./eb/eb_{self.dataset_name}') 186 | plt.savefig(f'./eb/eb_{self.dataset_name}/{self.dataset_name}_pca_2d.png', format='png', 187 | bbox_inches='tight') # change-16 命名 init='pca'//'random' 以及2d还是3d 188 | plt.show() 189 | 190 | logger.debug(f"Training time is : {datetime.now() - self.start_time}") 191 | 192 | -------------------------------------------------------------------------------- /models/TSTCC/utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import random 3 | import numpy as np 4 | import pandas as pd 5 | import os 6 | import sys 7 | import logging 8 | from sklearn.metrics import classification_report, cohen_kappa_score, confusion_matrix, accuracy_score 9 | from shutil import copy 10 | import matplotlib.pyplot as plt 11 | 12 | def plot_confusion_matrix(cm, labels_name, title): 13 | plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues) 14 | plt.title(title) 15 | plt.colorbar() 16 | num_local = np.array(range(len(labels_name))) 17 | plt.xticks(num_local, labels_name, rotation=90) 18 | plt.yticks(num_local, labels_name) 19 | plt.ylabel('True label') 20 | plt.xlabel('Predicted label') 21 | for first_index in range(len(cm)): 22 | for second_index in range(len(cm[first_index])): 23 | plt.text(first_index, second_index, cm[first_index][second_index]) 24 | 25 | def set_requires_grad(model, dict_, requires_grad=True): 26 | for param in model.named_parameters(): 27 | if param[0] in dict_: 28 | param[1].requires_grad = requires_grad 29 | 30 | 31 | def fix_randomness(SEED): 32 | random.seed(SEED) 33 | np.random.seed(SEED) 34 | torch.manual_seed(SEED) 35 | torch.cuda.manual_seed(SEED) 36 | torch.backends.cudnn.deterministic = True 37 | 38 | 39 | def epoch_time(start_time, end_time): 40 | elapsed_time = end_time - start_time 41 | elapsed_mins = int(elapsed_time / 60) 42 | elapsed_secs = int(elapsed_time - (elapsed_mins * 60)) 43 | return elapsed_mins, elapsed_secs 44 | 45 | 46 | def _calc_metrics(pred_labels, true_labels, log_dir, home_path, data_type): 47 | pred_labels = np.array(pred_labels).astype(int) 48 | true_labels = np.array(true_labels).astype(int) 49 | 50 | # save targets 51 | labels_save_path = os.path.join(log_dir, "labels") 52 | os.makedirs(labels_save_path, exist_ok=True) 53 | np.save(os.path.join(labels_save_path, "predicted_labels.npy"), pred_labels) 54 | np.save(os.path.join(labels_save_path, "true_labels.npy"), true_labels) 55 | 56 | r = classification_report(true_labels, pred_labels, digits=6, output_dict=True) 57 | cm = confusion_matrix(true_labels, pred_labels) 58 | df = pd.DataFrame(r) 59 | df["cohen"] = cohen_kappa_score(true_labels, pred_labels) 60 | df["accuracy"] = accuracy_score(true_labels, pred_labels) 61 | df = df * 100 62 | 63 | # save classification report 64 | exp_name = os.path.split(os.path.dirname(log_dir))[-1] 65 | training_mode = os.path.basename(log_dir) 66 | file_name = f"{exp_name}_{training_mode}_classification_report.xlsx" 67 | report_Save_path = os.path.join(home_path, log_dir, file_name) 68 | df.to_excel(report_Save_path) 69 | 70 | # save confusion matrix 71 | cm_file_name = f"{exp_name}_{training_mode}_confusion_matrix.torch" 72 | cm_Save_path = os.path.join(home_path, log_dir, cm_file_name) 73 | torch.save(cm, cm_Save_path) 74 | 75 | ### plot confusion matrix 76 | print('Starting to plot confusion matrix...') 77 | dataset_name = data_type 78 | if dataset_name == 'sleepEDF': 79 | labels_name = ['Wake (W)', 'Non-rapid eye movement (N1)', 'Non-rapid eye movement (N2)', 80 | 'Non-rapid eye movement (N3)', 'Rapid Eye Movement (REM)'] # sleepEDF 81 | elif dataset_name == 'HAR': 82 | labels_name = ['WALKING', 'WALKING_UPSTAIRS', 'WALKING_DOWNSTAIRS', 'SITTING', 'STANDING', 'LAYING'] # HAR 83 | elif dataset_name == 'epilepsy': 84 | labels_name = ['epilepsy', 'not epilepsy'] # epilepsy 85 | elif dataset_name == 'SHAR': 86 | labels_name = ['StandingUpFS', 'StandingUpFL', 'Walking', 'Running', 'GoingUpS', 'Jumping', 'GoingDownS', 'LyingDownFS', 'SittingDown', 'FallingForw', 'FallingRight', 'FallingBack', 'HittingObstacle', 'FallingWithPS', 'FallingBackSC', 'Syncope', 'FallingLeft'] # SHAR 87 | elif dataset_name == 'wisdm': 88 | labels_name = ['Walking', 'Jogging', 'Sitting', 'Standing', 'Upstairs', 'Downstairs'] # wisdm 89 | elif dataset_name == 'DuckDuckGeese': 90 | labels_name = ['Black-bellied Whistling Duck', 'Canadian Goose', 'Greylag Goose', 'Pink Footed Goose', 91 | 'White-faced Whistling Duck'] 92 | elif dataset_name == 'FingerMovements': 93 | labels_name = ['Left', 'Right'] 94 | elif dataset_name == 'PenDigits': 95 | labels_name = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'] 96 | elif dataset_name == 'PhonemeSpectra': 97 | labels_name = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', 98 | '18', '19', 99 | '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', 100 | '36', '37', '38'] 101 | elif dataset_name == 'StandWalkJump': 102 | labels_name = ['Standing', 'Walking', 'Jumping'] 103 | 104 | 105 | 106 | plot_confusion_matrix(cm, labels_name, f"{dataset_name}--- Confusion Matrix") 107 | plt.subplots_adjust(bottom=0.15) 108 | if not os.path.exists(f'./cm/cm_{dataset_name}'): 109 | os.makedirs(f'./cm/cm_{dataset_name}') 110 | plt.savefig(f'./cm/cm_{dataset_name}/{dataset_name}_cm.png', format='png', bbox_inches='tight') 111 | plt.show() 112 | 113 | 114 | 115 | def _logger(logger_name, level=logging.DEBUG): 116 | """ 117 | Method to return a custom logger with the given name and level 118 | """ 119 | logger = logging.getLogger(logger_name) 120 | logger.setLevel(level) 121 | # format_string = ("%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:" 122 | # "%(lineno)d — %(message)s") 123 | format_string = "%(message)s" 124 | log_format = logging.Formatter(format_string) 125 | # Creating and adding the console handler 126 | console_handler = logging.StreamHandler(sys.stdout) 127 | console_handler.setFormatter(log_format) 128 | logger.addHandler(console_handler) 129 | # Creating and adding the file handler 130 | file_handler = logging.FileHandler(logger_name, mode='a') 131 | file_handler.setFormatter(log_format) 132 | logger.addHandler(file_handler) 133 | return logger 134 | 135 | def copy_Files(destination, data_type): 136 | destination_dir = os.path.join(destination, "model_files") 137 | os.makedirs(destination_dir, exist_ok=True) 138 | copy("train_TSTCC.py", os.path.join(destination_dir, "train_TSTCC.py")) 139 | copy("trainer/trainer.py", os.path.join(destination_dir, "trainer.py")) 140 | copy(f"config_files/{data_type}_Configs.py", os.path.join(destination_dir, f"{data_type}_Configs.py")) 141 | copy("dataloader/augmentations.py", os.path.join(destination_dir, "augmentations.py")) 142 | copy("dataloader/dataloader.py", os.path.join(destination_dir, "dataloader.py")) 143 | copy(f"models/model.py", os.path.join(destination_dir, f"model.py")) 144 | copy("models/loss.py", os.path.join(destination_dir, "loss.py")) 145 | copy("models/TC.py", os.path.join(destination_dir, "TC.py")) 146 | -------------------------------------------------------------------------------- /models/TimeNet/framework.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | import torch 4 | import torch.nn as nn 5 | import torch.optim as optim 6 | from torch.utils.data import Dataset 7 | from sklearn.preprocessing import StandardScaler 8 | 9 | 10 | OUTPUT_DIR = "models" 11 | 12 | 13 | class SimpleSeriesDataset(Dataset): 14 | 15 | def __init__(self, series, maxlen=None, normalize=None): 16 | self.series = series 17 | self.normalize = normalize 18 | self.maxlen = maxlen 19 | self.scaler = StandardScaler() if normalize == 'zscore' else None 20 | 21 | def __len__(self): 22 | return len(self.series) 23 | 24 | def __getitem__(self, idx): 25 | series = self.series.iloc[idx] 26 | if self.maxlen is not None and len(series) > self.maxlen: 27 | series = series[:self.maxlen] 28 | if self.normalize: 29 | series = self.scaler.transform(series.values.reshape(-1, 1)) 30 | return torch.Tensor(series) 31 | 32 | 33 | class TimeNet(nn.Module): 34 | 35 | def __init__(self, size, num_layers, dropout=0.0): 36 | super(TimeNet, self).__init__() 37 | self.size = size 38 | self.num_layers = num_layers 39 | self.dropout = dropout 40 | self.encoder = self.build_encoder() 41 | self.decoder = self.build_decoder() 42 | 43 | def build_encoder(self): 44 | encoder_layers = [] 45 | encoder_layers.append(nn.GRU(1, self.size, num_layers=1, batch_first=True)) 46 | for i in range(2, self.num_layers + 1): 47 | encoder_layers.append(nn.GRU(self.size, self.size, num_layers=1, batch_first=True)) 48 | return nn.Sequential(*encoder_layers) 49 | 50 | def build_decoder(self): 51 | decoder_layers = [] 52 | decoder_layers.append(nn.GRU(self.size, self.size, num_layers=1, batch_first=True)) 53 | for i in range(2, self.num_layers + 1): 54 | decoder_layers.append(nn.GRU(self.size, self.size, num_layers=1, batch_first=True)) 55 | decoder_layers.append(nn.Linear(self.size, 1)) 56 | return nn.Sequential(*decoder_layers) 57 | 58 | def forward(self, x): 59 | # Encoder 60 | encode, _ = self.encoder(x) 61 | 62 | # Decoder 63 | decode, _ = self.decoder(torch.flip(encode, [1])) # Reverse the encoded sequence 64 | return decode 65 | 66 | 67 | def train(model, train_loader, optimizer, criterion, lr_scheduler=None, epochs=10, early_stop=5): 68 | run = model.get_run_id() 69 | log_dir = os.path.join(OUTPUT_DIR, run) 70 | weights_path = os.path.join(log_dir, 'weights.pt') 71 | 72 | loaded = False 73 | if os.path.exists(weights_path): 74 | print("Loading {}...".format(weights_path)) 75 | model.load_state_dict(torch.load(weights_path)) 76 | loaded = True 77 | 78 | if (not loaded): 79 | shutil.rmtree(log_dir, ignore_errors=True) 80 | os.makedirs(log_dir) 81 | 82 | weights_path = os.path.join(log_dir, 'weights.pt') 83 | 84 | if lr_scheduler is not None: 85 | optimizer = optim.lr_scheduler(optimizer, mode='min', patience=early_stop, verbose=True, threshold=0.0001, 86 | threshold_mode='rel') 87 | 88 | model.train() 89 | for epoch in range(epochs): 90 | for batch in train_loader: 91 | optimizer.zero_grad() 92 | outputs = model(batch.unsqueeze(2)) 93 | loss = criterion(outputs, batch.unsqueeze(2)) 94 | loss.backward() 95 | optimizer.step() 96 | 97 | # Save model weights 98 | torch.save(model.state_dict(), weights_path) 99 | 100 | return log_dir 101 | 102 | -------------------------------------------------------------------------------- /models/TimeNet/train_TimeNet.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.optim as optim 4 | from torch.optim.lr_scheduler import ReduceLROnPlateau 5 | from framework import TimeNet 6 | from ..data_loader.dataset_loader import data_generator 7 | 8 | class train_TimeNet(): 9 | 10 | def __init__(self, parser): 11 | self.args = parser.parse_args() 12 | self.log_dir = self.args.exp_dir 13 | self.dataset_name = self.args.dataset_name 14 | self.embeddings_dim = self.args.feature_size 15 | self.num_layers = self.args.layers 16 | self.batch_size = self.args.batch_size 17 | self.learning_rate = self.args.lr 18 | self.epochs = self.args.epochs 19 | 20 | def excute(self): 21 | if self.dataset_name == 'HAR': 22 | from config_files.HAR_Configs import Config as Configs 23 | configs = Configs() 24 | train_loader, eval_loader = data_generator('data/HAR', configs, 'self_supervised') 25 | elif self.dataset_name == 'wisdm': 26 | from config_files.wisdm_Configs import Config as Configs 27 | configs = Configs() 28 | train_loader, eval_loader = data_generator('data/wisdm', configs, 'self_supervised') 29 | elif self.dataset_name == 'epilepsy': 30 | from config_files.epilepsy_Configs import Config as Configs 31 | configs = Configs() 32 | train_loader, eval_loader = data_generator('data/epilepsy', configs, 'self_supervised') 33 | elif self.dataset_name == 'SHAR': 34 | from config_files.SHAR_Configs import Config as Configs 35 | configs = Configs() 36 | train_loader, eval_loader = data_generator('data/SHAR', configs, 'self_supervised') 37 | elif self.dataset_name == 'PenDigits': 38 | from config_files.PenDigits_Configs import Config as Configs 39 | configs = Configs() 40 | train_loader, eval_loader = data_generator('data/PenDigits', configs, 'self_supervised') 41 | elif self.dataset_name == 'EigenWorms': 42 | from config_files.EigenWorms_Configs import Config as Configs 43 | configs = Configs() 44 | train_loader, eval_loader = data_generator('data/EigenWorms', configs, 'self_supervised') 45 | elif self.dataset_name == 'FingerMovements': 46 | from config_files.FingerMovements_Configs import Config as Configs 47 | configs = Configs() 48 | train_loader, eval_loader = data_generator('data/FingerMovements', configs, 'self_supervised') 49 | 50 | enc = TimeNet(self.embeddings_dim, num_layers=self.num_layers, dropout=self.dropout) 51 | 52 | criterion = nn.MSELoss() 53 | optimizer = optim.Adam(enc.parameters(), lr=self.learning_rate) 54 | lr_scheduler = ReduceLROnPlateau(optimizer, patience=10) 55 | 56 | for epoch in range(self.epochs): 57 | enc.train() 58 | for data in train_loader: 59 | optimizer.zero_grad() 60 | outputs = enc(data.unsqueeze(2)) # Add a dimension for time steps 61 | loss = criterion(outputs, data.unsqueeze(2)) 62 | loss.backward() 63 | optimizer.step() 64 | 65 | torch.save(enc.state_dict(), 'timenet_model.pth') 66 | 67 | 68 | 69 | -------------------------------------------------------------------------------- /models/notes.txt: -------------------------------------------------------------------------------- 1 | More models will be uploaded soon 2 | -------------------------------------------------------------------------------- /organization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mqwfrog/ULTS/79f34003a9408e4db0b5a911ab0a27e1bcf6e6e8/organization.png -------------------------------------------------------------------------------- /taxonomy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mqwfrog/ULTS/79f34003a9408e4db0b5a911ab0a27e1bcf6e6e8/taxonomy.png --------------------------------------------------------------------------------