├── CNN_torch ├── text.txt ├── CNN_EEG.py ├── EEGNet_tor.py └── CNN_Vision.py ├── Pre_trained_models └── ast-finetuned-audioset │ ├── Download.txt │ ├── preprocessor_config.json │ ├── README.md │ └── config.json ├── images └── EAVlogo.png ├── requirements.txt ├── EAV_datasplit.py ├── Transformer_torch ├── Transformer_Audio.py ├── Transformer_Vision.py └── Transformer_EEG.py ├── Dataload_vision.py ├── Dataload_audio.py ├── CNN_tensorflow ├── CNN_video_emotion_recognition.ipynb ├── CNN_EEG_tf.py ├── CNN_audio_emotion_recognition.ipynb └── EEG_nb.ipynb ├── Dataload_eeg.py ├── README.md └── LICENSE /CNN_torch/text.txt: -------------------------------------------------------------------------------- 1 | This folder contains the CNNs with torch -------------------------------------------------------------------------------- /Pre_trained_models/ast-finetuned-audioset/Download.txt: -------------------------------------------------------------------------------- 1 | hello222 -------------------------------------------------------------------------------- /images/EAVlogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nubcico/EAV/HEAD/images/EAVlogo.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | facenet_pytorch 2 | keras 3 | librosa 4 | matplotlib 5 | numpy 6 | opencv-python 7 | pandas 8 | pillow 9 | pickle 10 | scipy 11 | seaborn 12 | tensorflow 13 | torch 14 | torchaudio 15 | torchvision 16 | transformers -------------------------------------------------------------------------------- /Pre_trained_models/ast-finetuned-audioset/preprocessor_config.json: -------------------------------------------------------------------------------- 1 | { 2 | "do_normalize": true, 3 | "feature_extractor_type": "ASTFeatureExtractor", 4 | "feature_size": 1, 5 | "max_length": 1024, 6 | "mean": -4.2677393, 7 | "num_mel_bins": 128, 8 | "padding_side": "right", 9 | "padding_value": 0.0, 10 | "return_attention_mask": false, 11 | "sampling_rate": 16000, 12 | "std": 4.5689974 13 | } 14 | -------------------------------------------------------------------------------- /Pre_trained_models/ast-finetuned-audioset/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | license: bsd-3-clause 3 | tags: 4 | - audio-classification 5 | --- 6 | 7 | # Audio Spectrogram Transformer (fine-tuned on AudioSet) 8 | 9 | Audio Spectrogram Transformer (AST) model fine-tuned on AudioSet. It was introduced in the paper [AST: Audio Spectrogram Transformer](https://arxiv.org/abs/2104.01778) by Gong et al. and first released in [this repository](https://github.com/YuanGongND/ast). 10 | 11 | Disclaimer: The team releasing Audio Spectrogram Transformer did not write a model card for this model so this model card has been written by the Hugging Face team. 12 | 13 | ## Model description 14 | 15 | The Audio Spectrogram Transformer is equivalent to [ViT](https://huggingface.co/docs/transformers/model_doc/vit), but applied on audio. Audio is first turned into an image (as a spectrogram), after which a Vision Transformer is applied. The model gets state-of-the-art results on several audio classification benchmarks. 16 | 17 | ## Usage 18 | 19 | You can use the raw model for classifying audio into one of the AudioSet classes. See the [documentation](https://huggingface.co/docs/transformers/main/en/model_doc/audio-spectrogram-transformer#transformers.ASTForAudioClassification.forward.example) for more info. -------------------------------------------------------------------------------- /CNN_torch/CNN_EEG.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from torch.utils.data import DataLoader, TensorDataset, random_split 5 | import torch.optim as optim 6 | 7 | 8 | import torch 9 | import torch.nn as nn 10 | import torch.nn.functional as F 11 | from torch.nn.parameter import Parameter 12 | 13 | 14 | class EEGNet(nn.Module): 15 | def __init__(self, nb_classes, Chans=64, Samples=128, dropoutRate=0.5, kernLength=64, F1=8, D=2, F2=16, norm_rate=0.25): 16 | super(EEGNet, self).__init__() 17 | self.drop_rate = dropoutRate 18 | 19 | self.block1 = nn.Sequential( 20 | nn.Conv2d(1, F1, (1, kernLength), padding='same', bias=False), 21 | nn.BatchNorm2d(F1), 22 | nn.Conv2d(F1, D*F1, (Chans, 1), groups=F1, bias=False), 23 | nn.BatchNorm2d(D*F1), 24 | nn.ELU(), 25 | nn.AvgPool2d((1, 4)), 26 | nn.Dropout(self.drop_rate) 27 | ) 28 | 29 | self.block2 = nn.Sequential( 30 | nn.Conv2d(D*F1, F2, (1, 16), groups=D*F1, bias=False), 31 | nn.Conv2d(F2, F2, 1, bias=False), 32 | nn.BatchNorm2d(F2), 33 | nn.ELU(), 34 | nn.AvgPool2d((1, 8)), 35 | nn.Dropout(self.drop_rate) 36 | ) 37 | 38 | self.classifier = nn.Sequential( 39 | nn.Flatten(), 40 | nn.Linear(832, nb_classes), 41 | nn.Softmax(dim=1) 42 | ) 43 | 44 | def forward(self, x): 45 | x = x.unsqueeze(1) 46 | x = self.block1(x) 47 | x = self.block2(x) 48 | x = self.classifier(x) 49 | return x 50 | 51 | class EEGNetTrainer: 52 | def __init__(self, model, train_dataset, val_dataset, batch_size=32, epochs=100): 53 | self.model = model 54 | self.train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True) 55 | self.test_loader = DataLoader(val_dataset, batch_size=batch_size) 56 | self.criterion = torch.nn.CrossEntropyLoss() 57 | self.optimizer = torch.optim.Adam(model.parameters()) 58 | self.epochs = epochs 59 | self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 60 | self.model.to(self.device) 61 | print(self.device) 62 | 63 | def train_epoch(self): 64 | self.model.train() 65 | running_loss = 0.0 66 | for inputs, labels in self.train_loader: 67 | self.optimizer.zero_grad() 68 | outputs = self.model(inputs) 69 | loss = self.criterion(outputs, labels) 70 | loss.backward() 71 | self.optimizer.step() 72 | running_loss += loss.item() 73 | return running_loss / len(self.train_loader) 74 | 75 | def validate_epoch(self): 76 | self.model.eval() 77 | val_loss = 0.0 78 | correct = 0 79 | total = 0 80 | with torch.no_grad(): 81 | for inputs, labels in self.test_loader: 82 | inputs, labels = inputs.to(self.device), labels.to(self.device) 83 | outputs = self.model(inputs) 84 | loss = self.criterion(outputs, labels) 85 | val_loss += loss.item() 86 | _, predicted = torch.max(outputs.data, 1) 87 | total += labels.size(0) 88 | correct += (predicted == labels).sum().item() 89 | accuracy = 100 * correct / total 90 | return val_loss / len(self.test_loader), accuracy 91 | 92 | def train(self): 93 | for epoch in range(self.epochs): 94 | train_loss = self.train_epoch() 95 | val_loss, accuracy = self.validate_epoch() 96 | print(f'Epoch {epoch+1}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}, Accuracy: {accuracy:.2f}%') 97 | 98 | def predict(self): 99 | predictions = [] 100 | self.model.eval() 101 | with torch.no_grad(): 102 | for inputs, labels in self.test_loader: 103 | inputs, labels = inputs.to(self.device), labels.to(self.device) 104 | outputs = self.model(inputs) 105 | _, predicted = torch.max(outputs.data, 1) 106 | predictions.extend(predicted.tolist()) 107 | return predictions 108 | -------------------------------------------------------------------------------- /EAV_datasplit.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import torch 4 | from torch.utils.data import TensorDataset 5 | from torch.utils.data import DataLoader 6 | 7 | 8 | class EAVDataSplit: 9 | def __init__(self, x, y, batch_size=32): 10 | self.x = np.array(x) 11 | self.y = np.array(y) 12 | self.batch_size = batch_size 13 | def _split_features_labels(self): 14 | # Splitting features and labels based on class, select each 80 samples per class in order 15 | features = [] 16 | labels = [] 17 | for class_idx in range(5): # Assuming there are 5 classes 18 | class_mask = np.where(self.y == class_idx) 19 | class_features = self.x[class_mask] 20 | class_labels = self.y[class_mask] 21 | 22 | features.append(class_features) 23 | labels.append(class_labels) 24 | 25 | return features, labels 26 | 27 | def get_split(self, h_idx=40): # update it if you want to use different ratio here we have 50/50 28 | [features, labels] = self._split_features_labels() 29 | # Splitting into training and testing 30 | train_features = np.concatenate([cls_features[:h_idx] for cls_features in features], axis=0) 31 | test_features = np.concatenate([cls_features[h_idx:] for cls_features in features], axis=0) 32 | train_labels = np.concatenate([cls_labels[:h_idx] for cls_labels in labels], axis=0) 33 | test_labels = np.concatenate([cls_labels[h_idx:] for cls_labels in labels], axis=0) 34 | 35 | # 36 | train_features = np.squeeze(train_features) 37 | test_features = np.squeeze(test_features) 38 | train_labels = train_labels 39 | test_labels = test_labels 40 | 41 | return train_features, train_labels, test_features, test_labels 42 | 43 | def get_loaders(self): 44 | self._split_features_labels() 45 | train_features, train_labels, test_features, test_labels = self.get_split() 46 | 47 | train_features = torch.Tensor(np.squeeze(train_features)) 48 | test_features = torch.Tensor(np.squeeze(test_features)) 49 | train_labels = torch.Tensor(train_labels).long() # Using .long() for labels 50 | test_labels = torch.Tensor(test_labels).long() 51 | 52 | # Creating TensorDatasets and DataLoaders 53 | train_dataset = TensorDataset(train_features, train_labels) 54 | test_dataset = TensorDataset(test_features, test_labels) 55 | 56 | loader_train = DataLoader(train_dataset, batch_size=self.batch_size, shuffle=True) 57 | loader_test = DataLoader(test_dataset, batch_size=self.batch_size, shuffle=False) 58 | 59 | return loader_train, loader_test 60 | 61 | ''' 62 | # 50/50 in sequential order, straight forward codes without loops 63 | def EAV_train_test(x, y): 64 | c1 = np.where(np.array(y) == 0) 65 | c2 = np.where(np.array(y) == 1) 66 | c3 = np.where(np.array(y) == 2) 67 | c4 = np.where(np.array(y) == 3) 68 | c5 = np.where(np.array(y) == 4) 69 | # (400, 1, 1024, 128), np.array(feature) 70 | f1 = np.array(x)[c1] 71 | f2 = np.array(x)[c2] 72 | f3 = np.array(x)[c3] 73 | f4 = np.array(x)[c4] 74 | f5 = np.array(x)[c5] 75 | 76 | h_idx = 40 77 | train_features = np.concatenate((f1[:h_idx], f2[:h_idx], f3[:h_idx], f4[:h_idx], f5[:h_idx]), axis=0) 78 | test_features = np.concatenate((f1[h_idx:], f2[h_idx:], f3[h_idx:], f4[h_idx:], f5[h_idx:]), axis=0) 79 | train_features = torch.Tensor(np.squeeze(train_features)) 80 | test_features = torch.Tensor(np.squeeze(test_features)) 81 | 82 | # do this to the label as well 83 | c1 = np.where(np.array(y) == 0) 84 | c2 = np.where(np.array(y) == 1) 85 | c3 = np.where(np.array(y) == 2) 86 | c4 = np.where(np.array(y) == 3) 87 | c5 = np.where(np.array(y) == 4) 88 | # (400, 1, 1024, 128), np.array(feature) 89 | l1 = np.array(y)[c1] 90 | l2 = np.array(y)[c2] 91 | l3 = np.array(y)[c3] 92 | l4 = np.array(y)[c4] 93 | l5 = np.array(y)[c5] 94 | 95 | train_labels = np.concatenate((l1[:h_idx], l2[:h_idx], l3[:h_idx], l4[:h_idx], l5[:h_idx]), axis=0) 96 | test_labels = np.concatenate((l1[h_idx:], l2[h_idx:], l3[h_idx:], l4[h_idx:], l5[h_idx:]), axis=0) 97 | train_labels = torch.Tensor(train_labels) 98 | test_labels = torch.Tensor(test_labels) 99 | 100 | train_dataset = TensorDataset(train_features, train_labels) 101 | test_dataset = TensorDataset(test_features, test_labels) 102 | 103 | loader_train = DataLoader(train_dataset, batch_size=32, shuffle=True) 104 | loader_test = DataLoader(test_dataset, batch_size=32, shuffle=True) 105 | return loader_train, loader_test 106 | ''' 107 | 108 | -------------------------------------------------------------------------------- /Transformer_torch/Transformer_Audio.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.optim as optim 4 | from transformers import AutoModelForAudioClassification 5 | from torch.utils.data import DataLoader, TensorDataset 6 | from transformers import ASTFeatureExtractor 7 | import numpy as np 8 | 9 | class AudioModelTrainer: 10 | def __init__(self, DATA, model_path, sub = '', num_classes=5, weight_decay=1e-5, lr=0.001, batch_size=128): 11 | 12 | self.tr, self.tr_y, self.te, self.te_y = DATA 13 | self.tr_x = self._feature_extract(self.tr) 14 | self.te_x = self._feature_extract(self.te) 15 | 16 | self.sub = sub 17 | self.batch_size = batch_size 18 | 19 | self.train_dataloader = self._prepare_dataloader(self.tr_x, self.tr_y, shuffle=True) 20 | self.test_dataloader = self._prepare_dataloader(self.te_x, self.te_y, shuffle=False) 21 | 22 | self.model = AutoModelForAudioClassification.from_pretrained(model_path) 23 | # Modify classifier to fit the number of classes 24 | self.model.classifier.dense = torch.nn.Linear(self.model.classifier.dense.in_features, num_classes) 25 | 26 | self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 27 | self.model = self.model.to(self.device) 28 | 29 | # Setup optimizer and loss function 30 | self.initial_lr = lr 31 | self.optimizer = optim.AdamW(self.model.parameters(), lr=self.initial_lr) 32 | self.loss_fn = nn.CrossEntropyLoss() 33 | 34 | def _prepare_dataloader(self, x, y, shuffle=False): 35 | dataset = TensorDataset(torch.tensor(x, dtype=torch.float32), torch.tensor(y, dtype=torch.long)) 36 | dataloader = DataLoader(dataset, batch_size=self.batch_size, shuffle=shuffle) 37 | return dataloader 38 | 39 | def _feature_extract(self, x): 40 | feature_extractor = ASTFeatureExtractor() 41 | ft = feature_extractor(x, sampling_rate=16000, padding='max_length', 42 | return_tensors='pt') 43 | return ft['input_values'] 44 | 45 | def train(self, epochs=20, lr=None, freeze=True): 46 | lr = lr if lr is not None else self.initial_lr 47 | if lr is not None: 48 | for param_group in self.optimizer.param_groups: 49 | param_group['lr'] = lr 50 | 51 | if isinstance(self.model, nn.DataParallel): 52 | self.model = self.model.module 53 | # Freeze or unfreeze model parameters based on the freeze flag 54 | for param in self.model.parameters(): 55 | param.requires_grad = not freeze 56 | for param in self.model.classifier.parameters(): 57 | param.requires_grad = True 58 | 59 | # Wrap the model with DataParallel 60 | if torch.cuda.device_count() > 1: 61 | self.model = nn.DataParallel(self.model) 62 | 63 | for epoch in range(epochs): 64 | self.model.train() 65 | train_correct, train_total = 0, 0 66 | 67 | total_batches = len(self.train_dataloader) 68 | for batch_idx, batch in enumerate(self.train_dataloader, start=1): 69 | #print(f'batch ({batch_idx}/{total_batches})') 70 | 71 | x, t = [b.to(self.device) for b in batch] 72 | self.optimizer.zero_grad() 73 | logits = self.model(x).logits 74 | loss = self.loss_fn(logits, t) 75 | if loss.dim() > 0: 76 | loss = loss.mean() 77 | else: 78 | loss = loss 79 | loss.backward() 80 | self.optimizer.step() 81 | 82 | train_correct += (logits.argmax(dim=-1) == t).sum().item() 83 | train_total += t.size(0) 84 | train_accuracy = train_correct / train_total 85 | 86 | self.model.eval() 87 | correct, total = 0, 0 88 | outputs_batch = [] 89 | with torch.no_grad(): 90 | for x, t in self.test_dataloader: 91 | x, t = x.to(self.device), t.long().to(self.device) 92 | logits = self.model(x).logits 93 | correct += (logits.argmax(dim=-1) == t).sum().item() 94 | total += t.size(0) 95 | 96 | logits_cpu = logits.detach().cpu().numpy() 97 | outputs_batch.append(logits_cpu) 98 | test_accuracy = correct / total 99 | if epoch == epochs-1 and not freeze: # we saved test prediction only at last epoch, and finetuning 100 | self.outputs_test = np.concatenate(outputs_batch, axis=0) 101 | 102 | print(f"Epoch {epoch + 1}/{epochs}, Training Accuracy: {train_accuracy * 100:.2f}%, Test Accuracy: {test_accuracy * 100:.2f}%") 103 | with open('training_performance_audio.txt', 'a') as f: 104 | f.write(f"{self.sub}, Epoch {epoch + 1}, Test Accuracy: {test_accuracy * 100:.2f}%\n") 105 | 106 | 107 | 108 | -------------------------------------------------------------------------------- /Dataload_vision.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import numpy as np 4 | import EAV_datasplit 5 | from facenet_pytorch import MTCNN 6 | import torch 7 | 8 | 9 | class DataLoadVision: 10 | def __init__(self, subject='all', parent_directory=r'C:\Users\minho.lee\Dropbox\EAV', face_detection=False, 11 | image_size=224): 12 | self.IMG_HEIGHT, self.IMG_WIDTH = 480, 640 13 | self.subject = subject 14 | self.parent_directory = parent_directory 15 | self.file_path = list() 16 | self.file_emotion = list() 17 | self.images = list() 18 | self.image_label = list() # actual class name 19 | self.image_label_idx = list() 20 | self.face_detection = face_detection 21 | self.image_size = image_size 22 | self.face_image_size = 56 # 23 | self.device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') 24 | 25 | self.mtcnn = MTCNN( 26 | image_size=self.face_image_size, margin=0, min_face_size=20, 27 | thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True, 28 | device=self.device 29 | ) 30 | 31 | def data_files(self): 32 | subject = f'subject{self.subject:02d}' 33 | print(subject, " Loading") 34 | file_emotion = [] 35 | subjects = [] 36 | path = os.path.join(self.parent_directory, subject, 'Video') 37 | for i in os.listdir(path): 38 | emotion = i.split('_')[4] 39 | self.file_emotion.append(emotion) 40 | self.file_path.append(os.path.join(path, i)) 41 | 42 | def data_load(self): 43 | 44 | for idx, file in enumerate(self.file_path): 45 | nm_class = file.split("_")[-1].split(".")[0] # we extract the class label from the file 46 | 47 | if "Speaking" in file and file.endswith(".mp4"): 48 | print(idx) 49 | cap = cv2.VideoCapture(file) 50 | # total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) # ~600 51 | # frame_rate = cap.get(cv2.CAP_PROP_FPS) # 30 frame 52 | a1 = [] 53 | if cap.isOpened(): 54 | frame_index = 1 55 | while True: 56 | ret, frame = cap.read() 57 | if not ret: 58 | break 59 | frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) 60 | # (30 framerate * 20s) * 100 Speaking, Select every 6th frame from the first 600 frames 61 | # face detection, we converted it into 0-255 again from the [-1 - 1] tensor, you can directly return the tensor 62 | if (frame_index - 1) % 6 == 0 and frame_index <= 600: 63 | if self.face_detection: 64 | with torch.no_grad(): 65 | x_aligned, prob = self.mtcnn(frame, return_prob=True) 66 | if prob > 0.3: 67 | x_aligned = (x_aligned + 1) / 2 68 | x_aligned = np.clip(x_aligned * 255, 0, 255) 69 | x_aligned = np.transpose(x_aligned.numpy().astype('uint8'), (1, 2, 0)) 70 | a1.append(x_aligned) 71 | else: 72 | print("Face is not detected, original is saved") 73 | a1.append(x_aligned) # incase that face has not been detected, add previous one 74 | pass 75 | else: 76 | resizedImg = cv2.resize(frame, (self.image_size, self.image_size)) 77 | a1.append(resizedImg) # sabina: dlkfjefoie 78 | 79 | if len(a1) == 25: # 25 frame is 5s each 80 | self.images.append(a1) # this will contain 400 samples [400, 25, (225, 225, 3)] 81 | a1 = [] 82 | self.image_label.append(nm_class) 83 | frame_index += 1 84 | cap.release() 85 | else: 86 | print(f"Error opening video file: {file}") 87 | emotion_to_index = { 88 | 'Neutral': 0, 89 | 'Happiness': 3, 90 | 'Sadness': 1, 91 | 'Anger': 2, 92 | 'Calmness': 4 93 | } 94 | self.image_label_idx = [emotion_to_index[emotion] for emotion in self.image_label] 95 | 96 | def process(self): 97 | self.data_files() 98 | self.data_load() 99 | return self.images, self.image_label_idx 100 | 101 | 102 | if __name__ == '__main__': 103 | 104 | for sub in range(1, 20): 105 | print(sub) 106 | file_path = "C:/Users/minho.lee/Dropbox/Datasets/EAV/Input_images/Vision/" 107 | file_name = f"subject_{sub:02d}_vis.pkl" 108 | file_ = os.path.join(file_path, file_name) 109 | #if not os.path.exists(file_): 110 | 111 | vis_loader = DataLoadVision(subject=sub, parent_directory=r'C:\Users\minho.lee\Dropbox\Datasets\EAV', face_detection=True) 112 | [data_vis, data_vis_y] = vis_loader.process() 113 | 114 | eav_loader = EAV_datasplit.EAVDataSplit(data_vis, data_vis_y) 115 | 116 | #each class contains 80 trials, 5/5 radio (h_idx=40), 7/3 ratio (h_dix=56) 117 | [tr_x_vis, tr_y_vis, te_x_vis, te_y_vis] = eav_loader.get_split(h_idx=56) # output(list): train, trlabel, test, telabel 118 | data = [tr_x_vis, tr_y_vis, te_x_vis, te_y_vis] 119 | 120 | ''' 121 | # Here you can write / load vision features tr:{280}(25, 56, 56, 3), te:{120}(25, 56, 56, 3): trials, frames, height, weight, channel 122 | import pickle 123 | Vis_list = [tr_x_vis, tr_y_vis, te_x_vis, te_y_vis] 124 | with open(file_, 'wb') as f: 125 | pickle.dump(Vis_list, f) 126 | 127 | # You can directly work from here 128 | with open(file_, 'rb') as f: 129 | Vis_list = pickle.load(f) 130 | tr_x_vis, tr_y_vis, te_x_vis, te_y_vis = Vis_list 131 | data = [tr_x_vis, tr_y_vis, te_x_vis, te_y_vis] 132 | ''' 133 | # Transformer for Vision 134 | from Transformer_torch import Transformer_Vision 135 | 136 | mod_path = os.path.join('C:\\Users\\minho.lee\\Dropbox\\Projects\\EAV', 'facial_emotions_image_detection') 137 | trainer = Transformer_Vision.ImageClassifierTrainer(data, 138 | model_path=mod_path, sub=f"subject_{sub:02d}", 139 | num_labels=5, lr=5e-5, batch_size=128) 140 | trainer.train(epochs=10, lr=5e-4, freeze=True) 141 | trainer.train(epochs=5, lr=5e-6, freeze=False) 142 | trainer.outputs_test 143 | 144 | # CNN for Vision 145 | from CNN_torch.CNN_Vision import ImageClassifierTrainer 146 | trainer = ImageClassifierTrainer(data, num_labels=5, lr=5e-5, batch_size=32) 147 | trainer.train(epochs=3, lr=5e-4, freeze=True) 148 | trainer.train(epochs=3, lr=5e-6, freeze=False) 149 | trainer._delete_dataloader() 150 | trainer.outputs_test 151 | 152 | 153 | 154 | 155 | 156 | 157 | -------------------------------------------------------------------------------- /CNN_torch/EEGNet_tor.py: -------------------------------------------------------------------------------- 1 | from Dataload_eeg import * 2 | import torch 3 | import torch.nn as nn 4 | from Fusion.VIT_audio.Transformer_audio import Trainer_uni 5 | #from tensorflow.keras.models import Model 6 | from torch.nn.utils import weight_norm 7 | 8 | # Define the EEGNet model 9 | import torch 10 | import torch.nn as nn 11 | import torch.nn.functional as F 12 | from torch.nn.utils.parametrizations import spectral_norm 13 | import torch.optim as optim 14 | 15 | class EEGNet_tor(nn.Module): 16 | def __init__(self, nb_classes, Chans=30, Samples=500, dropoutRate=0.5, kernLength=300, F1=8, D=8, F2=64, 17 | norm_rate=1.0, dropoutType='Dropout'): 18 | super(EEGNet_tor, self).__init__() 19 | 20 | # Configure dropout 21 | self.dropout = nn.Dropout(dropoutRate) if dropoutType == 'Dropout' else nn.Dropout2d(dropoutRate) 22 | 23 | # Block 1 24 | self.firstConv = nn.Conv2d(1, F1, (1, kernLength), padding='same', bias=False) 25 | self.firstBN = nn.BatchNorm2d(F1) 26 | self.elu = nn.ELU() 27 | 28 | self.depthwiseConv = nn.Conv2d(F1, F1 * D, (Chans, 1), groups=F1, padding=0, bias=False) 29 | self.depthwiseBN = nn.BatchNorm2d(F1 * D) 30 | self.depthwisePool = nn.AvgPool2d((1, 4)) 31 | 32 | # Applying max-norm constraint 33 | self.depthwiseConv.register_forward_hook( 34 | lambda module, inputs, outputs: module.weight.data.renorm_(p=2, dim=0, maxnorm=norm_rate)) 35 | 36 | # Block 2 37 | self.separableConv = nn.Conv2d(F1 * D, F2, (1, 16), padding='same', bias=False) 38 | self.separableBN = nn.BatchNorm2d(F2) 39 | self.separablePool = nn.AvgPool2d((1, 8)) 40 | 41 | # Final layers 42 | self.flatten = nn.Flatten() 43 | self.dense = nn.Linear(F2 * ((Samples // 4 // 8)), nb_classes) 44 | self.softmax = nn.Softmax(dim=1) 45 | 46 | # Applying max-norm constraint 47 | self.dense.register_forward_hook( 48 | lambda module, inputs, outputs: module.weight.data.renorm_(p=2, dim=0, maxnorm=norm_rate)) 49 | 50 | def forward(self, x): 51 | x = self.firstConv(x) 52 | x = self.firstBN(x) 53 | x = self.elu(x) 54 | x = self.depthwiseConv(x) 55 | x = self.depthwiseBN(x) 56 | x = self.elu(x) 57 | x = self.depthwisePool(x) 58 | x = self.dropout(x) 59 | x = self.separableConv(x) 60 | x = self.separableBN(x) 61 | x = self.elu(x) 62 | x = self.separablePool(x) 63 | x = self.dropout(x) 64 | x = self.flatten(x) 65 | x = self.dense(x) 66 | x = self.softmax(x) 67 | return x 68 | 69 | class Trainer_uni: 70 | def __init__(self, model, data, lr=1e-4, batch_size=32, num_epochs=10, device=None): 71 | 72 | self.lr = lr 73 | self.batch_size = batch_size 74 | self.num_epochs = num_epochs 75 | 76 | self.tr_x, self.tr_y, self.te_x, self.te_y = data 77 | self.train_dataloader = self._prepare_dataloader(self.tr_x, self.tr_y, shuffle=True) 78 | self.test_dataloader = self._prepare_dataloader(self.te_x, self.te_y, shuffle=False) 79 | 80 | self.model = model 81 | self.criterion = nn.CrossEntropyLoss() 82 | self.optimizer = optim.Adam(self.model.parameters(), lr=self.lr) 83 | 84 | self.device = device if device else torch.device("cuda" if torch.cuda.is_available() else "cpu") 85 | 86 | if torch.cuda.device_count() > 1: 87 | print(f"Using {torch.cuda.device_count()} GPUs!") 88 | self.model = nn.DataParallel(self.model) 89 | self.model.to(self.device) 90 | 91 | def _prepare_dataloader(self, x, y, shuffle=False): 92 | dataset = TensorDataset(torch.tensor(x, dtype=torch.float32), torch.tensor(y, dtype=torch.long)) 93 | dataloader = DataLoader(dataset, batch_size=self.batch_size, shuffle=shuffle) 94 | return dataloader 95 | 96 | def train(self): 97 | self.model.train() # Set model to training mode 98 | for epoch in range(self.num_epochs): 99 | for batch_idx, (data, targets) in enumerate(self.train_dataloader): 100 | data = data.to(self.device) 101 | targets = targets.to(self.device) 102 | 103 | # Forward pass 104 | scores = self.model(data) 105 | loss = self.criterion(scores, targets) 106 | 107 | # Backward pass and optimization 108 | self.optimizer.zero_grad() 109 | loss.backward() 110 | self.optimizer.step() 111 | 112 | if batch_idx % 100 == 0: 113 | print(f"Epoch [{epoch+1}/{self.num_epochs}], Step [{batch_idx}/{len(self.train_dataloader)}], Loss: {loss.item():.4f}") 114 | 115 | if self.test_dataloader: 116 | self.validate() 117 | 118 | def validate(self): 119 | self.model.eval() # Set model to evaluation mode 120 | total_loss = 0 121 | total_correct = 0 122 | with torch.no_grad(): 123 | for data, targets in self.test_dataloader: 124 | data = data.to(self.device) 125 | targets = targets.to(self.device) 126 | 127 | scores = self.model(data) 128 | loss = self.criterion(scores, targets) 129 | total_loss += loss.item() 130 | predictions = scores.argmax(dim=1) 131 | total_correct += (predictions == targets).sum().item() 132 | 133 | avg_loss = total_loss / len(self.test_dataloader) 134 | accuracy = total_correct / len(self.test_dataloader.dataset) 135 | print(f"Validation - Loss: {avg_loss:.4f}, Accuracy: {accuracy:.4f}") 136 | 137 | 138 | def _prepare_dataloader(self, x, y, shuffle=False): 139 | dataset = TensorDataset(torch.tensor(x, dtype=torch.float32), torch.tensor(y, dtype=torch.long)) 140 | dataloader = DataLoader(dataset, batch_size=self.batch_size, shuffle=shuffle) 141 | return dataloader 142 | 143 | 144 | if __name__ == "__main__": 145 | result_acc = list() 146 | for i in range(1, 43): 147 | eeg_loader = DataLoadEEG(subject=i, band=[5, 30], fs_orig=500, fs_target=100, 148 | parent_directory=r'C:\Users\minho.lee\Dropbox\Datasets\EAV') 149 | data_eeg, data_eeg_y = eeg_loader.data_prepare() 150 | 151 | division_eeg = EAVDataSplit(data_eeg, data_eeg_y) 152 | [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] = division_eeg.get_split() 153 | tr_x_eeg = torch.from_numpy(tr_x_eeg).float().unsqueeze(1) # Reshape to (200, 1, 30, 500) 154 | te_x_eeg = torch.from_numpy(te_x_eeg).float().unsqueeze(1) # Reshape to (200, 1, 30, 500) 155 | 156 | data = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 157 | 158 | ##################################################### 159 | model = EEGNet_tor(nb_classes=5, D=8, F2=64, Chans=30, kernLength=300, Samples=500, 160 | dropoutRate=0.5) 161 | trainer = Trainer_uni(model=model, data=data, lr=1e-5, batch_size=32, num_epochs=350) 162 | trainer.train() 163 | model.eval() # 모델을 평가 모드로 설정 164 | 165 | # 손실 함수 설정 166 | criterion = nn.CrossEntropyLoss() 167 | # 데이터를 PyTorch 텐서로 변환 및 GPU로 이동 168 | device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 169 | te_x_eeg = torch.tensor(te_x_eeg, dtype=torch.float32).to(device) 170 | te_y_eeg = torch.tensor(te_y_eeg, dtype=torch.long).to(device) 171 | model.to(device) 172 | # 평가 실행 173 | with torch.no_grad(): 174 | scores = model(te_x_eeg) 175 | predictions = scores.argmax(dim=1) 176 | correct = (predictions == te_y_eeg).sum().item() 177 | total = te_y_eeg.size(0) # 전체 데이터 개수 178 | accuracy = correct / total 179 | 180 | result_acc.append(accuracy) 181 | print(result_acc) 182 | 183 | 184 | -------------------------------------------------------------------------------- /Dataload_audio.py: -------------------------------------------------------------------------------- 1 | import os 2 | import torchaudio 3 | from torchaudio.transforms import Resample 4 | from transformers import ASTFeatureExtractor 5 | 6 | from EAV_datasplit import * 7 | from Transformer_torch import Transformer_Audio 8 | 9 | 10 | class DataLoadAudio: 11 | def __init__(self, subject='all', parent_directory=r'C:\Users\minho.lee\Dropbox\Datasets\EAV', target_sampling_rate=16000): 12 | self.parent_directory = parent_directory 13 | self.original_sampling_rate = int() 14 | self.target_sampling_rate = target_sampling_rate 15 | self.subject = subject 16 | self.file_path = list() 17 | self.file_emotion = list() 18 | 19 | self.seg_length = 5 # 5s 20 | self.feature = None 21 | self.label = None 22 | self.label_indexes = None 23 | self.test_prediction = list() 24 | 25 | def data_files(self): 26 | subject = f'subject{self.subject:02d}' 27 | file_emotion = [] 28 | subjects = [] 29 | path = os.path.join(self.parent_directory, subject, 'Audio') 30 | for i in os.listdir(path): 31 | emotion = i.split('_')[4] 32 | self.file_emotion.append(emotion) 33 | self.file_path.append(os.path.join(path, i)) 34 | 35 | def feature_extraction(self): 36 | x = [] 37 | y = [] 38 | feature_extractor = ASTFeatureExtractor() 39 | for idx, path in enumerate(self.file_path): 40 | waveform, sampling_rate = torchaudio.load(path) 41 | self.original_sampling_rate = sampling_rate 42 | if self.original_sampling_rate is not self.target_sampling_rate: 43 | resampler = Resample(orig_freq=sampling_rate, new_freq=self.target_sampling_rate) 44 | resampled_waveform = resampler(waveform) 45 | resampled_waveform = resampled_waveform.squeeze().numpy() 46 | else: 47 | resampled_waveform = waveform 48 | 49 | segment_length = self.target_sampling_rate * self.seg_length 50 | num_sections = int(np.floor(len(resampled_waveform) / segment_length)) 51 | 52 | for i in range(num_sections): 53 | t = resampled_waveform[i * segment_length: (i + 1) * segment_length] 54 | x.append(t) 55 | y.append(self.file_emotion[idx]) 56 | print(f"Original sf: {self.original_sampling_rate}, resampled into {self.target_sampling_rate}") 57 | 58 | emotion_to_index = { 59 | 'Neutral': 0, 60 | 'Happiness': 3, 61 | 'Sadness': 1, 62 | 'Anger': 2, 63 | 'Calmness': 4 64 | } 65 | y_idx = [emotion_to_index[emotion] for emotion in y] 66 | self.feature = np.squeeze(np.array(x)) 67 | self.label_indexes = np.array(y_idx) 68 | self.label = np.array(y) 69 | 70 | def process(self): 71 | self.data_files() 72 | self.feature_extraction() 73 | return self.feature, self.label_indexes 74 | 75 | def label_emotion(self): 76 | self.data_files() 77 | self.feature_extraction() 78 | return self.label 79 | 80 | if __name__ == "__main__": 81 | test_acc = [] 82 | for sub in range(1,43): 83 | file_path = "C:/Users/minho.lee/Dropbox/Datasets/EAV/Input_images/Audio/" 84 | file_name = f"subject_{sub:02d}_aud.pkl" 85 | file_ = os.path.join(file_path, file_name) 86 | 87 | aud_loader = DataLoadAudio(subject=sub, parent_directory=r'C:\Users\minho.lee\Dropbox\Datasets\EAV') 88 | [data_aud , data_aud_y] = aud_loader.process() 89 | # audio_loader.label_emotion() 90 | 91 | division_aud = EAVDataSplit(data_aud, data_aud_y) 92 | [tr_x_aud, tr_y_aud, te_x_aud , te_y_aud] = division_aud.get_split(h_idx=56) 93 | data = [tr_x_aud, tr_y_aud, te_x_aud , te_y_aud] 94 | 95 | ''' 96 | # Here you can write / load vision features tr:{280}(80000), te:{120}(80000): trials, frames, height, weight, channel 97 | # This code is to store the RAW audio input to the folder: (400, 80000), 16000Hz 98 | import pickle 99 | Aud_list = [tr_x_aud, tr_y_aud, te_x_aud, te_y_aud] 100 | with open(file_, 'wb') as f: 101 | pickle.dump(Aud_list, f) 102 | 103 | # You can directly work from here 104 | with open(file_, 'rb') as f: 105 | Aud_list = pickle.load(f) 106 | [tr_x_aud, tr_y_aud, te_x_aud, te_y_aud] = Aud_list 107 | data = [tr_x_aud, tr_y_aud, te_x_aud , te_y_aud] 108 | ''' 109 | mod_path = os.path.join(os.getcwd(), 'ast-finetuned-audioset') 110 | Trainer = Transformer_Audio.AudioModelTrainer(data, model_path=mod_path, sub =f"subject_{sub:02d}", 111 | num_classes=5, weight_decay=1e-5, lr=0.005, batch_size = 8) 112 | 113 | Trainer.train(epochs=10, lr=5e-4, freeze=True) 114 | Trainer.train(epochs=15, lr=5e-6, freeze=False) 115 | test_acc.append(Trainer.outputs_test) 116 | 117 | ## Add CNN - audio here, refer to the file Dataload_vision.py 118 | 119 | 120 | 121 | 122 | 123 | ''' This code is to store the RAW audio input to the folder: (400, 80000), 16000Hz 124 | if __name__ == "__main__": 125 | test_acc = [] 126 | for sub in range(1,43): 127 | print(sub) 128 | file_path = "C:/Users/minho.lee/Dropbox/Datasets/EAV/Input_images/Audio/" 129 | file_name = f"subject_{sub:02d}_aud.pkl" 130 | file_ = os.path.join(file_path, file_name) 131 | 132 | aud_loader = DataLoadAudio(subject=sub, parent_directory='C:/Users/minho.lee/Dropbox/Datasets/EAV') 133 | [data_aud , data_aud_y] = aud_loader.process() 134 | # audio_loader.label_emotion() 135 | 136 | division_aud = EAVDataSplit(data_aud, data_aud_y) 137 | [tr_x_aud, tr_y_aud, te_x_aud , te_y_aud] = division_aud.get_split(h_idx=56) 138 | 139 | Aud_list = [tr_x_aud, tr_y_aud, te_x_aud, te_y_aud] 140 | import pickle 141 | with open(file_, 'wb') as f: 142 | pickle.dump(Aud_list, f) 143 | ''' 144 | 145 | 146 | 147 | ''' test it with the current data 148 | import pickle 149 | with open("test_acc_audio.pkl", 'wb') as f: 150 | pickle.dump(test_acc, f) 151 | 152 | 153 | 154 | 155 | with open("test_acc_audio.pkl", 'rb') as f: 156 | testacc = pickle.load(f) 157 | 158 | # test accuracy for 200 trials 159 | ## acquire the test label from one subject, it is same for all subjects 160 | from sklearn.metrics import f1_score 161 | file_name = f"subject_{1:02d}_vis.pkl" 162 | file_ = os.path.join(os.getcwd(), 'Feature_vision', file_name) 163 | with open(file_, 'rb') as f: 164 | vis_list2 = pickle.load(f) 165 | te_y_vis = vis_list2[3] 166 | 167 | # load test accuracy for all subjects: 5000 (200, 25) predictions 168 | with open("test_acc_vision.pkl", 'rb') as f: 169 | testacc = pickle.load(f) 170 | 171 | test_acc_all = list() 172 | test_f1_all = list() 173 | for sub in range(42): 174 | aa = testacc[sub] 175 | out1 = np.argmax(aa, axis = 1) 176 | accuracy = np.mean(out1 == te_y_vis) 177 | test_acc_all.append(accuracy) 178 | 179 | f1 = f1_score(te_y_vis, out1, average='weighted') 180 | test_f1_all.append(f1) 181 | 182 | test_acc_all = np.reshape(np.array(test_acc_all), (42, 1)) 183 | test_f1_all = np.reshape(np.array(test_f1_all), (42, 1)) 184 | 185 | 186 | 187 | 188 | 189 | model = AutoModelForAudioClassification.from_pretrained("MIT/ast-finetuned-audioset") 190 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 191 | model = model.to(device) 192 | 193 | test_data = torch.tensor(data, dtype=torch.float32) 194 | test_data = test_data.to(device) 195 | aa = test_data[0:20] 196 | with torch.no_grad(): # 572 classes. 197 | logits = model(aa).logits 198 | 199 | probs = torch.nn.functional.softmax(logits, dim=-1) 200 | predicted_class_id = probs.argmax(dim=1) 201 | bb = np.array(probs.cpu()) 202 | config = model.config 203 | config.num_labels 204 | ''' -------------------------------------------------------------------------------- /Transformer_torch/Transformer_Vision.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.utils.data import DataLoader, TensorDataset 3 | from transformers import AutoImageProcessor, AutoModelForImageClassification 4 | import torch.nn as nn 5 | import numpy as np 6 | 7 | 8 | def calculate_accuracy(outputs, labels): 9 | _, predicted = torch.max(outputs.logits, 1) 10 | correct = (predicted == labels).sum().item() 11 | return correct / labels.size(0) 12 | 13 | 14 | class ImageClassifierTrainer: 15 | def __init__(self, DATA, model_path, sub = '', num_labels=5, lr=5e-5, batch_size=128): 16 | self.tr_x, self.tr_y, self.te_x, self.te_y = DATA 17 | self.model_path = model_path 18 | self.num_labels = num_labels 19 | self.initial_lr = lr # Storing initial learning rate for reference 20 | self.batch_size = batch_size 21 | self.frame_per_sample = np.shape(self.tr_x)[1] # Assuming tr_x is a numpy array or similar 22 | self.sub = sub # this is for log data 23 | self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 24 | 25 | self.test_prediction = list() 26 | 27 | # Initialize model and processor 28 | self.processor = AutoImageProcessor.from_pretrained(model_path) 29 | self.model = AutoModelForImageClassification.from_pretrained(model_path) 30 | self.model.classifier = torch.nn.Linear(self.model.config.hidden_size, self.num_labels) 31 | self.model.num_labels = self.num_labels 32 | 33 | self.model.to(self.device) 34 | 35 | # Initial optimizer setup with initial learning rate 36 | self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=self.initial_lr) 37 | 38 | # Prepare dataloaders, test doesn't need to be in dataloader 39 | print("Image preprocessing..") 40 | self.train_dataloader = self._prepare_dataloader(self.tr_x, self.tr_y, shuffle=True)[0] 41 | self.test_dataloader = self._prepare_dataloader(self.te_x, self.te_y, shuffle=False)[0] 42 | print("Ended..") 43 | 44 | def _prepare_dataloader(self, x, y, shuffle=True): 45 | processed_x = self.preprocess_images(x) 46 | y_repeated = torch.from_numpy(np.repeat(y, self.frame_per_sample)).long() 47 | 48 | dataset = TensorDataset(processed_x.view(-1, 3, 224, 224), y_repeated) 49 | dataloader = DataLoader(dataset, batch_size=self.batch_size, shuffle=shuffle) 50 | return dataloader, processed_x, y_repeated 51 | 52 | def preprocess_images(self, image_list): 53 | pixel_values_list = [] 54 | for img_set in image_list: 55 | for img in img_set: 56 | processed = self.processor(images=img, return_tensors="pt") 57 | pixel_values = processed.pixel_values.squeeze() 58 | pixel_values_list.append(pixel_values) 59 | return torch.stack(pixel_values_list).to(self.device) 60 | 61 | def train(self, epochs=3, lr=None, freeze=True, log = False): 62 | # Update learning rate if provided, otherwise use the initial learning rate 63 | lr = lr if lr is not None else self.initial_lr 64 | if lr is not None: 65 | for param_group in self.optimizer.param_groups: 66 | param_group['lr'] = lr 67 | 68 | for param_group in self.optimizer.param_groups: 69 | param_group['lr'] = lr 70 | 71 | if isinstance(self.model, nn.DataParallel): 72 | self.model = self.model.module 73 | # Freeze or unfreeze model parameters based on the freeze flag 74 | for param in self.model.parameters(): 75 | param.requires_grad = not freeze 76 | for param in self.model.classifier.parameters(): 77 | param.requires_grad = True 78 | 79 | print(f"Training with {'frozen' if freeze else 'unfrozen'} feature layers at lr={lr}") 80 | 81 | # Wrap the model with DataParallel 82 | if torch.cuda.device_count() > 1: 83 | self.model = nn.DataParallel(self.model) 84 | 85 | for epoch in range(epochs): 86 | # Training loop 87 | self.model.train() 88 | total_batches = len(self.train_dataloader) 89 | for batch_idx, batch in enumerate(self.train_dataloader, start=1): 90 | pixel_values, labels = [b.to(self.device) for b in batch] 91 | self.optimizer.zero_grad() 92 | 93 | outputs = self.model(pixel_values=pixel_values, labels=labels) 94 | 95 | if outputs.loss.dim() > 0: 96 | loss = outputs.loss.mean() 97 | else: 98 | loss = outputs.loss 99 | 100 | loss.backward() 101 | self.optimizer.step() 102 | 103 | print(f'batch ({batch_idx}/{total_batches})') 104 | 105 | # Evaluation loop 106 | self.model.eval() 107 | total_accuracy = 0 108 | outputs_batch = [] 109 | with torch.no_grad(): 110 | for batch in self.test_dataloader: 111 | pixel_values, labels = [b.to(self.device) for b in batch] 112 | outputs = self.model(pixel_values) 113 | 114 | accuracy = calculate_accuracy(outputs, labels) 115 | total_accuracy += accuracy 116 | 117 | logits = outputs.logits if hasattr(outputs, 'logits') else outputs 118 | logits_cpu = logits.detach().cpu().numpy() 119 | outputs_batch.append(logits_cpu) 120 | 121 | if epoch == epochs-1 and not freeze: # we saved test prediction only at last epoch, and finetuning 122 | self.outputs_test = np.concatenate(outputs_batch, axis=0) 123 | outputs_batch = [] 124 | 125 | avg_accuracy = total_accuracy / len(self.test_dataloader) 126 | print(f"Epoch {epoch + 1}, Test Accuracy: {avg_accuracy * 100:.2f}%") 127 | 128 | if log: 129 | with open('training_performance.txt', 'a') as f: 130 | f.write(f"{self.sub}, Epoch {epoch + 1}, Test Accuracy: {avg_accuracy * 100:.2f}%\n") 131 | 132 | 133 | if __name__ == '__main__': 134 | import pickle 135 | import os 136 | test_acc = [] 137 | for idx in range(1, 43): 138 | 139 | file_name = f"subject_{idx:02d}_vis.pkl" 140 | file_ = os.path.join(os.getcwd(), 'Feature_vision', file_name) 141 | 142 | with open(file_, 'rb') as f: 143 | vis_list2 = pickle.load(f) 144 | tr_x_vis, tr_y_vis, te_x_vis, te_y_vis = vis_list2 145 | 146 | mod_path = os.path.join(os.getcwd(), 'facial_emotions_image_detection') 147 | data = [tr_x_vis, tr_y_vis, te_x_vis, te_y_vis] 148 | trainer = ImageClassifierTrainer(data, 149 | model_path=mod_path, sub = f"subject_{idx:02d}", 150 | num_labels=5, lr=5e-5, batch_size=128) 151 | 152 | trainer.train(epochs=10, lr=5e-4, freeze=True) 153 | trainer.train(epochs=5, lr=5e-6, freeze=False) 154 | 155 | test_acc.append(trainer.outputs_test) 156 | 157 | import pickle 158 | with open("test_acc_vision.pkl", 'wb') as f: 159 | pickle.dump(test_acc, f) 160 | 161 | 162 | # test accuracy for 200 trials 163 | ## acquire the test label from one subject, it is same for all subjects 164 | from sklearn.metrics import f1_score 165 | file_name = f"subject_{1:02d}_vis.pkl" 166 | file_ = os.path.join(os.getcwd(), 'Feature_vision', file_name) 167 | with open(file_, 'rb') as f: 168 | vis_list2 = pickle.load(f) 169 | te_y_vis = vis_list2[3] 170 | 171 | # load test accuracy for all subjects: 5000 (200, 25) predictions 172 | with open("test_acc_vision.pkl", 'rb') as f: 173 | testacc = pickle.load(f) 174 | 175 | test_acc_all = list() 176 | test_f1_all = list() 177 | for sub in range(42): 178 | aa = testacc[sub] 179 | aa2 = np.reshape(aa, (200, 25, 5), 'C') 180 | aa3 = np.mean(aa2, 1) 181 | out1 = np.argmax(aa3, axis = 1) 182 | accuracy = np.mean(out1 == te_y_vis) 183 | test_acc_all.append(accuracy) 184 | 185 | f1 = f1_score(te_y_vis, out1, average='weighted') 186 | test_f1_all.append(f1) 187 | 188 | test_acc_all = np.reshape(np.array(test_acc_all), (42, 1)) 189 | test_f1_all = np.reshape(np.array(test_f1_all), (42, 1)) 190 | ''' 191 | # Creating toy data for training 192 | import numpy as np 193 | 194 | tr_x = np.random.randint(0, 256, (128, 25, 224, 224, 3)) # Training images 195 | te_x = np.random.randint(0, 256, (128, 25, 224, 224, 3)) # Test images 196 | 197 | tr_y = np.random.randint(0, 5, (128,)) # Training labels 198 | te_y = np.random.randint(0, 5, (128,)) # Test labels 199 | 200 | data = [tr_x, tr_y, te_x, te_y] 201 | trainer = ImageClassifierTrainer(data, 202 | model_path='C:/Users/minho.lee/Dropbox/zEmotion_fusion/pythonProject/facial_emotions_image_detection', 203 | num_labels=5, lr=5e-5, batch_size=128) 204 | 205 | trainer.train(epochs=3, lr=5e-5, freeze=True) 206 | trainer.train(epochs=3, lr=5e-6, freeze=False) 207 | ''' 208 | -------------------------------------------------------------------------------- /CNN_torch/CNN_Vision.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.utils.data import DataLoader, TensorDataset 3 | from transformers import AutoImageProcessor, AutoModelForImageClassification 4 | import torch.nn as nn 5 | import numpy as np 6 | import pickle 7 | import cv2 8 | from torchvision import transforms 9 | import torch.nn.functional as F 10 | from torchvision.models import resnet50 11 | from PIL import Image 12 | 13 | transform = transforms.Compose([ 14 | transforms.Resize((224, 224)), 15 | transforms.ToTensor(), 16 | transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) 17 | ]) 18 | 19 | 20 | class VideoModel(nn.Module): 21 | def __init__(self, num_labels=5, ratio=1): 22 | self.num_labels = num_labels 23 | super(VideoModel, self).__init__() 24 | base_model = resnet50(pretrained=True, progress=True) 25 | self.base_model = torch.nn.Sequential(*(list(base_model.children())[:-2])) 26 | self.avg_pool = nn.AdaptiveAvgPool2d(1) 27 | self.max_pool = nn.AdaptiveMaxPool2d(1) 28 | self.shared_layer_one = nn.Linear(2048 // ratio, 2048) 29 | self.shared_layer_two = nn.Linear(2048, 2048) 30 | self.global_avg_pool = nn.AdaptiveAvgPool2d(1) 31 | self.flatten = nn.Flatten() 32 | self.fc1 = nn.Linear(2048, 1024) 33 | self.fc2 = nn.Linear(1024, self.num_labels) 34 | self.ratio = ratio 35 | 36 | def channel_attention(self, x): 37 | avg_pool = self.avg_pool(x) 38 | max_pool = self.max_pool(x) 39 | avg_pool = self.shared_layer_one(avg_pool.view(avg_pool.size(0), -1)) 40 | max_pool = self.shared_layer_one(max_pool.view(max_pool.size(0), -1)) 41 | avg_pool = self.shared_layer_two(avg_pool) 42 | max_pool = self.shared_layer_two(max_pool) 43 | return avg_pool + max_pool 44 | 45 | def forward(self, x): 46 | x = self.base_model(x) 47 | #ipdb.set_trace() 48 | attention = self.channel_attention(x) 49 | x = x * attention.unsqueeze(2).unsqueeze(3) 50 | x = self.global_avg_pool(x) 51 | x = self.flatten(x) 52 | x = F.relu(self.fc1(x)) 53 | x = self.fc2(x) 54 | return x 55 | 56 | 57 | 58 | class ImageClassifierTrainer: 59 | def __init__(self, DATA, num_labels=5, lr=5e-5, batch_size=128): 60 | self.tr_x, self.tr_y, self.te_x, self.te_y = DATA 61 | self.num_labels = num_labels 62 | self.initial_lr = lr # Storing initial learning rate for reference 63 | self.batch_size = batch_size 64 | self.frame_per_sample = np.shape(self.tr_x)[1] # Assuming tr_x is a numpy array or similar 65 | 66 | self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 67 | 68 | # Initialize model and processor 69 | self.processor=transform 70 | self.model = VideoModel(self.num_labels) 71 | self.model.to(self.device) 72 | 73 | self.criterion = nn.CrossEntropyLoss() 74 | # Initial optimizer setup with initial learning rate 75 | self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=self.initial_lr) 76 | 77 | # Prepare dataloaders 78 | print("Image preprocessing..") 79 | self.train_dataloader = self._prepare_dataloader(self.tr_x, self.tr_y) 80 | self.test_dataloader = self._prepare_dataloader(self.te_x, self.te_y, shuffle=False) 81 | print("Ended..") 82 | def calculate_accuracy(self,outputs, labels): 83 | #ipdb.set_trace() 84 | #probabilities = torch.softmax(outputs, dim=1) 85 | _, predicted = torch.max(outputs, 1) 86 | correct = (predicted == labels).sum().item() 87 | return correct / labels.size(0) 88 | def _prepare_dataloader(self, x, y, shuffle=True): 89 | processed_x = self.preprocess_images(x) 90 | y_repeated = torch.from_numpy(np.repeat(y, self.frame_per_sample)).long() 91 | 92 | dataset = TensorDataset(processed_x.view(-1, 3, 224, 224), y_repeated) 93 | dataloader = DataLoader(dataset, batch_size=self.batch_size, shuffle=shuffle) 94 | del dataset 95 | return dataloader 96 | def _delete_dataloader(self): 97 | del self.train_dataloader 98 | del self.test_dataloader 99 | 100 | def preprocess_images(self, image_list): 101 | pixel_values_list = [] 102 | for img_set in image_list: 103 | for img in img_set: 104 | pil_img = Image.fromarray(img) 105 | processed = self.processor(pil_img) 106 | #ipdb.set_trace() 107 | pixel_values = processed.squeeze() #check if the shape is (_, 224, 224, 3) 108 | pixel_values_list.append(pixel_values) 109 | return torch.stack(pixel_values_list).to(self.device) 110 | 111 | def train(self, epochs=3, lr=None, freeze=True): 112 | # Update learning rate if provided, otherwise use the initial learning rate 113 | lr = lr if lr is not None else self.initial_lr 114 | for param_group in self.optimizer.param_groups: 115 | param_group['lr'] = lr 116 | 117 | # Freeze or unfreeze model parameters based on the freeze flag 118 | for param in self.model.base_model.parameters(): 119 | param.requires_grad = not freeze 120 | 121 | print(f"Training with {'frozen' if freeze else 'unfrozen'} feature layers at lr={lr}") 122 | 123 | # Wrap the model with DataParallel 124 | #if torch.cuda.device_count() > 1: 125 | # self.model = nn.DataParallel(self.model) 126 | 127 | for epoch in range(epochs): 128 | # Training loop 129 | self.model.train() 130 | total_accuracy_train = 0 131 | outputs_batch = [] 132 | for batch in self.train_dataloader: 133 | pixel_values, labels = [b.to(self.device) for b in batch] 134 | self.optimizer.zero_grad() 135 | #ipdb.set_trace() 136 | outputs = self.model(pixel_values) 137 | loss=self.criterion(outputs, labels) 138 | loss.backward() 139 | self.optimizer.step() 140 | 141 | accuracy_train = self.calculate_accuracy(outputs, labels) 142 | total_accuracy_train += accuracy_train 143 | # Evaluation loop 144 | avg_accuracy_train = total_accuracy_train / len(self.train_dataloader) 145 | self.model.eval() 146 | total_accuracy = 0 147 | with torch.no_grad(): 148 | for batch in self.test_dataloader: 149 | pixel_values, labels = [b.to(self.device) for b in batch] 150 | outputs = self.model(pixel_values) 151 | 152 | 153 | accuracy = self.calculate_accuracy(outputs, labels) 154 | total_accuracy += accuracy 155 | 156 | logits = outputs.logits if hasattr(outputs, 'logits') else outputs 157 | logits_cpu = logits.detach().cpu().numpy() 158 | outputs_batch.append(logits_cpu) 159 | 160 | if epoch == epochs-1 and not freeze: # we saved test prediction only at last epoch, and finetuning 161 | self.outputs_test = np.concatenate(outputs_batch, axis=0) 162 | 163 | outputs_batch = [] 164 | avg_accuracy = total_accuracy / len(self.test_dataloader) 165 | print(f"Epoch {epoch + 1}, Train Accuracy: {avg_accuracy_train * 100:.2f}%, Test Accuracy: {avg_accuracy * 100:.2f}%") 166 | 167 | 168 | # Example usage 169 | from sklearn.metrics import f1_score 170 | if __name__ == '__main__': 171 | 172 | import pickle 173 | import os 174 | 175 | test_acc_all = list() 176 | test_f1_all = list() 177 | for idx in range(1, 2): 178 | test_acc = [] 179 | torch.cuda.empty_cache() 180 | 181 | direct=r"C:\Users\minho.lee\Dropbox\Projects\EAV\Feature_vision" 182 | file_name = f"subject_{idx:02d}_vis.pkl" 183 | file_ = os.path.join(direct, file_name) 184 | 185 | with open(file_, 'rb') as f: 186 | vis_list2 = pickle.load(f) 187 | tr_x_vis, tr_y_vis, te_x_vis, te_y_vis = vis_list2 188 | 189 | mod_path = r'C:\Users\user.DESKTOP-HI4HHBR\Downloads\facial_emotions_image_detection (1)' 190 | data = [tr_x_vis, tr_y_vis, te_x_vis, te_y_vis] 191 | trainer = ImageClassifierTrainer(data,num_labels=5, lr=5e-5, batch_size=32) 192 | 193 | trainer.train(epochs=3, lr=5e-4, freeze=True) 194 | trainer.train(epochs=3, lr=5e-6, freeze=False) 195 | trainer._delete_dataloader() 196 | test_acc.append(trainer.outputs_test) 197 | 198 | #ipdb.set_trace() 199 | f = open("accuracy_cnn.txt", "a") 200 | f.write("\n Subject ") 201 | f.write(str(idx)) 202 | aa = test_acc[0] 203 | aa2 = np.reshape(aa, (200, 25, 5), 'C') 204 | aa3 = np.mean(aa2, 1) 205 | out1 = np.argmax(aa3, axis = 1) 206 | accuracy = np.mean(out1 == te_y_vis) 207 | test_acc_all.append(accuracy) 208 | f.write("\n") 209 | f.write(f"The accuracy of the {idx}-subject is ") 210 | f.write(str(accuracy)) 211 | print(f"The accuracy of the {idx}-subject is ") 212 | print(accuracy) 213 | f1 = f1_score(te_y_vis, out1, average='weighted') 214 | test_f1_all.append(f1) 215 | f.write("\n") 216 | f.write(f"The f1 score of the {idx}-subject is ") 217 | f.write(str(f1)) 218 | f.close() 219 | print(f"The f1 score of the {idx}-subject is ") 220 | print(f1) 221 | 222 | test_acc_all = np.reshape(np.array(test_acc_all), (42, 1)) 223 | test_f1_all = np.reshape(np.array(test_f1_all), (42, 1)) 224 | -------------------------------------------------------------------------------- /Transformer_torch/Transformer_EEG.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import torch.nn as nn 4 | import torch.optim as optim 5 | import torch.nn.functional as F 6 | from torch.utils.data import DataLoader, TensorDataset, random_split 7 | import os 8 | class PositionalEncoding(nn.Module): 9 | """Positional encoding. 10 | https://d2l.ai/chapter_attention-mechanisms-and-transformers/self-attention-and-positional-encoding.html 11 | """ 12 | def __init__(self, num_hiddens, dropout, max_len=1000): 13 | super().__init__() 14 | self.dropout = nn.Dropout(dropout) 15 | # Create a long enough P 16 | self.p = torch.zeros((1, max_len, num_hiddens)) 17 | x = torch.arange(max_len, dtype=torch.float32).reshape( 18 | -1, 1) / torch.pow(10000, torch.arange( 19 | 0, num_hiddens, 2, dtype=torch.float32) / num_hiddens) 20 | self.p[:, :, 0::2] = torch.sin(x) 21 | self.p[:, :, 1::2] = torch.cos(x) 22 | 23 | def forward(self, x): # note we carefully add the positional encoding, omitted 24 | x = x #+ self.p[:, :x.shape[1], :].to(x.device) 25 | return self.dropout(x) 26 | 27 | class TransformerBlock(nn.Module): 28 | def __init__(self, embed_dim, num_heads, dim_feedforward, dropout=0.1): 29 | super().__init__() 30 | 31 | self.attention = nn.MultiheadAttention( 32 | embed_dim, 33 | num_heads, 34 | dropout, 35 | batch_first=True, 36 | ) 37 | self.mlp = nn.Sequential( 38 | nn.Linear(embed_dim, dim_feedforward), 39 | nn.ReLU(True), 40 | nn.Dropout(dropout), 41 | nn.Linear(dim_feedforward, embed_dim), 42 | ) 43 | 44 | self.layernorm0 = nn.LayerNorm(embed_dim) 45 | self.layernorm1 = nn.LayerNorm(embed_dim) 46 | 47 | self.dropout = dropout 48 | 49 | def forward(self, x): 50 | y, att = self.attention(x, x, x) 51 | y = F.dropout(y, self.dropout, training=self.training) 52 | x = self.layernorm0(x + y) 53 | y = self.mlp(x) 54 | y = F.dropout(y, self.dropout, training=self.training) 55 | x = self.layernorm1(x + y) 56 | return x 57 | 58 | class EEGClassificationModel(nn.Module): 59 | def __init__(self, eeg_channel, dropout=0.1): 60 | super().__init__() 61 | 62 | self.conv = nn.Sequential( 63 | nn.Conv1d( 64 | eeg_channel, eeg_channel, 11, 1, padding=5, bias=False 65 | ), 66 | nn.BatchNorm1d(eeg_channel), 67 | nn.ReLU(True), 68 | nn.Dropout1d(dropout), 69 | nn.Conv1d( 70 | eeg_channel, eeg_channel * 2, 11, 1, padding=5, bias=False 71 | ), 72 | nn.BatchNorm1d(eeg_channel * 2), 73 | ) 74 | 75 | self.transformer = nn.Sequential( 76 | PositionalEncoding(eeg_channel * 2, dropout), 77 | TransformerBlock(eeg_channel * 2, 4, eeg_channel // 8, dropout), 78 | TransformerBlock(eeg_channel * 2, 4, eeg_channel // 8, dropout), 79 | TransformerBlock(eeg_channel * 2, 4, eeg_channel // 8, dropout), 80 | TransformerBlock(eeg_channel * 2, 4, eeg_channel // 8, dropout), 81 | TransformerBlock(eeg_channel * 2, 4, eeg_channel // 8, dropout), 82 | TransformerBlock(eeg_channel * 2, 4, eeg_channel // 8, dropout), 83 | ) 84 | 85 | self.mlp = nn.Sequential( 86 | nn.Linear(eeg_channel * 2, eeg_channel // 2), 87 | nn.ReLU(True), 88 | nn.Dropout(dropout), 89 | nn.Linear(eeg_channel // 2, 5), 90 | ) 91 | 92 | def forward(self, x): 93 | x = self.conv(x) 94 | x = x.permute(0, 2, 1) 95 | x = self.transformer(x) 96 | x = x.permute(0, 2, 1) 97 | x = x.mean(dim=-1) 98 | x = self.mlp(x) 99 | return x 100 | 101 | class EEGModelTrainer: 102 | def __init__(self, DATA, model = [], sub = '', lr=0.001, batch_size = 64): 103 | if model: 104 | self.model = model 105 | else: 106 | self.model = EEGClassificationModel(eeg_channel=30) 107 | 108 | self.tr, self.tr_y, self.te, self.te_y = DATA 109 | self.batch_size = batch_size 110 | self.test_acc = float() 111 | 112 | self.train_dataloader = self._prepare_dataloader(self.tr, self.tr_y, shuffle=True) 113 | self.test_dataloader = self._prepare_dataloader(self.te, self.te_y, shuffle=False) 114 | 115 | self.initial_lr = lr 116 | self.criterion = nn.CrossEntropyLoss() 117 | self.optimizer = optim.Adam(self.model.parameters(), lr=self.initial_lr) 118 | 119 | # Automatically use GPU if available, else fallback to CPU 120 | self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 121 | self.model.to(self.device) 122 | print(self.device) 123 | 124 | def _prepare_dataloader(self, x, y, shuffle=False): 125 | dataset = TensorDataset(torch.tensor(x, dtype=torch.float32), torch.tensor(y, dtype=torch.long)) 126 | dataloader = DataLoader(dataset, batch_size=self.batch_size, shuffle=shuffle) 127 | return dataloader 128 | 129 | def evaluate(self): 130 | self.model.eval() 131 | correct = 0 132 | total = 0 133 | predictions = [] 134 | accuracies = [] 135 | 136 | with torch.no_grad(): 137 | for inputs, labels in self.test_dataloader: 138 | inputs = inputs.to(self.device) 139 | labels = labels.to(self.device) 140 | outputs = self.model(inputs) 141 | _, predicted = torch.max(outputs.data, 1) 142 | total += labels.size(0) 143 | correct += (predicted == labels).sum().item() 144 | predictions.extend(predicted.cpu().numpy()) 145 | accuracies.extend((predicted == labels).cpu().numpy()) 146 | accuracy = correct / total 147 | print(f'Test Accuracy: {accuracy:.2f}') 148 | return accuracy, predictions 149 | 150 | def train(self, epochs=25, lr=None, freeze=False): 151 | lr = lr if lr is not None else self.initial_lr 152 | if lr is not None: 153 | for param_group in self.optimizer.param_groups: 154 | param_group['lr'] = lr 155 | 156 | if isinstance(self.model, nn.DataParallel): 157 | self.model = self.model.module 158 | # Freeze or unfreeze model parameters based on the freeze flag 159 | # we train the eeg model from the scratch 160 | for param in self.model.parameters(): 161 | param.requires_grad = not freeze 162 | 163 | # Wrap the model with DataParallel 164 | if torch.cuda.device_count() > 1: 165 | self.model = nn.DataParallel(self.model) 166 | print("GPU:", torch.cuda.device_count()) 167 | 168 | for epoch in range(epochs): 169 | # Variables to store performance metrics 170 | running_loss = 0.0 171 | correct_predictions = 0 172 | total_predictions = 0 173 | 174 | # Training phase 175 | self.model.train() 176 | for inputs, labels in self.train_dataloader: 177 | inputs, labels = inputs.to(self.device), labels.to(self.device) 178 | self.optimizer.zero_grad() 179 | outputs = self.model(inputs) 180 | loss = self.criterion(outputs, labels) 181 | loss.backward() 182 | self.optimizer.step() 183 | running_loss += loss.item() * inputs.size(0) 184 | 185 | _, predicted = torch.max(outputs.data, 1) 186 | total_predictions += labels.size(0) 187 | correct_predictions += (predicted == labels).sum().item() 188 | 189 | train_loss = running_loss / len(self.train_dataloader.dataset) 190 | train_accuracy = correct_predictions / total_predictions 191 | 192 | # Validation phase 193 | self.model.eval() 194 | running_val_loss = 0.0 195 | val_correct_predictions = 0 196 | val_total_predictions = 0 197 | with torch.no_grad(): 198 | for inputs, labels in self.test_dataloader: 199 | inputs, labels = inputs.to(self.device), labels.to(self.device) 200 | outputs = self.model(inputs) 201 | val_loss = self.criterion(outputs, labels) 202 | running_val_loss += val_loss.item() * inputs.size(0) 203 | 204 | _, predicted = torch.max(outputs.data, 1) 205 | val_total_predictions += labels.size(0) 206 | val_correct_predictions += (predicted == labels).sum().item() 207 | 208 | val_loss = running_val_loss / len(self.test_dataloader.dataset) 209 | val_accuracy = val_correct_predictions / val_total_predictions 210 | 211 | print(f'Epoch {epoch + 1}/{epochs}, Train Loss: {train_loss:.4f}, Train Accuracy: {train_accuracy:.4f}, Val Loss: {val_loss:.4f}, Val Accuracy: {val_accuracy:.4f}') 212 | self.test_acc = val_accuracy 213 | return self.model 214 | 215 | 216 | if __name__ == "__main__": 217 | # Toy dataste 218 | data = torch.randn(1000, 30, 500) 219 | labels = torch.randint(0, 5, (1000,)) 220 | 221 | dataset = TensorDataset(data, labels) 222 | 223 | train_size = int(0.5 * len(dataset)) 224 | val_size = len(dataset) - train_size 225 | train_dataset, val_dataset = random_split(dataset, [train_size, val_size]) 226 | 227 | # Create DataLoaders for training and validation sets 228 | batch_size = 128 229 | train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True) 230 | val_dataloader = DataLoader(val_dataset, batch_size=batch_size) 231 | 232 | model = EEGClassificationModel(eeg_channel=30) # Example: 64 EEG channels 233 | trainer = EEGModelTrainer(model, train_dataloader, val_dataloader) 234 | trainer.train_model(num_epochs=25) 235 | 236 | 237 | 238 | -------------------------------------------------------------------------------- /CNN_tensorflow/CNN_video_emotion_recognition.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "id": "f92db771", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "import os\n", 11 | "import numpy as np\n", 12 | "import cv2\n", 13 | "import tensorflow as tf\n", 14 | "import random\n", 15 | "import numpy as np\n", 16 | "import cv2\n", 17 | "import face_recognition\n", 18 | "from tensorflow.keras.applications import ResNet50\n", 19 | "from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Multiply\n", 20 | "from tensorflow.keras.models import Model\n", 21 | "from tensorflow.keras.utils import to_categorical\n", 22 | "\n", 23 | "from sklearn.metrics import confusion_matrix, f1_score, accuracy_score\n", 24 | "from scipy.stats import mode\n", 25 | "import gc # Import the garbage collection module\n", 26 | "from tensorflow.keras.models import Sequential\n", 27 | "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n", 28 | "import sklearn.metrics as metrics\n", 29 | "IMG_HEIGHT, IMG_WIDTH = 480, 640\n", 30 | "from scipy.stats import mode\n", 31 | "\n", 32 | "NEW_HEIGHT, NEW_WIDTH = IMG_HEIGHT // 2, IMG_WIDTH // 2\n", 33 | "start_row, start_col = (IMG_HEIGHT - NEW_HEIGHT) // 2, (IMG_WIDTH - NEW_WIDTH) // 2\n", 34 | "end_row, end_col = start_row + NEW_HEIGHT, start_col + NEW_WIDTH\n", 35 | "\n", 36 | "class ChannelAttention(tf.keras.layers.Layer):\n", 37 | " def __init__(self, ratio=8):\n", 38 | " super(ChannelAttention, self).__init__()\n", 39 | " self.ratio = ratio\n", 40 | "\n", 41 | " def build(self, input_shape):\n", 42 | " self.shared_layer_one = Dense(input_shape[-1] // self.ratio,\n", 43 | " activation='relu', \n", 44 | " kernel_initializer='he_normal',\n", 45 | " use_bias=True, \n", 46 | " bias_initializer='zeros')\n", 47 | " self.shared_layer_two = Dense(input_shape[-1],\n", 48 | " kernel_initializer='he_normal',\n", 49 | " use_bias=True, \n", 50 | " bias_initializer='zeros')\n", 51 | "\n", 52 | " def call(self, inputs):\n", 53 | " avg_pool = tf.keras.layers.GlobalAveragePooling2D()(inputs)\n", 54 | " max_pool = tf.keras.layers.GlobalMaxPooling2D()(inputs)\n", 55 | "\n", 56 | " avg_pool = self.shared_layer_one(avg_pool)\n", 57 | " max_pool = self.shared_layer_one(max_pool)\n", 58 | "\n", 59 | " avg_pool = self.shared_layer_two(avg_pool)\n", 60 | " max_pool = self.shared_layer_two(max_pool)\n", 61 | "\n", 62 | " return avg_pool + max_pool\n", 63 | "\n", 64 | "def sort_cmp (key):\n", 65 | " ans = 0\n", 66 | " for c in key:\n", 67 | " if (c.isdigit() == True):\n", 68 | " ans = ans * 10 + int (c)\n", 69 | " return ans\n", 70 | "\n", 71 | "acc = []\n", 72 | "fscore = []\n", 73 | "confusion = []\n", 74 | "\n", 75 | "root_folder = \"../Dropbox/EAV\"\n", 76 | "emotions = [\"Anger\", \"Neutral\", \"Sadness\", \"Calmness\", \"Happiness\"]\n", 77 | "IMG_HEIGHT, IMG_WIDTH = 480, 640\n", 78 | "for subfolder in sorted(os.listdir(root_folder), key = sort_cmp):\n", 79 | " subfolder_path = os.path.join(root_folder, subfolder)\n", 80 | " audio_video_folder_path = os.path.join(subfolder_path, \"\")\n", 81 | " video_path = os.path.join(subfolder_path, 'Video')\n", 82 | " data_array = np.zeros(shape=(20000, 224, 224, 3, 1))\n", 83 | " data, labels = [], []\n", 84 | " datav, labelsv = [], []\n", 85 | " if os.path.exists(video_path):\n", 86 | " \n", 87 | " all_files = sorted(os.listdir(video_path))\n", 88 | " categorized_files_v = {emotion: [] for emotion in emotions}\n", 89 | " for file in all_files:\n", 90 | " if \"Speaking\" in file and file.endswith(\".mp4\"):\n", 91 | " for emotion in emotions:\n", 92 | " if emotion in file:\n", 93 | " video_path_cur = os.path.join(video_path, file)\n", 94 | " categorized_files_v[emotion].append(video_path_cur)\n", 95 | " \n", 96 | " print(categorized_files_v[\"Anger\"])\n", 97 | " idx = 0\n", 98 | " for class_index, emotion in enumerate([\"Anger\", \"Neutral\", \"Sadness\", \"Calmness\", \"Happiness\"]):\n", 99 | " for file_index, video_path in enumerate(categorized_files_v[emotion]):\n", 100 | " cap = cv2.VideoCapture(video_path)\n", 101 | " total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n", 102 | " if cap.isOpened():\n", 103 | " frame_index = 1 \n", 104 | " while True: \n", 105 | " ret, frame = cap.read()\n", 106 | " if not ret:\n", 107 | " break\n", 108 | " if frame_index % 3 == 0 and frame_index < 601:\n", 109 | " resizedImg = cv2.resize(frame, (224, 224)).reshape(224, 224, 3, 1) / 255.0\n", 110 | " data_array[idx] = resizedImg \n", 111 | " idx += 1 \n", 112 | " labelsv.append(class_index)\n", 113 | " frame_index += 1\n", 114 | " cap.release()\n", 115 | " else:\n", 116 | " print(f\"Error opening video file: {video_path}\")\n", 117 | " x_train = []\n", 118 | " y_train = []\n", 119 | " x_test = []\n", 120 | " y_test = []\n", 121 | " \n", 122 | " x_trainv = []\n", 123 | " y_trainv = []\n", 124 | " x_testv = []\n", 125 | " y_testv = []\n", 126 | " \n", 127 | " for i, emotion in enumerate (emotions):\n", 128 | " class_start = i * 4000\n", 129 | " class_middle = class_start + 2000\n", 130 | " class_end = class_middle + 2000\n", 131 | "\n", 132 | " x_trainv.extend(data_array[class_start:class_middle])\n", 133 | " x_testv.extend(data_array[class_middle:class_end])\n", 134 | "\n", 135 | " y_trainv.extend(labelsv[class_start:class_middle])\n", 136 | " y_testv.extend(labelsv[class_middle:class_end])\n", 137 | " \n", 138 | " identity_matrix = np.eye(len(emotions))\n", 139 | " train_labels_one_hot = np.array([identity_matrix[label] for label in y_trainv])\n", 140 | " test_labels_one_hot = np.array([identity_matrix[label] for label in y_testv])\n", 141 | " \n", 142 | " x_trainv = np.array(x_trainv)\n", 143 | " x_testv = np.array(x_testv)\n", 144 | " y_trainv = np.array(y_trainv)\n", 145 | " y_testv = np.array(y_testv)\n", 146 | " base_model = ResNet50(weights='imagenet', include_top=False)\n", 147 | " # Add custom layers for 5-class classification\n", 148 | " x = base_model.output\n", 149 | " x = GlobalAveragePooling2D()(x)\n", 150 | " attention = ChannelAttention()(base_model.output)\n", 151 | " x = Multiply()([x, attention])\n", 152 | " x = Dense(1024, activation='relu')(x)\n", 153 | " predictions = Dense(5, activation='softmax')(x)\n", 154 | " model = Model(inputs=base_model.input, outputs=predictions)\n", 155 | "\n", 156 | " # Freeze the layers of the base model (so they don't get trained)\n", 157 | " for layer in base_model.layers:\n", 158 | " layer.trainable = False\n", 159 | "\n", 160 | " # Compile the model\n", 161 | " model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n", 162 | " \n", 163 | " # Train the model\n", 164 | " model.fit(x_trainv, train_labels_one_hot, epochs=100, batch_size=64)\n", 165 | "\n", 166 | " # Evaluate the model (with test data and labels)\n", 167 | " # Replace resized_data and labels with your test data and labels\n", 168 | " test_loss, test_acc = model.evaluate(x_testv, test_labels_one_hot, verbose=2)\n", 169 | " \n", 170 | " print('\\nTest accuracy:', test_acc)\n", 171 | " loss, accuracy = model.evaluate(x_testv, test_labels_one_hot)\n", 172 | " print(f\"Video Accuracy: {accuracy * 100:.2f}%\")\n", 173 | " y_pred = model.predict (x_testv)\n", 174 | " confusion_emotions = [\"Anger\", \"Neutral\", \"Sadness\", \"Calmness\", \"Happiness\"]\n", 175 | " predicted_classes = np.argmax(model.predict(x_testv), axis=1)\n", 176 | " true_classes = np.argmax(test_labels_one_hot, axis=1)\n", 177 | " trials_predicted = predicted_classes.reshape(-1, 50)\n", 178 | " predicted_most_frequent_classes = mode(trials_predicted, axis=1)[0].flatten()\n", 179 | " print(predicted_most_frequent_classes)\n", 180 | " trials_true = true_classes.reshape(-1, 50)\n", 181 | " true_most_frequent_classes = mode(trials_true, axis=1)[0].flatten()\n", 182 | " print(true_most_frequent_classes)\n", 183 | " accur = accuracy_score(true_most_frequent_classes, predicted_most_frequent_classes)\n", 184 | " f1 = f1_score(true_most_frequent_classes, predicted_most_frequent_classes, average='weighted')\n", 185 | " # Confusion Matrix\n", 186 | " cm = confusion_matrix(true_most_frequent_classes, predicted_most_frequent_classes)\n", 187 | " acc.append (accur)\n", 188 | " fscore.append (f1)\n", 189 | " confusion.append (cm)" 190 | ] 191 | } 192 | ], 193 | "metadata": { 194 | "kernelspec": { 195 | "display_name": "latest", 196 | "language": "python", 197 | "name": "latest" 198 | }, 199 | "language_info": { 200 | "codemirror_mode": { 201 | "name": "ipython", 202 | "version": 3 203 | }, 204 | "file_extension": ".py", 205 | "mimetype": "text/x-python", 206 | "name": "python", 207 | "nbconvert_exporter": "python", 208 | "pygments_lexer": "ipython3", 209 | "version": "3.10.0" 210 | } 211 | }, 212 | "nbformat": 4, 213 | "nbformat_minor": 5 214 | } 215 | -------------------------------------------------------------------------------- /CNN_tensorflow/CNN_EEG_tf.py: -------------------------------------------------------------------------------- 1 | import os 2 | import scipy.io 3 | import numpy as np 4 | #from tensorflow.keras.models import Model 5 | from tensorflow.keras.layers import (Dense, Activation, Permute, Dropout, Conv2D, 6 | MaxPooling2D, AveragePooling2D, SeparableConv2D, 7 | DepthwiseConv2D, BatchNormalization, SpatialDropout2D, 8 | Input, Flatten) 9 | from tensorflow.keras.models import Model 10 | from tensorflow.keras.regularizers import l1_l2 11 | from tensorflow.keras.constraints import max_norm 12 | from tensorflow.keras import backend as K 13 | from scipy import signal 14 | from scipy.signal import butter, sosfilt, sosfreqz 15 | from sklearn.metrics import accuracy_score, confusion_matrix, f1_score 16 | from scipy.linalg import eigh 17 | import tensorflow as tf 18 | from scipy.stats import entropy 19 | from sklearn.model_selection import train_test_split 20 | 21 | def EEGNet(nb_classes, Chans=64, Samples=128, 22 | dropoutRate=0.5, kernLength=64, F1=8, 23 | D=2, F2=16, norm_rate=0.25, dropoutType='Dropout'): 24 | if dropoutType == 'SpatialDropout2D': 25 | dropoutType = SpatialDropout2D 26 | elif dropoutType == 'Dropout': 27 | dropoutType = Dropout 28 | else: 29 | raise ValueError('dropoutType must be one of SpatialDropout2D ' 30 | 'or Dropout, passed as a string.') 31 | 32 | input1 = Input(shape=(Chans, Samples, 1)) 33 | 34 | ################################################################## 35 | block1 = Conv2D(F1, (1, kernLength), padding='same', 36 | input_shape=(Chans, Samples, 1), 37 | use_bias=False)(input1) 38 | block1 = BatchNormalization()(block1) 39 | block1 = DepthwiseConv2D((Chans, 1), use_bias=False, 40 | depth_multiplier=D, 41 | depthwise_constraint=max_norm(1.))(block1) 42 | block1 = BatchNormalization()(block1) 43 | block1 = Activation('elu')(block1) 44 | block1 = AveragePooling2D((1, 4))(block1) 45 | block1 = dropoutType(dropoutRate)(block1) 46 | 47 | block2 = SeparableConv2D(F2, (1, 16), 48 | use_bias=False, padding='same')(block1) 49 | block2 = BatchNormalization()(block2) 50 | block2 = Activation('elu')(block2) 51 | block2 = AveragePooling2D((1, 8))(block2) 52 | block2 = dropoutType(dropoutRate)(block2) 53 | 54 | flatten = Flatten(name='flatten')(block2) 55 | 56 | dense = Dense(nb_classes, name='dense', 57 | kernel_constraint=max_norm(norm_rate))(flatten) 58 | softmax = Activation('softmax', name='softmax')(dense) 59 | 60 | return Model(inputs=input1, outputs=softmax) 61 | 62 | # input = (10000, 30, 200) 63 | # out = (10000, 30, 200) 64 | def Bandpass(dat, freq=[5, 80], fs=500): 65 | [D, Ch, Tri] = dat.shape 66 | dat2 = np.transpose(dat, [0, 2, 1]) 67 | dat3 = np.reshape(dat2, [10000 * 200, 30], order='F') 68 | 69 | sos = butter(5, freq, 'band', fs=fs, output='sos') 70 | fdat = list() 71 | for i in range(np.size(dat3, 1)): 72 | tm = signal.sosfilt(sos, dat3[:, i]) 73 | fdat.append(tm) 74 | dat4 = np.array(fdat).transpose().reshape((D, Tri, Ch), order='F').transpose(0, 2, 1) 75 | return dat4 76 | # no need this function anymore 77 | def seg(dat, marker, ival=[0, 2000]): 78 | sdat = list() 79 | for i in range(np.size(marker, 1)): 80 | lag = range(marker[0, i] + ival[0], marker[0, i] + ival[1]) 81 | sdat.append(dat[lag, :]) 82 | return np.array(sdat) 83 | # data = (200, 2000, 30), label = (10, 160) 84 | def mysplit(data, label): 85 | # Original data and labels 86 | # data = np.random.rand(200, 30, 2000) # Replace with your actual data 87 | # labels = np.random.randint(0, 2, size=(160, 10)) # Replace with your actual labels (one-hot vectors) 88 | 89 | # Splitting parameters 90 | split_length = 500 # Length of each split 91 | num_splits = data.shape[1] // split_length # Number of splits 92 | 93 | a1 = np.transpose(data, [1, 0, 2]) 94 | a2 = np.reshape(a1, [500, 4, 200, 30], order='F') 95 | a3 = np.reshape(a2, [500, 4 * 200, 30], order='F') 96 | a4 = np.transpose(a3, [1, 0, 2]) 97 | 98 | labels_repeated = np.repeat(label, repeats=4, axis=1) 99 | 100 | return a4, labels_repeated 101 | 102 | result_conf = list() 103 | result_acc = list() 104 | result_f1 = list() 105 | 106 | for sub in range(1, 43): 107 | file_path = "C:/Users/minho.lee/Dropbox/Datasets/EAV/Input_images/EEG/" 108 | file_name = f"subject_{sub:02d}_eeg.pkl" 109 | file_ = os.path.join(file_path, file_name) 110 | import pickle 111 | with open(file_, 'rb') as f: 112 | eeg_list2 = pickle.load(f) 113 | tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg = eeg_list2 114 | data = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 115 | 116 | model = EEGNet(nb_classes=5, D=8, F2=64, Chans=30, kernLength=300, Samples=500, 117 | dropoutRate=0.5) 118 | 119 | model.compile(loss='categorical_crossentropy', optimizer='adam', 120 | metrics=['accuracy']) 121 | 122 | y_train = np.zeros((tr_y_eeg.shape[0], 5)) 123 | y_train[np.arange(tr_y_eeg.shape[0]), tr_y_eeg.flatten()] = 1 124 | y_test = np.zeros((te_y_eeg.shape[0], 5)) 125 | y_test[np.arange(te_y_eeg.shape[0]), te_y_eeg.flatten()] = 1 126 | x_train = np.reshape(tr_x_eeg, (280, 30, 500, 1)) 127 | x_test = np.reshape(te_x_eeg, (120, 30, 500, 1)) 128 | model.fit(x_train, y_train, batch_size=32, epochs=200, shuffle=True, validation_data=(x_test, y_test)) 129 | 130 | pred = model.predict(x_test) 131 | pred = np.argmax(pred, axis=1) 132 | 133 | y_test2 = np.argmax(y_test, axis=1) 134 | 135 | cm = confusion_matrix(pred, y_test2) 136 | 137 | accuracy = accuracy_score(pred, y_test2) 138 | f1 = f1_score(y_test2, pred, average='weighted') # 'weighted' for multiclass F1-Score 139 | 140 | result_conf.append(cm) 141 | result_acc.append(accuracy) 142 | result_f1.append(f1) # Append the F1-Score to your result list 143 | print(result_acc) 144 | 145 | result_conf_np = np.array(result_conf) 146 | summed_confusion_matrix = np.sum(result_conf_np, axis=0) 147 | summed_confusion_matrix_T = summed_confusion_matrix.T 148 | 149 | ''' 150 | 151 | 152 | for subject in sorted_subject_folders: 153 | # Remove trailing '__' if present 154 | # subject = subject.rstrip('__') 155 | cnt_ = []; 156 | # Construct the full path to the EEG file 157 | eeg_folder = os.path.join(parent_directory, subject, 'EEG') 158 | eeg_file_name = subject.rstrip('__') + '_eeg.mat' 159 | eeg_file_path = os.path.join(eeg_folder, eeg_file_name) 160 | 161 | label_file_name = subject.rstrip('__') + '_eeg_label.mat' 162 | label_file_path = os.path.join(eeg_folder, label_file_name) 163 | 164 | # Check if the EEG file exists 165 | if os.path.exists(eeg_file_path): 166 | mat = scipy.io.loadmat(eeg_file_path) 167 | cnt_ = np.array(mat.get('seg1')) 168 | if np.ndim(cnt_) == 3: 169 | cnt_ = np.array(mat.get('seg1')) 170 | else: 171 | cnt_ = np.array(mat.get('seg')) 172 | 173 | mat_Y = scipy.io.loadmat(label_file_path) 174 | Label = np.array(mat_Y.get('label')) 175 | 176 | print(f'Loaded EEG data for {subject}') 177 | else: 178 | print(f'EEG data not found for {subject}') 179 | 180 | cnt_f = Bandpass(cnt_, freq=[3, 50], fs=500) 181 | 182 | fs_original = 500 # Original sampling rate in Hz 183 | fs_target = 100 # Target sampling rate in Hz 184 | 185 | tm = np.transpose(cnt_f, [0, 2, 1]).reshape([10000 * 200, 30], order='F') 186 | 187 | downsampling_factor = fs_target / fs_original 188 | tm2 = signal.resample_poly(tm, up=1, down=int(fs_original / fs_target), axis=0) 189 | cnt_f2 = np.reshape(tm2, [2000, 200, 30], order='F') 190 | 191 | cnt_seg = np.transpose(cnt_f2, [1, 0, 2]) 192 | 193 | num_trials_per_class = np.sum(Label, axis=1) # check the balance. 194 | [cnt_seg_split, Label_split] = mysplit(cnt_seg, Label) 195 | 196 | dat = np.transpose(cnt_seg_split, (0, 2, 1)).reshape( 197 | (800, 30, 500, 1)) # This should be (800, 30, 500, 1) for balanced classes 198 | 199 | selected_classes = [1, 3, 5, 7, 9] # only listening classes 200 | 201 | selected_indices = np.isin(np.argmax(Label_split, axis=0), selected_classes) 202 | 203 | data_5class = dat[selected_indices] 204 | 205 | aa = Label_split[:, selected_indices] 206 | label_5class = aa[selected_classes, :] 207 | 208 | # x_train, x_test, y_train, y_test = train_test_split(data_5class, label_5class.T, test_size=0.5, random_state=42, stratify=label_5class.T) 209 | 210 | # Lists to collect train and test subsets 211 | x_train_list = [] 212 | x_test_list = [] 213 | y_train_list = [] 214 | y_test_list = [] 215 | 216 | for i in range(5): # Looping over each class 217 | class_indices = np.where(label_5class.T[:, i] == 1)[0] # Find indices where current class label is 1 218 | midpoint = len(class_indices) // 2 # Calculate the midpoint for 50% split 219 | 220 | # this random shuffle will decide "random order" or "sequential order in time" 221 | # np.random.shuffle(class_indices) # Shuffle the indices randomly 222 | # Split data based on found indices 223 | x_train_list.append(data_5class[class_indices[:midpoint]]) 224 | x_test_list.append(data_5class[class_indices[midpoint:]]) 225 | 226 | y_train_list.append(label_5class.T[class_indices[:midpoint]]) 227 | y_test_list.append(label_5class.T[class_indices[midpoint:]]) 228 | 229 | # Convert lists to numpy arrays 230 | x_train = np.concatenate(x_train_list, axis=0) 231 | x_test = np.concatenate(x_test_list, axis=0) 232 | y_train = np.concatenate(y_train_list, axis=0) 233 | y_test = np.concatenate(y_test_list, axis=0) 234 | 235 | model = EEGNet(nb_classes=5, D=8, F2=64, Chans=30, kernLength=300, Samples=500, 236 | dropoutRate=0.5) 237 | 238 | model.compile(loss='categorical_crossentropy', optimizer='adam', 239 | metrics=['accuracy']) 240 | 241 | model.fit(x_train, y_train, batch_size=32, epochs=100, shuffle=True, validation_data=(x_test, y_test)) 242 | 243 | pred = model.predict(x_test) 244 | pred = np.argmax(pred, axis=1) 245 | 246 | y_test2 = np.argmax(y_test, axis=1) 247 | 248 | cm = confusion_matrix(pred, y_test2) 249 | 250 | accuracy = accuracy_score(pred, y_test2) 251 | f1 = f1_score(y_test2, pred, average='weighted') # 'weighted' for multiclass F1-Score 252 | 253 | result_conf.append(cm) 254 | result_acc.append(accuracy) 255 | result_f1.append(f1) # Append the F1-Score to your result list 256 | print(result_acc) 257 | 258 | result_conf_np = np.array(result_conf) 259 | summed_confusion_matrix = np.sum(result_conf_np, axis=0) 260 | summed_confusion_matrix_T = summed_confusion_matrix.T 261 | ''' -------------------------------------------------------------------------------- /Dataload_eeg.py: -------------------------------------------------------------------------------- 1 | import os 2 | import scipy.io 3 | from scipy.signal import butter 4 | from scipy import signal 5 | 6 | from EAV_datasplit import * 7 | 8 | ''' 9 | NEU_SPE = 108, 0 10 | S_SPE = 1 11 | A_SPE = 2 12 | H_SPE = 3 13 | R_SPE = 4 ##### 14 | ''' 15 | 16 | class DataLoadEEG: 17 | def __init__(self, subject='all', band=[0.3, 50], fs_orig=500, fs_target=100, 18 | parent_directory=r'C:\Users\minho.lee\Dropbox\Datasets\EAV'): 19 | self.subject = subject 20 | self.band = band 21 | self.parent_directory = parent_directory 22 | self.fs_orig = fs_orig 23 | self.fs_target = fs_target 24 | self.seg = [] 25 | self.label = [] 26 | self.label_div = [] 27 | self.seg_f = [] 28 | self.seg_f_div = [] 29 | 30 | def data_mat(self): 31 | subject = f'subject{self.subject:02d}' 32 | eeg_folder = os.path.join(self.parent_directory, subject, 'EEG') 33 | eeg_file_name = subject.rstrip('__') + '_eeg.mat' 34 | eeg_file_path = os.path.join(eeg_folder, eeg_file_name) 35 | 36 | label_file_name = subject.rstrip('__') + '_eeg_label.mat' 37 | label_file_path = os.path.join(eeg_folder, label_file_name) 38 | 39 | if os.path.exists(eeg_file_path): 40 | mat = scipy.io.loadmat(eeg_file_path) 41 | cnt_ = np.array(mat.get('seg1')) 42 | if np.ndim(cnt_) == 3: 43 | cnt_ = np.array(mat.get('seg1')) 44 | else: 45 | cnt_ = np.array(mat.get('seg')) 46 | 47 | mat_y = scipy.io.loadmat(label_file_path) 48 | label = np.array(mat_y.get('label')) 49 | 50 | self.seg = np.transpose(cnt_, [1, 0, 2]) # (10000, 30, 200) -> (30ch, 10000t, 200trial) 51 | self.label = label 52 | 53 | print(f'Loaded EEG data for {subject}') 54 | else: 55 | print(f'EEG data not found for {subject}') 56 | 57 | def downsampling(self, fs_target=100): 58 | [ch, t, tri] = self.seg.shape 59 | factor = fs_target / self.fs_orig 60 | tm = np.reshape(self.seg, [ch, t * tri], order='F') 61 | tm2 = signal.resample_poly(tm, up=1, down=int(self.fs_orig / fs_target), axis=1) 62 | self.seg = np.reshape(tm2, [ch, int(t * factor), tri], order='F') 63 | 64 | def bandpass(self): 65 | [ch, t, tri] = self.seg.shape 66 | dat = np.reshape(self.seg, [ch, t * tri], order='F') 67 | # bandpass after the downsample -> fs_target 68 | sos = butter(5, self.band, btype='bandpass', fs=self.fs_target, output='sos') 69 | fdat = list() 70 | for i in range(np.size(dat, 0)): 71 | tm = signal.sosfilt(sos, dat[i, :]) 72 | fdat.append(tm) 73 | self.seg_f = np.array(fdat).reshape((ch, t, tri), order='F') 74 | 75 | def data_div(self): 76 | # Here 2000 (20seconds) are divided into 4 splits 77 | [ch, t, tri] = self.seg_f.shape 78 | tm1 = self.seg_f.reshape((30, 500, 4, 200), order='F') 79 | self.seg_f_div = tm1.reshape((30, 500, 4 * 200), order='F') 80 | self.label_div = np.repeat(self.label, repeats=4, axis=1) 81 | 82 | # Here we only select the listening classes 83 | selected_classes = [1, 3, 5, 7, 9] 84 | label = self.label_div[selected_classes, :] 85 | selected_indices = np.isin(np.argmax(self.label_div, axis=0), selected_classes) 86 | label = label[:, selected_indices] 87 | x = self.seg_f_div[:, :, selected_indices] 88 | 89 | 90 | self.seg_f_div = np.transpose(x, (2, 0, 1)) # (30, 500, 400) -> (400, 30, 500) 91 | class_indices = np.argmax(label, axis=0) 92 | 93 | #self.label_div = label 94 | self.label_div = class_indices 95 | 96 | def data_split(self): 97 | selected_classes = [1, 3, 5, 7, 9] # only listening classes 98 | label = self.label_div[selected_classes, :] 99 | 100 | selected_indices = np.isin(np.argmax(self.label_div, axis=0), selected_classes) 101 | label = label[:, selected_indices] 102 | 103 | x = self.seg_f_div[:, :, selected_indices] 104 | x_train_list = [] 105 | x_test_list = [] 106 | y_train_list = [] 107 | y_test_list = [] 108 | 109 | for i in range(5): # Looping over each class 110 | class_indices = np.where(label.T[:, i] == 1)[0] # Find indices where current class label is 1 111 | midpoint = len(class_indices) // 2 # Calculate the midpoint for 50% split 112 | 113 | # Split data based on found indices 114 | x_train_list.append(x[:, :, class_indices[:midpoint]]) 115 | x_test_list.append(x[:, :, class_indices[midpoint:]]) 116 | 117 | y_train_list.append(label.T[class_indices[:midpoint]]) 118 | y_test_list.append(label.T[class_indices[midpoint:]]) 119 | 120 | # Convert lists to numpy arrays 121 | x_train = np.concatenate(x_train_list, axis=0) 122 | x_test = np.concatenate(x_test_list, axis=0) 123 | y_train = np.concatenate(y_train_list, axis=0) 124 | y_test = np.concatenate(y_test_list, axis=0) 125 | 126 | def data_prepare(self): 127 | self.data_mat() 128 | self.downsampling() 129 | self.bandpass() 130 | self.data_div() 131 | return self.seg_f_div, self.label_div 132 | 133 | 134 | 135 | #create eeg pickle files 136 | if __name__ == "__main__": 137 | for sub in range(1,43): 138 | print(sub) 139 | file_path = "C:/Users/minho.lee/Dropbox/Datasets/EAV/Input_images/EEG/" 140 | file_name = f"subject_{sub:02d}_eeg.pkl" 141 | file_ = os.path.join(file_path, file_name) 142 | eeg_loader = DataLoadEEG(subject=sub, band=[0.5, 45], fs_orig=500, fs_target=100, 143 | parent_directory='C://Users//minho.lee//Dropbox//Datasets//EAV') 144 | data_eeg, data_eeg_y = eeg_loader.data_prepare() 145 | 146 | division_eeg = EAVDataSplit(data_eeg, data_eeg_y) 147 | [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] = division_eeg.get_split(h_idx=56) 148 | EEG_list = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 149 | 150 | ''' 151 | # Here you can write / load vision features tr:{280}(30, 500), te:{120}(30, 500) 152 | import pickle 153 | with open(file_, 'wb') as f: 154 | pickle.dump(EEG_list, f) 155 | 156 | # You can directly work from here 157 | with open(file_, 'rb') as f: 158 | eeg_list = pickle.load(f) 159 | tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg = eeg_list 160 | data = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 161 | ''' 162 | data = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 163 | 164 | # Transformer for EEG 165 | from Transformer_torch import Transformer_EEG 166 | model = Transformer_EEG.EEGClassificationModel(eeg_channel=30) 167 | trainer = Transformer_EEG.EEGModelTrainer(data, model = model, lr=0.001, batch_size = 64) 168 | trainer.train(epochs=100, lr=None, freeze=False) 169 | 170 | [accuracy, predictions] = trainer.evaluate() 171 | 172 | # CNN_tensorflow for EEG 173 | from CNN_tensorflow.CNN_EEG_tf import EEGNet 174 | from sklearn.metrics import accuracy_score, confusion_matrix 175 | 176 | model = EEGNet(nb_classes=5, D=8, F2=64, Chans=30, kernLength=300, Samples=500, 177 | dropoutRate=0.5) 178 | model.compile(loss='categorical_crossentropy', optimizer='adam', 179 | metrics=['accuracy']) 180 | y_train = np.zeros((tr_y_eeg.shape[0], 5)) 181 | y_train[np.arange(tr_y_eeg.shape[0]), tr_y_eeg.flatten()] = 1 182 | y_test = np.zeros((te_y_eeg.shape[0], 5)) 183 | y_test[np.arange(te_y_eeg.shape[0]), te_y_eeg.flatten()] = 1 184 | x_train = np.reshape(tr_x_eeg, (280, 30, 500, 1)) 185 | x_test = np.reshape(te_x_eeg, (120, 30, 500, 1)) 186 | model.fit(x_train, y_train, batch_size=32, epochs=200, shuffle=True, validation_data=(x_test, y_test)) 187 | 188 | pred = model.predict(x_test) 189 | pred = np.argmax(pred, axis=1) 190 | 191 | y_test2 = np.argmax(y_test, axis=1) 192 | cm = confusion_matrix(pred, y_test2) 193 | accuracy = accuracy_score(pred, y_test2) 194 | 195 | # CNN_pytorch for EEG, fix the error, and make the accuracy same 196 | from CNN_torch.EEGNet_tor import EEGNet_tor, Trainer_uni 197 | import torch.nn as nn 198 | 199 | model = EEGNet_tor(nb_classes=5, D=8, F2=64, Chans=30, kernLength=300, Samples=500, 200 | dropoutRate=0.5) 201 | trainer = Trainer_uni(model=model, data=data, lr=1e-5, batch_size=32, num_epochs=200) 202 | trainer.train() 203 | model.eval() 204 | 205 | criterion = nn.CrossEntropyLoss() 206 | device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 207 | te_x_eeg = torch.tensor(te_x_eeg, dtype=torch.float32).to(device) 208 | te_y_eeg = torch.tensor(te_y_eeg, dtype=torch.long).to(device) 209 | model.to(device) 210 | 211 | with torch.no_grad(): 212 | scores = model(te_x_eeg) 213 | predictions = scores.argmax(dim=1) 214 | correct = (predictions == te_y_eeg).sum().item() 215 | total = te_y_eeg.size(0) 216 | accuracy = correct / total 217 | print(accuracy) 218 | 219 | 220 | 221 | 222 | ''' Direct evaluation 223 | if __name__ == "__main__": 224 | eeg_loader = DataLoadEEG(subject=1, band=[0.5, 45], fs_orig=500, fs_target=100, 225 | parent_directory='C://Users//minho.lee//Dropbox//Datasets//EAV') 226 | data_eeg, data_eeg_y = eeg_loader.data_prepare() 227 | 228 | division_eeg = EAVDataSplit(data_eeg, data_eeg_y) 229 | [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] = division_eeg.get_split() 230 | data = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 231 | 232 | trainer = Transformer_EEG.EEGModelTrainer(data, lr=0.001, batch_size = 64) 233 | trainer.train(epochs=200, lr=None, freeze=False) 234 | ''' 235 | ''' 236 | from Transformer_EEG import EEGClassificationModel 237 | accuracy_all = list() 238 | prediction_all = list() 239 | if __name__ == "__main__": # from pickle data 240 | import pickle 241 | for sub in range(1, 43): 242 | file_path = "C:/Users/minho.lee/Dropbox/Datasets/EAV/Input_images/EEG/" 243 | file_name = f"subject_{sub:02d}_eeg.pkl" 244 | file_ = os.path.join(file_path, file_name) 245 | 246 | with open(file_, 'rb') as f: 247 | eeg_list2 = pickle.load(f) 248 | tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg = eeg_list2 249 | data = [tr_x_eeg, tr_y_eeg, te_x_eeg, te_y_eeg] 250 | 251 | model = EEGClassificationModel(eeg_channel=30) 252 | trainer = Transformer_EEG.EEGModelTrainer(data, model = model, lr=0.001, batch_size = 64) 253 | trainer.train(epochs=100, lr=None, freeze=False) 254 | 255 | [accuracy, predictions] = trainer.evaluate() 256 | accuracy_all.append(accuracy) 257 | prediction_all.append(predictions) 258 | ''' 259 | 260 | -------------------------------------------------------------------------------- /CNN_tensorflow/CNN_audio_emotion_recognition.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "id": "9bd7a220", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "import os\n", 11 | "import numpy as np\n", 12 | "import cv2\n", 13 | "import tensorflow as tf\n", 14 | "import random\n", 15 | "from sklearn.metrics import confusion_matrix, f1_score, accuracy_score\n", 16 | "from scipy.stats import mode\n", 17 | "import gc # Import the garbage collection module\n", 18 | "from tensorflow.keras.models import Sequential\n", 19 | "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n", 20 | "import librosa\n", 21 | "import soundfile\n", 22 | "import os, glob, pickle\n", 23 | "import numpy as np\n", 24 | "from sklearn.model_selection import train_test_split\n", 25 | "from sklearn.metrics import accuracy_score,classification_report, confusion_matrix, f1_score\n", 26 | "import pickle\n", 27 | "from tensorflow.keras import layers, Sequential\n", 28 | "from tensorflow.keras.layers import Conv1D, Activation, Dropout, Dense, Flatten, MaxPooling1D\n", 29 | "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n", 30 | "from tensorflow.keras import regularizers\n", 31 | "from matplotlib import pyplot as plt\n", 32 | "import seaborn as sn\n", 33 | "import pandas as pd\n", 34 | "import sklearn.metrics as metrics\n", 35 | "import keras\n", 36 | "\n", 37 | "def sort_cmp (key):\n", 38 | " ans = 0\n", 39 | " for c in key:\n", 40 | " if (c.isdigit() == True):\n", 41 | " ans = ans * 10 + int (c)\n", 42 | " return ans\n", 43 | "\n", 44 | "def load_test_data(dir_rec, save=False):\n", 45 | " x, y = [], [] \n", 46 | " print (dir_rec)\n", 47 | " for file in sorted(glob.glob(dir_rec + \"/*.wav\")):\n", 48 | " # load an audio file as a floating point time series. \n", 49 | " data, sr = librosa.load(file)\n", 50 | " #print ('shape:', data.shape, ' ', sr)\n", 51 | " # extract features from audio files into numpy array\n", 52 | " #feature = extract_feature(data, sr, mfcc=True/*, chroma=True, mel=True)\n", 53 | " seg = np.array (divide_into_segments (data, sr))\n", 54 | " i = 0\n", 55 | " for s in seg:\n", 56 | " feature = extract_feature(s, sr, mfcc=True, chroma=True, mel=True)\n", 57 | " x.append(feature)\n", 58 | " #x_orig.append (s)\n", 59 | " i+= 1\n", 60 | " if (i == 5):\n", 61 | " break\n", 62 | " file_name = os.path.basename(file)\n", 63 | " #print (file_name)\n", 64 | " if (file_name.count ('Neutral') > 0):\n", 65 | " emotion = file_name.split ('.')[0].split ('_')[4]\n", 66 | " else:\n", 67 | " emotion = file_name.split ('.')[0].split ('_')[4]\n", 68 | " \n", 69 | " if (emotion == 'Calmness'):\n", 70 | " emotion = 'Calmness' \n", 71 | " # get emotion label from the file name\n", 72 | " i = 0\n", 73 | " for s in seg:\n", 74 | " y.append(emotions.index (emotion))\n", 75 | " i += 1\n", 76 | " if (i == 5):\n", 77 | " break\n", 78 | " \n", 79 | " if save==True:\n", 80 | " np.save('X', np.array(x))\n", 81 | " np.save('y', y)\n", 82 | " \n", 83 | " return np.array(x), y\n", 84 | "def divide_into_segments (data, sr, seg_dur_seconds = 5):\n", 85 | " segment_length = sr * seg_dur_seconds\n", 86 | " num_sections = int(np.floor(len(data) / segment_length))\n", 87 | " split = []\n", 88 | " for i in range(num_sections):\n", 89 | " t = data[i * segment_length: (i + 1) * segment_length]\n", 90 | " split.append(t)\n", 91 | " return split \n", 92 | "\n", 93 | "def extract_feature(data, sr, mfcc, chroma, mel):\n", 94 | " if chroma: \n", 95 | " stft = np.abs(librosa.stft(data)) \n", 96 | " result = np.array([])\n", 97 | " if mfcc: \n", 98 | " mfccs = np.mean(librosa.feature.mfcc(y=data, sr=sr, n_mfcc=40).T, axis=0)\n", 99 | " result = np.hstack((result, mfccs))\n", 100 | " if chroma: \n", 101 | " chroma = np.mean(librosa.feature.chroma_stft(S=stft, sr=sr).T,axis=0)\n", 102 | " result = np.hstack((result, chroma))\n", 103 | " \n", 104 | " if mel: \n", 105 | " mel = np.mean(librosa.feature.melspectrogram(y=data, sr=sr).T,axis=0)\n", 106 | " result = np.hstack((result, mel))\n", 107 | " \n", 108 | " return result \n", 109 | "\n", 110 | "def proc_data (file, emotion):\n", 111 | " x = []\n", 112 | " y = []\n", 113 | " data, sr = librosa.load(file)\n", 114 | " seg = np.array (divide_into_segments (data, sr))\n", 115 | " i = 0\n", 116 | " for s in seg:\n", 117 | " feature = extract_feature(s, sr, mfcc=True, chroma=True, mel=True)\n", 118 | " x.append(feature)\n", 119 | " i+= 1\n", 120 | " if (i == 5):\n", 121 | " break\n", 122 | " i = 0\n", 123 | " for s in seg:\n", 124 | " y.append(emotions.index (emotion))\n", 125 | " i += 1\n", 126 | " if (i == 5):\n", 127 | " break\n", 128 | " return x, y\n", 129 | "\n", 130 | "def extract_index_video(filename):\n", 131 | " key = (int (filename.split ('/')[-1].split('_')[1]), int (filename.split ('/')[-1].split('_')[5]))\n", 132 | " return key\n", 133 | "\n", 134 | "def extract_index_audio(filename):\n", 135 | " if (filename.count ('neutral') > 0):\n", 136 | " return (int (filename.split ('/')[-1].split('_')[1]), 0)\n", 137 | " \n", 138 | " key = (int (filename.split ('/')[-1].split('_')[1]), int (filename.split ('/')[-1].split('_')[3]))\n", 139 | " \n", 140 | " return key\n", 141 | "\n", 142 | "acc = []\n", 143 | "fscore = []\n", 144 | "confusion = []\n", 145 | "\n", 146 | "root_folder = \"../Dropbox/EAV\"\n", 147 | "\n", 148 | "emotions = [\"Anger\", \"Neutral\", \"Sadness\", \"Calmness\", \"Happiness\"]\n", 149 | "for subfolder in sorted(os.listdir(root_folder), key = sort_cmp):\n", 150 | " print(subfolder)\n", 151 | " categorized_files = {emotion: [] for emotion in emotions}\n", 152 | " subfolder_path = os.path.join(root_folder, subfolder)\n", 153 | " audio_path = os.path.join(subfolder_path, 'Audio')\n", 154 | " data, labels = [], []\n", 155 | " if os.path.exists(audio_path):\n", 156 | " all_files = sorted(os.listdir(audio_path))\n", 157 | " for file in all_files:\n", 158 | " if file.endswith(\".wav\"):\n", 159 | " for emotion in emotions:\n", 160 | " if emotion in file:\n", 161 | " if (emotion != 'Neutral' and file.count ('Neutral') > 0):\n", 162 | " continue\n", 163 | " audio_path2 = os.path.join(audio_path, file)\n", 164 | " categorized_files[emotion].append(audio_path2)\n", 165 | " \n", 166 | " X, y = load_test_data(audio_path)\n", 167 | " idx = 0\n", 168 | " \n", 169 | " for class_index, emotion in enumerate(emotions):\n", 170 | " for file_index, audio_path2 in enumerate(categorized_files[emotion]):\n", 171 | " x,y = proc_data (audio_path2, emotion) \n", 172 | " data.extend(x)\n", 173 | " labels.extend(y)\n", 174 | " idx+=1\n", 175 | "\n", 176 | " x_train = []\n", 177 | " y_train = []\n", 178 | " x_test = []\n", 179 | " y_test = []\n", 180 | " \n", 181 | " for i, emotion in enumerate (emotions):\n", 182 | " x_train.extend (data [i*80: i*80 + 40])\n", 183 | " y_train.extend (labels [i*80: i*80 + 40])\n", 184 | " x_test.extend (data [i*80 + 40: i*80 + 80])\n", 185 | " y_test.extend (labels [i*80 + 40: i*80 + 80])\n", 186 | " \n", 187 | " x_train = np.array(x_train)\n", 188 | " x_test = np.array(x_test)\n", 189 | " y_train = np.array(y_train)\n", 190 | " y_test = np.array(y_test)\n", 191 | " x_train = np.expand_dims(x_train, axis=2)\n", 192 | " x_test = np.expand_dims(x_test, axis=2)\n", 193 | " \n", 194 | " identity_matrix = np.eye(len(emotions))\n", 195 | " train_labels_one_hot = np.array([identity_matrix[label] for label in y_train])\n", 196 | " test_labels_one_hot = np.array([identity_matrix[label] for label in y_test])\n", 197 | " \n", 198 | " audio_model = Sequential()\n", 199 | " audio_model.add(Conv1D(256, 5,padding='same', input_shape=(180,1))) # 1st layer\n", 200 | " audio_model.add(Activation('relu'))\n", 201 | " audio_model.add(Conv1D(128, 5,padding='same', kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4))) # 2nd layer\n", 202 | " audio_model.add(Activation('relu'))\n", 203 | " audio_model.add(Dropout(0.1))\n", 204 | " audio_model.add(MaxPooling1D(pool_size=(8)))\n", 205 | " audio_model.add(Conv1D(128, 5,padding='same', kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4))) # 3rd layer\n", 206 | " audio_model.add(Activation('relu'))\n", 207 | " audio_model.add(Conv1D(128, 5,padding='same', kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4))) # 4th layer\n", 208 | " audio_model.add(Activation('relu'))\n", 209 | " audio_model.add(Dropout(0.5))\n", 210 | " audio_model.add(Flatten())\n", 211 | " audio_model.add(Dense(units=5,\n", 212 | " kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),\n", 213 | " bias_regularizer=regularizers.l2(1e-4),\n", 214 | " activity_regularizer=regularizers.l2(1e-5)\n", 215 | " )\n", 216 | " ) # 7th layer\n", 217 | " audio_model.add(Activation('softmax'))\n", 218 | " audio_model.compile(optimizer=tf.keras.optimizers.Adam(), loss='categorical_crossentropy', metrics=['accuracy'])\n", 219 | " audio_model.fit(x_train, train_labels_one_hot, validation_data=(x_test, test_labels_one_hot), epochs=100, batch_size=64, verbose = False)\n", 220 | " loss, accuracy = audio_model.evaluate(x_test, test_labels_one_hot)\n", 221 | " y_pred = audio_model.predict (x_test)\n", 222 | " f1 = f1_score(y_test,np.argmax(y_pred, axis=-1),average='weighted')\n", 223 | " acc.append (accuracy)\n", 224 | " fscore.append (f1)\n", 225 | " confusion_emotions = [\"Anger\", \"Neutral\", \"Sadness\", \"Calmness\", \"Happiness\"]\n", 226 | " cm = metrics.confusion_matrix(y_test, np.argmax(y_pred, axis=-1))\n", 227 | " confusion.append (cm)\n", 228 | " print(f\"Audio Accuracy for {subfolder} : {accuracy * 100:.2f}%\")\n", 229 | " print ('F1: ', f1)\n", 230 | " print ('Confusion:', cm)" 231 | ] 232 | } 233 | ], 234 | "metadata": { 235 | "kernelspec": { 236 | "display_name": "latest", 237 | "language": "python", 238 | "name": "latest" 239 | }, 240 | "language_info": { 241 | "codemirror_mode": { 242 | "name": "ipython", 243 | "version": 3 244 | }, 245 | "file_extension": ".py", 246 | "mimetype": "text/x-python", 247 | "name": "python", 248 | "nbconvert_exporter": "python", 249 | "pygments_lexer": "ipython3", 250 | "version": "3.10.0" 251 | } 252 | }, 253 | "nbformat": 4, 254 | "nbformat_minor": 5 255 | } 256 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts 3 | 4 | We introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography 5 | (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, 6 | eliciting five distinct emotions: neutral(N), anger(A), happiness(H), sadness(S), and calmness(C). 7 | 8 | Participants engage in paired listen/speak sets with recordings of an experienced actor, seated in front of a 27-inch monitor displaying visual stimuli. 9 | The experiment is designed as a pseudo-random class-iteration sequence: [A, A, C, C, S, S, H, A, C, H, H, S, S, A, A, C, C, H, H, S]. 10 | Throughout the experiment, each participant contributed 200 interactions. This resulted in a cumulative total of 8,400 interactions across all participants. 11 | Please refer to the [paper](https://www.nature.com/articles/s41597-024-03838-4) 12 | for more details: 13 | https://www.nature.com/articles/s41597-024-03838-4 14 | 15 | ## Domains 16 | 17 | ### Video 18 | Each 'Video' subfolder contains 200 video clips, each is 20 sec in lengths, 30 fps and performs either ’listening’ or ’speaking’ tasks. 19 | The Video data adopts the structure - [5 emotion classes × 2 tasks × 20 iterations] 20 | 21 | File format: .mp4 22 | 23 | Baseline performance of DeepFace: Mean ACC = 52.8 %, Mean F1-score = 51.5 % 24 | 25 | ### Audio 26 | Each 'Audio' subfolder contains 100 audio files, each is 20 sec in lengths and performs only ’speaking’ task. 27 | The audio data adopts the structure - [5 classes × 1 task ('speaking') × 20 conversations] 28 | 29 | File format: .wav 30 | 31 | Baseline performance of SCNN: Mean ACC = 36.7 %, Mean F1-score = 34.1 % 32 | ### EEG 33 | Each 'EEG' subfolder contains 2 EEG data files. Each instance is 20 sec in lengths and an initial sampling rate of 500 Hz. Due to continuous recording, 34 | the processed EEG data adopts the structure - [200 instances × 10,000 time points(20s × 500 Hz) × 30 channels]. 35 | The labels for this data use a one-hot encoding format, structured as 200 trials by 10 classes (5 emotions multiplied by 2 36 | tasks). 37 | 38 | File format: .mat 39 | 40 | Baseline performance of EEGnet: Mean ACC = 36.7 %, Mean F1-score = 34.1 % 41 | 42 | _Note that the label information can be applied across all modalities since all recordings, regardless of the modality, were 43 | conducted synchronously. This ensures uniform annotations throughout the dataset._ 44 | 45 | 46 | ## Getting Started 47 | 48 | 49 | * conda environment 50 | ```sh 51 | conda create --name eav python=3.10 52 | conda activate eav 53 | ``` 54 | 55 | ### Installation 56 | 57 | 1. Clone the repo 58 | ```sh 59 | git clone https://github.com/nubcico/EAV.git 60 | cd EAV 61 | ``` 62 | 2. Install requirements 63 | ```sh 64 | pip install -r requirements.txt 65 | ``` 66 | 67 | 68 | ## Usage 69 | 70 | ### Dataset 71 | 72 | The raw dataset, along with the pre-extracted features, can be accessed and downloaded from [Zenodo](https://doi.org/10.5281/zenodo.10205702). 73 | 74 | ### Program Execution Process 75 | 76 | #### Selecting the Dataset Type 77 | 78 | After downloading the dataset, you must choose between utilizing the raw dataset or the pre-extracted features, as this decision will determine your subsequent steps. 79 | 80 | If you opt for the raw dataset, only a minor modification is required in the `Dataload_Audio.py` file: adjust the `parent_directory` parameter in the `DataLoadAudio` class to the directory location of the "EAV" folder on your system. 81 | 82 | Using the raw dataset enables customization of your training and testing data split ratio through the EAVDataSplit class. In our case, we employed a 70/30 split, calculated as `h_idx = 56`. If `x` is your desired training dataset percentage (e.g., x = 70), `h_idx` can be calculated using the formula `h_idx = (x * 80) / 100`. 83 | 84 | If you decide to work with the pre-extracted features, you need to modify the code as follows: comment out the lines currently used for the raw dataset before `aud_list.get_split`, then uncomment the section for the pre-extracted features. Additionally, set the `direct` variable to point to the path containing the "Audio" directory on your system. 85 | 86 | ```python 87 | aud_loader = DataLoadAudio(subject=sub, parent_directory=r'D:\EAV') 88 | [data_aud , data_aud_y] = aud_loader.process() 89 | aud_list = EAVDataSplit(data_aud, data_aud_y) 90 | [tr_x_aud, tr_y_aud, te_x_aud , te_y_aud] = aud_list.get_split(h_idx=56) 91 | 92 | # direct=r"D:\EAV\Inputs\Audio" 93 | # file_name = f"subject_{sub:02d}_aud.pkl" 94 | # file_ = os.path.join(direct, file_name) 95 | 96 | # with open(file_, 'rb') as f: 97 | # aud_list = pickle.load(f) 98 | # tr_x_aud, tr_y_aud, te_x_aud , te_y_aud = aud_list 99 | 100 | data = [tr_x_aud, tr_y_aud, te_x_aud , te_y_aud] 101 | ``` 102 | 103 | The same adjustments should be applied for each modality. 104 | 105 | #### Selecting the classification model 106 | 107 | For the classification of the audio modality, we employ the Audio Spectrogram Transformer (AST) model pretrained on the AudioSet dataset (the pre-trained model can be downloaded from [Hugging Face](https://huggingface.co/MIT/ast-finetuned-audioset-10-10-0.4593)), which we will subsequently fine-tune on our specific dataset, as implemented in the 'Dataload_audio.py' and 'Transformer_torch/Transformer_Audio.py' files. 108 | 109 | ```python 110 | from Transformer_torch import Transformer_Audio 111 | ... 112 | mod_path = os.path.join(os.getcwd(), 'ast-finetuned-audioset') 113 | Trainer = Transformer_Audio.AudioModelTrainer(data, model_path=mod_path, sub =f"subject_{sub:02d}", 114 | num_classes=5, weight_decay=1e-5, lr=0.005, batch_size = 8) 115 | Trainer.train(epochs=10, lr=5e-4, freeze=True) 116 | Trainer.train(epochs=15, lr=5e-6, freeze=False) 117 | test_acc.append(Trainer.outputs_test) 118 | ``` 119 | The 'AudioModelTrainer' class is designed to train and fine-tune this model effectively. It leverages PyTorch and the Hugging Face Transformers library to adapt the AST model for the emotion classification task. 120 | 121 | ```python 122 | from transformers import AutoModelForAudioClassification 123 | ... 124 | class AudioModelTrainer: 125 | def __init__(self, DATA, model_path, sub='', num_classes=5, weight_decay=1e-5, lr=0.001, batch_size=128): 126 | ... 127 | self.model = AutoModelForAudioClassification.from_pretrained(model_path) 128 | ``` 129 | 130 | For the video and EEG modalities, the framework allows the choice between Transformer-based and CNN-based models. Specifically, for video, we utilize the Vision Transformer model, which is pretrained on the facial_emotions_image_detection dataset(the pre-trained model can be downloaded from [Hugging Face](https://huggingface.co/dima806/facial_emotions_image_detection)). The following example from the 'Dataload_vision.py' file illustrates both options: 131 | 132 | ```python 133 | # Transformer for Vision 134 | from Transformer_torch import Transformer_Vision 135 | 136 | mod_path = os.path.join('C:\\Users\\minho.lee\\Dropbox\\Projects\\EAV', 'facial_emotions_image_detection') 137 | trainer = Transformer_Vision.ImageClassifierTrainer(data, 138 | model_path=mod_path, sub=f"subject_{sub:02d}", 139 | num_labels=5, lr=5e-5, batch_size=128) 140 | trainer.train(epochs=10, lr=5e-4, freeze=True) 141 | trainer.train(epochs=5, lr=5e-6, freeze=False) 142 | trainer.outputs_test 143 | ``` 144 | 145 | Alternatively, the CNN-based model can be utilized as follows: 146 | ```python 147 | # CNN for Vision 148 | from CNN_torch.CNN_Vision import ImageClassifierTrainer 149 | trainer = ImageClassifierTrainer(data, num_labels=5, lr=5e-5, batch_size=32) 150 | trainer.train(epochs=3, lr=5e-4, freeze=True) 151 | trainer.train(epochs=3, lr=5e-6, freeze=False) 152 | trainer._delete_dataloader() 153 | trainer.outputs_test 154 | ``` 155 | The same approach can be applied for the EEG modality, providing flexibility in choosing between Transformer and CNN architectures based on the requirements of the task. 156 | 157 | To execute the program via the command line, run the following commands for each modality respectively: 158 | 159 | ```sh 160 | python Dataload_audio.py 161 | ``` 162 | 163 | 164 | 165 | ## Data Preprocessing 166 | 167 | The data preprocessing for audio, EEG, and video modalities is designed to prepare the raw data for emotion classification. Each modality follows its own distinct workflow: 168 | 169 | 1. Audio Modality: 170 | The `DataLoadAudio` class handles audio data processing: 171 | - Data File Loading: The `data_files()` method retrieves audio file paths and their corresponding emotion labels. 172 | - Feature Extraction: The `feature_extraction()` method loads the audio files, resamples them to a target rate, and segments the audio into 5-second clips. Each segment is labeled according to the associated emotion. 173 | - Label Encoding: Emotion labels are converted to numerical indices for model compatibility. 174 | - Processing Coordination: The `process()` method orchestrates these steps, returning the extracted features and labels. 175 | 176 | ```python 177 | class DataLoadAudio: 178 | def process(self): 179 | self.data_files() # Load audio file paths and labels 180 | self.feature_extraction() # Extract audio features 181 | return self.feature, self.label_indexes # Return features and labels 182 | ``` 183 | 184 | 2. EEG Modality: 185 | The `DataLoadEEG` class manages EEG data processing: 186 | - Data File Loading: The `data_mat()` method loads EEG data and labels from MAT files. 187 | - Downsampling: The `downsampling()` method reduces the sampling frequency to a target rate. 188 | - Bandpass Filtering: The `bandpass()` method applies a bandpass filter to retain frequencies of interest. 189 | - Segmentation: The `data_div()` method divides the data into smaller segments for analysis. 190 | - Data Preparation: The `data_prepare()` method coordinates the above steps, returning the processed EEG segments and labels. 191 | 192 | ```python 193 | class DataLoadEEG: 194 | def data_prepare(self): 195 | self.data_mat() # Load EEG data and labels 196 | self.downsampling() # Downsample the data 197 | self.bandpass() # Apply bandpass filtering 198 | self.data_div() # Divide the data into segments 199 | return self.seg_f_div, self.label_div # Return filtered segments and labels 200 | ``` 201 | 202 | 3. Video Modality: 203 | The `DataLoadVision` class processes video data: 204 | - Data File Loading: The `data_files()` method gathers video file paths and emotion labels. 205 | - Face Detection and Frame Extraction: The `data_load()` method captures frames from each video, using face detection to align faces if enabled. It collects segments of 25 frames, each representing a 5-second interval. 206 | - Label Encoding: Emotion labels are converted to numerical indices for consistency. 207 | - Data Preparation: The `process()` method manages these tasks, returning the segmented images and their corresponding labels. 208 | 209 | ```python 210 | class DataLoadVision: 211 | def process(self): 212 | self.data_files() # Load video file paths and labels 213 | self.data_load() # Extract and process frames from videos 214 | return self.images, self.image_label_idx # Return processed image segments and labels 215 | ``` 216 | 217 | This structured approach ensures that each modality's data is appropriately preprocessed, facilitating effective training and evaluation of the emotion classification model. 218 | 219 | 220 | ## Roadmap 221 | 222 | - [x] CNN based Emotion recognition on Video, Audio and EEG domains using Tensorflow 223 | - [x] CNN based Emotion recognition on Video and EEG domains using PyTorch 224 | - [x] Transformer based Emotion recognition on Video, Audio and EEG domains using PyTorch 225 | - [ ] Create demo file 226 | - [ ] Add .pkl files of preprocessed video data (Feature_vision folder) 227 | - [ ] Add inference files 228 | 229 | 230 | 231 | ## Contact 232 | 233 | Minho Lee - minho.lee@nu.edu.kz 234 | 235 | Adai Shomanov - adai.shomanov@nu.edu.kz 236 | 237 | Zhuldyz Kabidenova - zhuldyz.kabidenova@nu.edu.kz 238 | 239 | Adnan Yazici - adnan.yazici@nu.edu.kz 240 | 241 | 242 | 243 | ## License 244 | 245 | Distributed under the MIT License. See `LICENSE.txt` for more information. 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | [product-screenshot]: images/EAVlogo.png 254 | -------------------------------------------------------------------------------- /Pre_trained_models/ast-finetuned-audioset/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "architectures": [ 3 | "ASTForAudioClassification" 4 | ], 5 | "attention_probs_dropout_prob": 0.0, 6 | "frequency_stride": 10, 7 | "hidden_act": "gelu", 8 | "hidden_dropout_prob": 0.0, 9 | "hidden_size": 768, 10 | "id2label": { 11 | "0": "Speech", 12 | "1": "Male speech, man speaking", 13 | "2": "Female speech, woman speaking", 14 | "3": "Child speech, kid speaking", 15 | "4": "Conversation", 16 | "5": "Narration, monologue", 17 | "6": "Babbling", 18 | "7": "Speech synthesizer", 19 | "8": "Shout", 20 | "9": "Bellow", 21 | "10": "Whoop", 22 | "11": "Yell", 23 | "12": "Battle cry", 24 | "13": "Children shouting", 25 | "14": "Screaming", 26 | "15": "Whispering", 27 | "16": "Laughter", 28 | "17": "Baby laughter", 29 | "18": "Giggle", 30 | "19": "Snicker", 31 | "20": "Belly laugh", 32 | "21": "Chuckle, chortle", 33 | "22": "Crying, sobbing", 34 | "23": "Baby cry, infant cry", 35 | "24": "Whimper", 36 | "25": "Wail, moan", 37 | "26": "Sigh", 38 | "27": "Singing", 39 | "28": "Choir", 40 | "29": "Yodeling", 41 | "30": "Chant", 42 | "31": "Mantra", 43 | "32": "Male singing", 44 | "33": "Female singing", 45 | "34": "Child singing", 46 | "35": "Synthetic singing", 47 | "36": "Rapping", 48 | "37": "Humming", 49 | "38": "Groan", 50 | "39": "Grunt", 51 | "40": "Whistling", 52 | "41": "Breathing", 53 | "42": "Wheeze", 54 | "43": "Snoring", 55 | "44": "Gasp", 56 | "45": "Pant", 57 | "46": "Snort", 58 | "47": "Cough", 59 | "48": "Throat clearing", 60 | "49": "Sneeze", 61 | "50": "Sniff", 62 | "51": "Run", 63 | "52": "Shuffle", 64 | "53": "Walk, footsteps", 65 | "54": "Chewing, mastication", 66 | "55": "Biting", 67 | "56": "Gargling", 68 | "57": "Stomach rumble", 69 | "58": "Burping, eructation", 70 | "59": "Hiccup", 71 | "60": "Fart", 72 | "61": "Hands", 73 | "62": "Finger snapping", 74 | "63": "Clapping", 75 | "64": "Heart sounds, heartbeat", 76 | "65": "Heart murmur", 77 | "66": "Cheering", 78 | "67": "Applause", 79 | "68": "Chatter", 80 | "69": "Crowd", 81 | "70": "Hubbub, speech noise, speech babble", 82 | "71": "Children playing", 83 | "72": "Animal", 84 | "73": "Domestic animals, pets", 85 | "74": "Dog", 86 | "75": "Bark", 87 | "76": "Yip", 88 | "77": "Howl", 89 | "78": "Bow-wow", 90 | "79": "Growling", 91 | "80": "Whimper (dog)", 92 | "81": "Cat", 93 | "82": "Purr", 94 | "83": "Meow", 95 | "84": "Hiss", 96 | "85": "Caterwaul", 97 | "86": "Livestock, farm animals, working animals", 98 | "87": "Horse", 99 | "88": "Clip-clop", 100 | "89": "Neigh, whinny", 101 | "90": "Cattle, bovinae", 102 | "91": "Moo", 103 | "92": "Cowbell", 104 | "93": "Pig", 105 | "94": "Oink", 106 | "95": "Goat", 107 | "96": "Bleat", 108 | "97": "Sheep", 109 | "98": "Fowl", 110 | "99": "Chicken, rooster", 111 | "100": "Cluck", 112 | "101": "Crowing, cock-a-doodle-doo", 113 | "102": "Turkey", 114 | "103": "Gobble", 115 | "104": "Duck", 116 | "105": "Quack", 117 | "106": "Goose", 118 | "107": "Honk", 119 | "108": "Wild animals", 120 | "109": "Roaring cats (lions, tigers)", 121 | "110": "Roar", 122 | "111": "Bird", 123 | "112": "Bird vocalization, bird call, bird song", 124 | "113": "Chirp, tweet", 125 | "114": "Squawk", 126 | "115": "Pigeon, dove", 127 | "116": "Coo", 128 | "117": "Crow", 129 | "118": "Caw", 130 | "119": "Owl", 131 | "120": "Hoot", 132 | "121": "Bird flight, flapping wings", 133 | "122": "Canidae, dogs, wolves", 134 | "123": "Rodents, rats, mice", 135 | "124": "Mouse", 136 | "125": "Patter", 137 | "126": "Insect", 138 | "127": "Cricket", 139 | "128": "Mosquito", 140 | "129": "Fly, housefly", 141 | "130": "Buzz", 142 | "131": "Bee, wasp, etc.", 143 | "132": "Frog", 144 | "133": "Croak", 145 | "134": "Snake", 146 | "135": "Rattle", 147 | "136": "Whale vocalization", 148 | "137": "Music", 149 | "138": "Musical instrument", 150 | "139": "Plucked string instrument", 151 | "140": "Guitar", 152 | "141": "Electric guitar", 153 | "142": "Bass guitar", 154 | "143": "Acoustic guitar", 155 | "144": "Steel guitar, slide guitar", 156 | "145": "Tapping (guitar technique)", 157 | "146": "Strum", 158 | "147": "Banjo", 159 | "148": "Sitar", 160 | "149": "Mandolin", 161 | "150": "Zither", 162 | "151": "Ukulele", 163 | "152": "Keyboard (musical)", 164 | "153": "Piano", 165 | "154": "Electric piano", 166 | "155": "Organ", 167 | "156": "Electronic organ", 168 | "157": "Hammond organ", 169 | "158": "Synthesizer", 170 | "159": "Sampler", 171 | "160": "Harpsichord", 172 | "161": "Percussion", 173 | "162": "Drum kit", 174 | "163": "Drum machine", 175 | "164": "Drum", 176 | "165": "Snare drum", 177 | "166": "Rimshot", 178 | "167": "Drum roll", 179 | "168": "Bass drum", 180 | "169": "Timpani", 181 | "170": "Tabla", 182 | "171": "Cymbal", 183 | "172": "Hi-hat", 184 | "173": "Wood block", 185 | "174": "Tambourine", 186 | "175": "Rattle (instrument)", 187 | "176": "Maraca", 188 | "177": "Gong", 189 | "178": "Tubular bells", 190 | "179": "Mallet percussion", 191 | "180": "Marimba, xylophone", 192 | "181": "Glockenspiel", 193 | "182": "Vibraphone", 194 | "183": "Steelpan", 195 | "184": "Orchestra", 196 | "185": "Brass instrument", 197 | "186": "French horn", 198 | "187": "Trumpet", 199 | "188": "Trombone", 200 | "189": "Bowed string instrument", 201 | "190": "String section", 202 | "191": "Violin, fiddle", 203 | "192": "Pizzicato", 204 | "193": "Cello", 205 | "194": "Double bass", 206 | "195": "Wind instrument, woodwind instrument", 207 | "196": "Flute", 208 | "197": "Saxophone", 209 | "198": "Clarinet", 210 | "199": "Harp", 211 | "200": "Bell", 212 | "201": "Church bell", 213 | "202": "Jingle bell", 214 | "203": "Bicycle bell", 215 | "204": "Tuning fork", 216 | "205": "Chime", 217 | "206": "Wind chime", 218 | "207": "Change ringing (campanology)", 219 | "208": "Harmonica", 220 | "209": "Accordion", 221 | "210": "Bagpipes", 222 | "211": "Didgeridoo", 223 | "212": "Shofar", 224 | "213": "Theremin", 225 | "214": "Singing bowl", 226 | "215": "Scratching (performance technique)", 227 | "216": "Pop music", 228 | "217": "Hip hop music", 229 | "218": "Beatboxing", 230 | "219": "Rock music", 231 | "220": "Heavy metal", 232 | "221": "Punk rock", 233 | "222": "Grunge", 234 | "223": "Progressive rock", 235 | "224": "Rock and roll", 236 | "225": "Psychedelic rock", 237 | "226": "Rhythm and blues", 238 | "227": "Soul music", 239 | "228": "Reggae", 240 | "229": "Country", 241 | "230": "Swing music", 242 | "231": "Bluegrass", 243 | "232": "Funk", 244 | "233": "Folk music", 245 | "234": "Middle Eastern music", 246 | "235": "Jazz", 247 | "236": "Disco", 248 | "237": "Classical music", 249 | "238": "Opera", 250 | "239": "Electronic music", 251 | "240": "House music", 252 | "241": "Techno", 253 | "242": "Dubstep", 254 | "243": "Drum and bass", 255 | "244": "Electronica", 256 | "245": "Electronic dance music", 257 | "246": "Ambient music", 258 | "247": "Trance music", 259 | "248": "Music of Latin America", 260 | "249": "Salsa music", 261 | "250": "Flamenco", 262 | "251": "Blues", 263 | "252": "Music for children", 264 | "253": "New-age music", 265 | "254": "Vocal music", 266 | "255": "A capella", 267 | "256": "Music of Africa", 268 | "257": "Afrobeat", 269 | "258": "Christian music", 270 | "259": "Gospel music", 271 | "260": "Music of Asia", 272 | "261": "Carnatic music", 273 | "262": "Music of Bollywood", 274 | "263": "Ska", 275 | "264": "Traditional music", 276 | "265": "Independent music", 277 | "266": "Song", 278 | "267": "Background music", 279 | "268": "Theme music", 280 | "269": "Jingle (music)", 281 | "270": "Soundtrack music", 282 | "271": "Lullaby", 283 | "272": "Video game music", 284 | "273": "Christmas music", 285 | "274": "Dance music", 286 | "275": "Wedding music", 287 | "276": "Happy music", 288 | "277": "Funny music", 289 | "278": "Sad music", 290 | "279": "Tender music", 291 | "280": "Exciting music", 292 | "281": "Angry music", 293 | "282": "Scary music", 294 | "283": "Wind", 295 | "284": "Rustling leaves", 296 | "285": "Wind noise (microphone)", 297 | "286": "Thunderstorm", 298 | "287": "Thunder", 299 | "288": "Water", 300 | "289": "Rain", 301 | "290": "Raindrop", 302 | "291": "Rain on surface", 303 | "292": "Stream", 304 | "293": "Waterfall", 305 | "294": "Ocean", 306 | "295": "Waves, surf", 307 | "296": "Steam", 308 | "297": "Gurgling", 309 | "298": "Fire", 310 | "299": "Crackle", 311 | "300": "Vehicle", 312 | "301": "Boat, Water vehicle", 313 | "302": "Sailboat, sailing ship", 314 | "303": "Rowboat, canoe, kayak", 315 | "304": "Motorboat, speedboat", 316 | "305": "Ship", 317 | "306": "Motor vehicle (road)", 318 | "307": "Car", 319 | "308": "Vehicle horn, car horn, honking", 320 | "309": "Toot", 321 | "310": "Car alarm", 322 | "311": "Power windows, electric windows", 323 | "312": "Skidding", 324 | "313": "Tire squeal", 325 | "314": "Car passing by", 326 | "315": "Race car, auto racing", 327 | "316": "Truck", 328 | "317": "Air brake", 329 | "318": "Air horn, truck horn", 330 | "319": "Reversing beeps", 331 | "320": "Ice cream truck, ice cream van", 332 | "321": "Bus", 333 | "322": "Emergency vehicle", 334 | "323": "Police car (siren)", 335 | "324": "Ambulance (siren)", 336 | "325": "Fire engine, fire truck (siren)", 337 | "326": "Motorcycle", 338 | "327": "Traffic noise, roadway noise", 339 | "328": "Rail transport", 340 | "329": "Train", 341 | "330": "Train whistle", 342 | "331": "Train horn", 343 | "332": "Railroad car, train wagon", 344 | "333": "Train wheels squealing", 345 | "334": "Subway, metro, underground", 346 | "335": "Aircraft", 347 | "336": "Aircraft engine", 348 | "337": "Jet engine", 349 | "338": "Propeller, airscrew", 350 | "339": "Helicopter", 351 | "340": "Fixed-wing aircraft, airplane", 352 | "341": "Bicycle", 353 | "342": "Skateboard", 354 | "343": "Engine", 355 | "344": "Light engine (high frequency)", 356 | "345": "Dental drill, dentist's drill", 357 | "346": "Lawn mower", 358 | "347": "Chainsaw", 359 | "348": "Medium engine (mid frequency)", 360 | "349": "Heavy engine (low frequency)", 361 | "350": "Engine knocking", 362 | "351": "Engine starting", 363 | "352": "Idling", 364 | "353": "Accelerating, revving, vroom", 365 | "354": "Door", 366 | "355": "Doorbell", 367 | "356": "Ding-dong", 368 | "357": "Sliding door", 369 | "358": "Slam", 370 | "359": "Knock", 371 | "360": "Tap", 372 | "361": "Squeak", 373 | "362": "Cupboard open or close", 374 | "363": "Drawer open or close", 375 | "364": "Dishes, pots, and pans", 376 | "365": "Cutlery, silverware", 377 | "366": "Chopping (food)", 378 | "367": "Frying (food)", 379 | "368": "Microwave oven", 380 | "369": "Blender", 381 | "370": "Water tap, faucet", 382 | "371": "Sink (filling or washing)", 383 | "372": "Bathtub (filling or washing)", 384 | "373": "Hair dryer", 385 | "374": "Toilet flush", 386 | "375": "Toothbrush", 387 | "376": "Electric toothbrush", 388 | "377": "Vacuum cleaner", 389 | "378": "Zipper (clothing)", 390 | "379": "Keys jangling", 391 | "380": "Coin (dropping)", 392 | "381": "Scissors", 393 | "382": "Electric shaver, electric razor", 394 | "383": "Shuffling cards", 395 | "384": "Typing", 396 | "385": "Typewriter", 397 | "386": "Computer keyboard", 398 | "387": "Writing", 399 | "388": "Alarm", 400 | "389": "Telephone", 401 | "390": "Telephone bell ringing", 402 | "391": "Ringtone", 403 | "392": "Telephone dialing, DTMF", 404 | "393": "Dial tone", 405 | "394": "Busy signal", 406 | "395": "Alarm clock", 407 | "396": "Siren", 408 | "397": "Civil defense siren", 409 | "398": "Buzzer", 410 | "399": "Smoke detector, smoke alarm", 411 | "400": "Fire alarm", 412 | "401": "Foghorn", 413 | "402": "Whistle", 414 | "403": "Steam whistle", 415 | "404": "Mechanisms", 416 | "405": "Ratchet, pawl", 417 | "406": "Clock", 418 | "407": "Tick", 419 | "408": "Tick-tock", 420 | "409": "Gears", 421 | "410": "Pulleys", 422 | "411": "Sewing machine", 423 | "412": "Mechanical fan", 424 | "413": "Air conditioning", 425 | "414": "Cash register", 426 | "415": "Printer", 427 | "416": "Camera", 428 | "417": "Single-lens reflex camera", 429 | "418": "Tools", 430 | "419": "Hammer", 431 | "420": "Jackhammer", 432 | "421": "Sawing", 433 | "422": "Filing (rasp)", 434 | "423": "Sanding", 435 | "424": "Power tool", 436 | "425": "Drill", 437 | "426": "Explosion", 438 | "427": "Gunshot, gunfire", 439 | "428": "Machine gun", 440 | "429": "Fusillade", 441 | "430": "Artillery fire", 442 | "431": "Cap gun", 443 | "432": "Fireworks", 444 | "433": "Firecracker", 445 | "434": "Burst, pop", 446 | "435": "Eruption", 447 | "436": "Boom", 448 | "437": "Wood", 449 | "438": "Chop", 450 | "439": "Splinter", 451 | "440": "Crack", 452 | "441": "Glass", 453 | "442": "Chink, clink", 454 | "443": "Shatter", 455 | "444": "Liquid", 456 | "445": "Splash, splatter", 457 | "446": "Slosh", 458 | "447": "Squish", 459 | "448": "Drip", 460 | "449": "Pour", 461 | "450": "Trickle, dribble", 462 | "451": "Gush", 463 | "452": "Fill (with liquid)", 464 | "453": "Spray", 465 | "454": "Pump (liquid)", 466 | "455": "Stir", 467 | "456": "Boiling", 468 | "457": "Sonar", 469 | "458": "Arrow", 470 | "459": "Whoosh, swoosh, swish", 471 | "460": "Thump, thud", 472 | "461": "Thunk", 473 | "462": "Electronic tuner", 474 | "463": "Effects unit", 475 | "464": "Chorus effect", 476 | "465": "Basketball bounce", 477 | "466": "Bang", 478 | "467": "Slap, smack", 479 | "468": "Whack, thwack", 480 | "469": "Smash, crash", 481 | "470": "Breaking", 482 | "471": "Bouncing", 483 | "472": "Whip", 484 | "473": "Flap", 485 | "474": "Scratch", 486 | "475": "Scrape", 487 | "476": "Rub", 488 | "477": "Roll", 489 | "478": "Crushing", 490 | "479": "Crumpling, crinkling", 491 | "480": "Tearing", 492 | "481": "Beep, bleep", 493 | "482": "Ping", 494 | "483": "Ding", 495 | "484": "Clang", 496 | "485": "Squeal", 497 | "486": "Creak", 498 | "487": "Rustle", 499 | "488": "Whir", 500 | "489": "Clatter", 501 | "490": "Sizzle", 502 | "491": "Clicking", 503 | "492": "Clickety-clack", 504 | "493": "Rumble", 505 | "494": "Plop", 506 | "495": "Jingle, tinkle", 507 | "496": "Hum", 508 | "497": "Zing", 509 | "498": "Boing", 510 | "499": "Crunch", 511 | "500": "Silence", 512 | "501": "Sine wave", 513 | "502": "Harmonic", 514 | "503": "Chirp tone", 515 | "504": "Sound effect", 516 | "505": "Pulse", 517 | "506": "Inside, small room", 518 | "507": "Inside, large room or hall", 519 | "508": "Inside, public space", 520 | "509": "Outside, urban or manmade", 521 | "510": "Outside, rural or natural", 522 | "511": "Reverberation", 523 | "512": "Echo", 524 | "513": "Noise", 525 | "514": "Environmental noise", 526 | "515": "Static", 527 | "516": "Mains hum", 528 | "517": "Distortion", 529 | "518": "Sidetone", 530 | "519": "Cacophony", 531 | "520": "White noise", 532 | "521": "Pink noise", 533 | "522": "Throbbing", 534 | "523": "Vibration", 535 | "524": "Television", 536 | "525": "Radio", 537 | "526": "Field recording" 538 | }, 539 | "initializer_range": 0.02, 540 | "intermediate_size": 3072, 541 | "label2id": { 542 | "A capella": 255, 543 | "Accelerating, revving, vroom": 353, 544 | "Accordion": 209, 545 | "Acoustic guitar": 143, 546 | "Afrobeat": 257, 547 | "Air brake": 317, 548 | "Air conditioning": 413, 549 | "Air horn, truck horn": 318, 550 | "Aircraft": 335, 551 | "Aircraft engine": 336, 552 | "Alarm": 388, 553 | "Alarm clock": 395, 554 | "Ambient music": 246, 555 | "Ambulance (siren)": 324, 556 | "Angry music": 281, 557 | "Animal": 72, 558 | "Applause": 67, 559 | "Arrow": 458, 560 | "Artillery fire": 430, 561 | "Babbling": 6, 562 | "Baby cry, infant cry": 23, 563 | "Baby laughter": 17, 564 | "Background music": 267, 565 | "Bagpipes": 210, 566 | "Bang": 466, 567 | "Banjo": 147, 568 | "Bark": 75, 569 | "Basketball bounce": 465, 570 | "Bass drum": 168, 571 | "Bass guitar": 142, 572 | "Bathtub (filling or washing)": 372, 573 | "Battle cry": 12, 574 | "Beatboxing": 218, 575 | "Bee, wasp, etc.": 131, 576 | "Beep, bleep": 481, 577 | "Bell": 200, 578 | "Bellow": 9, 579 | "Belly laugh": 20, 580 | "Bicycle": 341, 581 | "Bicycle bell": 203, 582 | "Bird": 111, 583 | "Bird flight, flapping wings": 121, 584 | "Bird vocalization, bird call, bird song": 112, 585 | "Biting": 55, 586 | "Bleat": 96, 587 | "Blender": 369, 588 | "Bluegrass": 231, 589 | "Blues": 251, 590 | "Boat, Water vehicle": 301, 591 | "Boiling": 456, 592 | "Boing": 498, 593 | "Boom": 436, 594 | "Bouncing": 471, 595 | "Bow-wow": 78, 596 | "Bowed string instrument": 189, 597 | "Brass instrument": 185, 598 | "Breaking": 470, 599 | "Breathing": 41, 600 | "Burping, eructation": 58, 601 | "Burst, pop": 434, 602 | "Bus": 321, 603 | "Busy signal": 394, 604 | "Buzz": 130, 605 | "Buzzer": 398, 606 | "Cacophony": 519, 607 | "Camera": 416, 608 | "Canidae, dogs, wolves": 122, 609 | "Cap gun": 431, 610 | "Car": 307, 611 | "Car alarm": 310, 612 | "Car passing by": 314, 613 | "Carnatic music": 261, 614 | "Cash register": 414, 615 | "Cat": 81, 616 | "Caterwaul": 85, 617 | "Cattle, bovinae": 90, 618 | "Caw": 118, 619 | "Cello": 193, 620 | "Chainsaw": 347, 621 | "Change ringing (campanology)": 207, 622 | "Chant": 30, 623 | "Chatter": 68, 624 | "Cheering": 66, 625 | "Chewing, mastication": 54, 626 | "Chicken, rooster": 99, 627 | "Child singing": 34, 628 | "Child speech, kid speaking": 3, 629 | "Children playing": 71, 630 | "Children shouting": 13, 631 | "Chime": 205, 632 | "Chink, clink": 442, 633 | "Chirp tone": 503, 634 | "Chirp, tweet": 113, 635 | "Choir": 28, 636 | "Chop": 438, 637 | "Chopping (food)": 366, 638 | "Chorus effect": 464, 639 | "Christian music": 258, 640 | "Christmas music": 273, 641 | "Chuckle, chortle": 21, 642 | "Church bell": 201, 643 | "Civil defense siren": 397, 644 | "Clang": 484, 645 | "Clapping": 63, 646 | "Clarinet": 198, 647 | "Classical music": 237, 648 | "Clatter": 489, 649 | "Clickety-clack": 492, 650 | "Clicking": 491, 651 | "Clip-clop": 88, 652 | "Clock": 406, 653 | "Cluck": 100, 654 | "Coin (dropping)": 380, 655 | "Computer keyboard": 386, 656 | "Conversation": 4, 657 | "Coo": 116, 658 | "Cough": 47, 659 | "Country": 229, 660 | "Cowbell": 92, 661 | "Crack": 440, 662 | "Crackle": 299, 663 | "Creak": 486, 664 | "Cricket": 127, 665 | "Croak": 133, 666 | "Crow": 117, 667 | "Crowd": 69, 668 | "Crowing, cock-a-doodle-doo": 101, 669 | "Crumpling, crinkling": 479, 670 | "Crunch": 499, 671 | "Crushing": 478, 672 | "Crying, sobbing": 22, 673 | "Cupboard open or close": 362, 674 | "Cutlery, silverware": 365, 675 | "Cymbal": 171, 676 | "Dance music": 274, 677 | "Dental drill, dentist's drill": 345, 678 | "Dial tone": 393, 679 | "Didgeridoo": 211, 680 | "Ding": 483, 681 | "Ding-dong": 356, 682 | "Disco": 236, 683 | "Dishes, pots, and pans": 364, 684 | "Distortion": 517, 685 | "Dog": 74, 686 | "Domestic animals, pets": 73, 687 | "Door": 354, 688 | "Doorbell": 355, 689 | "Double bass": 194, 690 | "Drawer open or close": 363, 691 | "Drill": 425, 692 | "Drip": 448, 693 | "Drum": 164, 694 | "Drum and bass": 243, 695 | "Drum kit": 162, 696 | "Drum machine": 163, 697 | "Drum roll": 167, 698 | "Dubstep": 242, 699 | "Duck": 104, 700 | "Echo": 512, 701 | "Effects unit": 463, 702 | "Electric guitar": 141, 703 | "Electric piano": 154, 704 | "Electric shaver, electric razor": 382, 705 | "Electric toothbrush": 376, 706 | "Electronic dance music": 245, 707 | "Electronic music": 239, 708 | "Electronic organ": 156, 709 | "Electronic tuner": 462, 710 | "Electronica": 244, 711 | "Emergency vehicle": 322, 712 | "Engine": 343, 713 | "Engine knocking": 350, 714 | "Engine starting": 351, 715 | "Environmental noise": 514, 716 | "Eruption": 435, 717 | "Exciting music": 280, 718 | "Explosion": 426, 719 | "Fart": 60, 720 | "Female singing": 33, 721 | "Female speech, woman speaking": 2, 722 | "Field recording": 526, 723 | "Filing (rasp)": 422, 724 | "Fill (with liquid)": 452, 725 | "Finger snapping": 62, 726 | "Fire": 298, 727 | "Fire alarm": 400, 728 | "Fire engine, fire truck (siren)": 325, 729 | "Firecracker": 433, 730 | "Fireworks": 432, 731 | "Fixed-wing aircraft, airplane": 340, 732 | "Flamenco": 250, 733 | "Flap": 473, 734 | "Flute": 196, 735 | "Fly, housefly": 129, 736 | "Foghorn": 401, 737 | "Folk music": 233, 738 | "Fowl": 98, 739 | "French horn": 186, 740 | "Frog": 132, 741 | "Frying (food)": 367, 742 | "Funk": 232, 743 | "Funny music": 277, 744 | "Fusillade": 429, 745 | "Gargling": 56, 746 | "Gasp": 44, 747 | "Gears": 409, 748 | "Giggle": 18, 749 | "Glass": 441, 750 | "Glockenspiel": 181, 751 | "Goat": 95, 752 | "Gobble": 103, 753 | "Gong": 177, 754 | "Goose": 106, 755 | "Gospel music": 259, 756 | "Groan": 38, 757 | "Growling": 79, 758 | "Grunge": 222, 759 | "Grunt": 39, 760 | "Guitar": 140, 761 | "Gunshot, gunfire": 427, 762 | "Gurgling": 297, 763 | "Gush": 451, 764 | "Hair dryer": 373, 765 | "Hammer": 419, 766 | "Hammond organ": 157, 767 | "Hands": 61, 768 | "Happy music": 276, 769 | "Harmonic": 502, 770 | "Harmonica": 208, 771 | "Harp": 199, 772 | "Harpsichord": 160, 773 | "Heart murmur": 65, 774 | "Heart sounds, heartbeat": 64, 775 | "Heavy engine (low frequency)": 349, 776 | "Heavy metal": 220, 777 | "Helicopter": 339, 778 | "Hi-hat": 172, 779 | "Hiccup": 59, 780 | "Hip hop music": 217, 781 | "Hiss": 84, 782 | "Honk": 107, 783 | "Hoot": 120, 784 | "Horse": 87, 785 | "House music": 240, 786 | "Howl": 77, 787 | "Hubbub, speech noise, speech babble": 70, 788 | "Hum": 496, 789 | "Humming": 37, 790 | "Ice cream truck, ice cream van": 320, 791 | "Idling": 352, 792 | "Independent music": 265, 793 | "Insect": 126, 794 | "Inside, large room or hall": 507, 795 | "Inside, public space": 508, 796 | "Inside, small room": 506, 797 | "Jackhammer": 420, 798 | "Jazz": 235, 799 | "Jet engine": 337, 800 | "Jingle (music)": 269, 801 | "Jingle bell": 202, 802 | "Jingle, tinkle": 495, 803 | "Keyboard (musical)": 152, 804 | "Keys jangling": 379, 805 | "Knock": 359, 806 | "Laughter": 16, 807 | "Lawn mower": 346, 808 | "Light engine (high frequency)": 344, 809 | "Liquid": 444, 810 | "Livestock, farm animals, working animals": 86, 811 | "Lullaby": 271, 812 | "Machine gun": 428, 813 | "Mains hum": 516, 814 | "Male singing": 32, 815 | "Male speech, man speaking": 1, 816 | "Mallet percussion": 179, 817 | "Mandolin": 149, 818 | "Mantra": 31, 819 | "Maraca": 176, 820 | "Marimba, xylophone": 180, 821 | "Mechanical fan": 412, 822 | "Mechanisms": 404, 823 | "Medium engine (mid frequency)": 348, 824 | "Meow": 83, 825 | "Microwave oven": 368, 826 | "Middle Eastern music": 234, 827 | "Moo": 91, 828 | "Mosquito": 128, 829 | "Motor vehicle (road)": 306, 830 | "Motorboat, speedboat": 304, 831 | "Motorcycle": 326, 832 | "Mouse": 124, 833 | "Music": 137, 834 | "Music for children": 252, 835 | "Music of Africa": 256, 836 | "Music of Asia": 260, 837 | "Music of Bollywood": 262, 838 | "Music of Latin America": 248, 839 | "Musical instrument": 138, 840 | "Narration, monologue": 5, 841 | "Neigh, whinny": 89, 842 | "New-age music": 253, 843 | "Noise": 513, 844 | "Ocean": 294, 845 | "Oink": 94, 846 | "Opera": 238, 847 | "Orchestra": 184, 848 | "Organ": 155, 849 | "Outside, rural or natural": 510, 850 | "Outside, urban or manmade": 509, 851 | "Owl": 119, 852 | "Pant": 45, 853 | "Patter": 125, 854 | "Percussion": 161, 855 | "Piano": 153, 856 | "Pig": 93, 857 | "Pigeon, dove": 115, 858 | "Ping": 482, 859 | "Pink noise": 521, 860 | "Pizzicato": 192, 861 | "Plop": 494, 862 | "Plucked string instrument": 139, 863 | "Police car (siren)": 323, 864 | "Pop music": 216, 865 | "Pour": 449, 866 | "Power tool": 424, 867 | "Power windows, electric windows": 311, 868 | "Printer": 415, 869 | "Progressive rock": 223, 870 | "Propeller, airscrew": 338, 871 | "Psychedelic rock": 225, 872 | "Pulleys": 410, 873 | "Pulse": 505, 874 | "Pump (liquid)": 454, 875 | "Punk rock": 221, 876 | "Purr": 82, 877 | "Quack": 105, 878 | "Race car, auto racing": 315, 879 | "Radio": 525, 880 | "Rail transport": 328, 881 | "Railroad car, train wagon": 332, 882 | "Rain": 289, 883 | "Rain on surface": 291, 884 | "Raindrop": 290, 885 | "Rapping": 36, 886 | "Ratchet, pawl": 405, 887 | "Rattle": 135, 888 | "Rattle (instrument)": 175, 889 | "Reggae": 228, 890 | "Reverberation": 511, 891 | "Reversing beeps": 319, 892 | "Rhythm and blues": 226, 893 | "Rimshot": 166, 894 | "Ringtone": 391, 895 | "Roar": 110, 896 | "Roaring cats (lions, tigers)": 109, 897 | "Rock and roll": 224, 898 | "Rock music": 219, 899 | "Rodents, rats, mice": 123, 900 | "Roll": 477, 901 | "Rowboat, canoe, kayak": 303, 902 | "Rub": 476, 903 | "Rumble": 493, 904 | "Run": 51, 905 | "Rustle": 487, 906 | "Rustling leaves": 284, 907 | "Sad music": 278, 908 | "Sailboat, sailing ship": 302, 909 | "Salsa music": 249, 910 | "Sampler": 159, 911 | "Sanding": 423, 912 | "Sawing": 421, 913 | "Saxophone": 197, 914 | "Scary music": 282, 915 | "Scissors": 381, 916 | "Scrape": 475, 917 | "Scratch": 474, 918 | "Scratching (performance technique)": 215, 919 | "Screaming": 14, 920 | "Sewing machine": 411, 921 | "Shatter": 443, 922 | "Sheep": 97, 923 | "Ship": 305, 924 | "Shofar": 212, 925 | "Shout": 8, 926 | "Shuffle": 52, 927 | "Shuffling cards": 383, 928 | "Sidetone": 518, 929 | "Sigh": 26, 930 | "Silence": 500, 931 | "Sine wave": 501, 932 | "Singing": 27, 933 | "Singing bowl": 214, 934 | "Single-lens reflex camera": 417, 935 | "Sink (filling or washing)": 371, 936 | "Siren": 396, 937 | "Sitar": 148, 938 | "Sizzle": 490, 939 | "Ska": 263, 940 | "Skateboard": 342, 941 | "Skidding": 312, 942 | "Slam": 358, 943 | "Slap, smack": 467, 944 | "Sliding door": 357, 945 | "Slosh": 446, 946 | "Smash, crash": 469, 947 | "Smoke detector, smoke alarm": 399, 948 | "Snake": 134, 949 | "Snare drum": 165, 950 | "Sneeze": 49, 951 | "Snicker": 19, 952 | "Sniff": 50, 953 | "Snoring": 43, 954 | "Snort": 46, 955 | "Sonar": 457, 956 | "Song": 266, 957 | "Soul music": 227, 958 | "Sound effect": 504, 959 | "Soundtrack music": 270, 960 | "Speech": 0, 961 | "Speech synthesizer": 7, 962 | "Splash, splatter": 445, 963 | "Splinter": 439, 964 | "Spray": 453, 965 | "Squawk": 114, 966 | "Squeak": 361, 967 | "Squeal": 485, 968 | "Squish": 447, 969 | "Static": 515, 970 | "Steam": 296, 971 | "Steam whistle": 403, 972 | "Steel guitar, slide guitar": 144, 973 | "Steelpan": 183, 974 | "Stir": 455, 975 | "Stomach rumble": 57, 976 | "Stream": 292, 977 | "String section": 190, 978 | "Strum": 146, 979 | "Subway, metro, underground": 334, 980 | "Swing music": 230, 981 | "Synthesizer": 158, 982 | "Synthetic singing": 35, 983 | "Tabla": 170, 984 | "Tambourine": 174, 985 | "Tap": 360, 986 | "Tapping (guitar technique)": 145, 987 | "Tearing": 480, 988 | "Techno": 241, 989 | "Telephone": 389, 990 | "Telephone bell ringing": 390, 991 | "Telephone dialing, DTMF": 392, 992 | "Television": 524, 993 | "Tender music": 279, 994 | "Theme music": 268, 995 | "Theremin": 213, 996 | "Throat clearing": 48, 997 | "Throbbing": 522, 998 | "Thump, thud": 460, 999 | "Thunder": 287, 1000 | "Thunderstorm": 286, 1001 | "Thunk": 461, 1002 | "Tick": 407, 1003 | "Tick-tock": 408, 1004 | "Timpani": 169, 1005 | "Tire squeal": 313, 1006 | "Toilet flush": 374, 1007 | "Tools": 418, 1008 | "Toot": 309, 1009 | "Toothbrush": 375, 1010 | "Traditional music": 264, 1011 | "Traffic noise, roadway noise": 327, 1012 | "Train": 329, 1013 | "Train horn": 331, 1014 | "Train wheels squealing": 333, 1015 | "Train whistle": 330, 1016 | "Trance music": 247, 1017 | "Trickle, dribble": 450, 1018 | "Trombone": 188, 1019 | "Truck": 316, 1020 | "Trumpet": 187, 1021 | "Tubular bells": 178, 1022 | "Tuning fork": 204, 1023 | "Turkey": 102, 1024 | "Typewriter": 385, 1025 | "Typing": 384, 1026 | "Ukulele": 151, 1027 | "Vacuum cleaner": 377, 1028 | "Vehicle": 300, 1029 | "Vehicle horn, car horn, honking": 308, 1030 | "Vibraphone": 182, 1031 | "Vibration": 523, 1032 | "Video game music": 272, 1033 | "Violin, fiddle": 191, 1034 | "Vocal music": 254, 1035 | "Wail, moan": 25, 1036 | "Walk, footsteps": 53, 1037 | "Water": 288, 1038 | "Water tap, faucet": 370, 1039 | "Waterfall": 293, 1040 | "Waves, surf": 295, 1041 | "Wedding music": 275, 1042 | "Whack, thwack": 468, 1043 | "Whale vocalization": 136, 1044 | "Wheeze": 42, 1045 | "Whimper": 24, 1046 | "Whimper (dog)": 80, 1047 | "Whip": 472, 1048 | "Whir": 488, 1049 | "Whispering": 15, 1050 | "Whistle": 402, 1051 | "Whistling": 40, 1052 | "White noise": 520, 1053 | "Whoop": 10, 1054 | "Whoosh, swoosh, swish": 459, 1055 | "Wild animals": 108, 1056 | "Wind": 283, 1057 | "Wind chime": 206, 1058 | "Wind instrument, woodwind instrument": 195, 1059 | "Wind noise (microphone)": 285, 1060 | "Wood": 437, 1061 | "Wood block": 173, 1062 | "Writing": 387, 1063 | "Yell": 11, 1064 | "Yip": 76, 1065 | "Yodeling": 29, 1066 | "Zing": 497, 1067 | "Zipper (clothing)": 378, 1068 | "Zither": 150 1069 | }, 1070 | "layer_norm_eps": 1e-12, 1071 | "max_length": 1024, 1072 | "model_type": "audio-spectrogram-transformer", 1073 | "num_attention_heads": 12, 1074 | "num_hidden_layers": 12, 1075 | "num_mel_bins": 128, 1076 | "patch_size": 16, 1077 | "qkv_bias": true, 1078 | "time_stride": 10, 1079 | "torch_dtype": "float32", 1080 | "transformers_version": "4.25.0.dev0" 1081 | } 1082 | -------------------------------------------------------------------------------- /CNN_tensorflow/EEG_nb.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 9, 6 | "id": "f503cb6f-cae0-4315-8b99-9ba1ed804323", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "import os\n", 11 | "import scipy.io\n", 12 | "import numpy as np\n", 13 | "#from tensorflow.keras.models import Model\n", 14 | "from tensorflow.keras.layers import (Dense, Activation, Permute, Dropout, Conv2D, \n", 15 | " MaxPooling2D, AveragePooling2D, SeparableConv2D, \n", 16 | " DepthwiseConv2D, BatchNormalization, SpatialDropout2D, \n", 17 | " Input, Flatten)\n", 18 | "from tensorflow.keras.models import Model\n", 19 | "from tensorflow.keras.regularizers import l1_l2\n", 20 | "from tensorflow.keras.constraints import max_norm\n", 21 | "from tensorflow.keras import backend as K\n", 22 | "from scipy import signal\n", 23 | "from scipy.signal import butter, sosfilt, sosfreqz\n", 24 | "from sklearn.metrics import accuracy_score, confusion_matrix, f1_score\n", 25 | "from scipy.linalg import eigh\n", 26 | "import tensorflow as tf\n", 27 | "from scipy.stats import entropy\n", 28 | "from sklearn.model_selection import train_test_split" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 3, 34 | "id": "5698591f-7382-4005-b8f9-5d960d3083c4", 35 | "metadata": {}, 36 | "outputs": [ 37 | { 38 | "name": "stdout", 39 | "output_type": "stream", 40 | "text": [ 41 | "['subject1', 'subject2', 'subject3', 'subject4', 'subject5', 'subject6', 'subject7', 'subject8', 'subject9', 'subject10', 'subject11', 'subject12', 'subject13', 'subject14', 'subject15', 'subject16', 'subject17', 'subject18', 'subject19', 'subject20', 'subject21', 'subject22', 'subject23', 'subject24', 'subject25', 'subject26', 'subject27', 'subject28', 'subject29', 'subject30', 'subject31', 'subject32', 'subject33', 'subject34', 'subject35', 'subject36', 'subject37', 'subject38', 'subject39', 'subject40']\n" 42 | ] 43 | } 44 | ], 45 | "source": [ 46 | "# Path to the root directory\n", 47 | "parent_directory = r'C:\\Users\\minho.lee\\Dropbox\\EAV'\n", 48 | "# Get all directories in the parent directory that start with \"subject\"\n", 49 | "subject_folders = [d for d in os.listdir(parent_directory) if os.path.isdir(os.path.join(parent_directory, d)) and d.startswith(\"subject\")]\n", 50 | "\n", 51 | "# Sort the list\n", 52 | "sorted_subject_folders = sorted(subject_folders, key=lambda s: int(s.replace(\"subject\", \"\")))\n", 53 | "\n", 54 | "print(sorted_subject_folders)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 10, 60 | "id": "cee58179-7707-461c-8fc3-f6fc3a58de86", 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "def EEGNet(nb_classes, Chans = 64, Samples = 128, \n", 65 | " dropoutRate = 0.5, kernLength = 64, F1 = 8, \n", 66 | " D = 2, F2 = 16, norm_rate = 0.25, dropoutType = 'Dropout'):\n", 67 | " \n", 68 | " if dropoutType == 'SpatialDropout2D':\n", 69 | " dropoutType = SpatialDropout2D\n", 70 | " elif dropoutType == 'Dropout':\n", 71 | " dropoutType = Dropout\n", 72 | " else:\n", 73 | " raise ValueError('dropoutType must be one of SpatialDropout2D '\n", 74 | " 'or Dropout, passed as a string.')\n", 75 | " \n", 76 | " input1 = Input(shape = (Chans, Samples, 1))\n", 77 | "\n", 78 | " ##################################################################\n", 79 | " block1 = Conv2D(F1, (1, kernLength), padding = 'same',\n", 80 | " input_shape = (Chans, Samples, 1),\n", 81 | " use_bias = False)(input1)\n", 82 | " block1 = BatchNormalization()(block1)\n", 83 | " block1 = DepthwiseConv2D((Chans, 1), use_bias = False, \n", 84 | " depth_multiplier = D,\n", 85 | " depthwise_constraint = max_norm(1.))(block1)\n", 86 | " block1 = BatchNormalization()(block1)\n", 87 | " block1 = Activation('elu')(block1)\n", 88 | " block1 = AveragePooling2D((1, 4))(block1)\n", 89 | " block1 = dropoutType(dropoutRate)(block1)\n", 90 | " \n", 91 | " block2 = SeparableConv2D(F2, (1, 16),\n", 92 | " use_bias = False, padding = 'same')(block1)\n", 93 | " block2 = BatchNormalization()(block2)\n", 94 | " block2 = Activation('elu')(block2)\n", 95 | " block2 = AveragePooling2D((1, 8))(block2)\n", 96 | " block2 = dropoutType(dropoutRate)(block2)\n", 97 | " \n", 98 | " flatten = Flatten(name = 'flatten')(block2)\n", 99 | " \n", 100 | " dense = Dense(nb_classes, name = 'dense', \n", 101 | " kernel_constraint = max_norm(norm_rate))(flatten)\n", 102 | " softmax = Activation('softmax', name = 'softmax')(dense)\n", 103 | " \n", 104 | " return Model(inputs=input1, outputs=softmax)\n" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 11, 110 | "id": "9b98c526-8f5e-490c-a6e0-769f44ef5dec", 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | " # input = (10000, 30, 200)\n", 115 | " # out = (10000, 30, 200)\n", 116 | " def Bandpass(dat, freq = [5, 80], fs = 500): \n", 117 | " [D, Ch, Tri] = dat.shape\n", 118 | " dat2 = np.transpose(dat, [0, 2, 1])\n", 119 | " dat3 = np.reshape(dat2, [10000*200, 30], order = 'F')\n", 120 | " \n", 121 | " sos = butter(5, freq, 'band', fs=fs, output='sos') \n", 122 | " fdat = list()\n", 123 | " for i in range(np.size(dat3, 1)):\n", 124 | " tm = signal.sosfilt(sos, dat3[:,i])\n", 125 | " fdat.append(tm) \n", 126 | " dat4 = np.array(fdat).transpose().reshape((D, Tri, Ch), order = 'F').transpose(0, 2, 1)\n", 127 | " return dat4\n", 128 | " \n", 129 | " # no need this function anymore\n", 130 | " def seg(dat, marker, ival = [0, 2000]): \n", 131 | " sdat = list()\n", 132 | " for i in range(np.size(marker, 1)):\n", 133 | " lag = range(marker[0, i]+ival[0], marker[0, i]+ival[1])\n", 134 | " sdat.append(dat[lag, :] ) \n", 135 | " return np.array(sdat)\n", 136 | " \n", 137 | " # data = (200, 2000, 30), label = (10, 160)\n", 138 | " def mysplit(data, label): \n", 139 | " # Original data and labels\n", 140 | " #data = np.random.rand(200, 30, 2000) # Replace with your actual data\n", 141 | " #labels = np.random.randint(0, 2, size=(160, 10)) # Replace with your actual labels (one-hot vectors)\n", 142 | " \n", 143 | " # Splitting parameters\n", 144 | " split_length = 500 # Length of each split\n", 145 | " num_splits = data.shape[1] // split_length # Number of splits\n", 146 | " \n", 147 | " a1 = np.transpose(data, [1, 0, 2])\n", 148 | " a2 = np.reshape(a1, [500, 4, 200, 30], order = 'F')\n", 149 | " a3 = np.reshape(a2, [500, 4*200, 30], order = 'F')\n", 150 | " a4 = np.transpose(a3, [1, 0, 2])\n", 151 | " \n", 152 | " labels_repeated = np.repeat(label, repeats = 4, axis=1)\n", 153 | " \n", 154 | " return a4, labels_repeated" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 12, 160 | "id": "9bbe6c3e-821d-4c12-81c8-14e3bb873e89", 161 | "metadata": {}, 162 | "outputs": [ 163 | { 164 | "name": "stdout", 165 | "output_type": "stream", 166 | "text": [ 167 | "Loaded EEG data for subject1\n", 168 | "Epoch 1/100\n", 169 | "7/7 [==============================] - 10s 655ms/step - loss: 1.5794 - accuracy: 0.2400 - val_loss: 1.5917 - val_accuracy: 0.3150\n", 170 | "Epoch 2/100\n", 171 | "7/7 [==============================] - 4s 545ms/step - loss: 1.4161 - accuracy: 0.3750 - val_loss: 1.5733 - val_accuracy: 0.3000\n", 172 | "Epoch 3/100\n", 173 | "7/7 [==============================] - 4s 534ms/step - loss: 1.2619 - accuracy: 0.4150 - val_loss: 1.5464 - val_accuracy: 0.2250\n", 174 | "Epoch 4/100\n", 175 | "7/7 [==============================] - 4s 537ms/step - loss: 1.1083 - accuracy: 0.4300 - val_loss: 1.5236 - val_accuracy: 0.3200\n", 176 | "Epoch 5/100\n", 177 | "7/7 [==============================] - 4s 534ms/step - loss: 1.0452 - accuracy: 0.4750 - val_loss: 1.5097 - val_accuracy: 0.4000\n", 178 | "Epoch 6/100\n", 179 | "7/7 [==============================] - 4s 533ms/step - loss: 0.9920 - accuracy: 0.5100 - val_loss: 1.4941 - val_accuracy: 0.4750\n", 180 | "Epoch 7/100\n", 181 | "7/7 [==============================] - 4s 536ms/step - loss: 1.0158 - accuracy: 0.5650 - val_loss: 1.4857 - val_accuracy: 0.4700\n", 182 | "Epoch 8/100\n", 183 | "7/7 [==============================] - 4s 559ms/step - loss: 0.9620 - accuracy: 0.5550 - val_loss: 1.4730 - val_accuracy: 0.4900\n", 184 | "Epoch 9/100\n", 185 | "7/7 [==============================] - 4s 533ms/step - loss: 0.9329 - accuracy: 0.5900 - val_loss: 1.4550 - val_accuracy: 0.5250\n", 186 | "Epoch 10/100\n", 187 | "7/7 [==============================] - 4s 535ms/step - loss: 0.8893 - accuracy: 0.6100 - val_loss: 1.4424 - val_accuracy: 0.5700\n", 188 | "Epoch 11/100\n", 189 | "7/7 [==============================] - 4s 535ms/step - loss: 0.8810 - accuracy: 0.6100 - val_loss: 1.4326 - val_accuracy: 0.5450\n", 190 | "Epoch 12/100\n", 191 | "7/7 [==============================] - 4s 537ms/step - loss: 0.8529 - accuracy: 0.5850 - val_loss: 1.4213 - val_accuracy: 0.5150\n", 192 | "Epoch 13/100\n", 193 | "7/7 [==============================] - ETA: 0s - loss: 0.8373 - accuracy: 0.6200" 194 | ] 195 | }, 196 | { 197 | "ename": "KeyboardInterrupt", 198 | "evalue": "", 199 | "output_type": "error", 200 | "traceback": [ 201 | "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", 202 | "\u001b[1;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", 203 | "Cell \u001b[1;32mIn[12], line 99\u001b[0m\n\u001b[0;32m 93\u001b[0m model \u001b[38;5;241m=\u001b[39m EEGNet(nb_classes \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m5\u001b[39m, D \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m8\u001b[39m, F2 \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m64\u001b[39m, Chans \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m30\u001b[39m, kernLength \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m300\u001b[39m, Samples \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m500\u001b[39m, \n\u001b[0;32m 94\u001b[0m dropoutRate \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0.5\u001b[39m)\n\u001b[0;32m 96\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcategorical_crossentropy\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124madam\u001b[39m\u001b[38;5;124m'\u001b[39m, \n\u001b[0;32m 97\u001b[0m metrics \u001b[38;5;241m=\u001b[39m [\u001b[38;5;124m'\u001b[39m\u001b[38;5;124maccuracy\u001b[39m\u001b[38;5;124m'\u001b[39m])\n\u001b[1;32m---> 99\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m32\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43mshuffle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvalidation_data\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[43m(\u001b[49m\u001b[43mx_test\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my_test\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m \n\u001b[0;32m 101\u001b[0m pred \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(x_test) \n\u001b[0;32m 102\u001b[0m pred \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39margmax(pred, axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m) \n", 204 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\keras\\utils\\traceback_utils.py:65\u001b[0m, in \u001b[0;36mfilter_traceback..error_handler\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m 63\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m 64\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m---> 65\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m fn(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m 66\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m 67\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n", 205 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\keras\\engine\\training.py:1606\u001b[0m, in \u001b[0;36mModel.fit\u001b[1;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[0;32m 1591\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mgetattr\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_eval_data_handler\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m) \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m 1592\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_eval_data_handler \u001b[38;5;241m=\u001b[39m data_adapter\u001b[38;5;241m.\u001b[39mget_data_handler(\n\u001b[0;32m 1593\u001b[0m x\u001b[38;5;241m=\u001b[39mval_x,\n\u001b[0;32m 1594\u001b[0m y\u001b[38;5;241m=\u001b[39mval_y,\n\u001b[1;32m (...)\u001b[0m\n\u001b[0;32m 1604\u001b[0m steps_per_execution\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_steps_per_execution,\n\u001b[0;32m 1605\u001b[0m )\n\u001b[1;32m-> 1606\u001b[0m val_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mevaluate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m 1607\u001b[0m \u001b[43m \u001b[49m\u001b[43mx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mval_x\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1608\u001b[0m \u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mval_y\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1609\u001b[0m \u001b[43m \u001b[49m\u001b[43msample_weight\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mval_sample_weight\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1610\u001b[0m \u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mvalidation_batch_size\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1611\u001b[0m \u001b[43m \u001b[49m\u001b[43msteps\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mvalidation_steps\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1612\u001b[0m \u001b[43m \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcallbacks\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1613\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_queue_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_queue_size\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1614\u001b[0m \u001b[43m \u001b[49m\u001b[43mworkers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mworkers\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1615\u001b[0m \u001b[43m \u001b[49m\u001b[43muse_multiprocessing\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43muse_multiprocessing\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 1616\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_dict\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[0;32m 1617\u001b[0m \u001b[43m \u001b[49m\u001b[43m_use_cached_eval_dataset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[0;32m 1618\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 1619\u001b[0m val_logs \u001b[38;5;241m=\u001b[39m {\n\u001b[0;32m 1620\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mval_\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m name: val \u001b[38;5;28;01mfor\u001b[39;00m name, val \u001b[38;5;129;01min\u001b[39;00m val_logs\u001b[38;5;241m.\u001b[39mitems()\n\u001b[0;32m 1621\u001b[0m }\n\u001b[0;32m 1622\u001b[0m epoch_logs\u001b[38;5;241m.\u001b[39mupdate(val_logs)\n", 206 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\keras\\utils\\traceback_utils.py:65\u001b[0m, in \u001b[0;36mfilter_traceback..error_handler\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m 63\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m 64\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m---> 65\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m fn(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m 66\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m 67\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n", 207 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\keras\\engine\\training.py:1947\u001b[0m, in \u001b[0;36mModel.evaluate\u001b[1;34m(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict, **kwargs)\u001b[0m\n\u001b[0;32m 1943\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[0;32m 1944\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtest\u001b[39m\u001b[38;5;124m\"\u001b[39m, step_num\u001b[38;5;241m=\u001b[39mstep, _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m\n\u001b[0;32m 1945\u001b[0m ):\n\u001b[0;32m 1946\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_test_batch_begin(step)\n\u001b[1;32m-> 1947\u001b[0m tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtest_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 1948\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[0;32m 1949\u001b[0m context\u001b[38;5;241m.\u001b[39masync_wait()\n", 208 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\util\\traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback..error_handler\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m fn(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n", 209 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\eager\\def_function.py:915\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[1;34m(self, *args, **kwds)\u001b[0m\n\u001b[0;32m 912\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 914\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[1;32m--> 915\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_call(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwds)\n\u001b[0;32m 917\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[0;32m 918\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n", 210 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\eager\\def_function.py:954\u001b[0m, in \u001b[0;36mFunction._call\u001b[1;34m(self, *args, **kwds)\u001b[0m\n\u001b[0;32m 951\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[0;32m 952\u001b[0m \u001b[38;5;66;03m# In this case we have not created variables on the first call. So we can\u001b[39;00m\n\u001b[0;32m 953\u001b[0m \u001b[38;5;66;03m# run the first trace but we should fail if variables are created.\u001b[39;00m\n\u001b[1;32m--> 954\u001b[0m results \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stateful_fn(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwds)\n\u001b[0;32m 955\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_created_variables \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m ALLOW_DYNAMIC_VARIABLE_CREATION:\n\u001b[0;32m 956\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCreating variables on a non-first call to a function\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 957\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m decorated with tf.function.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n", 211 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\eager\\function.py:2496\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m 2493\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[0;32m 2494\u001b[0m (graph_function,\n\u001b[0;32m 2495\u001b[0m filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[1;32m-> 2496\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m 2497\u001b[0m \u001b[43m \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n", 212 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\eager\\function.py:1862\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[1;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[0;32m 1858\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[0;32m 1859\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[0;32m 1860\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[0;32m 1861\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[1;32m-> 1862\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m 1863\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[0;32m 1864\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[0;32m 1865\u001b[0m args,\n\u001b[0;32m 1866\u001b[0m possible_gradient_type,\n\u001b[0;32m 1867\u001b[0m executing_eagerly)\n\u001b[0;32m 1868\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n", 213 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\eager\\function.py:499\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[1;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[0;32m 497\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[0;32m 498\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m--> 499\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m 500\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 501\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 502\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 503\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 504\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 505\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m 506\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[0;32m 507\u001b[0m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[0;32m 508\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[1;32m (...)\u001b[0m\n\u001b[0;32m 511\u001b[0m ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[0;32m 512\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n", 214 | "File \u001b[1;32m~\\AppData\\Local\\anaconda3\\envs\\Emotion_gpu2\\lib\\site-packages\\tensorflow\\python\\eager\\execute.py:54\u001b[0m, in \u001b[0;36mquick_execute\u001b[1;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[0;32m 52\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m 53\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[1;32m---> 54\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m 55\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 56\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m 57\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n", 215 | "\u001b[1;31mKeyboardInterrupt\u001b[0m: " 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "\n", 221 | "\n", 222 | "result_conf = list()\n", 223 | "result_acc = list()\n", 224 | "result_f1 = list()\n", 225 | "\n", 226 | "\n", 227 | "for subject in sorted_subject_folders:\n", 228 | " # Remove trailing '__' if present\n", 229 | " # subject = subject.rstrip('__')\n", 230 | " cnt_ = [];\n", 231 | " # Construct the full path to the EEG file\n", 232 | " eeg_folder = os.path.join(parent_directory, subject, 'EEG')\n", 233 | " eeg_file_name = subject.rstrip('__') + '_eeg.mat'\n", 234 | " eeg_file_path = os.path.join(eeg_folder, eeg_file_name)\n", 235 | " \n", 236 | " label_file_name = subject.rstrip('__') + '_eeg_label.mat'\n", 237 | " label_file_path = os.path.join(eeg_folder, label_file_name)\n", 238 | "\n", 239 | " # Check if the EEG file exists\n", 240 | " if os.path.exists(eeg_file_path): \n", 241 | " mat = scipy.io.loadmat(eeg_file_path)\n", 242 | " cnt_ = np.array(mat.get('seg1'))\n", 243 | " if np.ndim(cnt_) == 3:\n", 244 | " cnt_ = np.array(mat.get('seg1')) \n", 245 | " else:\n", 246 | " cnt_ = np.array(mat.get('seg')) \n", 247 | " \n", 248 | " mat_Y = scipy.io.loadmat(label_file_path)\n", 249 | " Label = np.array(mat_Y.get('label'))\n", 250 | " \n", 251 | " print(f'Loaded EEG data for {subject}')\n", 252 | " else:\n", 253 | " print(f'EEG data not found for {subject}')\n", 254 | "\n", 255 | " \n", 256 | " cnt_f = Bandpass(cnt_, freq = [3, 50], fs = 500)\n", 257 | " \n", 258 | " fs_original = 500 # Original sampling rate in Hz\n", 259 | " fs_target = 100 # Target sampling rate in Hz\n", 260 | " \n", 261 | " \n", 262 | " tm = np.transpose(cnt_f, [0, 2, 1]).reshape([10000*200, 30], order = 'F')\n", 263 | " \n", 264 | " downsampling_factor = fs_target / fs_original\n", 265 | " tm2 = signal.resample_poly(tm, up=1, down=int(fs_original/fs_target), axis=0)\n", 266 | " cnt_f2= np.reshape(tm2, [2000, 200, 30], order = 'F')\n", 267 | " \n", 268 | " cnt_seg = np.transpose(cnt_f2, [1, 0, 2])\n", 269 | " \n", 270 | " num_trials_per_class = np.sum(Label, axis=1) # check the balance. \n", 271 | " [cnt_seg_split, Label_split]= mysplit(cnt_seg, Label)\n", 272 | " \n", 273 | " \n", 274 | " dat = np.transpose(cnt_seg_split, (0, 2, 1)).reshape((800, 30, 500, 1)) # This should be (800, 30, 500, 1) for balanced classes\n", 275 | " \n", 276 | " \n", 277 | " selected_classes = [1, 3, 5, 7, 9] # only listening classes\n", 278 | " \n", 279 | " selected_indices = np.isin(np.argmax(Label_split, axis=0), selected_classes)\n", 280 | "\n", 281 | " data_5class = dat[selected_indices]\n", 282 | " \n", 283 | " aa = Label_split[:, selected_indices]\n", 284 | " label_5class = aa[selected_classes,:]\n", 285 | " \n", 286 | " #x_train, x_test, y_train, y_test = train_test_split(data_5class, label_5class.T, test_size=0.5, random_state=42, stratify=label_5class.T)\n", 287 | " \n", 288 | " \n", 289 | " # Lists to collect train and test subsets\n", 290 | " x_train_list = []\n", 291 | " x_test_list = []\n", 292 | " y_train_list = []\n", 293 | " y_test_list = []\n", 294 | " \n", 295 | " for i in range(5): # Looping over each class\n", 296 | " class_indices = np.where(label_5class.T[:, i] == 1)[0] # Find indices where current class label is 1\n", 297 | " midpoint = len(class_indices) // 2 # Calculate the midpoint for 50% split\n", 298 | " \n", 299 | " # this random shuffle will decide \"random order\" or \"sequential order in time\"\n", 300 | " #np.random.shuffle(class_indices) # Shuffle the indices randomly\n", 301 | " # Split data based on found indices\n", 302 | " x_train_list.append(data_5class[class_indices[:midpoint]])\n", 303 | " x_test_list.append(data_5class[class_indices[midpoint:]])\n", 304 | " \n", 305 | " y_train_list.append(label_5class.T[class_indices[:midpoint]])\n", 306 | " y_test_list.append(label_5class.T[class_indices[midpoint:]])\n", 307 | " \n", 308 | " # Convert lists to numpy arrays\n", 309 | " x_train = np.concatenate(x_train_list, axis=0)\n", 310 | " x_test = np.concatenate(x_test_list, axis=0)\n", 311 | " y_train = np.concatenate(y_train_list, axis=0)\n", 312 | " y_test = np.concatenate(y_test_list, axis=0)\n", 313 | " \n", 314 | " model = EEGNet(nb_classes = 5, D = 8, F2 = 64, Chans = 30, kernLength = 300, Samples = 500, \n", 315 | " dropoutRate = 0.5)\n", 316 | " \n", 317 | " model.compile(loss='categorical_crossentropy', optimizer='adam', \n", 318 | " metrics = ['accuracy'])\n", 319 | " \n", 320 | " model.fit(x_train, y_train, batch_size = 32, epochs = 100,shuffle=True, validation_data = (x_test, y_test)) \n", 321 | " \n", 322 | " pred = model.predict(x_test) \n", 323 | " pred = np.argmax(pred, axis=1) \n", 324 | " \n", 325 | " y_test2 = np.argmax(y_test, axis=1)\n", 326 | " \n", 327 | " cm = confusion_matrix(pred, y_test2)\n", 328 | " \n", 329 | " accuracy = accuracy_score(pred, y_test2)\n", 330 | " f1 = f1_score(y_test2, pred, average='weighted') # 'weighted' for multiclass F1-Score\n", 331 | " \n", 332 | " result_conf.append(cm)\n", 333 | " result_acc.append(accuracy)\n", 334 | " result_f1.append(f1) # Append the F1-Score to your result list\n", 335 | " \n", 336 | "result_conf_np = np.array(result_conf)\n", 337 | "summed_confusion_matrix = np.sum(result_conf_np, axis=0)\n", 338 | "summed_confusion_matrix_T = summed_confusion_matrix.T" 339 | ] 340 | } 341 | ], 342 | "metadata": { 343 | "kernelspec": { 344 | "display_name": "Python 3 (ipykernel)", 345 | "language": "python", 346 | "name": "python3" 347 | }, 348 | "language_info": { 349 | "codemirror_mode": { 350 | "name": "ipython", 351 | "version": 3 352 | }, 353 | "file_extension": ".py", 354 | "mimetype": "text/x-python", 355 | "name": "python", 356 | "nbconvert_exporter": "python", 357 | "pygments_lexer": "ipython3", 358 | "version": "3.10.13" 359 | } 360 | }, 361 | "nbformat": 4, 362 | "nbformat_minor": 5 363 | } 364 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | --------------------------------------------------------------------------------