├── README.md ├── constants.py ├── data └── README ├── data_process.py ├── dataset_loader.py ├── emojis ├── angry.png ├── disgusted.png ├── fearful.png ├── happy.png ├── neutral.png ├── sad.png └── surprised.png ├── haarcascades ├── haarcascade_eye.xml ├── haarcascade_eye_tree_eyeglasses.xml └── haarcascade_frontalface_default.xml ├── pics ├── 1happy.jpg ├── 1raf_aligned.png ├── 1sF=1.1.png ├── 1sF=1.3.png ├── 2happy.jpg ├── 2raf_aligned.png ├── 2sF=1.1.png ├── 2sF=1.3.png ├── 3raf_aligned.png ├── 3sF=1.1.png ├── 3sF=1.3.png ├── 3surprised.jpg ├── 4disguested.jpg ├── 4raf_aligned.png ├── 4sF=1.1.png ├── 4sF=1.3.png ├── 5raf_aligned.png ├── 5sF=1.1.png ├── 5sF=1.3.png └── 5surprised.jpg ├── predict_cam&video.py ├── predict_pic.py ├── tflearn_logs └── README └── train.py /README.md: -------------------------------------------------------------------------------- 1 | # Facial Expression Recognition 2 | 3 | Facial expression recognition using convolutional neural network. 4 | 5 | ## Overview 6 | ### Requirement 7 | - Python3.5 8 | - opencv 9 | - Keras 10 | - tensorflow-gpu 11 | - tflearn 12 | 13 | ### Data 14 | [Kaggle_fer2013](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data):Include 35587 lableled images, you can download `fer2013.tar.gz` and decompress `fer2013.csv` in the `data` folder. 15 | 16 | [RAF_Dataset](http://www.whdeng.cn/RAF/model1.html):Include 15399 basic images and 3954 compound images. 17 | 18 | Some processed data can be found here: https://pan.baidu.com/s/14xwd8YeTFk_LDKVn0YSbQA PWD: xxm5 19 | 20 | ### Howtouse 21 | - get data 22 | - run ```python3 data_process.py``` to generate npy files for training 23 | - run ```python3 train.py``` to train, you will get 3 'Gudi...' files after training 24 | - copy 'Gudi_model_100_epochs_20000_faces.data-00000-of-00001' and rename it 'Gudi_model_100_epochs_20000_faces' 25 | - run ```python3 predict_pic.py``` to predict faces of images 26 | - run ```python3 predict_cam&pic.py``` to predict faces in camera or video 27 | 28 | ### Results 29 | I find 5 images(happy,happy,suprised,disgusted,surprised) in [MS emotion_recognition API](https://azure.microsoft.com/zh-cn/services/cognitive-services/face/#recognition) and use them to test model. 30 | 31 | ![pics/1happy.jpg](pics/1happy.jpg)![pics/2happy.jpg](pics/2happy.jpg)![pics/3surprised.jpg](pics/3surprised.jpg)![pics/4disguested.jpg](pics/4disguested.jpg)![pics/5surprised.jpg](pics/5surprised.jpg) 32 | 33 | ![pics/1raf_aligned.png](pics/1raf_aligned.png)![pics/2raf_aligned.png](pics/2raf_aligned.png)![pics/3raf_aligned.png](pics/3raf_aligned.png)![pics/4raf_aligned.png](pics/4raf_aligned.png)![pics/5raf_aligned.png](pics/5raf_aligned.png) 34 | 35 | ![pics/1sF=1.3.png](pics/1sF=1.3.png)![pics/2sF=1.3.png](pics/2sF=1.3.png)![pics/3sF=1.3.png](pics/3sF=1.3.png)![pics/4sF=1.3.png](pics/4sF=1.3.png)![pics/5sF=1.3.png](pics/5sF=1.3.png) 36 | 37 | ![pics/1sF=1.1.png](pics/1sF=1.1.png)![pics/2sF=1.1.png](pics/2sF=1.1.png)![pics/3sF=1.1.png](pics/3sF=1.1.png)![pics/4sF=1.1.png](pics/4sF=1.1.png)![pics/5sF=1.1.png](pics/5sF=1.1.png) 38 | 39 | The first row shows original images, the second shows predicted images using raf_aligned dataset, the third and fourth show predicted images using Kaggle_fer dataset processed by opencv when scaleFactor=1.3&1.1 respectively. 40 | 41 | It seems that the results using raf_aligned dataset are not good. The reason might be that raf_aligned images are processed by humans and we predict faces using opencv. So using raf dataset processed by opencv to train the model might be better. 42 | 43 | And the results seem better when using processed Kaggle_fer2013 dataset by scaleFactor=1.1. I think that is bacause of more training images. 44 | 45 | ## Reference 46 | https://github.com/isseu/emotion-recognition-neural-networks 47 | 48 | ## Citation 49 | @inproceedings{li2017reliable, 50 | title={Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild}, 51 | author={Li, Shan and Deng, Weihong and Du, JunPing}, 52 | booktitle={2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 53 | pages={2584--2593}, 54 | year={2017}, 55 | organization={IEEE} 56 | } 57 | 58 | -------------------------------------------------------------------------------- /constants.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | # __ __ 3 | # /\ \__ /\ \__ 4 | # ___ ___ ___ ____\ \ ,_\ __ ___\ \ ,_\ ____ 5 | # /'___\ / __`\ /' _ `\ /',__\\ \ \/ /'__`\ /' _ `\ \ \/ /',__\ 6 | # /\ \__//\ \L\ \/\ \/\ \/\__, `\\ \ \_/\ \L\.\_/\ \/\ \ \ \_/\__, `\ 7 | # \ \____\ \____/\ \_\ \_\/\____/ \ \__\ \__/.\_\ \_\ \_\ \__\/\____/ 8 | # \/____/\/___/ \/_/\/_/\/___/ \/__/\/__/\/_/\/_/\/_/\/__/\/___/ .txt 9 | # 10 | # 11 | 12 | CASC_PATH = 'haarcascades/haarcascade_frontalface_default.xml' 13 | EYE_CASC = 'haarcascades/haarcascade_eye.xml' 14 | EYEGLASSSES_CASC = 'haarcascades/haarcascade_eye_tree_eyeglasses.xml' 15 | SIZE_FACE = 48 16 | SCALEFACTOR = 1.1 17 | EMOTIONS = ['angry', 'disgusted', 'fearful','happy', 'sad', 'surprised', 'neutral'] 18 | RAF_EMOTIONS = ['surprised', 'fearful', 'disgusted', 'happy', 'sad', 'angry', 'neutral'] 19 | SAVE_DIRECTORY = 'data' 20 | SAVE_MODEL_FILENAME = 'Gudi_model_100_epochs_20000_faces' 21 | DATASET_CSV_FILENAME = 'fer2013.csv' 22 | SAVE_DATASET_IMAGES_FILENAME = 'data_images.npy' 23 | SAVE_DATASET_LABELS_FILENAME = 'data_labels.npy' -------------------------------------------------------------------------------- /data/README: -------------------------------------------------------------------------------- 1 | DIR to save data 2 | Or you can download here: https://pan.baidu.com/s/14xwd8YeTFk_LDKVn0YSbQA pswd:xxm5 3 | -------------------------------------------------------------------------------- /data_process.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | from constants import * 4 | import cv2 5 | import pandas as pd 6 | import numpy as np 7 | from PIL import Image 8 | from os.path import join 9 | 10 | cascade_classifier = cv2.CascadeClassifier(CASC_PATH) 11 | 12 | 13 | def format_image(image, flag=0): 14 | if len(image.shape) > 2 and image.shape[2] == 3: 15 | image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) 16 | else: 17 | image = cv2.imdecode(image, cv2.IMREAD_GRAYSCALE) 18 | if flag == 0: 19 | gray_border = np.zeros((150, 150), np.uint8) 20 | gray_border[:, :] = 200 21 | gray_border[ 22 | int((150 / 2) - (SIZE_FACE / 2)): int((150 / 2) + (SIZE_FACE / 2)), 23 | int((150 / 2) - (SIZE_FACE / 2)): int((150 / 2) + (SIZE_FACE / 2)) 24 | ] = image 25 | image = gray_border 26 | faces = cascade_classifier.detectMultiScale( 27 | image, 28 | scaleFactor=SCALEFACTOR, 29 | minNeighbors=5 30 | ) 31 | 32 | # None is we don't found an image 33 | if not len(faces) > 0: 34 | return None 35 | max_area_face = faces[0] 36 | for face in faces: 37 | if face[2] * face[3] > max_area_face[2] * max_area_face[3]: 38 | max_area_face = face 39 | # Chop image to face 40 | face = max_area_face 41 | image = image[face[1]:(face[1] + face[2]), face[0]:(face[0] + face[3])] 42 | # Resize image to network size 43 | try: 44 | image = cv2.resize(image, (SIZE_FACE, SIZE_FACE), 45 | interpolation=cv2.INTER_CUBIC) / 255. 46 | except Exception: 47 | print("[+] Problem during resize") 48 | return None 49 | return image 50 | 51 | 52 | def emotion_to_vec(x): 53 | d = np.zeros(len(EMOTIONS)) 54 | d[x] = 1.0 55 | return d 56 | 57 | 58 | def flip_image(image): 59 | return cv2.flip(image, 1) 60 | 61 | 62 | def data_to_image(data, i): 63 | data_image = np.fromstring( 64 | str(data), dtype=np.uint8, sep=' ').reshape((SIZE_FACE, SIZE_FACE)) 65 | data_image = Image.fromarray(data_image).convert('RGB') 66 | data_image = np.array(data_image)[:, :, ::-1].copy() 67 | 68 | # if you want to save all images 69 | # cv2.imwrite(SAVE_DIRECTORY + '/images/' + str(i) + '.png', data_image) 70 | 71 | data_image = format_image(data_image) 72 | return data_image 73 | 74 | 75 | def get_fer(csv_path): 76 | data = pd.read_csv(csv_path) 77 | labels = [] 78 | images = [] 79 | count = 0 80 | total = data.shape[0] 81 | 82 | for index, row in data.iterrows(): 83 | emotion = emotion_to_vec(row['emotion']) 84 | image = data_to_image(row['pixels'], index) 85 | 86 | if image is not None: 87 | labels.append(emotion) 88 | images.append(image) 89 | 90 | # if you want to save faces 91 | # real_image = image * 255 92 | # cv2.imwrite(SAVE_DIRECTORY + '/faces/' + str(index) + '.png', real_image) 93 | 94 | count += 1 95 | print("Progress: {}/{} {:.2f}%".format(index, total, index * 100.0 / total)) 96 | 97 | print(index) # 共35587 98 | print("Total: " + str(len(images))) 99 | np.save(join(SAVE_DIRECTORY, 'sf=' + str(SCALEFACTOR) + '_' + SAVE_DATASET_IMAGES_FILENAME), images) 100 | np.save(join(SAVE_DIRECTORY, 'sf=' + str(SCALEFACTOR) + '_' + SAVE_DATASET_LABELS_FILENAME), labels) 101 | 102 | 103 | def rafnum2vec(x): 104 | d = np.zeros(7) 105 | d[EMOTIONS.index(RAF_EMOTIONS[x-1])] = 1.0 106 | return d 107 | 108 | 109 | def image2array(image_path): 110 | data = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE) 111 | array = cv2.resize(data, (SIZE_FACE, SIZE_FACE), interpolation=cv2.INTER_CUBIC) / 255. 112 | return array 113 | 114 | 115 | def get_raf_aligned(one_dir, one_txt): 116 | # 得到RAF aligned的npy文件 117 | dict_train, dict_test = {}, {} 118 | for line in open(one_txt, 'r'): 119 | line = line.strip() 120 | if line.startswith('train'): 121 | dict_train[line.split(' ')[0][:11] + '_aligned.jpg'] = int(line.split(' ')[1]) 122 | elif line.startswith('test'): 123 | dict_test[line.split(' ')[0][:9] + '_aligned.jpg'] = int(line.split(' ')[1]) 124 | else: 125 | print("wrong!!!") 126 | 127 | images, labels = [], [] 128 | for k, v in dict_train.items(): 129 | image_path = one_dir + k 130 | image = image2array(image_path) 131 | images.append(image) 132 | labels.append(rafnum2vec(v)) 133 | for k, v in dict_test.items(): 134 | image_path = one_dir + k 135 | images.append(image2array(image_path)) 136 | labels.append(rafnum2vec(v)) 137 | 138 | np.save(join(SAVE_DIRECTORY, 'raf_aligned_' + SAVE_DATASET_IMAGES_FILENAME), images) 139 | np.save(join(SAVE_DIRECTORY, 'raf_aligned_' + SAVE_DATASET_LABELS_FILENAME), labels) 140 | return "Get aligned!!!" 141 | 142 | 143 | def get_raf_original(one_dir, one_txt): 144 | # 用opencv得到RAF original的npy文件 145 | images, labels, count = [], [], 0 146 | for line in open(one_txt, 'r'): 147 | line = line.strip() 148 | image_path = one_dir + line.split(' ')[0] 149 | image = format_image(cv2.imread(image_path), flag=1) 150 | if image is not None: 151 | count += 1 152 | images.append(image) 153 | labels.append(rafnum2vec(int(line.split(' ')[1]))) 154 | 155 | print(count) # 12717 156 | np.save(join(SAVE_DIRECTORY, 'sf=' + str(SCALEFACTOR) + '_raf_origin_' + SAVE_DATASET_IMAGES_FILENAME), images) 157 | np.save(join(SAVE_DIRECTORY, 'sf=' + str(SCALEFACTOR) + '_raf_origin_' + SAVE_DATASET_LABELS_FILENAME), labels) 158 | return "Get original!!!" 159 | 160 | 161 | if __name__ == '__main__': 162 | get_fer(join(SAVE_DIRECTORY, DATASET_CSV_FILENAME)) 163 | raf_aligned_dir = '/data1/emotion_rec/Real-world Affective Faces (RAF) Database/basic/Image/aligned/' 164 | raf_original_dir = '/data1/emotion_rec/Real-world Affective Faces (RAF) Database/basic/Image/original/' 165 | label_txt = '/data1/emotion_rec/Real-world Affective Faces (RAF) Database/basic/EmoLabel/list_patition_label.txt' 166 | get_raf_aligned(raf_aligned_dir, label_txt) 167 | get_raf_original(raf_original_dir, label_txt) 168 | pass 169 | -------------------------------------------------------------------------------- /dataset_loader.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | from os.path import join 4 | import numpy as np 5 | from constants import * 6 | from sklearn.model_selection import train_test_split 7 | 8 | 9 | class DatasetLoader: 10 | def load_from_save(self): 11 | images = np.load(join(SAVE_DIRECTORY, 'sf=' + str(SCALEFACTOR) + '_' +SAVE_DATASET_IMAGES_FILENAME)) 12 | images = images.reshape([-1, SIZE_FACE, SIZE_FACE, 1]) 13 | labels = np.load(join(SAVE_DIRECTORY, 'sf=' + str(SCALEFACTOR) + '_' +SAVE_DATASET_LABELS_FILENAME)).\ 14 | reshape([-1, len(EMOTIONS)]) 15 | self._images, self._images_test, self._labels, self._labels_test = train_test_split\ 16 | (images, labels, test_size=0.10, random_state=42) 17 | 18 | @property 19 | def images(self): 20 | return self._images 21 | 22 | @property 23 | def labels(self): 24 | return self._labels 25 | 26 | @property 27 | def images_test(self): 28 | return self._images_test 29 | 30 | @property 31 | def labels_test(self): 32 | return self._labels_test 33 | -------------------------------------------------------------------------------- /emojis/angry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/angry.png -------------------------------------------------------------------------------- /emojis/disgusted.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/disgusted.png -------------------------------------------------------------------------------- /emojis/fearful.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/fearful.png -------------------------------------------------------------------------------- /emojis/happy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/happy.png -------------------------------------------------------------------------------- /emojis/neutral.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/neutral.png -------------------------------------------------------------------------------- /emojis/sad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/sad.png -------------------------------------------------------------------------------- /emojis/surprised.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/emojis/surprised.png -------------------------------------------------------------------------------- /pics/1happy.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/1happy.jpg -------------------------------------------------------------------------------- /pics/1raf_aligned.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/1raf_aligned.png -------------------------------------------------------------------------------- /pics/1sF=1.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/1sF=1.1.png -------------------------------------------------------------------------------- /pics/1sF=1.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/1sF=1.3.png -------------------------------------------------------------------------------- /pics/2happy.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/2happy.jpg -------------------------------------------------------------------------------- /pics/2raf_aligned.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/2raf_aligned.png -------------------------------------------------------------------------------- /pics/2sF=1.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/2sF=1.1.png -------------------------------------------------------------------------------- /pics/2sF=1.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/2sF=1.3.png -------------------------------------------------------------------------------- /pics/3raf_aligned.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/3raf_aligned.png -------------------------------------------------------------------------------- /pics/3sF=1.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/3sF=1.1.png -------------------------------------------------------------------------------- /pics/3sF=1.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/3sF=1.3.png -------------------------------------------------------------------------------- /pics/3surprised.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/3surprised.jpg -------------------------------------------------------------------------------- /pics/4disguested.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/4disguested.jpg -------------------------------------------------------------------------------- /pics/4raf_aligned.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/4raf_aligned.png -------------------------------------------------------------------------------- /pics/4sF=1.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/4sF=1.1.png -------------------------------------------------------------------------------- /pics/4sF=1.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/4sF=1.3.png -------------------------------------------------------------------------------- /pics/5raf_aligned.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/5raf_aligned.png -------------------------------------------------------------------------------- /pics/5sF=1.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/5sF=1.1.png -------------------------------------------------------------------------------- /pics/5sF=1.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/5sF=1.3.png -------------------------------------------------------------------------------- /pics/5surprised.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ShawDa/facial-expression-recognition/e9c5a57ab3c5068a2223f7be442fe1fba6895713/pics/5surprised.jpg -------------------------------------------------------------------------------- /predict_cam&video.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | import cv2 4 | from constants import * 5 | from train import EmotionRecognition 6 | import numpy as np 7 | from PIL import Image, ImageDraw, ImageFont 8 | import time 9 | 10 | global face 11 | cascade_classifier = cv2.CascadeClassifier(CASC_PATH) 12 | eye_cascade = cv2.CascadeClassifier(EYEGLASSSES_CASC) 13 | 14 | # Load Model 15 | network = EmotionRecognition() 16 | network.build_network() 17 | 18 | 19 | def format_image(image): 20 | global face 21 | 22 | if len(image.shape) > 2 and image.shape[2] == 3: 23 | image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) 24 | else: 25 | image = cv2.imdecode(image, cv2.IMREAD_GRAYSCALE) 26 | faces = cascade_classifier.detectMultiScale( 27 | image, 28 | scaleFactor=1.1, 29 | minNeighbors=5 30 | ) 31 | 32 | # None is we don't found an image 33 | if not len(faces) > 0: 34 | return None 35 | max_area_face = faces[0] 36 | for oneface in faces: 37 | if oneface[2] * oneface[3] > max_area_face[2] * max_area_face[3]: 38 | max_area_face = oneface 39 | 40 | # draw face 41 | face = max_area_face 42 | x, y, w, h = face 43 | cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2) 44 | 45 | # draw eyes 46 | roi_gray = image[y:y + h, x:x + w] 47 | roi_color = frame[y:y + h, x:x + w] 48 | eyes = eye_cascade.detectMultiScale(roi_gray) 49 | for (ex, ey, ew, eh) in eyes: 50 | cv2.rectangle(roi_color, (ex, ey), (ex + ew, ey + eh), (255, 0, 0), 2) 51 | 52 | # Chop image to face 53 | image = image[face[1]:(face[1] + face[2]), face[0]:(face[0] + face[3])] 54 | 55 | # Resize image to network size 56 | try: 57 | image = cv2.resize(image, (SIZE_FACE, SIZE_FACE), 58 | interpolation=cv2.INTER_CUBIC) / 255. 59 | except Exception: 60 | print("[+] Problem during resize") 61 | return None 62 | 63 | return image 64 | 65 | 66 | if __name__ == '__main__': 67 | mode = 'cam' # 'cam' default,or you can add a video path 68 | if mode == 'cam': # camera 69 | video_capture = cv2.VideoCapture(0) 70 | else: # video 71 | video_capture = cv2.VideoCapture(mode) 72 | 73 | # 设定摄像头分辨率 74 | video_capture.set(cv2.CAP_PROP_FRAME_WIDTH, 1000) 75 | video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT, 550) 76 | 77 | feelings_faces = [] 78 | 79 | for index, emotion in enumerate(EMOTIONS): 80 | feelings_faces.append(cv2.imread('./emojis/' + emotion + '.png', -1)) 81 | 82 | fourcc = cv2.VideoWriter_fourcc(*'XVID') 83 | out = cv2.VideoWriter('face.mp4', fourcc, 10, (640, 480)) 84 | fps_time = 0 85 | 86 | while True: 87 | # Capture frame-by-frame 88 | ret, frame = video_capture.read() 89 | 90 | # Predict result with network 91 | result = network.predict(format_image(frame)) 92 | 93 | # Write results in frame 94 | if result is not None: 95 | for index, emotion in enumerate(EMOTIONS): 96 | cv2.putText(frame, emotion, (10, index * 20 + 20), 97 | cv2.FONT_HERSHEY_PLAIN, 0.5, (0, 255, 0), 1) 98 | cv2.rectangle(frame, (130, index * 20 + 10), (130 + int(result[0][index] * 100), 99 | (index + 1) * 20 + 4), (255, 0, 0), -1) 100 | 101 | face_image = feelings_faces[np.argmax(result[0])] 102 | 103 | # Ugly transparent fix 104 | for c in range(0, 3): 105 | frame[200:320, 10:130, c] = face_image[:, :, c] * \ 106 | (face_image[:, :, 3] / 255.0) + frame[200:320, 107 | 10:130, c] * ( 108 | 1.0 - face_image[:, :, 3] / 255.0) 109 | # draw emotion text 110 | text = EMOTIONS[np.argmax(result[0])] 111 | # print(result, text) 112 | cv2.putText(frame, text, (face[0], face[1]), cv2.FONT_HERSHEY_PLAIN, 1, (0, 255, 0), 1) 113 | 114 | # put CH name on frames 115 | img_PIL = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)) 116 | # 字体 字体*.ttc的存放路径一般是： /usr/share/fonts/opentype/noto/ 查找指令locate *.ttc 117 | img_font = ImageFont.truetype('NotoSansCJK-Bold.ttc', 20) 118 | fillColor = (0, 255, 0) 119 | position = (face[0]+face[2]-40, face[1]-25) 120 | draw = ImageDraw.Draw(img_PIL) 121 | ch_str = '中国' 122 | draw.text(position, ch_str, font=img_font, fill=fillColor) 123 | frame = cv2.cvtColor(np.asarray(img_PIL), cv2.COLOR_RGB2BGR) 124 | 125 | # FPS 126 | cv2.putText(image, 127 | "FPS: %f" % (1.0 / (time.time() - fps_time)), 128 | (10, 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, 129 | (0, 255, 0), 2) 130 | 131 | out.write(frame) 132 | # Display the resulting frame 133 | cv2.imshow('Video', frame) 134 | fps_time = time.time() 135 | if cv2.waitKey(1) & 0xFF == ord('q'): 136 | break 137 | 138 | # When everything is done, release the capture 139 | video_capture.release() 140 | out.release() 141 | cv2.destroyAllWindows() 142 | -------------------------------------------------------------------------------- /predict_pic.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | import cv2 4 | from constants import * 5 | from train import EmotionRecognition 6 | import numpy as np 7 | 8 | cascade_classifier = cv2.CascadeClassifier(CASC_PATH) 9 | 10 | # Load Model 11 | network = EmotionRecognition() 12 | network.build_network() 13 | 14 | 15 | def face_predict(face, gray): 16 | gray = gray[face[1]:(face[1] + face[2]), face[0]:(face[0] + face[3])] 17 | try: 18 | gray = cv2.resize(gray, (SIZE_FACE, SIZE_FACE), 19 | interpolation=cv2.INTER_CUBIC) / 255. 20 | except Exception: 21 | print("[+] Problem during resize") 22 | return None 23 | 24 | # print(gray) 25 | result = network.predict(gray) 26 | # print(result) 27 | return result 28 | 29 | 30 | if __name__ == '__main__': 31 | pic_path = 'pics/1happy.jpg' 32 | 33 | image = cv2.imread(pic_path) 34 | 35 | if len(image.shape) > 2 and image.shape[2] == 3: 36 | gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) 37 | else: 38 | gray = cv2.imencode(image, cv2.IMREAD_GRAYSCALE) 39 | 40 | faces = cascade_classifier.detectMultiScale( 41 | gray, 42 | scaleFactor=1.1, 43 | minNeighbors=5 44 | ) 45 | for face in faces: 46 | (x, y, w, h) = face 47 | cv2.rectangle(image, (x,y), (x+w,y+h), (255,0,0), 2) 48 | print(x, y, w, h) 49 | result = face_predict(face, gray) 50 | text = EMOTIONS[np.argmax(result[0])] 51 | print(result, text) 52 | cv2.putText(image, text, (x, y), cv2.FONT_HERSHEY_PLAIN, 1, (0, 255, 0), 1) 53 | 54 | # cv2.imshow('happy.jpg', image) 55 | cv2.imwrite('detected_faces.png', image) -------------------------------------------------------------------------------- /tflearn_logs/README: -------------------------------------------------------------------------------- 1 | DIR to save logs 2 | -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | from __future__ import division, absolute_import 4 | from dataset_loader import DatasetLoader 5 | import tflearn 6 | from tflearn.layers.core import input_data, dropout, fully_connected 7 | from tflearn.layers.conv import conv_2d, max_pool_2d 8 | from tflearn.layers.estimator import regression 9 | from constants import * 10 | from os.path import isfile, join 11 | 12 | 13 | class EmotionRecognition: 14 | def __init__(self): 15 | self.dataset = DatasetLoader() 16 | 17 | def build_network(self): 18 | print('[+] Building CNN') 19 | self.network = input_data(shape=[None, SIZE_FACE, SIZE_FACE, 1]) 20 | self.network = conv_2d(self.network, 64, 5, activation='relu') 21 | self.network = max_pool_2d(self.network, 3, strides=2) 22 | self.network = conv_2d(self.network, 64, 5, activation='relu') 23 | self.network = max_pool_2d(self.network, 3, strides=2) 24 | self.network = conv_2d(self.network, 128, 4, activation='relu') 25 | self.network = dropout(self.network, 0.3) 26 | self.network = fully_connected(self.network, 3072, activation='relu') 27 | self.network = fully_connected( 28 | self.network, len(EMOTIONS), activation='softmax') 29 | self.network = regression( 30 | self.network, 31 | optimizer='momentum', 32 | loss='categorical_crossentropy' 33 | ) 34 | self.model = tflearn.DNN( 35 | self.network, 36 | max_checkpoints=1, 37 | tensorboard_verbose=2, 38 | tensorboard_dir="tflearn_logs" 39 | ) 40 | self.load_model() 41 | 42 | def load_saved_dataset(self): 43 | self.dataset.load_from_save() 44 | print('[+] Dataset found and loaded') 45 | 46 | def start_training(self): 47 | self.load_saved_dataset() 48 | self.build_network() 49 | if self.dataset is None: 50 | self.load_saved_dataset() 51 | # Training 52 | print('[+] Training network') 53 | self.model.fit( 54 | self.dataset.images, self.dataset.labels, 55 | validation_set=(self.dataset.images_test, 56 | self.dataset.labels_test), 57 | n_epoch=100, 58 | batch_size=50, 59 | shuffle=True, 60 | show_metric=True, 61 | snapshot_step=200, 62 | snapshot_epoch=True, 63 | run_id='FER' 64 | ) 65 | 66 | def predict(self, image): 67 | if image is None: 68 | return None 69 | image = image.reshape([-1, SIZE_FACE, SIZE_FACE, 1]) 70 | return self.model.predict(image) 71 | 72 | def save_model(self): 73 | self.model.save(join(SAVE_DIRECTORY, SAVE_MODEL_FILENAME)) 74 | print('[+] Model trained and saved at ' + SAVE_MODEL_FILENAME) 75 | 76 | def load_model(self): 77 | if isfile(join(SAVE_DIRECTORY, SAVE_MODEL_FILENAME)): 78 | self.model.load(join(SAVE_DIRECTORY, SAVE_MODEL_FILENAME)) 79 | print('[+] Model loaded from ' + SAVE_MODEL_FILENAME) 80 | 81 | 82 | if __name__ == "__main__": 83 | network = EmotionRecognition() 84 | network.start_training() 85 | network.save_model() 86 | 87 | --------------------------------------------------------------------------------