├── README.md ├── collect └── PalmTracker.py ├── config.py ├── data ├── down │ ├── saved_0000.jpg │ └── saved_0001.jpg ├── left │ ├── saved_0000.jpg │ └── saved_0001.jpg ├── pause │ ├── saved_0000.jpg │ └── saved_0001.jpg ├── right │ ├── saved_0000.jpg │ └── saved_0001.jpg └── up │ ├── saved_0000.jpg │ └── saved_0001.jpg ├── demo.py ├── game.py ├── images ├── Blinky.png ├── Clyde.png ├── Inky.png ├── Pinky.png ├── Trollman.png └── pacman.jpg ├── pacman.py ├── src └── freesansbold.ttf ├── test.jpg ├── tools ├── test.py └── train.py └── utils.py /README.md: -------------------------------------------------------------------------------- 1 | # PaddlePaddle实现手势识别玩转吃豆豆！ 2 | 3 | ![](https://ai-studio-static-online.cdn.bcebos.com/2207b0c19f2944cd997f003f3ddb8b8fc735d852ba584d1dadb12bb5c94d0697) 4 | 5 | 6 | ## 文章目录： 7 | 8 | ### 1. 手势数据采集 9 | ### 2. PaddleX训练模型 10 | ### 3. 测试手势识别模型 11 | ### 4. 测试游戏种手势控制 12 | ### 5. 大功告成~ 13 | 14 | 15 | ```python 16 | # 解压代码 17 | !unzip /home/aistudio/data/data41298/code.zip -d /home/aistudio/work/ 18 | ``` 19 | 20 | 21 | ```python 22 | !pip install paddlex 23 | ``` 24 | 25 | 拳头表示向下走： 26 | 27 | ![](https://ai-studio-static-online.cdn.bcebos.com/968be5f9b32f4840a0c3d7d4fd477359a6a41afec8cb400f81557827ae65b5de) 28 | 29 | 手掌表示向上走： 30 | 31 | ![](https://ai-studio-static-online.cdn.bcebos.com/a380aea8ff944a9fbc6f3677fb90b418aa8e05fdf6c1419495e1ad8cd49aa613) 32 | 33 | 下面两个分别是向左和向右： 34 | 35 | ![](https://ai-studio-static-online.cdn.bcebos.com/3b1a1c71eb9941e7ba41cc11cc3444f3994c17b8373f4d81b7ec7d8b18013e8f) 36 | ![](https://ai-studio-static-online.cdn.bcebos.com/0155857100374323bf167f0fdb90973e1ef9b0f4880d408080e5e83a1d740397) 37 | 38 | 空白表示按位不动： 39 | 40 | ![](https://ai-studio-static-online.cdn.bcebos.com/f0d252786c2a428abc721c786b53d0244ca803b928f541dc8b2cbb5085f204b2) 41 | 42 | 43 | 44 | ```python 45 | # 设置工作路径 46 | import os 47 | os.chdir('/home/aistudio/work/Pacman-master/') 48 | ``` 49 | 50 | ## 1. 手势数据采集： 51 | 52 | 这一步需要在本地运行collect文件夹下PalmTracker.py文件进行手势数据采集； 53 | 54 | 运行该程序时会打开摄像头，在指定区域做出手势，按s保存； 55 | 56 | ![](https://ai-studio-static-online.cdn.bcebos.com/06c664b20f794a4da1d3cffc5c57aad616dc037a5c8542fa802c879bb67fdac9) 57 | 58 | 59 | 60 | ```python 61 | # !python collect/PalmTracker.py 62 | ``` 63 | 64 | collect data game.py pacman.py test.jpg utils.py 65 | config.py demo.py images src tools weights 66 | 67 | 68 | ## 2. PaddleX训练模型 69 | 70 | 这一步使用PaddleX提供的ResNet18进行训练； 71 | 72 | 预训练模型使用在'IMAGENET'上训练的权重，PaddleX选择参数 pretrain_weights='IMAGENET' 即可； 73 | 74 | 我这里每种手势共收集了40张左右，训练结果准确率在93%以上； 75 | 76 | ### 2.1 定义数据集 77 | 78 | 79 | ```python 80 | from paddlex.cls import transforms 81 | import os 82 | import cv2 83 | import numpy as np 84 | import paddlex as pdx 85 | 86 | base = './data' 87 | 88 | with open(os.path.join('train_list.txt'), 'w') as f: 89 | for i, cls_fold in enumerate(os.listdir(base)): 90 | cls_base = os.path.join(base, cls_fold) 91 | files = os.listdir(cls_base) 92 | print('{} train num:'.format(cls_fold), len(files)) 93 | for pt in files: 94 | img = os.path.join(cls_fold, pt) 95 | info = img + ' ' + str(i) + '\n' 96 | f.write(info) 97 | 98 | with open(os.path.join('labels.txt'), 'w') as f: 99 | for i, cls_fold in enumerate(os.listdir(base)): 100 | f.write(cls_fold+'\n') 101 | 102 | train_transforms = transforms.Compose([ 103 | transforms.RandomCrop(crop_size=224), 104 | transforms.Normalize() 105 | ]) 106 | 107 | train_dataset = pdx.datasets.ImageNet( 108 | data_dir=base, 109 | file_list='train_list.txt', 110 | label_list='labels.txt', 111 | transforms=train_transforms, 112 | shuffle=True) 113 | ``` 114 | 115 | ### 2.2 使用ResNet18训练模型 116 | 117 | 此处训练20个epoch，初始学习率为2e-2 118 | 119 | 120 | ```python 121 | num_classes = len(train_dataset.labels) 122 | model = pdx.cls.ResNet18(num_classes=num_classes) 123 | model.train(num_epochs=20, 124 | train_dataset=train_dataset, 125 | train_batch_size=32, 126 | lr_decay_epochs=[5, 10, 15], 127 | learning_rate=2e-2, 128 | save_dir='w', 129 | log_interval_steps=5, 130 | save_interval_epochs=4) 131 | ``` 132 | 133 | ## 3 测试手势识别模型： 134 | 135 | 136 | ```python 137 | from paddlex.cls import transforms 138 | import matplotlib.pyplot as plt 139 | import paddlex 140 | import cv2 141 | import warnings 142 | 143 | warnings.filterwarnings('ignore') 144 | 145 | train_transforms = transforms.Compose([ 146 | transforms.RandomCrop(crop_size=224), 147 | transforms.Normalize() 148 | ]) 149 | 150 | model = paddlex.load_model('weights/final') 151 | im = cv2.imread('test.jpg') 152 | result = model.predict(im, topk=1, transforms=train_transforms) 153 | print("Predict Result:", result) 154 | 155 | %matplotlib inline 156 | plt.imshow(im) 157 | plt.show() 158 | ``` 159 | 160 | 2020-06-23 09:27:29 [INFO] Model[ResNet18] loaded. 161 | Predict Result: [{'category_id': 1, 'category': 'left', 'score': 0.9999609}] 162 | 163 | 164 | 165 | ![png](output_13_1.png) 166 | 167 | 168 | ## 4. 测试游戏中手势控制： 169 | 170 | 本地运行demo.py即可； 171 | 172 | ![](https://ai-studio-static-online.cdn.bcebos.com/39bb2143fff544e0b553e5499ec7f3f49346affa73b94f42902d2314b9d9c47d) 173 | 174 | 175 | 176 | ```python 177 | !python demo.py 178 | ``` 179 | 180 | ## 5. 大功告成 181 | 182 | 然后将该控制嵌入到游戏中即可~ 183 | 184 | 游戏代码来自：https://github.com/hbokmann/Pacman 185 | 186 | 187 | ```python 188 | !python game.py 189 | ``` 190 | 191 | ![](https://ai-studio-static-online.cdn.bcebos.com/0f06ca2879024729a5f3411a4c7fac5d370e1b7ce4754833b084b624de9187c2) 192 | 193 | 194 | ### 演示视频我放到Youtube了（因为B站审核太慢了，，，） 195 | 196 | 链接地址：[https://youtu.be/tlZT2WeaK1U](https://youtu.be/tlZT2WeaK1U) 197 | 198 | ## 更新，B站审核通过啦！ 199 | 200 | 链接地址：[https://www.bilibili.com/video/BV1xa4y1Y7Mb/](https://www.bilibili.com/video/BV1xa4y1Y7Mb/) 201 | 202 | ## 关于作者： 203 | > 北京理工大学 大二在读 204 | 205 | > 感兴趣的方向为：目标检测、人脸识别、EEG识别等 206 | 207 | > 将会定期分享一些小项目，感兴趣的朋友可以互相关注一下：[主页链接](http://aistudio.baidu.com/aistudio/personalcenter/thirdview/67156) 208 | 209 | > 也欢迎大家fork、评论交流 210 | 211 | > 作者博客主页：[https://blog.csdn.net/weixin_44936889](https://blog.csdn.net/weixin_44936889) 212 | 213 | -------------------------------------------------------------------------------- /collect/PalmTracker.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import imutils 3 | import numpy as np 4 | import argparse 5 | bg = None 6 | 7 | 8 | def run_avg(image, aWeight): 9 | global bg 10 | if bg is None: 11 | bg = image.copy().astype('float') 12 | return 13 | 14 | cv2.accumulateWeighted(image, bg, aWeight) 15 | 16 | 17 | def segment(image, threshold=25): 18 | global bg 19 | diff = cv2.absdiff(bg.astype('uint8'), image) 20 | 21 | thresholded = cv2.threshold(diff, 22 | threshold, 23 | 255, 24 | cv2.THRESH_BINARY)[1] 25 | 26 | (cnts, _) = cv2.findContours(thresholded.copy(), 27 | cv2.RETR_EXTERNAL, 28 | cv2.CHAIN_APPROX_SIMPLE) 29 | 30 | if len(cnts) == 0: 31 | return 32 | else: 33 | segmented = max(cnts, key=cv2.contourArea) 34 | return (thresholded, segmented) 35 | 36 | 37 | def main(dtype): 38 | aWeight = 0.5 39 | 40 | camera = cv2.VideoCapture(0) 41 | 42 | top, right, bottom, left = 90, 380, 285, 590 43 | 44 | num_frames = 0 45 | thresholded = None 46 | 47 | count = 0 48 | 49 | while(True): 50 | (grabbed, frame) = camera.read() 51 | if grabbed: 52 | 53 | frame = imutils.resize(frame, width=700) 54 | 55 | frame = cv2.flip(frame, 1) 56 | 57 | clone = frame.copy() 58 | 59 | (height, width) = frame.shape[:2] 60 | 61 | roi = frame[top:bottom, right:left] 62 | 63 | gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY) 64 | gray = cv2.GaussianBlur(gray, (7, 7), 0) 65 | 66 | if num_frames < 30: 67 | run_avg(gray, aWeight) 68 | else: 69 | hand = segment(gray) 70 | 71 | if hand is not None: 72 | (thresholded, segmented) = hand 73 | 74 | cv2.drawContours( 75 | clone, [segmented + (right, top)], -1, (0, 0, 255)) 76 | 77 | cv2.rectangle(clone, (left, top), (right, bottom), (0, 255, 0), 2) 78 | 79 | num_frames += 1 80 | 81 | cv2.imshow('Video Feed', clone) 82 | if not thresholded is None: 83 | cv2.imshow('Thesholded', thresholded) 84 | 85 | keypress = cv2.waitKey(1) & 0xFF 86 | 87 | if keypress == ord('q'): 88 | break 89 | 90 | if keypress == ord('s'): 91 | cv2.imwrite('data/{}/saved_v2_{:04}.jpg'.format(dtype, count), thresholded) 92 | count += 1 93 | print(count, 'saved.') 94 | 95 | else: 96 | camera.release() 97 | break 98 | 99 | if __name__ == '__main__': 100 | parser = argparse.ArgumentParser() 101 | parser.add_argument('--dtype', type=str, default='pause') 102 | args = parser.parse_args() 103 | main(args.dtype) 104 | cv2.destroyAllWindows() 105 | -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | import pygame 2 | 3 | Trollicon = pygame.image.load('images/Trollman.png') 4 | 5 | walls = [[0, 0, 6, 600], 6 | [0, 0, 600, 6], 7 | [0, 600, 606, 6], 8 | [600, 0, 6, 606], 9 | [300, 0, 6, 66], 10 | [60, 60, 186, 6], 11 | [360, 60, 186, 6], 12 | [60, 120, 66, 6], 13 | [60, 120, 6, 126], 14 | [180, 120, 246, 6], 15 | [300, 120, 6, 66], 16 | [480, 120, 66, 6], 17 | [540, 120, 6, 126], 18 | [120, 180, 126, 6], 19 | [120, 180, 6, 126], 20 | [360, 180, 126, 6], 21 | [480, 180, 6, 126], 22 | [180, 240, 6, 126], 23 | [180, 360, 246, 6], 24 | [420, 240, 6, 126], 25 | [240, 240, 42, 6], 26 | [324, 240, 42, 6], 27 | [240, 240, 6, 66], 28 | [240, 300, 126, 6], 29 | [360, 240, 6, 66], 30 | [0, 300, 66, 6], 31 | [540, 300, 66, 6], 32 | [60, 360, 66, 6], 33 | [60, 360, 6, 186], 34 | [480, 360, 66, 6], 35 | [540, 360, 6, 186], 36 | [120, 420, 366, 6], 37 | [120, 420, 6, 66], 38 | [480, 420, 6, 66], 39 | [180, 480, 246, 6], 40 | [300, 480, 6, 66], 41 | [120, 540, 126, 6], 42 | [360, 540, 126, 6] 43 | ] 44 | Pinky_directions = [ 45 | [0, -30, 4], 46 | [15, 0, 9], 47 | [0, 15, 11], 48 | [-15, 0, 23], 49 | [0, 15, 7], 50 | [15, 0, 3], 51 | [0, -15, 3], 52 | [15, 0, 19], 53 | [0, 15, 3], 54 | [15, 0, 3], 55 | [0, 15, 3], 56 | [15, 0, 3], 57 | [0, -15, 15], 58 | [-15, 0, 7], 59 | [0, 15, 3], 60 | [-15, 0, 19], 61 | [0, -15, 11], 62 | [15, 0, 9] 63 | ] 64 | 65 | Blinky_directions = [ 66 | [0, -15, 4], 67 | [15, 0, 9], 68 | [0, 15, 11], 69 | [15, 0, 3], 70 | [0, 15, 7], 71 | [-15, 0, 11], 72 | [0, 15, 3], 73 | [15, 0, 15], 74 | [0, -15, 15], 75 | [15, 0, 3], 76 | [0, -15, 11], 77 | [-15, 0, 3], 78 | [0, -15, 11], 79 | [-15, 0, 3], 80 | [0, -15, 3], 81 | [-15, 0, 7], 82 | [0, -15, 3], 83 | [15, 0, 15], 84 | [0, 15, 15], 85 | [-15, 0, 3], 86 | [0, 15, 3], 87 | [-15, 0, 3], 88 | [0, -15, 7], 89 | [-15, 0, 3], 90 | [0, 15, 7], 91 | [-15, 0, 11], 92 | [0, -15, 7], 93 | [15, 0, 5] 94 | ] 95 | 96 | Inky_directions = [ 97 | [30, 0, 2], 98 | [0, -15, 4], 99 | [15, 0, 10], 100 | [0, 15, 7], 101 | [15, 0, 3], 102 | [0, -15, 3], 103 | [15, 0, 3], 104 | [0, -15, 15], 105 | [-15, 0, 15], 106 | [0, 15, 3], 107 | [15, 0, 15], 108 | [0, 15, 11], 109 | [-15, 0, 3], 110 | [0, -15, 7], 111 | [-15, 0, 11], 112 | [0, 15, 3], 113 | [-15, 0, 11], 114 | [0, 15, 7], 115 | [-15, 0, 3], 116 | [0, -15, 3], 117 | [-15, 0, 3], 118 | [0, -15, 15], 119 | [15, 0, 15], 120 | [0, 15, 3], 121 | [-15, 0, 15], 122 | [0, 15, 11], 123 | [15, 0, 3], 124 | [0, -15, 11], 125 | [15, 0, 11], 126 | [0, 15, 3], 127 | [15, 0, 1], 128 | ] 129 | 130 | Clyde_directions = [ 131 | [-30, 0, 2], 132 | [0, -15, 4], 133 | [15, 0, 5], 134 | [0, 15, 7], 135 | [-15, 0, 11], 136 | [0, -15, 7], 137 | [-15, 0, 3], 138 | [0, 15, 7], 139 | [-15, 0, 7], 140 | [0, 15, 15], 141 | [15, 0, 15], 142 | [0, -15, 3], 143 | [-15, 0, 11], 144 | [0, -15, 7], 145 | [15, 0, 3], 146 | [0, -15, 11], 147 | [15, 0, 9], 148 | ] 149 | 150 | black = (0, 0, 0) 151 | white = (255, 255, 255) 152 | blue = (0, 0, 255) 153 | green = (0, 255, 0) 154 | red = (255, 0, 0) 155 | purple = (255, 0, 255) 156 | yellow = (255, 255, 0) 157 | CLASSES = ['pause', 'up', 'down', 'left', 'right'] 158 | 159 | w = 303-16 # Width 160 | p_h = (7*60)+19 # Pacman height 161 | m_h = (4*60)+19 # Monster height 162 | b_h = (3*60)+19 # Binky height 163 | i_w = 303-16-32 # Inky width 164 | c_w = 303+(32-16) # Clyde width -------------------------------------------------------------------------------- /data/down/saved_0000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/down/saved_0000.jpg -------------------------------------------------------------------------------- /data/down/saved_0001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/down/saved_0001.jpg -------------------------------------------------------------------------------- /data/left/saved_0000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/left/saved_0000.jpg -------------------------------------------------------------------------------- /data/left/saved_0001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/left/saved_0001.jpg -------------------------------------------------------------------------------- /data/pause/saved_0000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/pause/saved_0000.jpg -------------------------------------------------------------------------------- /data/pause/saved_0001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/pause/saved_0001.jpg -------------------------------------------------------------------------------- /data/right/saved_0000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/right/saved_0000.jpg -------------------------------------------------------------------------------- /data/right/saved_0001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/right/saved_0001.jpg -------------------------------------------------------------------------------- /data/up/saved_0000.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/up/saved_0000.jpg -------------------------------------------------------------------------------- /data/up/saved_0001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/data/up/saved_0001.jpg -------------------------------------------------------------------------------- /demo.py: -------------------------------------------------------------------------------- 1 | import paddlex 2 | from paddlex.cls import transforms 3 | import cv2 4 | import imutils 5 | import numpy as np 6 | 7 | bg = None 8 | 9 | train_transforms = transforms.Compose([ 10 | transforms.RandomCrop(crop_size=224), 11 | transforms.Normalize() 12 | ]) 13 | 14 | model = paddlex.load_model('weights/final') 15 | CLASSES = ['pause', 'up', 'down', 'left', 'right'] 16 | 17 | 18 | def run_avg(image, aWeight): 19 | global bg 20 | if bg is None: 21 | bg = image.copy().astype('float') 22 | return 23 | 24 | cv2.accumulateWeighted(image, bg, aWeight) 25 | 26 | 27 | def segment(image, threshold=25): 28 | global bg 29 | diff = cv2.absdiff(bg.astype('uint8'), image) 30 | 31 | thresholded = cv2.threshold(diff, 32 | threshold, 33 | 255, 34 | cv2.THRESH_BINARY)[1] 35 | 36 | (cnts, _) = cv2.findContours(thresholded.copy(), 37 | cv2.RETR_EXTERNAL, 38 | cv2.CHAIN_APPROX_SIMPLE) 39 | 40 | if len(cnts) == 0: 41 | return 42 | else: 43 | segmented = max(cnts, key=cv2.contourArea) 44 | return (thresholded, segmented) 45 | 46 | 47 | def main(): 48 | action = 'pause' 49 | aWeight = 0.5 50 | 51 | camera = cv2.VideoCapture(0) 52 | 53 | top, right, bottom, left = 90, 380, 285, 590 54 | 55 | num_frames = 0 56 | thresholded = None 57 | 58 | count = 0 59 | 60 | while(True): 61 | (grabbed, frame) = camera.read() 62 | if grabbed: 63 | 64 | frame = imutils.resize(frame, width=700) 65 | 66 | frame = cv2.flip(frame, 1) 67 | 68 | clone = frame.copy() 69 | 70 | (height, width) = frame.shape[:2] 71 | 72 | roi = frame[top:bottom, right:left] 73 | 74 | gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY) 75 | gray = cv2.GaussianBlur(gray, (7, 7), 0) 76 | 77 | if num_frames < 30: 78 | run_avg(gray, aWeight) 79 | else: 80 | hand = segment(gray) 81 | 82 | if hand is not None: 83 | (thresholded, segmented) = hand 84 | 85 | cv2.drawContours( 86 | clone, [segmented + (right, top)], -1, (0, 0, 255)) 87 | 88 | cv2.rectangle(clone, (left, top), (right, bottom), (0, 255, 0), 2) 89 | 90 | num_frames += 1 91 | 92 | cv2.imshow('Video Feed', clone) 93 | 94 | if not thresholded is None: 95 | 96 | input_im = cv2.merge( 97 | [thresholded, thresholded, thresholded]) 98 | result = model.predict( 99 | input_im, topk=5, transforms=train_transforms) 100 | action = result[0]['category'] 101 | cv2.putText(input_im, action, (0, 20), 102 | cv2.FONT_HERSHEY_PLAIN, 2.0, (0, 255, 0), 2) 103 | 104 | layout = np.zeros(input_im.shape) 105 | final = [] 106 | for clas in CLASSES: 107 | for v in result: 108 | if v['category'] == clas: 109 | final.append(v['score']) 110 | break 111 | 112 | for (i, score) in enumerate(final): 113 | # construct the label text 114 | text = "{}: {:.2f}%".format(CLASSES[i], score * 100) 115 | 116 | w = int(score * 300) 117 | cv2.rectangle(layout, (7, (i * 35) + 5), 118 | (w, (i * 35) + 35), (0, 0, 255), -1) 119 | cv2.putText(layout, text, (10, (i * 35) + 23), 120 | cv2.FONT_HERSHEY_SIMPLEX, 0.45, 121 | (255, 255, 255), 2) 122 | 123 | cv2.imshow('Thesholded', np.vstack([input_im, layout])) 124 | 125 | keypress = cv2.waitKey(1) & 0xFF 126 | 127 | if keypress == ord('q'): 128 | break 129 | else: 130 | camera.release() 131 | break 132 | 133 | 134 | main() 135 | cv2.destroyAllWindows() 136 | -------------------------------------------------------------------------------- /game.py: -------------------------------------------------------------------------------- 1 | 2 | import cv2 3 | import paddlex 4 | from paddlex.cls import transforms 5 | from utils import * 6 | from config import * 7 | import imutils 8 | import numpy as np 9 | 10 | train_transforms = transforms.Compose([ 11 | transforms.RandomCrop(crop_size=224), 12 | transforms.Normalize() 13 | ]) 14 | 15 | bg = None 16 | model = paddlex.load_model('weights/final') 17 | pygame.display.set_icon(Trollicon) 18 | 19 | 20 | def process_gesture(thresholded): 21 | input_im = cv2.merge( 22 | [thresholded, thresholded, thresholded]) 23 | result = model.predict( 24 | input_im, topk=5, transforms=train_transforms) 25 | gesture = result[0]['category'] 26 | cv2.putText(input_im, gesture, (0, 20), 27 | cv2.FONT_HERSHEY_PLAIN, 2.0, (0, 255, 0), 2) 28 | 29 | layout = np.zeros(input_im.shape) 30 | final = [] 31 | for clas in CLASSES: 32 | for v in result: 33 | if v['category'] == clas: 34 | final.append(v['score']) 35 | break 36 | 37 | for (i, score) in enumerate(final): 38 | # construct the label text 39 | text = "{}: {:.2f}%".format(CLASSES[i], score * 100) 40 | 41 | w = int(score * 300) 42 | cv2.rectangle(layout, (7, (i * 35) + 5), 43 | (w, (i * 35) + 35), (0, 0, 255), -1) 44 | cv2.putText(layout, text, (10, (i * 35) + 23), 45 | cv2.FONT_HERSHEY_SIMPLEX, 0.45, 46 | (255, 255, 255), 2) 47 | 48 | return gesture, input_im, layout 49 | 50 | 51 | def run_avg(image, aWeight): 52 | global bg 53 | if bg is None: 54 | bg = image.copy().astype('float') 55 | return 56 | 57 | cv2.accumulateWeighted(image, bg, aWeight) 58 | 59 | 60 | def segment(image, threshold=25): 61 | global bg 62 | diff = cv2.absdiff(bg.astype('uint8'), image) 63 | 64 | thresholded = cv2.threshold(diff, 65 | threshold, 66 | 255, 67 | cv2.THRESH_BINARY)[1] 68 | 69 | (cnts, _) = cv2.findContours(thresholded.copy(), 70 | cv2.RETR_EXTERNAL, 71 | cv2.CHAIN_APPROX_SIMPLE) 72 | 73 | if len(cnts) == 0: 74 | return 75 | else: 76 | segmented = max(cnts, key=cv2.contourArea) 77 | return (thresholded, segmented) 78 | 79 | 80 | def setupRoomOne(all_sprites_list): 81 | wall_list = pygame.sprite.RenderPlain() 82 | 83 | for item in walls: 84 | wall = Wall(item[0], item[1], item[2], item[3], blue) 85 | wall_list.add(wall) 86 | all_sprites_list.add(wall) 87 | 88 | return wall_list 89 | 90 | 91 | def setupGate(all_sprites_list): 92 | gate = pygame.sprite.RenderPlain() 93 | gate.add(Wall(282, 242, 42, 2, white)) 94 | all_sprites_list.add(gate) 95 | return gate 96 | 97 | 98 | def startGame(): 99 | 100 | all_sprites_list = pygame.sprite.RenderPlain() 101 | block_list = pygame.sprite.RenderPlain() 102 | monsta_list = pygame.sprite.RenderPlain() 103 | pacman_collide = pygame.sprite.RenderPlain() 104 | wall_list = setupRoomOne(all_sprites_list) 105 | gate = setupGate(all_sprites_list) 106 | 107 | p_turn = 0 108 | p_steps = 0 109 | b_turn = 0 110 | b_steps = 0 111 | i_turn = 0 112 | i_steps = 0 113 | c_turn = 0 114 | c_steps = 0 115 | 116 | top, right, bottom, left = 90, 360, 285, 580 117 | 118 | Pacman = Player(w, p_h, 'images/Trollman.png') 119 | all_sprites_list.add(Pacman) 120 | pacman_collide.add(Pacman) 121 | 122 | Blinky = Ghost(w, b_h, 'images/Blinky.png') 123 | monsta_list.add(Blinky) 124 | all_sprites_list.add(Blinky) 125 | 126 | Pinky = Ghost(w, m_h, 'images/Pinky.png') 127 | monsta_list.add(Pinky) 128 | all_sprites_list.add(Pinky) 129 | 130 | Inky = Ghost(i_w, m_h, 'images/Inky.png') 131 | monsta_list.add(Inky) 132 | all_sprites_list.add(Inky) 133 | 134 | Clyde = Ghost(c_w, m_h, 'images/Clyde.png') 135 | monsta_list.add(Clyde) 136 | all_sprites_list.add(Clyde) 137 | 138 | for row in range(19): 139 | for column in range(19): 140 | if (row == 7 or row == 8) and (column == 8 or column == 9 or column == 10): 141 | continue 142 | else: 143 | block = Block(yellow, 4, 4) 144 | 145 | # Set a random location for the block 146 | block.rect.x = (30*column+6)+26 147 | block.rect.y = (30*row+6)+26 148 | 149 | b_collide = pygame.sprite.spritecollide( 150 | block, wall_list, False) 151 | p_collide = pygame.sprite.spritecollide( 152 | block, pacman_collide, False) 153 | if b_collide: 154 | continue 155 | elif p_collide: 156 | continue 157 | else: 158 | # Add the block to the list of objects 159 | block_list.add(block) 160 | all_sprites_list.add(block) 161 | 162 | bll = len(block_list) 163 | score = 0 164 | num_frames = 0 165 | i = 0 166 | gesture = 'pause' 167 | thresholded = None 168 | 169 | camera = cv2.VideoCapture(0) 170 | 171 | while True: 172 | 173 | grabbed, frame = camera.read() 174 | if not grabbed: 175 | break 176 | 177 | frame = imutils.resize(frame, width=600) 178 | 179 | frame = cv2.flip(frame, 1) 180 | 181 | clone = frame.copy() 182 | 183 | (height, width) = frame.shape[:2] 184 | 185 | roi = frame[top:bottom, right:left] 186 | 187 | gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY) 188 | gray = cv2.GaussianBlur(gray, (7, 7), 0) 189 | 190 | if num_frames < 10: 191 | run_avg(gray, 0.5) 192 | 193 | else: 194 | hand = segment(gray) 195 | 196 | if hand is not None: 197 | (thresholded, segmented) = hand 198 | cv2.drawContours( 199 | clone, [segmented + (right, top)], -1, (0, 0, 255)) 200 | cv2.rectangle(clone, (left, top), (right, bottom), (0, 255, 0), 2) 201 | num_frames += 1 202 | 203 | if not thresholded is None: 204 | gesture, input_im, layout = process_gesture(thresholded) 205 | cv2.imshow('Thesholded', np.vstack([input_im, layout])) 206 | cv2.imshow('Video Feed', clone) 207 | 208 | for event in pygame.event.get(): 209 | if event.type == pygame.QUIT: 210 | pygame.quit() 211 | 212 | if gesture == 'left': 213 | Pacman.changespeed(-30, 0) 214 | if gesture == 'right': 215 | Pacman.changespeed(30, 0) 216 | if gesture == 'up': 217 | Pacman.changespeed(0, -30) 218 | if gesture == 'down': 219 | Pacman.changespeed(0, 30) 220 | if gesture == 'pause': 221 | Pacman.changespeed(0, 0) 222 | 223 | Pacman.update(wall_list, gate) 224 | 225 | returned = Pinky.changespeed( 226 | Pinky_directions, False, p_turn, p_steps, pl) 227 | p_turn = returned[0] 228 | p_steps = returned[1] 229 | Pinky.changespeed(Pinky_directions, False, p_turn, p_steps, pl) 230 | Pinky.update(wall_list, False) 231 | 232 | returned = Blinky.changespeed( 233 | Blinky_directions, False, b_turn, b_steps, bl) 234 | b_turn = returned[0] 235 | b_steps = returned[1] 236 | Blinky.changespeed(Blinky_directions, False, b_turn, b_steps, bl) 237 | Blinky.update(wall_list, False) 238 | 239 | returned = Inky.changespeed( 240 | Inky_directions, False, i_turn, i_steps, il) 241 | i_turn = returned[0] 242 | i_steps = returned[1] 243 | Inky.changespeed(Inky_directions, False, i_turn, i_steps, il) 244 | Inky.update(wall_list, False) 245 | 246 | returned = Clyde.changespeed( 247 | Clyde_directions, 'clyde', c_turn, c_steps, cl) 248 | c_turn = returned[0] 249 | c_steps = returned[1] 250 | Clyde.changespeed(Clyde_directions, 'clyde', c_turn, c_steps, cl) 251 | Clyde.update(wall_list, False) 252 | 253 | # See if the Pacman block has collided with anything. 254 | blocks_hit_list = pygame.sprite.spritecollide(Pacman, block_list, True) 255 | 256 | # Check the list of collisions. 257 | if len(blocks_hit_list) > 0: 258 | score += len(blocks_hit_list) 259 | 260 | screen.fill(black) 261 | 262 | wall_list.draw(screen) 263 | gate.draw(screen) 264 | all_sprites_list.draw(screen) 265 | monsta_list.draw(screen) 266 | 267 | text = font.render('Score: '+str(score)+'/'+str(bll), True, red) 268 | screen.blit(text, [10, 10]) 269 | 270 | if score == bll: 271 | doNext('Congratulations, you won!', 145, all_sprites_list, 272 | block_list, monsta_list, pacman_collide, wall_list, gate) 273 | 274 | monsta_hit_list = pygame.sprite.spritecollide( 275 | Pacman, monsta_list, False) 276 | 277 | if monsta_hit_list: 278 | doNext('Game Over', 235, all_sprites_list, block_list, 279 | monsta_list, pacman_collide, wall_list, gate) 280 | 281 | # ALL CODE TO DRAW SHOULD GO ABOVE THIS COMMENT 282 | try: 283 | pygame.display.flip() 284 | except Exception as e: 285 | return 286 | clock.tick(10) 287 | 288 | 289 | def doNext(message, left, all_sprites_list, block_list, monsta_list, pacman_collide, wall_list, gate): 290 | while True: 291 | try: 292 | # ALL EVENT PROCESSING SHOULD GO BELOW THIS COMMENT 293 | for event in pygame.event.get(): 294 | if event.type == pygame.QUIT: 295 | pygame.quit() 296 | if event.type == pygame.KEYDOWN: 297 | if event.key == pygame.K_ESCAPE: 298 | pygame.quit() 299 | if event.key == pygame.K_RETURN: 300 | del all_sprites_list 301 | del block_list 302 | del monsta_list 303 | del pacman_collide 304 | del wall_list 305 | del gate 306 | startGame() 307 | 308 | # Grey background 309 | w = pygame.Surface((400, 200)) # the size of your rect 310 | w.set_alpha(10) # alpha level 311 | w.fill((128, 128, 128)) # this fills the entire surface 312 | screen.blit(w, (100, 200)) # (0,0) are the top-left coordinates 313 | 314 | #Won or lost 315 | text1 = font.render(message, True, white) 316 | screen.blit(text1, [left, 233]) 317 | 318 | text2 = font.render('To play again, press ENTER.', True, white) 319 | screen.blit(text2, [135, 303]) 320 | text3 = font.render('To quit, press ESCAPE.', True, white) 321 | screen.blit(text3, [165, 333]) 322 | 323 | pygame.display.flip() 324 | 325 | clock.tick(10) 326 | 327 | except Exception as e: 328 | break 329 | 330 | 331 | if __name__ == '__main__': 332 | 333 | pl = len(Pinky_directions)-1 334 | bl = len(Blinky_directions)-1 335 | il = len(Inky_directions)-1 336 | cl = len(Clyde_directions)-1 337 | 338 | pygame.init() 339 | 340 | screen = pygame.display.set_mode([606, 606]) 341 | 342 | pygame.display.set_caption('Pacman') 343 | 344 | background = pygame.Surface(screen.get_size()) 345 | 346 | background = background.convert() 347 | 348 | background.fill(black) 349 | 350 | clock = pygame.time.Clock() 351 | 352 | pygame.font.init() 353 | font = pygame.font.Font('src/freesansbold.ttf', 24) 354 | 355 | startGame() 356 | 357 | pygame.quit() 358 | -------------------------------------------------------------------------------- /images/Blinky.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/images/Blinky.png -------------------------------------------------------------------------------- /images/Clyde.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/images/Clyde.png -------------------------------------------------------------------------------- /images/Inky.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/images/Inky.png -------------------------------------------------------------------------------- /images/Pinky.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/images/Pinky.png -------------------------------------------------------------------------------- /images/Trollman.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/images/Trollman.png -------------------------------------------------------------------------------- /images/pacman.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/images/pacman.jpg -------------------------------------------------------------------------------- /pacman.py: -------------------------------------------------------------------------------- 1 | # Pacman in Python with PyGame 2 | # https://github.com/hbokmann/Pacman 3 | 4 | import pygame 5 | from config import * 6 | 7 | black = (0, 0, 0) 8 | white = (255, 255, 255) 9 | blue = (0, 0, 255) 10 | green = (0, 255, 0) 11 | red = (255, 0, 0) 12 | purple = (255, 0, 255) 13 | yellow = (255, 255, 0) 14 | 15 | Trollicon = pygame.image.load('images/Trollman.png') 16 | pygame.display.set_icon(Trollicon) 17 | 18 | 19 | class Wall(pygame.sprite.Sprite): 20 | def __init__(self, x, y, width, height, color): 21 | pygame.sprite.Sprite.__init__(self) 22 | 23 | self.image = pygame.Surface([width, height]) 24 | self.image.fill(color) 25 | 26 | self.rect = self.image.get_rect() 27 | self.rect.top = y 28 | self.rect.left = x 29 | 30 | 31 | def setupRoomOne(all_sprites_list): 32 | wall_list = pygame.sprite.RenderPlain() 33 | 34 | for item in walls: 35 | wall = Wall(item[0], item[1], item[2], item[3], blue) 36 | wall_list.add(wall) 37 | all_sprites_list.add(wall) 38 | 39 | return wall_list 40 | 41 | 42 | def setupGate(all_sprites_list): 43 | gate = pygame.sprite.RenderPlain() 44 | gate.add(Wall(282, 242, 42, 2, white)) 45 | all_sprites_list.add(gate) 46 | return gate 47 | 48 | 49 | class Block(pygame.sprite.Sprite): 50 | def __init__(self, color, width, height): 51 | pygame.sprite.Sprite.__init__(self) 52 | 53 | self.image = pygame.Surface([width, height]) 54 | self.image.fill(white) 55 | self.image.set_colorkey(white) 56 | pygame.draw.ellipse(self.image, color, [0, 0, width, height]) 57 | 58 | self.rect = self.image.get_rect() 59 | 60 | 61 | class Player(pygame.sprite.Sprite): 62 | 63 | change_x = 0 64 | change_y = 0 65 | 66 | def __init__(self, x, y, filename): 67 | pygame.sprite.Sprite.__init__(self) 68 | 69 | self.image = pygame.image.load(filename).convert() 70 | 71 | self.rect = self.image.get_rect() 72 | self.rect.top = y 73 | self.rect.left = x 74 | self.prev_x = x 75 | self.prev_y = y 76 | 77 | def prevdirection(self): 78 | self.prev_x = self.change_x 79 | self.prev_y = self.change_y 80 | 81 | def changespeed(self, x, y): 82 | self.change_x += x 83 | self.change_y += y 84 | 85 | def update(self, walls, gate): 86 | 87 | old_x = self.rect.left 88 | new_x = old_x+self.change_x 89 | prev_x = old_x+self.prev_x 90 | self.rect.left = new_x 91 | 92 | old_y = self.rect.top 93 | new_y = old_y+self.change_y 94 | prev_y = old_y+self.prev_y 95 | 96 | x_collide = pygame.sprite.spritecollide(self, walls, False) 97 | if x_collide: 98 | self.rect.left = old_x 99 | else: 100 | self.rect.top = new_y 101 | 102 | y_collide = pygame.sprite.spritecollide(self, walls, False) 103 | if y_collide: 104 | self.rect.top = old_y 105 | 106 | if gate != False: 107 | gate_hit = pygame.sprite.spritecollide(self, gate, False) 108 | if gate_hit: 109 | self.rect.left = old_x 110 | self.rect.top = old_y 111 | 112 | 113 | class Ghost(Player): 114 | def changespeed(self, list, ghost, turn, steps, l): 115 | try: 116 | z = list[turn][2] 117 | if steps < z: 118 | self.change_x = list[turn][0] 119 | self.change_y = list[turn][1] 120 | steps += 1 121 | else: 122 | if turn < l: 123 | turn += 1 124 | elif ghost == 'clyde': 125 | turn = 2 126 | else: 127 | turn = 0 128 | self.change_x = list[turn][0] 129 | self.change_y = list[turn][1] 130 | steps = 0 131 | return [turn, steps] 132 | except IndexError: 133 | return [0, 0] 134 | 135 | 136 | pl = len(Pinky_directions)-1 137 | bl = len(Blinky_directions)-1 138 | il = len(Inky_directions)-1 139 | cl = len(Clyde_directions)-1 140 | 141 | pygame.init() 142 | 143 | screen = pygame.display.set_mode([606, 606]) 144 | 145 | pygame.display.set_caption('Pacman') 146 | 147 | background = pygame.Surface(screen.get_size()) 148 | 149 | background = background.convert() 150 | 151 | background.fill(black) 152 | 153 | clock = pygame.time.Clock() 154 | 155 | pygame.font.init() 156 | font = pygame.font.Font('src/freesansbold.ttf', 24) 157 | 158 | w = 303-16 # Width 159 | p_h = (7*60)+19 # Pacman height 160 | m_h = (4*60)+19 # Monster height 161 | b_h = (3*60)+19 # Binky height 162 | i_w = 303-16-32 # Inky width 163 | c_w = 303+(32-16) # Clyde width 164 | 165 | 166 | def startGame(): 167 | 168 | all_sprites_list = pygame.sprite.RenderPlain() 169 | 170 | block_list = pygame.sprite.RenderPlain() 171 | 172 | monsta_list = pygame.sprite.RenderPlain() 173 | 174 | pacman_collide = pygame.sprite.RenderPlain() 175 | 176 | wall_list = setupRoomOne(all_sprites_list) 177 | 178 | gate = setupGate(all_sprites_list) 179 | 180 | p_turn = 0 181 | p_steps = 0 182 | 183 | b_turn = 0 184 | b_steps = 0 185 | 186 | i_turn = 0 187 | i_steps = 0 188 | 189 | c_turn = 0 190 | c_steps = 0 191 | 192 | Pacman = Player(w, p_h, 'images/Trollman.png') 193 | all_sprites_list.add(Pacman) 194 | pacman_collide.add(Pacman) 195 | 196 | Blinky = Ghost(w, b_h, 'images/Blinky.png') 197 | monsta_list.add(Blinky) 198 | all_sprites_list.add(Blinky) 199 | 200 | Pinky = Ghost(w, m_h, 'images/Pinky.png') 201 | monsta_list.add(Pinky) 202 | all_sprites_list.add(Pinky) 203 | 204 | Inky = Ghost(i_w, m_h, 'images/Inky.png') 205 | monsta_list.add(Inky) 206 | all_sprites_list.add(Inky) 207 | 208 | Clyde = Ghost(c_w, m_h, 'images/Clyde.png') 209 | monsta_list.add(Clyde) 210 | all_sprites_list.add(Clyde) 211 | 212 | # Draw the grid 213 | for row in range(19): 214 | for column in range(19): 215 | if (row == 7 or row == 8) and (column == 8 or column == 9 or column == 10): 216 | continue 217 | else: 218 | block = Block(yellow, 4, 4) 219 | 220 | # Set a random location for the block 221 | block.rect.x = (30*column+6)+26 222 | block.rect.y = (30*row+6)+26 223 | 224 | b_collide = pygame.sprite.spritecollide( 225 | block, wall_list, False) 226 | p_collide = pygame.sprite.spritecollide( 227 | block, pacman_collide, False) 228 | if b_collide: 229 | continue 230 | elif p_collide: 231 | continue 232 | else: 233 | # Add the block to the list of objects 234 | block_list.add(block) 235 | all_sprites_list.add(block) 236 | 237 | bll = len(block_list) 238 | 239 | score = 0 240 | 241 | i = 0 242 | 243 | while True: 244 | # ALL EVENT PROCESSING SHOULD GO BELOW THIS COMMENT 245 | for event in pygame.event.get(): 246 | if event.type == pygame.QUIT: 247 | break 248 | 249 | if event.type == pygame.KEYDOWN: 250 | if event.key == pygame.K_LEFT: 251 | Pacman.changespeed(-30, 0) 252 | if event.key == pygame.K_RIGHT: 253 | Pacman.changespeed(30, 0) 254 | if event.key == pygame.K_UP: 255 | Pacman.changespeed(0, -30) 256 | if event.key == pygame.K_DOWN: 257 | Pacman.changespeed(0, 30) 258 | 259 | if event.type == pygame.KEYUP: 260 | if event.key == pygame.K_LEFT: 261 | Pacman.changespeed(30, 0) 262 | if event.key == pygame.K_RIGHT: 263 | Pacman.changespeed(-30, 0) 264 | if event.key == pygame.K_UP: 265 | Pacman.changespeed(0, 30) 266 | if event.key == pygame.K_DOWN: 267 | Pacman.changespeed(0, -30) 268 | 269 | Pacman.update(wall_list, gate) 270 | 271 | returned = Pinky.changespeed( 272 | Pinky_directions, False, p_turn, p_steps, pl) 273 | p_turn = returned[0] 274 | p_steps = returned[1] 275 | Pinky.changespeed(Pinky_directions, False, p_turn, p_steps, pl) 276 | Pinky.update(wall_list, False) 277 | 278 | returned = Blinky.changespeed( 279 | Blinky_directions, False, b_turn, b_steps, bl) 280 | b_turn = returned[0] 281 | b_steps = returned[1] 282 | Blinky.changespeed(Blinky_directions, False, b_turn, b_steps, bl) 283 | Blinky.update(wall_list, False) 284 | 285 | returned = Inky.changespeed( 286 | Inky_directions, False, i_turn, i_steps, il) 287 | i_turn = returned[0] 288 | i_steps = returned[1] 289 | Inky.changespeed(Inky_directions, False, i_turn, i_steps, il) 290 | Inky.update(wall_list, False) 291 | 292 | returned = Clyde.changespeed( 293 | Clyde_directions, 'clyde', c_turn, c_steps, cl) 294 | c_turn = returned[0] 295 | c_steps = returned[1] 296 | Clyde.changespeed(Clyde_directions, 'clyde', c_turn, c_steps, cl) 297 | Clyde.update(wall_list, False) 298 | 299 | # See if the Pacman block has collided with anything. 300 | blocks_hit_list = pygame.sprite.spritecollide(Pacman, block_list, True) 301 | 302 | # Check the list of collisions. 303 | if len(blocks_hit_list) > 0: 304 | score += len(blocks_hit_list) 305 | 306 | # ALL GAME LOGIC SHOULD GO ABOVE THIS COMMENT 307 | 308 | # ALL CODE TO DRAW SHOULD GO BELOW THIS COMMENT 309 | screen.fill(black) 310 | 311 | wall_list.draw(screen) 312 | gate.draw(screen) 313 | all_sprites_list.draw(screen) 314 | monsta_list.draw(screen) 315 | 316 | text = font.render('Score: '+str(score)+'/'+str(bll), True, red) 317 | screen.blit(text, [10, 10]) 318 | 319 | if score == bll: 320 | doNext('Congratulations, you won!', 145, all_sprites_list, 321 | block_list, monsta_list, pacman_collide, wall_list, gate) 322 | 323 | monsta_hit_list = pygame.sprite.spritecollide( 324 | Pacman, monsta_list, False) 325 | 326 | if monsta_hit_list: 327 | doNext('Game Over', 235, all_sprites_list, block_list, 328 | monsta_list, pacman_collide, wall_list, gate) 329 | 330 | # ALL CODE TO DRAW SHOULD GO ABOVE THIS COMMENT 331 | try: 332 | pygame.display.flip() 333 | except Exception as e: 334 | return 335 | clock.tick(10) 336 | 337 | 338 | def doNext(message, left, all_sprites_list, block_list, monsta_list, pacman_collide, wall_list, gate): 339 | while True: 340 | try: 341 | # ALL EVENT PROCESSING SHOULD GO BELOW THIS COMMENT 342 | for event in pygame.event.get(): 343 | if event.type == pygame.QUIT: 344 | pygame.quit() 345 | if event.type == pygame.KEYDOWN: 346 | if event.key == pygame.K_ESCAPE: 347 | pygame.quit() 348 | if event.key == pygame.K_RETURN: 349 | del all_sprites_list 350 | del block_list 351 | del monsta_list 352 | del pacman_collide 353 | del wall_list 354 | del gate 355 | startGame() 356 | 357 | # Grey background 358 | w = pygame.Surface((400, 200)) # the size of your rect 359 | w.set_alpha(10) # alpha level 360 | w.fill((128, 128, 128)) # this fills the entire surface 361 | screen.blit(w, (100, 200)) # (0,0) are the top-left coordinates 362 | 363 | #Won or lost 364 | text1 = font.render(message, True, white) 365 | screen.blit(text1, [left, 233]) 366 | 367 | text2 = font.render('To play again, press ENTER.', True, white) 368 | screen.blit(text2, [135, 303]) 369 | text3 = font.render('To quit, press ESCAPE.', True, white) 370 | screen.blit(text3, [165, 333]) 371 | 372 | pygame.display.flip() 373 | 374 | clock.tick(10) 375 | except Exception as e: 376 | break 377 | 378 | 379 | startGame() 380 | 381 | pygame.quit() 382 | -------------------------------------------------------------------------------- /src/freesansbold.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/src/freesansbold.ttf -------------------------------------------------------------------------------- /test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Sharpiless/play-Pacman-with-gesture-recognition-by-resnet18/7e946fc9ef5692105149952f369f6059ec4da0bf/test.jpg -------------------------------------------------------------------------------- /tools/test.py: -------------------------------------------------------------------------------- 1 | from paddlex.cls import transforms 2 | import paddlex 3 | import cv2 4 | 5 | train_transforms = transforms.Compose([ 6 | transforms.RandomCrop(crop_size=224), 7 | transforms.Normalize() 8 | ]) 9 | 10 | model = paddlex.load_model('weights/final') 11 | im = cv2.imread('test.jpg') 12 | result = model.predict(im, topk=3, transforms=train_transforms) 13 | print("Predict Result:", result) 14 | -------------------------------------------------------------------------------- /tools/train.py: -------------------------------------------------------------------------------- 1 | from paddlex.cls import transforms 2 | import os 3 | import cv2 4 | import numpy as np 5 | import paddlex as pdx 6 | 7 | base = './data' 8 | 9 | with open(os.path.join('train_list.txt'), 'w') as f: 10 | for i, cls_fold in enumerate(os.listdir(base)): 11 | cls_base = os.path.join(base, cls_fold) 12 | files = os.listdir(cls_base) 13 | print('{} train num:'.format(cls_fold), len(files)) 14 | for pt in files: 15 | img = os.path.join(cls_fold, pt) 16 | info = img + ' ' + str(i) + '\n' 17 | f.write(info) 18 | 19 | with open(os.path.join('labels.txt'), 'w') as f: 20 | for i, cls_fold in enumerate(os.listdir(base)): 21 | f.write(cls_fold+'\n') 22 | 23 | train_transforms = transforms.Compose([ 24 | transforms.RandomCrop(crop_size=224), 25 | transforms.Normalize() 26 | ]) 27 | 28 | train_dataset = pdx.datasets.ImageNet( 29 | data_dir=base, 30 | file_list='train_list.txt', 31 | label_list='labels.txt', 32 | transforms=train_transforms, 33 | shuffle=True) 34 | 35 | num_classes = len(train_dataset.labels) 36 | model = pdx.cls.ResNet18(num_classes=num_classes) 37 | model.train(num_epochs=20, 38 | train_dataset=train_dataset, 39 | train_batch_size=32, 40 | lr_decay_epochs=[5, 10, 15], 41 | learning_rate=2e-2, 42 | save_dir='w', 43 | log_interval_steps=5, 44 | save_interval_epochs=4) 45 | 46 | im = cv2.imread('test.jpg') 47 | result = model.predict('test.jpg', topk=1, transforms=train_transforms) 48 | print("Predict Result:", result) -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import pygame 2 | from config import * 3 | 4 | 5 | class Player(pygame.sprite.Sprite): 6 | 7 | change_x = 0 8 | change_y = 0 9 | 10 | def __init__(self, x, y, filename): 11 | pygame.sprite.Sprite.__init__(self) 12 | 13 | self.image = pygame.image.load(filename).convert() 14 | 15 | self.rect = self.image.get_rect() 16 | self.rect.top = y 17 | self.rect.left = x 18 | self.prev_x = x 19 | self.prev_y = y 20 | 21 | def prevdirection(self): 22 | self.prev_x = self.change_x 23 | self.prev_y = self.change_y 24 | 25 | def changespeed(self, x, y): 26 | self.change_x = x 27 | self.change_y = y 28 | 29 | def update(self, walls, gate): 30 | 31 | old_x = self.rect.left 32 | new_x = old_x+self.change_x 33 | prev_x = old_x+self.prev_x 34 | self.rect.left = new_x 35 | 36 | old_y = self.rect.top 37 | new_y = old_y+self.change_y 38 | prev_y = old_y+self.prev_y 39 | 40 | x_collide = pygame.sprite.spritecollide(self, walls, False) 41 | if x_collide: 42 | self.rect.left = old_x 43 | else: 44 | self.rect.top = new_y 45 | 46 | y_collide = pygame.sprite.spritecollide(self, walls, False) 47 | if y_collide: 48 | self.rect.top = old_y 49 | 50 | if gate != False: 51 | gate_hit = pygame.sprite.spritecollide(self, gate, False) 52 | if gate_hit: 53 | self.rect.left = old_x 54 | self.rect.top = old_y 55 | 56 | 57 | class Wall(pygame.sprite.Sprite): 58 | def __init__(self, x, y, width, height, color): 59 | pygame.sprite.Sprite.__init__(self) 60 | 61 | self.image = pygame.Surface([width, height]) 62 | self.image.fill(color) 63 | 64 | self.rect = self.image.get_rect() 65 | self.rect.top = y 66 | self.rect.left = x 67 | 68 | 69 | class Block(pygame.sprite.Sprite): 70 | def __init__(self, color, width, height): 71 | pygame.sprite.Sprite.__init__(self) 72 | 73 | self.image = pygame.Surface([width, height]) 74 | self.image.fill(white) 75 | self.image.set_colorkey(white) 76 | pygame.draw.ellipse(self.image, color, [0, 0, width, height]) 77 | 78 | self.rect = self.image.get_rect() 79 | 80 | 81 | class Ghost(Player): 82 | def changespeed(self, list, ghost, turn, steps, l): 83 | try: 84 | z = list[turn][2] 85 | if steps < z: 86 | self.change_x = list[turn][0] 87 | self.change_y = list[turn][1] 88 | steps += 1 89 | else: 90 | if turn < l: 91 | turn += 1 92 | elif ghost == 'clyde': 93 | turn = 2 94 | else: 95 | turn = 0 96 | self.change_x = list[turn][0] 97 | self.change_y = list[turn][1] 98 | steps = 0 99 | return [turn, steps] 100 | except IndexError: 101 | return [0, 0] 102 | --------------------------------------------------------------------------------