├── README.md ├── classification ├── README.md ├── __pycache__ │ ├── classification.cpython-37.pyc │ ├── classification.cpython-38.pyc │ ├── classification_copy.cpython-37.pyc │ ├── classification_copy.cpython-38.pyc │ └── classification_pose.cpython-37.pyc ├── classification.py ├── classification_pose.py ├── cls_test.txt ├── cls_train.txt ├── eval_top1.py ├── eval_top5.py ├── img │ ├── cat.jpg │ └── dog.jpg ├── logs │ ├── Epoch50-Total_Loss0.0870-Val_Loss0.0343.pth │ └── README.md ├── nets │ ├── __pycache__ │ │ ├── mobilenet.cpython-37.pyc │ │ ├── resnet50.cpython-37.pyc │ │ └── vgg16.cpython-37.pyc │ ├── mobilenet.py │ ├── resnet50.py │ └── vgg16.py ├── predict.py ├── requirements.txt ├── test.py ├── train.py ├── txt_annotation.py └── utils │ ├── __pycache__ │ ├── dataloader.cpython-37.pyc │ └── utils.cpython-37.pyc │ ├── dataloader.py │ └── utils.py ├── examples ├── media │ ├── COCO_val2014_000000000192.jpg │ ├── COCO_val2014_000000000241.jpg │ ├── COCO_val2014_000000000257.jpg │ ├── COCO_val2014_000000000294.jpg │ ├── COCO_val2014_000000000328.jpg │ ├── COCO_val2014_000000000338.jpg │ ├── COCO_val2014_000000000357.jpg │ ├── COCO_val2014_000000000360.jpg │ ├── COCO_val2014_000000000395.jpg │ ├── COCO_val2014_000000000415.jpg │ ├── COCO_val2014_000000000428.jpg │ ├── COCO_val2014_000000000459.jpg │ ├── COCO_val2014_000000000474.jpg │ ├── COCO_val2014_000000000488.jpg │ ├── COCO_val2014_000000000536.jpg │ ├── COCO_val2014_000000000544.jpg │ ├── COCO_val2014_000000000564.jpg │ ├── COCO_val2014_000000000569.jpg │ ├── COCO_val2014_000000000589.jpg │ ├── COCO_val2014_000000000623.jpg │ ├── test3.mp4 │ ├── test4.mp4 │ ├── test4_60fps.mp4 │ ├── test5.mp4 │ └── video.avi ├── tutorial_api_cpp │ ├── 01_body_from_image_default.cpp │ ├── 02_whole_body_from_image_default.cpp │ ├── 03_keypoints_from_image.cpp │ ├── 04_keypoints_from_images.cpp │ ├── 05_keypoints_from_images_multi_gpu.cpp │ ├── 06_face_from_image.cpp │ ├── 07_hand_from_image.cpp │ ├── 08_heatmaps_from_image.cpp │ ├── 09_keypoints_from_heatmaps.cpp │ ├── 10_asynchronous_custom_input.cpp │ ├── 11_asynchronous_custom_input_multi_camera.cpp │ ├── 12_asynchronous_custom_output.cpp │ ├── 13_asynchronous_custom_input_output_and_datum.cpp │ ├── 14_synchronous_custom_input.cpp │ ├── 15_synchronous_custom_preprocessing.cpp │ ├── 16_synchronous_custom_postprocessing.cpp │ ├── 17_synchronous_custom_output.cpp │ ├── 18_synchronous_custom_all_and_datum.cpp │ ├── CMakeLists.txt │ └── README.md └── tutorial_api_python │ ├── 01_body_from_image.py │ ├── 02_whole_body_from_image.py │ ├── 04_keypoints_from_images.py │ ├── 05_keypoints_from_images_multi_gpu.py │ ├── 06_face_from_image.py │ ├── 07_hand_from_image.py │ ├── 08_heatmaps_from_image.py │ ├── 09_keypoints_from_heatmaps.py │ ├── 12_asynchronous_custom_output.py │ ├── CMakeLists.txt │ ├── README.md │ └── openpose_python.py ├── keypoints_hand.png ├── keypoints_pose_25.png ├── models ├── cameraParameters │ └── flir │ │ └── 17012332.xml.example ├── face │ ├── haarcascade_frontalface_alt.xml │ └── pose_deploy.prototxt ├── getModels.bat ├── getModels.sh ├── hand │ └── pose_deploy.prototxt └── pose │ ├── body_25 │ └── pose_deploy.prototxt │ ├── coco │ └── pose_deploy_linevec.prototxt │ └── mpi │ ├── pose_deploy_linevec.prototxt │ └── pose_deploy_linevec_faster_4_stages.prototxt ├── requirements.txt ├── test.py ├── test_video.py ├── test_video_track_point.py ├── track_of_point.jpg └── 文本材料 ├── 1.作品简介.doc ├── 2.引用说明.docx └── 3.安装说明.docx /README.md: -------------------------------------------------------------------------------- 1 | # 手语识别 2 | --- 3 | ## 0、使用到的模型 4 | (1). openpose，作者：CMU-Perceptual-Computing-Lab 5 | ``` 6 | https://github.com/CMU-Perceptual-Computing-Lab/openpose 7 | ``` 8 | (2). 图像分类classification，作者：Bubbliiiing 9 | ``` 10 | https://github.com/bubbliiiing/classification-pytorch 11 | 12 | B站对应视频：https://www.bilibili.com/video/BV143411B7wg 13 | ``` 14 | (3). 手语教学视频，作者：二碳碳 15 | ``` 16 | https://www.bilibili.com/video/BV1XE41137LV 17 | ``` 18 | （感谢大佬们的开源项目和教程，都已star加三连） 19 | 20 |
21 |
22 | 23 | --- 24 | 25 | ## 1、大致思路 26 | 方法一： 将视频输入到openpose中，检测出关节点的变化轨迹，将轨迹绘制在一张图片上，把这张图片传到图像分类网络中检测属于哪个动作 27 | ``` 28 | 视频 -----> | openpose |-----> 关节点运动轨迹图-------> | 图像分类模型 | ----------> 单词分类 29 | ``` 30 | 方法二： 将视频输入到openpose中，检测出每一帧中关节点的位置，将多帧进行堆叠，形成一个三维张量，其中两个维度是图片的宽和高，一个维度是时间，然后对这个三维张量使用三维卷积进行训练和预测 31 | ``` 32 | 视频 -----> | openpose |-----> 多张关节点位置图 ---------> | 堆叠 | --------> 三维张量 -------> | 三维卷积网络 | ----------> 单词分类 33 | ``` 34 | 35 |
36 |
37 | 38 | --- 39 | 40 | ## 2、环境配置 41 | python：3.7（其他版本会导致openpose无法运行，建议使用anaconda的python环境） 42 | cuda：10 43 | cudnn：7或8应该都行 44 | （配置cuda和cudnn会比较麻烦，如果实在不想配，你可以去openpose的github网站下载使用cpu的版本，这里这个版本应该不支持cpu） 45 | 46 |
47 | 48 | 具体的配置环境方式： 49 | ### (0).python和cuda和cudnn自己装 50 |
51 | 52 | ### (1).下载文件 53 | 下载代码文件后，再从网盘下载模型和数据文件（没有这些跑不起来），网盘链接： 54 | ``` 55 | 链接：https://pan.baidu.com/s/1Q2aVVhMhSfWL4qKS9QslkQ 56 | 提取码：abcd 57 | ``` 58 | 将从github下载的文件夹和网盘下载的文件夹合并，然后就可以下一步了。 59 | （当然你大可直接找我要u盘拿完整的文件） 60 | 61 |
62 | 63 | ### (2).安装requirements.txt中的库 64 | cmd进入环境后，cd到项目文件夹下，执行指令： 65 | ``` 66 | pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple 67 | ``` 68 |
69 | 70 | ### (3).安装torch和torchvision 71 | 先下载好torch（1.2.0）和torchvision（0.4.0）的whl文件，下载地址： 72 | ``` 73 | 链接：https://pan.baidu.com/s/1QIuJfEE5qQFpXY8ZlHeLNQ 74 | 提取码：abcd 75 | 76 | （当然你依旧可直接找我拿u盘） 77 | ``` 78 | 下载好torch和torchvision的whl文件后，cmd进入环境，cd到下载文件夹下，执行指令： 79 | ``` 80 | pip install [torch或torchvision的whl文件的文件名] 81 | 82 | （先装torch再装torchvision，不然有可能会报错） 83 | ``` 84 | 85 |
86 |
87 | 88 | --- 89 | 90 | ## 3、测试运行openpose 91 | 项目文件夹下有三个文件： 92 | ``` 93 | test.py 94 | test_video.py 95 | test_video_track_point.py 96 | ``` 97 | 分别对应openpose的功能：检测图片、检测视频、检测视频并绘制关节点轨迹 98 | 具体的使用方法可以看文件中的注释部分 99 | 100 |
101 | 102 | 在test_video_track_point.py中，取消掉最后几行的注释，就可以将绘制的轨迹图送到classification中去做分类检测 103 | （不过现阶段分类器尚未做好） 104 | 105 |
106 |
107 | 108 | --- 109 | 110 | ## 4、classification的训练和使用 111 | 可以看下classification文件夹中的README.md文件，大佬已经在里边讲得很详细了 112 | 113 | -------------------------------------------------------------------------------- /classification/README.md: -------------------------------------------------------------------------------- 1 | ## Classification：分类模型在Pytorch当中的实现 2 | --- 3 | 4 | ## 目录 5 | 1. [所需环境 Environment](#所需环境) 6 | 2. [文件下载 Download](#文件下载) 7 | 3. [预测步骤 How2predict](#预测步骤) 8 | 4. [训练步骤 How2train](#训练步骤) 9 | 5. [评估步骤 How2eval](#评估步骤) 10 | 6. [参考资料 Reference](#Reference) 11 | 12 | ## 所需环境 13 | pytorch == 1.2.0 14 | 15 | ## 文件下载 16 | 训练所需的预训练权重都可以在百度云下载。 17 | 链接: https://pan.baidu.com/s/1KwH-O7EafSpfPXnCyX_DVw 18 | 提取码: 2g5y 19 | 20 | 训练所用的示例猫狗数据集也可以在百度云下载。 21 | 链接: https://pan.baidu.com/s/1Jz1nafsrRcw2Hqso4c8i0g 22 | 提取码: hmmx 23 | 24 | ## 预测步骤 25 | ### a、使用预训练权重 26 | 1. 下载完库后解压，model_data已经存在一个训练好的猫狗模型mobilenet025_catvsdog.h5，运行predict.py，输入 27 | ```python 28 | img/cat.jpg 29 | ``` 30 | ### b、使用自己训练的权重 31 | 1. 按照训练步骤训练。 32 | 2. 在classification.py文件里面，在如下部分修改model_path、classes_path、backbone和alpha使其对应训练好的文件；**model_path对应logs文件夹下面的权值文件，classes_path是model_path对应分的类，backbone对应使用的主干特征提取网络，alpha是当使用mobilenet的alpha值**。 33 | ```python 34 | _defaults = { 35 | "model_path" : 'model_data/mobilenet_catvsdog.pth', 36 | "classes_path" : 'model_data/cls_classes.txt', 37 | "input_shape" : [224,224,3], 38 | "backbone" : 'mobilenet', 39 | "cuda" : True 40 | } 41 | ``` 42 | 3. 运行predict.py，输入 43 | ```python 44 | img/cat.jpg 45 | ``` 46 | 47 | ## 训练步骤 48 | 1. datasets文件夹下存放的图片分为两部分，train里面是训练图片，test里面是测试图片。 49 | 2. 在训练之前需要首先准备好数据集，在train或者test文件里里面创建不同的文件夹，每个文件夹的名称为对应的类别名称，文件夹下面的图片为这个类的图片。文件格式可参考如下： 50 | ``` 51 | |-datasets 52 | |-train 53 | |-cat 54 | |-123.jpg 55 | |-234.jpg 56 | |-dog 57 | |-345.jpg 58 | |-456.jpg 59 | |-... 60 | |-test 61 | |-cat 62 | |-567.jpg 63 | |-678.jpg 64 | |-dog 65 | |-789.jpg 66 | |-890.jpg 67 | |-... 68 | ``` 69 | 3. 在准备好数据集后，需要在根目录运行txt_annotation.py生成训练所需的cls_train.txt，运行前需要修改其中的classes，将其修改成自己需要分的类。 70 | 4. 之后修改model_data文件夹下的cls_classes.txt，使其也对应自己需要分的类。 71 | 5. 在train.py里面调整自己要选择的网络和权重后，就可以开始训练了！ 72 | 73 | ## 评估步骤 74 | 1. datasets文件夹下存放的图片分为两部分，train里面是训练图片，test里面是测试图片，在评估的时候，我们使用的是test文件夹里面的图片。 75 | 2. 在评估之前需要首先准备好数据集，在train或者test文件里里面创建不同的文件夹，每个文件夹的名称为对应的类别名称，文件夹下面的图片为这个类的图片。文件格式可参考如下： 76 | ``` 77 | |-datasets 78 | |-train 79 | |-cat 80 | |-123.jpg 81 | |-234.jpg 82 | |-dog 83 | |-345.jpg 84 | |-456.jpg 85 | |-... 86 | |-test 87 | |-cat 88 | |-567.jpg 89 | |-678.jpg 90 | |-dog 91 | |-789.jpg 92 | |-890.jpg 93 | |-... 94 | ``` 95 | 3. 在准备好数据集后，需要在根目录运行txt_annotation.py生成评估所需的cls_test.txt，运行前需要修改其中的classes，将其修改成自己需要分的类。 96 | 4. 之后在classification.py文件里面修改如下部分model_path、classes_path、backbone和alpha使其对应训练好的文件；**model_path对应logs文件夹下面的权值文件，classes_path是model_path对应分的类，backbone对应使用的主干特征提取网络，alpha是当使用mobilenet的alpha值**。 97 | ```python 98 | _defaults = { 99 | "model_path" : 'model_data/mobilenet_catvsdog.pth', 100 | "classes_path" : 'model_data/cls_classes.txt', 101 | "input_shape" : [224,224,3], 102 | "backbone" : 'mobilenet', 103 | "cuda" : True 104 | } 105 | ``` 106 | 5. 运行eval_top1.py和eval_top5.py来进行模型准确率评估。 107 | 108 | ## Reference 109 | https://github.com/keras-team/keras-applications 110 | 111 | -------------------------------------------------------------------------------- /classification/__pycache__/classification.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/__pycache__/classification.cpython-37.pyc -------------------------------------------------------------------------------- /classification/__pycache__/classification.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/__pycache__/classification.cpython-38.pyc -------------------------------------------------------------------------------- /classification/__pycache__/classification_copy.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/__pycache__/classification_copy.cpython-37.pyc -------------------------------------------------------------------------------- /classification/__pycache__/classification_copy.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/__pycache__/classification_copy.cpython-38.pyc -------------------------------------------------------------------------------- /classification/__pycache__/classification_pose.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/__pycache__/classification_pose.cpython-37.pyc -------------------------------------------------------------------------------- /classification/classification.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import os 3 | 4 | import matplotlib.pyplot as plt 5 | import numpy as np 6 | import torch 7 | from torch import nn 8 | from torch.autograd import Variable 9 | 10 | from nets.mobilenet import mobilenet_v2 11 | from nets.resnet50 import resnet50 12 | from nets.vgg16 import vgg16 13 | from utils.utils import letterbox_image 14 | 15 | get_model_from_name = { 16 | "mobilenet":mobilenet_v2, 17 | "resnet50":resnet50, 18 | "vgg16":vgg16, 19 | } 20 | 21 | #----------------------------------------# 22 | # 预处理训练图片 23 | #----------------------------------------# 24 | def _preprocess_input(x,): 25 | x /= 127.5 26 | x -= 1. 27 | return x 28 | 29 | #--------------------------------------------# 30 | # 使用自己训练好的模型预测需要修改3个参数 31 | # model_path和classes_path和backbone都需要修改！ 32 | #--------------------------------------------# 33 | class Classification(object): 34 | _defaults = { 35 | "model_path" : 'classification/logs/Epoch50-Total_Loss0.0870-Val_Loss0.0343.pth', 36 | "classes_path" : 'classification/model_data/cls_classes.txt', 37 | "input_shape" : [224,224,3], 38 | "backbone" : 'mobilenet', 39 | "cuda" : True 40 | } 41 | 42 | @classmethod 43 | def get_defaults(cls, n): 44 | if n in cls._defaults: 45 | return cls._defaults[n] 46 | else: 47 | return "Unrecognized attribute name '" + n + "'" 48 | 49 | #---------------------------------------------------# 50 | # 初始化classification 51 | #---------------------------------------------------# 52 | def __init__(self, **kwargs): 53 | self.__dict__.update(self._defaults) 54 | self.class_names = self._get_class() 55 | self.generate() 56 | 57 | #---------------------------------------------------# 58 | # 获得所有的分类 59 | #---------------------------------------------------# 60 | def _get_class(self): 61 | classes_path = os.path.expanduser(self.classes_path) 62 | with open(classes_path) as f: 63 | class_names = f.readlines() 64 | class_names = [c.strip() for c in class_names] 65 | return class_names 66 | 67 | #---------------------------------------------------# 68 | # 获得所有的分类 69 | #---------------------------------------------------# 70 | def generate(self): 71 | model_path = os.path.expanduser(self.model_path) 72 | 73 | # 计算总的种类 74 | self.num_classes = len(self.class_names) 75 | 76 | assert self.backbone in ["mobilenet", "resnet50", "vgg16"] 77 | 78 | self.model = get_model_from_name[self.backbone](num_classes=self.num_classes, pretrained=False) 79 | 80 | self.model = self.model.eval() 81 | state_dict = torch.load(self.model_path) 82 | self.model.load_state_dict(state_dict) 83 | if self.cuda: 84 | self.model = nn.DataParallel(self.model) 85 | self.model = self.model.cuda() 86 | print('{} model, and classes loaded.'.format(model_path)) 87 | 88 | #---------------------------------------------------# 89 | # 检测图片 90 | #---------------------------------------------------# 91 | def detect_image(self, image): 92 | old_image = copy.deepcopy(image) 93 | 94 | crop_img = letterbox_image(image, [self.input_shape[0],self.input_shape[1]]) 95 | photo = np.array(crop_img,dtype = np.float32) 96 | 97 | # 图片预处理，归一化 98 | photo = np.reshape(_preprocess_input(photo),[1,self.input_shape[0],self.input_shape[1],self.input_shape[2]]) 99 | photo = np.transpose(photo,(0,3,1,2)) 100 | 101 | with torch.no_grad(): 102 | photo = Variable(torch.from_numpy(photo).type(torch.FloatTensor)) 103 | if self.cuda: 104 | photo = photo.cuda() 105 | preds = torch.softmax(self.model(photo)[0], dim=-1).cpu().numpy() 106 | 107 | class_name = self.class_names[np.argmax(preds)] 108 | probability = np.max(preds) 109 | 110 | # 显示图片 111 | plt.subplot(1, 1, 1) 112 | plt.imshow(np.array(old_image)) 113 | plt.title('Class:%s Probability:%.3f' %(class_name, probability)) 114 | plt.show() 115 | return class_name 116 | 117 | def close_session(self): 118 | self.sess.close() 119 | -------------------------------------------------------------------------------- /classification/classification_pose.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import os 3 | 4 | import matplotlib.pyplot as plt 5 | import numpy as np 6 | import torch 7 | from torch import nn 8 | from torch.autograd import Variable 9 | 10 | from classification.nets.mobilenet import mobilenet_v2 11 | from classification.nets.resnet50 import resnet50 12 | from classification.nets.vgg16 import vgg16 13 | from classification.utils.utils import letterbox_image 14 | 15 | get_model_from_name = { 16 | "mobilenet":mobilenet_v2, 17 | "resnet50":resnet50, 18 | "vgg16":vgg16, 19 | } 20 | 21 | #----------------------------------------# 22 | # 预处理训练图片 23 | #----------------------------------------# 24 | def _preprocess_input(x,): 25 | x /= 127.5 26 | x -= 1. 27 | return x 28 | 29 | #--------------------------------------------# 30 | # 使用自己训练好的模型预测需要修改3个参数 31 | # model_path和classes_path和backbone都需要修改！ 32 | #--------------------------------------------# 33 | class Classification(object): 34 | _defaults = { 35 | "model_path" : 'classification/logs/Epoch50-Total_Loss0.0870-Val_Loss0.0343.pth', 36 | "classes_path" : 'classification/model_data/cls_classes.txt', 37 | "input_shape" : [224,224,3], 38 | "backbone" : 'mobilenet', 39 | "cuda" : True 40 | } 41 | 42 | @classmethod 43 | def get_defaults(cls, n): 44 | if n in cls._defaults: 45 | return cls._defaults[n] 46 | else: 47 | return "Unrecognized attribute name '" + n + "'" 48 | 49 | #---------------------------------------------------# 50 | # 初始化classification 51 | #---------------------------------------------------# 52 | def __init__(self, **kwargs): 53 | self.__dict__.update(self._defaults) 54 | self.class_names = self._get_class() 55 | self.generate() 56 | 57 | #---------------------------------------------------# 58 | # 获得所有的分类 59 | #---------------------------------------------------# 60 | def _get_class(self): 61 | classes_path = os.path.expanduser(self.classes_path) 62 | with open(classes_path) as f: 63 | class_names = f.readlines() 64 | class_names = [c.strip() for c in class_names] 65 | return class_names 66 | 67 | #---------------------------------------------------# 68 | # 获得所有的分类 69 | #---------------------------------------------------# 70 | def generate(self): 71 | model_path = os.path.expanduser(self.model_path) 72 | 73 | # 计算总的种类 74 | self.num_classes = len(self.class_names) 75 | 76 | assert self.backbone in ["mobilenet", "resnet50", "vgg16"] 77 | 78 | self.model = get_model_from_name[self.backbone](num_classes=self.num_classes, pretrained=False) 79 | 80 | self.model = self.model.eval() 81 | state_dict = torch.load(self.model_path) 82 | self.model.load_state_dict(state_dict) 83 | if self.cuda: 84 | self.model = nn.DataParallel(self.model) 85 | self.model = self.model.cuda() 86 | print('{} model, and classes loaded.'.format(model_path)) 87 | 88 | #---------------------------------------------------# 89 | # 检测图片 90 | #---------------------------------------------------# 91 | def detect_image(self, image): 92 | old_image = copy.deepcopy(image) 93 | 94 | crop_img = letterbox_image(image, [self.input_shape[0],self.input_shape[1]]) 95 | photo = np.array(crop_img,dtype = np.float32) 96 | 97 | # 图片预处理，归一化 98 | photo = np.reshape(_preprocess_input(photo),[1,self.input_shape[0],self.input_shape[1],self.input_shape[2]]) 99 | photo = np.transpose(photo,(0,3,1,2)) 100 | 101 | with torch.no_grad(): 102 | photo = Variable(torch.from_numpy(photo).type(torch.FloatTensor)) 103 | if self.cuda: 104 | photo = photo.cuda() 105 | preds = torch.softmax(self.model(photo)[0], dim=-1).cpu().numpy() 106 | 107 | class_name = self.class_names[np.argmax(preds)] 108 | probability = np.max(preds) 109 | 110 | # 显示图片 111 | plt.subplot(1, 1, 1) 112 | plt.imshow(np.array(old_image)) 113 | plt.title('Class:%s Probability:%.3f' %(class_name, probability)) 114 | plt.show() 115 | return class_name 116 | 117 | def close_session(self): 118 | self.sess.close() 119 | -------------------------------------------------------------------------------- /classification/eval_top1.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | from PIL import Image 4 | from torch.autograd import Variable 5 | 6 | from classification import Classification, _preprocess_input 7 | from utils.utils import letterbox_image 8 | 9 | 10 | class top1_Classification(Classification): 11 | def detect_image(self, image): 12 | crop_img = letterbox_image(image, [self.input_shape[0],self.input_shape[1]]) 13 | photo = np.array(crop_img,dtype = np.float32) 14 | 15 | # 图片预处理，归一化 16 | photo = np.reshape(_preprocess_input(photo),[1,self.input_shape[0],self.input_shape[1],self.input_shape[2]]) 17 | photo = np.transpose(photo,(0,3,1,2)) 18 | 19 | with torch.no_grad(): 20 | photo = Variable(torch.from_numpy(photo).type(torch.FloatTensor)) 21 | if self.cuda: 22 | photo = photo.cuda() 23 | preds = torch.softmax(self.model(photo)[0], dim=-1).cpu().numpy() 24 | 25 | arg_pred = np.argmax(preds) 26 | return arg_pred 27 | 28 | def evaluteTop1(classfication, lines): 29 | correct = 0 30 | total = len(lines) 31 | for index, line in enumerate(lines): 32 | annotation_path = line.split(';')[1].split()[0] 33 | x = Image.open(annotation_path) 34 | y = int(line.split(';')[0]) 35 | 36 | pred = classfication.detect_image(x) 37 | correct += pred == y 38 | if index % 100 == 0: 39 | print("[%d/%d]"%(index,total)) 40 | return correct / total 41 | 42 | classfication = top1_Classification() 43 | with open(r"classification/cls_test.txt","r") as f: 44 | lines = f.readlines() 45 | top1 = evaluteTop1(classfication, lines) 46 | print("top-1 accuracy = %.2f%%" % (top1*100)) 47 | -------------------------------------------------------------------------------- /classification/eval_top5.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | from PIL import Image 4 | from torch.autograd import Variable 5 | 6 | from classification import Classification, _preprocess_input 7 | from utils.utils import letterbox_image 8 | 9 | 10 | class top5_Classification(Classification): 11 | def detect_image(self, image): 12 | crop_img = letterbox_image(image, [self.input_shape[0],self.input_shape[1]]) 13 | photo = np.array(crop_img,dtype = np.float32) 14 | 15 | # 图片预处理，归一化 16 | photo = np.reshape(_preprocess_input(photo),[1,self.input_shape[0],self.input_shape[1],self.input_shape[2]]) 17 | photo = np.transpose(photo,(0,3,1,2)) 18 | 19 | with torch.no_grad(): 20 | photo = Variable(torch.from_numpy(photo).type(torch.FloatTensor)) 21 | if self.cuda: 22 | photo = photo.cuda() 23 | preds = torch.softmax(self.model(photo)[0], dim=-1).cpu().numpy() 24 | 25 | arg_pred = np.argsort(preds)[::-1] 26 | arg_pred_top5 = arg_pred[:5] 27 | return arg_pred_top5 28 | 29 | def evaluteTop5(classfication, lines): 30 | correct = 0 31 | total = len(lines) 32 | for index, line in enumerate(lines): 33 | annotation_path = line.split(';')[1].split()[0] 34 | x = Image.open(annotation_path) 35 | y = int(line.split(';')[0]) 36 | 37 | pred = classfication.detect_image(x) 38 | correct += y in pred 39 | if index % 100 == 0: 40 | print("[%d/%d]"%(index,total)) 41 | return correct / total 42 | 43 | classfication = top5_Classification() 44 | with open(r"classification/cls_test.txt","r") as f: 45 | lines = f.readlines() 46 | top5 = evaluteTop5(classfication, lines) 47 | print("top-5 accuracy = %.2f%%" % (top5*100)) 48 | 49 | -------------------------------------------------------------------------------- /classification/img/cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/img/cat.jpg -------------------------------------------------------------------------------- /classification/img/dog.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/img/dog.jpg -------------------------------------------------------------------------------- /classification/logs/Epoch50-Total_Loss0.0870-Val_Loss0.0343.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/logs/Epoch50-Total_Loss0.0870-Val_Loss0.0343.pth -------------------------------------------------------------------------------- /classification/logs/README.md: -------------------------------------------------------------------------------- 1 | 存放训练后的模型 -------------------------------------------------------------------------------- /classification/nets/__pycache__/mobilenet.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/nets/__pycache__/mobilenet.cpython-37.pyc -------------------------------------------------------------------------------- /classification/nets/__pycache__/resnet50.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/nets/__pycache__/resnet50.cpython-37.pyc -------------------------------------------------------------------------------- /classification/nets/__pycache__/vgg16.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/nets/__pycache__/vgg16.cpython-37.pyc -------------------------------------------------------------------------------- /classification/nets/mobilenet.py: -------------------------------------------------------------------------------- 1 | from torch import nn 2 | from torchvision.models.utils import load_state_dict_from_url 3 | 4 | 5 | __all__ = ['MobileNetV2', 'mobilenet_v2'] 6 | 7 | 8 | model_urls = { 9 | 'mobilenet_v2': 'https://download.pytorch.org/models/mobilenet_v2-b0353104.pth', 10 | } 11 | 12 | 13 | def _make_divisible(v, divisor, min_value=None): 14 | if min_value is None: 15 | min_value = divisor 16 | new_v = max(min_value, int(v + divisor / 2) // divisor * divisor) 17 | if new_v < 0.9 * v: 18 | new_v += divisor 19 | return new_v 20 | 21 | class ConvBNReLU(nn.Sequential): 22 | def __init__(self, in_planes, out_planes, kernel_size=3, stride=1, groups=1): 23 | padding = (kernel_size - 1) // 2 24 | super(ConvBNReLU, self).__init__( 25 | nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=False), 26 | nn.BatchNorm2d(out_planes), 27 | nn.ReLU6(inplace=True) 28 | ) 29 | 30 | class InvertedResidual(nn.Module): 31 | def __init__(self, inp, oup, stride, expand_ratio): 32 | super(InvertedResidual, self).__init__() 33 | self.stride = stride 34 | assert stride in [1, 2] 35 | 36 | hidden_dim = int(round(inp * expand_ratio)) 37 | self.use_res_connect = self.stride == 1 and inp == oup 38 | 39 | layers = [] 40 | if expand_ratio != 1: 41 | layers.append(ConvBNReLU(inp, hidden_dim, kernel_size=1)) 42 | layers.extend([ 43 | ConvBNReLU(hidden_dim, hidden_dim, stride=stride, groups=hidden_dim), 44 | nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False), 45 | nn.BatchNorm2d(oup), 46 | ]) 47 | self.conv = nn.Sequential(*layers) 48 | 49 | def forward(self, x): 50 | if self.use_res_connect: 51 | return x + self.conv(x) 52 | else: 53 | return self.conv(x) 54 | 55 | 56 | class MobileNetV2(nn.Module): 57 | def __init__(self, num_classes=1000, width_mult=1.0, inverted_residual_setting=None, round_nearest=8): 58 | super(MobileNetV2, self).__init__() 59 | block = InvertedResidual 60 | input_channel = 32 61 | last_channel = 1280 62 | 63 | if inverted_residual_setting is None: 64 | inverted_residual_setting = [ 65 | # t, c, n, s 66 | # 112, 112, 32 -> 112, 112, 16 67 | [1, 16, 1, 1], 68 | # 112, 112, 16 -> 56, 56, 24 69 | [6, 24, 2, 2], 70 | # 56, 56, 24 -> 28, 28, 32 71 | [6, 32, 3, 2], 72 | # 28, 28, 32 -> 14, 14, 64 73 | [6, 64, 4, 2], 74 | # 14, 14, 64 -> 14, 14, 96 75 | [6, 96, 3, 1], 76 | # 14, 14, 96 -> 7, 7, 160 77 | [6, 160, 3, 2], 78 | # 7, 7, 160 -> 7, 7, 320 79 | [6, 320, 1, 1], 80 | ] 81 | 82 | if len(inverted_residual_setting) == 0 or len(inverted_residual_setting[0]) != 4: 83 | raise ValueError("inverted_residual_setting should be non-empty " 84 | "or a 4-element list, got {}".format(inverted_residual_setting)) 85 | 86 | input_channel = _make_divisible(input_channel * width_mult, round_nearest) 87 | self.last_channel = _make_divisible(last_channel * max(1.0, width_mult), round_nearest) 88 | 89 | # 224, 224, 3 -> 112, 112, 32 90 | features = [ConvBNReLU(3, input_channel, stride=2)] 91 | 92 | for t, c, n, s in inverted_residual_setting: 93 | output_channel = _make_divisible(c * width_mult, round_nearest) 94 | for i in range(n): 95 | stride = s if i == 0 else 1 96 | features.append(block(input_channel, output_channel, stride, expand_ratio=t)) 97 | input_channel = output_channel 98 | 99 | # 7, 7, 320 -> 7,7,1280 100 | features.append(ConvBNReLU(input_channel, self.last_channel, kernel_size=1)) 101 | self.features = nn.Sequential(*features) 102 | 103 | self.classifier = nn.Sequential( 104 | nn.Dropout(0.2), 105 | nn.Linear(self.last_channel, num_classes), 106 | ) 107 | 108 | for m in self.modules(): 109 | if isinstance(m, nn.Conv2d): 110 | nn.init.kaiming_normal_(m.weight, mode='fan_out') 111 | if m.bias is not None: 112 | nn.init.zeros_(m.bias) 113 | elif isinstance(m, nn.BatchNorm2d): 114 | nn.init.ones_(m.weight) 115 | nn.init.zeros_(m.bias) 116 | elif isinstance(m, nn.Linear): 117 | nn.init.normal_(m.weight, 0, 0.01) 118 | nn.init.zeros_(m.bias) 119 | 120 | def forward(self, x): 121 | x = self.features(x) 122 | # 1280 123 | x = x.mean([2, 3]) 124 | x = self.classifier(x) 125 | return x 126 | 127 | def freeze_backbone(self): 128 | for param in self.features.parameters(): 129 | param.requires_grad = False 130 | 131 | def Unfreeze_backbone(self): 132 | for param in self.features.parameters(): 133 | param.requires_grad = True 134 | 135 | 136 | def mobilenet_v2(pretrained=False, progress=True, num_classes=1000): 137 | model = MobileNetV2() 138 | if pretrained: 139 | state_dict = load_state_dict_from_url(model_urls['mobilenet_v2'], model_dir='./model_data', 140 | progress=progress) 141 | model.load_state_dict(state_dict) 142 | 143 | if num_classes!=1000: 144 | model.classifier = nn.Sequential( 145 | nn.Dropout(0.2), 146 | nn.Linear(model.last_channel, num_classes), 147 | ) 148 | return model 149 | -------------------------------------------------------------------------------- /classification/nets/resnet50.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from torchvision.models.utils import load_state_dict_from_url 4 | 5 | model_urls = { 6 | 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth', 7 | } 8 | 9 | 10 | def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1): 11 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, 12 | padding=dilation, groups=groups, bias=False, dilation=dilation) 13 | 14 | 15 | def conv1x1(in_planes, out_planes, stride=1): 16 | return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False) 17 | 18 | 19 | 20 | class Bottleneck(nn.Module): 21 | expansion = 4 22 | def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1, 23 | base_width=64, dilation=1, norm_layer=None): 24 | super(Bottleneck, self).__init__() 25 | if norm_layer is None: 26 | norm_layer = nn.BatchNorm2d 27 | width = int(planes * (base_width / 64.)) * groups 28 | # Both self.conv2 and self.downsample layers downsample the input when stride != 1 29 | self.conv1 = conv1x1(inplanes, width) 30 | self.bn1 = norm_layer(width) 31 | 32 | self.conv2 = conv3x3(width, width, stride, groups, dilation) 33 | self.bn2 = norm_layer(width) 34 | 35 | self.conv3 = conv1x1(width, planes * self.expansion) 36 | self.bn3 = norm_layer(planes * self.expansion) 37 | self.relu = nn.ReLU(inplace=True) 38 | self.downsample = downsample 39 | self.stride = stride 40 | 41 | def forward(self, x): 42 | identity = x 43 | 44 | out = self.conv1(x) 45 | out = self.bn1(out) 46 | out = self.relu(out) 47 | 48 | out = self.conv2(out) 49 | out = self.bn2(out) 50 | out = self.relu(out) 51 | 52 | out = self.conv3(out) 53 | out = self.bn3(out) 54 | 55 | if self.downsample is not None: 56 | identity = self.downsample(x) 57 | 58 | out += identity 59 | out = self.relu(out) 60 | 61 | return out 62 | 63 | 64 | class ResNet(nn.Module): 65 | 66 | def __init__(self, block, layers, num_classes=1000, zero_init_residual=False, 67 | groups=1, width_per_group=64, replace_stride_with_dilation=None, 68 | norm_layer=None): 69 | super(ResNet, self).__init__() 70 | if norm_layer is None: 71 | norm_layer = nn.BatchNorm2d 72 | self._norm_layer = norm_layer 73 | 74 | self.inplanes = 64 75 | self.dilation = 1 76 | if replace_stride_with_dilation is None: 77 | replace_stride_with_dilation = [False, False, False] 78 | 79 | if len(replace_stride_with_dilation) != 3: 80 | raise ValueError("replace_stride_with_dilation should be None " 81 | "or a 3-element tuple, got {}".format(replace_stride_with_dilation)) 82 | 83 | self.block = block 84 | self.groups = groups 85 | self.base_width = width_per_group 86 | 87 | # 224,224,3 -> 112,112,64 88 | self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3, 89 | bias=False) 90 | self.bn1 = norm_layer(self.inplanes) 91 | self.relu = nn.ReLU(inplace=True) 92 | 93 | # 112,112,64 -> 56,56,64 94 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 95 | 96 | # 56,56,64 -> 56,56,256 97 | self.layer1 = self._make_layer(block, 64, layers[0]) 98 | 99 | # 56,56,256 -> 28,28,512 100 | self.layer2 = self._make_layer(block, 128, layers[1], stride=2, 101 | dilate=replace_stride_with_dilation[0]) 102 | 103 | # 28,28,512 -> 14,14,1024 104 | self.layer3 = self._make_layer(block, 256, layers[2], stride=2, 105 | dilate=replace_stride_with_dilation[1]) 106 | 107 | # 14,14,1024 -> 7,7,2048 108 | self.layer4 = self._make_layer(block, 512, layers[3], stride=2, 109 | dilate=replace_stride_with_dilation[2]) 110 | 111 | # 7,7,2048 -> 2048 112 | self.avgpool = nn.AdaptiveAvgPool2d((1, 1)) 113 | 114 | # 2048 -> num_classes 115 | self.fc = nn.Linear(512 * block.expansion, num_classes) 116 | 117 | for m in self.modules(): 118 | if isinstance(m, nn.Conv2d): 119 | nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu') 120 | elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)): 121 | nn.init.constant_(m.weight, 1) 122 | nn.init.constant_(m.bias, 0) 123 | 124 | if zero_init_residual: 125 | for m in self.modules(): 126 | if isinstance(m, Bottleneck): 127 | nn.init.constant_(m.bn3.weight, 0) 128 | 129 | def _make_layer(self, block, planes, blocks, stride=1, dilate=False): 130 | norm_layer = self._norm_layer 131 | downsample = None 132 | previous_dilation = self.dilation 133 | if dilate: 134 | self.dilation *= stride 135 | stride = 1 136 | if stride != 1 or self.inplanes != planes * block.expansion: 137 | downsample = nn.Sequential( 138 | conv1x1(self.inplanes, planes * block.expansion, stride), 139 | norm_layer(planes * block.expansion), 140 | ) 141 | 142 | layers = [] 143 | # Conv_block 144 | layers.append(block(self.inplanes, planes, stride, downsample, self.groups, 145 | self.base_width, previous_dilation, norm_layer)) 146 | self.inplanes = planes * block.expansion 147 | for _ in range(1, blocks): 148 | # identity_block 149 | layers.append(block(self.inplanes, planes, groups=self.groups, 150 | base_width=self.base_width, dilation=self.dilation, 151 | norm_layer=norm_layer)) 152 | 153 | return nn.Sequential(*layers) 154 | 155 | def forward(self, x): 156 | x = self.conv1(x) 157 | x = self.bn1(x) 158 | x = self.relu(x) 159 | x = self.maxpool(x) 160 | 161 | x = self.layer1(x) 162 | x = self.layer2(x) 163 | x = self.layer3(x) 164 | x = self.layer4(x) 165 | 166 | x = self.avgpool(x) 167 | x = torch.flatten(x, 1) 168 | x = self.fc(x) 169 | 170 | return x 171 | 172 | def freeze_backbone(self): 173 | backbone = [self.conv1, self.bn1, self.layer1, self.layer2, self.layer3, self.layer4] 174 | for module in backbone: 175 | for param in module.parameters(): 176 | param.requires_grad = False 177 | 178 | def Unfreeze_backbone(self): 179 | backbone = [self.conv1, self.bn1, self.layer1, self.layer2, self.layer3, self.layer4] 180 | for module in backbone: 181 | for param in module.parameters(): 182 | param.requires_grad = True 183 | 184 | 185 | def resnet50(pretrained=False, progress=True, num_classes=1000): 186 | model = ResNet(Bottleneck, [3, 4, 6, 3]) 187 | if pretrained: 188 | state_dict = load_state_dict_from_url(model_urls['resnet50'], model_dir='./model_data', 189 | progress=progress) 190 | model.load_state_dict(state_dict) 191 | 192 | if num_classes!=1000: 193 | model.fc = nn.Linear(512 * model.block.expansion, num_classes) 194 | return model 195 | -------------------------------------------------------------------------------- /classification/nets/vgg16.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from torchvision.models.utils import load_state_dict_from_url 4 | 5 | model_urls = { 6 | 'vgg16': 'https://download.pytorch.org/models/vgg16-397923af.pth', 7 | } 8 | 9 | 10 | class VGG(nn.Module): 11 | def __init__(self, features, num_classes=1000, init_weights=True): 12 | super(VGG, self).__init__() 13 | self.features = features 14 | self.avgpool = nn.AdaptiveAvgPool2d((7, 7)) 15 | self.classifier = nn.Sequential( 16 | nn.Linear(512 * 7 * 7, 4096), 17 | nn.ReLU(True), 18 | nn.Dropout(), 19 | nn.Linear(4096, 4096), 20 | nn.ReLU(True), 21 | nn.Dropout(), 22 | nn.Linear(4096, num_classes), 23 | ) 24 | if init_weights: 25 | self._initialize_weights() 26 | 27 | def forward(self, x): 28 | x = self.features(x) 29 | x = self.avgpool(x) 30 | x = torch.flatten(x, 1) 31 | x = self.classifier(x) 32 | return x 33 | 34 | def _initialize_weights(self): 35 | for m in self.modules(): 36 | if isinstance(m, nn.Conv2d): 37 | nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu') 38 | if m.bias is not None: 39 | nn.init.constant_(m.bias, 0) 40 | elif isinstance(m, nn.BatchNorm2d): 41 | nn.init.constant_(m.weight, 1) 42 | nn.init.constant_(m.bias, 0) 43 | elif isinstance(m, nn.Linear): 44 | nn.init.normal_(m.weight, 0, 0.01) 45 | nn.init.constant_(m.bias, 0) 46 | 47 | def freeze_backbone(self): 48 | for param in self.features.parameters(): 49 | param.requires_grad = False 50 | 51 | def Unfreeze_backbone(self): 52 | for param in self.features.parameters(): 53 | param.requires_grad = True 54 | 55 | 56 | def make_layers(cfg, batch_norm=False): 57 | layers = [] 58 | in_channels = 3 59 | for v in cfg: 60 | if v == 'M': 61 | layers += [nn.MaxPool2d(kernel_size=2, stride=2)] 62 | else: 63 | conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1) 64 | if batch_norm: 65 | layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)] 66 | else: 67 | layers += [conv2d, nn.ReLU(inplace=True)] 68 | in_channels = v 69 | return nn.Sequential(*layers) 70 | 71 | # 224,224,3 -> 224,224,64 -> 112,112,64 -> 112,112,128 -> 56,56,128 -> 56,56,256 -> 28,28,256 -> 28,28,512 72 | # 14,14,512 -> 14,14,512 -> 7,7,512 73 | cfgs = { 74 | 'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'], 75 | } 76 | 77 | def vgg16(pretrained=False, progress=True, num_classes=1000): 78 | model = VGG(make_layers(cfgs['D'])) 79 | if pretrained: 80 | state_dict = load_state_dict_from_url(model_urls['vgg16'], model_dir='./model_data', 81 | progress=progress) 82 | model.load_state_dict(state_dict,strict=False) 83 | 84 | if num_classes!=1000: 85 | model.classifier = nn.Sequential( 86 | nn.Linear(512 * 7 * 7, 4096), 87 | nn.ReLU(True), 88 | nn.Dropout(), 89 | nn.Linear(4096, 4096), 90 | nn.ReLU(True), 91 | nn.Dropout(), 92 | nn.Linear(4096, num_classes), 93 | ) 94 | return model 95 | -------------------------------------------------------------------------------- /classification/predict.py: -------------------------------------------------------------------------------- 1 | from PIL import Image 2 | 3 | from classification import Classification 4 | 5 | classfication = Classification() 6 | 7 | while True: 8 | img = input('Input image filename:') 9 | try: 10 | image = Image.open(img) 11 | except: 12 | print('Open Error! Try again!') 13 | continue 14 | else: 15 | class_name = classfication.detect_image(image) 16 | print(class_name) 17 | -------------------------------------------------------------------------------- /classification/requirements.txt: -------------------------------------------------------------------------------- 1 | scipy==1.2.1 2 | numpy==1.17.0 3 | matplotlib==3.1.2 4 | opencv_python==4.1.2.30 5 | # torch==1.2.0 6 | # torchvision==0.4.0 7 | tqdm==4.60.0 8 | Pillow==8.2.0 9 | h5py==2.10.0 10 | -------------------------------------------------------------------------------- /classification/test.py: -------------------------------------------------------------------------------- 1 | from torchsummary import summary 2 | 3 | from nets.mobilenet import mobilenet_v2 4 | from nets.resnet50 import resnet50 5 | from nets.vgg16 import vgg16 6 | 7 | if __name__ == "__main__": 8 | model = vgg16(num_classes=1000, pretrained=False).train().cuda() 9 | summary(model,(3,473,473)) 10 | -------------------------------------------------------------------------------- /classification/train.py: -------------------------------------------------------------------------------- 1 | import time 2 | 3 | import numpy as np 4 | import torch 5 | import torch.backends.cudnn as cudnn 6 | import torch.nn.functional as F 7 | import torch.optim as optim 8 | from torch import nn 9 | from torch.utils.data import DataLoader 10 | from tqdm import tqdm 11 | 12 | from nets.mobilenet import mobilenet_v2 13 | from nets.resnet50 import resnet50 14 | from nets.vgg16 import vgg16 15 | from utils.utils import weights_init 16 | from utils.dataloader import DataGenerator, detection_collate 17 | 18 | get_model_from_name = { 19 | "mobilenet" : mobilenet_v2, 20 | "resnet50" : resnet50, 21 | "vgg16" : vgg16, 22 | } 23 | 24 | freeze_layers = { 25 | "mobilenet" :81, 26 | "resnet50" :173, 27 | "vgg16" :19, 28 | } 29 | 30 | def get_lr(optimizer): 31 | for param_group in optimizer.param_groups: 32 | return param_group['lr'] 33 | 34 | def fit_one_epoch(net,epoch,epoch_size,epoch_size_val,gen,genval,Epoch,cuda): 35 | total_loss = 0 36 | total_accuracy = 0 37 | 38 | val_toal_loss = 0 39 | with tqdm(total=epoch_size,desc=f'Epoch {epoch + 1}/{Epoch}',postfix=dict,mininterval=0.3) as pbar: 40 | for iteration, batch in enumerate(gen): 41 | if iteration >= epoch_size: 42 | break 43 | images, targets = batch 44 | with torch.no_grad(): 45 | images = torch.from_numpy(images).type(torch.FloatTensor) 46 | targets = torch.from_numpy(targets).type(torch.FloatTensor).long() 47 | if cuda: 48 | images = images.cuda() 49 | targets = targets.cuda() 50 | 51 | optimizer.zero_grad() 52 | 53 | outputs = net(images) 54 | loss = nn.CrossEntropyLoss()(outputs, targets) 55 | loss.backward() 56 | optimizer.step() 57 | 58 | total_loss += loss.item() 59 | with torch.no_grad(): 60 | accuracy = torch.mean((torch.argmax(F.softmax(outputs, dim=-1), dim=-1) == targets).type(torch.FloatTensor)) 61 | total_accuracy += accuracy.item() 62 | 63 | 64 | pbar.set_postfix(**{'total_loss': total_loss / (iteration + 1), 65 | 'accuracy' : total_accuracy / (iteration + 1), 66 | 'lr' : get_lr(optimizer)}) 67 | pbar.update(1) 68 | 69 | print('Start Validation') 70 | with tqdm(total=epoch_size_val, desc=f'Epoch {epoch + 1}/{Epoch}',postfix=dict,mininterval=0.3) as pbar: 71 | for iteration, batch in enumerate(genval): 72 | if iteration >= epoch_size_val: 73 | break 74 | images, targets = batch 75 | with torch.no_grad(): 76 | images = torch.from_numpy(images).type(torch.FloatTensor) 77 | targets = torch.from_numpy(targets).type(torch.FloatTensor).long() 78 | if cuda: 79 | images = images.cuda() 80 | targets = targets.cuda() 81 | 82 | optimizer.zero_grad() 83 | 84 | outputs = net(images) 85 | val_loss = nn.CrossEntropyLoss()(outputs, targets) 86 | 87 | val_toal_loss += val_loss.item() 88 | 89 | pbar.set_postfix(**{'total_loss': val_toal_loss / (iteration + 1), 90 | 'lr' : get_lr(optimizer)}) 91 | pbar.update(1) 92 | 93 | print('Finish Validation') 94 | print('Epoch:'+ str(epoch+1) + '/' + str(Epoch)) 95 | print('Total Loss: %.4f || Val Loss: %.4f ' % (total_loss/(epoch_size+1),val_toal_loss/(epoch_size_val+1))) 96 | 97 | print('Saving state, iter:', str(epoch+1)) 98 | torch.save(model.state_dict(), 'classification/logs/Epoch%d-Total_Loss%.4f-Val_Loss%.4f.pth'%((epoch+1),total_loss/(epoch_size+1),val_toal_loss/(epoch_size_val+1))) 99 | 100 | #---------------------------------------------------# 101 | # 获得类 102 | #---------------------------------------------------# 103 | def get_classes(classes_path): 104 | '''loads the classes''' 105 | with open(classes_path) as f: 106 | class_names = f.readlines() 107 | class_names = [c.strip() for c in class_names] 108 | return class_names 109 | 110 | #----------------------------------------# 111 | # 主函数 112 | #----------------------------------------# 113 | if __name__ == "__main__": 114 | log_dir = "classification/logs/" 115 | #---------------------# 116 | # 所用模型种类 117 | #---------------------# 118 | backbone = "mobilenet" 119 | #---------------------# 120 | # 输入的图片大小 121 | #---------------------# 122 | input_shape = [224,224,3] 123 | #-------------------------------# 124 | # Cuda的使用 125 | #-------------------------------# 126 | Cuda = True 127 | 128 | #-------------------------------# 129 | # 是否使用网络的imagenet 130 | # 预训练权重 131 | #-------------------------------# 132 | pretrained = True 133 | 134 | classes_path = 'classification/model_data/cls_classes.txt' 135 | class_names = get_classes(classes_path) 136 | num_classes = len(class_names) 137 | 138 | assert backbone in ["mobilenet", "resnet50", "vgg16"] 139 | 140 | model = get_model_from_name[backbone](num_classes=num_classes,pretrained=pretrained) 141 | if not pretrained: 142 | weights_init(model) 143 | 144 | #------------------------------------------# 145 | # 注释部分可用于断点续练 146 | # 将训练好的模型重新载入 147 | #------------------------------------------# 148 | # # 加快模型训练的效率 149 | # model_path = "model_data/Omniglot_vgg.pth" 150 | # print('Loading weights into state dict...') 151 | # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 152 | # model_dict = model.state_dict() 153 | # pretrained_dict = torch.load(model_path, map_location=device) 154 | # pretrained_dict = {k: v for k, v in pretrained_dict.items() if np.shape(model_dict[k]) == np.shape(v)} 155 | # model_dict.update(pretrained_dict) 156 | # model.load_state_dict(model_dict) 157 | 158 | with open(r"classification/cls_train.txt","r") as f: 159 | lines = f.readlines() 160 | 161 | np.random.seed(10101) 162 | np.random.shuffle(lines) 163 | np.random.seed(None) 164 | num_val = int(len(lines)*0.1) 165 | num_train = len(lines) - num_val 166 | 167 | net = model.train() 168 | if Cuda: 169 | net = torch.nn.DataParallel(model) 170 | cudnn.benchmark = True 171 | net = net.cuda() 172 | 173 | #------------------------------------------------------# 174 | # 主干特征提取网络特征通用，冻结训练可以加快训练速度 175 | # 也可以在训练初期防止权值被破坏。 176 | # Init_Epoch为起始世代 177 | # Freeze_Epoch为冻结训练的世代 178 | # Epoch总训练世代 179 | # 提示OOM或者显存不足请调小Batch_size 180 | #------------------------------------------------------# 181 | if True: 182 | #--------------------------------------------# 183 | # BATCH_SIZE不要太小，不然训练效果很差 184 | #--------------------------------------------# 185 | lr = 1e-3 186 | Batch_size = 32 187 | Init_Epoch = 0 188 | Freeze_Epoch = 50 189 | 190 | optimizer = optim.Adam(net.parameters(),lr,weight_decay=5e-4) 191 | lr_scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9) 192 | 193 | train_dataset = DataGenerator(input_shape,lines[:num_train]) 194 | val_dataset = DataGenerator(input_shape,lines[num_train:], False) 195 | gen = DataLoader(train_dataset, batch_size=Batch_size, num_workers=4, pin_memory=True, 196 | drop_last=True, collate_fn=detection_collate) 197 | gen_val = DataLoader(val_dataset, batch_size=Batch_size, num_workers=4, pin_memory=True, 198 | drop_last=True, collate_fn=detection_collate) 199 | 200 | epoch_size = train_dataset.get_len()//Batch_size 201 | epoch_size_val = val_dataset.get_len()//Batch_size 202 | 203 | print(train_dataset.get_len(), val_dataset.get_len()) 204 | 205 | if epoch_size == 0 or epoch_size_val == 0: 206 | raise ValueError("数据集过小，无法进行训练，请扩充数据集。") 207 | #------------------------------------# 208 | # 冻结一定部分训练 209 | #------------------------------------# 210 | model.freeze_backbone() 211 | 212 | for epoch in range(Init_Epoch,Freeze_Epoch): 213 | fit_one_epoch(model,epoch,epoch_size,epoch_size_val,gen,gen_val,Freeze_Epoch,Cuda) 214 | lr_scheduler.step() 215 | 216 | if True: 217 | #--------------------------------------------# 218 | # BATCH_SIZE不要太小，不然训练效果很差 219 | #--------------------------------------------# 220 | lr = 1e-4 221 | Batch_size = 16 222 | Freeze_Epoch = 50 223 | Epoch = 100 224 | 225 | optimizer = optim.Adam(net.parameters(),lr,weight_decay=5e-4) 226 | lr_scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9) 227 | 228 | train_dataset = DataGenerator(input_shape,lines[:num_train]) 229 | val_dataset = DataGenerator(input_shape,lines[num_train:], False) 230 | gen = DataLoader(train_dataset, batch_size=Batch_size, num_workers=2, pin_memory=True, 231 | drop_last=True, collate_fn=detection_collate) 232 | gen_val = DataLoader(val_dataset, batch_size=Batch_size, num_workers=2, pin_memory=True, 233 | drop_last=True, collate_fn=detection_collate) 234 | 235 | epoch_size = train_dataset.get_len()//Batch_size 236 | epoch_size_val = val_dataset.get_len()//Batch_size 237 | 238 | if epoch_size == 0 or epoch_size_val == 0: 239 | raise ValueError("数据集过小，无法进行训练，请扩充数据集。") 240 | #------------------------------------# 241 | # 解冻后训练 242 | #------------------------------------# 243 | model.Unfreeze_backbone() 244 | 245 | for epoch in range(Freeze_Epoch,Epoch): 246 | fit_one_epoch(model,epoch,epoch_size,epoch_size_val,gen,gen_val,Epoch,Cuda) 247 | lr_scheduler.step() 248 | -------------------------------------------------------------------------------- /classification/txt_annotation.py: -------------------------------------------------------------------------------- 1 | import os 2 | from os import getcwd 3 | 4 | #---------------------------------------------# 5 | # 训练前一定要注意修改classes 6 | # 种类顺序需要和model_data下的txt一样 7 | #---------------------------------------------# 8 | classes = ["camera", "chair", "cup", "schooner"] 9 | sets = ["train", "test"] 10 | 11 | wd = getcwd() 12 | for se in sets: 13 | list_file = open('classification/cls_' + se + '.txt', 'w') 14 | 15 | datasets_path = "classification/datasets/" + se 16 | types_name = os.listdir(datasets_path) 17 | for type_name in types_name: 18 | if type_name not in classes: 19 | continue 20 | cls_id = classes.index(type_name) 21 | 22 | photos_path = os.path.join(datasets_path, type_name) 23 | photos_name = os.listdir(photos_path) 24 | for photo_name in photos_name: 25 | _, postfix = os.path.splitext(photo_name) 26 | if postfix not in ['.jpg', '.png', '.jpeg']: 27 | continue 28 | list_file.write(str(cls_id) + ";" + '%s/%s'%(wd, os.path.join(photos_path, photo_name))) 29 | list_file.write('\n') 30 | list_file.close() 31 | 32 | -------------------------------------------------------------------------------- /classification/utils/__pycache__/dataloader.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/utils/__pycache__/dataloader.cpython-37.pyc -------------------------------------------------------------------------------- /classification/utils/__pycache__/utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/classification/utils/__pycache__/utils.cpython-37.pyc -------------------------------------------------------------------------------- /classification/utils/dataloader.py: -------------------------------------------------------------------------------- 1 | from random import shuffle 2 | 3 | import cv2 4 | import numpy as np 5 | import torch.utils.data as data 6 | from PIL import Image 7 | 8 | from utils.utils import letterbox_image 9 | 10 | 11 | #----------------------------------------# 12 | # 预处理训练图片 13 | #----------------------------------------# 14 | def _preprocess_input(x,): 15 | x /= 127.5 16 | x -= 1. 17 | return x 18 | 19 | def rand(a=0, b=1): 20 | return np.random.rand()*(b-a) + a 21 | 22 | def get_random_data(image, input_shape, jitter=.3, hue=.1, sat=1.5, val=1.5): 23 | image = image.convert("RGB") 24 | h, w = input_shape 25 | 26 | # 对图像进行缩放并且进行长和宽的扭曲 27 | new_ar = w/h * rand(1-jitter,1+jitter)/rand(1-jitter,1+jitter) 28 | scale = rand(.75, 1.25) 29 | if new_ar < 1: 30 | nh = int(scale*h) 31 | nw = int(nh*new_ar) 32 | else: 33 | nw = int(scale*w) 34 | nh = int(nw/new_ar) 35 | image = image.resize((nw,nh), Image.BICUBIC) 36 | 37 | # 将图像多余的部分加上灰条 38 | dx = int(rand(0, w-nw)) 39 | dy = int(rand(0, h-nh)) 40 | new_image = Image.new('RGB', (w,h), (128,128,128)) 41 | new_image.paste(image, (dx, dy)) 42 | image = new_image 43 | 44 | # 翻转图像 45 | flip = rand()<.5 46 | if flip: image = image.transpose(Image.FLIP_LEFT_RIGHT) 47 | 48 | rotate = rand()<.5 49 | if rotate: 50 | angle=np.random.randint(-15,15) 51 | a,b=w/2,h/2 52 | M=cv2.getRotationMatrix2D((a,b),angle,1) 53 | image=cv2.warpAffine(np.array(image),M,(w,h),borderValue=[128,128,128]) 54 | 55 | # 色域扭曲 56 | hue = rand(-hue, hue) 57 | sat = rand(1, sat) if rand()<.5 else 1/rand(1, sat) 58 | val = rand(1, val) if rand()<.5 else 1/rand(1, val) 59 | x = cv2.cvtColor(np.array(image,np.float32)/255, cv2.COLOR_RGB2HSV) 60 | x[..., 1] *= sat 61 | x[..., 2] *= val 62 | x[x[:,:, 0]>360, 0] = 360 63 | x[:, :, 1:][x[:, :, 1:]>1] = 1 64 | x[x<0] = 0 65 | image_data = cv2.cvtColor(x, cv2.COLOR_HSV2RGB)*255 66 | return image_data 67 | 68 | class DataGenerator(data.Dataset): 69 | def __init__(self, input_shape, lines, random=True): 70 | self.input_shape = input_shape 71 | self.lines = lines 72 | self.random = random 73 | 74 | def __len__(self): 75 | return len(self.lines) 76 | 77 | def get_len(self): 78 | return len(self.lines) 79 | 80 | def __getitem__(self, index): 81 | if index == 0: 82 | shuffle(self.lines) 83 | 84 | annotation_path = self.lines[index].split(';')[1].split()[0] 85 | img = Image.open(annotation_path) 86 | 87 | if self.random: 88 | img = get_random_data(img, [self.input_shape[0],self.input_shape[1]]) 89 | else: 90 | img = letterbox_image(img, [self.input_shape[0],self.input_shape[1]]) 91 | 92 | img = np.array(img).astype(np.float32) 93 | img = _preprocess_input(img) 94 | img = np.transpose(img,[2,0,1]) 95 | 96 | y = int(self.lines[index].split(';')[0]) 97 | return img, y 98 | 99 | def detection_collate(batch): 100 | images = [] 101 | targets = [] 102 | for img, y in batch: 103 | images.append(img) 104 | targets.append(y) 105 | images = np.array(images) 106 | targets = np.array(targets) 107 | return images, targets 108 | -------------------------------------------------------------------------------- /classification/utils/utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from PIL import Image 3 | 4 | 5 | def letterbox_image(image, size): 6 | '''resize image with unchanged aspect ratio using padding''' 7 | iw, ih = image.size 8 | h, w = size 9 | scale = min(w/iw, h/ih) 10 | nw = int(iw*scale) 11 | nh = int(ih*scale) 12 | 13 | image = image.resize((nw,nh), Image.BICUBIC) 14 | new_image = Image.new('RGB', size, (128,128,128)) 15 | new_image.paste(image, ((w-nw)//2, (h-nh)//2)) 16 | return new_image 17 | 18 | def weights_init(net, init_type='normal', init_gain=0.02): 19 | def init_func(m): 20 | classname = m.__class__.__name__ 21 | if hasattr(m, 'weight') and classname.find('Conv') != -1: 22 | if init_type == 'normal': 23 | torch.nn.init.normal_(m.weight.data, 0.0, init_gain) 24 | elif init_type == 'xavier': 25 | torch.nn.init.xavier_normal_(m.weight.data, gain=init_gain) 26 | elif init_type == 'kaiming': 27 | torch.nn.init.kaiming_normal_(m.weight.data, a=0, mode='fan_in') 28 | elif init_type == 'orthogonal': 29 | torch.nn.init.orthogonal_(m.weight.data, gain=init_gain) 30 | else: 31 | raise NotImplementedError('initialization method [%s] is not implemented' % init_type) 32 | elif classname.find('BatchNorm2d') != -1: 33 | torch.nn.init.normal_(m.weight.data, 1.0, 0.02) 34 | torch.nn.init.constant_(m.bias.data, 0.0) 35 | print('initialize network with %s type' % init_type) 36 | net.apply(init_func) 37 | -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000192.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000192.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000241.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000241.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000257.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000257.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000294.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000294.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000328.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000328.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000338.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000338.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000357.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000357.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000360.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000360.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000395.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000395.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000415.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000415.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000428.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000428.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000459.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000459.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000474.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000474.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000488.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000488.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000536.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000536.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000544.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000544.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000564.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000564.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000569.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000569.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000589.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000589.jpg -------------------------------------------------------------------------------- /examples/media/COCO_val2014_000000000623.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/COCO_val2014_000000000623.jpg -------------------------------------------------------------------------------- /examples/media/test3.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/test3.mp4 -------------------------------------------------------------------------------- /examples/media/test4.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/test4.mp4 -------------------------------------------------------------------------------- /examples/media/test4_60fps.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/test4_60fps.mp4 -------------------------------------------------------------------------------- /examples/media/test5.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/test5.mp4 -------------------------------------------------------------------------------- /examples/media/video.avi: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/examples/media/video.avi -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/01_body_from_image_default.cpp: -------------------------------------------------------------------------------- 1 | // ----------------------------- OpenPose C++ API Tutorial - Example 1 - Body from image ----------------------------- 2 | // It reads an image, process it, and displays it with the pose keypoints. 3 | 4 | // Third-party dependencies 5 | #include 6 | // Command-line user interface 7 | #define OPENPOSE_FLAGS_DISABLE_POSE 8 | #include 9 | // OpenPose dependencies 10 | #include 11 | 12 | // Custom OpenPose flags 13 | // Producer 14 | DEFINE_string(image_path, "examples/media/COCO_val2014_000000000192.jpg", 15 | "Process an image. Read all standard formats (jpg, png, bmp, etc.)."); 16 | // Display 17 | DEFINE_bool(no_display, false, 18 | "Enable to disable the visual display."); 19 | 20 | // This worker will just read and return all the jpg files in a directory 21 | void display(const std::shared_ptr>>& datumsPtr) 22 | { 23 | try 24 | { 25 | // User's displaying/saving/other processing here 26 | // datum.cvOutputData: rendered frame with pose or heatmaps 27 | // datum.poseKeypoints: Array with the estimated pose 28 | if (datumsPtr != nullptr && !datumsPtr->empty()) 29 | { 30 | // Display image 31 | const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); 32 | if (!cvMat.empty()) 33 | { 34 | cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); 35 | cv::waitKey(0); 36 | } 37 | else 38 | op::opLog("Empty cv::Mat as output.", op::Priority::High, __LINE__, __FUNCTION__, __FILE__); 39 | } 40 | else 41 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 42 | } 43 | catch (const std::exception& e) 44 | { 45 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 46 | } 47 | } 48 | 49 | void printKeypoints(const std::shared_ptr>>& datumsPtr) 50 | { 51 | try 52 | { 53 | // Example: How to use the pose keypoints 54 | if (datumsPtr != nullptr && !datumsPtr->empty()) 55 | { 56 | // Alternative 1 57 | op::opLog("Body keypoints: " + datumsPtr->at(0)->poseKeypoints.toString(), op::Priority::High); 58 | 59 | // // Alternative 2 60 | // op::opLog(datumsPtr->at(0)->poseKeypoints, op::Priority::High); 61 | 62 | // // Alternative 3 63 | // std::cout << datumsPtr->at(0)->poseKeypoints << std::endl; 64 | 65 | // // Alternative 4 - Accesing each element of the keypoints 66 | // op::opLog("\nKeypoints:", op::Priority::High); 67 | // const auto& poseKeypoints = datumsPtr->at(0)->poseKeypoints; 68 | // op::opLog("Person pose keypoints:", op::Priority::High); 69 | // for (auto person = 0 ; person < poseKeypoints.getSize(0) ; person++) 70 | // { 71 | // op::opLog("Person " + std::to_string(person) + " (x, y, score):", op::Priority::High); 72 | // for (auto bodyPart = 0 ; bodyPart < poseKeypoints.getSize(1) ; bodyPart++) 73 | // { 74 | // std::string valueToPrint; 75 | // for (auto xyscore = 0 ; xyscore < poseKeypoints.getSize(2) ; xyscore++) 76 | // valueToPrint += std::to_string( poseKeypoints[{person, bodyPart, xyscore}] ) + " "; 77 | // op::opLog(valueToPrint, op::Priority::High); 78 | // } 79 | // } 80 | // op::opLog(" ", op::Priority::High); 81 | } 82 | else 83 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 84 | } 85 | catch (const std::exception& e) 86 | { 87 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 88 | } 89 | } 90 | 91 | int tutorialApiCpp() 92 | { 93 | try 94 | { 95 | op::opLog("Starting OpenPose demo...", op::Priority::High); 96 | const auto opTimer = op::getTimerInit(); 97 | 98 | // Configuring OpenPose 99 | op::opLog("Configuring OpenPose...", op::Priority::High); 100 | op::Wrapper opWrapper{op::ThreadManagerMode::Asynchronous}; 101 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 102 | if (FLAGS_disable_multi_thread) 103 | opWrapper.disableMultiThreading(); 104 | 105 | // Starting OpenPose 106 | op::opLog("Starting thread(s)...", op::Priority::High); 107 | opWrapper.start(); 108 | 109 | // Process and display image 110 | const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); 111 | const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); 112 | auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); 113 | if (datumProcessed != nullptr) 114 | { 115 | printKeypoints(datumProcessed); 116 | if (!FLAGS_no_display) 117 | display(datumProcessed); 118 | } 119 | else 120 | op::opLog("Image could not be processed.", op::Priority::High); 121 | 122 | // Measuring total time 123 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 124 | 125 | // Return 126 | return 0; 127 | } 128 | catch (const std::exception&) 129 | { 130 | return -1; 131 | } 132 | } 133 | 134 | int main(int argc, char *argv[]) 135 | { 136 | // Parsing command line flags 137 | gflags::ParseCommandLineFlags(&argc, &argv, true); 138 | 139 | // Running tutorialApiCpp 140 | return tutorialApiCpp(); 141 | } 142 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/02_whole_body_from_image_default.cpp: -------------------------------------------------------------------------------- 1 | // -------------------------- OpenPose C++ API Tutorial - Example 2 - Whole body from image -------------------------- 2 | // It reads an image, process it, and displays it with the pose, hand, and face keypoints. 3 | 4 | // Third-party dependencies 5 | #include 6 | // Command-line user interface 7 | #define OPENPOSE_FLAGS_DISABLE_POSE 8 | #include 9 | // OpenPose dependencies 10 | #include 11 | 12 | // Custom OpenPose flags 13 | // Producer 14 | DEFINE_string(image_path, "examples/media/COCO_val2014_000000000241.jpg", 15 | "Process an image. Read all standard formats (jpg, png, bmp, etc.)."); 16 | // Display 17 | DEFINE_bool(no_display, false, 18 | "Enable to disable the visual display."); 19 | 20 | // This worker will just read and return all the jpg files in a directory 21 | void display(const std::shared_ptr>>& datumsPtr) 22 | { 23 | try 24 | { 25 | // User's displaying/saving/other processing here 26 | // datum.cvOutputData: rendered frame with pose or heatmaps 27 | // datum.poseKeypoints: Array with the estimated pose 28 | if (datumsPtr != nullptr && !datumsPtr->empty()) 29 | { 30 | // Display image 31 | const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); 32 | if (!cvMat.empty()) 33 | { 34 | cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); 35 | cv::waitKey(0); 36 | } 37 | else 38 | op::opLog("Empty cv::Mat as output.", op::Priority::High, __LINE__, __FUNCTION__, __FILE__); 39 | } 40 | else 41 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 42 | } 43 | catch (const std::exception& e) 44 | { 45 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 46 | } 47 | } 48 | 49 | void printKeypoints(const std::shared_ptr>>& datumsPtr) 50 | { 51 | try 52 | { 53 | // Example: How to use the pose keypoints 54 | if (datumsPtr != nullptr && !datumsPtr->empty()) 55 | { 56 | op::opLog("Body keypoints: " + datumsPtr->at(0)->poseKeypoints.toString(), op::Priority::High); 57 | op::opLog("Face keypoints: " + datumsPtr->at(0)->faceKeypoints.toString(), op::Priority::High); 58 | op::opLog("Left hand keypoints: " + datumsPtr->at(0)->handKeypoints[0].toString(), op::Priority::High); 59 | op::opLog("Right hand keypoints: " + datumsPtr->at(0)->handKeypoints[1].toString(), op::Priority::High); 60 | } 61 | else 62 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 63 | } 64 | catch (const std::exception& e) 65 | { 66 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 67 | } 68 | } 69 | 70 | int tutorialApiCpp() 71 | { 72 | try 73 | { 74 | op::opLog("Starting OpenPose demo...", op::Priority::High); 75 | const auto opTimer = op::getTimerInit(); 76 | 77 | // Configuring OpenPose 78 | op::opLog("Configuring OpenPose...", op::Priority::High); 79 | op::Wrapper opWrapper{op::ThreadManagerMode::Asynchronous}; 80 | // Add hand and face 81 | opWrapper.configure(op::WrapperStructFace{true}); 82 | opWrapper.configure(op::WrapperStructHand{true}); 83 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 84 | if (FLAGS_disable_multi_thread) 85 | opWrapper.disableMultiThreading(); 86 | 87 | // Starting OpenPose 88 | op::opLog("Starting thread(s)...", op::Priority::High); 89 | opWrapper.start(); 90 | 91 | // Process and display image 92 | const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); 93 | const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); 94 | auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); 95 | if (datumProcessed != nullptr) 96 | { 97 | printKeypoints(datumProcessed); 98 | if (!FLAGS_no_display) 99 | display(datumProcessed); 100 | } 101 | else 102 | op::opLog("Image could not be processed.", op::Priority::High); 103 | 104 | // Measuring total time 105 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 106 | 107 | // Return 108 | return 0; 109 | } 110 | catch (const std::exception&) 111 | { 112 | return -1; 113 | } 114 | } 115 | 116 | int main(int argc, char *argv[]) 117 | { 118 | // Parsing command line flags 119 | gflags::ParseCommandLineFlags(&argc, &argv, true); 120 | 121 | // Running tutorialApiCpp 122 | return tutorialApiCpp(); 123 | } 124 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/03_keypoints_from_image.cpp: -------------------------------------------------------------------------------- 1 | // ----------------------- OpenPose C++ API Tutorial - Example 3 - Body from image ----------------------- 2 | // It reads an image, process it, and displays it with the pose (and optionally hand and face) keypoints. In addition, 3 | // it includes all the OpenPose configuration flags (enable/disable hand, face, output saving, etc.). 4 | 5 | // Third-party dependencies 6 | #include 7 | // Command-line user interface 8 | #define OPENPOSE_FLAGS_DISABLE_PRODUCER 9 | #define OPENPOSE_FLAGS_DISABLE_DISPLAY 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // Custom OpenPose flags 15 | // Producer 16 | DEFINE_string(image_path, "examples/media/COCO_val2014_000000000294.jpg", 17 | "Process an image. Read all standard formats (jpg, png, bmp, etc.)."); 18 | // Display 19 | DEFINE_bool(no_display, false, 20 | "Enable to disable the visual display."); 21 | 22 | // This worker will just read and return all the jpg files in a directory 23 | void display(const std::shared_ptr>>& datumsPtr) 24 | { 25 | try 26 | { 27 | // User's displaying/saving/other processing here 28 | // datum.cvOutputData: rendered frame with pose or heatmaps 29 | // datum.poseKeypoints: Array with the estimated pose 30 | if (datumsPtr != nullptr && !datumsPtr->empty()) 31 | { 32 | // Display image 33 | const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); 34 | if (!cvMat.empty()) 35 | { 36 | cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); 37 | cv::waitKey(0); 38 | } 39 | else 40 | op::opLog("Empty cv::Mat as output.", op::Priority::High, __LINE__, __FUNCTION__, __FILE__); 41 | } 42 | else 43 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 44 | } 45 | catch (const std::exception& e) 46 | { 47 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 48 | } 49 | } 50 | 51 | void printKeypoints(const std::shared_ptr>>& datumsPtr) 52 | { 53 | try 54 | { 55 | // Example: How to use the pose keypoints 56 | if (datumsPtr != nullptr && !datumsPtr->empty()) 57 | { 58 | op::opLog("Body keypoints: " + datumsPtr->at(0)->poseKeypoints.toString(), op::Priority::High); 59 | op::opLog("Face keypoints: " + datumsPtr->at(0)->faceKeypoints.toString(), op::Priority::High); 60 | op::opLog("Left hand keypoints: " + datumsPtr->at(0)->handKeypoints[0].toString(), op::Priority::High); 61 | op::opLog("Right hand keypoints: " + datumsPtr->at(0)->handKeypoints[1].toString(), op::Priority::High); 62 | } 63 | else 64 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 65 | } 66 | catch (const std::exception& e) 67 | { 68 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 69 | } 70 | } 71 | 72 | void configureWrapper(op::Wrapper& opWrapper) 73 | { 74 | try 75 | { 76 | // Configuring OpenPose 77 | 78 | // logging_level 79 | op::checkBool( 80 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 81 | __LINE__, __FUNCTION__, __FILE__); 82 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 83 | op::Profiler::setDefaultX(FLAGS_profile_speed); 84 | 85 | // Applying user defined configuration - GFlags to program variables 86 | // outputSize 87 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 88 | // netInputSize 89 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 90 | // faceNetInputSize 91 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 92 | // handNetInputSize 93 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 94 | // poseMode 95 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 96 | // poseModel 97 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 98 | // JSON saving 99 | if (!FLAGS_write_keypoint.empty()) 100 | op::opLog( 101 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 102 | " instead.", op::Priority::Max); 103 | // keypointScaleMode 104 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 105 | // heatmaps to add 106 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 107 | FLAGS_heatmaps_add_PAFs); 108 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 109 | // >1 camera view? 110 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1); 111 | // Face and hand detectors 112 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 113 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 114 | // Enabling Google Logging 115 | const bool enableGoogleLogging = true; 116 | 117 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 118 | const op::WrapperStructPose wrapperStructPose{ 119 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 120 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 121 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 122 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 123 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 124 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 125 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 126 | opWrapper.configure(wrapperStructPose); 127 | // Face configuration (use op::WrapperStructFace{} to disable it) 128 | const op::WrapperStructFace wrapperStructFace{ 129 | FLAGS_face, faceDetector, faceNetInputSize, 130 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 131 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 132 | opWrapper.configure(wrapperStructFace); 133 | // Hand configuration (use op::WrapperStructHand{} to disable it) 134 | const op::WrapperStructHand wrapperStructHand{ 135 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 136 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 137 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 138 | opWrapper.configure(wrapperStructHand); 139 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 140 | const op::WrapperStructExtra wrapperStructExtra{ 141 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 142 | opWrapper.configure(wrapperStructExtra); 143 | // Output (comment or use default argument to disable any output) 144 | const op::WrapperStructOutput wrapperStructOutput{ 145 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 146 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 147 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 148 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 149 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 150 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 151 | op::String(FLAGS_udp_port)}; 152 | opWrapper.configure(wrapperStructOutput); 153 | // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{}); 154 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 155 | if (FLAGS_disable_multi_thread) 156 | opWrapper.disableMultiThreading(); 157 | } 158 | catch (const std::exception& e) 159 | { 160 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 161 | } 162 | } 163 | 164 | int tutorialApiCpp() 165 | { 166 | try 167 | { 168 | op::opLog("Starting OpenPose demo...", op::Priority::High); 169 | const auto opTimer = op::getTimerInit(); 170 | 171 | // Configuring OpenPose 172 | op::opLog("Configuring OpenPose...", op::Priority::High); 173 | op::Wrapper opWrapper{op::ThreadManagerMode::Asynchronous}; 174 | configureWrapper(opWrapper); 175 | 176 | // Starting OpenPose 177 | op::opLog("Starting thread(s)...", op::Priority::High); 178 | opWrapper.start(); 179 | 180 | // Process and display image 181 | const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); 182 | const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); 183 | auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); 184 | if (datumProcessed != nullptr) 185 | { 186 | printKeypoints(datumProcessed); 187 | if (!FLAGS_no_display) 188 | display(datumProcessed); 189 | } 190 | else 191 | op::opLog("Image could not be processed.", op::Priority::High); 192 | 193 | // Measuring total time 194 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 195 | 196 | // Return 197 | return 0; 198 | } 199 | catch (const std::exception&) 200 | { 201 | return -1; 202 | } 203 | } 204 | 205 | int main(int argc, char *argv[]) 206 | { 207 | // Parsing command line flags 208 | gflags::ParseCommandLineFlags(&argc, &argv, true); 209 | 210 | // Running tutorialApiCpp 211 | return tutorialApiCpp(); 212 | } 213 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/04_keypoints_from_images.cpp: -------------------------------------------------------------------------------- 1 | // ----------------------- OpenPose C++ API Tutorial - Example 4 - Body from images ---------------------- 2 | // It reads images, process them, and display them with the pose (and optionally hand and face) keypoints. In addition, 3 | // it includes all the OpenPose configuration flags (enable/disable hand, face, output saving, etc.). 4 | 5 | // Third-party dependencies 6 | #include 7 | // Command-line user interface 8 | #define OPENPOSE_FLAGS_DISABLE_PRODUCER 9 | #define OPENPOSE_FLAGS_DISABLE_DISPLAY 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // Custom OpenPose flags 15 | // Producer 16 | DEFINE_string(image_dir, "examples/media/", 17 | "Process a directory of images. Read all standard formats (jpg, png, bmp, etc.)."); 18 | // Display 19 | DEFINE_bool(no_display, false, 20 | "Enable to disable the visual display."); 21 | 22 | // This worker will just read and return all the jpg files in a directory 23 | bool display(const std::shared_ptr>>& datumsPtr) 24 | { 25 | try 26 | { 27 | // User's displaying/saving/other processing here 28 | // datum.cvOutputData: rendered frame with pose or heatmaps 29 | // datum.poseKeypoints: Array with the estimated pose 30 | if (datumsPtr != nullptr && !datumsPtr->empty()) 31 | { 32 | // Display image and sleeps at least 1 ms (it usually sleeps ~5-10 msec to display the image) 33 | const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); 34 | if (!cvMat.empty()) 35 | cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); 36 | else 37 | op::opLog("Empty cv::Mat as output.", op::Priority::High, __LINE__, __FUNCTION__, __FILE__); 38 | } 39 | else 40 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 41 | const auto key = (char)cv::waitKey(1); 42 | return (key == 27); 43 | } 44 | catch (const std::exception& e) 45 | { 46 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 47 | return true; 48 | } 49 | } 50 | 51 | void printKeypoints(const std::shared_ptr>>& datumsPtr) 52 | { 53 | try 54 | { 55 | // Example: How to use the pose keypoints 56 | if (datumsPtr != nullptr && !datumsPtr->empty()) 57 | { 58 | op::opLog("Body keypoints: " + datumsPtr->at(0)->poseKeypoints.toString(), op::Priority::High); 59 | op::opLog("Face keypoints: " + datumsPtr->at(0)->faceKeypoints.toString(), op::Priority::High); 60 | op::opLog("Left hand keypoints: " + datumsPtr->at(0)->handKeypoints[0].toString(), op::Priority::High); 61 | op::opLog("Right hand keypoints: " + datumsPtr->at(0)->handKeypoints[1].toString(), op::Priority::High); 62 | } 63 | else 64 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 65 | } 66 | catch (const std::exception& e) 67 | { 68 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 69 | } 70 | } 71 | 72 | void configureWrapper(op::Wrapper& opWrapper) 73 | { 74 | try 75 | { 76 | // Configuring OpenPose 77 | 78 | // logging_level 79 | op::checkBool( 80 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 81 | __LINE__, __FUNCTION__, __FILE__); 82 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 83 | op::Profiler::setDefaultX(FLAGS_profile_speed); 84 | 85 | // Applying user defined configuration - GFlags to program variables 86 | // outputSize 87 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 88 | // netInputSize 89 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 90 | // faceNetInputSize 91 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 92 | // handNetInputSize 93 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 94 | // poseMode 95 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 96 | // poseModel 97 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 98 | // JSON saving 99 | if (!FLAGS_write_keypoint.empty()) 100 | op::opLog( 101 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 102 | " instead.", op::Priority::Max); 103 | // keypointScaleMode 104 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 105 | // heatmaps to add 106 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 107 | FLAGS_heatmaps_add_PAFs); 108 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 109 | // >1 camera view? 110 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1); 111 | // Face and hand detectors 112 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 113 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 114 | // Enabling Google Logging 115 | const bool enableGoogleLogging = true; 116 | 117 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 118 | const op::WrapperStructPose wrapperStructPose{ 119 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 120 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 121 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 122 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 123 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 124 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 125 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 126 | opWrapper.configure(wrapperStructPose); 127 | // Face configuration (use op::WrapperStructFace{} to disable it) 128 | const op::WrapperStructFace wrapperStructFace{ 129 | FLAGS_face, faceDetector, faceNetInputSize, 130 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 131 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 132 | opWrapper.configure(wrapperStructFace); 133 | // Hand configuration (use op::WrapperStructHand{} to disable it) 134 | const op::WrapperStructHand wrapperStructHand{ 135 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 136 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 137 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 138 | opWrapper.configure(wrapperStructHand); 139 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 140 | const op::WrapperStructExtra wrapperStructExtra{ 141 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 142 | opWrapper.configure(wrapperStructExtra); 143 | // Output (comment or use default argument to disable any output) 144 | const op::WrapperStructOutput wrapperStructOutput{ 145 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 146 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 147 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 148 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 149 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 150 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 151 | op::String(FLAGS_udp_port)}; 152 | opWrapper.configure(wrapperStructOutput); 153 | // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{}); 154 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 155 | if (FLAGS_disable_multi_thread) 156 | opWrapper.disableMultiThreading(); 157 | } 158 | catch (const std::exception& e) 159 | { 160 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 161 | } 162 | } 163 | 164 | int tutorialApiCpp() 165 | { 166 | try 167 | { 168 | op::opLog("Starting OpenPose demo...", op::Priority::High); 169 | const auto opTimer = op::getTimerInit(); 170 | 171 | // Configuring OpenPose 172 | op::opLog("Configuring OpenPose...", op::Priority::High); 173 | op::Wrapper opWrapper{op::ThreadManagerMode::Asynchronous}; 174 | configureWrapper(opWrapper); 175 | 176 | // Starting OpenPose 177 | op::opLog("Starting thread(s)...", op::Priority::High); 178 | opWrapper.start(); 179 | 180 | // Read frames on directory 181 | const auto imagePaths = op::getFilesOnDirectory(FLAGS_image_dir, op::Extensions::Images); 182 | 183 | // Process and display images 184 | for (const auto& imagePath : imagePaths) 185 | { 186 | const cv::Mat cvImageToProcess = cv::imread(imagePath); 187 | const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); 188 | auto datumProcessed = opWrapper.emplaceAndPop(imageToProcess); 189 | if (datumProcessed != nullptr) 190 | { 191 | printKeypoints(datumProcessed); 192 | if (!FLAGS_no_display) 193 | { 194 | const auto userWantsToExit = display(datumProcessed); 195 | if (userWantsToExit) 196 | { 197 | op::opLog("User pressed Esc to exit demo.", op::Priority::High); 198 | break; 199 | } 200 | } 201 | } 202 | else 203 | op::opLog("Image " + imagePath + " could not be processed.", op::Priority::High); 204 | } 205 | 206 | // Measuring total time 207 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 208 | 209 | // Return 210 | return 0; 211 | } 212 | catch (const std::exception&) 213 | { 214 | return -1; 215 | } 216 | } 217 | 218 | int main(int argc, char *argv[]) 219 | { 220 | // Parsing command line flags 221 | gflags::ParseCommandLineFlags(&argc, &argv, true); 222 | 223 | // Running tutorialApiCpp 224 | return tutorialApiCpp(); 225 | } 226 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/06_face_from_image.cpp: -------------------------------------------------------------------------------- 1 | // ----------------------------- OpenPose C++ API Tutorial - Example 6 - Face from Image ----------------------------- 2 | // It reads an image and the face location, process it, and displays the face keypoints. In addition, 3 | // it includes all the OpenPose configuration flags. 4 | // Input: An image and the face rectangle locations. 5 | // Output: OpenPose face keypoint detection. 6 | // NOTE: This demo is auto-selecting the following flags: `--body 0 --face --face_detector 2` 7 | 8 | // Third-party dependencies 9 | #include 10 | // Command-line user interface 11 | #define OPENPOSE_FLAGS_DISABLE_PRODUCER 12 | #define OPENPOSE_FLAGS_DISABLE_DISPLAY 13 | #include 14 | // OpenPose dependencies 15 | #include 16 | 17 | // Custom OpenPose flags 18 | // Producer 19 | DEFINE_string(image_path, "examples/media/COCO_val2014_000000000241.jpg", 20 | "Process an image. Read all standard formats (jpg, png, bmp, etc.)."); 21 | // Display 22 | DEFINE_bool(no_display, false, 23 | "Enable to disable the visual display."); 24 | 25 | // This worker will just read and return all the jpg files in a directory 26 | void display(const std::shared_ptr>>& datumsPtr) 27 | { 28 | try 29 | { 30 | // User's displaying/saving/other processing here 31 | // datum.cvOutputData: rendered frame with pose or heatmaps 32 | // datum.poseKeypoints: Array with the estimated pose 33 | if (datumsPtr != nullptr && !datumsPtr->empty()) 34 | { 35 | // Display image 36 | const cv::Mat cvMat = OP_OP2CVCONSTMAT(datumsPtr->at(0)->cvOutputData); 37 | if (!cvMat.empty()) 38 | { 39 | cv::imshow(OPEN_POSE_NAME_AND_VERSION + " - Tutorial C++ API", cvMat); 40 | cv::waitKey(0); 41 | } 42 | else 43 | op::opLog("Empty cv::Mat as output.", op::Priority::High, __LINE__, __FUNCTION__, __FILE__); 44 | } 45 | else 46 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 47 | } 48 | catch (const std::exception& e) 49 | { 50 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 51 | } 52 | } 53 | 54 | void printKeypoints(const std::shared_ptr>>& datumsPtr) 55 | { 56 | try 57 | { 58 | // Example: How to use the pose keypoints 59 | if (datumsPtr != nullptr && !datumsPtr->empty()) 60 | { 61 | op::opLog("Body keypoints: " + datumsPtr->at(0)->poseKeypoints.toString(), op::Priority::High); 62 | op::opLog("Face keypoints: " + datumsPtr->at(0)->faceKeypoints.toString(), op::Priority::High); 63 | op::opLog("Left hand keypoints: " + datumsPtr->at(0)->handKeypoints[0].toString(), op::Priority::High); 64 | op::opLog("Right hand keypoints: " + datumsPtr->at(0)->handKeypoints[1].toString(), op::Priority::High); 65 | } 66 | else 67 | op::opLog("Nullptr or empty datumsPtr found.", op::Priority::High); 68 | } 69 | catch (const std::exception& e) 70 | { 71 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 72 | } 73 | } 74 | 75 | void configureWrapper(op::Wrapper& opWrapper) 76 | { 77 | try 78 | { 79 | // Configuring OpenPose 80 | 81 | // logging_level 82 | op::checkBool( 83 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 84 | __LINE__, __FUNCTION__, __FILE__); 85 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 86 | op::Profiler::setDefaultX(FLAGS_profile_speed); 87 | 88 | // Applying user defined configuration - GFlags to program variables 89 | // outputSize 90 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 91 | // netInputSize 92 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 93 | // faceNetInputSize 94 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 95 | // handNetInputSize 96 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 97 | // poseMode 98 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 99 | // poseModel 100 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 101 | // JSON saving 102 | if (!FLAGS_write_keypoint.empty()) 103 | op::opLog( 104 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 105 | " instead.", op::Priority::Max); 106 | // keypointScaleMode 107 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 108 | // heatmaps to add 109 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 110 | FLAGS_heatmaps_add_PAFs); 111 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 112 | // >1 camera view? 113 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1); 114 | // Face and hand detectors 115 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 116 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 117 | // Enabling Google Logging 118 | const bool enableGoogleLogging = true; 119 | 120 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 121 | const op::WrapperStructPose wrapperStructPose{ 122 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 123 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 124 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 125 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 126 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 127 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 128 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 129 | opWrapper.configure(wrapperStructPose); 130 | // Face configuration (use op::WrapperStructFace{} to disable it) 131 | const op::WrapperStructFace wrapperStructFace{ 132 | FLAGS_face, faceDetector, faceNetInputSize, 133 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 134 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 135 | opWrapper.configure(wrapperStructFace); 136 | // Hand configuration (use op::WrapperStructHand{} to disable it) 137 | const op::WrapperStructHand wrapperStructHand{ 138 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 139 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 140 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 141 | opWrapper.configure(wrapperStructHand); 142 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 143 | const op::WrapperStructExtra wrapperStructExtra{ 144 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 145 | opWrapper.configure(wrapperStructExtra); 146 | // Output (comment or use default argument to disable any output) 147 | const op::WrapperStructOutput wrapperStructOutput{ 148 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 149 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 150 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 151 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 152 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 153 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 154 | op::String(FLAGS_udp_port)}; 155 | opWrapper.configure(wrapperStructOutput); 156 | // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{}); 157 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 158 | if (FLAGS_disable_multi_thread) 159 | opWrapper.disableMultiThreading(); 160 | } 161 | catch (const std::exception& e) 162 | { 163 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 164 | } 165 | } 166 | 167 | int tutorialApiCpp() 168 | { 169 | try 170 | { 171 | op::opLog("Starting OpenPose demo...", op::Priority::High); 172 | const auto opTimer = op::getTimerInit(); 173 | 174 | // Required flags to enable heatmaps 175 | FLAGS_body = 0; 176 | FLAGS_face = true; 177 | FLAGS_face_detector = 2; 178 | 179 | // Configuring OpenPose 180 | op::opLog("Configuring OpenPose...", op::Priority::High); 181 | op::Wrapper opWrapper{op::ThreadManagerMode::Asynchronous}; 182 | configureWrapper(opWrapper); 183 | 184 | // Starting OpenPose 185 | op::opLog("Starting thread(s)...", op::Priority::High); 186 | opWrapper.start(); 187 | 188 | // Read image and face rectangle locations 189 | const cv::Mat cvImageToProcess = cv::imread(FLAGS_image_path); 190 | const op::Matrix imageToProcess = OP_CV2OPCONSTMAT(cvImageToProcess); 191 | const std::vector> faceRectangles{ 192 | op::Rectangle{330.119385f, 277.532715f, 48.717274f, 48.717274f}, // Face of person 0 193 | op::Rectangle{24.036991f, 267.918793f, 65.175171f, 65.175171f}, // Face of person 1 194 | op::Rectangle{151.803436f, 32.477852f, 108.295761f, 108.295761f} // Face of person 2 195 | }; 196 | 197 | // Create new datum 198 | auto datumsPtr = std::make_shared>>(); 199 | datumsPtr->emplace_back(); 200 | auto& datumPtr = datumsPtr->at(0); 201 | datumPtr = std::make_shared(); 202 | // Fill datum with image and faceRectangles 203 | datumPtr->cvInputData = imageToProcess; 204 | datumPtr->faceRectangles = faceRectangles; 205 | 206 | // Process and display image 207 | opWrapper.emplaceAndPop(datumsPtr); 208 | if (datumsPtr != nullptr) 209 | { 210 | printKeypoints(datumsPtr); 211 | if (!FLAGS_no_display) 212 | display(datumsPtr); 213 | } 214 | else 215 | op::opLog("Image could not be processed.", op::Priority::High); 216 | 217 | // Info 218 | op::opLog("NOTE: In addition with the user flags, this demo has auto-selected the following flags:\n" 219 | "\t`--body 0 --face --face_detector 2`", op::Priority::High); 220 | 221 | // Measuring total time 222 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 223 | 224 | // Return 225 | return 0; 226 | } 227 | catch (const std::exception&) 228 | { 229 | return -1; 230 | } 231 | } 232 | 233 | int main(int argc, char *argv[]) 234 | { 235 | // Parsing command line flags 236 | gflags::ParseCommandLineFlags(&argc, &argv, true); 237 | 238 | // Running tutorialApiCpp 239 | return tutorialApiCpp(); 240 | } 241 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/10_asynchronous_custom_input.cpp: -------------------------------------------------------------------------------- 1 | // ------------------------- OpenPose C++ API Tutorial - Example 10 - Custom Input ------------------------- 2 | // Asynchronous mode: ideal for fast prototyping when performance is not an issue. 3 | // In this function, the user can implement its own way to create frames (e.g., reading his own folder of images) 4 | // and emplaces/pushes the frames to OpenPose. 5 | 6 | // Third-party dependencies 7 | #include 8 | // Command-line user interface 9 | #define OPENPOSE_FLAGS_DISABLE_PRODUCER 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // Custom OpenPose flags 15 | // Producer 16 | DEFINE_string(image_dir, "examples/media/", 17 | "Process a directory of images. Read all standard formats (jpg, png, bmp, etc.)."); 18 | 19 | // This worker will just read and return all the basic image file formats in a directory 20 | class UserInputClass 21 | { 22 | public: 23 | UserInputClass(const std::string& directoryPath) : 24 | mImageFiles{op::getFilesOnDirectory(directoryPath, op::Extensions::Images)}, // For all basic image formats 25 | // If we want only e.g., "jpg" + "png" images 26 | // mImageFiles{op::getFilesOnDirectory(directoryPath, std::vector{"jpg", "png"})}, 27 | mCounter{0}, 28 | mClosed{false} 29 | { 30 | if (mImageFiles.empty()) 31 | op::error("No images found on: " + directoryPath, __LINE__, __FUNCTION__, __FILE__); 32 | } 33 | 34 | std::shared_ptr>> createDatum() 35 | { 36 | // Close program when empty frame 37 | if (mClosed || mImageFiles.size() <= mCounter) 38 | { 39 | op::opLog("Last frame read and added to queue. Closing program after it is processed.", op::Priority::High); 40 | // This function stops this worker, which will eventually stop the whole thread system once all the frames 41 | // have been processed 42 | mClosed = true; 43 | return nullptr; 44 | } 45 | else // if (!mClosed) 46 | { 47 | // Create new datum 48 | auto datumsPtr = std::make_shared>>(); 49 | datumsPtr->emplace_back(); 50 | auto& datumPtr = datumsPtr->at(0); 51 | datumPtr = std::make_shared(); 52 | 53 | // Fill datum 54 | const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); 55 | datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); 56 | 57 | // If empty frame -> return nullptr 58 | if (datumPtr->cvInputData.empty()) 59 | { 60 | op::opLog("Empty frame detected on path: " + mImageFiles.at(mCounter-1) + ". Closing program.", 61 | op::Priority::High); 62 | mClosed = true; 63 | datumsPtr = nullptr; 64 | } 65 | 66 | return datumsPtr; 67 | } 68 | } 69 | 70 | bool isFinished() const 71 | { 72 | return mClosed; 73 | } 74 | 75 | private: 76 | const std::vector mImageFiles; 77 | unsigned long long mCounter; 78 | bool mClosed; 79 | }; 80 | 81 | void configureWrapper(op::Wrapper& opWrapper) 82 | { 83 | try 84 | { 85 | // Configuring OpenPose 86 | 87 | // logging_level 88 | op::checkBool( 89 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 90 | __LINE__, __FUNCTION__, __FILE__); 91 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 92 | op::Profiler::setDefaultX(FLAGS_profile_speed); 93 | 94 | // Applying user defined configuration - GFlags to program variables 95 | // outputSize 96 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 97 | // netInputSize 98 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 99 | // faceNetInputSize 100 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 101 | // handNetInputSize 102 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 103 | // poseMode 104 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 105 | // poseModel 106 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 107 | // JSON saving 108 | if (!FLAGS_write_keypoint.empty()) 109 | op::opLog( 110 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 111 | " instead.", op::Priority::Max); 112 | // keypointScaleMode 113 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 114 | // heatmaps to add 115 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 116 | FLAGS_heatmaps_add_PAFs); 117 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 118 | // >1 camera view? 119 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1); 120 | // Face and hand detectors 121 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 122 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 123 | // Enabling Google Logging 124 | const bool enableGoogleLogging = true; 125 | 126 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 127 | const op::WrapperStructPose wrapperStructPose{ 128 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 129 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 130 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 131 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 132 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 133 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 134 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 135 | opWrapper.configure(wrapperStructPose); 136 | // Face configuration (use op::WrapperStructFace{} to disable it) 137 | const op::WrapperStructFace wrapperStructFace{ 138 | FLAGS_face, faceDetector, faceNetInputSize, 139 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 140 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 141 | opWrapper.configure(wrapperStructFace); 142 | // Hand configuration (use op::WrapperStructHand{} to disable it) 143 | const op::WrapperStructHand wrapperStructHand{ 144 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 145 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 146 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 147 | opWrapper.configure(wrapperStructHand); 148 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 149 | const op::WrapperStructExtra wrapperStructExtra{ 150 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 151 | opWrapper.configure(wrapperStructExtra); 152 | // Output (comment or use default argument to disable any output) 153 | const op::WrapperStructOutput wrapperStructOutput{ 154 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 155 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 156 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 157 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 158 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 159 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 160 | op::String(FLAGS_udp_port)}; 161 | opWrapper.configure(wrapperStructOutput); 162 | // GUI (comment or use default argument to disable any visual output) 163 | const op::WrapperStructGui wrapperStructGui{ 164 | op::flagsToDisplayMode(FLAGS_display, FLAGS_3d), !FLAGS_no_gui_verbose, FLAGS_fullscreen}; 165 | opWrapper.configure(wrapperStructGui); 166 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 167 | if (FLAGS_disable_multi_thread) 168 | opWrapper.disableMultiThreading(); 169 | } 170 | catch (const std::exception& e) 171 | { 172 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 173 | } 174 | } 175 | 176 | int tutorialApiCpp() 177 | { 178 | try 179 | { 180 | op::opLog("Starting OpenPose demo...", op::Priority::High); 181 | const auto opTimer = op::getTimerInit(); 182 | 183 | // Configuring OpenPose 184 | op::opLog("Configuring OpenPose...", op::Priority::High); 185 | op::Wrapper opWrapper{op::ThreadManagerMode::AsynchronousIn}; 186 | configureWrapper(opWrapper); 187 | 188 | // Start, run, and stop processing - exec() blocks this thread until OpenPose wrapper has finished 189 | op::opLog("Starting thread(s)...", op::Priority::High); 190 | opWrapper.start(); 191 | 192 | // User processing 193 | UserInputClass userInputClass(FLAGS_image_dir); 194 | bool userWantsToExit = false; 195 | while (!userWantsToExit && !userInputClass.isFinished()) 196 | { 197 | // Push frame 198 | auto datumToProcess = userInputClass.createDatum(); 199 | if (datumToProcess != nullptr) 200 | { 201 | auto successfullyEmplaced = opWrapper.waitAndEmplace(datumToProcess); 202 | if (!successfullyEmplaced) 203 | op::opLog("Processed datum could not be emplaced.", op::Priority::High); 204 | } 205 | } 206 | 207 | op::opLog("Stopping thread(s)", op::Priority::High); 208 | opWrapper.stop(); 209 | 210 | // Measuring total time 211 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 212 | 213 | // Return 214 | return 0; 215 | } 216 | catch (const std::exception&) 217 | { 218 | return -1; 219 | } 220 | } 221 | 222 | int main(int argc, char *argv[]) 223 | { 224 | // Parsing command line flags 225 | gflags::ParseCommandLineFlags(&argc, &argv, true); 226 | 227 | // Running tutorialApiCpp 228 | return tutorialApiCpp(); 229 | } 230 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/11_asynchronous_custom_input_multi_camera.cpp: -------------------------------------------------------------------------------- 1 | // ------------------------- OpenPose C++ API Tutorial - Example 10 - Custom Input ------------------------- 2 | // Asynchronous mode: ideal for fast prototyping when performance is not an issue. 3 | // In this function, the user can implement its own way to create frames (e.g., reading his own folder of images) 4 | // and emplaces/pushes the frames to OpenPose. 5 | 6 | // Third-party dependencies 7 | #include 8 | // Command-line user interface 9 | #define OPENPOSE_FLAGS_DISABLE_PRODUCER 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // Custom OpenPose flags 15 | // Producer 16 | DEFINE_string(video, "3d_4camera_video.avi", 17 | "Use a video file instead of the camera. Use `examples/media/video.avi` for our default example video."); 18 | DEFINE_string(camera_parameter_path, "models/cameraParameters/flir/", 19 | "String with the folder where the camera parameters are located. If there is only 1 XML file (for single" 20 | " video, webcam, or images from the same camera), you must specify the whole XML file path (ending in .xml)."); 21 | 22 | // This worker will just read and return all the basic image file formats in a directory 23 | class UserInputClass 24 | { 25 | public: 26 | UserInputClass(const std::string& videoPath, const std::string& cameraParameterPath) : 27 | mClosed{false}, 28 | mFrameCounter{0ull}, 29 | mVideoCapture{videoPath} 30 | { 31 | if (!mVideoCapture.isOpened()) 32 | { 33 | mClosed = true; 34 | op::error("No video " + videoPath + " opened.", __LINE__, __FUNCTION__, __FILE__); 35 | } 36 | 37 | // Create CameraParameterReader 38 | mCameraParameterReader.readParameters(cameraParameterPath); 39 | } 40 | 41 | std::shared_ptr>> createDatum() 42 | { 43 | if (mClosed) 44 | { 45 | op::opLog("Video already closed, nullptr returned.", op::Priority::High); 46 | return nullptr; 47 | } 48 | 49 | // Read cv::Mat 50 | cv::Mat cvInputData; 51 | mVideoCapture >> cvInputData; 52 | // If empty frame -> return nullptr 53 | if (cvInputData.empty()) 54 | { 55 | // Close program when empty frame 56 | op::opLog("Empty frame detected, closing program.", op::Priority::High); 57 | mClosed = true; 58 | return nullptr; 59 | } 60 | 61 | // Create new datum and add 3D information (cv::Mat splitted and camera parameters) 62 | auto datumsPtr = std::make_shared>>(); 63 | op::createMultiviewTDatum(datumsPtr, mFrameCounter, mCameraParameterReader, (void*)&cvInputData); 64 | 65 | return datumsPtr; 66 | } 67 | 68 | bool isFinished() const 69 | { 70 | return mClosed; 71 | } 72 | 73 | private: 74 | bool mClosed; 75 | unsigned long long mFrameCounter; 76 | cv::VideoCapture mVideoCapture; 77 | op::CameraParameterReader mCameraParameterReader; 78 | }; 79 | 80 | void configureWrapper(op::Wrapper& opWrapper) 81 | { 82 | try 83 | { 84 | // Configuring OpenPose 85 | 86 | // logging_level 87 | op::checkBool( 88 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 89 | __LINE__, __FUNCTION__, __FILE__); 90 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 91 | op::Profiler::setDefaultX(FLAGS_profile_speed); 92 | 93 | // Applying user defined configuration - GFlags to program variables 94 | // outputSize 95 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 96 | // netInputSize 97 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 98 | // faceNetInputSize 99 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 100 | // handNetInputSize 101 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 102 | // poseMode 103 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 104 | // poseModel 105 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 106 | // JSON saving 107 | if (!FLAGS_write_keypoint.empty()) 108 | op::opLog( 109 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 110 | " instead.", op::Priority::Max); 111 | // keypointScaleMode 112 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 113 | // heatmaps to add 114 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 115 | FLAGS_heatmaps_add_PAFs); 116 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 117 | // >1 camera view? 118 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1); 119 | // Face and hand detectors 120 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 121 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 122 | // Enabling Google Logging 123 | const bool enableGoogleLogging = true; 124 | 125 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 126 | const op::WrapperStructPose wrapperStructPose{ 127 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 128 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 129 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 130 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 131 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 132 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 133 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 134 | opWrapper.configure(wrapperStructPose); 135 | // Face configuration (use op::WrapperStructFace{} to disable it) 136 | const op::WrapperStructFace wrapperStructFace{ 137 | FLAGS_face, faceDetector, faceNetInputSize, 138 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 139 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 140 | opWrapper.configure(wrapperStructFace); 141 | // Hand configuration (use op::WrapperStructHand{} to disable it) 142 | const op::WrapperStructHand wrapperStructHand{ 143 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 144 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 145 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 146 | opWrapper.configure(wrapperStructHand); 147 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 148 | const op::WrapperStructExtra wrapperStructExtra{ 149 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 150 | opWrapper.configure(wrapperStructExtra); 151 | // Output (comment or use default argument to disable any output) 152 | const op::WrapperStructOutput wrapperStructOutput{ 153 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 154 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 155 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 156 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 157 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 158 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 159 | op::String(FLAGS_udp_port)}; 160 | opWrapper.configure(wrapperStructOutput); 161 | // GUI (comment or use default argument to disable any visual output) 162 | const op::WrapperStructGui wrapperStructGui{ 163 | op::flagsToDisplayMode(FLAGS_display, FLAGS_3d), !FLAGS_no_gui_verbose, FLAGS_fullscreen}; 164 | opWrapper.configure(wrapperStructGui); 165 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 166 | if (FLAGS_disable_multi_thread) 167 | opWrapper.disableMultiThreading(); 168 | } 169 | catch (const std::exception& e) 170 | { 171 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 172 | } 173 | } 174 | 175 | int tutorialApiCpp() 176 | { 177 | try 178 | { 179 | op::opLog("Starting OpenPose demo...", op::Priority::High); 180 | const auto opTimer = op::getTimerInit(); 181 | 182 | // Required flags to enable 3-D 183 | FLAGS_3d = true; 184 | FLAGS_number_people_max = 1; 185 | FLAGS_3d_min_views = 3; 186 | FLAGS_output_resolution = "320x256"; // Optional, but otherwise it gets too big to render in real time 187 | // FLAGS_3d_views = X; // Not required because it only affects OpenPose producers (rather than custom ones) 188 | 189 | // Configuring OpenPose 190 | op::opLog("Configuring OpenPose...", op::Priority::High); 191 | op::Wrapper opWrapper{op::ThreadManagerMode::AsynchronousIn}; 192 | configureWrapper(opWrapper); 193 | 194 | // Start, run, and stop processing - exec() blocks this thread until OpenPose wrapper has finished 195 | op::opLog("Starting thread(s)...", op::Priority::High); 196 | opWrapper.start(); 197 | 198 | // User processing 199 | UserInputClass userInputClass(FLAGS_video, FLAGS_camera_parameter_path); 200 | bool userWantsToExit = false; 201 | while (!userWantsToExit && !userInputClass.isFinished()) 202 | { 203 | if (!opWrapper.isRunning()) 204 | { 205 | op::opLog("OpenPose wrapper is no longer running, exiting video.", op::Priority::High); 206 | break; 207 | } 208 | // Push frame 209 | auto datumToProcess = userInputClass.createDatum(); 210 | if (datumToProcess != nullptr) 211 | { 212 | auto successfullyEmplaced = opWrapper.waitAndEmplace(datumToProcess); 213 | if (!successfullyEmplaced) 214 | op::opLog("Processed datum could not be emplaced.", op::Priority::High); 215 | } 216 | } 217 | 218 | op::opLog("Stopping thread(s)", op::Priority::High); 219 | opWrapper.stop(); 220 | 221 | // Measuring total time 222 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 223 | 224 | // Return 225 | return 0; 226 | } 227 | catch (const std::exception&) 228 | { 229 | return -1; 230 | } 231 | } 232 | 233 | int main(int argc, char *argv[]) 234 | { 235 | // Parsing command line flags 236 | gflags::ParseCommandLineFlags(&argc, &argv, true); 237 | 238 | // Running tutorialApiCpp 239 | return tutorialApiCpp(); 240 | } 241 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/14_synchronous_custom_input.cpp: -------------------------------------------------------------------------------- 1 | // ------------------------- OpenPose C++ API Tutorial - Example 13 - Custom Input ------------------------- 2 | // Synchronous mode: ideal for production integration. It provides the fastest results with respect to runtime 3 | // performance. 4 | // In this function, the user can implement its own way to create frames (e.g., reading his own folder of images). 5 | 6 | // Third-party dependencies 7 | #include 8 | // Command-line user interface 9 | #define OPENPOSE_FLAGS_DISABLE_PRODUCER 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // Custom OpenPose flags 15 | // Producer 16 | DEFINE_string(image_dir, "examples/media/", 17 | "Process a directory of images. Read all standard formats (jpg, png, bmp, etc.)."); 18 | 19 | // This worker will just read and return all the basic image file formats in a directory 20 | class WUserInput : public op::WorkerProducer>>> 21 | { 22 | public: 23 | WUserInput(const std::string& directoryPath) : 24 | mImageFiles{op::getFilesOnDirectory(directoryPath, op::Extensions::Images)}, // For all basic image formats 25 | // If we want only e.g., "jpg" + "png" images 26 | // mImageFiles{op::getFilesOnDirectory(directoryPath, std::vector{"jpg", "png"})}, 27 | mCounter{0} 28 | { 29 | if (mImageFiles.empty()) 30 | op::error("No images found on: " + directoryPath, __LINE__, __FUNCTION__, __FILE__); 31 | } 32 | 33 | void initializationOnThread() {} 34 | 35 | std::shared_ptr>> workProducer() 36 | { 37 | try 38 | { 39 | // Close program when empty frame 40 | if (mImageFiles.size() <= mCounter) 41 | { 42 | op::opLog("Last frame read and added to queue. Closing program after it is processed.", 43 | op::Priority::High); 44 | // This funtion stops this worker, which will eventually stop the whole thread system once all the 45 | // frames have been processed 46 | this->stop(); 47 | return nullptr; 48 | } 49 | else 50 | { 51 | // Create new datum 52 | auto datumsPtr = std::make_shared>>(); 53 | datumsPtr->emplace_back(); 54 | auto& datumPtr = datumsPtr->at(0); 55 | datumPtr = std::make_shared(); 56 | 57 | // Fill datum 58 | const cv::Mat cvInputData = cv::imread(mImageFiles.at(mCounter++)); 59 | datumPtr->cvInputData = OP_CV2OPCONSTMAT(cvInputData); 60 | 61 | // If empty frame -> return nullptr 62 | if (datumPtr->cvInputData.empty()) 63 | { 64 | op::opLog("Empty frame detected on path: " + mImageFiles.at(mCounter-1) + ". Closing program.", 65 | op::Priority::High); 66 | this->stop(); 67 | datumsPtr = nullptr; 68 | } 69 | 70 | return datumsPtr; 71 | } 72 | } 73 | catch (const std::exception& e) 74 | { 75 | this->stop(); 76 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 77 | return nullptr; 78 | } 79 | } 80 | 81 | private: 82 | const std::vector mImageFiles; 83 | unsigned long long mCounter; 84 | }; 85 | 86 | void configureWrapper(op::Wrapper& opWrapper) 87 | { 88 | try 89 | { 90 | // Configuring OpenPose 91 | 92 | // logging_level 93 | op::checkBool( 94 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 95 | __LINE__, __FUNCTION__, __FILE__); 96 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 97 | op::Profiler::setDefaultX(FLAGS_profile_speed); 98 | 99 | // Applying user defined configuration - GFlags to program variables 100 | // outputSize 101 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 102 | // netInputSize 103 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 104 | // faceNetInputSize 105 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 106 | // handNetInputSize 107 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 108 | // poseMode 109 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 110 | // poseModel 111 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 112 | // JSON saving 113 | if (!FLAGS_write_keypoint.empty()) 114 | op::opLog( 115 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 116 | " instead.", op::Priority::Max); 117 | // keypointScaleMode 118 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 119 | // heatmaps to add 120 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 121 | FLAGS_heatmaps_add_PAFs); 122 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 123 | // >1 camera view? 124 | // const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1 || FLAGS_flir_camera); 125 | const auto multipleView = false; 126 | // Face and hand detectors 127 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 128 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 129 | // Enabling Google Logging 130 | const bool enableGoogleLogging = true; 131 | 132 | // Initializing the user custom classes 133 | // Frames producer (e.g., video, webcam, ...) 134 | auto wUserInput = std::make_shared(FLAGS_image_dir); 135 | // Add custom processing 136 | const auto workerInputOnNewThread = true; 137 | opWrapper.setWorker(op::WorkerType::Input, wUserInput, workerInputOnNewThread); 138 | 139 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 140 | const op::WrapperStructPose wrapperStructPose{ 141 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 142 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 143 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 144 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 145 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 146 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 147 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 148 | opWrapper.configure(wrapperStructPose); 149 | // Face configuration (use op::WrapperStructFace{} to disable it) 150 | const op::WrapperStructFace wrapperStructFace{ 151 | FLAGS_face, faceDetector, faceNetInputSize, 152 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 153 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 154 | opWrapper.configure(wrapperStructFace); 155 | // Hand configuration (use op::WrapperStructHand{} to disable it) 156 | const op::WrapperStructHand wrapperStructHand{ 157 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 158 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 159 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 160 | opWrapper.configure(wrapperStructHand); 161 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 162 | const op::WrapperStructExtra wrapperStructExtra{ 163 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 164 | opWrapper.configure(wrapperStructExtra); 165 | // Output (comment or use default argument to disable any output) 166 | const op::WrapperStructOutput wrapperStructOutput{ 167 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 168 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 169 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 170 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 171 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 172 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 173 | op::String(FLAGS_udp_port)}; 174 | opWrapper.configure(wrapperStructOutput); 175 | // GUI (comment or use default argument to disable any visual output) 176 | const op::WrapperStructGui wrapperStructGui{ 177 | op::flagsToDisplayMode(FLAGS_display, FLAGS_3d), !FLAGS_no_gui_verbose, FLAGS_fullscreen}; 178 | opWrapper.configure(wrapperStructGui); 179 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 180 | if (FLAGS_disable_multi_thread) 181 | opWrapper.disableMultiThreading(); 182 | } 183 | catch (const std::exception& e) 184 | { 185 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 186 | } 187 | } 188 | 189 | int tutorialApiCpp() 190 | { 191 | try 192 | { 193 | op::opLog("Starting OpenPose demo...", op::Priority::High); 194 | const auto opTimer = op::getTimerInit(); 195 | 196 | // OpenPose wrapper 197 | op::opLog("Configuring OpenPose...", op::Priority::High); 198 | op::Wrapper opWrapper; 199 | configureWrapper(opWrapper); 200 | 201 | // Start, run, and stop processing - exec() blocks this thread until OpenPose wrapper has finished 202 | op::opLog("Starting thread(s)...", op::Priority::High); 203 | opWrapper.exec(); 204 | 205 | // Measuring total time 206 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 207 | 208 | // Return 209 | return 0; 210 | } 211 | catch (const std::exception&) 212 | { 213 | return -1; 214 | } 215 | } 216 | 217 | int main(int argc, char *argv[]) 218 | { 219 | // Parsing command line flags 220 | gflags::ParseCommandLineFlags(&argc, &argv, true); 221 | 222 | // Running tutorialApiCpp 223 | return tutorialApiCpp(); 224 | } 225 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/15_synchronous_custom_preprocessing.cpp: -------------------------------------------------------------------------------- 1 | // ------------------------- OpenPose C++ API Tutorial - Example 14 - Custom Pre-processing ------------------------- 2 | // Synchronous mode: ideal for production integration. It provides the fastest results with respect to runtime 3 | // performance. 4 | // In this function, the user can implement its own pre-processing, i.e., his function will be called after the image 5 | // has been read by OpenPose but before OpenPose processes the frames. 6 | 7 | // Third-party dependencies 8 | #include 9 | // Command-line user interface 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // This worker will just invert the image 15 | class WUserPreProcessing : public op::Worker>>> 16 | { 17 | public: 18 | WUserPreProcessing() 19 | { 20 | // User's constructor here 21 | } 22 | 23 | void initializationOnThread() {} 24 | 25 | void work(std::shared_ptr>>& datumsPtr) 26 | { 27 | try 28 | { 29 | // User's pre-processing (after OpenPose read the input image & before OpenPose processing) here 30 | // datumPtr->cvInputData: input frame 31 | if (datumsPtr != nullptr && !datumsPtr->empty()) 32 | { 33 | for (auto& datumPtr : *datumsPtr) 34 | { 35 | cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); 36 | cv::bitwise_not(cvOutputData, cvOutputData); 37 | } 38 | } 39 | } 40 | catch (const std::exception& e) 41 | { 42 | this->stop(); 43 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 44 | } 45 | } 46 | }; 47 | 48 | void configureWrapper(op::Wrapper& opWrapper) 49 | { 50 | try 51 | { 52 | // Configuring OpenPose 53 | 54 | // logging_level 55 | op::checkBool( 56 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 57 | __LINE__, __FUNCTION__, __FILE__); 58 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 59 | op::Profiler::setDefaultX(FLAGS_profile_speed); 60 | 61 | // Applying user defined configuration - GFlags to program variables 62 | // producerType 63 | op::ProducerType producerType; 64 | op::String producerString; 65 | std::tie(producerType, producerString) = op::flagsToProducer( 66 | op::String(FLAGS_image_dir), op::String(FLAGS_video), op::String(FLAGS_ip_camera), FLAGS_camera, 67 | FLAGS_flir_camera, FLAGS_flir_camera_index); 68 | // cameraSize 69 | const auto cameraSize = op::flagsToPoint(op::String(FLAGS_camera_resolution), "-1x-1"); 70 | // outputSize 71 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 72 | // netInputSize 73 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 74 | // faceNetInputSize 75 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 76 | // handNetInputSize 77 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 78 | // poseMode 79 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 80 | // poseModel 81 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 82 | // JSON saving 83 | if (!FLAGS_write_keypoint.empty()) 84 | op::opLog( 85 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 86 | " instead.", op::Priority::Max); 87 | // keypointScaleMode 88 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 89 | // heatmaps to add 90 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 91 | FLAGS_heatmaps_add_PAFs); 92 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 93 | // >1 camera view? 94 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1 || FLAGS_flir_camera); 95 | // Face and hand detectors 96 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 97 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 98 | // Enabling Google Logging 99 | const bool enableGoogleLogging = true; 100 | 101 | // Initializing the user custom classes 102 | // Processing 103 | auto wUserPreProcessing = std::make_shared(); 104 | // Add custom processing 105 | const auto workerProcessingOnNewThread = true; 106 | opWrapper.setWorker(op::WorkerType::PreProcessing, wUserPreProcessing, workerProcessingOnNewThread); 107 | 108 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 109 | const op::WrapperStructPose wrapperStructPose{ 110 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 111 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 112 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 113 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 114 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 115 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 116 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 117 | opWrapper.configure(wrapperStructPose); 118 | // Face configuration (use op::WrapperStructFace{} to disable it) 119 | const op::WrapperStructFace wrapperStructFace{ 120 | FLAGS_face, faceDetector, faceNetInputSize, 121 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 122 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 123 | opWrapper.configure(wrapperStructFace); 124 | // Hand configuration (use op::WrapperStructHand{} to disable it) 125 | const op::WrapperStructHand wrapperStructHand{ 126 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 127 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 128 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 129 | opWrapper.configure(wrapperStructHand); 130 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 131 | const op::WrapperStructExtra wrapperStructExtra{ 132 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 133 | opWrapper.configure(wrapperStructExtra); 134 | // Producer (use default to disable any input) 135 | const op::WrapperStructInput wrapperStructInput{ 136 | producerType, producerString, FLAGS_frame_first, FLAGS_frame_step, FLAGS_frame_last, 137 | FLAGS_process_real_time, FLAGS_frame_flip, FLAGS_frame_rotate, FLAGS_frames_repeat, 138 | cameraSize, op::String(FLAGS_camera_parameter_path), FLAGS_frame_undistort, FLAGS_3d_views}; 139 | opWrapper.configure(wrapperStructInput); 140 | // Output (comment or use default argument to disable any output) 141 | const op::WrapperStructOutput wrapperStructOutput{ 142 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 143 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 144 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 145 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 146 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 147 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 148 | op::String(FLAGS_udp_port)}; 149 | opWrapper.configure(wrapperStructOutput); 150 | // GUI (comment or use default argument to disable any visual output) 151 | const op::WrapperStructGui wrapperStructGui{ 152 | op::flagsToDisplayMode(FLAGS_display, FLAGS_3d), !FLAGS_no_gui_verbose, FLAGS_fullscreen}; 153 | opWrapper.configure(wrapperStructGui); 154 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 155 | if (FLAGS_disable_multi_thread) 156 | opWrapper.disableMultiThreading(); 157 | } 158 | catch (const std::exception& e) 159 | { 160 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 161 | } 162 | } 163 | 164 | int tutorialApiCpp() 165 | { 166 | try 167 | { 168 | op::opLog("Starting OpenPose demo...", op::Priority::High); 169 | const auto opTimer = op::getTimerInit(); 170 | 171 | // OpenPose wrapper 172 | op::opLog("Configuring OpenPose...", op::Priority::High); 173 | op::Wrapper opWrapper; 174 | configureWrapper(opWrapper); 175 | 176 | // Start, run, and stop processing - exec() blocks this thread until OpenPose wrapper has finished 177 | op::opLog("Starting thread(s)...", op::Priority::High); 178 | opWrapper.exec(); 179 | 180 | // Measuring total time 181 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 182 | 183 | // Return 184 | return 0; 185 | } 186 | catch (const std::exception&) 187 | { 188 | return -1; 189 | } 190 | } 191 | 192 | int main(int argc, char *argv[]) 193 | { 194 | // Parsing command line flags 195 | gflags::ParseCommandLineFlags(&argc, &argv, true); 196 | 197 | // Running tutorialApiCpp 198 | return tutorialApiCpp(); 199 | } 200 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/16_synchronous_custom_postprocessing.cpp: -------------------------------------------------------------------------------- 1 | // ------------------------- OpenPose C++ API Tutorial - Example 15 - Custom Post-processing ------------------------- 2 | // Synchronous mode: ideal for production integration. It provides the fastest results with respect to runtime 3 | // performance. 4 | // In this function, the user can implement its own post-processing, i.e., his function will be called after OpenPose 5 | // has processed the frames but before saving or visualizing any result. 6 | 7 | // Third-party dependencies 8 | #include 9 | // Command-line user interface 10 | #include 11 | // OpenPose dependencies 12 | #include 13 | 14 | // This worker will just invert the image 15 | class WUserPostProcessing : public op::Worker>>> 16 | { 17 | public: 18 | WUserPostProcessing() 19 | { 20 | // User's constructor here 21 | } 22 | 23 | void initializationOnThread() {} 24 | 25 | void work(std::shared_ptr>>& datumsPtr) 26 | { 27 | try 28 | { 29 | // User's post-processing (after OpenPose processing & before OpenPose outputs) here 30 | // datumPtr->cvOutputData: rendered frame with pose or heatmaps 31 | // datumPtr->poseKeypoints: Array with the estimated pose 32 | if (datumsPtr != nullptr && !datumsPtr->empty()) 33 | { 34 | for (auto& datumPtr : *datumsPtr) 35 | { 36 | cv::Mat cvOutputData = OP_OP2CVMAT(datumPtr->cvOutputData); 37 | cv::bitwise_not(cvOutputData, cvOutputData); 38 | } 39 | } 40 | } 41 | catch (const std::exception& e) 42 | { 43 | this->stop(); 44 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 45 | } 46 | } 47 | }; 48 | 49 | void configureWrapper(op::Wrapper& opWrapper) 50 | { 51 | try 52 | { 53 | // Configuring OpenPose 54 | 55 | // logging_level 56 | op::checkBool( 57 | 0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.", 58 | __LINE__, __FUNCTION__, __FILE__); 59 | op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level); 60 | op::Profiler::setDefaultX(FLAGS_profile_speed); 61 | 62 | // Applying user defined configuration - GFlags to program variables 63 | // producerType 64 | op::ProducerType producerType; 65 | op::String producerString; 66 | std::tie(producerType, producerString) = op::flagsToProducer( 67 | op::String(FLAGS_image_dir), op::String(FLAGS_video), op::String(FLAGS_ip_camera), FLAGS_camera, 68 | FLAGS_flir_camera, FLAGS_flir_camera_index); 69 | // cameraSize 70 | const auto cameraSize = op::flagsToPoint(op::String(FLAGS_camera_resolution), "-1x-1"); 71 | // outputSize 72 | const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1"); 73 | // netInputSize 74 | const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368"); 75 | // faceNetInputSize 76 | const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)"); 77 | // handNetInputSize 78 | const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)"); 79 | // poseMode 80 | const auto poseMode = op::flagsToPoseMode(FLAGS_body); 81 | // poseModel 82 | const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose)); 83 | // JSON saving 84 | if (!FLAGS_write_keypoint.empty()) 85 | op::opLog( 86 | "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`" 87 | " instead.", op::Priority::Max); 88 | // keypointScaleMode 89 | const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale); 90 | // heatmaps to add 91 | const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg, 92 | FLAGS_heatmaps_add_PAFs); 93 | const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale); 94 | // >1 camera view? 95 | const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1 || FLAGS_flir_camera); 96 | // Face and hand detectors 97 | const auto faceDetector = op::flagsToDetector(FLAGS_face_detector); 98 | const auto handDetector = op::flagsToDetector(FLAGS_hand_detector); 99 | // Enabling Google Logging 100 | const bool enableGoogleLogging = true; 101 | 102 | // Initializing the user custom classes 103 | // Processing 104 | auto wUserPostProcessing = std::make_shared(); 105 | // Add custom processing 106 | const auto workerProcessingOnNewThread = true; 107 | opWrapper.setWorker(op::WorkerType::PostProcessing, wUserPostProcessing, workerProcessingOnNewThread); 108 | 109 | // Pose configuration (use WrapperStructPose{} for default and recommended configuration) 110 | const op::WrapperStructPose wrapperStructPose{ 111 | poseMode, netInputSize, FLAGS_net_resolution_dynamic, outputSize, keypointScaleMode, FLAGS_num_gpu, 112 | FLAGS_num_gpu_start, FLAGS_scale_number, (float)FLAGS_scale_gap, 113 | op::flagsToRenderMode(FLAGS_render_pose, multipleView), poseModel, !FLAGS_disable_blending, 114 | (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap, FLAGS_part_to_show, op::String(FLAGS_model_folder), 115 | heatMapTypes, heatMapScaleMode, FLAGS_part_candidates, (float)FLAGS_render_threshold, 116 | FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max, op::String(FLAGS_prototxt_path), 117 | op::String(FLAGS_caffemodel_path), (float)FLAGS_upsampling_ratio, enableGoogleLogging}; 118 | opWrapper.configure(wrapperStructPose); 119 | // Face configuration (use op::WrapperStructFace{} to disable it) 120 | const op::WrapperStructFace wrapperStructFace{ 121 | FLAGS_face, faceDetector, faceNetInputSize, 122 | op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose), 123 | (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold}; 124 | opWrapper.configure(wrapperStructFace); 125 | // Hand configuration (use op::WrapperStructHand{} to disable it) 126 | const op::WrapperStructHand wrapperStructHand{ 127 | FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range, 128 | op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose, 129 | (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold}; 130 | opWrapper.configure(wrapperStructHand); 131 | // Extra functionality configuration (use op::WrapperStructExtra{} to disable it) 132 | const op::WrapperStructExtra wrapperStructExtra{ 133 | FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads}; 134 | opWrapper.configure(wrapperStructExtra); 135 | // Producer (use default to disable any input) 136 | const op::WrapperStructInput wrapperStructInput{ 137 | producerType, producerString, FLAGS_frame_first, FLAGS_frame_step, FLAGS_frame_last, 138 | FLAGS_process_real_time, FLAGS_frame_flip, FLAGS_frame_rotate, FLAGS_frames_repeat, 139 | cameraSize, op::String(FLAGS_camera_parameter_path), FLAGS_frame_undistort, FLAGS_3d_views}; 140 | opWrapper.configure(wrapperStructInput); 141 | // Output (comment or use default argument to disable any output) 142 | const op::WrapperStructOutput wrapperStructOutput{ 143 | FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format), 144 | op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants, 145 | FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format), 146 | op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio, 147 | op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d), 148 | op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host), 149 | op::String(FLAGS_udp_port)}; 150 | opWrapper.configure(wrapperStructOutput); 151 | // GUI (comment or use default argument to disable any visual output) 152 | const op::WrapperStructGui wrapperStructGui{ 153 | op::flagsToDisplayMode(FLAGS_display, FLAGS_3d), !FLAGS_no_gui_verbose, FLAGS_fullscreen}; 154 | opWrapper.configure(wrapperStructGui); 155 | // Set to single-thread (for sequential processing and/or debugging and/or reducing latency) 156 | if (FLAGS_disable_multi_thread) 157 | opWrapper.disableMultiThreading(); 158 | } 159 | catch (const std::exception& e) 160 | { 161 | op::error(e.what(), __LINE__, __FUNCTION__, __FILE__); 162 | } 163 | } 164 | 165 | int tutorialApiCpp() 166 | { 167 | try 168 | { 169 | op::opLog("Starting OpenPose demo...", op::Priority::High); 170 | const auto opTimer = op::getTimerInit(); 171 | 172 | // OpenPose wrapper 173 | op::opLog("Configuring OpenPose...", op::Priority::High); 174 | op::Wrapper opWrapper; 175 | configureWrapper(opWrapper); 176 | 177 | // Start, run, and stop processing - exec() blocks this thread until OpenPose wrapper has finished 178 | op::opLog("Starting thread(s)...", op::Priority::High); 179 | opWrapper.exec(); 180 | 181 | // Measuring total time 182 | op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High); 183 | 184 | // Return 185 | return 0; 186 | } 187 | catch (const std::exception&) 188 | { 189 | return -1; 190 | } 191 | } 192 | 193 | int main(int argc, char *argv[]) 194 | { 195 | // Parsing command line flags 196 | gflags::ParseCommandLineFlags(&argc, &argv, true); 197 | 198 | // Running tutorialApiCpp 199 | return tutorialApiCpp(); 200 | } 201 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | set(EXAMPLE_FILES 2 | 01_body_from_image_default.cpp 3 | 02_whole_body_from_image_default.cpp 4 | 03_keypoints_from_image.cpp 5 | 04_keypoints_from_images.cpp 6 | 05_keypoints_from_images_multi_gpu.cpp 7 | 06_face_from_image.cpp 8 | 07_hand_from_image.cpp 9 | 08_heatmaps_from_image.cpp 10 | 09_keypoints_from_heatmaps.cpp 11 | 10_asynchronous_custom_input.cpp 12 | 11_asynchronous_custom_input_multi_camera.cpp 13 | 12_asynchronous_custom_output.cpp 14 | 13_asynchronous_custom_input_output_and_datum.cpp 15 | 14_synchronous_custom_input.cpp 16 | 15_synchronous_custom_preprocessing.cpp 17 | 16_synchronous_custom_postprocessing.cpp 18 | 17_synchronous_custom_output.cpp 19 | 18_synchronous_custom_all_and_datum.cpp) 20 | 21 | include(${CMAKE_SOURCE_DIR}/cmake/Utils.cmake) 22 | 23 | foreach(EXAMPLE_FILE ${EXAMPLE_FILES}) 24 | 25 | get_filename_component(SOURCE_NAME ${EXAMPLE_FILE} NAME_WE) 26 | 27 | if (UNIX OR APPLE) 28 | set(EXE_NAME "${SOURCE_NAME}.bin") 29 | elseif (WIN32) 30 | set(EXE_NAME "${SOURCE_NAME}") 31 | endif () 32 | 33 | message(STATUS "Adding Example ${EXE_NAME}") 34 | add_executable(${EXE_NAME} ${EXAMPLE_FILE}) 35 | target_link_libraries(${EXE_NAME} openpose ${examples_3rdparty_libraries}) 36 | 37 | if (WIN32) 38 | set_property(TARGET ${EXE_NAME} PROPERTY FOLDER "Examples/Tutorial/C++ API") 39 | configure_file(${CMAKE_SOURCE_DIR}/cmake/OpenPose${VCXPROJ_FILE_GPU_MODE}.vcxproj.user 40 | ${CMAKE_CURRENT_BINARY_DIR}/${EXE_NAME}.vcxproj.user @ONLY) 41 | # Properties->General->Output Directory 42 | set_property(TARGET ${EXE_NAME} PROPERTY RUNTIME_OUTPUT_DIRECTORY_RELEASE ${PROJECT_BINARY_DIR}/$(Platform)/$(Configuration)) 43 | set_property(TARGET ${EXE_NAME} PROPERTY RUNTIME_OUTPUT_DIRECTORY_DEBUG ${PROJECT_BINARY_DIR}/$(Platform)/$(Configuration)) 44 | endif (WIN32) 45 | 46 | endforeach() 47 | -------------------------------------------------------------------------------- /examples/tutorial_api_cpp/README.md: -------------------------------------------------------------------------------- 1 | # C++ API Examples 2 | See the [OpenPose C++ API doc](../../doc/04_cpp_api.md) for more details on this folder. 3 | 4 | This folder provides examples to the basic OpenPose C++ API. The analogous Python API is exposed in [examples/tutorial_api_python/](../tutorial_api_python/). 5 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/01_body_from_image.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | 9 | try: 10 | # Import Openpose (Windows/Ubuntu/OSX) 11 | dir_path = os.path.dirname(os.path.realpath(__file__)) 12 | try: 13 | # Windows Import 14 | if platform == "win32": 15 | # Change these variables to point to the correct folder (Release/x64 etc.) 16 | sys.path.append(dir_path + '/../../python/openpose/Release'); 17 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 18 | import pyopenpose as op 19 | else: 20 | # Change these variables to point to the correct folder (Release/x64 etc.) 21 | sys.path.append('../../python'); 22 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 23 | # sys.path.append('/usr/local/python') 24 | from openpose import pyopenpose as op 25 | except ImportError as e: 26 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 27 | raise e 28 | 29 | # Flags 30 | parser = argparse.ArgumentParser() 31 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000192.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 32 | args = parser.parse_known_args() 33 | 34 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 35 | params = dict() 36 | params["model_folder"] = "../../../models/" 37 | 38 | # Add others in path? 39 | for i in range(0, len(args[1])): 40 | curr_item = args[1][i] 41 | if i != len(args[1])-1: next_item = args[1][i+1] 42 | else: next_item = "1" 43 | if "--" in curr_item and "--" in next_item: 44 | key = curr_item.replace('-','') 45 | if key not in params: params[key] = "1" 46 | elif "--" in curr_item and "--" not in next_item: 47 | key = curr_item.replace('-','') 48 | if key not in params: params[key] = next_item 49 | 50 | # Construct it from system arguments 51 | # op.init_argv(args[1]) 52 | # oppython = op.OpenposePython() 53 | 54 | # Starting OpenPose 55 | opWrapper = op.WrapperPython() 56 | opWrapper.configure(params) 57 | opWrapper.start() 58 | 59 | # Process Image 60 | datum = op.Datum() 61 | imageToProcess = cv2.imread(args[0].image_path) 62 | datum.cvInputData = imageToProcess 63 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 64 | 65 | # Display Image 66 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 67 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 68 | cv2.waitKey(0) 69 | except Exception as e: 70 | print(e) 71 | sys.exit(-1) 72 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/02_whole_body_from_image.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | 9 | try: 10 | # Import Openpose (Windows/Ubuntu/OSX) 11 | dir_path = os.path.dirname(os.path.realpath(__file__)) 12 | try: 13 | # Windows Import 14 | if platform == "win32": 15 | # Change these variables to point to the correct folder (Release/x64 etc.) 16 | sys.path.append(dir_path + '/../../python/openpose/Release'); 17 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 18 | import pyopenpose as op 19 | else: 20 | # Change these variables to point to the correct folder (Release/x64 etc.) 21 | sys.path.append('../../python'); 22 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 23 | # sys.path.append('/usr/local/python') 24 | from openpose import pyopenpose as op 25 | except ImportError as e: 26 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 27 | raise e 28 | 29 | # Flags 30 | parser = argparse.ArgumentParser() 31 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000241.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 32 | args = parser.parse_known_args() 33 | 34 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 35 | params = dict() 36 | params["model_folder"] = "../../../models/" 37 | params["face"] = True 38 | params["hand"] = True 39 | 40 | # Add others in path? 41 | for i in range(0, len(args[1])): 42 | curr_item = args[1][i] 43 | if i != len(args[1])-1: next_item = args[1][i+1] 44 | else: next_item = "1" 45 | if "--" in curr_item and "--" in next_item: 46 | key = curr_item.replace('-','') 47 | if key not in params: params[key] = "1" 48 | elif "--" in curr_item and "--" not in next_item: 49 | key = curr_item.replace('-','') 50 | if key not in params: params[key] = next_item 51 | 52 | # Construct it from system arguments 53 | # op.init_argv(args[1]) 54 | # oppython = op.OpenposePython() 55 | 56 | # Starting OpenPose 57 | opWrapper = op.WrapperPython() 58 | opWrapper.configure(params) 59 | opWrapper.start() 60 | 61 | # Process Image 62 | datum = op.Datum() 63 | imageToProcess = cv2.imread(args[0].image_path) 64 | datum.cvInputData = imageToProcess 65 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 66 | 67 | # Display Image 68 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 69 | print("Face keypoints: \n" + str(datum.faceKeypoints)) 70 | print("Left hand keypoints: \n" + str(datum.handKeypoints[0])) 71 | print("Right hand keypoints: \n" + str(datum.handKeypoints[1])) 72 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 73 | cv2.waitKey(0) 74 | except Exception as e: 75 | print(e) 76 | sys.exit(-1) 77 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/04_keypoints_from_images.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | import time 9 | 10 | try: 11 | # Import Openpose (Windows/Ubuntu/OSX) 12 | dir_path = os.path.dirname(os.path.realpath(__file__)) 13 | try: 14 | # Windows Import 15 | if platform == "win32": 16 | # Change these variables to point to the correct folder (Release/x64 etc.) 17 | sys.path.append(dir_path + '/../../python/openpose/Release'); 18 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 19 | import pyopenpose as op 20 | else: 21 | # Change these variables to point to the correct folder (Release/x64 etc.) 22 | sys.path.append('../../python'); 23 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 24 | # sys.path.append('/usr/local/python') 25 | from openpose import pyopenpose as op 26 | except ImportError as e: 27 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 28 | raise e 29 | 30 | # Flags 31 | parser = argparse.ArgumentParser() 32 | parser.add_argument("--image_dir", default="../../../examples/media/", help="Process a directory of images. Read all standard formats (jpg, png, bmp, etc.).") 33 | parser.add_argument("--no_display", default=False, help="Enable to disable the visual display.") 34 | args = parser.parse_known_args() 35 | 36 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 37 | params = dict() 38 | params["model_folder"] = "../../../models/" 39 | 40 | # Add others in path? 41 | for i in range(0, len(args[1])): 42 | curr_item = args[1][i] 43 | if i != len(args[1])-1: next_item = args[1][i+1] 44 | else: next_item = "1" 45 | if "--" in curr_item and "--" in next_item: 46 | key = curr_item.replace('-','') 47 | if key not in params: params[key] = "1" 48 | elif "--" in curr_item and "--" not in next_item: 49 | key = curr_item.replace('-','') 50 | if key not in params: params[key] = next_item 51 | 52 | # Construct it from system arguments 53 | # op.init_argv(args[1]) 54 | # oppython = op.OpenposePython() 55 | 56 | # Starting OpenPose 57 | opWrapper = op.WrapperPython() 58 | opWrapper.configure(params) 59 | opWrapper.start() 60 | 61 | # Read frames on directory 62 | imagePaths = op.get_images_on_directory(args[0].image_dir); 63 | start = time.time() 64 | 65 | # Process and display images 66 | for imagePath in imagePaths: 67 | datum = op.Datum() 68 | imageToProcess = cv2.imread(imagePath) 69 | datum.cvInputData = imageToProcess 70 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 71 | 72 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 73 | 74 | if not args[0].no_display: 75 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 76 | key = cv2.waitKey(15) 77 | if key == 27: break 78 | 79 | end = time.time() 80 | print("OpenPose demo successfully finished. Total time: " + str(end - start) + " seconds") 81 | except Exception as e: 82 | print(e) 83 | sys.exit(-1) 84 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/05_keypoints_from_images_multi_gpu.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | import time 9 | 10 | try: 11 | # Import Openpose (Windows/Ubuntu/OSX) 12 | dir_path = os.path.dirname(os.path.realpath(__file__)) 13 | try: 14 | # Windows Import 15 | if platform == "win32": 16 | # Change these variables to point to the correct folder (Release/x64 etc.) 17 | sys.path.append(dir_path + '/../../python/openpose/Release'); 18 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 19 | import pyopenpose as op 20 | else: 21 | # Change these variables to point to the correct folder (Release/x64 etc.) 22 | sys.path.append('../../python'); 23 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 24 | # sys.path.append('/usr/local/python') 25 | from openpose import pyopenpose as op 26 | except ImportError as e: 27 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 28 | raise e 29 | 30 | # Flags 31 | parser = argparse.ArgumentParser() 32 | parser.add_argument("--image_dir", default="../../../examples/media/", help="Process a directory of images. Read all standard formats (jpg, png, bmp, etc.).") 33 | parser.add_argument("--no_display", default=False, help="Enable to disable the visual display.") 34 | parser.add_argument("--num_gpu", default=op.get_gpu_number(), help="Number of GPUs.") 35 | args = parser.parse_known_args() 36 | 37 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 38 | params = dict() 39 | params["model_folder"] = "../../../models/" 40 | params["num_gpu"] = int(vars(args[0])["num_gpu"]) 41 | numberGPUs = int(params["num_gpu"]) 42 | 43 | # Add others in path? 44 | for i in range(0, len(args[1])): 45 | curr_item = args[1][i] 46 | if i != len(args[1])-1: next_item = args[1][i+1] 47 | else: next_item = "1" 48 | if "--" in curr_item and "--" in next_item: 49 | key = curr_item.replace('-','') 50 | if key not in params: params[key] = "1" 51 | elif "--" in curr_item and "--" not in next_item: 52 | key = curr_item.replace('-','') 53 | if key not in params: params[key] = next_item 54 | 55 | # Construct it from system arguments 56 | # op.init_argv(args[1]) 57 | # oppython = op.OpenposePython() 58 | 59 | # Starting OpenPose 60 | opWrapper = op.WrapperPython() 61 | opWrapper.configure(params) 62 | opWrapper.start() 63 | 64 | # Read frames on directory 65 | imagePaths = op.get_images_on_directory(args[0].image_dir); 66 | 67 | # Read number of GPUs in your system 68 | start = time.time() 69 | 70 | # Process and display images 71 | for imageBaseId in range(0, len(imagePaths), numberGPUs): 72 | 73 | # Create datums 74 | images = [] 75 | 76 | # Read and push images into OpenPose wrapper 77 | for gpuId in range(0, numberGPUs): 78 | 79 | imageId = imageBaseId+gpuId 80 | if imageId < len(imagePaths): 81 | 82 | imagePath = imagePaths[imageBaseId+gpuId] 83 | datum = op.Datum() 84 | images.append(cv2.imread(imagePath)) 85 | datum.cvInputData = images[-1] 86 | opWrapper.waitAndEmplace(op.VectorDatum([datum])) 87 | 88 | # Retrieve processed results from OpenPose wrapper 89 | for gpuId in range(0, numberGPUs): 90 | 91 | imageId = imageBaseId+gpuId 92 | if imageId < len(imagePaths): 93 | 94 | datums = op.VectorDatum() 95 | opWrapper.waitAndPop(datums) 96 | datum = datums[0] 97 | 98 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 99 | 100 | if not args[0].no_display: 101 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 102 | key = cv2.waitKey(15) 103 | if key == 27: break 104 | 105 | end = time.time() 106 | print("OpenPose demo successfully finished. Total time: " + str(end - start) + " seconds") 107 | except Exception as e: 108 | print(e) 109 | sys.exit(-1) 110 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/06_face_from_image.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | import time 9 | 10 | try: 11 | # Import Openpose (Windows/Ubuntu/OSX) 12 | dir_path = os.path.dirname(os.path.realpath(__file__)) 13 | try: 14 | # Windows Import 15 | if platform == "win32": 16 | # Change these variables to point to the correct folder (Release/x64 etc.) 17 | sys.path.append(dir_path + '/../../python/openpose/Release'); 18 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 19 | import pyopenpose as op 20 | else: 21 | # Change these variables to point to the correct folder (Release/x64 etc.) 22 | sys.path.append('../../python'); 23 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 24 | # sys.path.append('/usr/local/python') 25 | from openpose import pyopenpose as op 26 | except ImportError as e: 27 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 28 | raise e 29 | 30 | # Flags 31 | parser = argparse.ArgumentParser() 32 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000241.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 33 | args = parser.parse_known_args() 34 | 35 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 36 | params = dict() 37 | params["model_folder"] = "../../../models/" 38 | params["face"] = True 39 | params["face_detector"] = 2 40 | params["body"] = 0 41 | 42 | # Add others in path? 43 | for i in range(0, len(args[1])): 44 | curr_item = args[1][i] 45 | if i != len(args[1])-1: next_item = args[1][i+1] 46 | else: next_item = "1" 47 | if "--" in curr_item and "--" in next_item: 48 | key = curr_item.replace('-','') 49 | if key not in params: params[key] = "1" 50 | elif "--" in curr_item and "--" not in next_item: 51 | key = curr_item.replace('-','') 52 | if key not in params: params[key] = next_item 53 | 54 | # Construct it from system arguments 55 | # op.init_argv(args[1]) 56 | # oppython = op.OpenposePython() 57 | 58 | # Starting OpenPose 59 | opWrapper = op.WrapperPython() 60 | opWrapper.configure(params) 61 | opWrapper.start() 62 | 63 | # Read image and face rectangle locations 64 | imageToProcess = cv2.imread(args[0].image_path) 65 | faceRectangles = [ 66 | op.Rectangle(330.119385, 277.532715, 48.717274, 48.717274), 67 | op.Rectangle(24.036991, 267.918793, 65.175171, 65.175171), 68 | op.Rectangle(151.803436, 32.477852, 108.295761, 108.295761), 69 | ] 70 | 71 | # Create new datum 72 | datum = op.Datum() 73 | datum.cvInputData = imageToProcess 74 | datum.faceRectangles = faceRectangles 75 | 76 | # Process and display image 77 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 78 | print("Face keypoints: \n" + str(datum.faceKeypoints)) 79 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 80 | cv2.waitKey(0) 81 | except Exception as e: 82 | print(e) 83 | sys.exit(-1) 84 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/07_hand_from_image.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | import time 9 | 10 | try: 11 | # Import Openpose (Windows/Ubuntu/OSX) 12 | dir_path = os.path.dirname(os.path.realpath(__file__)) 13 | try: 14 | # Windows Import 15 | if platform == "win32": 16 | # Change these variables to point to the correct folder (Release/x64 etc.) 17 | sys.path.append(dir_path + '/../../python/openpose/Release'); 18 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 19 | import pyopenpose as op 20 | else: 21 | # Change these variables to point to the correct folder (Release/x64 etc.) 22 | sys.path.append('../../python'); 23 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 24 | # sys.path.append('/usr/local/python') 25 | from openpose import pyopenpose as op 26 | except ImportError as e: 27 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 28 | raise e 29 | 30 | # Flags 31 | parser = argparse.ArgumentParser() 32 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000241.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 33 | args = parser.parse_known_args() 34 | 35 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 36 | params = dict() 37 | params["model_folder"] = "../../../models/" 38 | params["hand"] = True 39 | params["hand_detector"] = 2 40 | params["body"] = 0 41 | 42 | # Add others in path? 43 | for i in range(0, len(args[1])): 44 | curr_item = args[1][i] 45 | if i != len(args[1])-1: next_item = args[1][i+1] 46 | else: next_item = "1" 47 | if "--" in curr_item and "--" in next_item: 48 | key = curr_item.replace('-','') 49 | if key not in params: params[key] = "1" 50 | elif "--" in curr_item and "--" not in next_item: 51 | key = curr_item.replace('-','') 52 | if key not in params: params[key] = next_item 53 | 54 | # Construct it from system arguments 55 | # op.init_argv(args[1]) 56 | # oppython = op.OpenposePython() 57 | 58 | # Starting OpenPose 59 | opWrapper = op.WrapperPython() 60 | opWrapper.configure(params) 61 | opWrapper.start() 62 | 63 | # Read image and face rectangle locations 64 | imageToProcess = cv2.imread(args[0].image_path) 65 | handRectangles = [ 66 | # Left/Right hands person 0 67 | [ 68 | op.Rectangle(320.035889, 377.675049, 69.300949, 69.300949), 69 | op.Rectangle(0., 0., 0., 0.), 70 | ], 71 | # Left/Right hands person 1 72 | [ 73 | op.Rectangle(80.155792, 407.673492, 80.812706, 80.812706), 74 | op.Rectangle(46.449715, 404.559753, 98.898178, 98.898178), 75 | ], 76 | # Left/Right hands person 2 77 | [ 78 | op.Rectangle(185.692673, 303.112244, 157.587555, 157.587555), 79 | op.Rectangle(88.984360, 268.866547, 117.818230, 117.818230), 80 | ] 81 | ] 82 | 83 | # Create new datum 84 | datum = op.Datum() 85 | datum.cvInputData = imageToProcess 86 | datum.handRectangles = handRectangles 87 | 88 | # Process and display image 89 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 90 | print("Left hand keypoints: \n" + str(datum.handKeypoints[0])) 91 | print("Right hand keypoints: \n" + str(datum.handKeypoints[1])) 92 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 93 | cv2.waitKey(0) 94 | except Exception as e: 95 | print(e) 96 | sys.exit(-1) 97 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/08_heatmaps_from_image.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | 9 | try: 10 | # Import Openpose (Windows/Ubuntu/OSX) 11 | dir_path = os.path.dirname(os.path.realpath(__file__)) 12 | try: 13 | # Windows Import 14 | if platform == "win32": 15 | # Change these variables to point to the correct folder (Release/x64 etc.) 16 | sys.path.append(dir_path + '/../../python/openpose/Release'); 17 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 18 | import pyopenpose as op 19 | else: 20 | # Change these variables to point to the correct folder (Release/x64 etc.) 21 | sys.path.append('../../python'); 22 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 23 | # sys.path.append('/usr/local/python') 24 | from openpose import pyopenpose as op 25 | except ImportError as e: 26 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 27 | raise e 28 | 29 | # Flags 30 | parser = argparse.ArgumentParser() 31 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000192.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 32 | args = parser.parse_known_args() 33 | 34 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 35 | params = dict() 36 | params["model_folder"] = "../../../models/" 37 | params["heatmaps_add_parts"] = True 38 | params["heatmaps_add_bkg"] = True 39 | params["heatmaps_add_PAFs"] = True 40 | params["heatmaps_scale"] = 2 41 | 42 | # Add others in path? 43 | for i in range(0, len(args[1])): 44 | curr_item = args[1][i] 45 | if i != len(args[1])-1: next_item = args[1][i+1] 46 | else: next_item = "1" 47 | if "--" in curr_item and "--" in next_item: 48 | key = curr_item.replace('-','') 49 | if key not in params: params[key] = "1" 50 | elif "--" in curr_item and "--" not in next_item: 51 | key = curr_item.replace('-','') 52 | if key not in params: params[key] = next_item 53 | 54 | # Construct it from system arguments 55 | # op.init_argv(args[1]) 56 | # oppython = op.OpenposePython() 57 | 58 | # Starting OpenPose 59 | opWrapper = op.WrapperPython() 60 | opWrapper.configure(params) 61 | opWrapper.start() 62 | 63 | # Process Image 64 | datum = op.Datum() 65 | imageToProcess = cv2.imread(args[0].image_path) 66 | datum.cvInputData = imageToProcess 67 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 68 | 69 | # Process outputs 70 | outputImageF = (datum.inputNetData[0].copy())[0,:,:,:] + 0.5 71 | outputImageF = cv2.merge([outputImageF[0,:,:], outputImageF[1,:,:], outputImageF[2,:,:]]) 72 | outputImageF = (outputImageF*255.).astype(dtype='uint8') 73 | heatmaps = datum.poseHeatMaps.copy() 74 | heatmaps = (heatmaps).astype(dtype='uint8') 75 | 76 | # Display Image 77 | counter = 0 78 | while 1: 79 | num_maps = heatmaps.shape[0] 80 | heatmap = heatmaps[counter, :, :].copy() 81 | heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET) 82 | combined = cv2.addWeighted(outputImageF, 0.5, heatmap, 0.5, 0) 83 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", combined) 84 | key = cv2.waitKey(-1) 85 | if key == 27: 86 | break 87 | counter += 1 88 | counter = counter % num_maps 89 | except Exception as e: 90 | print(e) 91 | sys.exit(-1) 92 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/09_keypoints_from_heatmaps.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | import numpy as np 9 | 10 | try: 11 | # Import Openpose (Windows/Ubuntu/OSX) 12 | dir_path = os.path.dirname(os.path.realpath(__file__)) 13 | try: 14 | # Windows Import 15 | if platform == "win32": 16 | # Change these variables to point to the correct folder (Release/x64 etc.) 17 | sys.path.append(dir_path + '/../../python/openpose/Release'); 18 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 19 | import pyopenpose as op 20 | else: 21 | # Change these variables to point to the correct folder (Release/x64 etc.) 22 | sys.path.append('../../python'); 23 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 24 | # sys.path.append('/usr/local/python') 25 | from openpose import pyopenpose as op 26 | except ImportError as e: 27 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 28 | raise e 29 | 30 | # Flags 31 | parser = argparse.ArgumentParser() 32 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000294.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 33 | args = parser.parse_known_args() 34 | 35 | # Load image 36 | imageToProcess = cv2.imread(args[0].image_path) 37 | 38 | def get_sample_heatmaps(): 39 | # These parameters are globally set. You need to unset variables set here if you have a new OpenPose object. See * 40 | params = dict() 41 | params["model_folder"] = "../../../models/" 42 | params["heatmaps_add_parts"] = True 43 | params["heatmaps_add_bkg"] = True 44 | params["heatmaps_add_PAFs"] = True 45 | params["heatmaps_scale"] = 3 46 | params["upsampling_ratio"] = 1 47 | params["body"] = 1 48 | 49 | # Starting OpenPose 50 | opWrapper = op.WrapperPython() 51 | opWrapper.configure(params) 52 | opWrapper.start() 53 | 54 | # Process Image and get heatmap 55 | datum = op.Datum() 56 | imageToProcess = cv2.imread(args[0].image_path) 57 | datum.cvInputData = imageToProcess 58 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 59 | poseHeatMaps = datum.poseHeatMaps.copy() 60 | opWrapper.stop() 61 | 62 | return poseHeatMaps 63 | 64 | # Get Heatmap 65 | poseHeatMaps = get_sample_heatmaps() 66 | 67 | # Starting OpenPose 68 | params = dict() 69 | params["model_folder"] = "../../../models/" 70 | params["body"] = 2 # Disable OP Network 71 | params["upsampling_ratio"] = 0 # * Unset this variable 72 | opWrapper = op.WrapperPython() 73 | opWrapper.configure(params) 74 | opWrapper.start() 75 | 76 | # Pass Heatmap and Run OP 77 | datum = op.Datum() 78 | datum.cvInputData = imageToProcess 79 | datum.poseNetOutput = poseHeatMaps 80 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 81 | 82 | # Display Image 83 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 84 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 85 | cv2.waitKey(0) 86 | except Exception as e: 87 | print(e) 88 | sys.exit(-1) 89 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/12_asynchronous_custom_output.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | 9 | 10 | def display(datums): 11 | datum = datums[0] 12 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 13 | key = cv2.waitKey(1) 14 | return (key == 27) 15 | 16 | 17 | def printKeypoints(datums): 18 | datum = datums[0] 19 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 20 | print("Face keypoints: \n" + str(datum.faceKeypoints)) 21 | print("Left hand keypoints: \n" + str(datum.handKeypoints[0])) 22 | print("Right hand keypoints: \n" + str(datum.handKeypoints[1])) 23 | 24 | 25 | try: 26 | # Import Openpose (Windows/Ubuntu/OSX) 27 | dir_path = os.path.dirname(os.path.realpath(__file__)) 28 | try: 29 | # Windows Import 30 | if platform == "win32": 31 | # Change these variables to point to the correct folder (Release/x64 etc.) 32 | sys.path.append(dir_path + '/../../python/openpose/Release'); 33 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 34 | import pyopenpose as op 35 | else: 36 | # Change these variables to point to the correct folder (Release/x64 etc.) 37 | sys.path.append('../../python'); 38 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 39 | # sys.path.append('/usr/local/python') 40 | from openpose import pyopenpose as op 41 | except ImportError as e: 42 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 43 | raise e 44 | 45 | # Flags 46 | parser = argparse.ArgumentParser() 47 | parser.add_argument("--no-display", action="store_true", help="Disable display.") 48 | args = parser.parse_known_args() 49 | 50 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 51 | params = dict() 52 | params["model_folder"] = "../../../models/" 53 | 54 | # Add others in path? 55 | for i in range(0, len(args[1])): 56 | curr_item = args[1][i] 57 | if i != len(args[1])-1: next_item = args[1][i+1] 58 | else: next_item = "1" 59 | if "--" in curr_item and "--" in next_item: 60 | key = curr_item.replace('-','') 61 | if key not in params: params[key] = "1" 62 | elif "--" in curr_item and "--" not in next_item: 63 | key = curr_item.replace('-','') 64 | if key not in params: params[key] = next_item 65 | 66 | # Construct it from system arguments 67 | # op.init_argv(args[1]) 68 | # oppython = op.OpenposePython() 69 | 70 | # Starting OpenPose 71 | opWrapper = op.WrapperPython(op.ThreadManagerMode.AsynchronousOut) 72 | opWrapper.configure(params) 73 | opWrapper.start() 74 | 75 | # Main loop 76 | userWantsToExit = False 77 | while not userWantsToExit: 78 | # Pop frame 79 | datumProcessed = op.VectorDatum() 80 | if opWrapper.waitAndPop(datumProcessed): 81 | if not args[0].no_display: 82 | # Display image 83 | userWantsToExit = display(datumProcessed) 84 | printKeypoints(datumProcessed) 85 | else: 86 | break 87 | except Exception as e: 88 | print(e) 89 | sys.exit(-1) 90 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | ### Add Python Test 2 | configure_file(01_body_from_image.py 01_body_from_image.py) 3 | configure_file(02_whole_body_from_image.py 02_whole_body_from_image.py) 4 | configure_file(04_keypoints_from_images.py 04_keypoints_from_images.py) 5 | configure_file(05_keypoints_from_images_multi_gpu.py 05_keypoints_from_images_multi_gpu.py) 6 | configure_file(06_face_from_image.py 06_face_from_image.py) 7 | configure_file(07_hand_from_image.py 07_hand_from_image.py) 8 | configure_file(08_heatmaps_from_image.py 08_heatmaps_from_image.py) 9 | configure_file(09_keypoints_from_heatmaps.py 09_keypoints_from_heatmaps.py) 10 | configure_file(openpose_python.py openpose_python.py) 11 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/README.md: -------------------------------------------------------------------------------- 1 | # Python API Examples 2 | See the [OpenPose Python API doc](../../doc/03_python_api.md) for more details on this folder. 3 | 4 | This folder provides examples to the basic OpenPose Python API. The analogous C++ API is exposed in [examples/tutorial_api_cpp/](../tutorial_api_cpp/). 5 | -------------------------------------------------------------------------------- /examples/tutorial_api_python/openpose_python.py: -------------------------------------------------------------------------------- 1 | # From Python 2 | # It requires OpenCV installed for Python 3 | import sys 4 | import cv2 5 | import os 6 | from sys import platform 7 | import argparse 8 | 9 | try: 10 | # Import Openpose (Windows/Ubuntu/OSX) 11 | dir_path = os.path.dirname(os.path.realpath(__file__)) 12 | try: 13 | # Windows Import 14 | if platform == "win32": 15 | # Change these variables to point to the correct folder (Release/x64 etc.) 16 | sys.path.append(dir_path + '/../../python/openpose/Release'); 17 | os.environ['PATH'] = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' + dir_path + '/../../bin;' 18 | import pyopenpose as op 19 | else: 20 | # Change these variables to point to the correct folder (Release/x64 etc.) 21 | sys.path.append('../../python'); 22 | # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it. 23 | # sys.path.append('/usr/local/python') 24 | from openpose import pyopenpose as op 25 | except ImportError as e: 26 | print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?') 27 | raise e 28 | 29 | # Flags 30 | parser = argparse.ArgumentParser() 31 | parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000192.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).") 32 | args = parser.parse_known_args() 33 | 34 | # Custom Params (refer to include/openpose/flags.hpp for more parameters) 35 | params = dict() 36 | params["model_folder"] = "../../../models/" 37 | 38 | # Add others in path? 39 | for i in range(0, len(args[1])): 40 | curr_item = args[1][i] 41 | if i != len(args[1])-1: next_item = args[1][i+1] 42 | else: next_item = "1" 43 | if "--" in curr_item and "--" in next_item: 44 | key = curr_item.replace('-','') 45 | if key not in params: params[key] = "1" 46 | elif "--" in curr_item and "--" not in next_item: 47 | key = curr_item.replace('-','') 48 | if key not in params: params[key] = next_item 49 | 50 | # Construct it from system arguments 51 | # op.init_argv(args[1]) 52 | # oppython = op.OpenposePython() 53 | 54 | # Starting OpenPose 55 | opWrapper = op.WrapperPython(op.ThreadManagerMode.Synchronous) 56 | opWrapper.configure(params) 57 | opWrapper.execute() 58 | except Exception as e: 59 | print(e) 60 | sys.exit(-1) 61 | -------------------------------------------------------------------------------- /keypoints_hand.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/keypoints_hand.png -------------------------------------------------------------------------------- /keypoints_pose_25.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/keypoints_pose_25.png -------------------------------------------------------------------------------- /models/cameraParameters/flir/17012332.xml.example: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 3 6 | 4 7 |
d
8 | 9 | 1. 0. 0. 0. 10 | 0. 1. 0. 0. 11 | 0. 0. 1. 0. 12 | 13 | 3 14 | 3 15 |
d
16 | 17 | 8.1793481631740565e+02 0. 6.0070689997785121e+02 0. 18 | 8.1651774059837908e+02 5.1784529566329593e+02 0. 0. 1. 19 | 20 | 8 21 | 1 22 |
d
23 | 24 | -1.8102158829399091e+00 9.1966147162623262e+00 25 | -4.4293900343777355e-04 1.3638377686816653e-03 26 | 1.3303863414979364e+00 -1.4189051636354870e+00 27 | 8.4725535468475819e+00 4.7911023525901033e+00 28 | 29 | -------------------------------------------------------------------------------- /models/getModels.bat: -------------------------------------------------------------------------------- 1 | :: Avoid printing all the comments in the Windows cmd 2 | @echo off 3 | 4 | echo ------------------------- BODY, FOOT, FACE, AND HAND MODELS ------------------------- 5 | echo ----- Downloading body pose (COCO and MPI), face and hand models ----- 6 | SET WGET_EXE=..\3rdparty\windows\wget\wget.exe 7 | SET OPENPOSE_URL=http://posefs1.perception.cs.cmu.edu/OpenPose/models/ 8 | SET POSE_FOLDER=pose/ 9 | SET FACE_FOLDER=face/ 10 | SET HAND_FOLDER=hand/ 11 | 12 | echo: 13 | echo ------------------------- POSE (BODY+FOOT) MODELS ------------------------- 14 | echo Body (BODY_25) 15 | set BODY_25_FOLDER=%POSE_FOLDER%body_25/ 16 | set BODY_25_MODEL=%BODY_25_FOLDER%pose_iter_584000.caffemodel 17 | %WGET_EXE% -c %OPENPOSE_URL%%BODY_25_MODEL% -P %BODY_25_FOLDER% 18 | 19 | echo Body (COCO) 20 | SET COCO_FOLDER=%POSE_FOLDER%coco/ 21 | SET COCO_MODEL=%COCO_FOLDER%pose_iter_440000.caffemodel 22 | %WGET_EXE% -c %OPENPOSE_URL%%COCO_MODEL% -P %COCO_FOLDER% 23 | 24 | echo: 25 | echo Body (MPI) 26 | SET MPI_FOLDER=%POSE_FOLDER%mpi/ 27 | SET MPI_MODEL=%MPI_FOLDER%pose_iter_160000.caffemodel 28 | %WGET_EXE% -c %OPENPOSE_URL%%MPI_MODEL% -P %MPI_FOLDER% 29 | echo ----------------------- POSE DOWNLOADED ----------------------- 30 | 31 | echo: 32 | echo ------------------------- FACE MODELS ------------------------- 33 | echo Face 34 | SET FACE_MODEL=%FACE_FOLDER%pose_iter_116000.caffemodel 35 | %WGET_EXE% -c %OPENPOSE_URL%%FACE_MODEL% -P %FACE_FOLDER% 36 | echo ----------------------- FACE DOWNLOADED ----------------------- 37 | 38 | echo: 39 | echo ------------------------- HAND MODELS ------------------------- 40 | echo Hand 41 | SET HAND_MODEL=%HAND_FOLDER%pose_iter_102000.caffemodel 42 | %WGET_EXE% -c %OPENPOSE_URL%%HAND_MODEL% -P %HAND_FOLDER% 43 | echo ----------------------- HAND DOWNLOADED ----------------------- 44 | -------------------------------------------------------------------------------- /models/getModels.sh: -------------------------------------------------------------------------------- 1 | # ------------------------- BODY, FOOT, FACE, AND HAND MODELS ------------------------- 2 | # Downloading body pose (COCO and MPI), face and hand models 3 | OPENPOSE_URL="http://posefs1.perception.cs.cmu.edu/OpenPose/models/" 4 | POSE_FOLDER="pose/" 5 | FACE_FOLDER="face/" 6 | HAND_FOLDER="hand/" 7 | 8 | # ------------------------- POSE (BODY+FOOT) MODELS ------------------------- 9 | # Body (BODY_25) 10 | BODY_25_FOLDER=${POSE_FOLDER}"body_25/" 11 | BODY_25_MODEL=${BODY_25_FOLDER}"pose_iter_584000.caffemodel" 12 | wget -c ${OPENPOSE_URL}${BODY_25_MODEL} -P ${BODY_25_FOLDER} 13 | 14 | # Body (COCO) 15 | COCO_FOLDER=${POSE_FOLDER}"coco/" 16 | COCO_MODEL=${COCO_FOLDER}"pose_iter_440000.caffemodel" 17 | wget -c ${OPENPOSE_URL}${COCO_MODEL} -P ${COCO_FOLDER} 18 | # Alternative: it will not check whether file was fully downloaded 19 | # if [ ! -f $COCO_MODEL ]; then 20 | # wget ${OPENPOSE_URL}$COCO_MODEL -P $COCO_FOLDER 21 | # fi 22 | 23 | # Body (MPI) 24 | MPI_FOLDER=${POSE_FOLDER}"mpi/" 25 | MPI_MODEL=${MPI_FOLDER}"pose_iter_160000.caffemodel" 26 | wget -c ${OPENPOSE_URL}${MPI_MODEL} -P ${MPI_FOLDER} 27 | 28 | # "------------------------- FACE MODELS -------------------------" 29 | # Face 30 | FACE_MODEL=${FACE_FOLDER}"pose_iter_116000.caffemodel" 31 | wget -c ${OPENPOSE_URL}${FACE_MODEL} -P ${FACE_FOLDER} 32 | 33 | # "------------------------- HAND MODELS -------------------------" 34 | # Hand 35 | HAND_MODEL=$HAND_FOLDER"pose_iter_102000.caffemodel" 36 | wget -c ${OPENPOSE_URL}${HAND_MODEL} -P ${HAND_FOLDER} 37 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # torch and torchvision can only be installed with .whl, otherwise it will not install correctly 2 | # torch==1.2.0 3 | # torchvision==0.4.0 4 | opencv_python==4.1.2.30 5 | numpy==1.17.0 6 | matplotlib==3.1.2 7 | tqdm==4.60.0 8 | Pillow==8.3.2 9 | torchsummary==1.5.1 10 | scipy==1.2.1 11 | h5py==2.10.0 12 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | # 此文件用来进行图片骨架检测的测试 2 | 3 | import bin.pyopenpose as op # 注意要3.7才能正确import 4 | import matplotlib.pyplot as plt 5 | import cv2 6 | import numpy as np 7 | 8 | filePath = 'examples/media/COCO_val2014_000000000395.jpg' 9 | 10 | # 找到模型文件 11 | params = dict() 12 | params["model_folder"] = "models/" 13 | params["net_resolution"] = "256x256" # 大小要设置成1：1的，且是2的倍数，1660ti超过256就爆显存了 14 | params["hand"] = True 15 | 16 | # opWrapper用来启动openpose 17 | opWrapper = op.WrapperPython() 18 | opWrapper.configure(params) 19 | opWrapper.start() 20 | 21 | # 加载图片到openpose的运行时里 22 | imageToProcess = cv2.imread(filePath) 23 | datum = op.Datum() 24 | datum.cvInputData = imageToProcess 25 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 26 | 27 | # 打印出点的位置 28 | # datum.poseKeypoints的类型是ndarray 29 | print("Body keypoints: \n" + str(datum.poseKeypoints)) 30 | print("Left hand keypoints: \n" + str(datum.handKeypoints[0])) 31 | print("Right hand keypoints: \n" + str(datum.handKeypoints[1])) 32 | 33 | # 显示图片 34 | cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData) 35 | cv2.waitKey(0) 36 | -------------------------------------------------------------------------------- /test_video.py: -------------------------------------------------------------------------------- 1 | # 此文件用来进行视频骨架检测的测试 2 | 3 | 4 | import bin.pyopenpose as op # 注意要3.7才能正确import，版本与文件名对应 5 | import matplotlib.pyplot as plt 6 | import cv2 7 | import time 8 | 9 | # 视频路径，若要打开摄像头，设为0 10 | filePath = 'examples/media/test.mp4' 11 | 12 | # 找到模型文件 13 | params = dict() 14 | params["model_folder"] = "models/" 15 | params["net_resolution"] = "256x256" # 大小要设置成1：1的，且是2的倍数，1660ti超过256就爆显存了 16 | params["hand"] = True 17 | 18 | # opWrapper用来启动openpose 19 | opWrapper = op.WrapperPython() 20 | opWrapper.configure(params) 21 | opWrapper.start() 22 | 23 | 24 | # 打开视频 25 | cap = cv2.VideoCapture(filePath) 26 | 27 | fps = 0 28 | while True: 29 | t0 = time.time() 30 | # 截取一帧图片 31 | flag, frame = cap.read() 32 | if flag == False: 33 | break 34 | # 将图片到加载openpose的运行时里 35 | datum = op.Datum() 36 | datum.cvInputData = frame 37 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 38 | 39 | # 打印出点的位置 40 | # datum.poseKeypoints的类型是ndarray 41 | # print("Body keypoints: \n" + str(datum.poseKeypoints)) 42 | 43 | 44 | # 显示图片 45 | cv2.imshow('', datum.cvOutputData) 46 | if cv2.waitKey(10) & 0xFF == 27: 47 | break 48 | 49 | # 计算并打印fps 50 | t1 = time.time() 51 | fps = 1/(t1 - t0) 52 | print('fps = {:.3f}'.format(fps)) 53 | -------------------------------------------------------------------------------- /test_video_track_point.py: -------------------------------------------------------------------------------- 1 | import bin.pyopenpose as op # 注意要3.7才能正确import，版本与文件名对应 2 | 3 | import matplotlib.pyplot as plt 4 | import numpy as np 5 | import cv2 6 | import time 7 | from PIL import Image 8 | 9 | # 视频路径，若要打开摄像头，设为0 10 | filePath = 'examples/media/test4.mp4' 11 | 12 | # 要追踪的关节点，点的定义在kepoints_hand.png和kepoints_pose_25.png里 13 | body_point_list = [1, 2, 3, 4, 5, 6, 7] # 7个 14 | left_hand_point_lsit = [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 20] # 16个 15 | right_hand_point_list = [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 20] # 16个 16 | # left_hand_point_lsit = [0] # 1 17 | # right_hand_point_list = [0] # 1 18 | 19 | 20 | # 身体和手部的颜色定义 21 | body_color = [int(36.4*(i+1)) for i in range(7)] 22 | hand_color = [int(15.9*(i+1)) for i in range(16)] 23 | # print(body_color) 24 | # print(hand_color) 25 | 26 | # 找到模型文件 27 | params = dict() 28 | params["model_folder"] = "models/" 29 | params["net_resolution"] = "256x256" # 大小要设置成1：1的，且是2的倍数，1660ti超过256就爆显存了 30 | params["hand"] = True 31 | 32 | # opWrapper用来启动openpose 33 | opWrapper = op.WrapperPython() 34 | opWrapper.configure(params) 35 | opWrapper.start() 36 | 37 | 38 | # 打开视频 39 | cap = cv2.VideoCapture(filePath) 40 | 41 | # 轨迹图 42 | track_img = np.zeros(shape=(1080, 1920, 3), dtype=np.int8) 43 | 44 | fps = 0 45 | while True: 46 | t0 = time.time() 47 | # 截取一帧图片 48 | flag, frame = cap.read() 49 | if flag == False: 50 | break 51 | # 将图片到加载openpose的运行时里做检测 52 | datum = op.Datum() 53 | datum.cvInputData = frame 54 | opWrapper.emplaceAndPop(op.VectorDatum([datum])) 55 | 56 | # 打印出点的位置 57 | # datum.poseKeypoints的类型是ndarray 58 | # print("Body keypoints: \n" + str(datum.poseKeypoints)) 59 | 60 | # 获取关节点 61 | body_point = datum.poseKeypoints 62 | left_hand_point = datum.handKeypoints[0] 63 | right_hand_point = datum.handKeypoints[1] 64 | 65 | 66 | # 绘制点轨迹图 67 | color_idx = 0 68 | for idx in body_point_list: 69 | cv2.circle(track_img, (int(body_point[0][idx][0]),int(body_point[0][idx][1])), 1, (0, 0, body_color[color_idx]), 5) 70 | color_idx += 1 71 | 72 | color_idx = 0 73 | for idx in left_hand_point_lsit: 74 | cv2.circle(track_img, (int(left_hand_point[0][idx][0]),int(left_hand_point[0][idx][1])), 1, (0, hand_color[color_idx], 0), 5) 75 | color_idx += 1 76 | 77 | color_idx = 0 78 | for idx in right_hand_point_list: 79 | cv2.circle(track_img, (int(right_hand_point[0][idx][0]),int(right_hand_point[0][idx][1])), 1, (hand_color[color_idx], 0, 0), 5) 80 | color_idx += 1 81 | 82 | 83 | # 显示图片 84 | cv2.imshow('', datum.cvOutputData) 85 | if cv2.waitKey(10) & 0xFF == 27: 86 | break 87 | 88 | # 计算并打印fps 89 | t1 = time.time() 90 | # fps = 1/(t1 - t0) 91 | # print('fps = {:.3f}'.format(fps)) 92 | 93 | # 保存图片 94 | track_img = np.uint8(track_img) 95 | cv2.imwrite('track_of_point.jpg', track_img) 96 | 97 | 98 | 99 | ################################################################################ 100 | ################################################################################ 101 | 102 | # 取消注释下面这几行，就可以把图片送到classification中去做分类 103 | 104 | # from classification.classification_pose import Classification 105 | 106 | # # 将BGR转成RGB 107 | # track_img = cv2.cvtColor(track_img, cv2.COLOR_BGR2RGB) 108 | 109 | # # 分类器 110 | # classification = Classification() 111 | 112 | # # 传到classification中去分类 113 | # track_img = Image.fromarray(track_img) 114 | # class_name = classification.detect_image(track_img) 115 | # print(class_name) -------------------------------------------------------------------------------- /track_of_point.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/track_of_point.jpg -------------------------------------------------------------------------------- /文本材料/1.作品简介.doc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/文本材料/1.作品简介.doc -------------------------------------------------------------------------------- /文本材料/2.引用说明.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/文本材料/2.引用说明.docx -------------------------------------------------------------------------------- /文本材料/3.安装说明.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owo12321/Sign-Language-Recognition/e54c8a777b7ba64ecabde5ff219a5dec35e0d776/文本材料/3.安装说明.docx --------------------------------------------------------------------------------