├── README.md ├── yolov8pose_herizon ├── 01_check.sh ├── 02_preprocess.sh ├── 03_build.sh ├── 04_inference.sh ├── 05_evaluate.sh ├── cal_data │ └── test.jpg ├── data_preprocess.py ├── hb_mapper_checker.log ├── hb_mapper_makertbin.log ├── inference_image_demo.py ├── model │ └── yolov8pos_relu_zq.onnx ├── model_output │ ├── yolov8seg.bin │ ├── yolov8seg_optimized_float_model.onnx │ ├── yolov8seg_original_float_model.onnx │ └── yolov8seg_quantized_model.onnx ├── preprocess.py ├── src_data │ └── test.jpg ├── test.jpg ├── test_horizon_result.jpg └── yolov8pose_config.yaml ├── yolov8pose_onnx ├── test.jpg ├── test_onnx_result.jpg ├── test_pytorch_result.jpg ├── yolov8-pose_onnx_demo.py └── yolov8pos_relu_zq.onnx ├── yolov8pose_rknn ├── data │ └── test.jpg ├── dataset.txt ├── onnx2rknn_demo_ZQ.py ├── test.jpg ├── test_rknn_result.jpg ├── yolov8pos_relu_zq.onnx └── yolov8pos_relu_zq.rknn └── yolov8pose_tensorRT ├── onnx2trt_rt7.py ├── tensorRT_inferenc_demo.py ├── test.jpg ├── test_result_tensorRT.jpg ├── yolov8pos_relu_zq.onnx └── yolov8pos_relu_zq.trt /README.md: -------------------------------------------------------------------------------- 1 | # yolov8pose_onnx_tensorRT_rknn_horizon 2 | yolov8pose 部署版本，便于移植不同平台（onnx、tensorRT、rknn、Horizon）。 3 | 4 | ***特别说明：本示例提供的代码只适用按照本链接提供的方式导出的onnx，导出onnx方式参考后文链接。本示例中模型的训练使用的数据不多，模型效果无法保证，只是用来测试部署用的，如果换其他图像可能存在检测不到的现象。*** 5 | 6 | # 文件夹结构说明 7 | 8 | yolov8pose_onnx：onnx模型、测试图像、测试结果、pytorch测试结果、测试demo脚本 9 | 10 | yolov8pose_TensorRT：TensorRT版本模型、测试图像、测试结果、测试demo脚本、onnx模型、onnx2tensorRT脚本(tensorRT-7.2.3.4) 11 | 12 | yolov8pose_rknn：rknn模型、测试（量化）图像、测试结果、onnx2rknn转换测试脚本 13 | 14 | yolov8pose_horizon：地平线模型、测试（量化）图像、测试结果、转换测试脚本、测试量化后onnx模型脚本 15 | 16 | # 测试结果 17 | 18 | pytyorch 测试结果 19 | 20 | ![image](https://github.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/blob/main/yolov8pose_onnx/test_pytorch_result.jpg) 21 | 22 | onnx测试结果 23 | 24 | ![image](https://github.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/blob/main/yolov8pose_onnx/test_onnx_result.jpg) 25 | 26 | （注：图片来源coco数据集） 27 | 28 | 说明：推理测试预处理没有考虑等比率缩放，激活函数 SiLU 用 Relu 进行了替换。由于模型训练使用的数据并不多，且迭代的次数不多，效果并不是很好，仅供测试流程用。 29 | 30 | 导出onnx参考 [yolov8pose 瑞芯微RKNN芯片、地平线Horizon芯片、TensorRT部署](https://blog.csdn.net/zhangqian_1/article/details/131857506) 31 | 32 | # 相关链接 33 | 34 | ## yolov8 检测部署 35 | 36 | 部署导出方高效方式参考 [类似本实例中导出检测方式](https://blog.csdn.net/zhangqian_1/article/details/128918268) 37 | 38 | 官方导出onnx方式板端部署方式参考 [官方导出onnx方式部署](https://blog.csdn.net/zhangqian_1/article/details/130754564) 39 | 40 | rknn的板端C++部署参考 [C++部署](https://github.com/cqu20160901/yolov8n_onnx_tensorRT_rknn_horizon) 41 | 42 | ## yolov8seg 部署 43 | 44 | yolov8seg 部署 [模型+代码](https://github.com/cqu20160901/yolov8seg_onnx_tensorRT_rknn_horizon) 45 | 46 | 47 | 48 | -------------------------------------------------------------------------------- /yolov8pose_herizon/01_check.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 3 | # 4 | # The material in this file is confidential and contains trade secrets 5 | # of Horizon Robotics Inc. This is proprietary information owned by 6 | # Horizon Robotics Inc. No part of this work may be disclosed, 7 | # reproduced, copied, transmitted, or used in any way for any purpose, 8 | # without the express written permission of Horizon Robotics Inc. 9 | 10 | set -e -v 11 | cd $(dirname $0) || exit 12 | 13 | model_type="onnx" 14 | onnx_model="./model/yolov8pos_relu_zq.onnx" 15 | output="./yolov8_checker.log" 16 | march="bernoulli2" 17 | 18 | hb_mapper checker --model-type ${model_type} \ 19 | --model ${onnx_model} \ 20 | --output ${output} --march ${march} 21 | -------------------------------------------------------------------------------- /yolov8pose_herizon/02_preprocess.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 3 | # 4 | # The material in this file is confidential and contains trade secrets 5 | # of Horizon Robotics Inc. This is proprietary information owned by 6 | # Horizon Robotics Inc. No part of this work may be disclosed, 7 | # reproduced, copied, transmitted, or used in any way for any purpose, 8 | # without the express written permission of Horizon Robotics Inc. 9 | 10 | set -e -v 11 | cd $(dirname $0) || exit 12 | 13 | python3 data_preprocess.py \ 14 | --src_dir ./src_data \ 15 | --dst_dir ./cal_data \ 16 | --pic_ext .rgb \ 17 | --read_mode opencv 18 | -------------------------------------------------------------------------------- /yolov8pose_herizon/03_build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 3 | # 4 | # The material in this file is confidential and contains trade secrets 5 | # of Horizon Robotics Inc. This is proprietary information owned by 6 | # Horizon Robotics Inc. No part of this work may be disclosed, 7 | # reproduced, copied, transmitted, or used in any way for any purpose, 8 | # without the express written permission of Horizon Robotics Inc. 9 | 10 | set -e -v 11 | cd $(dirname $0) 12 | config_file="./yolov8pose_config.yaml" 13 | model_type="onnx" 14 | # build model 15 | hb_mapper makertbin --config ${config_file} \ 16 | --model-type ${model_type} 17 | -------------------------------------------------------------------------------- /yolov8pose_herizon/04_inference.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 3 | # 4 | # The material in this file is confidential and contains trade secrets 5 | # of Horizon Robotics Inc. This is proprietary information owned by 6 | # Horizon Robotics Inc. No part of this work may be disclosed, 7 | # reproduced, copied, transmitted, or used in any way for any purpose, 8 | # without the express written permission of Horizon Robotics Inc. 9 | 10 | python3 -u inference_image_demo.py 11 | -------------------------------------------------------------------------------- /yolov8pose_herizon/05_evaluate.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 3 | # 4 | # The material in this file is confidential and contains trade secrets 5 | # of Horizon Robotics Inc. This is proprietary information owned by 6 | # Horizon Robotics Inc. No part of this work may be disclosed, 7 | # reproduced, copied, transmitted, or used in any way for any purpose, 8 | # without the express written permission of Horizon Robotics Inc. 9 | 10 | set -v -e 11 | cd $(dirname $0) || exit 12 | 13 | #for converted quanti model evaluation 14 | quanti_model_file="./model_output/yolov5_672x672_nv12_quantized_model.onnx" 15 | quanti_input_layout="NHWC" 16 | 17 | original_model_file="./model_output/yolov5_672x672_nv12_original_float_model.onnx" 18 | original_input_layout="NCHW" 19 | 20 | if [[ $1 =~ "origin" ]]; then 21 | model=$original_model_file 22 | layout=$original_input_layout 23 | input_offset=128 24 | else 25 | model=$quanti_model_file 26 | layout=$quanti_input_layout 27 | input_offset=128 28 | fi 29 | 30 | image_path="../../../01_common/data/coco/coco_val2017/images/" 31 | anno_path="../../../01_common/data/coco/coco_val2017/annotations/instances_val2017.json" 32 | 33 | 34 | if [ -z $2 ]; then 35 | total_image_number=5000 36 | else 37 | total_image_number=$2 38 | fi 39 | 40 | # ------------------------------------------------------------------------------------------------------------- 41 | # shell command "sh 05_evaluate.sh" runs quanti full evaluation by default 42 | # If quanti model eval is intended, please run the shell via command "sh 05_evaluate.sh quanti" 43 | # If float model eval is intended, please run the shell via command "sh 05_evaluate.sh origin"# 44 | # If quanti model quick eval test is intended, please run the shell via command "sh 05_evaluate.sh quanti 20" 45 | # If float model quick eval test is intended, please run the shell via command "sh 05_evaluate.sh origin 20" 46 | # ------------------------------------------------------------------------------------------------------------- 47 | # quanti model eval 48 | python3 -u ../../det_evaluate.py \ 49 | --model=${model} \ 50 | --image_path=${image_path} \ 51 | --annotation_path=${anno_path} \ 52 | --input_layout=${layout} \ 53 | --total_image_number=${total_image_number} \ 54 | --input_offset ${input_offset} 55 | -------------------------------------------------------------------------------- /yolov8pose_herizon/cal_data/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/cal_data/test.jpg -------------------------------------------------------------------------------- /yolov8pose_herizon/data_preprocess.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 2 | # 3 | # The material in this file is confidential and contains trade secrets 4 | # of Horizon Robotics Inc. This is proprietary information owned by 5 | # Horizon Robotics Inc. No part of this work may be disclosed, 6 | # reproduced, copied, transmitted, or used in any way for any purpose, 7 | # without the express written permission of Horizon Robotics Inc. 8 | 9 | import os 10 | import sys 11 | sys.path.append('.') 12 | 13 | import click 14 | import numpy as np 15 | from preprocess import calibration_transformers 16 | import skimage.io 17 | import cv2 18 | 19 | transformers = calibration_transformers() 20 | 21 | sys.path.append("../../../01_common/python/data/") 22 | from dataloader import DataLoader 23 | from dataset import CifarDataset 24 | 25 | regular_process_list = [ 26 | ".rgb", 27 | ".rgbp", 28 | ".bgr", 29 | ".bgrp", 30 | ".yuv", 31 | ".feature", 32 | ".cali", 33 | ] 34 | 35 | 36 | def read_image(src_file, read_mode): 37 | if read_mode == "skimage": 38 | image = skimage.img_as_float(skimage.io.imread(src_file)).astype( 39 | np.float32) 40 | elif read_mode == "opencv": 41 | image = cv2.imread(src_file) 42 | else: 43 | raise ValueError(f"Invalid read mode {read_mode}") 44 | if image.ndim != 3: # expend gray scale image to three channels 45 | image = image[..., np.newaxis] 46 | image = np.concatenate([image, image, image], axis=-1) 47 | return image 48 | 49 | 50 | def regular_preprocess(src_file, transformers, dst_dir, pic_ext, read_mode): 51 | image = [read_image(src_file, read_mode)] 52 | for trans in transformers: 53 | image = trans(image) 54 | 55 | filename = os.path.basename(src_file) 56 | short_name, ext = os.path.splitext(filename) 57 | pic_name = os.path.join(dst_dir, short_name + pic_ext) 58 | print("write:%s" % pic_name) 59 | dtype = np.float32 if dst_dir.endswith("_f32") else np.uint8 60 | image[0].astype(dtype).tofile(pic_name) 61 | 62 | 63 | def cifar_preprocess(src_file, data_loader, dst_dir, pic_ext, cal_img_num): 64 | for i in range(cal_img_num): 65 | image, label = next(data_loader) 66 | filename = os.path.basename(src_file) 67 | pic_name = os.path.join(dst_dir + '/' + str(i) + pic_ext) 68 | print("write:%s" % pic_name) 69 | image[0].astype(np.uint8).tofile(pic_name) 70 | 71 | 72 | @click.command(help=''' 73 | A Tool used to generate preprocess pics for calibration. 74 | ''') 75 | @click.option('--src_dir', type=str, help='calibration source file') 76 | @click.option('--dst_dir', type=str, help='generated calibration file') 77 | @click.option('--pic_ext', 78 | type=str, 79 | default=".cali", 80 | help='picture extension.') 81 | @click.option('--read_mode', 82 | type=click.Choice(["skimage", "opencv"]), 83 | default="opencv", 84 | help='picture extension.') 85 | @click.option('--cal_img_num', type=int, default=100, help='cali picture num.') 86 | def main(src_dir, dst_dir, pic_ext, read_mode, cal_img_num): 87 | '''A Tool used to generate preprocess pics for calibration.''' 88 | pic_num = 0 89 | os.makedirs(dst_dir, exist_ok=True) 90 | if pic_ext.strip().split('_')[0] in regular_process_list: 91 | print("regular preprocess") 92 | for src_name in sorted(os.listdir(src_dir)): 93 | pic_num += 1 94 | if pic_num > cal_img_num: 95 | break 96 | src_file = os.path.join(src_dir, src_name) 97 | regular_preprocess(src_file, transformers, dst_dir, pic_ext, 98 | read_mode) 99 | elif pic_ext.strip().split('_')[0] == ".cifar": 100 | print("cifar preprocess") 101 | data_loader = DataLoader(CifarDataset(src_dir), transformers, 1) 102 | cifar_preprocess(src_dir, data_loader, dst_dir, pic_ext, cal_img_num) 103 | else: 104 | raise ValueError(f"invalid pic_ext {pic_ext}") 105 | 106 | 107 | if __name__ == '__main__': 108 | main() 109 | -------------------------------------------------------------------------------- /yolov8pose_herizon/hb_mapper_checker.log: -------------------------------------------------------------------------------- 1 | 2023-07-21 17:35:57,336 file: hb_mapper.py func: hb_mapper line No: 70 Start hb_mapper.... 2 | 2023-07-21 17:35:57,337 file: hb_mapper.py func: hb_mapper line No: 71 log will be stored in /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/hb_mapper_checker.log 3 | 2023-07-21 17:35:57,337 file: hb_mapper.py func: hb_mapper line No: 72 hbdk version 3.27.4 4 | 2023-07-21 17:35:57,337 file: hb_mapper.py func: hb_mapper line No: 73 horizon_nn version 0.13.3 5 | 2023-07-21 17:35:57,337 file: hb_mapper.py func: hb_mapper line No: 74 hb_mapper version 1.6.8a 6 | 2023-07-21 17:35:57,337 file: hb_mapper.py func: hb_mapper line No: 76 parameter [output] is deprecated 7 | 2023-07-21 17:35:57,429 file: helper.py func: helper line No: 124 Model input names: ['data'] 8 | 2023-07-21 17:35:57,430 file: hb_mapper_checker.py func: hb_mapper_checker line No: 104 Model type: onnx 9 | 2023-07-21 17:35:57,430 file: hb_mapper_checker.py func: hb_mapper_checker line No: 105 march: bernoulli2 10 | 2023-07-21 17:35:57,430 file: hb_mapper_checker.py func: hb_mapper_checker line No: 110 input names [] 11 | 2023-07-21 17:35:57,430 file: hb_mapper_checker.py func: hb_mapper_checker line No: 111 input shapes {} 12 | 2023-07-21 17:35:57,431 file: hb_mapper_checker.py func: hb_mapper_checker line No: 117 Begin model checking.... 13 | 2023-07-21 17:35:57,431 file: build.py func: build line No: 36 [Fri Jul 21 17:35:57 2023] Start to Horizon NN Model Convert. 14 | 2023-07-21 17:35:57,431 file: dict_parser.py func: dict_parser line No: 28 The input parameter is not specified, convert with default parameters. 15 | 2023-07-21 17:35:57,431 file: dict_parser.py func: dict_parser line No: 513 The hbdk parameter is not specified, and the submodel will be compiled with the default parameter. 16 | 2023-07-21 17:35:57,432 file: build.py func: build line No: 143 HorizonNN version: 0.13.3 17 | 2023-07-21 17:35:57,432 file: build.py func: build line No: 147 HBDK version: 3.27.4 18 | 2023-07-21 17:35:57,432 file: build.py func: build line No: 36 [Fri Jul 21 17:35:57 2023] Start to parse the onnx model. 19 | 2023-07-21 17:35:57,709 file: onnx_parser.py func: onnx_parser line No: 146 ONNX model info: 20 | ONNX IR version: 6 21 | Opset version: 11 22 | Input name: data, [1, 3, 640, 640] 23 | 2023-07-21 17:35:57,905 file: build.py func: build line No: 39 [Fri Jul 21 17:35:57 2023] End to parse the onnx model. 24 | 2023-07-21 17:35:57,906 file: build.py func: build line No: 266 Model input names: ['data'] 25 | 2023-07-21 17:35:57,927 file: build.py func: build line No: 537 Saving the original float model: ./.hb_check/original_float_model.onnx. 26 | 2023-07-21 17:35:57,928 file: build.py func: build line No: 36 [Fri Jul 21 17:35:57 2023] Start to optimize the model. 27 | 2023-07-21 17:35:58,924 file: build.py func: build line No: 39 [Fri Jul 21 17:35:58 2023] End to optimize the model. 28 | 2023-07-21 17:35:58,948 file: build.py func: build line No: 548 Saving the optimized model: ./.hb_check/optimized_float_model.onnx. 29 | 2023-07-21 17:35:58,949 file: build.py func: build line No: 36 [Fri Jul 21 17:35:58 2023] Start to calibrate the model. 30 | 2023-07-21 17:35:59,124 file: calibration_data_set.py func: calibration_data_set line No: 67 There are 1 samples in the calibration data set. 31 | 2023-07-21 17:35:59,126 file: max_calibrater.py func: max_calibrater line No: 68 Run calibration model with max method. 32 | 2023-07-21 17:35:59,580 file: build.py func: build line No: 39 [Fri Jul 21 17:35:59 2023] End to calibrate the model. 33 | 2023-07-21 17:35:59,581 file: build.py func: build line No: 36 [Fri Jul 21 17:35:59 2023] Start to quantize the model. 34 | 2023-07-21 17:36:00,463 file: build.py func: build line No: 39 [Fri Jul 21 17:36:00 2023] End to quantize the model. 35 | 2023-07-21 17:36:00,583 file: build.py func: build line No: 562 Saving the quantized model: ./.hb_check/quantized_model.onnx. 36 | 2023-07-21 17:36:00,583 file: build.py func: build line No: 36 [Fri Jul 21 17:36:00 2023] Start to compile the model with march bernoulli2. 37 | 2023-07-21 17:36:00,583 file: dict_parser.py func: dict_parser line No: 518 Parsing the hbdk parameter:{'hbdk_pass_through_params': '--O0'} 38 | 2023-07-21 17:36:00,879 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_0 39 | 2023-07-21 17:36:01,235 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--O0', '--input-layout', 'NHWC', '--output-layout', 'NCHW'] 40 | 2023-07-21 17:36:02,398 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_1 41 | 2023-07-21 17:36:02,417 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--O0', '--input-layout', 'NHWC', '--output-layout', 'NCHW'] 42 | 2023-07-21 17:36:02,484 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_2 43 | 2023-07-21 17:36:02,502 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--O0', '--input-layout', 'NHWC', '--output-layout', 'NCHW'] 44 | 2023-07-21 17:36:02,546 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_3 45 | 2023-07-21 17:36:02,563 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--O0', '--input-layout', 'NHWC', '--output-layout', 'NCHW'] 46 | 2023-07-21 17:36:02,782 file: build.py func: build line No: 39 [Fri Jul 21 17:36:02 2023] End to compile the model with march bernoulli2. 47 | 2023-07-21 17:36:02,784 file: node_info.py func: node_info line No: 54 The converted model node information: 48 | ============================================================================================ 49 | Node ON Subgraph Type 50 | --------------------------------------------------------------------------------------------- 51 | Conv_0 BPU id(0) HzSQuantizedConv 52 | Conv_2 BPU id(0) HzSQuantizedConv 53 | Conv_4 BPU id(0) HzSQuantizedConv 54 | Split_6 BPU id(0) Split 55 | Conv_7 BPU id(0) HzSQuantizedConv 56 | Conv_9 BPU id(0) HzSQuantizedConv 57 | UNIT_CONV_FOR_Add_11 BPU id(0) HzSQuantizedConv 58 | UNIT_CONV_FOR_153_0.102091550827026_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 59 | UNIT_CONV_FOR_154_0.102091550827026_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 60 | Concat_12 BPU id(0) Concat 61 | Conv_13 BPU id(0) HzSQuantizedConv 62 | Conv_15 BPU id(0) HzSQuantizedConv 63 | Conv_17 BPU id(0) HzSQuantizedConv 64 | Split_19 BPU id(0) Split 65 | Conv_20 BPU id(0) HzSQuantizedConv 66 | Conv_22 BPU id(0) HzSQuantizedConv 67 | UNIT_CONV_FOR_Add_24 BPU id(0) HzSQuantizedConv 68 | Conv_25 BPU id(0) HzSQuantizedConv 69 | Conv_27 BPU id(0) HzSQuantizedConv 70 | UNIT_CONV_FOR_Add_29 BPU id(0) HzSQuantizedConv 71 | UNIT_CONV_FOR_167_0.058226555585861_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 72 | UNIT_CONV_FOR_168_0.058226555585861_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 73 | UNIT_CONV_FOR_173_0.058226555585861_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 74 | Concat_30 BPU id(0) Concat 75 | Conv_31 BPU id(0) HzSQuantizedConv 76 | Conv_33 BPU id(0) HzSQuantizedConv 77 | Conv_35 BPU id(0) HzSQuantizedConv 78 | Split_37 BPU id(0) Split 79 | Conv_38 BPU id(0) HzSQuantizedConv 80 | Conv_40 BPU id(0) HzSQuantizedConv 81 | UNIT_CONV_FOR_Add_42 BPU id(0) HzSQuantizedConv 82 | Conv_43 BPU id(0) HzSQuantizedConv 83 | Conv_45 BPU id(0) HzSQuantizedConv 84 | UNIT_CONV_FOR_Add_47 BPU id(0) HzSQuantizedConv 85 | UNIT_CONV_FOR_186_0.045044675469398_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 86 | UNIT_CONV_FOR_187_0.045044675469398_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 87 | UNIT_CONV_FOR_192_0.045044675469398_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 88 | Concat_48 BPU id(0) Concat 89 | Conv_49 BPU id(0) HzSQuantizedConv 90 | Conv_51 BPU id(0) HzSQuantizedConv 91 | Conv_53 BPU id(0) HzSQuantizedConv 92 | Split_55 BPU id(0) Split 93 | Conv_56 BPU id(0) HzSQuantizedConv 94 | Conv_58 BPU id(0) HzSQuantizedConv 95 | UNIT_CONV_FOR_Add_60 BPU id(0) HzSQuantizedConv 96 | UNIT_CONV_FOR_205_0.044145390391350_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 97 | UNIT_CONV_FOR_206_0.044145390391350_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 98 | Concat_61 BPU id(0) Concat 99 | Conv_62 BPU id(0) HzSQuantizedConv 100 | Conv_64 BPU id(0) HzSQuantizedConv 101 | MaxPool_66 BPU id(0) HzQuantizedMaxPool 102 | MaxPool_67 BPU id(0) HzQuantizedMaxPool 103 | MaxPool_68 BPU id(0) HzQuantizedMaxPool 104 | Concat_69 BPU id(0) Concat 105 | Conv_70 BPU id(0) HzSQuantizedConv 106 | Resize_73 BPU id(0) HzQuantizedResizeUpsample 107 | UNIT_CONV_FOR_200_0.038292985409498_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 108 | Concat_74 BPU id(0) Concat 109 | Conv_75 BPU id(0) HzSQuantizedConv 110 | Split_77 BPU id(0) Split 111 | Conv_78 BPU id(0) HzSQuantizedConv 112 | Conv_80 BPU id(0) HzSQuantizedConv 113 | Concat_82 BPU id(0) Concat 114 | Conv_83 BPU id(0) HzSQuantizedConv 115 | Resize_86 BPU id(0) HzQuantizedResizeUpsample 116 | UNIT_CONV_FOR_181_0.037524033337831_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 117 | Concat_87 BPU id(0) Concat 118 | Conv_88 BPU id(0) HzSQuantizedConv 119 | Split_90 BPU id(0) Split 120 | Conv_91 BPU id(0) HzSQuantizedConv 121 | Conv_93 BPU id(0) HzSQuantizedConv 122 | Concat_95 BPU id(0) Concat 123 | Conv_96 BPU id(0) HzSQuantizedConv 124 | Conv_98 BPU id(0) HzSQuantizedConv 125 | UNIT_CONV_FOR_239_0.044660877436399_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 126 | Concat_100 BPU id(0) Concat 127 | Conv_101 BPU id(0) HzSQuantizedConv 128 | Split_103 BPU id(0) Split 129 | Conv_104 BPU id(0) HzSQuantizedConv 130 | Conv_106 BPU id(0) HzSQuantizedConv 131 | UNIT_CONV_FOR_262_0.037210706621408_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 132 | UNIT_CONV_FOR_263_0.037210706621408_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 133 | Concat_108 BPU id(0) Concat 134 | Conv_109 BPU id(0) HzSQuantizedConv 135 | Conv_111 BPU id(0) HzSQuantizedConv 136 | Concat_113 BPU id(0) Concat 137 | Conv_114 BPU id(0) HzSQuantizedConv 138 | Split_116 BPU id(0) Split 139 | Conv_117 BPU id(0) HzSQuantizedConv 140 | Conv_119 BPU id(0) HzSQuantizedConv 141 | Concat_121 BPU id(0) Concat 142 | Conv_122 BPU id(0) HzSQuantizedConv 143 | Conv_124 BPU id(0) HzSQuantizedConv 144 | Conv_126 BPU id(0) HzSQuantizedConv 145 | Conv_128 BPU id(0) HzSQuantizedConv 146 | Conv_129 BPU id(0) HzSQuantizedConv 147 | Conv_131 BPU id(0) HzSQuantizedConv 148 | Conv_133 BPU id(0) HzSQuantizedConv 149 | Conv_134 BPU id(0) HzSQuantizedConv 150 | Conv_136 BPU id(0) HzSQuantizedConv 151 | Conv_138 BPU id(0) HzSQuantizedConv 152 | Conv_139 BPU id(0) HzSQuantizedConv 153 | Conv_141 BPU id(0) HzSQuantizedConv 154 | Conv_143 BPU id(0) HzSQuantizedConv 155 | Conv_144 BPU id(0) HzSQuantizedConv 156 | Conv_146 BPU id(0) HzSQuantizedConv 157 | Conv_148 BPU id(0) HzSQuantizedConv 158 | Reshape_154 CPU -- Reshape 159 | Transpose_155 CPU -- Transpose 160 | Softmax_156 CPU -- Softmax 161 | Conv_158 BPU id(1) HzSQuantizedConv 162 | Conv_159 BPU id(0) HzSQuantizedConv 163 | Conv_161 BPU id(0) HzSQuantizedConv 164 | Conv_163 BPU id(0) HzSQuantizedConv 165 | Conv_164 BPU id(0) HzSQuantizedConv 166 | Conv_166 BPU id(0) HzSQuantizedConv 167 | Conv_168 BPU id(0) HzSQuantizedConv 168 | Reshape_174 CPU -- Reshape 169 | Transpose_175 CPU -- Transpose 170 | Softmax_176 CPU -- Softmax 171 | Conv_178 BPU id(2) HzSQuantizedConv 172 | Conv_179 BPU id(0) HzSQuantizedConv 173 | Conv_181 BPU id(0) HzSQuantizedConv 174 | Conv_183 BPU id(0) HzSQuantizedConv 175 | Conv_184 BPU id(0) HzSQuantizedConv 176 | Conv_186 BPU id(0) HzSQuantizedConv 177 | Conv_188 BPU id(0) HzSQuantizedConv 178 | Reshape_194 CPU -- Reshape 179 | Transpose_195 CPU -- Transpose 180 | Softmax_196 CPU -- Softmax 181 | Conv_198 BPU id(3) HzSQuantizedConv 182 | 2023-07-21 17:36:02,784 file: build.py func: build line No: 39 [Fri Jul 21 17:36:02 2023] End to Horizon NN Model Convert. 183 | 2023-07-21 17:36:02,790 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2841 ONNX model output num : 9 184 | 2023-07-21 17:36:02,800 file: hb_mapper_checker.py func: hb_mapper_checker line No: 141 End model checking.... 185 | -------------------------------------------------------------------------------- /yolov8pose_herizon/hb_mapper_makertbin.log: -------------------------------------------------------------------------------- 1 | 2023-07-21 17:37:00,289 file: hb_mapper.py func: hb_mapper line No: 111 Start hb_mapper.... 2 | 2023-07-21 17:37:00,290 file: hb_mapper.py func: hb_mapper line No: 112 log will be stored in /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/hb_mapper_makertbin.log 3 | 2023-07-21 17:37:00,290 file: hb_mapper.py func: hb_mapper line No: 113 hbdk version 3.27.4 4 | 2023-07-21 17:37:00,290 file: hb_mapper.py func: hb_mapper line No: 114 horizon_nn version 0.13.3 5 | 2023-07-21 17:37:00,290 file: hb_mapper.py func: hb_mapper line No: 115 hb_mapper version 1.6.8a 6 | 2023-07-21 17:37:00,290 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 590 Start Model Convert.... 7 | 2023-07-21 17:37:00,305 file: mapper_conf_parser.py func: mapper_conf_parser line No: 811 Using abs path /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/model/yolov8pos_relu_zq.onnx 8 | 2023-07-21 17:37:00,306 file: mapper_conf_parser.py func: mapper_conf_parser line No: 137 validating model_parameters... 9 | 2023-07-21 17:37:00,317 file: mapper_conf_parser.py func: mapper_conf_parser line No: 811 Using abs path /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/model_output 10 | 2023-07-21 17:37:00,317 file: mapper_conf_parser.py func: mapper_conf_parser line No: 149 validating model_parameters finished 11 | 2023-07-21 17:37:00,317 file: mapper_conf_parser.py func: mapper_conf_parser line No: 153 validating input_parameters... 12 | 2023-07-21 17:37:00,317 file: helper.py func: helper line No: 124 Model input names: ['data'] 13 | 2023-07-21 17:37:00,317 file: mapper_conf_parser.py func: mapper_conf_parser line No: 273 input num is set to 1 according to input_names 14 | 2023-07-21 17:37:00,317 file: mapper_conf_parser.py func: mapper_conf_parser line No: 279 model name missing, using model name from model file: ['data'] 15 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 325 model input shape missing, using shape from model file: [[1, 3, 640, 640]] 16 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 164 validating input_parameters finished 17 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 168 validating calibration_parameters... 18 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 811 Using abs path /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/cal_data 19 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 182 validating calibration_parameters finished 20 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 186 validating custom_op... 21 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 724 custom_op does not exist, skipped 22 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 192 validating custom_op finished 23 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 195 validating compiler_parameters... 24 | 2023-07-21 17:37:00,318 file: mapper_conf_parser.py func: mapper_conf_parser line No: 201 validating compiler_parameters finished 25 | 2023-07-21 17:37:00,319 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 56 Dump config: 26 | 2023-07-21 17:37:00,319 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 57 calibration_parameters: 27 | cal_data_dir: ./cal_data 28 | calibration_type: default 29 | preprocess_on: true 30 | compiler_parameters: 31 | compile_mode: latency 32 | debug: false 33 | optimize_level: O3 34 | input_parameters: 35 | input_layout_rt: NCHW 36 | input_layout_train: NCHW 37 | input_name: '' 38 | input_shape: '' 39 | input_type_rt: rgb 40 | input_type_train: rgb 41 | mean_value: '' 42 | norm_type: data_scale 43 | scale_value: '0.003921568627451' 44 | model_parameters: 45 | layer_out_dump: false 46 | log_level: debug 47 | march: bernoulli2 48 | onnx_model: /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/model/yolov8pos_relu_zq.onnx 49 | output_model_file_prefix: yolov8seg 50 | working_dir: model_output 51 | 52 | 2023-07-21 17:37:00,322 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 63 input 'data' : original model shape: [1, 3, 640, 640] 53 | 2023-07-21 17:37:00,322 file: loader.py func: loader line No: 44 ******************************************* 54 | 2023-07-21 17:37:00,322 file: loader.py func: loader line No: 45 First calibration picture name: test.jpg 55 | 2023-07-21 17:37:00,322 file: loader.py func: loader line No: 47 First calibration picture md5: 56 | 2023-07-21 17:37:00,340 file: loader.py func: loader line No: 51 ******************************************* 57 | 2023-07-21 17:37:00,504 file: tool_utils.py func: tool_utils line No: 320 calibration data shape: (1, 3, 640, 640) 58 | 2023-07-21 17:37:00,506 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 579 call build params: 59 | {'march': 'bernoulli2', 'debug_mode': False, 'save_model': True, 'name_prefix': 'yolov8seg', 'input_dict': {'data': {'input_shape': [1, 3, 640, 640], 'expected_input_type': 'RGB_128', 'original_input_type': 'RGB', 'original_input_layout': 'NCHW', 'scales': array([0.00392157], dtype=float32)}}, 'cali_dict': {'calibration_type': 'default', 'calibration_data': {'data': array([[[[75., 76., 75., ..., 91., 91., 91.], 60 | [75., 75., 75., ..., 90., 90., 90.], 61 | [76., 76., 76., ..., 90., 90., 90.], 62 | ..., 63 | [93., 91., 86., ..., 63., 64., 66.], 64 | [89., 87., 90., ..., 60., 60., 61.], 65 | [90., 83., 94., ..., 60., 60., 63.]], 66 | 67 | [[78., 79., 78., ..., 91., 91., 91.], 68 | [78., 78., 78., ..., 90., 90., 90.], 69 | [79., 79., 79., ..., 90., 90., 90.], 70 | ..., 71 | [93., 92., 87., ..., 66., 67., 69.], 72 | [89., 88., 91., ..., 63., 63., 64.], 73 | [90., 83., 95., ..., 63., 63., 66.]], 74 | 75 | [[85., 86., 85., ..., 93., 93., 93.], 76 | [85., 85., 85., ..., 92., 92., 92.], 77 | [86., 86., 86., ..., 92., 92., 92.], 78 | ..., 79 | [95., 94., 89., ..., 73., 72., 74.], 80 | [91., 90., 93., ..., 70., 68., 69.], 81 | [92., 85., 97., ..., 70., 70., 71.]]]], dtype=float32)}}, 'hbdk_dict': {'hbdk_pass_through_params': '--fast --O3', 'input-source': {'data': 'ddr', '_default_value': 'ddr'}}, 'node_dict': {}} 82 | 2023-07-21 17:37:00,517 file: build.py func: build line No: 36 [Fri Jul 21 17:37:00 2023] Start to Horizon NN Model Convert. 83 | 2023-07-21 17:37:00,517 file: dict_parser.py func: dict_parser line No: 32 Parsing the input parameter:{'data': {'input_shape': [1, 3, 640, 640], 'expected_input_type': 'RGB_128', 'original_input_type': 'RGB', 'original_input_layout': 'NCHW', 'scales': array([0.00392157], dtype=float32)}} 84 | 2023-07-21 17:37:00,517 file: build.py func: build line No: 237 Parsing the calibration parameter 85 | 2023-07-21 17:37:00,517 file: dict_parser.py func: dict_parser line No: 518 Parsing the hbdk parameter:{'hbdk_pass_through_params': '--fast --O3', 'input-source': {'data': 'ddr', '_default_value': 'ddr'}} 86 | 2023-07-21 17:37:00,518 file: build.py func: build line No: 143 HorizonNN version: 0.13.3 87 | 2023-07-21 17:37:00,518 file: build.py func: build line No: 147 HBDK version: 3.27.4 88 | 2023-07-21 17:37:00,518 file: build.py func: build line No: 36 [Fri Jul 21 17:37:00 2023] Start to parse the onnx model. 89 | 2023-07-21 17:37:00,559 file: onnx_parser.py func: onnx_parser line No: 146 ONNX model info: 90 | ONNX IR version: 6 91 | Opset version: 11 92 | Input name: data, [1, 3, 640, 640] 93 | 2023-07-21 17:37:00,629 file: build.py func: build line No: 39 [Fri Jul 21 17:37:00 2023] End to parse the onnx model. 94 | 2023-07-21 17:37:00,630 file: build.py func: build line No: 266 Model input names: ['data'] 95 | 2023-07-21 17:37:00,630 file: dict_parser.py func: dict_parser line No: 288 Create a preprocessing operator for input_name data with means=None, std=[254.99998492], original_input_layout=NCHW, color convert from 'RGB' to 'RGB'. 96 | 2023-07-21 17:37:00,674 file: build.py func: build line No: 537 Saving the original float model: yolov8seg_original_float_model.onnx. 97 | 2023-07-21 17:37:00,674 file: build.py func: build line No: 36 [Fri Jul 21 17:37:00 2023] Start to optimize the model. 98 | 2023-07-21 17:37:00,858 file: build.py func: build line No: 39 [Fri Jul 21 17:37:00 2023] End to optimize the model. 99 | 2023-07-21 17:37:00,868 file: build.py func: build line No: 548 Saving the optimized model: yolov8seg_optimized_float_model.onnx. 100 | 2023-07-21 17:37:00,868 file: build.py func: build line No: 36 [Fri Jul 21 17:37:00 2023] Start to calibrate the model. 101 | 2023-07-21 17:37:01,008 file: calibration_data_set.py func: calibration_data_set line No: 67 There are 1 samples in the calibration data set. 102 | 2023-07-21 17:37:01,011 file: default_calibrater.py func: default_calibrater line No: 145 Run calibration model with default calibration method. 103 | 2023-07-21 17:37:15,405 file: default_calibrater.py func: default_calibrater line No: 214 Select max-percentile:percentile=0.99995 method. 104 | 2023-07-21 17:37:15,427 file: build.py func: build line No: 39 [Fri Jul 21 17:37:15 2023] End to calibrate the model. 105 | 2023-07-21 17:37:15,428 file: build.py func: build line No: 36 [Fri Jul 21 17:37:15 2023] Start to quantize the model. 106 | 2023-07-21 17:37:18,840 file: build.py func: build line No: 39 [Fri Jul 21 17:37:18 2023] End to quantize the model. 107 | 2023-07-21 17:37:18,940 file: build.py func: build line No: 562 Saving the quantized model: yolov8seg_quantized_model.onnx. 108 | 2023-07-21 17:37:18,940 file: build.py func: build line No: 36 [Fri Jul 21 17:37:18 2023] Start to compile the model with march bernoulli2. 109 | 2023-07-21 17:37:19,083 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_0 110 | 2023-07-21 17:37:19,284 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--fast', '--O3', '--input-layout', 'NHWC', '--output-layout', 'NCHW', '--input-source', 'ddr'] 111 | 2023-07-21 17:37:45,104 file: tool_utils.py func: tool_utils line No: 293 consumed time 25.7879 112 | 2023-07-21 17:37:45,399 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_1 113 | 2023-07-21 17:37:45,417 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--fast', '--O3', '--input-layout', 'NHWC', '--output-layout', 'NCHW', '--input-source', 'ddr'] 114 | 2023-07-21 17:37:45,634 file: tool_utils.py func: tool_utils line No: 293 consumed time 0.184087 115 | 2023-07-21 17:37:45,679 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_2 116 | 2023-07-21 17:37:45,697 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--fast', '--O3', '--input-layout', 'NHWC', '--output-layout', 'NCHW', '--input-source', 'ddr'] 117 | 2023-07-21 17:37:45,777 file: tool_utils.py func: tool_utils line No: 293 consumed time 0.0433857 118 | 2023-07-21 17:37:45,820 file: hybrid_build.py func: hybrid_build line No: 123 Compile submodel: torch-jit-export_subgraph_3 119 | 2023-07-21 17:37:45,839 file: hbdk_cc.py func: hbdk_cc line No: 119 hbdk-cc parameters:['--fast', '--O3', '--input-layout', 'NHWC', '--output-layout', 'NCHW', '--input-source', 'ddr'] 120 | 2023-07-21 17:37:45,891 file: tool_utils.py func: tool_utils line No: 293 consumed time 0.0185339 121 | 2023-07-21 17:37:46,133 file: build.py func: build line No: 39 [Fri Jul 21 17:37:46 2023] End to compile the model with march bernoulli2. 122 | 2023-07-21 17:37:46,137 file: node_info.py func: node_info line No: 54 The converted model node information: 123 | ============================================================================================================================================ 124 | Node ON Subgraph Type Cosine Similarity Threshold 125 | -------------------------------------------------------------------------------------------------------------------------------------------- 126 | HZ_PREPROCESS_FOR_data BPU id(0) HzSQuantizedPreprocess 1.000029 127.000000 127 | Conv_0 BPU id(0) HzSQuantizedConv 0.999547 1.000000 128 | 148_pad BPU id(0) HzPad 129 | Conv_2 BPU id(0) HzSQuantizedConv 0.998810 6.224448 130 | Conv_4 BPU id(0) HzSQuantizedConv 0.998677 8.146382 131 | Split_6 BPU id(0) Split 0.998597 15.298190 132 | Conv_7 BPU id(0) HzSQuantizedConv 0.990820 15.298190 133 | 156_pad BPU id(0) HzPad 134 | Conv_9 BPU id(0) HzSQuantizedConv 0.994607 5.086967 135 | UNIT_CONV_FOR_Add_11 BPU id(0) HzSQuantizedConv 0.997572 15.298190 136 | UNIT_CONV_FOR_153_0.111463353037834_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 137 | UNIT_CONV_FOR_154_0.111463353037834_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 138 | Concat_12 BPU id(0) Concat 0.997805 15.298190 139 | Conv_13 BPU id(0) HzSQuantizedConv 0.995284 14.155846 140 | 162_pad BPU id(0) HzPad 141 | Conv_15 BPU id(0) HzSQuantizedConv 0.992841 4.297529 142 | Conv_17 BPU id(0) HzSQuantizedConv 0.992801 3.949197 143 | Split_19 BPU id(0) Split 0.990334 6.935760 144 | Conv_20 BPU id(0) HzSQuantizedConv 0.988006 6.935760 145 | 170_pad BPU id(0) HzPad 146 | Conv_22 BPU id(0) HzSQuantizedConv 0.990882 3.144512 147 | UNIT_CONV_FOR_Add_24 BPU id(0) HzSQuantizedConv 0.995302 6.935760 148 | Conv_25 BPU id(0) HzSQuantizedConv 0.985636 7.577655 149 | 175_pad BPU id(0) HzPad 150 | Conv_27 BPU id(0) HzSQuantizedConv 0.983965 2.642028 151 | UNIT_CONV_FOR_Add_29 BPU id(0) HzSQuantizedConv 0.993850 7.577655 152 | UNIT_CONV_FOR_167_0.076023690402508_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 153 | UNIT_CONV_FOR_168_0.076023690402508_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 154 | UNIT_CONV_FOR_173_0.076023690402508_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 155 | Concat_30 BPU id(0) Concat 0.993944 6.935760 156 | Conv_31 BPU id(0) HzSQuantizedConv 0.985220 9.655008 157 | Conv_33 BPU id(0) HzSQuantizedConv 0.989865 6.068075 158 | Conv_35 BPU id(0) HzSQuantizedConv 0.989802 2.571192 159 | Split_37 BPU id(0) Split 0.988267 4.649628 160 | Conv_38 BPU id(0) HzSQuantizedConv 0.991324 4.649628 161 | 189_pad BPU id(0) HzPad 162 | Conv_40 BPU id(0) HzSQuantizedConv 0.991065 2.246605 163 | UNIT_CONV_FOR_Add_42 BPU id(0) HzSQuantizedConv 0.993074 4.649628 164 | Conv_43 BPU id(0) HzSQuantizedConv 0.987420 6.525288 165 | 194_pad BPU id(0) HzPad 166 | Conv_45 BPU id(0) HzSQuantizedConv 0.988220 2.016948 167 | UNIT_CONV_FOR_Add_47 BPU id(0) HzSQuantizedConv 0.993165 6.525288 168 | UNIT_CONV_FOR_186_0.062861151993275_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 169 | UNIT_CONV_FOR_187_0.062861151993275_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 170 | UNIT_CONV_FOR_192_0.062861151993275_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 171 | Concat_48 BPU id(0) Concat 0.992398 4.649628 172 | Conv_49 BPU id(0) HzSQuantizedConv 0.987996 7.983366 173 | Conv_51 BPU id(0) HzSQuantizedConv 0.992062 5.696781 174 | Conv_53 BPU id(0) HzSQuantizedConv 0.990632 2.428639 175 | Split_55 BPU id(0) Split 0.990048 4.060051 176 | Conv_56 BPU id(0) HzSQuantizedConv 0.992909 4.060051 177 | 208_pad BPU id(0) HzPad 178 | Conv_58 BPU id(0) HzSQuantizedConv 0.994361 2.120098 179 | UNIT_CONV_FOR_Add_60 BPU id(0) HzSQuantizedConv 0.994972 4.060051 180 | UNIT_CONV_FOR_205_0.046630416065454_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 181 | UNIT_CONV_FOR_206_0.046630416065454_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 182 | Concat_61 BPU id(0) Concat 0.993569 4.060051 183 | Conv_62 BPU id(0) HzSQuantizedConv 0.992342 5.922063 184 | Conv_64 BPU id(0) HzSQuantizedConv 0.992678 2.105112 185 | MaxPool_66 BPU id(0) HzQuantizedMaxPool 0.997603 5.160666 186 | MaxPool_67 BPU id(0) HzQuantizedMaxPool 0.998712 5.160666 187 | MaxPool_68 BPU id(0) HzQuantizedMaxPool 0.999146 5.160666 188 | Concat_69 BPU id(0) Concat 0.998449 5.160666 189 | Conv_70 BPU id(0) HzSQuantizedConv 0.992550 5.160666 190 | Resize_73 BPU id(0) HzQuantizedResizeUpsample 0.992551 4.875719 191 | UNIT_CONV_FOR_200_0.038391489535570_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 192 | Concat_74 BPU id(0) Concat 0.990941 4.875719 193 | Conv_75 BPU id(0) HzSQuantizedConv 0.988078 4.875719 194 | Split_77 BPU id(0) Split 0.988382 4.869437 195 | Conv_78 BPU id(0) HzSQuantizedConv 0.985914 4.869437 196 | 234_pad BPU id(0) HzPad 197 | Conv_80 BPU id(0) HzSQuantizedConv 0.985413 2.005218 198 | UNIT_CONV_FOR_231_0.037235010415316_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 199 | UNIT_CONV_FOR_232_0.037235010415316_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 200 | Concat_82 BPU id(0) Concat 0.986985 4.869437 201 | Conv_83 BPU id(0) HzSQuantizedConv 0.983665 4.728847 202 | Resize_86 BPU id(0) HzQuantizedResizeUpsample 0.983668 5.474870 203 | UNIT_CONV_FOR_181_0.043109212070704_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 204 | Concat_87 BPU id(0) Concat 0.984172 5.474870 205 | Conv_88 BPU id(0) HzSQuantizedConv 0.983652 5.474870 206 | Split_90 BPU id(0) Split 0.982433 4.726348 207 | Conv_91 BPU id(0) HzSQuantizedConv 0.973238 4.726348 208 | 251_pad BPU id(0) HzPad 209 | Conv_93 BPU id(0) HzSQuantizedConv 0.970932 2.259508 210 | UNIT_CONV_FOR_248_0.047050915658474_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 211 | UNIT_CONV_FOR_249_0.047050915658474_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 212 | Concat_95 BPU id(0) Concat 0.978685 4.726348 213 | Conv_96 BPU id(0) HzSQuantizedConv 0.968553 5.975466 214 | 256_pad BPU id(0) HzPad 215 | Conv_98 BPU id(0) HzSQuantizedConv 0.966500 2.621284 216 | UNIT_CONV_FOR_239_0.041822936385870_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 217 | Concat_100 BPU id(0) Concat 0.977314 5.311513 218 | Conv_101 BPU id(0) HzSQuantizedConv 0.972657 5.311513 219 | Split_103 BPU id(0) Split 0.970772 4.329791 220 | Conv_104 BPU id(0) HzSQuantizedConv 0.970739 4.329791 221 | 265_pad BPU id(0) HzPad 222 | Conv_106 BPU id(0) HzSQuantizedConv 0.970102 2.280810 223 | UNIT_CONV_FOR_262_0.044201314449310_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 224 | UNIT_CONV_FOR_263_0.044201314449310_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 225 | Concat_108 BPU id(0) Concat 0.971342 4.329791 226 | Conv_109 BPU id(0) HzSQuantizedConv 0.963690 5.613567 227 | 270_pad BPU id(0) HzPad 228 | Conv_111 BPU id(0) HzSQuantizedConv 0.974700 2.451806 229 | UNIT_CONV_FOR_222_0.039076458662748_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 230 | Concat_113 BPU id(0) Concat 0.987153 4.962710 231 | Conv_114 BPU id(0) HzSQuantizedConv 0.986313 4.962710 232 | Split_116 BPU id(0) Split 0.985562 5.044684 233 | Conv_117 BPU id(0) HzSQuantizedConv 0.987110 5.044684 234 | 279_pad BPU id(0) HzPad 235 | Conv_119 BPU id(0) HzSQuantizedConv 0.987470 2.502454 236 | UNIT_CONV_FOR_276_0.044283170253038_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 237 | UNIT_CONV_FOR_277_0.044283170253038_TO_FUSE_SCALE BPU id(0) HzSQuantizedConv 238 | Concat_121 BPU id(0) Concat 0.986773 5.044684 239 | Conv_122 BPU id(0) HzSQuantizedConv 0.984443 5.623962 240 | 256_pad_1 BPU id(0) HzPad 241 | Conv_124 BPU id(0) HzSQuantizedConv 0.967918 2.621284 242 | 286_pad BPU id(0) HzPad 243 | Conv_126 BPU id(0) HzSQuantizedConv 0.959518 2.141563 244 | Conv_128 BPU id(0) HzSQuantizedConv 0.976597 2.586977 245 | 270_pad_1 BPU id(0) HzPad 246 | Conv_129 BPU id(0) HzSQuantizedConv 0.970885 2.451806 247 | 291_pad BPU id(0) HzPad 248 | Conv_131 BPU id(0) HzSQuantizedConv 0.967315 1.985868 249 | Conv_133 BPU id(0) HzSQuantizedConv 0.975665 2.140175 250 | 284_pad BPU id(0) HzPad 251 | Conv_134 BPU id(0) HzSQuantizedConv 0.985670 2.448336 252 | 296_pad BPU id(0) HzPad 253 | Conv_136 BPU id(0) HzSQuantizedConv 0.985284 2.325466 254 | Conv_138 BPU id(0) HzSQuantizedConv 0.992255 3.629121 255 | 256_pad_2 BPU id(0) HzPad 256 | Conv_139 BPU id(0) HzSQuantizedConv 0.966122 2.621284 257 | 301_pad BPU id(0) HzPad 258 | Conv_141 BPU id(0) HzSQuantizedConv 0.971887 2.396647 259 | Conv_143 BPU id(0) HzSQuantizedConv 0.985289 4.596310 260 | 256_pad_3 BPU id(0) HzPad 261 | Conv_144 BPU id(0) HzSQuantizedConv 0.966138 2.621284 262 | 306_pad BPU id(0) HzPad 263 | Conv_146 BPU id(0) HzSQuantizedConv 0.964980 2.323879 264 | Conv_148 BPU id(0) HzSQuantizedConv 0.999622 2.175680 265 | Reshape_154 CPU -- Reshape 0.985289 -- 266 | Transpose_155 CPU -- Transpose 0.985285 -- 267 | Softmax_156 CPU -- Softmax 0.964429 -- 268 | Conv_158 BPU id(1) HzSQuantizedConv 0.988179 0.863565 269 | 270_pad_2 BPU id(0) HzPad 270 | Conv_159 BPU id(0) HzSQuantizedConv 0.966308 2.451806 271 | 327_pad BPU id(0) HzPad 272 | Conv_161 BPU id(0) HzSQuantizedConv 0.973635 2.449803 273 | Conv_163 BPU id(0) HzSQuantizedConv 0.976529 4.849308 274 | 270_pad_3 BPU id(0) HzPad 275 | Conv_164 BPU id(0) HzSQuantizedConv 0.964083 2.451806 276 | 332_pad BPU id(0) HzPad 277 | Conv_166 BPU id(0) HzSQuantizedConv 0.962786 2.232537 278 | Conv_168 BPU id(0) HzSQuantizedConv 0.999565 2.256133 279 | Reshape_174 CPU -- Reshape 0.976529 -- 280 | Transpose_175 CPU -- Transpose 0.976537 -- 281 | Softmax_176 CPU -- Softmax 0.951247 -- 282 | Conv_178 BPU id(2) HzSQuantizedConv 0.985842 0.893731 283 | 284_pad_1 BPU id(0) HzPad 284 | Conv_179 BPU id(0) HzSQuantizedConv 0.986837 2.448336 285 | 353_pad BPU id(0) HzPad 286 | Conv_181 BPU id(0) HzSQuantizedConv 0.992849 2.617640 287 | Conv_183 BPU id(0) HzSQuantizedConv 0.993666 6.437304 288 | 284_pad_2 BPU id(0) HzPad 289 | Conv_184 BPU id(0) HzSQuantizedConv 0.987403 2.448336 290 | 358_pad BPU id(0) HzPad 291 | Conv_186 BPU id(0) HzSQuantizedConv 0.989589 2.514280 292 | Conv_188 BPU id(0) HzSQuantizedConv 0.999736 3.099249 293 | Reshape_194 CPU -- Reshape 0.993666 -- 294 | Transpose_195 CPU -- Transpose 0.993663 -- 295 | Softmax_196 CPU -- Softmax 0.986020 -- 296 | Conv_198 BPU id(3) HzSQuantizedConv 0.994720 0.875396 297 | 2023-07-21 17:37:46,140 file: build.py func: build line No: 621 The quantify model output: 298 | =========================================================================== 299 | Node Cosine Similarity L1 Distance L2 Distance Chebyshev Distance 300 | --------------------------------------------------------------------------- 301 | Conv_128 0.976597 0.037352 0.000110 0.828023 302 | Conv_133 0.975665 0.036191 0.000227 0.916981 303 | Conv_138 0.992255 0.045201 0.000504 0.700299 304 | Conv_148 0.999622 0.160476 0.003057 1.701255 305 | Conv_158 0.988179 0.408214 0.004347 8.627660 306 | Conv_168 0.999565 0.146725 0.005521 1.458210 307 | Conv_178 0.985842 0.460769 0.011443 7.091931 308 | Conv_188 0.999736 0.107689 0.007131 0.595237 309 | Conv_198 0.994720 0.254044 0.014422 5.312317 310 | 2023-07-21 17:37:46,140 file: build.py func: build line No: 39 [Fri Jul 21 17:37:46 2023] End to Horizon NN Model Convert. 311 | 2023-07-21 17:37:46,206 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 641 start convert to *.bin file.... 312 | 2023-07-21 17:37:46,251 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2841 ONNX model output num : 9 313 | 2023-07-21 17:37:46,252 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2717 ############# model deps info ############# 314 | 2023-07-21 17:37:46,252 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2719 hb_mapper version : 1.6.8a 315 | 2023-07-21 17:37:46,252 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2722 hbdk version : 3.27.4 316 | 2023-07-21 17:37:46,252 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2724 hbdk runtime version: 3.13.27 317 | 2023-07-21 17:37:46,252 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2727 horizon_nn version : 0.13.3 318 | 2023-07-21 17:37:46,252 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2730 ############# model_parameters info ############# 319 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2736 onnx_model : /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/model/yolov8pos_relu_zq.onnx 320 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2737 BPU march : bernoulli2 321 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2738 layer_out_dump : False 322 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2739 log_level : DEBUG 323 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2740 working dir : /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/model_output 324 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2741 output_model_file_prefix: yolov8seg 325 | 2023-07-21 17:37:46,253 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2748 ############# input_parameters info ############# 326 | 2023-07-21 17:37:46,254 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2765 ------------------------------------------ 327 | 2023-07-21 17:37:46,254 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2767 ---------input info : data --------- 328 | 2023-07-21 17:37:46,254 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2768 input_name : data 329 | 2023-07-21 17:37:46,254 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2769 input_type_rt : rgb 330 | 2023-07-21 17:37:46,254 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2772 input_space&range : regular 331 | 2023-07-21 17:37:46,254 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2773 input_layout_rt : NCHW 332 | 2023-07-21 17:37:46,255 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2774 input_type_train : rgb 333 | 2023-07-21 17:37:46,255 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2775 input_layout_train : NCHW 334 | 2023-07-21 17:37:46,255 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2776 norm_type : data_scale 335 | 2023-07-21 17:37:46,255 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2777 input_shape : 1x3x640x640 336 | 2023-07-21 17:37:46,255 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2783 scale_value : 0.003921568627451, 337 | 2023-07-21 17:37:46,256 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2785 cal_data_dir : /convert_model/horizon_model_convert_sample/04_detection/03_yolov8pose_zq/mapper/cal_data 338 | 2023-07-21 17:37:46,256 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2786 ---------input info : data end ------- 339 | 2023-07-21 17:37:46,256 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2787 ------------------------------------------ 340 | 2023-07-21 17:37:46,256 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2789 ############# calibration_parameters info ############# 341 | 2023-07-21 17:37:46,256 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2790 preprocess_on : True 342 | 2023-07-21 17:37:46,256 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2791 calibration_type: : default 343 | 2023-07-21 17:37:46,257 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2815 ############# compiler_parameters info ############# 344 | 2023-07-21 17:37:46,257 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2819 hbdk_pass_through_params: --fast --O3 345 | 2023-07-21 17:37:46,257 file: onnx2horizonrt.py func: onnx2horizonrt line No: 2819 input-source : {'data': 'ddr', '_default_value': 'ddr'} 346 | 2023-07-21 17:37:46,263 file: layout_util.py func: layout_util line No: 13 set_featuremap_layout start 347 | 2023-07-21 17:37:46,268 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 768 Convert to runtime bin file sucessfully! 348 | 2023-07-21 17:37:46,269 file: hb_mapper_makertbin.py func: hb_mapper_makertbin line No: 769 End Model Convert 349 | -------------------------------------------------------------------------------- /yolov8pose_herizon/inference_image_demo.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from horizon_tc_ui import HB_ONNXRuntime 3 | from horizon_tc_ui.utils.tool_utils import init_root_logger 4 | from math import exp 5 | import cv2 6 | import numpy as np 7 | 8 | 9 | color_list = [(0, 0, 255), (0, 255, 0), (255, 0, 0)] 10 | skeleton = [[16, 14], [14, 12], [17, 15], [15, 13], [12, 13], [6, 12], [7, 13], [6, 7], [6, 8], [7, 9], [8, 10], [9, 11], 11 | [2, 3], [1, 2], [1, 3], [2, 4], [3, 5], [4, 6], [5, 7]] 12 | 13 | CLASSES = ['person'] 14 | 15 | meshgrid = [] 16 | 17 | class_num = len(CLASSES) 18 | headNum = 3 19 | keypoint_num = 17 20 | 21 | strides = [8, 16, 32] 22 | mapSize = [[80, 80], [40, 40], [20, 20]] 23 | nmsThresh = 0.55 24 | objectThresh = 0.5 25 | 26 | input_imgH = 640 27 | input_imgW = 640 28 | 29 | 30 | class DetectBox: 31 | def __init__(self, classId, score, xmin, ymin, xmax, ymax, pose): 32 | self.classId = classId 33 | self.score = score 34 | self.xmin = xmin 35 | self.ymin = ymin 36 | self.xmax = xmax 37 | self.ymax = ymax 38 | self.pose = pose 39 | 40 | 41 | def GenerateMeshgrid(): 42 | for index in range(headNum): 43 | for i in range(mapSize[index][0]): 44 | for j in range(mapSize[index][1]): 45 | meshgrid.append(j + 0.5) 46 | meshgrid.append(i + 0.5) 47 | 48 | 49 | def IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2): 50 | xmin = max(xmin1, xmin2) 51 | ymin = max(ymin1, ymin2) 52 | xmax = min(xmax1, xmax2) 53 | ymax = min(ymax1, ymax2) 54 | 55 | innerWidth = xmax - xmin 56 | innerHeight = ymax - ymin 57 | 58 | innerWidth = innerWidth if innerWidth > 0 else 0 59 | innerHeight = innerHeight if innerHeight > 0 else 0 60 | 61 | innerArea = innerWidth * innerHeight 62 | 63 | area1 = (xmax1 - xmin1) * (ymax1 - ymin1) 64 | area2 = (xmax2 - xmin2) * (ymax2 - ymin2) 65 | 66 | total = area1 + area2 - innerArea 67 | 68 | return innerArea / total 69 | 70 | 71 | def NMS(detectResult): 72 | predBoxs = [] 73 | 74 | sort_detectboxs = sorted(detectResult, key=lambda x: x.score, reverse=True) 75 | 76 | for i in range(len(sort_detectboxs)): 77 | xmin1 = sort_detectboxs[i].xmin 78 | ymin1 = sort_detectboxs[i].ymin 79 | xmax1 = sort_detectboxs[i].xmax 80 | ymax1 = sort_detectboxs[i].ymax 81 | classId = sort_detectboxs[i].classId 82 | 83 | if sort_detectboxs[i].classId != -1: 84 | predBoxs.append(sort_detectboxs[i]) 85 | for j in range(i + 1, len(sort_detectboxs), 1): 86 | if classId == sort_detectboxs[j].classId: 87 | xmin2 = sort_detectboxs[j].xmin 88 | ymin2 = sort_detectboxs[j].ymin 89 | xmax2 = sort_detectboxs[j].xmax 90 | ymax2 = sort_detectboxs[j].ymax 91 | iou = IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2) 92 | if iou > nmsThresh: 93 | sort_detectboxs[j].classId = -1 94 | return predBoxs 95 | 96 | 97 | def sigmoid(x): 98 | return 1 / (1 + exp(-x)) 99 | 100 | 101 | def postprocess(out, img_h, img_w): 102 | print('postprocess ... ') 103 | 104 | detectResult = [] 105 | 106 | output = [] 107 | for i in range(len(out)): 108 | output.append(out[i].reshape((-1))) 109 | 110 | scale_h = img_h / input_imgH 111 | scale_w = img_w / input_imgW 112 | 113 | gridIndex = -2 114 | 115 | for index in range(headNum): 116 | reg = output[index * 2 + 0] 117 | cls = output[index * 2 + 1] 118 | pose = output[headNum * 2 + index] 119 | 120 | for h in range(mapSize[index][0]): 121 | for w in range(mapSize[index][1]): 122 | gridIndex += 2 123 | 124 | for cl in range(class_num): 125 | cls_val = sigmoid(cls[cl * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 126 | 127 | if cls_val > objectThresh: 128 | x1 = (meshgrid[gridIndex + 0] - reg[0 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 129 | y1 = (meshgrid[gridIndex + 1] - reg[1 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 130 | x2 = (meshgrid[gridIndex + 0] + reg[2 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 131 | y2 = (meshgrid[gridIndex + 1] + reg[3 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 132 | 133 | xmin = x1 * scale_w 134 | ymin = y1 * scale_h 135 | xmax = x2 * scale_w 136 | ymax = y2 * scale_h 137 | 138 | xmin = xmin if xmin > 0 else 0 139 | ymin = ymin if ymin > 0 else 0 140 | xmax = xmax if xmax < img_w else img_w 141 | ymax = ymax if ymax < img_h else img_h 142 | 143 | poseResult = [] 144 | for kc in range(keypoint_num): 145 | px = pose[(kc * 3 + 0) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 146 | py = pose[(kc * 3 + 1) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 147 | vs = sigmoid(pose[(kc * 3 + 2) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 148 | 149 | x = (px * 2.0 + (meshgrid[gridIndex + 0] - 0.5)) * strides[index] * scale_w 150 | y = (py * 2.0 + (meshgrid[gridIndex + 1] - 0.5)) * strides[index] * scale_h 151 | 152 | poseResult.append(vs) 153 | poseResult.append(x) 154 | poseResult.append(y) 155 | # print(poseResult) 156 | box = DetectBox(cl, cls_val, xmin, ymin, xmax, ymax, poseResult) 157 | detectResult.append(box) 158 | 159 | # NMS 160 | print('detectResult:', len(detectResult)) 161 | predBox = NMS(detectResult) 162 | 163 | return predBox 164 | 165 | 166 | def preprocess(src_image): 167 | src_image = cv2.cvtColor(src_image, cv2.COLOR_BGR2RGB) 168 | img = cv2.resize(src_image, (input_imgW, input_imgH)) 169 | return img 170 | 171 | 172 | def inference(model_path, image_path, input_layout, input_offset): 173 | # init_root_logger("inference.log", console_level=logging.INFO, file_level=logging.DEBUG) 174 | 175 | sess = HB_ONNXRuntime(model_file=model_path) 176 | sess.set_dim_param(0, 0, '?') 177 | 178 | if input_layout is None: 179 | logging.warning(f"input_layout not provided. Using {sess.layout[0]}") 180 | input_layout = sess.layout[0] 181 | 182 | origimg = cv2.imread(image_path) 183 | img_h, img_w = origimg.shape[:2] 184 | image_data = preprocess(origimg) 185 | 186 | # image_data = image_data.transpose((2, 0, 1)) 187 | image_data = np.expand_dims(image_data, axis=0) 188 | 189 | input_name = sess.input_names[0] 190 | output_name = sess.output_names 191 | output = sess.run(output_name, {input_name: image_data}, input_offset=input_offset) 192 | 193 | print('inference finished, output len is:', len(output)) 194 | 195 | out = [] 196 | for i in range(len(output)): 197 | out.append(output[i]) 198 | predbox = postprocess(out, img_h, img_w) 199 | 200 | print('obj num is :', len(predbox)) 201 | 202 | for i in range(len(predbox)): 203 | xmin = int(predbox[i].xmin) 204 | ymin = int(predbox[i].ymin) 205 | xmax = int(predbox[i].xmax) 206 | ymax = int(predbox[i].ymax) 207 | classId = predbox[i].classId 208 | score = predbox[i].score 209 | 210 | cv2.rectangle(origimg, (xmin, ymin), (xmax, ymax), (128, 128, 0), 2) 211 | ptext = (xmin, ymin) 212 | title = CLASSES[classId] + "%.2f" % score 213 | cv2.putText(origimg, title, ptext, cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2, cv2.LINE_AA) 214 | 215 | pose = predbox[i].pose 216 | for i in range(0, keypoint_num): 217 | if pose[i * 3] > 0.5: 218 | if i < 5: 219 | color = color_list[0] 220 | elif 5 <= i < 12: 221 | color = color_list[1] 222 | else: 223 | color = color_list[2] 224 | cv2.circle(origimg, (int(pose[i * 3 + 1]), int(pose[i * 3 + 2])), 2, color, 5) 225 | 226 | for i, sk in enumerate(skeleton): 227 | if pose[(sk[0] - 1) * 3] > 0.5: 228 | pos1 = (int(pose[(sk[0] - 1) * 3 + 1]), int(pose[(sk[0] - 1) * 3 + 2])) 229 | pos2 = (int(pose[(sk[1] - 1) * 3 + 1]), int(pose[(sk[1] - 1) * 3 + 2])) 230 | if (sk[0] - 1) < 5: 231 | color = color_list[0] 232 | elif 5 <= sk[0] - 1 < 12: 233 | color = color_list[1] 234 | else: 235 | color = color_list[2] 236 | cv2.line(origimg, pos1, pos2, color, thickness=2, lineType=cv2.LINE_AA) 237 | 238 | cv2.imwrite('./test_horizon_result.jpg', origimg) 239 | # cv2.imshow("test", origimg) 240 | # cv2.waitKey(0) 241 | 242 | 243 | if __name__ == '__main__': 244 | print('This main ... ') 245 | GenerateMeshgrid() 246 | 247 | model_path = './model_output/yolov8seg_quantized_model.onnx' 248 | image_path = './test.jpg' 249 | input_layout = 'NHWC' 250 | input_offset = 128 251 | 252 | inference(model_path, image_path, input_layout, input_offset) 253 | 254 | -------------------------------------------------------------------------------- /yolov8pose_herizon/model/yolov8pos_relu_zq.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/model/yolov8pos_relu_zq.onnx -------------------------------------------------------------------------------- /yolov8pose_herizon/model_output/yolov8seg.bin: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/model_output/yolov8seg.bin -------------------------------------------------------------------------------- /yolov8pose_herizon/model_output/yolov8seg_optimized_float_model.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/model_output/yolov8seg_optimized_float_model.onnx -------------------------------------------------------------------------------- /yolov8pose_herizon/model_output/yolov8seg_original_float_model.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/model_output/yolov8seg_original_float_model.onnx -------------------------------------------------------------------------------- /yolov8pose_herizon/model_output/yolov8seg_quantized_model.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/model_output/yolov8seg_quantized_model.onnx -------------------------------------------------------------------------------- /yolov8pose_herizon/preprocess.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021 Horizon Robotics.All Rights Reserved. 2 | # 3 | # The material in this file is confidential and contains trade secrets 4 | # of Horizon Robotics Inc. This is proprietary information owned by 5 | # Horizon Robotics Inc. No part of this work may be disclosed, 6 | # reproduced, copied, transmitted, or used in any way for any purpose, 7 | # without the express written permission of Horizon Robotics Inc. 8 | 9 | import sys 10 | sys.path.append("../../../01_common/python/data/") 11 | from transformer import * 12 | from dataloader import * 13 | 14 | 15 | def calibration_transformers(): 16 | """ 17 | step： 18 | 1、pad resize to 672 * 672 19 | 2、NHWC to NCHW 20 | 3、bgr to rgb 21 | """ 22 | transformers = [ 23 | PadResizeTransformer(target_size=(672, 672)), 24 | HWC2CHWTransformer(), 25 | BGR2RGBTransformer(data_format="CHW"), 26 | ] 27 | return transformers 28 | 29 | 30 | def infer_transformers(input_shape, input_layout="NHWC"): 31 | """ 32 | step： 33 | 1、pad resize to target_size(input_shape) 34 | 2、bgr to rgb 35 | 3、rgb to nv12 36 | 3、nv12 to yuv444 37 | :param input_shape: input shape(target size) 38 | :param input_layout: NCHW / NHWC 39 | """ 40 | transformers = [ 41 | PadResizeTransformer(target_size=input_shape), 42 | BGR2RGBTransformer(data_format="HWC"), 43 | RGB2NV12Transformer(data_format="HWC"), 44 | NV12ToYUV444Transformer(target_size=input_shape, 45 | yuv444_output_layout=input_layout[1:]), 46 | ] 47 | return transformers 48 | 49 | 50 | def infer_image_preprocess(image_file, input_layout, input_shape): 51 | """ 52 | image for single image inference 53 | note: imread_mode [skimage / opencv] 54 | opencv read image as 8-bit unsigned integers BGR in range [0, 255] 55 | skimage read image as float32 RGB in range [0, 1] 56 | make sure to use the same imread_mode as the model training 57 | :param image_file: image file 58 | :param input_layout: NCHW / NHWC 59 | :param input_shape: input shape（target size） 60 | :return: origin image, processed image (uint8, 0-255) 61 | """ 62 | transformers = infer_transformers(input_shape, input_layout) 63 | origin_image, processed_image = SingleImageDataLoaderWithOrigin( 64 | transformers, image_file, imread_mode="opencv") 65 | return origin_image, processed_image 66 | 67 | 68 | def eval_image_preprocess(image_path, annotation_path, input_shape, 69 | input_layout): 70 | """ 71 | image for full scale evaluation 72 | note: imread_mode [skimage / opencv] 73 | opencv read image as 8-bit unsigned integers BGR in range [0, 255] 74 | skimage read image as float32 RGB in range [0, 1] 75 | make sure to use the same imread_mode as the model training 76 | :param image_path: image path 77 | :param annotation_path: annotation path 78 | :param input_shape: input shape（target size） 79 | :param input_layout: input layout 80 | :return: data loader 81 | """ 82 | transformers = infer_transformers(input_shape, input_layout) 83 | data_loader = COCODataLoader(transformers, 84 | image_path, 85 | annotation_path, 86 | imread_mode='opencv') 87 | 88 | return data_loader 89 | -------------------------------------------------------------------------------- /yolov8pose_herizon/src_data/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/src_data/test.jpg -------------------------------------------------------------------------------- /yolov8pose_herizon/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/test.jpg -------------------------------------------------------------------------------- /yolov8pose_herizon/test_horizon_result.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_herizon/test_horizon_result.jpg -------------------------------------------------------------------------------- /yolov8pose_herizon/yolov8pose_config.yaml: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2020 Horizon Robotics.All Rights Reserved. 2 | # 3 | # The material in this file is confidential and contains trade secrets 4 | # of Horizon Robotics Inc. This is proprietary information owned by 5 | # Horizon Robotics Inc. No part of this work may be disclosed, 6 | # reproduced, copied, transmitted, or used in any way for any purpose, 7 | # without the express written permission of Horizon Robotics Inc. 8 | 9 | # 模型转化相关的参数 10 | # ------------------------------------ 11 | # model conversion related parameters 12 | model_parameters: 13 | # Onnx浮点网络数据模型文件 14 | # ----------------------------------------------------------- 15 | # the model file of floating-point ONNX neural network data 16 | onnx_model: './model/yolov8pos_relu_zq.onnx' 17 | 18 | # 适用BPU架构 19 | # -------------------------------- 20 | # the applicable BPU architecture 21 | march: "bernoulli2" 22 | 23 | # 指定模型转换过程中是否输出各层的中间结果，如果为True，则输出所有层的中间输出结果， 24 | # -------------------------------------------------------------------------------------- 25 | # specifies whether or not to dump the intermediate results of all layers in conversion 26 | # if set to True, then the intermediate results of all layers shall be dumped 27 | layer_out_dump: False 28 | 29 | # 日志文件的输出控制参数， 30 | # debug输出模型转换的详细信息 31 | # info只输出关键信息 32 | # warn输出警告和错误级别以上的信息 33 | # ---------------------------------------------------------------------------------------- 34 | # output control parameter of log file(s), 35 | # if set to 'debug', then details of model conversion will be dumped 36 | # if set to 'info', then only important imformation will be dumped 37 | # if set to 'warn', then information ranked higher than 'warn' and 'error' will be dumped 38 | log_level: 'debug' 39 | 40 | # 模型转换输出的结果的存放目录 41 | # ----------------------------------------------------------- 42 | # the directory in which model conversion results are stored 43 | working_dir: 'model_output' 44 | 45 | # 模型转换输出的用于上板执行的模型文件的名称前缀 46 | # ----------------------------------------------------------------------------------------- 47 | # model conversion generated name prefix of those model files used for dev board execution 48 | output_model_file_prefix: 'yolov8seg' 49 | 50 | # 模型输入相关参数, 若输入多个节点, 则应使用';'进行分隔, 使用默认缺省设置则写None 51 | # -------------------------------------------------------------------------- 52 | # model input related parameters, 53 | # please use ";" to seperate when inputting multiple nodes, 54 | # please use None for default setting 55 | input_parameters: 56 | 57 | # (选填) 模型输入的节点名称, 此名称应与模型文件中的名称一致, 否则会报错, 不填则会使用模型文件中的节点名称 58 | # -------------------------------------------------------------------------------------------------------- 59 | # (Optional) node name of model input, 60 | # it shall be the same as the name of model file, otherwise an error will be reported, 61 | # the node name of model file will be used when left blank 62 | input_name: "" 63 | 64 | # 网络实际执行时，输入给网络的数据格式，包括 nv12/rgb/bgr/yuv444/gray/featuremap, 65 | # ------------------------------------------------------------------------------------------ 66 | # the data formats to be passed into neural network when actually performing neural network 67 | # available options: nv12/rgb/bgr/yuv444/gray/featuremap, 68 | input_type_rt: 'rgb' 69 | 70 | # 网络实际执行时输入的数据排布, 可选值为 NHWC/NCHW 71 | # 若input_type_rt配置为nv12，则此处参数不需要配置 72 | # ------------------------------------------------------------------ 73 | # the data layout formats to be passed into neural network when actually performing neural network, available options: NHWC/NCHW 74 | # If input_type_rt is configured as nv12, then this parameter does not need to be configured 75 | input_layout_rt: 'NCHW' 76 | 77 | # 网络训练时输入的数据格式，可选的值为rgb/bgr/gray/featuremap/yuv444 78 | # -------------------------------------------------------------------- 79 | # the data formats in network training 80 | # available options: rgb/bgr/gray/featuremap/yuv444 81 | input_type_train: 'rgb' 82 | 83 | # 网络训练时输入的数据排布, 可选值为 NHWC/NCHW 84 | # ------------------------------------------------------------------ 85 | # the data layout in network training, available options: NHWC/NCHW 86 | input_layout_train: 'NCHW' 87 | 88 | # (选填) 模型网络的输入大小, 以'x'分隔, 不填则会使用模型文件中的网络输入大小，否则会覆盖模型文件中输入大小 89 | # ------------------------------------------------------------------------------------------- 90 | # (Optional)the input size of model network, seperated by 'x' 91 | # note that the network input size of model file will be used if left blank 92 | # otherwise it will overwrite the input size of model file 93 | input_shape: '' 94 | 95 | # 网络实际执行时，输入给网络的batch_size, 默认值为1 96 | # --------------------------------------------------------------------- 97 | # the data batch_size to be passed into neural network when actually performing neural network, default value: 1 98 | #input_batch: 1 99 | 100 | # 网络输入的预处理方法，主要有以下几种： 101 | # no_preprocess 不做任何操作 102 | # data_mean 减去通道均值mean_value 103 | # data_scale 对图像像素乘以data_scale系数 104 | # data_mean_and_scale 减去通道均值后再乘以scale系数 105 | # ------------------------------------------------------------------------------------------- 106 | # preprocessing methods of network input, available options: 107 | # 'no_preprocess' indicates that no preprocess will be made 108 | # 'data_mean' indicates that to minus the channel mean, i.e. mean_value 109 | # 'data_scale' indicates that image pixels to multiply data_scale ratio 110 | # 'data_mean_and_scale' indicates that to multiply scale ratio after channel mean is minused 111 | norm_type: 'data_scale' 112 | 113 | # 图像减去的均值, 如果是通道均值，value之间必须用空格分隔 114 | # -------------------------------------------------------------------------- 115 | # the mean value minused by image 116 | # note that values must be seperated by space if channel mean value is used 117 | mean_value: '' 118 | 119 | # 图像预处理缩放比例，如果是通道缩放比例，value之间必须用空格分隔 120 | # --------------------------------------------------------------------------- 121 | # scale value of image preprocess 122 | # note that values must be seperated by space if channel scale value is used 123 | scale_value: 0.003921568627451 124 | 125 | # 模型量化相关参数 126 | # ----------------------------- 127 | # model calibration parameters 128 | calibration_parameters: 129 | 130 | # 模型量化的参考图像的存放目录，图片格式支持Jpeg、Bmp等格式，输入的图片 131 | # 应该是使用的典型场景，一般是从测试集中选择20~100张图片，另外输入 132 | # 的图片要覆盖典型场景，不要是偏僻场景，如过曝光、饱和、模糊、纯黑、纯白等图片 133 | # 若有多个输入节点, 则应使用';'进行分隔 134 | # ------------------------------------------------------------------------------------------------- 135 | # the directory where reference images of model quantization are stored 136 | # image formats include JPEG, BMP etc. 137 | # should be classic application scenarios, usually 20~100 images are picked out from test datasets 138 | # in addition, note that input images should cover typical scenarios 139 | # and try to avoid those overexposed, oversaturated, vague, 140 | # pure blank or pure white images 141 | # use ';' to seperate when there are multiple input nodes 142 | cal_data_dir: './cal_data' 143 | 144 | # 如果输入的图片文件尺寸和模型训练的尺寸不一致时，并且preprocess_on为true， 145 | # 则将采用默认预处理方法(skimage resize)， 146 | # 将输入图片缩放或者裁减到指定尺寸，否则，需要用户提前把图片处理为训练时的尺寸 147 | # --------------------------------------------------------------------------------- 148 | # In case the size of input image file is different from that of in model training 149 | # and that preprocess_on is set to True, 150 | # shall the default preprocess method(skimage resize) be used 151 | # i.e., to resize or crop input image into specified size 152 | # otherwise user must keep image size as that of in training in advance 153 | preprocess_on: True 154 | 155 | # 模型量化的算法类型，支持kl、max、default、load，通常采用default即可满足要求, 若为QAT导出的模型, 则应选择load 156 | # ---------------------------------------------------------------------------------- 157 | # types of model quantization algorithms, usually default will meet the need 158 | # available options:kl, max, default and load 159 | # if converted model is quanti model exported from QAT , then choose `load` 160 | calibration_type: 'default' 161 | 162 | # 编译器相关参数 163 | # ---------------------------- 164 | # compiler related parameters 165 | compiler_parameters: 166 | 167 | # 编译策略，支持bandwidth和latency两种优化模式; 168 | # bandwidth以优化ddr的访问带宽为目标； 169 | # latency以优化推理时间为目标 170 | # ------------------------------------------------------------------------------------------- 171 | # compilation strategy, there are 2 available optimization modes: 'bandwidth' and 'lantency' 172 | # the 'bandwidth' mode aims to optimize ddr access bandwidth 173 | # while the 'lantency' mode aims to optimize inference duration 174 | compile_mode: 'latency' 175 | 176 | # 设置debug为True将打开编译器的debug模式，能够输出性能仿真的相关信息，如帧率、DDR带宽占用等 177 | # ----------------------------------------------------------------------------------- 178 | # the compiler's debug mode will be enabled by setting to True 179 | # this will dump performance simulation related information 180 | # such as: frame rate, DDR bandwidth usage etc. 181 | debug: False 182 | 183 | # 编译模型指定核数，不指定默认编译单核模型, 若编译双核模型，将下边注释打开即可 184 | # ------------------------------------------------------------------------------------- 185 | # specifies number of cores to be used in model compilation 186 | # as default, single core is used as this value left blank 187 | # please delete the "# " below to enable dual-core mode when compiling dual-core model 188 | # core_num: 2 189 | 190 | # 优化等级可选范围为O0~O3 191 | # O0不做任何优化, 编译速度最快，优化程度最低, 192 | # O1-O3随着优化等级提高，预期编译后的模型的执行速度会更快，但是所需编译时间也会变长。 193 | # 推荐用O2做最快验证 194 | # ---------------------------------------------------------------------------------------------------------- 195 | # optimization level ranges between O0~O3 196 | # O0 indicates that no optimization will be made 197 | # the faster the compilation, the lower optimization level will be 198 | # O1-O3: as optimization levels increase gradually, model execution, after compilation, shall become faster 199 | # while compilation will be prolonged 200 | # it is recommended to use O2 for fastest verification 201 | optimize_level: 'O3' 202 | -------------------------------------------------------------------------------- /yolov8pose_onnx/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_onnx/test.jpg -------------------------------------------------------------------------------- /yolov8pose_onnx/test_onnx_result.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_onnx/test_onnx_result.jpg -------------------------------------------------------------------------------- /yolov8pose_onnx/test_pytorch_result.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_onnx/test_pytorch_result.jpg -------------------------------------------------------------------------------- /yolov8pose_onnx/yolov8-pose_onnx_demo.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding:utf-8 -*- 3 | from math import exp 4 | import cv2 5 | import numpy as np 6 | import onnxruntime as ort 7 | 8 | 9 | color_list = [(0, 0, 255), (0, 255, 0), (255, 0, 0)] 10 | skeleton = [[16, 14], [14, 12], [17, 15], [15, 13], [12, 13], [6, 12], [7, 13], [6, 7], [6, 8], [7, 9], [8, 10], [9, 11], 11 | [2, 3], [1, 2], [1, 3], [2, 4], [3, 5], [4, 6], [5, 7]] 12 | 13 | CLASSES = ['person'] 14 | 15 | meshgrid = [] 16 | 17 | class_num = len(CLASSES) 18 | headNum = 3 19 | keypoint_num = 17 20 | 21 | strides = [8, 16, 32] 22 | mapSize = [[80, 80], [40, 40], [20, 20]] 23 | nmsThresh = 0.55 24 | objectThresh = 0.5 25 | 26 | input_imgH = 640 27 | input_imgW = 640 28 | 29 | 30 | class DetectBox: 31 | def __init__(self, classId, score, xmin, ymin, xmax, ymax, pose): 32 | self.classId = classId 33 | self.score = score 34 | self.xmin = xmin 35 | self.ymin = ymin 36 | self.xmax = xmax 37 | self.ymax = ymax 38 | self.pose = pose 39 | 40 | 41 | def GenerateMeshgrid(): 42 | for index in range(headNum): 43 | for i in range(mapSize[index][0]): 44 | for j in range(mapSize[index][1]): 45 | meshgrid.append(j + 0.5) 46 | meshgrid.append(i + 0.5) 47 | 48 | 49 | def IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2): 50 | xmin = max(xmin1, xmin2) 51 | ymin = max(ymin1, ymin2) 52 | xmax = min(xmax1, xmax2) 53 | ymax = min(ymax1, ymax2) 54 | 55 | innerWidth = xmax - xmin 56 | innerHeight = ymax - ymin 57 | 58 | innerWidth = innerWidth if innerWidth > 0 else 0 59 | innerHeight = innerHeight if innerHeight > 0 else 0 60 | 61 | innerArea = innerWidth * innerHeight 62 | 63 | area1 = (xmax1 - xmin1) * (ymax1 - ymin1) 64 | area2 = (xmax2 - xmin2) * (ymax2 - ymin2) 65 | 66 | total = area1 + area2 - innerArea 67 | 68 | return innerArea / total 69 | 70 | 71 | def NMS(detectResult): 72 | predBoxs = [] 73 | 74 | sort_detectboxs = sorted(detectResult, key=lambda x: x.score, reverse=True) 75 | 76 | for i in range(len(sort_detectboxs)): 77 | xmin1 = sort_detectboxs[i].xmin 78 | ymin1 = sort_detectboxs[i].ymin 79 | xmax1 = sort_detectboxs[i].xmax 80 | ymax1 = sort_detectboxs[i].ymax 81 | classId = sort_detectboxs[i].classId 82 | 83 | if sort_detectboxs[i].classId != -1: 84 | predBoxs.append(sort_detectboxs[i]) 85 | for j in range(i + 1, len(sort_detectboxs), 1): 86 | if classId == sort_detectboxs[j].classId: 87 | xmin2 = sort_detectboxs[j].xmin 88 | ymin2 = sort_detectboxs[j].ymin 89 | xmax2 = sort_detectboxs[j].xmax 90 | ymax2 = sort_detectboxs[j].ymax 91 | iou = IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2) 92 | if iou > nmsThresh: 93 | sort_detectboxs[j].classId = -1 94 | return predBoxs 95 | 96 | 97 | def sigmoid(x): 98 | return 1 / (1 + exp(-x)) 99 | 100 | 101 | def postprocess(out, img_h, img_w): 102 | print('postprocess ... ') 103 | 104 | detectResult = [] 105 | 106 | output = [] 107 | for i in range(len(out)): 108 | output.append(out[i].reshape((-1))) 109 | 110 | scale_h = img_h / input_imgH 111 | scale_w = img_w / input_imgW 112 | 113 | gridIndex = -2 114 | 115 | for index in range(headNum): 116 | reg = output[index * 2 + 0] 117 | cls = output[index * 2 + 1] 118 | pose = output[headNum * 2 + index] 119 | 120 | for h in range(mapSize[index][0]): 121 | for w in range(mapSize[index][1]): 122 | gridIndex += 2 123 | 124 | for cl in range(class_num): 125 | cls_val = sigmoid(cls[cl * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 126 | 127 | if cls_val > objectThresh: 128 | x1 = (meshgrid[gridIndex + 0] - reg[0 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 129 | y1 = (meshgrid[gridIndex + 1] - reg[1 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 130 | x2 = (meshgrid[gridIndex + 0] + reg[2 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 131 | y2 = (meshgrid[gridIndex + 1] + reg[3 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 132 | 133 | xmin = x1 * scale_w 134 | ymin = y1 * scale_h 135 | xmax = x2 * scale_w 136 | ymax = y2 * scale_h 137 | 138 | xmin = xmin if xmin > 0 else 0 139 | ymin = ymin if ymin > 0 else 0 140 | xmax = xmax if xmax < img_w else img_w 141 | ymax = ymax if ymax < img_h else img_h 142 | 143 | poseResult = [] 144 | for kc in range(keypoint_num): 145 | px = pose[(kc * 3 + 0) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 146 | py = pose[(kc * 3 + 1) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 147 | vs = sigmoid(pose[(kc * 3 + 2) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 148 | 149 | x = (px * 2.0 + (meshgrid[gridIndex + 0] - 0.5)) * strides[index] * scale_w 150 | y = (py * 2.0 + (meshgrid[gridIndex + 1] - 0.5)) * strides[index] * scale_h 151 | 152 | poseResult.append(vs) 153 | poseResult.append(x) 154 | poseResult.append(y) 155 | # print(poseResult) 156 | box = DetectBox(cl, cls_val, xmin, ymin, xmax, ymax, poseResult) 157 | detectResult.append(box) 158 | 159 | # NMS 160 | print('detectResult:', len(detectResult)) 161 | predBox = NMS(detectResult) 162 | 163 | return predBox 164 | 165 | 166 | def precess_image(img_src, resize_w, resize_h): 167 | image = cv2.resize(img_src, (resize_w, resize_h), interpolation=cv2.INTER_LINEAR) 168 | image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) 169 | image = image.astype(np.float32) 170 | image /= 255 171 | return image 172 | 173 | 174 | def detect(img_path): 175 | 176 | orig = cv2.imread(img_path) 177 | img_h, img_w = orig.shape[:2] 178 | image = precess_image(orig, input_imgW, input_imgH) 179 | 180 | image = image.transpose((2, 0, 1)) 181 | image = np.expand_dims(image, axis=0) 182 | 183 | ort_session = ort.InferenceSession('./yolov8pos_relu_zq.onnx') 184 | pred_results = (ort_session.run(None, {'data': image})) 185 | 186 | out = [] 187 | for i in range(len(pred_results)): 188 | out.append(pred_results[i]) 189 | predbox = postprocess(out, img_h, img_w) 190 | 191 | print('obj num is :', len(predbox)) 192 | 193 | for i in range(len(predbox)): 194 | xmin = int(predbox[i].xmin) 195 | ymin = int(predbox[i].ymin) 196 | xmax = int(predbox[i].xmax) 197 | ymax = int(predbox[i].ymax) 198 | classId = predbox[i].classId 199 | score = predbox[i].score 200 | 201 | cv2.rectangle(orig, (xmin, ymin), (xmax, ymax), (128, 128, 0), 2) 202 | ptext = (xmin, ymin) 203 | title = CLASSES[classId] + "%.2f" % score 204 | cv2.putText(orig, title, ptext, cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2, cv2.LINE_AA) 205 | 206 | pose = predbox[i].pose 207 | for i in range(0, keypoint_num): 208 | if pose[i * 3] > 0.5: 209 | if i < 5: 210 | color = color_list[0] 211 | elif 5 <= i < 12: 212 | color = color_list[1] 213 | else: 214 | color = color_list[2] 215 | cv2.circle(orig, (int(pose[i * 3 + 1]), int(pose[i * 3 + 2])), 2, color, 5) 216 | 217 | for i, sk in enumerate(skeleton): 218 | if pose[(sk[0] - 1) * 3] > 0.5: 219 | pos1 = (int(pose[(sk[0] - 1) * 3 + 1]), int(pose[(sk[0] - 1) * 3 + 2])) 220 | pos2 = (int(pose[(sk[1] - 1) * 3 + 1]), int(pose[(sk[1] - 1) * 3 + 2])) 221 | if (sk[0] - 1) < 5: 222 | color = color_list[0] 223 | elif 5 <= sk[0] - 1 < 12: 224 | color = color_list[1] 225 | else: 226 | color = color_list[2] 227 | cv2.line(orig, pos1, pos2, color, thickness=2, lineType=cv2.LINE_AA) 228 | 229 | cv2.imwrite('./test_onnx_result.jpg', orig) 230 | 231 | 232 | if __name__ == '__main__': 233 | print('This is main ....') 234 | GenerateMeshgrid() 235 | img_path = './test.jpg' 236 | detect(img_path) 237 | 238 | 239 | -------------------------------------------------------------------------------- /yolov8pose_onnx/yolov8pos_relu_zq.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_onnx/yolov8pos_relu_zq.onnx -------------------------------------------------------------------------------- /yolov8pose_rknn/data/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_rknn/data/test.jpg -------------------------------------------------------------------------------- /yolov8pose_rknn/dataset.txt: -------------------------------------------------------------------------------- 1 | ./data/test.jpg 2 | -------------------------------------------------------------------------------- /yolov8pose_rknn/onnx2rknn_demo_ZQ.py: -------------------------------------------------------------------------------- 1 | import os 2 | import urllib 3 | import traceback 4 | import time 5 | import sys 6 | import numpy as np 7 | import cv2 8 | from rknn.api import RKNN 9 | from math import exp 10 | 11 | ONNX_MODEL = './yolov8pos_relu_zq.onnx' 12 | RKNN_MODEL = './yolov8pos_relu_zq.rknn' 13 | DATASET = './dataset.txt' 14 | 15 | QUANTIZE_ON = True 16 | 17 | 18 | color_list = [(0, 0, 255), (0, 255, 0), (255, 0, 0)] 19 | skeleton = [[16, 14], [14, 12], [17, 15], [15, 13], [12, 13], [6, 12], [7, 13], [6, 7], [6, 8], [7, 9], [8, 10], [9, 11], 20 | [2, 3], [1, 2], [1, 3], [2, 4], [3, 5], [4, 6], [5, 7]] 21 | 22 | CLASSES = ['person'] 23 | 24 | meshgrid = [] 25 | 26 | class_num = len(CLASSES) 27 | headNum = 3 28 | keypoint_num = 17 29 | 30 | strides = [8, 16, 32] 31 | mapSize = [[80, 80], [40, 40], [20, 20]] 32 | nmsThresh = 0.55 33 | objectThresh = 0.5 34 | 35 | input_imgH = 640 36 | input_imgW = 640 37 | 38 | 39 | class DetectBox: 40 | def __init__(self, classId, score, xmin, ymin, xmax, ymax, pose): 41 | self.classId = classId 42 | self.score = score 43 | self.xmin = xmin 44 | self.ymin = ymin 45 | self.xmax = xmax 46 | self.ymax = ymax 47 | self.pose = pose 48 | 49 | 50 | def GenerateMeshgrid(): 51 | for index in range(headNum): 52 | for i in range(mapSize[index][0]): 53 | for j in range(mapSize[index][1]): 54 | meshgrid.append(j + 0.5) 55 | meshgrid.append(i + 0.5) 56 | 57 | 58 | def IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2): 59 | xmin = max(xmin1, xmin2) 60 | ymin = max(ymin1, ymin2) 61 | xmax = min(xmax1, xmax2) 62 | ymax = min(ymax1, ymax2) 63 | 64 | innerWidth = xmax - xmin 65 | innerHeight = ymax - ymin 66 | 67 | innerWidth = innerWidth if innerWidth > 0 else 0 68 | innerHeight = innerHeight if innerHeight > 0 else 0 69 | 70 | innerArea = innerWidth * innerHeight 71 | 72 | area1 = (xmax1 - xmin1) * (ymax1 - ymin1) 73 | area2 = (xmax2 - xmin2) * (ymax2 - ymin2) 74 | 75 | total = area1 + area2 - innerArea 76 | 77 | return innerArea / total 78 | 79 | 80 | def NMS(detectResult): 81 | predBoxs = [] 82 | 83 | sort_detectboxs = sorted(detectResult, key=lambda x: x.score, reverse=True) 84 | 85 | for i in range(len(sort_detectboxs)): 86 | xmin1 = sort_detectboxs[i].xmin 87 | ymin1 = sort_detectboxs[i].ymin 88 | xmax1 = sort_detectboxs[i].xmax 89 | ymax1 = sort_detectboxs[i].ymax 90 | classId = sort_detectboxs[i].classId 91 | 92 | if sort_detectboxs[i].classId != -1: 93 | predBoxs.append(sort_detectboxs[i]) 94 | for j in range(i + 1, len(sort_detectboxs), 1): 95 | if classId == sort_detectboxs[j].classId: 96 | xmin2 = sort_detectboxs[j].xmin 97 | ymin2 = sort_detectboxs[j].ymin 98 | xmax2 = sort_detectboxs[j].xmax 99 | ymax2 = sort_detectboxs[j].ymax 100 | iou = IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2) 101 | if iou > nmsThresh: 102 | sort_detectboxs[j].classId = -1 103 | return predBoxs 104 | 105 | 106 | def sigmoid(x): 107 | return 1 / (1 + exp(-x)) 108 | 109 | 110 | def postprocess(out, img_h, img_w): 111 | print('postprocess ... ') 112 | 113 | detectResult = [] 114 | 115 | output = [] 116 | for i in range(len(out)): 117 | output.append(out[i].reshape((-1))) 118 | 119 | scale_h = img_h / input_imgH 120 | scale_w = img_w / input_imgW 121 | 122 | gridIndex = -2 123 | 124 | for index in range(headNum): 125 | reg = output[index * 2 + 0] 126 | cls = output[index * 2 + 1] 127 | pose = output[headNum * 2 + index] 128 | 129 | for h in range(mapSize[index][0]): 130 | for w in range(mapSize[index][1]): 131 | gridIndex += 2 132 | 133 | for cl in range(class_num): 134 | cls_val = sigmoid(cls[cl * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 135 | 136 | if cls_val > objectThresh: 137 | x1 = (meshgrid[gridIndex + 0] - reg[0 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 138 | y1 = (meshgrid[gridIndex + 1] - reg[1 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 139 | x2 = (meshgrid[gridIndex + 0] + reg[2 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 140 | y2 = (meshgrid[gridIndex + 1] + reg[3 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 141 | 142 | xmin = x1 * scale_w 143 | ymin = y1 * scale_h 144 | xmax = x2 * scale_w 145 | ymax = y2 * scale_h 146 | 147 | xmin = xmin if xmin > 0 else 0 148 | ymin = ymin if ymin > 0 else 0 149 | xmax = xmax if xmax < img_w else img_w 150 | ymax = ymax if ymax < img_h else img_h 151 | 152 | poseResult = [] 153 | for kc in range(keypoint_num): 154 | px = pose[(kc * 3 + 0) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 155 | py = pose[(kc * 3 + 1) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 156 | vs = sigmoid(pose[(kc * 3 + 2) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 157 | 158 | x = (px * 2.0 + (meshgrid[gridIndex + 0] - 0.5)) * strides[index] * scale_w 159 | y = (py * 2.0 + (meshgrid[gridIndex + 1] - 0.5)) * strides[index] * scale_h 160 | 161 | poseResult.append(vs) 162 | poseResult.append(x) 163 | poseResult.append(y) 164 | # print(poseResult) 165 | box = DetectBox(cl, cls_val, xmin, ymin, xmax, ymax, poseResult) 166 | detectResult.append(box) 167 | 168 | # NMS 169 | print('detectResult:', len(detectResult)) 170 | predBox = NMS(detectResult) 171 | 172 | return predBox 173 | 174 | 175 | def export_rknn_inference(img): 176 | # Create RKNN object 177 | rknn = RKNN(verbose=True) 178 | 179 | # pre-process config 180 | print('--> Config model') 181 | rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], quantized_algorithm='normal', quantized_method='channel', target_platform='rk3588') 182 | print('done') 183 | 184 | # Load ONNX model 185 | print('--> Loading model') 186 | ret = rknn.load_onnx(model=ONNX_MODEL, outputs=["cls1", "reg1", "cls2", "reg2", "cls3", "reg3", "ps1", "ps2", "ps3"]) 187 | if ret != 0: 188 | print('Load model failed!') 189 | exit(ret) 190 | print('done') 191 | 192 | # Build model 193 | print('--> Building model') 194 | ret = rknn.build(do_quantization=QUANTIZE_ON, dataset=DATASET, rknn_batch_size=1) 195 | if ret != 0: 196 | print('Build model failed!') 197 | exit(ret) 198 | print('done') 199 | 200 | # Export RKNN model 201 | print('--> Export rknn model') 202 | ret = rknn.export_rknn(RKNN_MODEL) 203 | if ret != 0: 204 | print('Export rknn model failed!') 205 | exit(ret) 206 | print('done') 207 | 208 | # Init runtime environment 209 | print('--> Init runtime environment') 210 | ret = rknn.init_runtime() 211 | # ret = rknn.init_runtime(target='rk3566') 212 | if ret != 0: 213 | print('Init runtime environment failed!') 214 | exit(ret) 215 | print('done') 216 | 217 | # Inference 218 | print('--> Running model') 219 | outputs = rknn.inference(inputs=[img]) 220 | rknn.release() 221 | print('done') 222 | 223 | return outputs 224 | 225 | 226 | if __name__ == '__main__': 227 | print('This is main ...') 228 | GenerateMeshgrid() 229 | img_path = './test.jpg' 230 | orig_img = cv2.imread(img_path) 231 | img_h, img_w = orig_img.shape[:2] 232 | 233 | 234 | origimg = cv2.resize(orig_img, (input_imgW, input_imgH), interpolation=cv2.INTER_LINEAR) 235 | origimg = cv2.cvtColor(origimg, cv2.COLOR_BGR2RGB) 236 | 237 | img = np.expand_dims(origimg, 0) 238 | 239 | outputs = export_rknn_inference(img) 240 | 241 | out = [] 242 | for i in range(len(outputs)): 243 | out.append(outputs[i]) 244 | predbox = postprocess(out, img_h, img_w) 245 | 246 | print('obj num is :', len(predbox)) 247 | 248 | for i in range(len(predbox)): 249 | xmin = int(predbox[i].xmin) 250 | ymin = int(predbox[i].ymin) 251 | xmax = int(predbox[i].xmax) 252 | ymax = int(predbox[i].ymax) 253 | classId = predbox[i].classId 254 | score = predbox[i].score 255 | 256 | cv2.rectangle(orig_img, (xmin, ymin), (xmax, ymax), (128, 128, 0), 2) 257 | ptext = (xmin, ymin) 258 | title = CLASSES[classId] + "%.2f" % score 259 | cv2.putText(orig_img, title, ptext, cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2, cv2.LINE_AA) 260 | 261 | pose = predbox[i].pose 262 | for i in range(0, keypoint_num): 263 | if pose[i * 3] > 0.5: 264 | if i < 5: 265 | color = color_list[0] 266 | elif 5 <= i < 12: 267 | color = color_list[1] 268 | else: 269 | color = color_list[2] 270 | cv2.circle(orig_img, (int(pose[i * 3 + 1]), int(pose[i * 3 + 2])), 2, color, 5) 271 | 272 | for i, sk in enumerate(skeleton): 273 | if pose[(sk[0] - 1) * 3] > 0.5: 274 | pos1 = (int(pose[(sk[0] - 1) * 3 + 1]), int(pose[(sk[0] - 1) * 3 + 2])) 275 | pos2 = (int(pose[(sk[1] - 1) * 3 + 1]), int(pose[(sk[1] - 1) * 3 + 2])) 276 | if (sk[0] - 1) < 5: 277 | color = color_list[0] 278 | elif 5 <= sk[0] - 1 < 12: 279 | color = color_list[1] 280 | else: 281 | color = color_list[2] 282 | cv2.line(orig_img, pos1, pos2, color, thickness=2, lineType=cv2.LINE_AA) 283 | 284 | cv2.imwrite('./test_rknn_result.jpg', orig_img) 285 | # cv2.imshow("test", orig_img) 286 | # cv2.waitKey(0) 287 | 288 | -------------------------------------------------------------------------------- /yolov8pose_rknn/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_rknn/test.jpg -------------------------------------------------------------------------------- /yolov8pose_rknn/test_rknn_result.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_rknn/test_rknn_result.jpg -------------------------------------------------------------------------------- /yolov8pose_rknn/yolov8pos_relu_zq.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_rknn/yolov8pos_relu_zq.onnx -------------------------------------------------------------------------------- /yolov8pose_rknn/yolov8pos_relu_zq.rknn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_rknn/yolov8pos_relu_zq.rknn -------------------------------------------------------------------------------- /yolov8pose_tensorRT/onnx2trt_rt7.py: -------------------------------------------------------------------------------- 1 | import tensorrt as trt 2 | 3 | G_LOGGER = trt.Logger() 4 | 5 | batch_size = 1 6 | imput_h = 640 7 | imput_w = 640 8 | 9 | 10 | def get_engine(onnx_model_name, trt_model_name): 11 | explicit_batch = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) 12 | with trt.Builder(G_LOGGER) as builder, builder.create_network(explicit_batch) as network, trt.OnnxParser(network, 13 | G_LOGGER) as parser: 14 | builder.max_batch_size = batch_size 15 | builder.max_workspace_size = 2 << 30 16 | print('Loading ONNX file from path {}...'.format(onnx_model_name)) 17 | with open(onnx_model_name, 'rb') as model: 18 | print('Beginning ONNX file parsing') 19 | if not parser.parse(model.read()): 20 | for error in range(parser.num_errors): 21 | print(parser.get_error(error)) 22 | 23 | print('Completed parsing of ONNX file') 24 | print('Building an engine from file {}; this may take a while...'.format(onnx_model_name)) 25 | 26 | #### 27 | # builder.int8_mode = True 28 | # builder.int8_calibrator = calib 29 | builder.fp16_mode = True 30 | #### 31 | 32 | print("num layers:", network.num_layers) 33 | # last_layer = network.get_layer(network.num_layers - 1) 34 | # if not last_layer.get_output(0): 35 | # network.mark_output(network.get_layer(network.num_layers - 1).get_output(0))//有的模型需要，有的模型在转onnx的之后已经指定了，就不需要这行 36 | 37 | network.get_input(0).shape = [batch_size, 3, imput_h, imput_w] 38 | engine = builder.build_cuda_engine(network) 39 | print("engine:", engine) 40 | print("Completed creating Engine") 41 | with open(trt_model_name, "wb") as f: 42 | f.write(engine.serialize()) 43 | return engine 44 | 45 | 46 | def main(): 47 | onnx_file_path = './yolov8pos_relu_zq.onnx' 48 | engine_file_path = './yolov8pos_relu_zq.trt' 49 | 50 | engine = get_engine(onnx_file_path, engine_file_path) 51 | 52 | 53 | if __name__ == '__main__': 54 | print("This is main ...") 55 | main() 56 | -------------------------------------------------------------------------------- /yolov8pose_tensorRT/tensorRT_inferenc_demo.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | import tensorrt as trt 4 | import pycuda.driver as cuda 5 | import pycuda.autoinit 6 | from math import exp 7 | from math import sqrt 8 | 9 | TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE) 10 | 11 | 12 | color_list = [(0, 0, 255), (0, 255, 0), (255, 0, 0)] 13 | skeleton = [[16, 14], [14, 12], [17, 15], [15, 13], [12, 13], [6, 12], [7, 13], [6, 7], [6, 8], [7, 9], [8, 10], [9, 11], 14 | [2, 3], [1, 2], [1, 3], [2, 4], [3, 5], [4, 6], [5, 7]] 15 | 16 | CLASSES = ['person'] 17 | 18 | meshgrid = [] 19 | 20 | class_num = len(CLASSES) 21 | headNum = 3 22 | keypoint_num = 17 23 | 24 | strides = [8, 16, 32] 25 | mapSize = [[80, 80], [40, 40], [20, 20]] 26 | nmsThresh = 0.55 27 | objectThresh = 0.5 28 | 29 | input_imgH = 640 30 | input_imgW = 640 31 | 32 | 33 | # Simple helper data class that's a little nicer to use than a 2-tuple. 34 | class HostDeviceMem(object): 35 | def __init__(self, host_mem, device_mem): 36 | self.host = host_mem 37 | self.device = device_mem 38 | 39 | def __str__(self): 40 | return "Host:\n" + str(self.host) + "\nDevice:\n" + str(self.device) 41 | 42 | def __repr__(self): 43 | return self.__str__() 44 | 45 | 46 | def allocate_buffers(engine): 47 | inputs = [] 48 | outputs = [] 49 | bindings = [] 50 | stream = cuda.Stream() 51 | for binding in engine: 52 | size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size 53 | dtype = trt.nptype(engine.get_binding_dtype(binding)) 54 | # Allocate host and device buffers 55 | host_mem = cuda.pagelocked_empty(size, dtype) 56 | device_mem = cuda.mem_alloc(host_mem.nbytes) 57 | # Append the device buffer to device bindings. 58 | bindings.append(int(device_mem)) 59 | # Append to the appropriate list. 60 | if engine.binding_is_input(binding): 61 | inputs.append(HostDeviceMem(host_mem, device_mem)) 62 | else: 63 | outputs.append(HostDeviceMem(host_mem, device_mem)) 64 | return inputs, outputs, bindings, stream 65 | 66 | 67 | def get_engine_from_bin(engine_file_path): 68 | print('Reading engine from file {}'.format(engine_file_path)) 69 | with open(engine_file_path, 'rb') as f, trt.Runtime(TRT_LOGGER) as runtime: 70 | return runtime.deserialize_cuda_engine(f.read()) 71 | 72 | 73 | # This function is generalized for multiple inputs/outputs. 74 | # inputs and outputs are expected to be lists of HostDeviceMem objects. 75 | def do_inference(context, bindings, inputs, outputs, stream, batch_size=1): 76 | # Transfer input data to the GPU. 77 | [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs] 78 | # Run inference. 79 | context.execute_async(batch_size=batch_size, bindings=bindings, stream_handle=stream.handle) 80 | # Transfer predictions back from the GPU. 81 | [cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs] 82 | # Synchronize the stream 83 | stream.synchronize() 84 | # Return only the host outputs. 85 | return [out.host for out in outputs] 86 | 87 | 88 | class DetectBox: 89 | def __init__(self, classId, score, xmin, ymin, xmax, ymax, pose): 90 | self.classId = classId 91 | self.score = score 92 | self.xmin = xmin 93 | self.ymin = ymin 94 | self.xmax = xmax 95 | self.ymax = ymax 96 | self.pose = pose 97 | 98 | 99 | def GenerateMeshgrid(): 100 | for index in range(headNum): 101 | for i in range(mapSize[index][0]): 102 | for j in range(mapSize[index][1]): 103 | meshgrid.append(j + 0.5) 104 | meshgrid.append(i + 0.5) 105 | 106 | 107 | def IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2): 108 | xmin = max(xmin1, xmin2) 109 | ymin = max(ymin1, ymin2) 110 | xmax = min(xmax1, xmax2) 111 | ymax = min(ymax1, ymax2) 112 | 113 | innerWidth = xmax - xmin 114 | innerHeight = ymax - ymin 115 | 116 | innerWidth = innerWidth if innerWidth > 0 else 0 117 | innerHeight = innerHeight if innerHeight > 0 else 0 118 | 119 | innerArea = innerWidth * innerHeight 120 | 121 | area1 = (xmax1 - xmin1) * (ymax1 - ymin1) 122 | area2 = (xmax2 - xmin2) * (ymax2 - ymin2) 123 | 124 | total = area1 + area2 - innerArea 125 | 126 | return innerArea / total 127 | 128 | 129 | def NMS(detectResult): 130 | predBoxs = [] 131 | 132 | sort_detectboxs = sorted(detectResult, key=lambda x: x.score, reverse=True) 133 | 134 | for i in range(len(sort_detectboxs)): 135 | xmin1 = sort_detectboxs[i].xmin 136 | ymin1 = sort_detectboxs[i].ymin 137 | xmax1 = sort_detectboxs[i].xmax 138 | ymax1 = sort_detectboxs[i].ymax 139 | classId = sort_detectboxs[i].classId 140 | 141 | if sort_detectboxs[i].classId != -1: 142 | predBoxs.append(sort_detectboxs[i]) 143 | for j in range(i + 1, len(sort_detectboxs), 1): 144 | if classId == sort_detectboxs[j].classId: 145 | xmin2 = sort_detectboxs[j].xmin 146 | ymin2 = sort_detectboxs[j].ymin 147 | xmax2 = sort_detectboxs[j].xmax 148 | ymax2 = sort_detectboxs[j].ymax 149 | iou = IOU(xmin1, ymin1, xmax1, ymax1, xmin2, ymin2, xmax2, ymax2) 150 | if iou > nmsThresh: 151 | sort_detectboxs[j].classId = -1 152 | return predBoxs 153 | 154 | 155 | def sigmoid(x): 156 | return 1 / (1 + exp(-x)) 157 | 158 | 159 | def postprocess(out, img_h, img_w): 160 | print('postprocess ... ') 161 | 162 | detectResult = [] 163 | 164 | output = [] 165 | for i in range(len(out)): 166 | output.append(out[i].reshape((-1))) 167 | 168 | scale_h = img_h / input_imgH 169 | scale_w = img_w / input_imgW 170 | 171 | gridIndex = -2 172 | 173 | for index in range(headNum): 174 | cls = output[headNum + index * 2 + 0] 175 | reg = output[headNum + index * 2 + 1] 176 | pose = output[index] 177 | 178 | for h in range(mapSize[index][0]): 179 | for w in range(mapSize[index][1]): 180 | gridIndex += 2 181 | 182 | for cl in range(class_num): 183 | cls_val = sigmoid(cls[cl * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 184 | 185 | if cls_val > objectThresh: 186 | x1 = (meshgrid[gridIndex + 0] - reg[0 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 187 | y1 = (meshgrid[gridIndex + 1] - reg[1 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 188 | x2 = (meshgrid[gridIndex + 0] + reg[2 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 189 | y2 = (meshgrid[gridIndex + 1] + reg[3 * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) * strides[index] 190 | 191 | xmin = x1 * scale_w 192 | ymin = y1 * scale_h 193 | xmax = x2 * scale_w 194 | ymax = y2 * scale_h 195 | 196 | xmin = xmin if xmin > 0 else 0 197 | ymin = ymin if ymin > 0 else 0 198 | xmax = xmax if xmax < img_w else img_w 199 | ymax = ymax if ymax < img_h else img_h 200 | 201 | poseResult = [] 202 | for kc in range(keypoint_num): 203 | px = pose[(kc * 3 + 0) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 204 | py = pose[(kc * 3 + 1) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w] 205 | vs = sigmoid(pose[(kc * 3 + 2) * mapSize[index][0] * mapSize[index][1] + h * mapSize[index][1] + w]) 206 | 207 | x = (px * 2.0 + (meshgrid[gridIndex + 0] - 0.5)) * strides[index] * scale_w 208 | y = (py * 2.0 + (meshgrid[gridIndex + 1] - 0.5)) * strides[index] * scale_h 209 | 210 | poseResult.append(vs) 211 | poseResult.append(x) 212 | poseResult.append(y) 213 | # print(poseResult) 214 | box = DetectBox(cl, cls_val, xmin, ymin, xmax, ymax, poseResult) 215 | detectResult.append(box) 216 | 217 | # NMS 218 | print('detectResult:', len(detectResult)) 219 | predBox = NMS(detectResult) 220 | 221 | return predBox 222 | 223 | 224 | 225 | def preprocess(src): 226 | img = cv2.resize(src, (input_imgW, input_imgH)).astype(np.float32) 227 | img = img * 0.00392156 228 | img = img.transpose(2, 0, 1) 229 | img_input = img.copy() 230 | return img_input 231 | 232 | 233 | def main(): 234 | engine_file_path = 'yolov8pos_relu_zq.trt' 235 | input_image_path = 'test.jpg' 236 | 237 | orig_image = cv2.imread(input_image_path) 238 | orig = cv2.cvtColor(orig_image, cv2.COLOR_BGR2RGB) 239 | img_h, img_w = orig.shape[:2] 240 | image = preprocess(orig) 241 | 242 | with get_engine_from_bin(engine_file_path) as engine, engine.create_execution_context() as context: 243 | inputs, outputs, bindings, stream = allocate_buffers(engine) 244 | 245 | inputs[0].host = image 246 | trt_outputs = do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream, batch_size=1) 247 | print(len(trt_outputs)) 248 | 249 | out = [] 250 | for i in range(len(trt_outputs)): 251 | out.append(trt_outputs[i]) 252 | predbox = postprocess(out, img_h, img_w) 253 | 254 | print('obj num is :', len(predbox)) 255 | 256 | for i in range(len(predbox)): 257 | xmin = int(predbox[i].xmin) 258 | ymin = int(predbox[i].ymin) 259 | xmax = int(predbox[i].xmax) 260 | ymax = int(predbox[i].ymax) 261 | classId = predbox[i].classId 262 | score = predbox[i].score 263 | 264 | cv2.rectangle(orig_image, (xmin, ymin), (xmax, ymax), (128, 128, 0), 2) 265 | ptext = (xmin, ymin) 266 | title = CLASSES[classId] + "%.2f" % score 267 | cv2.putText(orig_image, title, ptext, cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2, cv2.LINE_AA) 268 | 269 | pose = predbox[i].pose 270 | for i in range(0, keypoint_num): 271 | if pose[i * 3] > 0.5: 272 | if i < 5: 273 | color = color_list[0] 274 | elif 5 <= i < 12: 275 | color = color_list[1] 276 | else: 277 | color = color_list[2] 278 | cv2.circle(orig_image, (int(pose[i * 3 + 1]), int(pose[i * 3 + 2])), 2, color, 5) 279 | 280 | for i, sk in enumerate(skeleton): 281 | if pose[(sk[0] - 1) * 3] > 0.5: 282 | pos1 = (int(pose[(sk[0] - 1) * 3 + 1]), int(pose[(sk[0] - 1) * 3 + 2])) 283 | pos2 = (int(pose[(sk[1] - 1) * 3 + 1]), int(pose[(sk[1] - 1) * 3 + 2])) 284 | if (sk[0] - 1) < 5: 285 | color = color_list[0] 286 | elif 5 <= sk[0] - 1 < 12: 287 | color = color_list[1] 288 | else: 289 | color = color_list[2] 290 | cv2.line(orig_image, pos1, pos2, color, thickness=2, lineType=cv2.LINE_AA) 291 | 292 | cv2.imwrite('./test_result_tensorRT.jpg', orig_image) 293 | # cv2.imshow("test", orig_image) 294 | # cv2.waitKey(0) 295 | 296 | 297 | if __name__ == '__main__': 298 | print('This is main ...') 299 | GenerateMeshgrid() 300 | main() 301 | -------------------------------------------------------------------------------- /yolov8pose_tensorRT/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_tensorRT/test.jpg -------------------------------------------------------------------------------- /yolov8pose_tensorRT/test_result_tensorRT.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_tensorRT/test_result_tensorRT.jpg -------------------------------------------------------------------------------- /yolov8pose_tensorRT/yolov8pos_relu_zq.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_tensorRT/yolov8pos_relu_zq.onnx -------------------------------------------------------------------------------- /yolov8pose_tensorRT/yolov8pos_relu_zq.trt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cqu20160901/yolov8pose_onnx_tensorRT_rknn_horizon/6571163f5d3400de36167983be01c5721f84bf76/yolov8pose_tensorRT/yolov8pos_relu_zq.trt --------------------------------------------------------------------------------