├── .github ├── CODEOWNERS ├── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md └── workflows │ └── typos.yml ├── .gitignore ├── .vscode └── settings.json ├── LICENSE ├── README.en.md ├── README.md ├── README.zh-cn.md ├── _typos.toml ├── configs └── .gitkeep ├── images └── スクリーンショット 2023-02-19 131430.png ├── javascript ├── fabric.js └── main.js └── scripts ├── main.py └── openpose ├── body.py ├── model.py └── util.py /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | # default reviewer 2 | * @fkunn1326 3 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Environment** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | 30 | **Additional context** 31 | Add any other context about the problem here. 32 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Additional context** 17 | Add any other context or screenshots about the feature request here. 18 | -------------------------------------------------------------------------------- /.github/workflows/typos.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # yamllint disable rule:line-length 3 | name: Typos 4 | 5 | on: # yamllint disable-line rule:truthy 6 | push: 7 | pull_request: 8 | types: 9 | - opened 10 | - synchronize 11 | - reopened 12 | 13 | jobs: 14 | build: 15 | runs-on: ubuntu-latest 16 | 17 | steps: 18 | - uses: actions/checkout@v3 19 | 20 | - name: typos-action 21 | uses: crate-ci/typos@v1.13.10 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | models/ 3 | -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "nuxt.isNuxtApp": false, 3 | "editor.tabCompletion": "on", 4 | "diffEditor.codeLens": true 5 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Fkunn1326 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.en.md: -------------------------------------------------------------------------------- 1 | ## Openpose Editor 2 | 3 | [日本語](README.md) | English | [简体中文](README.zh-cn.md) 4 | 5 | ![image](https://user-images.githubusercontent.com/92153597/219921945-468b2e4f-a3a0-4d44-a923-13ceb0258ddc.png) 6 | 7 | Openpose Editor for Automatic1111/stable-diffusion-webui 8 | 9 | - Pose editing 10 | - Pose detection 11 | 12 | This can: 13 | 14 | - Add a new person 15 | - Detect pose from an image 16 | - Add background image 17 | 18 | - Save as a PNG 19 | - Send to ControlNet extension 20 | 21 | ## Installation 22 | 23 | 1. Open the "Extension" tab 24 | 2. Click on "Install from URL" 25 | 3. In "URL for extension's git repository" enter this extension, https://github.com/fkunn1326/openpose-editor.git 26 | 4. Click "Install" 27 | 5. Restart WebUI 28 | 29 | ## Attention 30 | 31 | Do not select anything for the Preprocessor in ControlNet. 32 | 33 | 34 | ## Fix Error 35 | > urllib.error.URLError: 36 | 37 | Run 38 | ``` 39 | /Applications/Python\ $version /Install\ Certificates.command 40 | ``` 41 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Openpose Editor 2 | 3 | 日本語 | [English](README.en.md) | [简体中文](README.zh-cn.md) 4 | 5 | ![image](https://user-images.githubusercontent.com/92153597/219921945-468b2e4f-a3a0-4d44-a923-13ceb0258ddc.png) 6 | 7 | Automatic1111/stable-diffusion-webui用のOpenpose Editor 8 | 9 | - ポーズの編集 10 | - ポーズの検出 11 | 12 | ができます 13 | 14 | - 「Add」: 人を追加する 15 | - 「Detect from image」: 画像からポーズを検出する 16 | - 「Add Background image」: 背景を追加する 17 | 18 | - 「Save PNG」: PNGで保存する 19 | - 「Send to ControlNet」: Controlnet拡張機能がインストールされている場合、画像をそこに送る 20 | 21 | ## インストール方法 22 | 23 | 1. "Extension" タブを開く 24 | 2. "Install from URL" タブを開く 25 | 3. "URL for extension's git repository" 欄にこのリポジトリの URL (https://github.com/fkunn1326/openpose-editor.git) を入れます。 26 | 4. "Install" ボタンを押す 27 | 5. WebUIを再起動する 28 | 29 | ## 注意 30 | 31 | ControlNetの "Preprocessor" には、何も指定しないようにしてください。 32 | 33 | ## エラーの対策 34 | 35 | > urllib.error.URLError: 36 | 37 | 38 | 以下のファイルを開いてださい 39 | ``` 40 | /Applications/Python\ $version /Install\ Certificates.command 41 | ``` 42 | -------------------------------------------------------------------------------- /README.zh-cn.md: -------------------------------------------------------------------------------- 1 | ## Openpose Editor 2 | 3 | [日本語](README.md) | [English](README.en.md)|中文 4 | 5 | ![image](https://user-images.githubusercontent.com/92153597/219921945-468b2e4f-a3a0-4d44-a923-13ceb0258ddc.png) 6 | 7 | 适用于Automatic1111/stable-diffusion-webui 的Openpose Editor 插件。 8 | 9 | 功能： 10 | - 直接编辑骨骼动作 11 | - 从图像识别姿势 12 | 13 | 本插件实现以下操作： 14 | 15 | - 「Add」：添加一个新骨骼 16 | - 「Detect from image」: 从图片中识别姿势 17 | - 「Add Background image」: 添加背景图片 18 | - 「Load JSON」：载入JSON文件 19 | 20 | - 「Save PNG」: 保存为PNG格式图片 21 | - 「Send to ControlNet」:将骨骼姿势发送到 ControlNet 22 | - 「Save JSON」：将骨骼保存为JSON 23 | ## 安装方法 24 | 25 | 1. 打开扩展（Extension）标签。 26 | 2. 点击从网址安装（Install from URL） 27 | 3. 在扩展的 git 仓库网址（URL for extension's git repository）处输入 https://github.com/fkunn1326/openpose-editor.git 28 | 4. 点击安装（Install） 29 | 5. 重启 WebUI 30 | ## 注意 31 | 32 | 不要给ConrtolNet 的 "Preprocessor" 选项指定任何值，请保持在none状态 33 | 34 | ## 常见问题 35 | Mac OS可能会出现： 36 | > urllib.error.URLError: 37 | 38 | 请执行文件 39 | ``` 40 | /Applications/Python\ $version /Install\ Certificates.command 41 | ``` 42 | -------------------------------------------------------------------------------- /_typos.toml: -------------------------------------------------------------------------------- 1 | # Files for typos 2 | # Instruction: https://github.com/marketplace/actions/typos-action#getting-started 3 | 4 | [default.extend-identifiers] 5 | 6 | [default.extend-words] 7 | 8 | 9 | 10 | [files] 11 | extend-exclude = ["_typos.toml", "fabric.js"] 12 | -------------------------------------------------------------------------------- /configs/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fkunn1326/openpose-editor/c935771507f57201e26f4ed507aa47f3812ca491/configs/.gitkeep -------------------------------------------------------------------------------- /images/スクリーンショット 2023-02-19 131430.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fkunn1326/openpose-editor/c935771507f57201e26f4ed507aa47f3812ca491/images/スクリーンショット 2023-02-19 131430.png -------------------------------------------------------------------------------- /javascript/main.js: -------------------------------------------------------------------------------- 1 | fabric.Object.prototype.transparentCorners = false; 2 | fabric.Object.prototype.cornerColor = '#108ce6'; 3 | fabric.Object.prototype.borderColor = '#108ce6'; 4 | fabric.Object.prototype.cornerSize = 10; 5 | 6 | let count = 0; 7 | let executed_openpose_editor = false; 8 | 9 | let lockMode = false; 10 | const undo_history = []; 11 | const redo_history = []; 12 | 13 | coco_body_keypoints = [ 14 | "nose", 15 | "neck", 16 | "right_shoulder", 17 | "right_elbow", 18 | "right_wrist", 19 | "left_shoulder", 20 | "left_elbow", 21 | "left_wrist", 22 | "right_hip", 23 | "right_knee", 24 | "right_ankle", 25 | "left_hip", 26 | "left_knee", 27 | "left_ankle", 28 | "right_eye", 29 | "left_eye", 30 | "right_ear", 31 | "left_ear", 32 | ] 33 | 34 | let connect_keypoints = [[0, 1], [1, 2], [2, 3], [3, 4], [1, 5], [5, 6], [6, 7], [1, 8], [8, 9], [9, 10], [1, 11], [11, 12], [12, 13], [14, 0], [14, 16], [15, 0], [15, 17]] 35 | 36 | let connect_color = [[0, 0, 255], [255, 0, 0], [255, 170, 0], [255, 255, 0], [255, 85, 0], [170, 255, 0], [85, 255, 0], [0, 255, 0], 37 | [0, 255, 85], [0, 255, 170], [0, 255, 255], [0, 170, 255], [0, 85, 255], [85, 0, 255], 38 | [170, 0, 255], [255, 0, 255], [255, 0, 170], [255, 0, 85]] 39 | 40 | let openpose_obj = { 41 | // width, height 42 | resolution: [512, 512], 43 | // fps...? 44 | fps: 1, 45 | // frames 46 | frames: [ 47 | { 48 | frame_current: 1, 49 | // armatures 50 | armatures: { 51 | }, 52 | } 53 | ] 54 | } 55 | 56 | let visibleEyes = true; 57 | let flipped = false; 58 | const default_keypoints = [[241,77],[241,120],[191,118],[177,183],[163,252],[298,118],[317,182],[332,245],[225,241],[213,359],[215,454],[270,240],[282,360],[286,456],[232,59],[253,60],[225,70],[260,72]] 59 | 60 | async function fileToDataUrl(file) { 61 | const elem = gradioApp().querySelector(`#${file}`) 62 | const input = elem.previousElementSibling 63 | return await new Promise(r => { 64 | let a = new FileReader(); 65 | a.onload = r; 66 | a.readAsDataURL(input.files[0]) 67 | }).then(e => e.target.result) 68 | } 69 | 70 | function calcResolution(width, height){ 71 | const viewportWidth = window.innerWidth / 2.25; 72 | const viewportHeight = window.innerHeight * 0.75; 73 | const ratio = Math.min(viewportWidth / width, viewportHeight / height); 74 | return {width: width * ratio, height: height * ratio} 75 | } 76 | 77 | function resizeCanvas(width, height){ 78 | const elem = openpose_editor_elem; 79 | const canvas = openpose_editor_canvas; 80 | 81 | let resolution = calcResolution(width, height) 82 | 83 | canvas.setWidth(width); 84 | canvas.setHeight(height); 85 | elem.style.width = resolution["width"] + "px" 86 | elem.style.height = resolution["height"] + "px" 87 | elem.nextElementSibling.style.width = resolution["width"] + "px" 88 | elem.nextElementSibling.style.height = resolution["height"] + "px" 89 | elem.parentElement.style.width = resolution["width"] + "px" 90 | elem.parentElement.style.height = resolution["height"] + "px" 91 | } 92 | 93 | function undo() { 94 | const canvas = openpose_editor_canvas; 95 | if (undo_history.length > 0) { 96 | lockMode = true; 97 | if (undo_history.length > 1) redo_history.push(undo_history.pop()); 98 | const content = undo_history[undo_history.length - 1]; 99 | canvas.loadFromJSON(content, function () { 100 | canvas.renderAll(); 101 | lockMode = false; 102 | }); 103 | } 104 | } 105 | 106 | function redo() { 107 | const canvas = openpose_editor_canvas; 108 | if (redo_history.length > 0) { 109 | lockMode = true; 110 | const content = redo_history.pop(); 111 | undo_history.push(content); 112 | canvas.loadFromJSON(content, function () { 113 | canvas.renderAll(); 114 | lockMode = false; 115 | }); 116 | } 117 | } 118 | 119 | function setPose(keypoints, clear = true){ 120 | const canvas = openpose_editor_canvas; 121 | if (clear) 122 | canvas.clear() 123 | 124 | canvas.backgroundColor = "#000" 125 | 126 | const res = []; 127 | for (let i = 0; i < keypoints.length; i += 18) { 128 | const chunk = keypoints.slice(i, i + 18); 129 | res.push(chunk); 130 | } 131 | 132 | for (item of res){ 133 | addPose(item) 134 | openpose_editor_canvas.discardActiveObject(); 135 | } 136 | } 137 | 138 | function setPoseV2(people, w, h, clear = true){ 139 | const canvas = openpose_editor_canvas; 140 | 141 | if (clear) 142 | canvas.clear() 143 | 144 | canvas.backgroundColor = "#000" 145 | 146 | const res = []; 147 | for (keypoints of people) { 148 | const pos = []; 149 | for (let i = 0; i < keypoints["pose_keypoints_2d"].length - 1; i += 3) { 150 | const chunk = keypoints["pose_keypoints_2d"].slice(i, i + 3); 151 | if (chunk[2] == 0.0) 152 | pos.push([-1, -1]); 153 | else 154 | pos.push([chunk[0] * w * chunk[2], chunk[1] * h * chunk[2]]); 155 | } 156 | res.push(pos); 157 | } 158 | 159 | let default_relative_keypoints = [] 160 | for(i=0;i{ 169 | for(idx=0;idxx===-1&&y===-1)){ 181 | for(i=0;i { 259 | if(item.includes(i)){ 260 | list.push(lines[idx]) 261 | return idx 262 | } 263 | }) 264 | circle = makeCircle(`rgb(${connect_color[i].join(", ")})`, keypoints[i][0], keypoints[i][1], ...list) 265 | circle["id"] = i 266 | circles.push(circle) 267 | // canvas.add(circle) 268 | group.addWithUpdate(circle); 269 | } 270 | 271 | canvas.discardActiveObject(); 272 | canvas.setActiveObject(group); 273 | canvas.add(group); 274 | group.toActiveSelection(); 275 | canvas.requestRenderAll(); 276 | } 277 | 278 | function initCanvas(elem){ 279 | const canvas = window.openpose_editor_canvas = new fabric.Canvas(elem, { 280 | backgroundColor: '#000', 281 | // selection: false, 282 | preserveObjectStacking: true 283 | }); 284 | 285 | window.openpose_editor_elem = elem 286 | 287 | function updateLines(target) { 288 | if ("_objects" in target) { 289 | const flipX = target.flipX ? -1 : 1; 290 | const flipY = target.flipY ? -1 : 1; 291 | flipped = flipX * flipY === -1; 292 | const showEyes = flipped ? !visibleEyes : visibleEyes; 293 | 294 | if (target.angle === 0) { 295 | const rtop = target.top 296 | const rleft = target.left 297 | for (const item of target._objects){ 298 | let p = item; 299 | p.scaleX = 1; 300 | p.scaleY = 1; 301 | const top = rtop + p.top * target.scaleY * flipY + target.height * target.scaleY / 2; 302 | const left = rleft + p.left * target.scaleX * flipX + (target.width * target.scaleX / 2); 303 | p['_top'] = top; 304 | p['_left'] = left; 305 | if (p["id"] === 0) { 306 | p.line1 && p.line1.set({ 'x1': left, 'y1': top }); 307 | }else{ 308 | p.line1 && p.line1.set({ 'x2': left, 'y2': top }); 309 | } 310 | if (p['id'] === 14 || p['id'] === 15) { 311 | p.radius = showEyes ? 5 : 0; 312 | if (p.line1) p.line1.strokeWidth = showEyes ? 10 : 0; 313 | if (p.line2) p.line2.strokeWidth = showEyes ? 10 : 0; 314 | } 315 | p.line2 && p.line2.set({ 'x1': left, 'y1': top }); 316 | p.line3 && p.line3.set({ 'x1': left, 'y1': top }); 317 | p.line4 && p.line4.set({ 'x1': left, 'y1': top }); 318 | p.line5 && p.line5.set({ 'x1': left, 'y1': top }); 319 | 320 | } 321 | } else { 322 | const aCoords = target.aCoords; 323 | const center = {'x': (aCoords.tl.x + aCoords.br.x)/2, 'y': (aCoords.tl.y + aCoords.br.y)/2}; 324 | const rad = target.angle * Math.PI / 180; 325 | const sin = Math.sin(rad); 326 | const cos = Math.cos(rad); 327 | 328 | for (const item of target._objects){ 329 | let p = item; 330 | const p_top = p.top * target.scaleY * flipY; 331 | const p_left = p.left * target.scaleX * flipX; 332 | const left = center.x + p_left * cos - p_top * sin; 333 | const top = center.y + p_left * sin + p_top * cos; 334 | p['_top'] = top; 335 | p['_left'] = left; 336 | if (p["id"] === 0) { 337 | p.line1 && p.line1.set({ 'x1': left, 'y1': top }); 338 | }else{ 339 | p.line1 && p.line1.set({ 'x2': left, 'y2': top }); 340 | } 341 | if (p['id'] === 14 || p['id'] === 15) { 342 | p.radius = showEyes ? 5 : 0.3; 343 | if (p.line1) p.line1.strokeWidth = showEyes ? 10 : 0; 344 | if (p.line2) p.line2.strokeWidth = showEyes ? 10 : 0; 345 | } 346 | p.line2 && p.line2.set({ 'x1': left, 'y1': top }); 347 | p.line3 && p.line3.set({ 'x1': left, 'y1': top }); 348 | p.line4 && p.line4.set({ 'x1': left, 'y1': top }); 349 | p.line5 && p.line5.set({ 'x1': left, 'y1': top }); 350 | } 351 | } 352 | } else { 353 | var p = target; 354 | if (p["id"] === 0) { 355 | p.line1 && p.line1.set({ 'x1': p.left, 'y1': p.top }); 356 | }else{ 357 | p.line1 && p.line1.set({ 'x2': p.left, 'y2': p.top }); 358 | } 359 | p.line2 && p.line2.set({ 'x1': p.left, 'y1': p.top }); 360 | p.line3 && p.line3.set({ 'x1': p.left, 'y1': p.top }); 361 | p.line4 && p.line4.set({ 'x1': p.left, 'y1': p.top }); 362 | p.line5 && p.line5.set({ 'x1': p.left, 'y1': p.top }); 363 | } 364 | canvas.renderAll(); 365 | } 366 | 367 | canvas.on('object:moving', function(e) { 368 | updateLines(e.target); 369 | }); 370 | 371 | canvas.on('object:scaling', function(e) { 372 | updateLines(e.target); 373 | canvas.renderAll(); 374 | }); 375 | 376 | canvas.on('object:rotating', function(e) { 377 | updateLines(e.target); 378 | canvas.renderAll(); 379 | }); 380 | 381 | canvas.on("object:modified", function () { 382 | if (lockMode) return; 383 | undo_history.push(JSON.stringify(canvas)); 384 | redo_history.length = 0; 385 | }); 386 | 387 | resizeCanvas(...openpose_obj.resolution) 388 | 389 | setPose(default_keypoints) 390 | 391 | undo_history.push(JSON.stringify(canvas)); 392 | 393 | const json_observer = new MutationObserver((m) => { 394 | if(gradioApp().querySelector('#tab_openpose_editor').style.display!=='block') return; 395 | try { 396 | const json = gradioApp().querySelector("#jsonbox").querySelector("textarea") 397 | if(json.value.length!==0) detectImage(json.value); 398 | 399 | // reset #jsonbox after detectImage 400 | json.value = ""; 401 | } catch(e){console.log(e)} 402 | }) 403 | json_observer.observe(gradioApp().querySelector("#jsonbox"), { attributes: true, subtree: true }) 404 | 405 | // document.addEventListener('keydown', function(e) { 406 | // if (e.key !== undefined) { 407 | // if((e.key == "z" && (e.metaKey || e.ctrlKey || e.altKey))) undo() 408 | // if((e.key == "y" && (e.metaKey || e.ctrlKey || e.altKey))) redo() 409 | // } 410 | // }) 411 | } 412 | 413 | function resetCanvas(){ 414 | const canvas = openpose_editor_canvas; 415 | canvas.clear() 416 | canvas.backgroundColor = "#000" 417 | } 418 | 419 | function savePNG(){ 420 | openpose_editor_canvas.getObjects("image").forEach((img) => { 421 | img.set({ 422 | opacity: 0 423 | }); 424 | }) 425 | if (openpose_editor_canvas.backgroundImage) openpose_editor_canvas.backgroundImage.opacity = 0 426 | openpose_editor_canvas.discardActiveObject(); 427 | openpose_editor_canvas.renderAll() 428 | openpose_editor_elem.toBlob((blob) => { 429 | const a = document.createElement("a"); 430 | a.href = URL.createObjectURL(blob); 431 | a.download = "pose.png"; 432 | a.click(); 433 | URL.revokeObjectURL(a.href); 434 | }); 435 | openpose_editor_canvas.getObjects("image").forEach((img) => { 436 | img.set({ 437 | opacity: 1 438 | }); 439 | }) 440 | if (openpose_editor_canvas.backgroundImage) openpose_editor_canvas.backgroundImage.opacity = 0.5 441 | openpose_editor_canvas.renderAll() 442 | return openpose_editor_canvas 443 | } 444 | 445 | function serializeJSON(){ 446 | const json = JSON.stringify({ 447 | "width": openpose_editor_canvas.width, 448 | "height": openpose_editor_canvas.height, 449 | "keypoints": openpose_editor_canvas.getObjects().filter((item) => { 450 | if (item.type === "circle") return item 451 | }).map((item) => { 452 | return [Math.round(item.left), Math.round(item.top)] 453 | }) 454 | }, null, 4) 455 | return json; 456 | } 457 | 458 | function saveJSON(){ 459 | const json = serializeJSON() 460 | const blob = new Blob([json], { 461 | type: "application/json" 462 | }); 463 | const filename = "pose-" + Date.now().toString() + ".json" 464 | const a = document.createElement("a"); 465 | a.href = URL.createObjectURL(blob); 466 | a.download = filename; 467 | a.click(); 468 | URL.revokeObjectURL(a.href); 469 | } 470 | 471 | async function loadJSON(file){ 472 | const url = await fileToDataUrl(file) 473 | const response = await fetch(url) 474 | const json = await response.json() 475 | var w = 0, h = 0; 476 | if (json["canvas_width"] && json["canvas_height"]) { 477 | // new format 478 | resizeCanvas(json["canvas_width"], json["canvas_height"]) 479 | w = json["canvas_width"]; 480 | h = json["canvas_height"]; 481 | }else if (json["width"] && json["height"]) { 482 | resizeCanvas(json["width"], json["height"]) 483 | w = json["width"]; 484 | h = json["height"]; 485 | }else{ 486 | throw new Error('width, height is invalid'); 487 | } 488 | if (json["people"] && json["people"][0]["pose_keypoints_2d"]) { 489 | setPoseV2(json["people"], w, h) 490 | }else if (json["keypoints"].length % 18 === 0) { 491 | setPose(json["keypoints"]) 492 | }else{ 493 | throw new Error('keypoints is invalid') 494 | } 495 | return [w, h] 496 | } 497 | 498 | function savePreset(){ 499 | var name = prompt("Preset Name") 500 | const json = serializeJSON() 501 | return [name, json] 502 | } 503 | 504 | function loadPreset(json){ 505 | try { 506 | json = JSON.parse(json) 507 | if (json["width"] && json["height"]) { 508 | resizeCanvas(json["width"], json["height"]) 509 | }else{ 510 | throw new Error('width, height is invalid'); 511 | } 512 | if (json["keypoints"].length % 18 === 0) { 513 | setPose(json["keypoints"]) 514 | }else{ 515 | throw new Error('keypoints is invalid') 516 | } 517 | return [json["width"], json["height"]] 518 | }catch(e){ 519 | console.error(e) 520 | alert("Invalid JSON") 521 | } 522 | } 523 | 524 | async function addBackground(file){ 525 | const url = await fileToDataUrl(file) 526 | openpose_editor_canvas.setBackgroundImage(url, openpose_editor_canvas.renderAll.bind(openpose_editor_canvas), { 527 | opacity: 0.5 528 | }); 529 | const img = new Image(); 530 | await (img.src = url); 531 | resizeCanvas(img.width, img.height) 532 | return [img.width, img.height] 533 | } 534 | 535 | function detectImage(raw){ 536 | const json = JSON.parse(raw) 537 | 538 | let candidate = json["candidate"] 539 | let subset = json["subset"] 540 | const subsets = [] 541 | 542 | for (i=0; i < subset.length; i+= 20) { 543 | const sub = subset.slice(i, i + 20); 544 | subsets.push(sub); 545 | } 546 | 547 | let clear = true; 548 | for (subset of subsets) { 549 | detectSubset(candidate, subset, clear); 550 | clear = false; 551 | } 552 | } 553 | 554 | function detectSubset(candidate, subset, clear = true){ 555 | const li = [] 556 | for (i=0; 18 > i; i++){ 557 | if (Number.isInteger(subset[i]) && subset[i] >= 0){ 558 | li.push(candidate[subset[i]]) 559 | }else{ 560 | li.push([-1,-1]) 561 | } 562 | } 563 | if(li.length === 0){ 564 | const bgimage = openpose_editor_canvas.backgroundImage 565 | setPose(li); 566 | openpose_editor_canvas.backgroundImage = bgimage 567 | return; 568 | } 569 | if(li.every(([x,y])=>x===-1&&y===-1)){ 570 | const ra_width = Math.floor(Math.random() * openpose_editor_canvas.width) 571 | const ra_height = Math.floor(Math.random() * openpose_editor_canvas.height) 572 | li[0] = [ra_width, ra_height] 573 | } 574 | default_relative_keypoints = [] 575 | for(i=0;i{ 584 | for(idx=0;idxx===-1&&y===-1)){ 594 | for(i=0;i { 616 | img.set({ 617 | opacity: 0 618 | }); 619 | }) 620 | if (openpose_editor_canvas.backgroundImage) openpose_editor_canvas.backgroundImage.opacity = 0 621 | openpose_editor_canvas.discardActiveObject(); 622 | openpose_editor_canvas.renderAll() 623 | openpose_editor_elem.toBlob((blob) => { 624 | const file = new File(([blob]), "pose.png") 625 | const dt = new DataTransfer(); 626 | dt.items.add(file); 627 | const list = dt.files 628 | const selector = type === "txt2img" ? "#txt2img_script_container" : "#img2img_script_container" 629 | if (type === "txt2img"){ 630 | switch_to_txt2img() 631 | }else if(type === "img2img"){ 632 | switch_to_img2img() 633 | } 634 | 635 | const isNew = window.gradio_config.version.replace("\n", "") >= "3.23.0" 636 | const accordion_selector = isNew ? "#controlnet > .label-wrap > .icon" : "#controlnet .transition" 637 | const accordion = gradioApp().querySelector(selector).querySelector(accordion_selector) 638 | if (isNew ? accordion.style.transform == "rotate(90deg)" : accordion.classList.contains("rotate-90")) { 639 | accordion.click() 640 | } 641 | 642 | let input = gradioApp().querySelector(selector).querySelector("#controlnet").querySelector("input[type='file']"); 643 | 644 | const input_image = (input) =>{ 645 | try { 646 | if(input.previousElementSibling 647 | && input.previousElementSibling.previousElementSibling 648 | && input.previousElementSibling.previousElementSibling.querySelector("button[aria-label='Clear']")) { 649 | input.previousElementSibling.previousElementSibling.querySelector("button[aria-label='Clear']").click() 650 | } 651 | } catch (e) { 652 | console.error(e) 653 | } 654 | input.value = ""; 655 | input.files = list; 656 | const event = new Event('change', { 'bubbles': true, "composed": true }); 657 | input.dispatchEvent(event); 658 | } 659 | 660 | if (input == null){ 661 | const callback = (observer) => { 662 | input = gradioApp().querySelector(selector).querySelector("#controlnet").querySelector("input[type='file']"); 663 | if (input == null) { 664 | console.error('input[type=file] NOT exists') 665 | return 666 | }else{ 667 | input_image(input) 668 | observer.disconnect() 669 | } 670 | } 671 | const observer = new MutationObserver(callback); 672 | observer.observe(gradioApp().querySelector(selector).querySelector("#controlnet"), { childList: true }); 673 | }else{ 674 | input_image(input) 675 | } 676 | }); 677 | openpose_editor_canvas.getObjects("image").forEach((img) => { 678 | img.set({ 679 | opacity: 1 680 | }); 681 | }) 682 | if (openpose_editor_canvas.backgroundImage) openpose_editor_canvas.backgroundImage.opacity = 0.5 683 | openpose_editor_canvas.renderAll() 684 | } 685 | 686 | function canvas_onDragOver(event) { 687 | canvas_drag_overlay = gradioApp().querySelector("#canvas_drag_overlay"); 688 | 689 | if (event.dataTransfer.items[0].type.startsWith("image/")) { 690 | event.preventDefault(); 691 | canvas_drag_overlay.textContent = "Add Background"; 692 | canvas_drag_overlay.style.visibility = "visible"; 693 | } else if (event.dataTransfer.items[0].type == "application/json") { 694 | event.preventDefault(); 695 | canvas_drag_overlay.textContent = "Load JSON"; 696 | canvas_drag_overlay.style.visibility = "visible"; 697 | } 698 | } 699 | 700 | function canvas_onDrop(event) { 701 | canvas_drag_overlay = gradioApp().querySelector("#canvas_drag_overlay"); 702 | 703 | if (event.dataTransfer.items[0].type.startsWith("image/")) { 704 | event.preventDefault(); 705 | input = gradioApp().querySelector("#openpose_bg_button").previousElementSibling; 706 | input.files = event.dataTransfer.files; 707 | const changeEvent = new Event('change', { 'bubbles': true, "composed": true }); 708 | input.dispatchEvent(changeEvent); 709 | canvas_drag_overlay.style.visibility = "hidden"; 710 | } else if (event.dataTransfer.items[0].type == "application/json") { 711 | event.preventDefault(); 712 | input = gradioApp().querySelector("#openpose_json_button").previousElementSibling; 713 | input.files = event.dataTransfer.files; 714 | const changeEvent = new Event('change', { 'bubbles': true, "composed": true }); 715 | input.dispatchEvent(changeEvent); 716 | canvas_drag_overlay.style.visibility = "hidden"; 717 | } 718 | } 719 | 720 | function button_onDragOver(event) { 721 | if (((event.target.id == "openpose_detect_button" || event.target.id == "openpose_bg_button") && event.dataTransfer.items[0].type.startsWith("image/")) || 722 | (event.target.id == "openpose_json_button" && event.dataTransfer.items[0].type == "application/json")) { 723 | event.preventDefault(); 724 | event.target.classList.remove("gr-button-secondary"); 725 | } 726 | } 727 | 728 | function button_onDragLeave(event) { 729 | event.target.classList.add("gr-button-secondary"); 730 | } 731 | 732 | function detect_onDrop(event) { 733 | if (event.dataTransfer.items[0].type.startsWith("image/")) { 734 | event.preventDefault(); 735 | input = event.target.previousElementSibling; 736 | input.files = event.dataTransfer.files; 737 | const changeEvent = new Event('change', { 'bubbles': true, "composed": true }); 738 | input.dispatchEvent(changeEvent); 739 | } 740 | } 741 | 742 | function json_onDrop(event) { 743 | if (event.dataTransfer.items[0].type == "application/json") { 744 | event.preventDefault(); 745 | input = event.target.previousElementSibling; 746 | input.files = event.dataTransfer.files; 747 | const changeEvent = new Event('change', { 'bubbles': true, "composed": true }); 748 | input.dispatchEvent(changeEvent); 749 | } 750 | } 751 | 752 | onUiLoaded(function() { 753 | initCanvas(gradioApp().querySelector('#openpose_editor_canvas')) 754 | 755 | var canvas_drag_overlay = document.createElement("div"); 756 | canvas_drag_overlay.id = "canvas_drag_overlay" 757 | canvas_drag_overlay.style = "pointer-events: none; visibility: hidden; display: flex; align-items: center; justify-content: center; width: 100%; height: 100%; color: white; font-size: 2.5em; font-family: inherit; font-weight: 600; line-height: 100%; background: rgba(0,0,0,0.5); margin: 0.25rem; border-radius: 0.25rem; border: 0.5px solid; position: absolute;" 758 | 759 | var canvas = gradioApp().querySelector("#tab_openpose_editor .canvas-container") 760 | canvas.appendChild(canvas_drag_overlay) 761 | canvas.addEventListener("dragover", canvas_onDragOver); 762 | canvas.addEventListener("dragleave", () => gradioApp().querySelector("#canvas_drag_overlay").style.visibility = "hidden"); 763 | canvas.addEventListener("drop", canvas_onDrop); 764 | 765 | var bg_button = gradioApp().querySelector("#openpose_bg_button") 766 | bg_button.addEventListener("dragover", button_onDragOver); 767 | bg_button.addEventListener("dragleave", button_onDragLeave); 768 | bg_button.addEventListener("drop", canvas_onDrop); 769 | bg_button.addEventListener("drop", event => event.target.classList.add("gr-button-secondary")); 770 | bg_button.classList.add("gr-button-secondary"); 771 | 772 | var detect_button = gradioApp().querySelector("#openpose_detect_button") 773 | detect_button.addEventListener("dragover", button_onDragOver); 774 | detect_button.addEventListener("dragleave", button_onDragLeave); 775 | detect_button.addEventListener("drop", detect_onDrop); 776 | detect_button.classList.add("gr-button-secondary"); 777 | 778 | var json_button = gradioApp().querySelector("#openpose_json_button") 779 | json_button.addEventListener("dragover", button_onDragOver); 780 | json_button.addEventListener("dragleave", button_onDragLeave); 781 | json_button.addEventListener("drop", json_onDrop); 782 | json_button.classList.add("gr-button-secondary"); 783 | 784 | }) 785 | -------------------------------------------------------------------------------- /scripts/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import io 3 | import json 4 | import numpy as np 5 | import cv2 6 | 7 | import gradio as gr 8 | 9 | import modules.scripts as scripts 10 | from modules import script_callbacks 11 | from modules.shared import opts 12 | from modules.paths import models_path 13 | 14 | from basicsr.utils.download_util import load_file_from_url 15 | 16 | from scripts.openpose.body import Body 17 | 18 | from PIL import Image 19 | 20 | body_estimation = None 21 | presets_file = os.path.join(scripts.basedir(), "presets.json") 22 | presets = {} 23 | 24 | try: 25 | with open(presets_file) as file: 26 | presets = json.load(file) 27 | except FileNotFoundError: 28 | pass 29 | 30 | def pil2cv(in_image): 31 | out_image = np.array(in_image, dtype=np.uint8) 32 | 33 | if out_image.shape[2] == 3: 34 | out_image = cv2.cvtColor(out_image, cv2.COLOR_RGB2BGR) 35 | return out_image 36 | 37 | def candidate2li(li): 38 | res = [] 39 | for x, y, *_ in li: 40 | res.append([x, y]) 41 | return res 42 | 43 | def subset2li(li): 44 | res = [] 45 | for r in li: 46 | for c in r: 47 | res.append(c) 48 | return res 49 | 50 | class Script(scripts.Script): 51 | def __init__(self) -> None: 52 | super().__init__() 53 | 54 | def title(self): 55 | return "OpenPose Editor" 56 | 57 | def show(self, is_img2img): 58 | return scripts.AlwaysVisible 59 | 60 | def ui(self, is_img2img): 61 | return () 62 | 63 | def on_ui_tabs(): 64 | with gr.Blocks(analytics_enabled=False) as openpose_editor: 65 | with gr.Row(): 66 | with gr.Column(): 67 | width = gr.Slider(label="width", minimum=64, maximum=2048, value=512, step=64, interactive=True) 68 | height = gr.Slider(label="height", minimum=64, maximum=2048, value=512, step=64, interactive=True) 69 | with gr.Row(): 70 | add = gr.Button(value="Add", variant="primary") 71 | # delete = gr.Button(value="Delete") 72 | with gr.Row(): 73 | reset_btn = gr.Button(value="Reset") 74 | json_input = gr.UploadButton(label="Load from JSON", file_types=[".json"], elem_id="openpose_json_button") 75 | png_input = gr.UploadButton(label="Detect from Image", file_types=["image"], type="bytes", elem_id="openpose_detect_button") 76 | bg_input = gr.UploadButton(label="Add Background Image", file_types=["image"], elem_id="openpose_bg_button") 77 | with gr.Row(): 78 | preset_list = gr.Dropdown(label="Presets", choices=sorted(presets.keys()), interactive=True) 79 | preset_load = gr.Button(value="Load Preset") 80 | preset_save = gr.Button(value="Save Preset") 81 | 82 | with gr.Column(): 83 | # gradioooooo... 84 | canvas = gr.HTML('') 85 | jsonbox = gr.Text(label="json", elem_id="jsonbox", visible=False) 86 | with gr.Row(): 87 | json_output = gr.Button(value="Save JSON") 88 | png_output = gr.Button(value="Save PNG") 89 | send_t2t = gr.Button(value="Send to txt2img") 90 | send_i2i = gr.Button(value="Send to img2img") 91 | control_net_max_models_num = getattr(opts, 'control_net_max_models_num', 0) 92 | select_target_index = gr.Dropdown([str(i) for i in range(control_net_max_models_num)], label="Send to", value="0", interactive=True, visible=(control_net_max_models_num > 1)) 93 | 94 | def estimate(file): 95 | global body_estimation 96 | 97 | if body_estimation is None: 98 | model_path = os.path.join(models_path, "openpose", "body_pose_model.pth") 99 | if not os.path.isfile(model_path): 100 | body_model_path = "https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/body_pose_model.pth" 101 | load_file_from_url(body_model_path, model_dir=os.path.join(models_path, "openpose")) 102 | body_estimation = Body(model_path) 103 | 104 | stream = io.BytesIO(file) 105 | img = Image.open(stream) 106 | candidate, subset = body_estimation(pil2cv(img)) 107 | 108 | result = { 109 | "candidate": candidate2li(candidate), 110 | "subset": subset2li(subset), 111 | } 112 | 113 | return str(result).replace("'", '"') 114 | 115 | def savePreset(name, data): 116 | if name: 117 | presets[name] = json.loads(data) 118 | with open(presets_file, "w") as file: 119 | json.dump(presets, file) 120 | return gr.update(choices=sorted(presets.keys()), value=name), json.dumps(data) 121 | return gr.update(), gr.update() 122 | 123 | dummy_component = gr.Label(visible=False) 124 | preset = gr.Text(visible=False) 125 | width.change(None, [width, height], None, _js="(w, h) => {resizeCanvas(w, h)}") 126 | height.change(None, [width, height], None, _js="(w, h) => {resizeCanvas(w, h)}") 127 | png_output.click(None, [], None, _js="savePNG") 128 | bg_input.upload(None, [], [width, height], _js="() => {addBackground('openpose_bg_button')}") 129 | png_input.upload(estimate, png_input, jsonbox) 130 | png_input.upload(None, [], [width, height], _js="() => {addBackground('openpose_detect_button')}") 131 | add.click(None, [], None, _js="addPose") 132 | send_t2t.click(None, select_target_index, None, _js="(i) => {sendImage('txt2img', i)}") 133 | send_i2i.click(None, select_target_index, None, _js="(i) => {sendImage('img2img', i)}") 134 | reset_btn.click(None, [], None, _js="resetCanvas") 135 | json_input.upload(None, json_input, [width, height], _js="() => {loadJSON('openpose_json_button')}") 136 | json_output.click(None, None, None, _js="saveJSON") 137 | preset_save.click(savePreset, [dummy_component, dummy_component], [preset_list, preset], _js="savePreset") 138 | preset_load.click(None, preset, [width, height], _js="loadPreset") 139 | preset_list.change(lambda selected: json.dumps(presets[selected]), preset_list, preset) 140 | 141 | return [(openpose_editor, "OpenPose Editor", "openpose_editor")] 142 | 143 | script_callbacks.on_ui_tabs(on_ui_tabs) 144 | -------------------------------------------------------------------------------- /scripts/openpose/body.py: -------------------------------------------------------------------------------- 1 | # This code from https://github.com/lllyasviel/ControlNet 2 | 3 | import cv2 4 | import numpy as np 5 | import math 6 | import time 7 | from scipy.ndimage.filters import gaussian_filter 8 | import matplotlib.pyplot as plt 9 | import matplotlib 10 | import torch 11 | from torchvision import transforms 12 | 13 | from . import util 14 | from .model import bodypose_model 15 | 16 | class Body(object): 17 | def __init__(self, model_path): 18 | self.model = bodypose_model() 19 | if torch.cuda.is_available(): 20 | self.model = self.model.cuda() 21 | model_dict = util.transfer(self.model, torch.load(model_path)) 22 | self.model.load_state_dict(model_dict) 23 | self.model.eval() 24 | 25 | def __call__(self, oriImg): 26 | # scale_search = [0.5, 1.0, 1.5, 2.0] 27 | scale_search = [0.5] 28 | boxsize = 368 29 | stride = 8 30 | padValue = 128 31 | threshold1 = 0.1 32 | threshold2 = 0.05 33 | multiplier = [x * boxsize / oriImg.shape[0] for x in scale_search] 34 | heatmap_avg = np.zeros((oriImg.shape[0], oriImg.shape[1], 19)) 35 | paf_avg = np.zeros((oriImg.shape[0], oriImg.shape[1], 38)) 36 | 37 | for m in range(len(multiplier)): 38 | scale = multiplier[m] 39 | imageToTest = cv2.resize(oriImg, (0, 0), fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC) 40 | imageToTest_padded, pad = util.padRightDownCorner(imageToTest, stride, padValue) 41 | im = np.transpose(np.float32(imageToTest_padded[:, :, :, np.newaxis]), (3, 2, 0, 1)) / 256 - 0.5 42 | im = np.ascontiguousarray(im) 43 | 44 | data = torch.from_numpy(im).float() 45 | if torch.cuda.is_available(): 46 | data = data.cuda() 47 | # data = data.permute([2, 0, 1]).unsqueeze(0).float() 48 | with torch.no_grad(): 49 | Mconv7_stage6_L1, Mconv7_stage6_L2 = self.model(data) 50 | Mconv7_stage6_L1 = Mconv7_stage6_L1.cpu().numpy() 51 | Mconv7_stage6_L2 = Mconv7_stage6_L2.cpu().numpy() 52 | 53 | # extract outputs, resize, and remove padding 54 | # heatmap = np.transpose(np.squeeze(net.blobs[output_blobs.keys()[1]].data), (1, 2, 0)) # output 1 is heatmaps 55 | heatmap = np.transpose(np.squeeze(Mconv7_stage6_L2), (1, 2, 0)) # output 1 is heatmaps 56 | heatmap = cv2.resize(heatmap, (0, 0), fx=stride, fy=stride, interpolation=cv2.INTER_CUBIC) 57 | heatmap = heatmap[:imageToTest_padded.shape[0] - pad[2], :imageToTest_padded.shape[1] - pad[3], :] 58 | heatmap = cv2.resize(heatmap, (oriImg.shape[1], oriImg.shape[0]), interpolation=cv2.INTER_CUBIC) 59 | 60 | # paf = np.transpose(np.squeeze(net.blobs[output_blobs.keys()[0]].data), (1, 2, 0)) # output 0 is PAFs 61 | paf = np.transpose(np.squeeze(Mconv7_stage6_L1), (1, 2, 0)) # output 0 is PAFs 62 | paf = cv2.resize(paf, (0, 0), fx=stride, fy=stride, interpolation=cv2.INTER_CUBIC) 63 | paf = paf[:imageToTest_padded.shape[0] - pad[2], :imageToTest_padded.shape[1] - pad[3], :] 64 | paf = cv2.resize(paf, (oriImg.shape[1], oriImg.shape[0]), interpolation=cv2.INTER_CUBIC) 65 | 66 | heatmap_avg += heatmap_avg + heatmap / len(multiplier) 67 | paf_avg += + paf / len(multiplier) 68 | 69 | all_peaks = [] 70 | peak_counter = 0 71 | 72 | for part in range(18): 73 | map_ori = heatmap_avg[:, :, part] 74 | one_heatmap = gaussian_filter(map_ori, sigma=3) 75 | 76 | map_left = np.zeros(one_heatmap.shape) 77 | map_left[1:, :] = one_heatmap[:-1, :] 78 | map_right = np.zeros(one_heatmap.shape) 79 | map_right[:-1, :] = one_heatmap[1:, :] 80 | map_up = np.zeros(one_heatmap.shape) 81 | map_up[:, 1:] = one_heatmap[:, :-1] 82 | map_down = np.zeros(one_heatmap.shape) 83 | map_down[:, :-1] = one_heatmap[:, 1:] 84 | 85 | peaks_binary = np.logical_and.reduce( 86 | (one_heatmap >= map_left, one_heatmap >= map_right, one_heatmap >= map_up, one_heatmap >= map_down, one_heatmap > threshold1)) 87 | peaks = list(zip(np.nonzero(peaks_binary)[1], np.nonzero(peaks_binary)[0])) # note reverse 88 | peaks_with_score = [x + (map_ori[x[1], x[0]],) for x in peaks] 89 | peak_id = range(peak_counter, peak_counter + len(peaks)) 90 | peaks_with_score_and_id = [peaks_with_score[i] + (peak_id[i],) for i in range(len(peak_id))] 91 | 92 | all_peaks.append(peaks_with_score_and_id) 93 | peak_counter += len(peaks) 94 | 95 | # find connection in the specified sequence, center 29 is in the position 15 96 | limbSeq = [[2, 3], [2, 6], [3, 4], [4, 5], [6, 7], [7, 8], [2, 9], [9, 10], \ 97 | [10, 11], [2, 12], [12, 13], [13, 14], [2, 1], [1, 15], [15, 17], \ 98 | [1, 16], [16, 18], [3, 17], [6, 18]] 99 | # the middle joints heatmap correpondence 100 | mapIdx = [[31, 32], [39, 40], [33, 34], [35, 36], [41, 42], [43, 44], [19, 20], [21, 22], \ 101 | [23, 24], [25, 26], [27, 28], [29, 30], [47, 48], [49, 50], [53, 54], [51, 52], \ 102 | [55, 56], [37, 38], [45, 46]] 103 | 104 | connection_all = [] 105 | special_k = [] 106 | mid_num = 10 107 | 108 | for k in range(len(mapIdx)): 109 | score_mid = paf_avg[:, :, [x - 19 for x in mapIdx[k]]] 110 | candA = all_peaks[limbSeq[k][0] - 1] 111 | candB = all_peaks[limbSeq[k][1] - 1] 112 | nA = len(candA) 113 | nB = len(candB) 114 | indexA, indexB = limbSeq[k] 115 | if (nA != 0 and nB != 0): 116 | connection_candidate = [] 117 | for i in range(nA): 118 | for j in range(nB): 119 | vec = np.subtract(candB[j][:2], candA[i][:2]) 120 | norm = math.sqrt(vec[0] * vec[0] + vec[1] * vec[1]) 121 | norm = max(0.001, norm) 122 | vec = np.divide(vec, norm) 123 | 124 | startend = list(zip(np.linspace(candA[i][0], candB[j][0], num=mid_num), \ 125 | np.linspace(candA[i][1], candB[j][1], num=mid_num))) 126 | 127 | vec_x = np.array([score_mid[int(round(startend[I][1])), int(round(startend[I][0])), 0] \ 128 | for I in range(len(startend))]) 129 | vec_y = np.array([score_mid[int(round(startend[I][1])), int(round(startend[I][0])), 1] \ 130 | for I in range(len(startend))]) 131 | 132 | score_midpts = np.multiply(vec_x, vec[0]) + np.multiply(vec_y, vec[1]) 133 | score_with_dist_prior = sum(score_midpts) / len(score_midpts) + min( 134 | 0.5 * oriImg.shape[0] / norm - 1, 0) 135 | criterion1 = len(np.nonzero(score_midpts > threshold2)[0]) > 0.8 * len(score_midpts) 136 | criterion2 = score_with_dist_prior > 0 137 | if criterion1 and criterion2: 138 | connection_candidate.append( 139 | [i, j, score_with_dist_prior, score_with_dist_prior + candA[i][2] + candB[j][2]]) 140 | 141 | connection_candidate = sorted(connection_candidate, key=lambda x: x[2], reverse=True) 142 | connection = np.zeros((0, 5)) 143 | for c in range(len(connection_candidate)): 144 | i, j, s = connection_candidate[c][0:3] 145 | if (i not in connection[:, 3] and j not in connection[:, 4]): 146 | connection = np.vstack([connection, [candA[i][3], candB[j][3], s, i, j]]) 147 | if (len(connection) >= min(nA, nB)): 148 | break 149 | 150 | connection_all.append(connection) 151 | else: 152 | special_k.append(k) 153 | connection_all.append([]) 154 | 155 | # last number in each row is the total parts number of that person 156 | # the second last number in each row is the score of the overall configuration 157 | subset = -1 * np.ones((0, 20)) 158 | candidate = np.array([item for sublist in all_peaks for item in sublist]) 159 | 160 | for k in range(len(mapIdx)): 161 | if k not in special_k: 162 | partAs = connection_all[k][:, 0] 163 | partBs = connection_all[k][:, 1] 164 | indexA, indexB = np.array(limbSeq[k]) - 1 165 | 166 | for i in range(len(connection_all[k])): # = 1:size(temp,1) 167 | found = 0 168 | subset_idx = [-1, -1] 169 | for j in range(len(subset)): # 1:size(subset,1): 170 | if subset[j][indexA] == partAs[i] or subset[j][indexB] == partBs[i]: 171 | subset_idx[found] = j 172 | found += 1 173 | 174 | if found == 1: 175 | j = subset_idx[0] 176 | if subset[j][indexB] != partBs[i]: 177 | subset[j][indexB] = partBs[i] 178 | subset[j][-1] += 1 179 | subset[j][-2] += candidate[partBs[i].astype(int), 2] + connection_all[k][i][2] 180 | elif found == 2: # if found 2 and disjoint, merge them 181 | j1, j2 = subset_idx 182 | membership = ((subset[j1] >= 0).astype(int) + (subset[j2] >= 0).astype(int))[:-2] 183 | if len(np.nonzero(membership == 2)[0]) == 0: # merge 184 | subset[j1][:-2] += (subset[j2][:-2] + 1) 185 | subset[j1][-2:] += subset[j2][-2:] 186 | subset[j1][-2] += connection_all[k][i][2] 187 | subset = np.delete(subset, j2, 0) 188 | else: # as like found == 1 189 | subset[j1][indexB] = partBs[i] 190 | subset[j1][-1] += 1 191 | subset[j1][-2] += candidate[partBs[i].astype(int), 2] + connection_all[k][i][2] 192 | 193 | # if find no partA in the subset, create a new subset 194 | elif not found and k < 17: 195 | row = -1 * np.ones(20) 196 | row[indexA] = partAs[i] 197 | row[indexB] = partBs[i] 198 | row[-1] = 2 199 | row[-2] = sum(candidate[connection_all[k][i, :2].astype(int), 2]) + connection_all[k][i][2] 200 | subset = np.vstack([subset, row]) 201 | # delete some rows of subset which has few parts occur 202 | deleteIdx = [] 203 | for i in range(len(subset)): 204 | if subset[i][-1] < 4 or subset[i][-2] / subset[i][-1] < 0.4: 205 | deleteIdx.append(i) 206 | subset = np.delete(subset, deleteIdx, axis=0) 207 | 208 | # subset: n*20 array, 0-17 is the index in candidate, 18 is the total score, 19 is the total parts 209 | # candidate: x, y, score, id 210 | return candidate, subset 211 | 212 | if __name__ == "__main__": 213 | body_estimation = Body('../model/body_pose_model.pth') 214 | 215 | test_image = '../images/ski.jpg' 216 | oriImg = cv2.imread(test_image) # B,G,R order 217 | candidate, subset = body_estimation(oriImg) 218 | canvas = util.draw_bodypose(oriImg, candidate, subset) 219 | plt.imshow(canvas[:, :, [2, 1, 0]]) 220 | plt.show() 221 | -------------------------------------------------------------------------------- /scripts/openpose/model.py: -------------------------------------------------------------------------------- 1 | # This code from https://github.com/lllyasviel/ControlNet 2 | 3 | import torch 4 | from collections import OrderedDict 5 | 6 | import torch 7 | import torch.nn as nn 8 | 9 | def make_layers(block, no_relu_layers): 10 | layers = [] 11 | for layer_name, v in block.items(): 12 | if 'pool' in layer_name: 13 | layer = nn.MaxPool2d(kernel_size=v[0], stride=v[1], 14 | padding=v[2]) 15 | layers.append((layer_name, layer)) 16 | else: 17 | conv2d = nn.Conv2d(in_channels=v[0], out_channels=v[1], 18 | kernel_size=v[2], stride=v[3], 19 | padding=v[4]) 20 | layers.append((layer_name, conv2d)) 21 | if layer_name not in no_relu_layers: 22 | layers.append(('relu_'+layer_name, nn.ReLU(inplace=True))) 23 | 24 | return nn.Sequential(OrderedDict(layers)) 25 | 26 | class bodypose_model(nn.Module): 27 | def __init__(self): 28 | super(bodypose_model, self).__init__() 29 | 30 | # these layers have no relu layer 31 | no_relu_layers = ['conv5_5_CPM_L1', 'conv5_5_CPM_L2', 'Mconv7_stage2_L1',\ 32 | 'Mconv7_stage2_L2', 'Mconv7_stage3_L1', 'Mconv7_stage3_L2',\ 33 | 'Mconv7_stage4_L1', 'Mconv7_stage4_L2', 'Mconv7_stage5_L1',\ 34 | 'Mconv7_stage5_L2', 'Mconv7_stage6_L1', 'Mconv7_stage6_L1'] 35 | blocks = {} 36 | block0 = OrderedDict([ 37 | ('conv1_1', [3, 64, 3, 1, 1]), 38 | ('conv1_2', [64, 64, 3, 1, 1]), 39 | ('pool1_stage1', [2, 2, 0]), 40 | ('conv2_1', [64, 128, 3, 1, 1]), 41 | ('conv2_2', [128, 128, 3, 1, 1]), 42 | ('pool2_stage1', [2, 2, 0]), 43 | ('conv3_1', [128, 256, 3, 1, 1]), 44 | ('conv3_2', [256, 256, 3, 1, 1]), 45 | ('conv3_3', [256, 256, 3, 1, 1]), 46 | ('conv3_4', [256, 256, 3, 1, 1]), 47 | ('pool3_stage1', [2, 2, 0]), 48 | ('conv4_1', [256, 512, 3, 1, 1]), 49 | ('conv4_2', [512, 512, 3, 1, 1]), 50 | ('conv4_3_CPM', [512, 256, 3, 1, 1]), 51 | ('conv4_4_CPM', [256, 128, 3, 1, 1]) 52 | ]) 53 | 54 | 55 | # Stage 1 56 | block1_1 = OrderedDict([ 57 | ('conv5_1_CPM_L1', [128, 128, 3, 1, 1]), 58 | ('conv5_2_CPM_L1', [128, 128, 3, 1, 1]), 59 | ('conv5_3_CPM_L1', [128, 128, 3, 1, 1]), 60 | ('conv5_4_CPM_L1', [128, 512, 1, 1, 0]), 61 | ('conv5_5_CPM_L1', [512, 38, 1, 1, 0]) 62 | ]) 63 | 64 | block1_2 = OrderedDict([ 65 | ('conv5_1_CPM_L2', [128, 128, 3, 1, 1]), 66 | ('conv5_2_CPM_L2', [128, 128, 3, 1, 1]), 67 | ('conv5_3_CPM_L2', [128, 128, 3, 1, 1]), 68 | ('conv5_4_CPM_L2', [128, 512, 1, 1, 0]), 69 | ('conv5_5_CPM_L2', [512, 19, 1, 1, 0]) 70 | ]) 71 | blocks['block1_1'] = block1_1 72 | blocks['block1_2'] = block1_2 73 | 74 | self.model0 = make_layers(block0, no_relu_layers) 75 | 76 | # Stages 2 - 6 77 | for i in range(2, 7): 78 | blocks['block%d_1' % i] = OrderedDict([ 79 | ('Mconv1_stage%d_L1' % i, [185, 128, 7, 1, 3]), 80 | ('Mconv2_stage%d_L1' % i, [128, 128, 7, 1, 3]), 81 | ('Mconv3_stage%d_L1' % i, [128, 128, 7, 1, 3]), 82 | ('Mconv4_stage%d_L1' % i, [128, 128, 7, 1, 3]), 83 | ('Mconv5_stage%d_L1' % i, [128, 128, 7, 1, 3]), 84 | ('Mconv6_stage%d_L1' % i, [128, 128, 1, 1, 0]), 85 | ('Mconv7_stage%d_L1' % i, [128, 38, 1, 1, 0]) 86 | ]) 87 | 88 | blocks['block%d_2' % i] = OrderedDict([ 89 | ('Mconv1_stage%d_L2' % i, [185, 128, 7, 1, 3]), 90 | ('Mconv2_stage%d_L2' % i, [128, 128, 7, 1, 3]), 91 | ('Mconv3_stage%d_L2' % i, [128, 128, 7, 1, 3]), 92 | ('Mconv4_stage%d_L2' % i, [128, 128, 7, 1, 3]), 93 | ('Mconv5_stage%d_L2' % i, [128, 128, 7, 1, 3]), 94 | ('Mconv6_stage%d_L2' % i, [128, 128, 1, 1, 0]), 95 | ('Mconv7_stage%d_L2' % i, [128, 19, 1, 1, 0]) 96 | ]) 97 | 98 | for k in blocks.keys(): 99 | blocks[k] = make_layers(blocks[k], no_relu_layers) 100 | 101 | self.model1_1 = blocks['block1_1'] 102 | self.model2_1 = blocks['block2_1'] 103 | self.model3_1 = blocks['block3_1'] 104 | self.model4_1 = blocks['block4_1'] 105 | self.model5_1 = blocks['block5_1'] 106 | self.model6_1 = blocks['block6_1'] 107 | 108 | self.model1_2 = blocks['block1_2'] 109 | self.model2_2 = blocks['block2_2'] 110 | self.model3_2 = blocks['block3_2'] 111 | self.model4_2 = blocks['block4_2'] 112 | self.model5_2 = blocks['block5_2'] 113 | self.model6_2 = blocks['block6_2'] 114 | 115 | 116 | def forward(self, x): 117 | 118 | out1 = self.model0(x) 119 | 120 | out1_1 = self.model1_1(out1) 121 | out1_2 = self.model1_2(out1) 122 | out2 = torch.cat([out1_1, out1_2, out1], 1) 123 | 124 | out2_1 = self.model2_1(out2) 125 | out2_2 = self.model2_2(out2) 126 | out3 = torch.cat([out2_1, out2_2, out1], 1) 127 | 128 | out3_1 = self.model3_1(out3) 129 | out3_2 = self.model3_2(out3) 130 | out4 = torch.cat([out3_1, out3_2, out1], 1) 131 | 132 | out4_1 = self.model4_1(out4) 133 | out4_2 = self.model4_2(out4) 134 | out5 = torch.cat([out4_1, out4_2, out1], 1) 135 | 136 | out5_1 = self.model5_1(out5) 137 | out5_2 = self.model5_2(out5) 138 | out6 = torch.cat([out5_1, out5_2, out1], 1) 139 | 140 | out6_1 = self.model6_1(out6) 141 | out6_2 = self.model6_2(out6) 142 | 143 | return out6_1, out6_2 144 | 145 | class handpose_model(nn.Module): 146 | def __init__(self): 147 | super(handpose_model, self).__init__() 148 | 149 | # these layers have no relu layer 150 | no_relu_layers = ['conv6_2_CPM', 'Mconv7_stage2', 'Mconv7_stage3',\ 151 | 'Mconv7_stage4', 'Mconv7_stage5', 'Mconv7_stage6'] 152 | # stage 1 153 | block1_0 = OrderedDict([ 154 | ('conv1_1', [3, 64, 3, 1, 1]), 155 | ('conv1_2', [64, 64, 3, 1, 1]), 156 | ('pool1_stage1', [2, 2, 0]), 157 | ('conv2_1', [64, 128, 3, 1, 1]), 158 | ('conv2_2', [128, 128, 3, 1, 1]), 159 | ('pool2_stage1', [2, 2, 0]), 160 | ('conv3_1', [128, 256, 3, 1, 1]), 161 | ('conv3_2', [256, 256, 3, 1, 1]), 162 | ('conv3_3', [256, 256, 3, 1, 1]), 163 | ('conv3_4', [256, 256, 3, 1, 1]), 164 | ('pool3_stage1', [2, 2, 0]), 165 | ('conv4_1', [256, 512, 3, 1, 1]), 166 | ('conv4_2', [512, 512, 3, 1, 1]), 167 | ('conv4_3', [512, 512, 3, 1, 1]), 168 | ('conv4_4', [512, 512, 3, 1, 1]), 169 | ('conv5_1', [512, 512, 3, 1, 1]), 170 | ('conv5_2', [512, 512, 3, 1, 1]), 171 | ('conv5_3_CPM', [512, 128, 3, 1, 1]) 172 | ]) 173 | 174 | block1_1 = OrderedDict([ 175 | ('conv6_1_CPM', [128, 512, 1, 1, 0]), 176 | ('conv6_2_CPM', [512, 22, 1, 1, 0]) 177 | ]) 178 | 179 | blocks = {} 180 | blocks['block1_0'] = block1_0 181 | blocks['block1_1'] = block1_1 182 | 183 | # stage 2-6 184 | for i in range(2, 7): 185 | blocks['block%d' % i] = OrderedDict([ 186 | ('Mconv1_stage%d' % i, [150, 128, 7, 1, 3]), 187 | ('Mconv2_stage%d' % i, [128, 128, 7, 1, 3]), 188 | ('Mconv3_stage%d' % i, [128, 128, 7, 1, 3]), 189 | ('Mconv4_stage%d' % i, [128, 128, 7, 1, 3]), 190 | ('Mconv5_stage%d' % i, [128, 128, 7, 1, 3]), 191 | ('Mconv6_stage%d' % i, [128, 128, 1, 1, 0]), 192 | ('Mconv7_stage%d' % i, [128, 22, 1, 1, 0]) 193 | ]) 194 | 195 | for k in blocks.keys(): 196 | blocks[k] = make_layers(blocks[k], no_relu_layers) 197 | 198 | self.model1_0 = blocks['block1_0'] 199 | self.model1_1 = blocks['block1_1'] 200 | self.model2 = blocks['block2'] 201 | self.model3 = blocks['block3'] 202 | self.model4 = blocks['block4'] 203 | self.model5 = blocks['block5'] 204 | self.model6 = blocks['block6'] 205 | 206 | def forward(self, x): 207 | out1_0 = self.model1_0(x) 208 | out1_1 = self.model1_1(out1_0) 209 | concat_stage2 = torch.cat([out1_1, out1_0], 1) 210 | out_stage2 = self.model2(concat_stage2) 211 | concat_stage3 = torch.cat([out_stage2, out1_0], 1) 212 | out_stage3 = self.model3(concat_stage3) 213 | concat_stage4 = torch.cat([out_stage3, out1_0], 1) 214 | out_stage4 = self.model4(concat_stage4) 215 | concat_stage5 = torch.cat([out_stage4, out1_0], 1) 216 | out_stage5 = self.model5(concat_stage5) 217 | concat_stage6 = torch.cat([out_stage5, out1_0], 1) 218 | out_stage6 = self.model6(concat_stage6) 219 | return out_stage6 220 | 221 | 222 | -------------------------------------------------------------------------------- /scripts/openpose/util.py: -------------------------------------------------------------------------------- 1 | # This code from https://github.com/lllyasviel/ControlNet 2 | 3 | import math 4 | import numpy as np 5 | import matplotlib 6 | import cv2 7 | 8 | 9 | def padRightDownCorner(img, stride, padValue): 10 | h = img.shape[0] 11 | w = img.shape[1] 12 | 13 | pad = 4 * [None] 14 | pad[0] = 0 # up 15 | pad[1] = 0 # left 16 | pad[2] = 0 if (h % stride == 0) else stride - (h % stride) # down 17 | pad[3] = 0 if (w % stride == 0) else stride - (w % stride) # right 18 | 19 | img_padded = img 20 | pad_up = np.tile(img_padded[0:1, :, :]*0 + padValue, (pad[0], 1, 1)) 21 | img_padded = np.concatenate((pad_up, img_padded), axis=0) 22 | pad_left = np.tile(img_padded[:, 0:1, :]*0 + padValue, (1, pad[1], 1)) 23 | img_padded = np.concatenate((pad_left, img_padded), axis=1) 24 | pad_down = np.tile(img_padded[-2:-1, :, :]*0 + padValue, (pad[2], 1, 1)) 25 | img_padded = np.concatenate((img_padded, pad_down), axis=0) 26 | pad_right = np.tile(img_padded[:, -2:-1, :]*0 + padValue, (1, pad[3], 1)) 27 | img_padded = np.concatenate((img_padded, pad_right), axis=1) 28 | 29 | return img_padded, pad 30 | 31 | # transfer caffe model to pytorch which will match the layer name 32 | def transfer(model, model_weights): 33 | transferred_model_weights = {} 34 | for weights_name in model.state_dict().keys(): 35 | transferred_model_weights[weights_name] = model_weights['.'.join(weights_name.split('.')[1:])] 36 | return transferred_model_weights 37 | 38 | # draw the body keypoint and lims 39 | def draw_bodypose(canvas, candidate, subset): 40 | stickwidth = 4 41 | limbSeq = [[2, 3], [2, 6], [3, 4], [4, 5], [6, 7], [7, 8], [2, 9], [9, 10], \ 42 | [10, 11], [2, 12], [12, 13], [13, 14], [2, 1], [1, 15], [15, 17], \ 43 | [1, 16], [16, 18], [3, 17], [6, 18]] 44 | 45 | colors = [[255, 0, 0], [255, 85, 0], [255, 170, 0], [255, 255, 0], [170, 255, 0], [85, 255, 0], [0, 255, 0], \ 46 | [0, 255, 85], [0, 255, 170], [0, 255, 255], [0, 170, 255], [0, 85, 255], [0, 0, 255], [85, 0, 255], \ 47 | [170, 0, 255], [255, 0, 255], [255, 0, 170], [255, 0, 85]] 48 | for i in range(18): 49 | for n in range(len(subset)): 50 | index = int(subset[n][i]) 51 | if index == -1: 52 | continue 53 | x, y = candidate[index][0:2] 54 | cv2.circle(canvas, (int(x), int(y)), 4, colors[i], thickness=-1) 55 | for i in range(17): 56 | for n in range(len(subset)): 57 | index = subset[n][np.array(limbSeq[i]) - 1] 58 | if -1 in index: 59 | continue 60 | cur_canvas = canvas.copy() 61 | Y = candidate[index.astype(int), 0] 62 | X = candidate[index.astype(int), 1] 63 | mX = np.mean(X) 64 | mY = np.mean(Y) 65 | length = ((X[0] - X[1]) ** 2 + (Y[0] - Y[1]) ** 2) ** 0.5 66 | angle = math.degrees(math.atan2(X[0] - X[1], Y[0] - Y[1])) 67 | polygon = cv2.ellipse2Poly((int(mY), int(mX)), (int(length / 2), stickwidth), int(angle), 0, 360, 1) 68 | cv2.fillConvexPoly(cur_canvas, polygon, colors[i]) 69 | canvas = cv2.addWeighted(canvas, 0.4, cur_canvas, 0.6, 0) 70 | # plt.imsave("preview.jpg", canvas[:, :, [2, 1, 0]]) 71 | # plt.imshow(canvas[:, :, [2, 1, 0]]) 72 | return canvas 73 | 74 | 75 | # image drawn by opencv is not good. 76 | def draw_handpose(canvas, all_hand_peaks, show_number=False): 77 | edges = [[0, 1], [1, 2], [2, 3], [3, 4], [0, 5], [5, 6], [6, 7], [7, 8], [0, 9], [9, 10], \ 78 | [10, 11], [11, 12], [0, 13], [13, 14], [14, 15], [15, 16], [0, 17], [17, 18], [18, 19], [19, 20]] 79 | 80 | for peaks in all_hand_peaks: 81 | for ie, e in enumerate(edges): 82 | if np.sum(np.all(peaks[e], axis=1)==0)==0: 83 | x1, y1 = peaks[e[0]] 84 | x2, y2 = peaks[e[1]] 85 | cv2.line(canvas, (x1, y1), (x2, y2), matplotlib.colors.hsv_to_rgb([ie/float(len(edges)), 1.0, 1.0])*255, thickness=2) 86 | 87 | for i, keyponit in enumerate(peaks): 88 | x, y = keyponit 89 | cv2.circle(canvas, (x, y), 4, (0, 0, 255), thickness=-1) 90 | if show_number: 91 | cv2.putText(canvas, str(i), (x, y), cv2.FONT_HERSHEY_SIMPLEX, 0.3, (0, 0, 0), lineType=cv2.LINE_AA) 92 | return canvas 93 | 94 | # detect hand according to body pose keypoints 95 | # please refer to https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/src/openpose/hand/handDetector.cpp 96 | def handDetect(candidate, subset, oriImg): 97 | # right hand: wrist 4, elbow 3, shoulder 2 98 | # left hand: wrist 7, elbow 6, shoulder 5 99 | ratioWristElbow = 0.33 100 | detect_result = [] 101 | image_height, image_width = oriImg.shape[0:2] 102 | for person in subset.astype(int): 103 | # if any of three not detected 104 | has_left = np.sum(person[[5, 6, 7]] == -1) == 0 105 | has_right = np.sum(person[[2, 3, 4]] == -1) == 0 106 | if not (has_left or has_right): 107 | continue 108 | hands = [] 109 | #left hand 110 | if has_left: 111 | left_shoulder_index, left_elbow_index, left_wrist_index = person[[5, 6, 7]] 112 | x1, y1 = candidate[left_shoulder_index][:2] 113 | x2, y2 = candidate[left_elbow_index][:2] 114 | x3, y3 = candidate[left_wrist_index][:2] 115 | hands.append([x1, y1, x2, y2, x3, y3, True]) 116 | # right hand 117 | if has_right: 118 | right_shoulder_index, right_elbow_index, right_wrist_index = person[[2, 3, 4]] 119 | x1, y1 = candidate[right_shoulder_index][:2] 120 | x2, y2 = candidate[right_elbow_index][:2] 121 | x3, y3 = candidate[right_wrist_index][:2] 122 | hands.append([x1, y1, x2, y2, x3, y3, False]) 123 | 124 | for x1, y1, x2, y2, x3, y3, is_left in hands: 125 | # pos_hand = pos_wrist + ratio * (pos_wrist - pos_elbox) = (1 + ratio) * pos_wrist - ratio * pos_elbox 126 | # handRectangle.x = posePtr[wrist*3] + ratioWristElbow * (posePtr[wrist*3] - posePtr[elbow*3]); 127 | # handRectangle.y = posePtr[wrist*3+1] + ratioWristElbow * (posePtr[wrist*3+1] - posePtr[elbow*3+1]); 128 | # const auto distanceWristElbow = getDistance(poseKeypoints, person, wrist, elbow); 129 | # const auto distanceElbowShoulder = getDistance(poseKeypoints, person, elbow, shoulder); 130 | # handRectangle.width = 1.5f * fastMax(distanceWristElbow, 0.9f * distanceElbowShoulder); 131 | x = x3 + ratioWristElbow * (x3 - x2) 132 | y = y3 + ratioWristElbow * (y3 - y2) 133 | distanceWristElbow = math.sqrt((x3 - x2) ** 2 + (y3 - y2) ** 2) 134 | distanceElbowShoulder = math.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2) 135 | width = 1.5 * max(distanceWristElbow, 0.9 * distanceElbowShoulder) 136 | # x-y refers to the center --> offset to topLeft point 137 | # handRectangle.x -= handRectangle.width / 2.f; 138 | # handRectangle.y -= handRectangle.height / 2.f; 139 | x -= width / 2 140 | y -= width / 2 # width = height 141 | # overflow the image 142 | if x < 0: x = 0 143 | if y < 0: y = 0 144 | width1 = width 145 | width2 = width 146 | if x + width > image_width: width1 = image_width - x 147 | if y + width > image_height: width2 = image_height - y 148 | width = min(width1, width2) 149 | # the max hand box value is 20 pixels 150 | if width >= 20: 151 | detect_result.append([int(x), int(y), int(width), is_left]) 152 | 153 | ''' 154 | return value: [[x, y, w, True if left hand else False]]. 155 | width=height since the network require squared input. 156 | x, y is the coordinate of top left 157 | ''' 158 | return detect_result 159 | 160 | # get max index of 2d array 161 | def npmax(array): 162 | arrayindex = array.argmax(1) 163 | arrayvalue = array.max(1) 164 | i = arrayvalue.argmax() 165 | j = arrayindex[i] 166 | return i, j 167 | --------------------------------------------------------------------------------