├── Matlab ├── NTU_RGBD120_samples_with_missing_skeletons.txt ├── NTU_RGBD_samples_with_missing_skeletons.txt ├── read_skeleton_file.m ├── show_skeleton_on_IR_frames.m ├── show_skeleton_on_RGB_frames.m └── show_skeleton_on_depthmaps.m ├── Python ├── .gitignore ├── readme.md └── txt2npy.py └── README.md /Matlab/NTU_RGBD120_samples_with_missing_skeletons.txt: -------------------------------------------------------------------------------- 1 | 535 of the captured samples in the "NTU RGB+D 120" dataset have missing or incomplete skeleton data. 2 | If you are working on skeleton-based analysis, please ignore these files in your training and testing procedures. 3 | 4 | S001C002P005R002A008 5 | S001C002P006R001A008 6 | S001C003P002R001A055 7 | S001C003P002R002A012 8 | S001C003P005R002A004 9 | S001C003P005R002A005 10 | S001C003P005R002A006 11 | S001C003P006R002A008 12 | S002C002P011R002A030 13 | S002C003P008R001A020 14 | S002C003P010R002A010 15 | S002C003P011R002A007 16 | S002C003P011R002A011 17 | S002C003P014R002A007 18 | S003C001P019R001A055 19 | S003C002P002R002A055 20 | S003C002P018R002A055 21 | S003C003P002R001A055 22 | S003C003P016R001A055 23 | S003C003P018R002A024 24 | S004C002P003R001A013 25 | S004C002P008R001A009 26 | S004C002P020R001A003 27 | S004C002P020R001A004 28 | S004C002P020R001A012 29 | S004C002P020R001A020 30 | S004C002P020R001A021 31 | S004C002P020R001A036 32 | S005C002P004R001A001 33 | S005C002P004R001A003 34 | S005C002P010R001A016 35 | S005C002P010R001A017 36 | S005C002P010R001A048 37 | S005C002P010R001A049 38 | S005C002P016R001A009 39 | S005C002P016R001A010 40 | S005C002P018R001A003 41 | S005C002P018R001A028 42 | S005C002P018R001A029 43 | S005C003P016R002A009 44 | S005C003P018R002A013 45 | S005C003P021R002A057 46 | S006C001P001R002A055 47 | S006C002P007R001A005 48 | S006C002P007R001A006 49 | S006C002P016R001A043 50 | S006C002P016R001A051 51 | S006C002P016R001A052 52 | S006C002P022R001A012 53 | S006C002P023R001A020 54 | S006C002P023R001A021 55 | S006C002P023R001A022 56 | S006C002P023R001A023 57 | S006C002P024R001A018 58 | S006C002P024R001A019 59 | S006C003P001R002A013 60 | S006C003P007R002A009 61 | S006C003P007R002A010 62 | S006C003P007R002A025 63 | S006C003P016R001A060 64 | S006C003P017R001A055 65 | S006C003P017R002A013 66 | S006C003P017R002A014 67 | S006C003P017R002A015 68 | S006C003P022R002A013 69 | S007C001P018R002A050 70 | S007C001P025R002A051 71 | S007C001P028R001A050 72 | S007C001P028R001A051 73 | S007C001P028R001A052 74 | S007C002P008R002A008 75 | S007C002P015R002A055 76 | S007C002P026R001A008 77 | S007C002P026R001A009 78 | S007C002P026R001A010 79 | S007C002P026R001A011 80 | S007C002P026R001A012 81 | S007C002P026R001A050 82 | S007C002P027R001A011 83 | S007C002P027R001A013 84 | S007C002P028R002A055 85 | S007C003P007R001A002 86 | S007C003P007R001A004 87 | S007C003P019R001A060 88 | S007C003P027R002A001 89 | S007C003P027R002A002 90 | S007C003P027R002A003 91 | S007C003P027R002A004 92 | S007C003P027R002A005 93 | S007C003P027R002A006 94 | S007C003P027R002A007 95 | S007C003P027R002A008 96 | S007C003P027R002A009 97 | S007C003P027R002A010 98 | S007C003P027R002A011 99 | S007C003P027R002A012 100 | S007C003P027R002A013 101 | S008C002P001R001A009 102 | S008C002P001R001A010 103 | S008C002P001R001A014 104 | S008C002P001R001A015 105 | S008C002P001R001A016 106 | S008C002P001R001A018 107 | S008C002P001R001A019 108 | S008C002P008R002A059 109 | S008C002P025R001A060 110 | S008C002P029R001A004 111 | S008C002P031R001A005 112 | S008C002P031R001A006 113 | S008C002P032R001A018 114 | S008C002P034R001A018 115 | S008C002P034R001A019 116 | S008C002P035R001A059 117 | S008C002P035R002A002 118 | S008C002P035R002A005 119 | S008C003P007R001A009 120 | S008C003P007R001A016 121 | S008C003P007R001A017 122 | S008C003P007R001A018 123 | S008C003P007R001A019 124 | S008C003P007R001A020 125 | S008C003P007R001A021 126 | S008C003P007R001A022 127 | S008C003P007R001A023 128 | S008C003P007R001A025 129 | S008C003P007R001A026 130 | S008C003P007R001A028 131 | S008C003P007R001A029 132 | S008C003P007R002A003 133 | S008C003P008R002A050 134 | S008C003P025R002A002 135 | S008C003P025R002A011 136 | S008C003P025R002A012 137 | S008C003P025R002A016 138 | S008C003P025R002A020 139 | S008C003P025R002A022 140 | S008C003P025R002A023 141 | S008C003P025R002A030 142 | S008C003P025R002A031 143 | S008C003P025R002A032 144 | S008C003P025R002A033 145 | S008C003P025R002A049 146 | S008C003P025R002A060 147 | S008C003P031R001A001 148 | S008C003P031R002A004 149 | S008C003P031R002A014 150 | S008C003P031R002A015 151 | S008C003P031R002A016 152 | S008C003P031R002A017 153 | S008C003P032R002A013 154 | S008C003P033R002A001 155 | S008C003P033R002A011 156 | S008C003P033R002A012 157 | S008C003P034R002A001 158 | S008C003P034R002A012 159 | S008C003P034R002A022 160 | S008C003P034R002A023 161 | S008C003P034R002A024 162 | S008C003P034R002A044 163 | S008C003P034R002A045 164 | S008C003P035R002A016 165 | S008C003P035R002A017 166 | S008C003P035R002A018 167 | S008C003P035R002A019 168 | S008C003P035R002A020 169 | S008C003P035R002A021 170 | S009C002P007R001A001 171 | S009C002P007R001A003 172 | S009C002P007R001A014 173 | S009C002P008R001A014 174 | S009C002P015R002A050 175 | S009C002P016R001A002 176 | S009C002P017R001A028 177 | S009C002P017R001A029 178 | S009C003P017R002A030 179 | S009C003P025R002A054 180 | S010C001P007R002A020 181 | S010C002P016R002A055 182 | S010C002P017R001A005 183 | S010C002P017R001A018 184 | S010C002P017R001A019 185 | S010C002P019R001A001 186 | S010C002P025R001A012 187 | S010C003P007R002A043 188 | S010C003P008R002A003 189 | S010C003P016R001A055 190 | S010C003P017R002A055 191 | S011C001P002R001A008 192 | S011C001P018R002A050 193 | S011C002P008R002A059 194 | S011C002P016R002A055 195 | S011C002P017R001A020 196 | S011C002P017R001A021 197 | S011C002P018R002A055 198 | S011C002P027R001A009 199 | S011C002P027R001A010 200 | S011C002P027R001A037 201 | S011C003P001R001A055 202 | S011C003P002R001A055 203 | S011C003P008R002A012 204 | S011C003P015R001A055 205 | S011C003P016R001A055 206 | S011C003P019R001A055 207 | S011C003P025R001A055 208 | S011C003P028R002A055 209 | S012C001P019R001A060 210 | S012C001P019R002A060 211 | S012C002P015R001A055 212 | S012C002P017R002A012 213 | S012C002P025R001A060 214 | S012C003P008R001A057 215 | S012C003P015R001A055 216 | S012C003P015R002A055 217 | S012C003P016R001A055 218 | S012C003P017R002A055 219 | S012C003P018R001A055 220 | S012C003P018R001A057 221 | S012C003P019R002A011 222 | S012C003P019R002A012 223 | S012C003P025R001A055 224 | S012C003P027R001A055 225 | S012C003P027R002A009 226 | S012C003P028R001A035 227 | S012C003P028R002A055 228 | S013C001P015R001A054 229 | S013C001P017R002A054 230 | S013C001P018R001A016 231 | S013C001P028R001A040 232 | S013C002P015R001A054 233 | S013C002P017R002A054 234 | S013C002P028R001A040 235 | S013C003P008R002A059 236 | S013C003P015R001A054 237 | S013C003P017R002A054 238 | S013C003P025R002A022 239 | S013C003P027R001A055 240 | S013C003P028R001A040 241 | S014C001P027R002A040 242 | S014C002P015R001A003 243 | S014C002P019R001A029 244 | S014C002P025R002A059 245 | S014C002P027R002A040 246 | S014C002P039R001A050 247 | S014C003P007R002A059 248 | S014C003P015R002A055 249 | S014C003P019R002A055 250 | S014C003P025R001A048 251 | S014C003P027R002A040 252 | S015C001P008R002A040 253 | S015C001P016R001A055 254 | S015C001P017R001A055 255 | S015C001P017R002A055 256 | S015C002P007R001A059 257 | S015C002P008R001A003 258 | S015C002P008R001A004 259 | S015C002P008R002A040 260 | S015C002P015R001A002 261 | S015C002P016R001A001 262 | S015C002P016R002A055 263 | S015C003P008R002A007 264 | S015C003P008R002A011 265 | S015C003P008R002A012 266 | S015C003P008R002A028 267 | S015C003P008R002A040 268 | S015C003P025R002A012 269 | S015C003P025R002A017 270 | S015C003P025R002A020 271 | S015C003P025R002A021 272 | S015C003P025R002A030 273 | S015C003P025R002A033 274 | S015C003P025R002A034 275 | S015C003P025R002A036 276 | S015C003P025R002A037 277 | S015C003P025R002A044 278 | S016C001P019R002A040 279 | S016C001P025R001A011 280 | S016C001P025R001A012 281 | S016C001P025R001A060 282 | S016C001P040R001A055 283 | S016C001P040R002A055 284 | S016C002P008R001A011 285 | S016C002P019R002A040 286 | S016C002P025R002A012 287 | S016C003P008R001A011 288 | S016C003P008R002A002 289 | S016C003P008R002A003 290 | S016C003P008R002A004 291 | S016C003P008R002A006 292 | S016C003P008R002A009 293 | S016C003P019R002A040 294 | S016C003P039R002A016 295 | S017C001P016R002A031 296 | S017C002P007R001A013 297 | S017C002P008R001A009 298 | S017C002P015R001A042 299 | S017C002P016R002A031 300 | S017C002P016R002A055 301 | S017C003P007R002A013 302 | S017C003P008R001A059 303 | S017C003P016R002A031 304 | S017C003P017R001A055 305 | S017C003P020R001A059 306 | S019C001P046R001A075 307 | S019C002P042R001A094 308 | S019C002P042R001A095 309 | S019C002P042R001A096 310 | S019C002P042R001A097 311 | S019C002P042R001A098 312 | S019C002P042R001A099 313 | S019C002P042R001A100 314 | S019C002P042R001A101 315 | S019C002P042R001A102 316 | S019C002P049R002A074 317 | S019C002P049R002A079 318 | S019C002P051R001A061 319 | S019C003P046R001A061 320 | S019C003P046R002A061 321 | S019C003P046R002A062 322 | S020C002P041R001A063 323 | S020C002P041R001A064 324 | S020C002P044R001A063 325 | S020C002P044R001A064 326 | S020C002P044R001A066 327 | S020C002P044R001A084 328 | S020C002P054R001A081 329 | S021C001P059R001A108 330 | S021C002P055R001A065 331 | S021C002P055R001A092 332 | S021C002P055R001A093 333 | S021C002P057R001A064 334 | S021C002P058R001A063 335 | S021C002P058R001A064 336 | S021C002P059R001A074 337 | S021C002P059R001A075 338 | S021C002P059R001A076 339 | S021C002P059R001A077 340 | S021C002P059R001A078 341 | S021C002P059R001A079 342 | S021C003P057R002A078 343 | S021C003P057R002A079 344 | S021C003P057R002A094 345 | S022C002P061R001A113 346 | S022C003P061R002A061 347 | S022C003P061R002A062 348 | S022C003P063R002A061 349 | S022C003P063R002A062 350 | S022C003P063R002A063 351 | S022C003P063R002A064 352 | S022C003P063R002A078 353 | S022C003P064R002A061 354 | S022C003P064R002A062 355 | S022C003P065R002A061 356 | S022C003P065R002A062 357 | S022C003P065R002A119 358 | S022C003P067R002A064 359 | S023C002P055R001A114 360 | S023C002P055R002A092 361 | S023C002P059R001A075 362 | S023C002P063R001A075 363 | S023C003P055R002A093 364 | S023C003P055R002A094 365 | S023C003P061R002A061 366 | S023C003P064R001A092 367 | S024C001P063R001A109 368 | S024C002P062R002A074 369 | S024C002P067R001A100 370 | S024C002P067R001A101 371 | S024C002P067R001A102 372 | S024C002P067R001A103 373 | S024C003P062R002A074 374 | S024C003P063R002A061 375 | S024C003P063R002A062 376 | S025C001P055R002A119 377 | S025C003P056R002A119 378 | S025C003P059R002A115 379 | S026C002P044R001A061 380 | S026C002P044R001A062 381 | S026C002P070R001A092 382 | S026C003P069R002A075 383 | S026C003P074R002A061 384 | S026C003P074R002A062 385 | S026C003P075R001A117 386 | S026C003P075R001A118 387 | S027C001P082R001A063 388 | S027C002P044R002A092 389 | S027C002P079R001A061 390 | S027C002P079R001A062 391 | S027C002P079R001A063 392 | S027C002P079R001A064 393 | S027C002P082R001A092 394 | S027C002P084R001A061 395 | S027C002P084R001A062 396 | S027C002P086R001A061 397 | S027C003P041R002A087 398 | S027C003P080R002A061 399 | S027C003P082R002A061 400 | S027C003P082R002A062 401 | S027C003P086R002A061 402 | S027C003P086R002A062 403 | S028C001P087R001A061 404 | S028C002P041R001A091 405 | S028C002P087R001A061 406 | S028C003P042R002A064 407 | S028C003P046R002A063 408 | S028C003P046R002A066 409 | S028C003P046R002A067 410 | S028C003P046R002A068 411 | S028C003P046R002A069 412 | S028C003P046R002A070 413 | S028C003P046R002A071 414 | S028C003P046R002A072 415 | S028C003P046R002A074 416 | S028C003P046R002A075 417 | S028C003P046R002A077 418 | S028C003P046R002A081 419 | S028C003P046R002A082 420 | S028C003P046R002A083 421 | S028C003P046R002A084 422 | S028C003P048R002A061 423 | S028C003P048R002A062 424 | S028C003P048R002A073 425 | S028C003P073R002A073 426 | S028C003P087R001A061 427 | S028C003P087R002A061 428 | S028C003P087R002A062 429 | S029C001P043R002A092 430 | S029C001P044R002A092 431 | S029C001P048R001A073 432 | S029C001P089R001A063 433 | S029C002P041R001A074 434 | S029C002P041R001A084 435 | S029C002P044R001A091 436 | S029C002P048R001A075 437 | S029C002P048R001A081 438 | S029C002P074R001A081 439 | S029C002P074R001A095 440 | S029C002P074R001A096 441 | S029C002P080R001A091 442 | S029C002P088R001A066 443 | S029C002P089R001A065 444 | S029C002P090R001A067 445 | S029C003P008R002A065 446 | S029C003P008R002A067 447 | S029C003P041R001A089 448 | S029C003P043R001A080 449 | S029C003P043R001A092 450 | S029C003P043R001A105 451 | S029C003P043R002A085 452 | S029C003P043R002A086 453 | S029C003P044R002A106 454 | S029C003P048R001A065 455 | S029C003P048R002A073 456 | S029C003P048R002A074 457 | S029C003P048R002A075 458 | S029C003P048R002A076 459 | S029C003P048R002A092 460 | S029C003P048R002A094 461 | S029C003P051R002A073 462 | S029C003P051R002A074 463 | S029C003P051R002A075 464 | S029C003P051R002A076 465 | S029C003P051R002A077 466 | S029C003P051R002A078 467 | S029C003P051R002A079 468 | S029C003P051R002A080 469 | S029C003P051R002A081 470 | S029C003P051R002A082 471 | S029C003P051R002A083 472 | S029C003P051R002A084 473 | S029C003P051R002A085 474 | S029C003P051R002A086 475 | S029C003P051R002A110 476 | S029C003P067R001A098 477 | S029C003P074R002A110 478 | S029C003P080R002A066 479 | S029C003P088R002A078 480 | S029C003P089R001A075 481 | S029C003P089R002A061 482 | S029C003P089R002A062 483 | S029C003P089R002A063 484 | S029C003P090R002A092 485 | S029C003P090R002A095 486 | S030C002P091R002A091 487 | S030C002P091R002A092 488 | S030C002P091R002A093 489 | S030C002P091R002A094 490 | S030C002P091R002A095 491 | S030C002P091R002A096 492 | S030C002P091R002A097 493 | S030C002P091R002A098 494 | S030C002P091R002A099 495 | S030C002P091R002A100 496 | S030C002P091R002A101 497 | S030C002P091R002A102 498 | S030C002P091R002A103 499 | S030C002P091R002A104 500 | S030C002P091R002A105 501 | S030C003P044R002A065 502 | S030C003P044R002A081 503 | S030C003P044R002A084 504 | S031C002P042R001A111 505 | S031C002P051R001A061 506 | S031C002P051R001A062 507 | S031C002P067R001A067 508 | S031C002P067R001A068 509 | S031C002P067R001A069 510 | S031C002P067R001A070 511 | S031C002P067R001A071 512 | S031C002P067R001A072 513 | S031C002P082R001A075 514 | S031C002P082R002A117 515 | S031C002P097R001A061 516 | S031C002P097R001A062 517 | S031C003P043R002A074 518 | S031C003P043R002A075 519 | S031C003P044R002A094 520 | S031C003P082R002A067 521 | S031C003P082R002A068 522 | S031C003P082R002A069 523 | S031C003P082R002A070 524 | S031C003P082R002A071 525 | S031C003P082R002A072 526 | S031C003P082R002A073 527 | S031C003P082R002A075 528 | S031C003P082R002A076 529 | S031C003P082R002A077 530 | S031C003P082R002A084 531 | S031C003P082R002A085 532 | S031C003P082R002A086 533 | S032C002P067R001A092 534 | S032C003P067R002A066 535 | S032C003P067R002A067 536 | S032C003P067R002A075 537 | S032C003P067R002A076 538 | S032C003P067R002A077 539 | -------------------------------------------------------------------------------- /Matlab/NTU_RGBD_samples_with_missing_skeletons.txt: -------------------------------------------------------------------------------- 1 | 302 of the captured samples in the "NTU RGB+D" dataset have missing or incomplete skeleton data. 2 | If you are working on skeleton-based analysis, please ignore these files in your training and testing procedures. 3 | 4 | S001C002P005R002A008 5 | S001C002P006R001A008 6 | S001C003P002R001A055 7 | S001C003P002R002A012 8 | S001C003P005R002A004 9 | S001C003P005R002A005 10 | S001C003P005R002A006 11 | S001C003P006R002A008 12 | S002C002P011R002A030 13 | S002C003P008R001A020 14 | S002C003P010R002A010 15 | S002C003P011R002A007 16 | S002C003P011R002A011 17 | S002C003P014R002A007 18 | S003C001P019R001A055 19 | S003C002P002R002A055 20 | S003C002P018R002A055 21 | S003C003P002R001A055 22 | S003C003P016R001A055 23 | S003C003P018R002A024 24 | S004C002P003R001A013 25 | S004C002P008R001A009 26 | S004C002P020R001A003 27 | S004C002P020R001A004 28 | S004C002P020R001A012 29 | S004C002P020R001A020 30 | S004C002P020R001A021 31 | S004C002P020R001A036 32 | S005C002P004R001A001 33 | S005C002P004R001A003 34 | S005C002P010R001A016 35 | S005C002P010R001A017 36 | S005C002P010R001A048 37 | S005C002P010R001A049 38 | S005C002P016R001A009 39 | S005C002P016R001A010 40 | S005C002P018R001A003 41 | S005C002P018R001A028 42 | S005C002P018R001A029 43 | S005C003P016R002A009 44 | S005C003P018R002A013 45 | S005C003P021R002A057 46 | S006C001P001R002A055 47 | S006C002P007R001A005 48 | S006C002P007R001A006 49 | S006C002P016R001A043 50 | S006C002P016R001A051 51 | S006C002P016R001A052 52 | S006C002P022R001A012 53 | S006C002P023R001A020 54 | S006C002P023R001A021 55 | S006C002P023R001A022 56 | S006C002P023R001A023 57 | S006C002P024R001A018 58 | S006C002P024R001A019 59 | S006C003P001R002A013 60 | S006C003P007R002A009 61 | S006C003P007R002A010 62 | S006C003P007R002A025 63 | S006C003P016R001A060 64 | S006C003P017R001A055 65 | S006C003P017R002A013 66 | S006C003P017R002A014 67 | S006C003P017R002A015 68 | S006C003P022R002A013 69 | S007C001P018R002A050 70 | S007C001P025R002A051 71 | S007C001P028R001A050 72 | S007C001P028R001A051 73 | S007C001P028R001A052 74 | S007C002P008R002A008 75 | S007C002P015R002A055 76 | S007C002P026R001A008 77 | S007C002P026R001A009 78 | S007C002P026R001A010 79 | S007C002P026R001A011 80 | S007C002P026R001A012 81 | S007C002P026R001A050 82 | S007C002P027R001A011 83 | S007C002P027R001A013 84 | S007C002P028R002A055 85 | S007C003P007R001A002 86 | S007C003P007R001A004 87 | S007C003P019R001A060 88 | S007C003P027R002A001 89 | S007C003P027R002A002 90 | S007C003P027R002A003 91 | S007C003P027R002A004 92 | S007C003P027R002A005 93 | S007C003P027R002A006 94 | S007C003P027R002A007 95 | S007C003P027R002A008 96 | S007C003P027R002A009 97 | S007C003P027R002A010 98 | S007C003P027R002A011 99 | S007C003P027R002A012 100 | S007C003P027R002A013 101 | S008C002P001R001A009 102 | S008C002P001R001A010 103 | S008C002P001R001A014 104 | S008C002P001R001A015 105 | S008C002P001R001A016 106 | S008C002P001R001A018 107 | S008C002P001R001A019 108 | S008C002P008R002A059 109 | S008C002P025R001A060 110 | S008C002P029R001A004 111 | S008C002P031R001A005 112 | S008C002P031R001A006 113 | S008C002P032R001A018 114 | S008C002P034R001A018 115 | S008C002P034R001A019 116 | S008C002P035R001A059 117 | S008C002P035R002A002 118 | S008C002P035R002A005 119 | S008C003P007R001A009 120 | S008C003P007R001A016 121 | S008C003P007R001A017 122 | S008C003P007R001A018 123 | S008C003P007R001A019 124 | S008C003P007R001A020 125 | S008C003P007R001A021 126 | S008C003P007R001A022 127 | S008C003P007R001A023 128 | S008C003P007R001A025 129 | S008C003P007R001A026 130 | S008C003P007R001A028 131 | S008C003P007R001A029 132 | S008C003P007R002A003 133 | S008C003P008R002A050 134 | S008C003P025R002A002 135 | S008C003P025R002A011 136 | S008C003P025R002A012 137 | S008C003P025R002A016 138 | S008C003P025R002A020 139 | S008C003P025R002A022 140 | S008C003P025R002A023 141 | S008C003P025R002A030 142 | S008C003P025R002A031 143 | S008C003P025R002A032 144 | S008C003P025R002A033 145 | S008C003P025R002A049 146 | S008C003P025R002A060 147 | S008C003P031R001A001 148 | S008C003P031R002A004 149 | S008C003P031R002A014 150 | S008C003P031R002A015 151 | S008C003P031R002A016 152 | S008C003P031R002A017 153 | S008C003P032R002A013 154 | S008C003P033R002A001 155 | S008C003P033R002A011 156 | S008C003P033R002A012 157 | S008C003P034R002A001 158 | S008C003P034R002A012 159 | S008C003P034R002A022 160 | S008C003P034R002A023 161 | S008C003P034R002A024 162 | S008C003P034R002A044 163 | S008C003P034R002A045 164 | S008C003P035R002A016 165 | S008C003P035R002A017 166 | S008C003P035R002A018 167 | S008C003P035R002A019 168 | S008C003P035R002A020 169 | S008C003P035R002A021 170 | S009C002P007R001A001 171 | S009C002P007R001A003 172 | S009C002P007R001A014 173 | S009C002P008R001A014 174 | S009C002P015R002A050 175 | S009C002P016R001A002 176 | S009C002P017R001A028 177 | S009C002P017R001A029 178 | S009C003P017R002A030 179 | S009C003P025R002A054 180 | S010C001P007R002A020 181 | S010C002P016R002A055 182 | S010C002P017R001A005 183 | S010C002P017R001A018 184 | S010C002P017R001A019 185 | S010C002P019R001A001 186 | S010C002P025R001A012 187 | S010C003P007R002A043 188 | S010C003P008R002A003 189 | S010C003P016R001A055 190 | S010C003P017R002A055 191 | S011C001P002R001A008 192 | S011C001P018R002A050 193 | S011C002P008R002A059 194 | S011C002P016R002A055 195 | S011C002P017R001A020 196 | S011C002P017R001A021 197 | S011C002P018R002A055 198 | S011C002P027R001A009 199 | S011C002P027R001A010 200 | S011C002P027R001A037 201 | S011C003P001R001A055 202 | S011C003P002R001A055 203 | S011C003P008R002A012 204 | S011C003P015R001A055 205 | S011C003P016R001A055 206 | S011C003P019R001A055 207 | S011C003P025R001A055 208 | S011C003P028R002A055 209 | S012C001P019R001A060 210 | S012C001P019R002A060 211 | S012C002P015R001A055 212 | S012C002P017R002A012 213 | S012C002P025R001A060 214 | S012C003P008R001A057 215 | S012C003P015R001A055 216 | S012C003P015R002A055 217 | S012C003P016R001A055 218 | S012C003P017R002A055 219 | S012C003P018R001A055 220 | S012C003P018R001A057 221 | S012C003P019R002A011 222 | S012C003P019R002A012 223 | S012C003P025R001A055 224 | S012C003P027R001A055 225 | S012C003P027R002A009 226 | S012C003P028R001A035 227 | S012C003P028R002A055 228 | S013C001P015R001A054 229 | S013C001P017R002A054 230 | S013C001P018R001A016 231 | S013C001P028R001A040 232 | S013C002P015R001A054 233 | S013C002P017R002A054 234 | S013C002P028R001A040 235 | S013C003P008R002A059 236 | S013C003P015R001A054 237 | S013C003P017R002A054 238 | S013C003P025R002A022 239 | S013C003P027R001A055 240 | S013C003P028R001A040 241 | S014C001P027R002A040 242 | S014C002P015R001A003 243 | S014C002P019R001A029 244 | S014C002P025R002A059 245 | S014C002P027R002A040 246 | S014C002P039R001A050 247 | S014C003P007R002A059 248 | S014C003P015R002A055 249 | S014C003P019R002A055 250 | S014C003P025R001A048 251 | S014C003P027R002A040 252 | S015C001P008R002A040 253 | S015C001P016R001A055 254 | S015C001P017R001A055 255 | S015C001P017R002A055 256 | S015C002P007R001A059 257 | S015C002P008R001A003 258 | S015C002P008R001A004 259 | S015C002P008R002A040 260 | S015C002P015R001A002 261 | S015C002P016R001A001 262 | S015C002P016R002A055 263 | S015C003P008R002A007 264 | S015C003P008R002A011 265 | S015C003P008R002A012 266 | S015C003P008R002A028 267 | S015C003P008R002A040 268 | S015C003P025R002A012 269 | S015C003P025R002A017 270 | S015C003P025R002A020 271 | S015C003P025R002A021 272 | S015C003P025R002A030 273 | S015C003P025R002A033 274 | S015C003P025R002A034 275 | S015C003P025R002A036 276 | S015C003P025R002A037 277 | S015C003P025R002A044 278 | S016C001P019R002A040 279 | S016C001P025R001A011 280 | S016C001P025R001A012 281 | S016C001P025R001A060 282 | S016C001P040R001A055 283 | S016C001P040R002A055 284 | S016C002P008R001A011 285 | S016C002P019R002A040 286 | S016C002P025R002A012 287 | S016C003P008R001A011 288 | S016C003P008R002A002 289 | S016C003P008R002A003 290 | S016C003P008R002A004 291 | S016C003P008R002A006 292 | S016C003P008R002A009 293 | S016C003P019R002A040 294 | S016C003P039R002A016 295 | S017C001P016R002A031 296 | S017C002P007R001A013 297 | S017C002P008R001A009 298 | S017C002P015R001A042 299 | S017C002P016R002A031 300 | S017C002P016R002A055 301 | S017C003P007R002A013 302 | S017C003P008R001A059 303 | S017C003P016R002A031 304 | S017C003P017R001A055 305 | S017C003P020R001A059 306 | -------------------------------------------------------------------------------- /Matlab/read_skeleton_file.m: -------------------------------------------------------------------------------- 1 | function bodyinfo = read_skeleton_file(filename) 2 | % Reads an .skeleton file from "NTU RGB+D 3D Action Recognition Dataset". 3 | % 4 | % Argrument: 5 | % filename: full adress and filename of the .skeleton file. 6 | % 7 | % For further information please refer to: 8 | % NTU RGB+D dataset's webpage: 9 | % http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp 10 | % NTU RGB+D dataset's github page: 11 | % https://github.com/shahroudy/NTURGB-D 12 | % CVPR 2016 paper: 13 | % Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang, 14 | % "NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis", 15 | % in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 16 | % 17 | % For more details about the provided data, please refer to: 18 | % https://msdn.microsoft.com/en-us/library/dn799271.aspx 19 | % https://msdn.microsoft.com/en-us/library/dn782037.aspx 20 | 21 | fileid = fopen(filename); 22 | framecount = fscanf(fileid,'%d',1); % no of the recorded frames 23 | 24 | bodyinfo=[]; % to store multiple skeletons per frame 25 | 26 | for f=1:framecount 27 | bodycount = fscanf(fileid,'%d',1); % no of observerd skeletons in current frame 28 | for b=1:bodycount 29 | clear body; 30 | body.bodyID = fscanf(fileid,'%ld',1); % tracking id of the skeleton 31 | arrayint = fscanf(fileid,'%d',6); % read 6 integers 32 | body.clipedEdges = arrayint(1); 33 | body.handLeftConfidence = arrayint(2); 34 | body.handLeftState = arrayint(3); 35 | body.handRightConfidence = arrayint(4); 36 | body.handRightState = arrayint(5); 37 | body.isResticted = arrayint(6); 38 | lean = fscanf(fileid,'%f',2); 39 | body.leanX = lean(1); 40 | body.leanY = lean(2); 41 | body.trackingState = fscanf(fileid,'%d',1); 42 | 43 | body.jointCount = fscanf(fileid,'%d',1); % no of joints (25) 44 | joints=[]; 45 | for j=1:body.jointCount 46 | jointinfo = fscanf(fileid,'%f',11); 47 | joint=[]; 48 | 49 | % 3D location of the joint j 50 | joint.x = jointinfo(1); 51 | joint.y = jointinfo(2); 52 | joint.z = jointinfo(3); 53 | 54 | % 2D location of the joint j in corresponding depth/IR frame 55 | joint.depthX = jointinfo(4); 56 | joint.depthY = jointinfo(5); 57 | 58 | % 2D location of the joint j in corresponding RGB frame 59 | joint.colorX = jointinfo(6); 60 | joint.colorY = jointinfo(7); 61 | 62 | % The orientation of the joint j 63 | joint.orientationW = jointinfo(8); 64 | joint.orientationX = jointinfo(9); 65 | joint.orientationY = jointinfo(10); 66 | joint.orientationZ = jointinfo(11); 67 | 68 | % The tracking state of the joint j 69 | joint.trackingState = fscanf(fileid,'%d',1); 70 | 71 | body.joints(j)=joint; 72 | end 73 | bodyinfo(f).bodies(b)=body; 74 | end 75 | end 76 | fclose(fileid); 77 | end -------------------------------------------------------------------------------- /Matlab/show_skeleton_on_IR_frames.m: -------------------------------------------------------------------------------- 1 | function []=show_skeleton_on_IR_frames(... 2 | skeletonfilename,irfilename,outputvideofilename) 3 | % Draws the skeleton data on IR frames. 4 | % 5 | % Argrument: 6 | % skeletonfilename: full adress and filename of the .skeleton file. 7 | % irfilename: corresponding IR video file 8 | % outputvideofilename (optional): the filename for output video file. 9 | % 10 | % For further information please refer to: 11 | % NTU RGB+D dataset's webpage: 12 | % http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp 13 | % NTU RGB+D dataset's github page: 14 | % https://github.com/shahroudy/NTURGB-D 15 | % CVPR 2016 paper: 16 | % Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang, 17 | % "NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis", 18 | % in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 19 | 20 | bodyinfo = read_skeleton_file(skeletonfilename); 21 | 22 | videofile = VideoReader(irfilename); 23 | irvid = read(videofile); 24 | 25 | if nargin>2 % if the output video file is given 26 | writerObj = VideoWriter(outputvideofilename); 27 | open(writerObj); 28 | end 29 | 30 | % in the skeleton structure, each joint is connected to some other joint: 31 | connecting_joint = ... 32 | [2 1 21 3 21 5 6 7 21 9 10 11 1 13 14 15 1 17 18 19 2 8 8 12 12]; 33 | 34 | % reapeat this for every frame 35 | for f=1:numel(bodyinfo) 36 | try 37 | imir = irvid(:,:,:,f); 38 | 39 | % for all the detected skeletons in the current frame: 40 | for b=1:numel(bodyinfo(f).bodies) 41 | % for all the 25 joints within each skeleton: 42 | for j=1:25 43 | try 44 | % use red color for drawing joint connections 45 | rv=255; 46 | gv=0; 47 | bv=0; 48 | 49 | k = connecting_joint(j); 50 | 51 | joint = bodyinfo(f).bodies(b).joints(j); 52 | dx = joint.depthX; 53 | dy = joint.depthY; 54 | joint2 = bodyinfo(f).bodies(b).joints(k); 55 | dx2 = joint2.depthX; 56 | dy2 = joint2.depthY; 57 | 58 | xdist=abs(dx-dx2); 59 | ydist=abs(dy-dy2); 60 | 61 | % locate the pixels of the connecting line between the 62 | % two joints 63 | if xdist>ydist 64 | xrange = [dx:sign(dx2-dx):dx2]; 65 | yrange = [dy:sign(dy2-dy)*abs((dy2-dy)/(dx2-dx)):dy2]; 66 | else 67 | yrange = [dy:sign(dy2-dy):dy2]; 68 | xrange = [dx:sign(dx2-dx)*abs((dx2-dx)/(dy2-dy)):dx2]; 69 | end 70 | % draw the line! 71 | for i=1:numel(xrange) 72 | dx = int32(round(xrange(i))); 73 | dy = int32(round(yrange(i))); 74 | imir(dy-3:dy+3,dx-3:dx+3,1)=rv; 75 | imir(dy-3:dy+3,dx-3:dx+3,2)=gv; 76 | imir(dy-3:dy+3,dx-3:dx+3,3)=bv; 77 | end 78 | 79 | joint = bodyinfo(f).bodies(b).joints(j); 80 | dx = int32(round(joint.depthX)); 81 | dy = int32(round(joint.depthY)); 82 | 83 | % use green color to draw joints 84 | rv=0; 85 | gv=255; 86 | bv=0; 87 | imir(dy-3:dy+3,dx-3:dx+3,1)=rv; 88 | imir(dy-3:dy+3,dx-3:dx+3,2)=gv; 89 | imir(dy-3:dy+3,dx-3:dx+3,3)=bv; 90 | catch err1 91 | disp(err1); 92 | end 93 | end 94 | end 95 | imir = imir(1:424,1:512,:); 96 | imshow(imir); 97 | if nargin>2 98 | writeVideo(writerObj,imir); 99 | end 100 | pause(0.003); 101 | catch err2 102 | disp(err2); 103 | end 104 | end 105 | 106 | if nargin>2 107 | close(writerObj); 108 | end 109 | end 110 | -------------------------------------------------------------------------------- /Matlab/show_skeleton_on_RGB_frames.m: -------------------------------------------------------------------------------- 1 | function []=show_skeleton_on_RGB_frames(... 2 | skeletonfilename,rgbfilename,outputvideofilename) 3 | % Draws the skeleton data on RGB frames. 4 | % 5 | % Argrument: 6 | % skeletonfilename: full adress and filename of the .skeleton file. 7 | % rgbfilename: corresponding RGB video file 8 | % outputvideofilename (optional): the filename for output video file. 9 | % 10 | % For further information please refer to: 11 | % NTU RGB+D dataset's webpage: 12 | % http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp 13 | % NTU RGB+D dataset's github page: 14 | % https://github.com/shahroudy/NTURGB-D 15 | % CVPR 2016 paper: 16 | % Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang, 17 | % "NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis", 18 | % in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 19 | 20 | bodyinfo = read_skeleton_file(skeletonfilename); 21 | 22 | videofile = VideoReader(rgbfilename); 23 | rgbvid = read(videofile); 24 | 25 | if nargin>2 % if the output video file is given 26 | writerObj = VideoWriter(outputvideofilename); 27 | open(writerObj); 28 | end 29 | 30 | % in the skeleton structure, each joint is connected to some other joint: 31 | connecting_joint = ... 32 | [2 1 21 3 21 5 6 7 21 9 10 11 1 13 14 15 1 17 18 19 2 8 8 12 12]; 33 | 34 | % reapeat this for every frame 35 | for f=1:numel(bodyinfo) 36 | try 37 | imrgb = rgbvid(:,:,:,f); 38 | 39 | % for all the detected skeletons in the current frame: 40 | for b=1:numel(bodyinfo(f).bodies) 41 | % for all the 25 joints within each skeleton: 42 | for j=1:25 43 | try 44 | % use red color for drawing joint connections 45 | rv=255; 46 | gv=0; 47 | bv=0; 48 | 49 | k = connecting_joint(j); 50 | 51 | joint = bodyinfo(f).bodies(b).joints(j); 52 | dx = joint.colorX; 53 | dy = joint.colorY; 54 | joint2 = bodyinfo(f).bodies(b).joints(k); 55 | dx2 = joint2.colorX; 56 | dy2 = joint2.colorY; 57 | 58 | xdist=abs(dx-dx2); 59 | ydist=abs(dy-dy2); 60 | 61 | % locate the pixels of the connecting line between the 62 | % two joints 63 | if xdist>ydist 64 | xrange = [dx:sign(dx2-dx):dx2]; 65 | yrange = [dy:sign(dy2-dy)*abs((dy2-dy)/(dx2-dx)):dy2]; 66 | else 67 | yrange = [dy:sign(dy2-dy):dy2]; 68 | xrange = [dx:sign(dx2-dx)*abs((dx2-dx)/(dy2-dy)):dx2]; 69 | end 70 | % draw the line! 71 | for i=1:numel(xrange) 72 | dx = int32(round(xrange(i))); 73 | dy = int32(round(yrange(i))); 74 | imrgb(dy-3:dy+3,dx-3:dx+3,1)=rv; 75 | imrgb(dy-3:dy+3,dx-3:dx+3,2)=gv; 76 | imrgb(dy-3:dy+3,dx-3:dx+3,3)=bv; 77 | end 78 | 79 | joint = bodyinfo(f).bodies(b).joints(j); 80 | dx = int32(round(joint.colorX)); 81 | dy = int32(round(joint.colorY)); 82 | 83 | % use green color to draw joints 84 | rv=0; 85 | gv=255; 86 | bv=0; 87 | imrgb(dy-7:dy+7,dx-7:dx+7,1)=rv; 88 | imrgb(dy-7:dy+7,dx-7:dx+7,2)=gv; 89 | imrgb(dy-7:dy+7,dx-7:dx+7,3)=bv; 90 | catch err1 91 | disp(err1); 92 | end 93 | end 94 | end 95 | imrgb = imrgb(1:1080,1:1920,:); 96 | imshow(imrgb); 97 | writeVideo(writerObj,imrgb); 98 | pause(0.001); 99 | catch err2 100 | disp(err2); 101 | end 102 | end 103 | close(writerObj); 104 | end 105 | -------------------------------------------------------------------------------- /Matlab/show_skeleton_on_depthmaps.m: -------------------------------------------------------------------------------- 1 | function []=show_skeleton_on_depthmaps(... 2 | skeletonfilename,depthmapsfolder,outputvideofilename) 3 | % Draws the skeleton data on depthmaps. 4 | % 5 | % Argrument: 6 | % skeletonfilename: full adress and filename of the .skeleton file. 7 | % depthmapsfolder: corresponding depthmaps folder (full or masked) 8 | % outputvideofilename (optional): the filename for output video file. 9 | % 10 | % For further information please refer to: 11 | % NTU RGB+D dataset's webpage: 12 | % http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp 13 | % NTU RGB+D dataset's github page: 14 | % https://github.com/shahroudy/NTURGB-D 15 | % CVPR 2016 paper: 16 | % Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang, 17 | % "NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis", 18 | % in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 19 | 20 | bodyinfo = read_skeleton_file(skeletonfilename); 21 | 22 | if nargin>2 % if the output video file is given 23 | writerObj = VideoWriter(outputvideofilename); 24 | open(writerObj); 25 | end 26 | 27 | % in the skeleton structure, each joint is connected to some other joint: 28 | connecting_joint = ... 29 | [2 1 21 3 21 5 6 7 21 9 10 11 1 13 14 15 1 17 18 19 2 8 8 12 12]; 30 | 31 | % reapeat this for every frame 32 | for f=1:numel(bodyinfo) 33 | try 34 | imdepth = imread(fullfile(depthmapsfolder,sprintf('Depth-%08d.png',f))); 35 | % depth maps are in millimeters we need to scale them to [0,255] 36 | % for visualization: 37 | imdepth = double(imdepth)/5000; 38 | imdepth(imdepth>1)=1; 39 | imdepth = uint8(255*imdepth); 40 | 41 | % make it a 3-channel color image... 42 | depthframe = repmat(imdepth,1,1,3); 43 | 44 | % for all the detected skeletons in the current frame: 45 | for b=1:numel(bodyinfo(f).bodies) 46 | % for all the 25 joints within each skeleton: 47 | for j=1:25 48 | try 49 | % use red color for drawing joint connections 50 | rv=255; 51 | gv=0; 52 | bv=0; 53 | 54 | k = connecting_joint(j); 55 | 56 | joint = bodyinfo(f).bodies(b).joints(j); 57 | dx = joint.depthX; 58 | dy = joint.depthY; 59 | joint2 = bodyinfo(f).bodies(b).joints(k); 60 | dx2 = joint2.depthX; 61 | dy2 = joint2.depthY; 62 | 63 | xdist=abs(dx-dx2); 64 | ydist=abs(dy-dy2); 65 | 66 | % locate the pixels of the connecting line between the 67 | % two joints 68 | if xdist>ydist 69 | xrange = [dx:sign(dx2-dx):dx2]; 70 | yrange = [dy:sign(dy2-dy)*abs((dy2-dy)/(dx2-dx)):dy2]; 71 | else 72 | yrange = [dy:sign(dy2-dy):dy2]; 73 | xrange = [dx:sign(dx2-dx)*abs((dx2-dx)/(dy2-dy)):dx2]; 74 | end 75 | % draw the line! 76 | for i=1:numel(xrange) 77 | dx = int32(round(xrange(i))); 78 | dy = int32(round(yrange(i))); 79 | depthframe(dy-1:dy+1,dx-1:dx+1,1)=rv; 80 | depthframe(dy-1:dy+1,dx-1:dx+1,2)=gv; 81 | depthframe(dy-1:dy+1,dx-1:dx+1,3)=bv; 82 | end 83 | 84 | joint = bodyinfo(f).bodies(b).joints(j); 85 | dx = int32(round(joint.depthX)); 86 | dy = int32(round(joint.depthY)); 87 | 88 | % use green color to draw joints 89 | rv=0; 90 | gv=255; 91 | bv=0; 92 | depthframe(dy-3:dy+3,dx-3:dx+3,1)=rv; 93 | depthframe(dy-3:dy+3,dx-3:dx+3,2)=gv; 94 | depthframe(dy-3:dy+3,dx-3:dx+3,3)=bv; 95 | catch err1 96 | disp(err1); 97 | end 98 | end 99 | end 100 | 101 | % cut the extra boundaries if any! 102 | depthframe = depthframe(1:424,1:512,:); 103 | 104 | imshow(depthframe); 105 | if nargin>2 106 | writeVideo(writerObj,depthframe); 107 | end 108 | pause(0.01); 109 | catch err2 110 | disp(err2); 111 | end 112 | end 113 | 114 | if nargin>2 115 | close(writerObj); 116 | end 117 | end 118 | -------------------------------------------------------------------------------- /Python/.gitignore: -------------------------------------------------------------------------------- 1 | ./raw_npy/ 2 | -------------------------------------------------------------------------------- /Python/readme.md: -------------------------------------------------------------------------------- 1 | # NTU RGBD 120 database skeleton data parser in python 2 | 3 | 4 | 5 | This repository is to parse the skeleton data in NTU RGB+D 120 database[1]. Although the database already contains the skeleton data parser, It's written in MATLAB which is dependent on the large, commercial platform MATLAB. Also, the script in MATLAB is quite slow and hard to be parallel, making the process of extract the valid data in txt files tedious and time-consuming. To address those problems and make it easy to embed in the whole python project, I write the skeleton data parser in python. 6 | 7 | # Files in Project 8 | 9 | The structure of the project looks like: 10 | 11 | ```shell 12 | read_ntu_rgbd/ 13 | ├── ntu_rgb120_missings.txt # The missing files list, each item in this list needs to be ignored 14 | ├── raw_txt/ # the folder contains the original .txt skeleton data 15 | ├── raw_npy/ # the folder contains the parsed numpy arrays 16 | ├── readme.md # it's me :) 17 | └── txt2npy.py # the main script to parse the data 18 | ``` 19 | 20 | You need to move your original txt data in the `raw_txt/` folder first. 21 | 22 | 23 | 24 | # Run the script 25 | 26 | Before running the script, there are some configs you may need to modify to fit your system. Edit the `txt2npy.py` 27 | 28 | You need to specify the `save_npy_path`, `load_txt_path` to load and save the data. 29 | 30 | After the prepare, just run: 31 | 32 | ```shell 33 | python txt2npy.py 34 | ``` 35 | 36 | and wait for the ending. 37 | 38 | 39 | 40 | # the data structure in array 41 | 42 | In the data parsing process, I mainly concern some but not all items, there are what I most concern: 43 | 44 | 1. skeleton X, Y, Z coordinate 45 | 2. the skeleton coordinate projection on RGBs, it's a 2D coordinate X,Y 46 | 3. the skeleton coordinate projection on Depth, it's a 2D coordinate X,Y 47 | 48 | There are still other items left, like the ` handRightConfidence `, ` handLeftState `, ` trackingState `, ` handLeftConfidence ` etc. But I just ignore them in this parser, if you need them you need to modify the code. Don't worry, it's rather easy. 49 | 50 | Each sample is saved as the array in numpy with name like `SxxxCxxxPxxxRxxxAxxx.skeleton.npy` . You can read the data by: 51 | 52 | ```python 53 | data = np.load('./SxxxCxxxPxxxRxxxAxxx.skeleton.npy',allow_pickle=True).item() 54 | ``` 55 | 56 | Inside the array, it's actually a dictionary with some keys and each keys is a certain item you need. I list the item as following: 57 | 58 | - `file_name`: file's name 59 | - `nbodys`: it's a list with same length of the sequence. it represents the number of the actors in each frame. 60 | - `njoints`: the number of the joint node in the skeleton, it's a constant here 61 | - `skel_bodyx`: the skeleton coordinate with the shape of `(nframe, njoint, 3)`, the x denotes the id of the acting person in each frame. 62 | - `rgb_bodyx`: the projection of the skeleton coordinate in RGBs. 63 | - `depth_bodyx`: the projection of the skeleton coordinate in Depths 64 | 65 | 66 | 67 | # Reference 68 | 69 | [1]. https://github.com/shahroudy/NTURGB-D -------------------------------------------------------------------------------- /Python/txt2npy.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | 4 | ''' 5 | transform the skeleton data in NTU RGB+D dataset into the numpy arrays for a more efficient data loading 6 | ''' 7 | 8 | import numpy as np 9 | import os 10 | import sys 11 | 12 | user_name = 'user' 13 | save_npy_path = '/home/user/AI_workspace/datasets/NTU/raw_npy120/' 14 | load_txt_path = '/home/user/AI_workspace/datasets/NTU/raw_txt120/' 15 | missing_file_path = './ntu_rgb120_missings.txt' 16 | step_ranges = list(range(0,100)) # just parse range, for the purpose of paralle running. 17 | 18 | 19 | toolbar_width = 50 20 | def _print_toolbar(rate, annotation=''): 21 | sys.stdout.write("{}[".format(annotation)) 22 | for i in range(toolbar_width): 23 | if i * 1.0 / toolbar_width > rate: 24 | sys.stdout.write(' ') 25 | else: 26 | sys.stdout.write('-') 27 | sys.stdout.flush() 28 | sys.stdout.write(']\r') 29 | 30 | def _end_toolbar(): 31 | sys.stdout.write('\n') 32 | 33 | def _load_missing_file(path): 34 | missing_files = dict() 35 | with open(path, 'r') as f: 36 | lines = f.readlines() 37 | for line in lines: 38 | line = line[:-1] 39 | if line not in missing_files: 40 | missing_files[line] = True 41 | return missing_files 42 | 43 | def _read_skeleton(file_path, save_skelxyz=True, save_rgbxy=True, save_depthxy=True): 44 | f = open(file_path, 'r') 45 | datas = f.readlines() 46 | f.close() 47 | max_body = 4 48 | njoints = 25 49 | 50 | # specify the maximum number of the body shown in the sequence, according to the certain sequence, need to pune the 51 | # abundant bodys. 52 | # read all lines into the pool to speed up, less io operation. 53 | nframe = int(datas[0][:-1]) 54 | bodymat = dict() 55 | bodymat['file_name'] = file_path[-29:-9] 56 | nbody = int(datas[1][:-1]) 57 | bodymat['nbodys'] = [] 58 | bodymat['njoints'] = njoints 59 | for body in range(max_body): 60 | if save_skelxyz: 61 | bodymat['skel_body{}'.format(body)] = np.zeros(shape=(nframe, njoints, 3)) 62 | if save_rgbxy: 63 | bodymat['rgb_body{}'.format(body)] = np.zeros(shape=(nframe, njoints, 2)) 64 | if save_depthxy: 65 | bodymat['depth_body{}'.format(body)] = np.zeros(shape=(nframe, njoints, 2)) 66 | # above prepare the data holder 67 | cursor = 0 68 | for frame in range(nframe): 69 | cursor += 1 70 | bodycount = int(datas[cursor][:-1]) 71 | if bodycount == 0: 72 | continue 73 | # skip the empty frame 74 | bodymat['nbodys'].append(bodycount) 75 | for body in range(bodycount): 76 | cursor += 1 77 | skel_body = 'skel_body{}'.format(body) 78 | rgb_body = 'rgb_body{}'.format(body) 79 | depth_body = 'depth_body{}'.format(body) 80 | 81 | bodyinfo = datas[cursor][:-1].split(' ') 82 | cursor += 1 83 | 84 | njoints = int(datas[cursor][:-1]) 85 | for joint in range(njoints): 86 | cursor += 1 87 | jointinfo = datas[cursor][:-1].split(' ') 88 | jointinfo = np.array(list(map(float, jointinfo))) 89 | if save_skelxyz: 90 | bodymat[skel_body][frame,joint] = jointinfo[:3] 91 | if save_depthxy: 92 | bodymat[depth_body][frame,joint] = jointinfo[3:5] 93 | if save_rgbxy: 94 | bodymat[rgb_body][frame,joint] = jointinfo[5:7] 95 | # prune the abundant bodys 96 | for each in range(max_body): 97 | if each >= max(bodymat['nbodys']): 98 | if save_skelxyz: 99 | del bodymat['skel_body{}'.format(each)] 100 | if save_rgbxy: 101 | del bodymat['rgb_body{}'.format(each)] 102 | if save_depthxy: 103 | del bodymat['depth_body{}'.format(each)] 104 | return bodymat 105 | 106 | 107 | 108 | 109 | if __name__ == '__main__': 110 | missing_files = _load_missing_file(missing_file_path) 111 | datalist = os.listdir(load_txt_path) 112 | alread_exist = os.listdir(save_npy_path) 113 | alread_exist_dict = dict(zip(alread_exist, len(alread_exist) * [True])) 114 | 115 | for ind, each in enumerate(datalist): 116 | _print_toolbar(ind * 1.0 / len(datalist), 117 | '({:>5}/{:<5})'.format( 118 | ind + 1, len(datalist) 119 | )) 120 | S = int(each[1:4]) 121 | if S not in step_ranges: 122 | continue 123 | if each+'.skeleton.npy' in alread_exist_dict: 124 | print('file already existed !') 125 | continue 126 | if each[:20] in missing_files: 127 | print('file missing') 128 | continue 129 | loadname = load_txt_path+each 130 | print(each) 131 | mat = _read_skeleton(loadname) 132 | mat = np.array(mat) 133 | save_path = save_npy_path+'{}.npy'.format(each) 134 | np.save(save_path, mat) 135 | # raise ValueError() 136 | _end_toolbar() 137 | 138 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # "NTU RGB+D" Action Recognition Dataset 2 | # "NTU RGB+D 120" Action Recognition Dataset 3 | 4 | "NTU RGB+D" is a large-scale dataset for human action recognition. 5 | It is introduced in our CVPR 2016 paper [[PDF]](http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Shahroudy_NTU_RGBD_A_CVPR_2016_paper.pdf). 6 | 7 | "NTU RGB+D 120" is the extended version of the "NTU RGB+D" dataset. 8 | It is introduced in our TPAMI 2020 paper [[PDF]](https://arxiv.org/pdf/1905.04757.pdf). 9 | 10 | For any possible query regarding the datasets, please contact the first author of the paper. 11 | 12 | ### How to download the datasets 13 | 14 | The full datasets can be downloaded via: 15 | 16 | [https://rose1.ntu.edu.sg/dataset/actionRecognition/](https://rose1.ntu.edu.sg/dataset/actionRecognition/) 17 | 18 | If you need the skeleton data only, you could also obtain it via: 19 | 20 | https://drive.google.com/open?id=1CUZnBtYwifVXS21yVg62T-vrPVayso5H 21 | 22 | https://drive.google.com/open?id=1tEbuaEqMxAV7dNc4fqu1O4M7mC6CJ50w 23 | 24 | ### Structures of the datasets 25 | 26 | "NTU RGB+D" and NTU RGB+D 120" datasets contain 56,880 and 114,480 action samples, respectively. Both datasets include 4 different modalities of data for each sample: 27 | * RGB videos 28 | * depth map sequences 29 | * 3D skeletal data 30 | * infrared (IR) videos 31 | 32 | Video samples have been captured by three Microsoft Kinect V2 cameras concurrently. 33 | The resolutions of RGB videos are 1920×1080, depth maps and IR videos are all in 512×424, and 3D skeletal data contains the 3D locations of 25 major body joints at each frame. 34 | 35 | **Each file/folder name in both datasets is in the format of SsssCcccPpppRrrrAaaa (e.g., S001C002P003R002A013), in which sss is the setup number, ccc is the camera ID, ppp is the performer (subject) ID, rrr is the replication number (1 or 2), and aaa is the action class label.** 36 | 37 | The "NTU RGB+D" dataset includes the files/folders with setup numbers between S001 and S017, while the "NTU RGB+D 120" dataset includes the files/folders with setup numbers between S001 and S032. 38 | 39 | For more details about the setups, camera IDs, ..., please refer to the ["NTU RGB+D" dataset paper](http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Shahroudy_NTU_RGBD_A_CVPR_2016_paper.pdf) and the ["NTU RGB+D 120" dataset paper](https://arxiv.org/pdf/1905.04757.pdf). 40 | 41 | ### Samples with missing skeletons 42 | 43 | 302 samples in "NTU RGB+D" dataset and 535 samples in "NTU RGB+D 120" dataset have missing or incomplete skeleton data. 44 | If you are working on skeleton-based analysis, please ignore these files in your training and testing procedures. 45 | The list of these samples in "NTU RGB+D" dataset are provided [here](https://github.com/shahroudy/NTURGB-D/blob/master/Matlab/NTU_RGBD_samples_with_missing_skeletons.txt). 46 | The list of these samples in "NTU RGB+D 120" dataset are provided [here](https://github.com/shahroudy/NTURGB-D/blob/master/Matlab/NTU_RGBD120_samples_with_missing_skeletons.txt). 47 | 48 | ### Sample codes 49 | 50 | We have provided some MATLAB codes [here](https://github.com/shahroudy/NTURGB-D/tree/master/Matlab) to demonstrate how to read the skeleton files, map them to other modalities (RGB, depth, and IR frames), and visualize the skeleton data. The [codes](https://github.com/shahroudy/NTURGB-D/tree/master/Matlab) are suitable for both "NTU RGB+D" and "NTU RGB+D 120". 51 | 52 | ### Action Classes 53 | 54 | "NTU RGB+D" dataset contains 60 action classes, and "NTU RGB+D 120" dataset contains 120 action classes. The actions in these two datasets are listed below. Note that actions labelled from A1 to A60 are in "NTU RGB+D", while actions labelled from A1 to A120 are in "NTU RGB+D 120". 55 | 56 | * A1. drink water. 57 | * A2. eat meal/snack. 58 | * A3. brushing teeth. 59 | * A4. brushing hair. 60 | * A5. drop. 61 | * A6. pickup. 62 | * A7. throw. 63 | * A8. sitting down. 64 | * A9. standing up (from sitting position). 65 | * A10. clapping. 66 | * A11. reading. 67 | * A12. writing. 68 | * A13. tear up paper. 69 | * A14. wear jacket. 70 | * A15. take off jacket. 71 | * A16. wear a shoe. 72 | * A17. take off a shoe. 73 | * A18. wear on glasses. 74 | * A19. take off glasses. 75 | * A20. put on a hat/cap. 76 | * A21. take off a hat/cap. 77 | * A22. cheer up. 78 | * A23. hand waving. 79 | * A24. kicking something. 80 | * A25. reach into pocket. 81 | * A26. hopping (one foot jumping). 82 | * A27. jump up. 83 | * A28. make a phone call/answer phone. 84 | * A29. playing with phone/tablet. 85 | * A30. typing on a keyboard. 86 | * A31. pointing to something with finger. 87 | * A32. taking a selfie. 88 | * A33. check time (from watch). 89 | * A34. rub two hands together. 90 | * A35. nod head/bow. 91 | * A36. shake head. 92 | * A37. wipe face. 93 | * A38. salute. 94 | * A39. put the palms together. 95 | * A40. cross hands in front (say stop). 96 | * A41. sneeze/cough. 97 | * A42. staggering. 98 | * A43. falling. 99 | * A44. touch head (headache). 100 | * A45. touch chest (stomachache/heart pain). 101 | * A46. touch back (backache). 102 | * A47. touch neck (neckache). 103 | * A48. nausea or vomiting condition. 104 | * A49. use a fan (with hand or paper)/feeling warm. 105 | * A50. punching/slapping other person. 106 | * A51. kicking other person. 107 | * A52. pushing other person. 108 | * A53. pat on back of other person. 109 | * A54. point finger at the other person. 110 | * A55. hugging other person. 111 | * A56. giving something to other person. 112 | * A57. touch other person's pocket. 113 | * A58. handshaking. 114 | * A59. walking towards each other. 115 | * A60. walking apart from each other. 116 | * A61. put on headphone. 117 | * A62. take off headphone. 118 | * A63. shoot at the basket. 119 | * A64. bounce ball. 120 | * A65. tennis bat swing. 121 | * A66. juggling table tennis balls. 122 | * A67. hush (quite). 123 | * A68. flick hair. 124 | * A69. thumb up. 125 | * A70. thumb down. 126 | * A71. make ok sign. 127 | * A72. make victory sign. 128 | * A73. staple book. 129 | * A74. counting money. 130 | * A75. cutting nails. 131 | * A76. cutting paper (using scissors). 132 | * A77. snapping fingers. 133 | * A78. open bottle. 134 | * A79. sniff (smell). 135 | * A80. squat down. 136 | * A81. toss a coin. 137 | * A82. fold paper. 138 | * A83. ball up paper. 139 | * A84. play magic cube. 140 | * A85. apply cream on face. 141 | * A86. apply cream on hand back. 142 | * A87. put on bag. 143 | * A88. take off bag. 144 | * A89. put something into a bag. 145 | * A90. take something out of a bag. 146 | * A91. open a box. 147 | * A92. move heavy objects. 148 | * A93. shake fist. 149 | * A94. throw up cap/hat. 150 | * A95. hands up (both hands). 151 | * A96. cross arms. 152 | * A97. arm circles. 153 | * A98. arm swings. 154 | * A99. running on the spot. 155 | * A100. butt kicks (kick backward). 156 | * A101. cross toe touch. 157 | * A102. side kick. 158 | * A103. yawn. 159 | * A104. stretch oneself. 160 | * A105. blow nose. 161 | * A106. hit other person with something. 162 | * A107. wield knife towards other person. 163 | * A108. knock over other person (hit with body). 164 | * A109. grab other person’s stuff. 165 | * A110. shoot at other person with a gun. 166 | * A111. step on foot. 167 | * A112. high-five. 168 | * A113. cheers and drink. 169 | * A114. carry something with other person. 170 | * A115. take a photo of other person. 171 | * A116. follow other person. 172 | * A117. whisper in other person’s ear. 173 | * A118. exchange things with other person. 174 | * A119. support somebody with hand. 175 | * A120. finger-guessing game (playing rock-paper-scissors). 176 | 177 | ### Evaluation Protocol of One-Shot Action Recognition on "NTU RGB+D 120" 178 | 179 | In ["NTU RGB+D 120" dataset paper](https://arxiv.org/pdf/1905.04757.pdf), we introduced the one-shot recognition setting, in which "NTU RGB+D 120" dataset is split to two parts: auxiliary set and one-shot evaluation set. 180 | **Auxiliary set** contains 100 classes, and all samples of these classes can be used for learning. 181 | **Evaluation set** consists of 20 novel classes, and one sample from each novel class is picked as the exemplar, while all the remaining samples of these classes are used to test the recognition performance. 182 | 183 | **Evaluation set.** 20 novel classes, namely, A1, A7, A13, A19, A25, A31, A37, A43, A49, A55, A61, A67, A73, A79, A85, A91, A97, A103, A109, A115. The following 20 samples are the exemplars: 184 | (01)**S001**C003P008R001**A001**, (02)**S001**C003P008R001**A007**, (03)**S001**C003P008R001**A013**, (04)**S001**C003P008R001**A019**, (05)**S001**C003P008R001**A025**, (06)**S001**C003P008R001**A031**, (07)**S001**C003P008R001**A037**, (08)**S001**C003P008R001**A043**, (09)**S001**C003P008R001**A049**, (10)**S001**C003P008R001**A055**, 185 | (11)**S018**C003P008R001**A061**, (12)**S018**C003P008R001**A067**, (13)**S018**C003P008R001**A073**, (14)**S018**C003P008R001**A079**, (15)**S018**C003P008R001**A085**, (16)**S018**C003P008R001**A091**, (17)**S018**C003P008R001**A097**, (18)**S018**C003P008R001**A103**, (19)**S018**C003P008R001**A109**, (20)**S018**C003P008R001**A115**. 186 | 187 | **Auxiliary set.** 100 classes (the remaining 100 classes of "NTU RGB+D 120" excluding the 20 classes in evaluation set). 188 | 189 | ### Citation 190 | 191 | To cite our datasets, please use the following bibtex records: 192 | 193 | ``` 194 | @inproceedings{shahroudy2016ntu, 195 | title={NTU RGB+D: A large scale dataset for 3D human activity analysis}, 196 | author={Shahroudy, Amir and Liu, Jun and Ng, Tian-Tsong and Wang, Gang}, 197 | booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition}, 198 | pages={1010--1019}, 199 | year={2016} 200 | } 201 | 202 | @article{liu2020ntu, 203 | title={NTU RGB+D 120: A large-scale benchmark for 3D human activity understanding}, 204 | author={Liu, Jun and Shahroudy, Amir and Perez, Mauricio and Wang, Gang and Duan, Ling-Yu and Kot, Alex C}, 205 | journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 206 | volume={42}, 207 | number={10}, 208 | pages={2684--2701}, 209 | year={2020} 210 | } 211 | ``` 212 | 213 | ### Mailing List 214 | 215 | If you are interested to recieve news, updates, and future events about this dataset, please subscribe in the Google group of the dataset at: https://groups.google.com/d/forum/ntu-rgbd. 216 | If you cannot access the group's page, please email me, I will add your email to the list. 217 | You can find my email in the [paper](http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Shahroudy_NTU_RGBD_A_CVPR_2016_paper.pdf), and I'm the first author :) 218 | 219 | ### 3D Human Activity Analysis Challenge 220 | 221 | We organized the action recognition challenge "Large Scale 3D Human Activity Analysis Challenge in Depth Videos" based on the "NTU RGB+D" dataset in ACCV 2016. 222 | Details about this challenge can be found [here](https://rose1.ntu.edu.sg/challenge/ActionRecognitionChallenge/). 223 | 224 | ### Reported results on "NTU RGB+D" benchmark and "NTU RGB+D 120" benchmark 225 | 226 | Coming soon :) 227 | 228 | ### FAQs 229 | **(1) When I've sent the request for the dataset, I recieved '500 - Internal server error'.**
230 | *This error happens some of the time and is a technical problem in the hosting server. 231 | Please contact us via email, so that we can follow this up. 232 | The best person to email is the technician of our lab: Chai Ooy Mei (EOMChai@ntu.edu.sg). 233 | Please CC us (amir3@ntu.edu.sg, jliu029@ntu.edu.sg) in your email so that we can follow it up and ensure you will get access as soon as possible.* 234 | 235 | **(2) I want to align RGB and depth frames. Are there any camera calibration data recorded?**
236 | Unfortunately no camera calibration info is recorded. 237 | However, one applicable solution for this is to use the skeletal data. 238 | For each video sample, the skeletal data includes a big number of body joints and their precise locations in both RGB and depth frames. 239 | So for each sample you have a big number of mappings. 240 | Keep in mind that the cameras were fixed during each setup (Sxxx in the file names mean this sample is from setup xxx). 241 | So for each camera at each setup you have a huge number of mappings between RGB and depth cameras (and also between the three sensors!). 242 | Finding a transformation between the cameras will be as easy as solving a linear system with a lot of known points! 243 | 244 | **(3) There are some extra values recorded for each skeletal joint like orientation, lean, etc. What do they mean?**
245 | In almost all of the applications, the 3D locations of the joints are enough... 246 | We tried to keep everything generated by the SDK, so we recorded all. 247 | For more info about the meaning of those extra values, you can read this: 248 | https://medium.com/@lisajamhoury/understanding-kinect-v2-joints-and-coordinate-system-4f4b90b9df16 249 | 250 | **(4) What are masked depthmap in the download page?**
251 | The main purpose of providing masked depthmaps were to have a smaller sized version of the original depth maps. 252 | We used the position of the body skeletons to find regions of interest in depthmaps. 253 | We copied the depth values for the regions of interest (from the original depthmaps) and set the other regions' depth to zero. 254 | This helped to achieve a much more efficient frame-wise compression ratios. 255 | 256 | **(5) Why the individual and mutual actions are considered together? Isn't it better to separate them in our evaluations?**
257 | *Having these classes of human actions together is a part of our dataset design to cope with more realistic scenarios of human action analysis. Therefore, the ideal evalution should not provide any prior info about the type of the action.* 258 | 259 | **(6) How did you handle the variable subject numbers (one or two) in the input of the network?**
260 | *Our inputs initially includes two sets of joints (for two skeletons). 261 | When we observed just one, the second set was filled with zeros. 262 | When we observed two or more, we decided about which one to be the main subject and which one to be the second one, by measuring the amount of motion of their joints. 263 | Also, some of the detected skeletons are noise, like tables and seats. 264 | You can eliminate them by filtering out the skeletons that do not have reasonable Y spread over X spread values over all of their joints.* 265 | 266 | **(7) How did you choose the main actor in the preprocessing step?**
267 | *We used a heuristic method. It's very simple (but not necessarily correct for all the samples). 268 | We consider the variance of the X, Y, and Z values of all the joints and add them up. 269 | We took the body with the higher value as the main subject.* 270 | 271 | **(8) How important is the skeleton normalization step, described in experimantal setup section?**
272 | *In the extension of our experiments, we found out the normalization is not vital. 273 | You can skip the normalization step and it should work fine. 274 | Actually the network is supposed to learn how to normalize the data by itself.* 275 | 276 | **(9) The provided MATLAB code cannot read .avi files on my Linux machine.**
277 | *Most probably it's a missing codec problem. 278 | I used [this solution](http://askubuntu.com/questions/575869/how-do-i-install-gstreamer0-10-ffmpeg-on-ubuntu-14-10/707612#707612), and it worked on my own machine. Hope it would help you also.* 279 | --------------------------------------------------------------------------------