├── img └── overview.png ├── LICENSE ├── requirements.txt ├── README.md ├── utils ├── process_cme_SG.py └── process_SG.py ├── Data-process.py └── MoCos.py /img/overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Kali-Hac/MoCos/HEAD/img/overview.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Kali-Hac 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # This file may be used to create an environment using: 2 | # $ conda create --name --file 3 | # platform: linux-64 4 | _libgcc_mutex=0.1=main 5 | _tflow_select=2.1.0=gpu 6 | absl-py=0.12.0=py37h06a4308_0 7 | astor=0.8.1=py37h06a4308_0 8 | blas=1.0=mkl 9 | c-ares=1.17.1=h27cfd23_0 10 | ca-certificates=2021.5.25=h06a4308_1 11 | cached-property=1.5.2=py_0 12 | certifi=2020.12.5=py37h06a4308_0 13 | cffi=1.14.5=py37h261ae71_0 14 | coverage=5.5=py37h27cfd23_2 15 | cudatoolkit=9.0=h13b8566_0 16 | cudnn=7.6.5=cuda9.0_0 17 | cupti=9.0.176=0 18 | cython=0.29.23=py37h2531618_0 19 | decorator=4.4.2=pyhd3eb1b0_0 20 | faiss-gpu=1.6.3=py37h32c130d_0 21 | fastrlock=0.6=py37h2531618_0 22 | freetype=2.10.4=h5ab3b9f_0 23 | gast=0.4.0=py_0 24 | google-pasta=0.2.0=py_0 25 | grpcio=1.36.1=py37h2157cd5_1 26 | h5py=3.2.1=py37h6c542dc_0 27 | hdf5=1.10.6=hb1b8bf9_0 28 | importlib-metadata=3.10.0=py37h06a4308_0 29 | intel-openmp=2021.2.0=h06a4308_610 30 | jpeg=9b=h024ee3a_2 31 | keras-applications=1.0.8=py_1 32 | keras-preprocessing=1.1.2=pyhd3eb1b0_0 33 | lcms2=2.12=h3be6417_0 34 | ld_impl_linux-64=2.33.1=h53a641e_7 35 | libffi=3.3=he6710b0_2 36 | libgcc-ng=9.1.0=hdf63c60_0 37 | libgfortran-ng=7.3.0=hdf63c60_0 38 | libpng=1.6.37=hbc83047_0 39 | libprotobuf=3.14.0=h8c45485_0 40 | libstdcxx-ng=9.1.0=hdf63c60_0 41 | libtiff=4.1.0=h2733197_1 42 | lz4-c=1.9.3=h2531618_0 43 | markdown=3.3.4=py37h06a4308_0 44 | mkl=2021.2.0=h06a4308_296 45 | mkl-service=2.3.0=py37h27cfd23_1 46 | mkl_fft=1.3.0=py37h42c9631_2 47 | mkl_random=1.2.1=py37ha9443f7_2 48 | nccl=1.3.5=cuda9.0_0 49 | ncurses=6.2=he6710b0_1 50 | networkx=2.5.1=pyhd3eb1b0_0 51 | ninja=1.10.2=hff7bd54_1 52 | numpy=1.20.2=py37h2d18471_0 53 | numpy-base=1.20.2=py37hfae3a4d_0 54 | olefile=0.46=py_0 55 | openssl=1.1.1k=h27cfd23_0 56 | pillow=6.1.0=py37h34e0f95_0 57 | pip=21.1.1=py37h06a4308_0 58 | protobuf=3.14.0=py37h2531618_1 59 | pycparser=2.20=py_2 60 | python=3.7.10=hdb3f193_0 61 | pytorch=1.1.0=py3.7_cuda9.0.176_cudnn7.5.1_0 62 | readline=8.1=h27cfd23_0 63 | scipy=1.6.2=py37had2a1c9_1 64 | setuptools=52.0.0=py37h06a4308_0 65 | six=1.15.0=pyhd3eb1b0_0 66 | sqlite=3.35.4=hdfb4753_0 67 | tensorboard=1.14.0=py37hf484d3e_0 68 | tensorflow=1.14.0=gpu_py37hae64822_0 69 | tensorflow-base=1.14.0=gpu_py37h8f37b9b_0 70 | tensorflow-estimator=1.14.0=py_0 71 | tensorflow-gpu=1.14.0=h0d30ee6_0 72 | termcolor=1.1.0=py37h06a4308_1 73 | tk=8.6.10=hbc83047_0 74 | torchvision=0.3.0=py37_cu9.0.176_1 75 | typing_extensions=3.7.4.3=pyha847dfd_0 76 | werkzeug=1.0.1=pyhd3eb1b0_0 77 | wheel=0.36.2=pyhd3eb1b0_0 78 | wrapt=1.12.1=py37h7b6447c_1 79 | xz=5.2.5=h7b6447c_0 80 | zipp=3.4.1=pyhd3eb1b0_0 81 | zlib=1.2.11=h7b6447c_3 82 | zstd=1.4.9=haebb681_0 83 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![Python >=3.5](https://img.shields.io/badge/Python->=3.5-blue.svg) 2 | ![Tensorflow >=1.14.0](https://img.shields.io/badge/Tensorflow->=1.14.0-yellow.svg) 3 | ![Pytorch >=1.1.0](https://img.shields.io/badge/Pytorch->=1.1.0-green.svg) 4 | ![Faiss-gpu >= 1.6.3](https://img.shields.io/badge/Faiss->=1.6.3-orange.svg) 5 | 6 | # Motif Guided Graph Transformer with Combinatorial Skeleton Prototype Learning for Skeleton-Based Person Re-Identification 7 | By Haocong Rao and Chunyan Miao. In AAAI 2025 ([**Arxiv**](https://arxiv.org/abs/2412.09044)), 8 | 9 | ## Introduction 10 | This is the official implementation of MoCos presented by "Motif Guided Graph Transformer with Combinatorial Skeleton Prototype Learning for Skeleton-Based Person Re-Identification". The codes are used to reproduce experimental results of the proposed TranSG framework in the paper. 11 | 12 | ![image](https://github.com/Kali-Hac/MoCos/blob/main/img/overview.png) 13 | **Abstract**: 14 | 15 | Person re-identification (re-ID) via 3D skeleton data is a challenging task with significant value in many scenarios. Existing skeleton-based methods typically assume virtual motion relations between all joints, and adopt average joint or sequence representations for learning. However, they rarely explore key body structure and motion such as gait to focus on more important body joints or limbs, while lacking the ability to fully mine valuable spatial-temporal sub-patterns of skeletons to enhance model learning. This paper presents a generic Motif guided graph transformer with Combinatorial skeleton prototype learning (MoCos) that exploits *structure-specific* and *gait-related* body relations as well as combinatorial features of skeleton graphs to learn effective skeleton representations for person re-ID. In particular, motivated by the locality within joints' structure and the body-component collaboration in gait, we first propose the *motif guided graph transformer (MGT)* that incorporates hierarchical structural motifs and gait collaborative motifs, which simultaneously focuses on multi-order local joint correlations and key cooperative body parts to enhance skeleton relation learning. Then, we devise the *combinatorial skeleton prototype learning (CSP)* that leverages random spatial-temporal combinations of joint nodes and skeleton graphs to generate diverse *sub-skeleton* and *sub-tracklet* representations, which are contrasted with the most representative features (*prototypes*) of each identity to learn class-related semantics and discriminative skeleton representations. Extensive experiments validate the superior performance of MoCos over existing state-of-the-art models. We further show its generality under RGB-estimated skeletons, different graph modeling, and unsupervised scenarios. 16 | 17 | 18 | ## Environment 19 | - Python >= 3.5 20 | - Tensorflow-gpu >= 1.14.0 21 | - Pytorch >= 1.1.0 22 | - Faiss-gpu >= 1.6.3 23 | 24 | Here we provide a configuration file to install the extra requirements (if needed): 25 | ```bash 26 | conda install --file requirements.txt 27 | ``` 28 | 29 | **Note**: This file will not install tensorflow/tensorflow-gpu, faiss-gpu, pytroch/torch, please install them according to the cuda version of your graphic cards: [**Tensorflow**](https://www.tensorflow.org/install/pip), [**Pytorch**](https://pytorch.org/get-started/locally/). Take cuda 9.0 for example: 30 | ```bash 31 | conda install faiss-gpu cuda90 -c pytorch 32 | conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch 33 | conda install tensorflow==1.14 34 | conda install scikit-learn 35 | ``` 36 | 37 | ## Datasets and Models 38 | We provide three already **pre-processed datasets** (IAS-Lab, BIWI, KGBD) with various sequence lengths (**f=4/6/8/10/12**) [**here (pwd: 7je2)**](https://pan.baidu.com/s/1R7CEsyMJsEnZGFLqwvchBg) and the **pre-trained models** [**here (pwd: weww)**](https://pan.baidu.com/s/1DZtYtLVAbhZPtJ58POackg). Since we report the average performance of our approach on all datasets, here the provided models may produce better results than the paper.
39 | 40 | Please download the pre-processed datasets and model files while unzipping them to ``Datasets/`` and ``ReID_Models/`` folders in the current directory.
41 | 42 | **Note**: The access to the Vislab Multi-view KS20 dataset and large-scale RGB-based gait dataset CASIA-B are available upon request. If you have signed the license agreement and been granted the right to use them, please email us with the signed agreement and we will share the complete pre-processed KS20 and CASIA-B data. The original datasets can be downloaded here: [IAS-Lab](http://robotics.dei.unipd.it/reid/index.php/downloads), [BIWI](http://robotics.dei.unipd.it/reid/index.php/downloads), [KGBD](https://www.researchgate.net/publication/275023745_Kinect_Gait_Biometry_Dataset_-_data_from_164_individuals_walking_in_front_of_a_X-Box_360_Kinect_Sensor), [KS20](http://vislab.isr.ist.utl.pt/datasets/#ks20), [CASIA-B](http://www.cbsr.ia.ac.cn/english/Gait%20Databases.asp). We also provide the ``Data-process.py`` for directly transforming original datasets to the formatted training and testing data.
43 | 44 | ## Dataset Pre-Processing 45 | To (1) extract 3D skeleton sequences of length **f=6** from original datasets and (2) process them in a unified format (``.npy``) for the model inputs, please simply run the following command: 46 | ```bash 47 | python Data-process.py 6 48 | ``` 49 | **Note**: If you hope to preprocess manually (or *you can get the [already preprocessed data (pwd: 7je2)](https://pan.baidu.com/s/1R7CEsyMJsEnZGFLqwvchBg)*), please frist download and unzip the original datasets to the current directory with following folder structure: 50 | ```bash 51 | [Current Directory] 52 | ├─ BIWI 53 | │ ├─ Testing 54 | │ │ ├─ Still 55 | │ │ └─ Walking 56 | │ └─ Training 57 | ├─ IAS 58 | │ ├─ TestingA 59 | │ ├─ TestingB 60 | │ └─ Training 61 | ├─ KGBD 62 | │ └─ kinect gait raw dataset 63 | └─ KS20 64 | ├─ frontal 65 | ├─ left_diagonal 66 | ├─ left_lateral 67 | ├─ right_diagonal 68 | └─ right_lateral 69 | ``` 70 | After dataset preprocessing, the auto-generated folder structure of datasets is as follows: 71 | ```bash 72 | Datasets 73 | ├─ BIWI 74 | │ └─ 6 75 | │ ├─ test_npy_data 76 | │ │ ├─ Still 77 | │ │ └─ Walking 78 | │ └─ train_npy_data 79 | ├─ IAS 80 | │ └─ 6 81 | │ ├─ test_npy_data 82 | │ │ ├─ A 83 | │ │ └─ B 84 | │ └─ train_npy_data 85 | ├─ KGBD 86 | │ └─ 6 87 | │ ├─ test_npy_data 88 | │ │ ├─ gallery 89 | │ │ └─ probe 90 | │ └─ train_npy_data 91 | └─ KS20 92 | └─ 6 93 | ├─ test_npy_data 94 | │ ├─ gallery 95 | │ └─ probe 96 | └─ train_npy_data 97 | ``` 98 | **Note**: KS20 data need first transforming ".mat" to ".txt". If you are interested in the complete preprocessing of KS20 and CASIA-B, please contact us and we will share. We recommend to directly download the preprocessed data [**here (pwd: 7je2)**](https://pan.baidu.com/s/1R7CEsyMJsEnZGFLqwvchBg). 99 | 100 | ## Model Usage 101 | 102 | To (1) train MoCos to obtain skeleton representations and (2) validate their effectiveness on the person re-ID task on a specific dataset (probe), please simply run the following command: 103 | 104 | ```bash 105 | python MoCos.py --dataset KS20 --probe probe 106 | 107 | # Default options: --dataset KS20 --probe probe --length 6 --gpu 0 108 | # --dataset [IAS, KS20, BIWI, KGBD] 109 | # --probe ['probe' (the only probe for KS20 or KGBD), 'A' (for IAS-A probe), 'B' (for IAS-B probe), 'Walking' (for BIWI-Walking probe), 'Still' (for BIWI-Still probe)] 110 | # --length [4, 6, 8, 10, 12] 111 | # --(H, n_heads, L_transfomer, fusion_lambda, prob_s, prob_t, lr, etc.) with default settings for each dataset 112 | # --mode [Train (for training), Eval (for testing)] 113 | # --gpu [0, 1, ...] 114 | 115 | ``` 116 | Please see ```MoCos.py``` for more details. 117 | 118 | To print evaluation results (Top-1, Top-5, Top-10 Accuracy, mAP) of the best model saved in default directory (```ReID_Models/(Dataset)/(Probe)```), run: 119 | 120 | ```bash 121 | python MoCos.py --dataset KS20 --probe probe --mode Eval 122 | ``` 123 | 124 | 125 | ## Application to Model-Estimated Skeleton Data 126 | 127 | ### Estimate 3D Skeletons from RGB-Based Scenes 128 | To apply our MoCos to person re-ID under the large-scale RGB scenes (CASIA B), we exploit pose estimation methods to extract 3D skeletons from RGB videos of CASIA B as follows: 129 | - Step 1: Download [CASIA-B Dataset](http://www.cbsr.ia.ac.cn/english/Gait%20Databases.asp) 130 | - Step 2: Extract the 2D human body joints by using [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose) 131 | - Step 3: Estimate the 3D human body joints by using [3DHumanPose](https://github.com/flyawaychase/3DHumanPose) 132 | 133 | 134 | We provide already pre-processed skeleton data of CASIA B for **single-condition** (Nm-Nm, Cl-Cl, Bg-Bg) and **cross-condition evaluation** (Cl-Nm, Bg-Nm) (**f=40/50/60**) [**here (pwd: 07id)**](https://pan.baidu.com/s/1_Licrunki68r7F3EWQwYng). 135 | Please download the pre-processed datasets into the directory ``Datasets/``.
136 | 137 | ### Usage 138 | To (1) train the MoCos to obtain skeleton representations and (2) validate their effectiveness on the person re-ID task on CASIA B under **single-condition** and **cross-condition** settings, please simply run the following command: 139 | 140 | ```bash 141 | python MoCos.py --dataset CAISA_B --probe_type nm.nm --length 40 142 | 143 | # --length [40, 50, 60] 144 | # --probe_type ['nm.nm' (for 'Nm' probe and 'Nm' gallery), 'cl.cl', 'bg.bg', 'cl.nm' (for 'Cl' probe and 'Nm' gallery), 'bg.nm'] 145 | # --(H, n_heads, L_transfomer, fusion_lambda, prob_s, prob_t, lr, etc.) with default settings 146 | # --gpu [0, 1, ...] 147 | 148 | ``` 149 | 150 | Please see ```MoCos.py``` for more details. 151 | 152 | ## Citation 153 | If you find our work useful for your research, please cite our paper 154 | ```bash 155 | @inproceedings{rao2025motif, 156 | title = {Motif Guided Graph Transformer with Combinatorial Skeleton Prototype Learning for Skeleton-Based Person Re-Identification}, 157 | author = {Rao, Haocong and Miao, Chunyan}, 158 | booktitle = {Proceedings of the 39th Annual AAAI Conference on Artificial Intelligence (AAAI)}, 159 | year = {2025} 160 | } 161 | ``` 162 | A professionally curated list of resources (paper, code, data, etc.) on 3D Skeleton Based Person Re-ID (SRID) is available at [SRID Survey](https://github.com/Kali-Hac/SRID). 163 | 164 | More awesome skeleton-based models are collected in our [Awesome-Skeleton-Based-Models](https://github.com/Kali-Hac/Awesome-Skeleton-Based-Models). 165 | 166 | ## License 167 | 168 | MoCos is released under the MIT License. Our models and codes must only be used for the purpose of research. 169 | 170 | -------------------------------------------------------------------------------- /utils/process_cme_SG.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pickle as pkl 3 | import networkx as nx 4 | import scipy.sparse as sp 5 | import copy 6 | from scipy.sparse.linalg.eigen.arpack import eigsh 7 | import sys 8 | from sklearn.preprocessing import label_binarize 9 | from sklearn.metrics import roc_curve, auc 10 | 11 | """ 12 | Generate training data for each dataset. 13 | """ 14 | 15 | def gen_train_data(dataset, split, time_step, nb_nodes, nhood, global_att, batch_size, PG_type, reverse='0'): 16 | def get_data(dimension, fr): 17 | if reverse == '1': 18 | used_data = 'target' 19 | else: 20 | used_data = 'source' 21 | input_data = np.load( 22 | 'Datasets/' + frames_ps + dataset + '_train_npy_data/' + used_data + '_' + dimension + '_' + dataset + '_' + str( 23 | fr) + '.npy') 24 | input_data = input_data.reshape([-1, time_step, nb_nodes]) 25 | spine_pos = input_data[:, :, 0] 26 | spine_pos = np.expand_dims(spine_pos, -1) 27 | input_data = input_data - spine_pos 28 | if dataset == 'IAS': 29 | t_input_data = np.load( 30 | 'Datasets/' + frames_ps + dataset + '_test_npy_data/t_' + used_data + '_' + dimension + '_IAS-' + split + '_' + str( 31 | fr) + '.npy') 32 | else: 33 | t_input_data = np.load( 34 | 'Datasets/' + frames_ps + dataset + '_test_npy_data/' + PG_type + '/t_' + used_data + '_' + dimension + '_' + dataset + '_' + str( 35 | fr) + '.npy') 36 | t_input_data = t_input_data.reshape([-1, time_step, nb_nodes]) 37 | # Normalize 38 | t_spine_pos = t_input_data[:, :, 0] 39 | t_spine_pos = np.expand_dims(t_spine_pos, -1) 40 | t_input_data = t_input_data - t_spine_pos 41 | 42 | return input_data, t_input_data 43 | 44 | def unnormalized_laplacian(adj_matrix): 45 | R = np.sum(adj_matrix, axis=1) 46 | degreeMatrix = np.diag(R) 47 | return degreeMatrix - adj_matrix 48 | 49 | def normalized_laplacian(adj_matrix): 50 | R = np.sum(adj_matrix, axis=1) 51 | R_sqrt = 1 / np.sqrt(R) 52 | D_sqrt = np.diag(R_sqrt) 53 | I = np.eye(adj_matrix.shape[0]) 54 | return I - np.matmul(np.matmul(D_sqrt, adj_matrix), D_sqrt) 55 | 56 | # For CASIA B CME 57 | frames_ps = dataset + '_match/' + str(time_step) + '/' 58 | input_data_x, t_input_data_x = get_data('x', fr=time_step) 59 | input_data_y, t_input_data_y = get_data('y', fr=time_step) 60 | if dataset == 'OUMVLP': 61 | input_data_z, t_input_data_z = np.zeros_like(input_data_x), np.zeros_like(t_input_data_x) 62 | else: 63 | input_data_z, t_input_data_z = get_data('z', fr=time_step) 64 | 65 | X_train = np.concatenate([input_data_x, input_data_y, input_data_z], axis=-1) 66 | X_test = np.concatenate([t_input_data_x, t_input_data_y, t_input_data_z], axis=-1) 67 | 68 | ids = np.load( 69 | 'Datasets/' + frames_ps + dataset + '_train_npy_data/ids_' + dataset + '_' + str(time_step) + '.npy') 70 | ids = ids.item() 71 | if dataset == 'IAS': 72 | t_ids = np.load( 73 | 'Datasets/' + frames_ps + 'IAS_test_npy_data/ids_IAS-' + split + '_' + str(time_step) + '.npy') 74 | else: 75 | t_ids = np.load( 76 | 'Datasets/' + frames_ps + dataset + '_test_npy_data/' + PG_type + '/ids_' + dataset + '_' + str(time_step) + '.npy') 77 | t_ids = t_ids.item() 78 | 79 | y_train = np.load( 80 | 'Datasets/' + frames_ps + dataset + '_train_npy_data/frame_id_' + dataset + '_' + str(time_step) + '.npy') 81 | if dataset == 'IAS': 82 | y_test = np.load( 83 | 'Datasets/' + frames_ps + 'IAS_test_npy_data/frame_id_IAS-' + split + '_' + str(time_step) + '.npy') 84 | else: 85 | y_test = np.load( 86 | 'Datasets/' + frames_ps + dataset + '_test_npy_data/' + PG_type + '/frame_id_' + dataset + '_' + str(time_step) + '.npy') 87 | 88 | if dataset != 'OUMVLP': 89 | X_train, y_train = class_samp_gen(X_train, y_train, ids, batch_size) 90 | 91 | # randomly shuffle 92 | rand_p = np.random.permutation(X_train.shape[0]) 93 | X_train = X_train[rand_p] 94 | y_train = y_train[rand_p] 95 | # print(X_train.shape, y_train.shape) 96 | 97 | ids_keys = sorted(list(ids.keys())) 98 | classes = [i for i in ids_keys] 99 | y_train = label_binarize(y_train, classes=classes) 100 | t_ids_keys = sorted(list(t_ids.keys())) 101 | classes = [i for i in t_ids_keys] 102 | y_test = label_binarize(y_test, classes=classes) 103 | 104 | X_train_J = X_train.reshape([-1, time_step, 3, nb_nodes]) 105 | X_train_J = np.transpose(X_train_J, [0, 1, 3, 2]) 106 | X_train_P = reduce2part(X_train_J, nb_nodes) 107 | X_train_B = reduce2body(X_train_J, nb_nodes) 108 | X_train_H_B = reduce2h_body(X_train_J, nb_nodes) 109 | if dataset == 'KS20': 110 | X_train_In = interpolation(X_train_J, nb_nodes) 111 | 112 | X_test_J = X_test.reshape([-1, time_step, 3, nb_nodes]) 113 | X_test_J = np.transpose(X_test_J, [0, 1, 3, 2]) 114 | X_test_P = reduce2part(X_test_J, nb_nodes) 115 | X_test_B = reduce2body(X_test_J, nb_nodes) 116 | X_test_H_B = reduce2h_body(X_test_J, nb_nodes) 117 | if dataset == 'KS20': 118 | X_test_In = interpolation(X_test_J, nb_nodes) 119 | 120 | def generate_denser_adj(adj): 121 | adj_temp = copy.deepcopy(adj).tolist() 122 | node_num = len(adj_temp) 123 | new_adj = np.zeros([node_num * 2 - 1, node_num * 2 - 1]) 124 | cnt = node_num 125 | for i in range(node_num): 126 | for j in range(node_num): 127 | if adj_temp[i][j] == 1: 128 | new_adj[i, cnt] = new_adj[cnt, i] = new_adj[j, cnt] = new_adj[cnt, j] = 1 129 | adj_temp[i][j] = adj_temp[j][i] = 0 130 | # print(i, j, cnt) 131 | cnt += 1 132 | for i in range(node_num): 133 | for j in range(node_num): 134 | if adj_temp[i][j] == 1: 135 | assert new_adj[i, j] == new_adj[j, i] == 0 136 | if global_att: 137 | new_adj = np.ones([node_num*2-1, node_num*2-1]) 138 | # print(cnt) 139 | # print(new_adj) 140 | return new_adj 141 | import scipy.sparse 142 | if dataset == 'KS20': 143 | # Joint-Level adjacent matrix 144 | j_pair_1 = np.array([3, 2, 20, 8, 8, 9, 10, 9, 11, 10, 4, 20, 4, 5, 5, 6, 6, 7, 1, 20, 1, 0, 16, 0, 145 | 12, 0, 16, 17, 12, 13, 17, 18, 19, 18, 13, 14, 14, 15, 2, 20, 11, 23, 10, 24, 7, 21, 6, 22]) 146 | j_pair_2 = np.array([2, 3, 8, 20, 9, 8, 9, 10, 10, 11, 20, 4, 5, 4, 6, 5, 7, 6, 20, 1, 0, 1, 0, 16, 147 | 0, 12, 17, 16, 13, 12, 18, 17, 18, 19, 14, 13, 15, 14, 20, 2, 23, 11, 24, 10, 21, 7, 22, 6]) 148 | con_matrix = np.ones([48]) 149 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 150 | if global_att: 151 | adj_joint = np.ones([25, 25]) 152 | # hyper-Level (interpolation) adjacent matrix, NOT used, replaced by generate_denser_adj 153 | # i_pair_1 = np.array( 154 | # [3, 31, 31, 2, 2, 32, 32, 20, 20, 42, 42, 8, 8, 37, 37, 9, 9, 44, 44, 10, 10, 38, 38, 11, 11, 155 | # 39, 39, 23, 10, 40, 40, 24, 20, 41, 41, 4, 4, 33, 33, 5, 5, 43, 43, 6, 6, 34, 34, 7, 7, 35, 156 | # 35, 21, 6, 36, 36, 22, 157 | # 20, 45, 45, 1, 1, 30, 30, 0, 0, 46, 46, 12, 12, 26, 26, 13, 13, 48, 48, 14, 14, 27, 27, 15, 158 | # 0, 47, 47, 16, 16, 28, 28, 17, 17, 49, 49, 18, 18, 29, 29, 19]) 159 | # # miss 25 160 | # i_pair_1 = i_pair_1.tolist() 161 | # for i in range(len(i_pair_1)): 162 | # if i_pair_1[i] > 24: 163 | # i_pair_1[i] -= 1 164 | # i_pair_1 = np.array(i_pair_1) 165 | # i_pair_2 = np.array( 166 | # [31, 3, 2, 31, 32, 2, 20, 32, 42, 20, 8, 42, 37, 8, 9, 37, 44, 9, 10, 44, 38, 10, 11, 38, 39, 167 | # 11, 23, 39, 40, 10, 24, 40, 41, 20, 4, 41, 33, 4, 5, 33, 43, 5, 6, 43, 34, 6, 7, 34, 35, 7, 168 | # 21, 35, 36, 6, 22, 36, 169 | # 45, 20, 1, 45, 30, 1, 0, 30, 46, 0, 12, 46, 26, 12, 13, 26, 48, 13, 14, 48, 27, 14, 15, 27, 170 | # 47, 0, 16, 47, 28, 16, 17, 28, 49, 17, 18, 49, 29, 18, 19, 29]) 171 | # i_pair_2 = i_pair_2.tolist() 172 | # for i in range(len(i_pair_2)): 173 | # if i_pair_2[i] > 24: 174 | # i_pair_2[i] -= 1 175 | # i_pair_2 = np.array(i_pair_2) 176 | # # print(i_pair_1.shape, i_pair_2.shape) 177 | # con_matrix = np.ones([96]) 178 | # adj_interp = scipy.sparse.coo_matrix((con_matrix, (i_pair_1, i_pair_2)), 179 | # shape=(49, 49)).toarray() 180 | adj_interp = generate_denser_adj(adj_joint) 181 | 182 | 183 | elif dataset == 'CASIA_B': 184 | # Joint-Level adjacent matrix 185 | j_pair_1 = np.array([0, 1, 1, 2, 2, 3, 3, 4, 1, 5, 5, 6, 6, 7, 1, 8, 8, 9, 9, 10, 1, 11, 11, 12, 12, 13]) 186 | j_pair_2 = np.array([1, 0, 2, 1, 3, 2, 4, 3, 5, 1, 6, 5, 7, 6, 8, 1, 9, 8, 10, 9, 11, 1, 12, 11, 13, 12]) 187 | con_matrix = np.ones([26]) 188 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 189 | if global_att: 190 | adj_joint = np.ones([14, 14]) 191 | # adj_interp = generate_denser_adj(adj_joint) 192 | elif dataset == 'OUMVLP': 193 | # Joint-Level adjacent matrix 194 | j_pair_1 = np.array([0, 1, 1, 2, 2, 3, 3, 4, 1, 5, 5, 6, 6, 7, 1, 8, 8, 9, 9, 10, 1, 11, 11, 12, 12, 13, 14, 0, 15, 0, 195 | 15, 17, 16, 14]) 196 | j_pair_2 = np.array([1, 0, 2, 1, 3, 2, 4, 3, 5, 1, 6, 5, 7, 6, 8, 1, 9, 8, 10, 9, 11, 1, 12, 11, 13, 12, 0, 14, 0, 15, 197 | 17, 15, 14, 16]) 198 | con_matrix = np.ones([34]) 199 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 200 | if global_att: 201 | adj_joint = np.ones([18, 18]) 202 | adj_interp = generate_denser_adj(adj_joint) 203 | else: 204 | # Joint-Level adjacent matrix 205 | j_pair_1 = np.array([3, 2, 2, 8, 8, 9, 10, 9, 11, 10, 4, 2, 4, 5, 5, 6, 6, 7, 1, 2, 1, 0, 16, 0, 206 | 12, 0, 16, 17, 12, 13, 17, 18, 19, 18, 13, 14, 14, 15]) 207 | j_pair_2 = np.array([2, 3, 8, 2, 9, 8, 9, 10, 10, 11, 2, 4, 5, 4, 6, 5, 7, 6, 2, 1, 0, 1, 0, 16, 208 | 0, 12, 17, 16, 13, 12, 18, 17, 18, 19, 14, 13, 15, 14]) 209 | con_matrix = np.ones([38]) 210 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 211 | if global_att: 212 | adj_joint = np.ones([20, 20]) 213 | adj_interp = generate_denser_adj(adj_joint) 214 | 215 | # compute Laplacian matrix, use the k smallest non-trivial eigen-vectors 216 | k = 10 217 | L = normalized_laplacian(adj_joint) 218 | # only real 219 | EigVal, EigVec = np.linalg.eig(L) 220 | idx = EigVal.argsort() 221 | EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 222 | # print(EigVal.shape, EigVec.shape) 223 | # exit() 224 | pos_enc_ori = EigVec[:, 1:k + 1] 225 | 226 | # Part-Level adjacent matrix 227 | p_pair_1 = np.array([5, 6, 5, 8, 6, 7, 8, 9, 5, 4, 4, 2, 4, 0, 2, 3, 1, 0]) 228 | p_pair_2 = np.array([6, 5, 8, 5, 7, 6, 9, 8, 4, 5, 2, 4, 0, 4, 3, 2, 0, 1]) 229 | con_matrix = np.ones([18]) 230 | adj_part = scipy.sparse.coo_matrix((con_matrix, (p_pair_1, p_pair_2)), shape=(10, 10)).toarray() 231 | 232 | # Body-Level adjacent matrix 233 | b_pair_1 = np.array([2, 3, 2, 4, 2, 1, 2, 0]) 234 | b_pair_2 = np.array([3, 2, 4, 2, 1, 2, 0, 2]) 235 | con_matrix = np.ones([8]) 236 | adj_body = scipy.sparse.coo_matrix((con_matrix, (b_pair_1, b_pair_2)), shape=(5, 5)).toarray() 237 | 238 | # Hyper-Body-Level adjacent matrix 239 | # h_b_pair_1 = np.array([0, 1, 2, 1]) 240 | # h_b_pair_2 = np.array([1, 0, 1, 2]) 241 | # con_matrix = np.ones([4]) 242 | # adj_hyper_body = scipy.sparse.coo_matrix((con_matrix, (h_b_pair_1, h_b_pair_2)), shape=(3, 3)).toarray() 243 | 244 | # compute Laplacian matrix for part-level and body-level, use the k (1/2 node number) smallest non-trivial eigen-vectors 245 | # part-level 246 | k = 9 247 | L = normalized_laplacian(adj_part) 248 | # only real 249 | EigVal, EigVec = np.linalg.eig(L) 250 | idx = EigVal.argsort() 251 | EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 252 | pos_enc_part = EigVec[:, 1:k + 1] 253 | 254 | # body-level 255 | k = 4 256 | L = normalized_laplacian(adj_body) 257 | # only real 258 | EigVal, EigVec = np.linalg.eig(L) 259 | idx = EigVal.argsort() 260 | EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 261 | pos_enc_body = EigVec[:, 1:k + 1] 262 | 263 | if global_att: 264 | adj_part = np.ones([10, 10]) 265 | adj_body = np.ones([5, 5]) 266 | 267 | # if dataset != 'KS20': 268 | # X_train_In = generate_denser_graph_data(X_train_J, adj_joint, nb_nodes) 269 | # X_test_In = generate_denser_graph_data(X_test_J, adj_joint, nb_nodes) 270 | 271 | if dataset == 'IAS': 272 | nb_classes = 11 273 | elif dataset == 'KGBD': 274 | nb_classes = 164 275 | elif dataset == 'BIWI': 276 | nb_classes = 28 277 | elif dataset == 'KS20': 278 | nb_classes = 20 279 | elif dataset == 'CASIA_B': 280 | nb_classes = 62 281 | elif dataset == 'OUMVLP': 282 | nb_classes = 10295 283 | 284 | adj_joint = adj_joint[np.newaxis] 285 | biases_joint = adj_to_bias(adj_joint, [nb_nodes], nhood=nhood) 286 | 287 | # adj_part = adj_part[np.newaxis] 288 | # biases_part = adj_to_bias(adj_part, [10], nhood=1) 289 | # 290 | # adj_body = adj_body[np.newaxis] 291 | # biases_body = adj_to_bias(adj_body, [5], nhood=1) 292 | 293 | # adj_interp = adj_interp[np.newaxis] 294 | # biases_interp = adj_to_bias(adj_interp, [nb_nodes*2-1], nhood=1) 295 | 296 | # adj_hyper_body = adj_hyper_body[np.newaxis] 297 | # biases_hyper_body = adj_to_bias(adj_hyper_body, [3], nhood=1) 298 | 299 | # return X_train_J, X_train_P, X_train_B, X_train_H_B, X_train_In, y_train, X_test_J, X_test_P, X_test_B, X_test_H_B, X_test_In, y_test, \ 300 | # adj_joint, biases_joint, adj_part, biases_part, adj_body, biases_body, adj_hyper_body, biases_hyper_body, adj_interp, biases_interp, nb_classes 301 | # return X_train_J, X_train_P, X_train_B, 0, 0, y_train, X_test_J, X_test_P, X_test_B, 0, 0, y_test, \ 302 | # adj_joint, biases_joint, pos_enc_ori, 0, 0, 0, 0, 0, 0, 0, 0, nb_classes 303 | 304 | # return X_train_J, 0, 0, 0, 0, y_train, X_test_J, 0, 0, 0, 0, y_test, \ 305 | # adj_joint, biases_joint, pos_enc_ori, 0, 0, 0, 0, 0, 0, 0, 0, nb_classes 306 | return X_train_J, X_train_P, X_train_B, 0, 0, y_train, X_test_J, X_test_P, X_test_B, 0, 0, y_test, \ 307 | adj_joint, 0, pos_enc_ori, adj_part, 0, pos_enc_part, adj_body, 0, pos_enc_body, 0, 0, 0, 0, nb_classes 308 | 309 | """ 310 | Generate part-level skeleton graphs. 311 | """ 312 | 313 | def reduce2part(X, joint_num=20): 314 | if joint_num == 25: 315 | left_leg_up = [12, 13] 316 | left_leg_down = [14, 15] 317 | right_leg_up = [16, 17] 318 | right_leg_down = [18, 19] 319 | torso = [0, 1] 320 | head = [2, 3, 20] 321 | left_arm_up = [4, 5] 322 | left_arm_down = [6, 7, 21, 22] 323 | right_arm_up = [8, 9] 324 | right_arm_down = [10, 11, 23, 24] 325 | elif joint_num == 20: 326 | left_leg_up = [12, 13] 327 | left_leg_down = [14, 15] 328 | right_leg_up = [16, 17] 329 | right_leg_down = [18, 19] 330 | torso = [0, 1] 331 | head = [2, 3] 332 | left_arm_up = [4, 5] 333 | left_arm_down = [6, 7] 334 | right_arm_up = [8, 9] 335 | right_arm_down = [10, 11] 336 | elif joint_num == 14: 337 | left_leg_up = [11] 338 | left_leg_down = [12, 13] 339 | right_leg_up = [8] 340 | right_leg_down = [9, 10] 341 | torso = [1] 342 | head = [0] 343 | left_arm_up = [5] 344 | left_arm_down = [6, 7] 345 | right_arm_up = [2] 346 | right_arm_down = [3, 4] 347 | elif joint_num == 18: 348 | left_leg_up = [11] 349 | left_leg_down = [12, 13] 350 | right_leg_up = [8] 351 | right_leg_down = [9, 10] 352 | torso = [1] 353 | head = [0, 14, 15, 16, 17] 354 | left_arm_up = [5] 355 | left_arm_down = [6, 7] 356 | right_arm_up = [2] 357 | right_arm_down = [3, 4] 358 | 359 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 360 | x_leftlegup = np.mean(X[:, :, left_leg_up, :], axis=2) 361 | x_leftlegdown = np.mean(X[:, :, left_leg_down, :], axis=2) 362 | x_rightlegup = np.mean(X[:, :, right_leg_up, :], axis=2) 363 | x_rightlegdown = np.mean(X[:, :, right_leg_down, :], axis=2) 364 | x_head = np.mean(X[:, :, head, :], axis=2) 365 | x_leftarmup = np.mean(X[:, :, left_arm_up, :], axis=2) 366 | x_leftarmdown = np.mean(X[:, :, left_arm_down, :], axis=2) 367 | x_rightarmup = np.mean(X[:, :, right_arm_up, :], axis=2) 368 | x_rightarmdown = np.mean(X[:, :, right_arm_down, :], axis=2) 369 | X_part = np.concatenate((x_leftlegup, x_leftlegdown, x_rightlegup, x_rightlegdown, x_torso, x_head, x_leftarmup, 370 | x_leftarmdown, x_rightarmup, x_rightarmdown), axis=-1) \ 371 | .reshape([X.shape[0], X.shape[1], 10, 3]) 372 | return X_part 373 | 374 | """ 375 | Generate body-level skeleton graphs. 376 | """ 377 | 378 | def reduce2body(X, joint_num=20): 379 | if joint_num == 25: 380 | left_leg = [12, 13, 14, 15] 381 | right_leg = [16, 17, 18, 19] 382 | torso = [0, 1, 2, 3, 20] 383 | left_arm = [4, 5, 6, 7, 21, 22] 384 | right_arm = [8, 9, 10, 11, 23, 24] 385 | elif joint_num == 20: 386 | left_leg = [12, 13, 14, 15] 387 | right_leg = [16, 17, 18, 19] 388 | torso = [0, 1, 2, 3] 389 | left_arm = [4, 5, 6, 7] 390 | right_arm = [8, 9, 10, 11] 391 | elif joint_num == 14: 392 | left_leg = [11, 12, 13] 393 | right_leg = [8, 9, 10] 394 | torso = [0, 1] 395 | left_arm = [5, 6, 7] 396 | right_arm = [2, 3, 4] 397 | elif joint_num == 18: 398 | left_leg = [11, 12, 13] 399 | right_leg = [8, 9, 10] 400 | torso = [0, 1, 14, 15, 16, 17] 401 | left_arm = [5, 6, 7] 402 | right_arm = [2, 3, 4] 403 | 404 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 405 | x_leftleg = np.mean(X[:, :, left_leg, :], axis=2) 406 | x_rightleg = np.mean(X[:, :, right_leg, :], axis=2) 407 | x_leftarm = np.mean(X[:, :, left_arm, :], axis=2) 408 | x_rightarm = np.mean(X[:, :, right_arm, :], axis=2) 409 | X_body = np.concatenate((x_leftleg, x_rightleg, x_torso, x_leftarm, x_rightarm), axis=-1)\ 410 | .reshape([X.shape[0], X.shape[1], 5, 3]) 411 | return X_body 412 | 413 | """ 414 | Generate hyper-body-level skeleton graphs. 415 | """ 416 | 417 | def reduce2h_body(X, joint_num=20): 418 | if joint_num == 25: 419 | # left_leg = [12, 13, 14, 15] 420 | # right_leg = [16, 17, 18, 19] 421 | # torso = [0, 1, 2, 3, 20] 422 | # left_arm = [4, 5, 6, 7, 21, 22] 423 | # right_arm = [8, 9, 10, 11, 23, 24] 424 | upper = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 20, 21, 22, 23, 24] 425 | middle = [0] 426 | lower = [12, 13, 14, 15, 16, 17, 18, 19] 427 | elif joint_num == 20: 428 | # left_leg = [12, 13, 14, 15] 429 | # right_leg = [16, 17, 18, 19] 430 | # torso = [0, 1, 2, 3] 431 | # left_arm = [4, 5, 6, 7] 432 | # right_arm = [8, 9, 10, 11] 433 | upper = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 434 | middle = [0] 435 | lower = [12, 13, 14, 15, 16, 17, 18, 19] 436 | elif joint_num == 14: 437 | # left_leg = [11, 12, 13] 438 | # right_leg = [8, 9, 10] 439 | # torso = [0, 1] 440 | # left_arm = [5, 6, 7] 441 | # right_arm = [2, 3, 4] 442 | upper = [0, 1, 2, 3, 4, 5, 6, 7] 443 | middle = [8, 11] 444 | lower = [9, 10, 12, 13] 445 | elif joint_num == 18: 446 | # left_leg = [11, 12, 13] 447 | # right_leg = [8, 9, 10] 448 | # torso = [0, 1] 449 | # left_arm = [5, 6, 7] 450 | # right_arm = [2, 3, 4] 451 | upper = [0, 1, 2, 3, 4, 5, 6, 7, 14, 15, 16, 17] 452 | middle = [8, 11] 453 | lower = [9, 10, 12, 13] 454 | 455 | x_upper = np.mean(X[:, :, upper, :], axis=2) # [N * T, V=1] 456 | x_middle = np.mean(X[:, :, middle, :], axis=2) 457 | x_lower = np.mean(X[:, :, lower, :], axis=2) 458 | X_body = np.concatenate((x_upper, x_middle, x_lower), axis=-1)\ 459 | .reshape([X.shape[0], X.shape[1], 3, 3]) 460 | return X_body 461 | 462 | """ 463 | Linear interpolation 464 | """ 465 | 466 | def interpolation(X, joint_num=20): 467 | if joint_num == 25: 468 | left_leg_up = [12, 13] 469 | left_leg_down = [14, 15] 470 | right_leg_up = [16, 17] 471 | right_leg_down = [18, 19] 472 | torso = [0, 1] 473 | head_1 = [2, 3] 474 | head_2 = [2, 20] 475 | left_arm_up = [4, 5] 476 | left_arm_down_1 = [6, 7] 477 | left_arm_down_2 = [7, 21] 478 | left_arm_down_3 = [6, 22] 479 | right_arm_up = [8, 9] 480 | right_arm_down_1 = [10, 11] 481 | right_arm_down_2 = [11, 23] 482 | right_arm_down_3 = [10, 24] 483 | shoulder_1 = [4, 20] 484 | shoulder_2 = [8, 20] 485 | elbow_1 = [5, 6] 486 | elbow_2 = [9, 10] 487 | spine_mm = [20, 1] 488 | hip_1 = [0, 12] 489 | hip_2 = [0, 16] 490 | knee_1 = [13, 14] 491 | knee_2 = [17, 18] 492 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 493 | x_leftlegup = np.mean(X[:, :, left_leg_up, :], axis=2) 494 | x_leftlegdown = np.mean(X[:, :, left_leg_down, :], axis=2) 495 | x_rightlegup = np.mean(X[:, :, right_leg_up, :], axis=2) 496 | x_rightlegdown = np.mean(X[:, :, right_leg_down, :], axis=2) 497 | x_head_1 = np.mean(X[:, :, head_1, :], axis=2) 498 | x_head_2 = np.mean(X[:, :, head_2, :], axis=2) 499 | x_leftarmup = np.mean(X[:, :, left_arm_up, :], axis=2) 500 | x_leftarmdown_1 = np.mean(X[:, :, left_arm_down_1, :], axis=2) 501 | x_leftarmdown_2 = np.mean(X[:, :, left_arm_down_2, :], axis=2) 502 | x_leftarmdown_3 = np.mean(X[:, :, left_arm_down_3, :], axis=2) 503 | x_rightarmup = np.mean(X[:, :, right_arm_up, :], axis=2) 504 | x_rightarmdown_1 = np.mean(X[:, :, right_arm_down_1, :], axis=2) 505 | x_rightarmdown_2 = np.mean(X[:, :, right_arm_down_2, :], axis=2) 506 | x_rightarmdown_3 = np.mean(X[:, :, right_arm_down_3, :], axis=2) 507 | shoulder_1 = np.mean(X[:, :, shoulder_1, :], axis=2) 508 | shoulder_2 = np.mean(X[:, :, shoulder_2, :], axis=2) 509 | elbow_1 = np.mean(X[:, :, elbow_1, :], axis=2) 510 | elbow_2 = np.mean(X[:, :, elbow_2, :], axis=2) 511 | spine_mm = np.mean(X[:, :, spine_mm, :], axis=2) 512 | hip_1 = np.mean(X[:, :, hip_1, :], axis=2) 513 | hip_2 = np.mean(X[:, :, hip_2, :], axis=2) 514 | knee_1 = np.mean(X[:, :, knee_1, :], axis=2) 515 | knee_2 = np.mean(X[:, :, knee_2, :], axis=2) 516 | X_part = np.concatenate((x_leftlegup, x_leftlegdown, x_rightlegup, 517 | x_rightlegdown, x_torso, x_head_1, x_head_2, x_leftarmup, 518 | x_leftarmdown_1, x_leftarmdown_2, x_leftarmdown_3, 519 | x_rightarmup, x_rightarmdown_1, x_rightarmdown_2, x_rightarmdown_3, 520 | shoulder_1, shoulder_2, elbow_1, elbow_2, spine_mm, 521 | hip_1, hip_2, knee_1, knee_2), axis=-1) \ 522 | .reshape([X.shape[0], X.shape[1], 24, 3]) 523 | # 25+24 524 | X_interp = np.concatenate((X, X_part), axis=-2) 525 | # Changed 526 | elif joint_num == 20: 527 | # left_leg_up = [12, 13] 528 | # left_leg_down = [14, 15] 529 | # right_leg_up = [16, 17] 530 | # right_leg_down = [18, 19] 531 | # torso = [0, 1] 532 | # head = [2, 3] 533 | # left_arm_up = [4, 5] 534 | # left_arm_down = [6, 7] 535 | # right_arm_up = [8, 9] 536 | # right_arm_down = [10, 11] 537 | # 538 | left_leg_up = [12, 13] 539 | left_leg_down = [14, 15] 540 | right_leg_up = [16, 17] 541 | right_leg_down = [18, 19] 542 | torso = [0, 1] 543 | head_1 = [2, 3] 544 | # head_2 = [2, 20] 545 | left_arm_up = [4, 5] 546 | left_arm_down_1 = [6, 7] 547 | # left_arm_down_2 = [7, 21] 548 | # left_arm_down_3 = [6, 22] 549 | right_arm_up = [8, 9] 550 | right_arm_down_1 = [10, 11] 551 | # right_arm_down_2 = [11, 23] 552 | # right_arm_down_3 = [10, 24] 553 | # shoulder_1 = [4, 20] 554 | # shoulder_2 = [8, 20] 555 | shoulder_1 = [4, 2] 556 | shoulder_2 = [8, 2] 557 | elbow_1 = [5, 6] 558 | elbow_2 = [9, 10] 559 | # spine_mm = [20, 1] 560 | spine_mm = [2, 1] 561 | hip_1 = [0, 12] 562 | hip_2 = [0, 16] 563 | knee_1 = [13, 14] 564 | knee_2 = [17, 18] 565 | 566 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 567 | x_leftlegup = np.mean(X[:, :, left_leg_up, :], axis=2) 568 | x_leftlegdown = np.mean(X[:, :, left_leg_down, :], axis=2) 569 | x_rightlegup = np.mean(X[:, :, right_leg_up, :], axis=2) 570 | x_rightlegdown = np.mean(X[:, :, right_leg_down, :], axis=2) 571 | x_head_1 = np.mean(X[:, :, head_1, :], axis=2) 572 | # x_head_2 = np.mean(X[:, :, head_2, :], axis=2) 573 | x_leftarmup = np.mean(X[:, :, left_arm_up, :], axis=2) 574 | x_leftarmdown_1 = np.mean(X[:, :, left_arm_down_1, :], axis=2) 575 | # x_leftarmdown_2 = np.mean(X[:, :, left_arm_down_2, :], axis=2) 576 | # x_leftarmdown_3 = np.mean(X[:, :, left_arm_down_3, :], axis=2) 577 | x_rightarmup = np.mean(X[:, :, right_arm_up, :], axis=2) 578 | x_rightarmdown_1 = np.mean(X[:, :, right_arm_down_1, :], axis=2) 579 | # x_rightarmdown_2 = np.mean(X[:, :, right_arm_down_2, :], axis=2) 580 | # x_rightarmdown_3 = np.mean(X[:, :, right_arm_down_3, :], axis=2) 581 | shoulder_1 = np.mean(X[:, :, shoulder_1, :], axis=2) 582 | shoulder_2 = np.mean(X[:, :, shoulder_2, :], axis=2) 583 | elbow_1 = np.mean(X[:, :, elbow_1, :], axis=2) 584 | elbow_2 = np.mean(X[:, :, elbow_2, :], axis=2) 585 | spine_mm = np.mean(X[:, :, spine_mm, :], axis=2) 586 | hip_1 = np.mean(X[:, :, hip_1, :], axis=2) 587 | hip_2 = np.mean(X[:, :, hip_2, :], axis=2) 588 | knee_1 = np.mean(X[:, :, knee_1, :], axis=2) 589 | knee_2 = np.mean(X[:, :, knee_2, :], axis=2) 590 | X_part = np.concatenate((x_leftlegup, x_leftlegdown, x_rightlegup, 591 | x_rightlegdown, x_torso, x_head_1, x_leftarmup, 592 | x_leftarmdown_1, 593 | x_rightarmup, x_rightarmdown_1, 594 | shoulder_1, shoulder_2, elbow_1, elbow_2, spine_mm, 595 | hip_1, hip_2, knee_1, knee_2), axis=-1) \ 596 | .reshape([X.shape[0], X.shape[1], 19, 3]) 597 | # 25+24 598 | X_interp = np.concatenate((X, X_part), axis=-2) 599 | return X_interp 600 | 601 | def generate_denser_graph_data(X, adj, joint_num=20): 602 | adj_temp = copy.deepcopy(adj) 603 | adj_temp = adj_temp.tolist() 604 | node_num = len(adj_temp) 605 | cnt = node_num 606 | for i in range(node_num): 607 | for j in range(node_num): 608 | if adj_temp[i][j] == 1: 609 | adj_temp[i][j] = adj_temp[j][i] = 0 610 | new_node = np.mean(X[:, :, [i, j], :], axis=2) 611 | # print(new_node.shape) 612 | if cnt == node_num: 613 | X_interp = new_node 614 | else: 615 | X_interp = np.concatenate((X_interp, new_node), axis=-1) 616 | # print(X_interp.shape) 617 | # print(i, j) 618 | # print(i, j, cnt) 619 | cnt += 1 620 | # print(X_interp.shape) 621 | X_interp = np.reshape(X_interp, [X.shape[0], X.shape[1], node_num-1, 3]) 622 | X_interp = np.concatenate((X, X_interp), axis=-2) 623 | return X_interp 624 | 625 | """ 626 | Calculate normalized area under curves. 627 | """ 628 | def cal_nAUC(scores, labels): 629 | scores = np.array(scores) 630 | labels = np.array(labels) 631 | # Compute micro-average ROC curve and ROC area 632 | fpr, tpr, thresholds = roc_curve(labels.ravel(), scores.ravel()) 633 | roc_auc = auc(fpr, tpr) 634 | return roc_auc 635 | 636 | """ 637 | Generate training data with evenly distributed classes. 638 | """ 639 | # replaced by random shuffling 640 | def class_samp_gen(X, y, ids_, batch_size): 641 | class_num = len(ids_.keys()) 642 | ids_ = sorted(ids_.items(), key=lambda item: item[0]) 643 | cnt = 0 644 | all_batch_X = [] 645 | all_batch_y = [] 646 | total = y.shape[0] 647 | batch_num = total // batch_size * 2 648 | batch_num = total // batch_size * 2 649 | class_in_bacth = class_num 650 | batch_per_class = batch_size // class_in_bacth 651 | class_cnt = class_in_bacth 652 | # print(total, batch_num, batch_per_class) 653 | for i in range(batch_num): 654 | batch_X = [] 655 | batch_y = [] 656 | for k, v in ids_[class_cnt-class_in_bacth:class_cnt]: 657 | # print(k, len(v)) 658 | # cnt += len(v) 659 | if len(v[batch_per_class*i:batch_per_class*(i+1)]) < batch_per_class: 660 | rand_ind = np.random.choice(len(v), batch_per_class) 661 | v_array = np.array(v) 662 | samp_per_class = v_array[rand_ind].tolist() 663 | batch_X.extend(samp_per_class) 664 | else: 665 | batch_X.extend(v[batch_per_class*i:batch_per_class*(i+1)]) 666 | batch_y.extend(batch_per_class * [k]) 667 | if class_cnt + class_in_bacth > class_num and class_cnt <= class_num: 668 | class_cnt = class_num 669 | else: 670 | class_cnt = class_cnt + class_in_bacth 671 | all_batch_X.extend(batch_X) 672 | all_batch_y.extend(batch_y) 673 | # print(len(all_batch_X), len(all_batch_y)) 674 | X_train = X[all_batch_X] 675 | y_train = np.array(all_batch_y) 676 | return X_train, y_train 677 | 678 | """ 679 | Prepare adjacency matrix by expanding up to a given neighbourhood. 680 | This will insert loops on every node. 681 | Finally, the matrix is converted to bias vectors. 682 | Expected shape: [graph, nodes, nodes] 683 | """ 684 | 685 | 686 | def adj_to_bias(adj, sizes, nhood=1): 687 | nb_graphs = adj.shape[0] 688 | mt = np.empty(adj.shape) 689 | for g in range(nb_graphs): 690 | mt[g] = np.eye(adj.shape[1]) 691 | for _ in range(nhood): 692 | mt[g] = np.matmul(mt[g], (adj[g] + np.eye(adj.shape[1]))) 693 | for i in range(sizes[g]): 694 | for j in range(sizes[g]): 695 | if mt[g][i][j] > 0.0: 696 | mt[g][i][j] = 1.0 697 | return -1e9 * (1.0 - mt) 698 | 699 | 700 | ############################################### 701 | # This section of code adapted from tkipf/gcn # 702 | ############################################### 703 | 704 | def parse_index_file(filename): 705 | """Parse index file.""" 706 | index = [] 707 | for line in open(filename): 708 | index.append(int(line.strip())) 709 | return index 710 | 711 | 712 | def sample_mask(idx, l): 713 | """Create mask.""" 714 | mask = np.zeros(l) 715 | mask[idx] = 1 716 | return np.array(mask, dtype=np.bool) 717 | 718 | 719 | def load_data(dataset_str): # {'pubmed', 'citeseer', 'cora'} 720 | """Load data.""" 721 | names = ['x', 'y', 'tx', 'ty', 'allx', 'ally', 'graph'] 722 | objects = [] 723 | for i in range(len(names)): 724 | with open("data/ind.{}.{}".format(dataset_str, names[i]), 'rb') as f: 725 | if sys.version_info > (3, 0): 726 | objects.append(pkl.load(f, encoding='latin1')) 727 | else: 728 | objects.append(pkl.load(f)) 729 | 730 | x, y, tx, ty, allx, ally, graph = tuple(objects) 731 | test_idx_reorder = parse_index_file("data/ind.{}.test.index".format(dataset_str)) 732 | test_idx_range = np.sort(test_idx_reorder) 733 | 734 | if dataset_str == 'citeseer': 735 | # Fix citeseer dataset (there are some isolated nodes in the graph) 736 | # Find isolated nodes, add them as zero-vecs into the right position 737 | test_idx_range_full = range(min(test_idx_reorder), max(test_idx_reorder) + 1) 738 | tx_extended = sp.lil_matrix((len(test_idx_range_full), x.shape[1])) 739 | tx_extended[test_idx_range - min(test_idx_range), :] = tx 740 | tx = tx_extended 741 | ty_extended = np.zeros((len(test_idx_range_full), y.shape[1])) 742 | ty_extended[test_idx_range - min(test_idx_range), :] = ty 743 | ty = ty_extended 744 | 745 | features = sp.vstack((allx, tx)).tolil() 746 | features[test_idx_reorder, :] = features[test_idx_range, :] 747 | adj = nx.adjacency_matrix(nx.from_dict_of_lists(graph)) 748 | 749 | labels = np.vstack((ally, ty)) 750 | labels[test_idx_reorder, :] = labels[test_idx_range, :] 751 | 752 | idx_test = test_idx_range.tolist() 753 | idx_train = range(len(y)) 754 | idx_val = range(len(y), len(y) + 500) 755 | 756 | train_mask = sample_mask(idx_train, labels.shape[0]) 757 | val_mask = sample_mask(idx_val, labels.shape[0]) 758 | test_mask = sample_mask(idx_test, labels.shape[0]) 759 | 760 | y_train = np.zeros(labels.shape) 761 | y_val = np.zeros(labels.shape) 762 | y_test = np.zeros(labels.shape) 763 | y_train[train_mask, :] = labels[train_mask, :] 764 | y_val[val_mask, :] = labels[val_mask, :] 765 | y_test[test_mask, :] = labels[test_mask, :] 766 | 767 | # print(adj.shape) 768 | # print(features.shape) 769 | 770 | return adj, features, y_train, y_val, y_test, train_mask, val_mask, test_mask 771 | 772 | 773 | def load_random_data(size): 774 | adj = sp.random(size, size, density=0.002) # density similar to cora 775 | features = sp.random(size, 1000, density=0.015) 776 | int_labels = np.random.randint(7, size=(size)) 777 | labels = np.zeros((size, 7)) # Nx7 778 | labels[np.arange(size), int_labels] = 1 779 | 780 | train_mask = np.zeros((size,)).astype(bool) 781 | train_mask[np.arange(size)[0:int(size / 2)]] = 1 782 | 783 | val_mask = np.zeros((size,)).astype(bool) 784 | val_mask[np.arange(size)[int(size / 2):]] = 1 785 | 786 | test_mask = np.zeros((size,)).astype(bool) 787 | test_mask[np.arange(size)[int(size / 2):]] = 1 788 | 789 | y_train = np.zeros(labels.shape) 790 | y_val = np.zeros(labels.shape) 791 | y_test = np.zeros(labels.shape) 792 | y_train[train_mask, :] = labels[train_mask, :] 793 | y_val[val_mask, :] = labels[val_mask, :] 794 | y_test[test_mask, :] = labels[test_mask, :] 795 | 796 | # sparse NxN, sparse NxF, norm NxC, ..., norm Nx1, ... 797 | return adj, features, y_train, y_val, y_test, train_mask, val_mask, test_mask 798 | 799 | 800 | def sparse_to_tuple(sparse_mx): 801 | """Convert sparse matrix to tuple representation.""" 802 | 803 | def to_tuple(mx): 804 | if not sp.isspmatrix_coo(mx): 805 | mx = mx.tocoo() 806 | coords = np.vstack((mx.row, mx.col)).transpose() 807 | values = mx.data 808 | shape = mx.shape 809 | return coords, values, shape 810 | 811 | if isinstance(sparse_mx, list): 812 | for i in range(len(sparse_mx)): 813 | sparse_mx[i] = to_tuple(sparse_mx[i]) 814 | else: 815 | sparse_mx = to_tuple(sparse_mx) 816 | 817 | return sparse_mx 818 | 819 | 820 | def standardize_data(f, train_mask): 821 | """Standardize feature matrix and convert to tuple representation""" 822 | # standardize data 823 | f = f.todense() 824 | mu = f[train_mask == True, :].mean(axis=0) 825 | sigma = f[train_mask == True, :].std(axis=0) 826 | f = f[:, np.squeeze(np.array(sigma > 0))] 827 | mu = f[train_mask == True, :].mean(axis=0) 828 | sigma = f[train_mask == True, :].std(axis=0) 829 | f = (f - mu) / sigma 830 | return f 831 | 832 | 833 | def preprocess_features(features): 834 | """Row-normalize feature matrix and convert to tuple representation""" 835 | rowsum = np.array(features.sum(1)) 836 | r_inv = np.power(rowsum, -1).flatten() 837 | r_inv[np.isinf(r_inv)] = 0. 838 | r_mat_inv = sp.diags(r_inv) 839 | features = r_mat_inv.dot(features) 840 | return features.todense(), sparse_to_tuple(features) 841 | 842 | 843 | def normalize_adj(adj): 844 | """Symmetrically normalize adjacency matrix.""" 845 | adj = sp.coo_matrix(adj) 846 | rowsum = np.array(adj.sum(1)) 847 | d_inv_sqrt = np.power(rowsum, -0.5).flatten() 848 | d_inv_sqrt[np.isinf(d_inv_sqrt)] = 0. 849 | d_mat_inv_sqrt = sp.diags(d_inv_sqrt) 850 | return adj.dot(d_mat_inv_sqrt).transpose().dot(d_mat_inv_sqrt).tocoo() 851 | 852 | 853 | def preprocess_adj(adj): 854 | """Preprocessing of adjacency matrix for simple GCN model and conversion to tuple representation.""" 855 | adj_normalized = normalize_adj(adj + sp.eye(adj.shape[0])) 856 | return sparse_to_tuple(adj_normalized) 857 | 858 | 859 | def preprocess_adj_bias(adj): 860 | num_nodes = adj.shape[0] 861 | adj = adj + sp.eye(num_nodes) # self-loop 862 | adj[adj > 0.0] = 1.0 863 | if not sp.isspmatrix_coo(adj): 864 | adj = adj.tocoo() 865 | adj = adj.astype(np.float32) 866 | indices = np.vstack( 867 | (adj.col, adj.row)).transpose() # This is where I made a mistake, I used (adj.row, adj.col) instead 868 | # return tf.SparseTensor(indices=indices, values=adj.data, dense_shape=adj.shape) 869 | return indices, adj.data, adj.shape 870 | -------------------------------------------------------------------------------- /utils/process_SG.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pickle as pkl 3 | import networkx as nx 4 | import scipy.sparse as sp 5 | import copy 6 | from scipy.sparse.linalg.eigen.arpack import eigsh 7 | import sys 8 | from scipy.spatial.distance import pdist 9 | from sklearn.preprocessing import label_binarize 10 | from sklearn.metrics import roc_curve, auc 11 | import torch 12 | import torch.nn as nn 13 | 14 | """ 15 | Generate training data for each dataset. 16 | """ 17 | 18 | 19 | def gen_train_data(dataset, split, time_step, nb_nodes, nhood, global_att, batch_size, view='', reverse='0', D=False, graph=False, norm=True, L3=True, enc_k=10): 20 | def get_data(dimension, fr): 21 | if reverse == '1': 22 | used_data = 'target' 23 | else: 24 | used_data = 'source' 25 | input_data = np.load( 26 | 'Datasets/' + frames_ps + 'train_npy_data/' + used_data + '_' + dimension + '_' + dataset + '_' + str( 27 | fr) + '.npy', allow_pickle=True) 28 | input_data = input_data.reshape([-1, time_step, nb_nodes]) 29 | spine_pos = input_data[:, :, 0] 30 | spine_pos = np.expand_dims(spine_pos, -1) 31 | input_data = input_data - spine_pos 32 | # if dataset == 'IAS' or dataset == 'BIWI' or dataset == 'KGBD': 33 | t_input_data = np.load( 34 | 'Datasets/' + frames_ps + 'test_npy_data/' + split + '/t_' + used_data + '_' + dimension + '_' + dataset + '_' + str( 35 | fr) + '.npy', allow_pickle=True) 36 | # else: 37 | # t_input_data = np.load( 38 | # 'Datasets/' + frames_ps + 'test_npy_data/t_' + used_data + '_' + dimension + '_' + dataset + '_' + str( 39 | # fr) + '.npy') 40 | t_input_data = t_input_data.reshape([-1, time_step, nb_nodes]) 41 | if norm == True: 42 | # Normalize 43 | t_spine_pos = t_input_data[:, :, 0] 44 | t_spine_pos = np.expand_dims(t_spine_pos, -1) 45 | t_input_data = t_input_data - t_spine_pos 46 | 47 | return input_data, t_input_data 48 | 49 | def cal_features_ave(X, Y, Z): 50 | f = [] 51 | # 2014 52 | for i in range(X.shape[0]): 53 | ave_f = [] 54 | eights = [] 55 | nines = [] 56 | for j in range(X.shape[1]): 57 | sub_f = [] 58 | # head height 59 | h_1 = (2 * Y[i, j, 3] - Y[i, j, 15] - Y[i, j, 19]) / 2 60 | # neck height 61 | h_2 = (2 * Y[i, j, 2] - Y[i, j, 15] - Y[i, j, 19]) / 2 62 | sub_f.append(h_1) 63 | sub_f.append(h_2) 64 | for a, b in [[4,2], [8,2], [1,8], [8,10], [4,6], [16,17], [13,12],[2,1],[16,12]]: 65 | sub_f.append(np.sqrt((X[i,j,a]-X[i,j,b])**2+(Y[i,j,a]-Y[i,j,b])**2+(Z[i,j,a]-Z[i,j,b])**2)) 66 | eights.append(np.sqrt((X[i,j,16]-X[i,j,17])**2+(Y[i,j,16]-Y[i,j,17])**2+(Z[i,j,16]-Z[i,j,17])**2)) 67 | nines.append(np.sqrt((X[i,j,13]-X[i,j,12])**2+(Y[i,j,13]-Y[i,j,12])**2+(Z[i,j,13]-Z[i,j,12])**2)) 68 | ratio_1 = np.sqrt((X[i,j,2]-X[i,j,1])**2+(Y[i,j,2]-Y[i,j,1])**2+(Z[i,j,2]-Z[i,j,1])**2) / \ 69 | max(np.sqrt((X[i,j,16]-X[i,j,17])**2+(Y[i,j,16]-Y[i,j,17])**2+(Z[i,j,16]-Z[i,j,17])**2), 0.1) 70 | ratio_2 = np.sqrt((X[i,j,2]-X[i,j,1])**2+(Y[i,j,2]-Y[i,j,1])**2+(Z[i,j,2]-Z[i,j,1])**2) / \ 71 | max(np.sqrt((X[i,j,13]-X[i,j,12])**2+(Y[i,j,13]-Y[i,j,12])**2+(Z[i,j,13]-Z[i,j,12])**2), 0.1) 72 | sub_f.append(ratio_1) 73 | sub_f.append(ratio_2) 74 | ave_f.append(sub_f) 75 | ave_f = np.array(ave_f) 76 | # print (ave_f.shape) 77 | ave_f = np.mean(ave_f, axis=0) 78 | # print(ave_f.shape) 79 | # exit(1) 80 | ave_f = ave_f.tolist() 81 | f.append(ave_f) 82 | 83 | # 2019 84 | ff = [] 85 | for i in range(X.shape[0]): 86 | ave_f = [] 87 | for j in range(X.shape[1]): 88 | sub_ff = [] 89 | for a, b in [[3, 2], [1, 2], [1, 0], [8, 2], [8, 9], [9, 10], [2, 4], [4, 5], [5, 6], 90 | [16,0],[16,17],[17,19],[12,0],[12,13],[13,15]]: 91 | sub_ff.append(np.sqrt((X[i,j, a] - X[i, j,b]) ** 2 + (Y[i, j,a] - Y[i, j,b]) ** 2 + (Z[i,j, a] - Z[i,j, b]) ** 2)) 92 | h = h_1 = (2 * Y[i, j, 3] - Y[i, j, 15] - Y[i, j, 19]) / 2 93 | sub_ff.append(h) 94 | ave_f.append(sub_ff) 95 | ave_f = np.array(ave_f) 96 | ave_f = np.mean(ave_f, axis=0) 97 | ave_f = ave_f.tolist() 98 | ff.append(ave_f) 99 | f = np.array(f) 100 | ff = np.array(ff) 101 | # print(f.shape) 102 | # print(ff.shape) 103 | return f, ff 104 | 105 | def unnormalized_laplacian(adj_matrix): 106 | R = np.sum(adj_matrix, axis=1) 107 | degreeMatrix = np.diag(R) 108 | return degreeMatrix - adj_matrix 109 | 110 | def normalized_laplacian(adj_matrix): 111 | R = np.sum(adj_matrix, axis=1) 112 | R_sqrt = 1 / np.sqrt(R) 113 | D_sqrt = np.diag(R_sqrt) 114 | I = np.eye(adj_matrix.shape[0]) 115 | return I - np.matmul(np.matmul(D_sqrt, adj_matrix), D_sqrt) 116 | 117 | 118 | if view == '': 119 | frames_ps = dataset + '/' + str(time_step) + '/' 120 | else: 121 | frames_ps = dataset + '/' + str(time_step) + '/view_' + str(view) + '/' 122 | input_data_x, t_input_data_x = get_data('x', fr=time_step) 123 | input_data_y, t_input_data_y = get_data('y', fr=time_step) 124 | input_data_z, t_input_data_z = get_data('z', fr=time_step) 125 | 126 | 127 | if D: 128 | D_13_test, D_16_test = cal_features_ave(t_input_data_x, t_input_data_y, t_input_data_z) 129 | 130 | 131 | X_train = np.concatenate([input_data_x, input_data_y, input_data_z], axis=-1) 132 | X_test = np.concatenate([t_input_data_x, t_input_data_y, t_input_data_z], axis=-1) 133 | 134 | X_train_d = np.concatenate([input_data_x.reshape([-1, time_step, nb_nodes, 1]), 135 | input_data_y.reshape([-1, time_step, nb_nodes, 1]), 136 | input_data_z.reshape([-1, time_step, nb_nodes, 1])], axis=-1).reshape([-1, nb_nodes, 3]) 137 | X_test_d = np.concatenate([t_input_data_x.reshape([-1, time_step, nb_nodes, 1]), 138 | t_input_data_y.reshape([-1, time_step, nb_nodes, 1]), 139 | t_input_data_z.reshape([-1, time_step, nb_nodes, 1])], axis=-1).reshape([-1, nb_nodes, 3]) 140 | X_train_d_all = [] 141 | X_test_d_all = [] 142 | for i in range(X_train_d.shape[0]): 143 | X_train_d_all.append(pdist(X_train_d[i])) 144 | for i in range(X_test_d.shape[0]): 145 | X_test_d_all.append(pdist(X_test_d[i])) 146 | X_train_d_all = np.reshape(np.array(X_train_d_all), [X_train.shape[0], time_step, -1]) 147 | X_test_d_all = np.reshape(np.array(X_test_d_all), [X_test.shape[0], time_step, -1]) 148 | 149 | ids = np.load( 150 | 'Datasets/' + frames_ps + 'train_npy_data/ids_' + dataset + '_' + str(time_step) + '.npy', allow_pickle=True) 151 | ids = ids.item() 152 | # if dataset == 'IAS' or dataset == 'BIWI' or dataset == 'KGBD': 153 | t_ids = np.load( 154 | 'Datasets/' + frames_ps + 'test_npy_data/' + split + '/ids_' + dataset + '_' + str(time_step) + '.npy', allow_pickle=True) 155 | # else: 156 | # t_ids = np.load( 157 | # 'Datasets/' + frames_ps + 'test_npy_data/ids_' + dataset + '_' + str(time_step) + '.npy') 158 | t_ids = t_ids.item() 159 | 160 | y_train = np.load( 161 | 'Datasets/' + frames_ps + 'train_npy_data/frame_id_' + dataset + '_' + str(time_step) + '.npy', allow_pickle=True) 162 | # if dataset == 'IAS' or dataset == 'BIWI' or dataset == 'KGBD': 163 | y_test = np.load( 164 | 'Datasets/' + frames_ps + 'test_npy_data/' + split + '/frame_id_' + dataset + '_' + str(time_step) + '.npy', allow_pickle=True) 165 | # else: 166 | # y_test = np.load( 167 | # 'Datasets/' + frames_ps + 'test_npy_data/frame_id_' + dataset + '_' + str(time_step) + '.npy') 168 | 169 | # if dataset != 'CASIA_B': 170 | # print(X_train.shape, X_test.shape) 171 | if dataset != 'KGBD': 172 | X_train, y_train = class_samp_gen(X_train, y_train, ids, batch_size) 173 | 174 | # randomly shuffle 175 | rand_p = np.random.permutation(X_train.shape[0]) 176 | X_train = X_train[rand_p] 177 | y_train = y_train[rand_p] 178 | if D: 179 | D_13_train, D_16_train = cal_features_ave(X_train[:, :, :nb_nodes], X_train[:, :, nb_nodes:nb_nodes*2], 180 | X_train[:, :, nb_nodes*2:]) 181 | # print(X_train.shape, X_test.shape) 182 | 183 | ids_keys = sorted(list(ids.keys())) 184 | classes = [i for i in ids_keys] 185 | y_train = label_binarize(y_train, classes=classes) 186 | # t_ids_keys = sorted(list(t_ids.keys())) 187 | # classes = [i for i in t_ids_keys] 188 | y_test = label_binarize(y_test, classes=classes) 189 | 190 | X_train_J = X_train.reshape([-1, time_step, 3, nb_nodes]) 191 | X_train_J = np.transpose(X_train_J, [0, 1, 3, 2]) 192 | 193 | X_test_J = X_test.reshape([-1, time_step, 3, nb_nodes]) 194 | X_test_J = np.transpose(X_test_J, [0, 1, 3, 2]) 195 | 196 | # Mapping KGBD skeleton data into identical format 197 | if dataset == 'KGBD': 198 | X_train_J = X_train_J[:, :, [11, 10, 1, 0, 2, 4, 6, 8, 3, 5, 7, 9, 12, 14, 16, 18, 13, 15, 17, 19], :] 199 | X_test_J = X_test_J[:, :, [11, 10, 1, 0, 2, 4, 6, 8, 3, 5, 7, 9, 12, 14, 16, 18, 13, 15, 17, 19], :] 200 | 201 | if graph or L3: 202 | X_train_P = reduce2part(X_train_J, nb_nodes) 203 | X_train_B = reduce2body(X_train_J, nb_nodes) 204 | # X_train_H_B = reduce2h_body(X_train_J, nb_nodes) 205 | # if dataset == 'KS20': 206 | # X_train_In = interpolation(X_train_J, nb_nodes) 207 | 208 | X_test_P = reduce2part(X_test_J, nb_nodes) 209 | X_test_B = reduce2body(X_test_J, nb_nodes) 210 | # X_test_H_B = reduce2h_body(X_test_J, nb_nodes) 211 | # if dataset == 'KS20': 212 | # X_test_In = interpolation(X_test_J, nb_nodes) 213 | 214 | def generate_denser_adj(adj): 215 | adj_temp = copy.deepcopy(adj).tolist() 216 | node_num = len(adj_temp) 217 | # if global_att: 218 | # new_adj = np.ones([node_num * 2 - 1, node_num * 2 - 1]) 219 | # return new_adj 220 | new_adj = np.zeros([node_num * 2 - 1, node_num * 2 - 1]) 221 | cnt = node_num 222 | for i in range(node_num): 223 | for j in range(node_num): 224 | if adj_temp[i][j] == 1: 225 | new_adj[i, cnt] = new_adj[cnt, i] = new_adj[j, cnt] = new_adj[cnt, j] = 1 226 | adj_temp[i][j] = adj_temp[j][i] = 0 227 | # print(i, j, cnt) 228 | cnt += 1 229 | for i in range(node_num): 230 | for j in range(node_num): 231 | if adj_temp[i][j] == 1: 232 | assert new_adj[i, j] == new_adj[j, i] == 0 233 | # print(cnt) 234 | # np.save('interp_graph_adj.npy', new_adj) 235 | return new_adj 236 | import scipy.sparse 237 | if dataset == 'KS20': 238 | # Joint-Level adjacent matrix 239 | j_pair_1 = np.array([3, 2, 20, 8, 8, 9, 10, 9, 11, 10, 4, 20, 4, 5, 5, 6, 6, 7, 1, 20, 1, 0, 16, 0, 240 | 12, 0, 16, 17, 12, 13, 17, 18, 19, 18, 13, 14, 14, 15, 2, 20, 11, 23, 10, 24, 7, 21, 6, 22]) 241 | j_pair_2 = np.array([2, 3, 8, 20, 9, 8, 9, 10, 10, 11, 20, 4, 5, 4, 6, 5, 7, 6, 20, 1, 0, 1, 0, 16, 242 | 0, 12, 17, 16, 13, 12, 18, 17, 18, 19, 14, 13, 15, 14, 20, 2, 23, 11, 24, 10, 21, 7, 22, 6]) 243 | con_matrix = np.ones([48]) 244 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 245 | 246 | # L = normalized_laplacian(adj_joint) 247 | # # only real 248 | # EigVal, EigVec = np.linalg.eig(L) 249 | # idx = EigVal.argsort() # increasing order 250 | # EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 251 | # 252 | # # real + complex 253 | # # EigVal, EigVec = sp.linalg.eigs(L, k=10 + 1, which='SR', tol=1e-2) # for 40 PEs 254 | # # EigVec = EigVec[:, EigVal.argsort()] # increasing order 255 | # pos_enc_ori = EigVec[:,1:10+1] 256 | 257 | # print(pos_enc_dim.size()) 258 | 259 | # test 260 | # embedding_lap_pos_enc = nn.Linear(10, 64) 261 | # embedding_h = nn.Embedding(3, 64) 262 | # h = np.random.random([3]).astype(np.float32) 263 | # pos_enc_dim = pos_enc_dim.astype(np.float32) 264 | # input = torch.LongTensor([1]) 265 | # pos_enc_dim = torch.from_numpy(pos_enc_dim) 266 | # h = embedding_h(input) 267 | # h_lap_pos_enc = embedding_lap_pos_enc(pos_enc_dim) 268 | # h_new = h + h_lap_pos_enc 269 | # print(h_new.size(), h.size(), h_lap_pos_enc.size()) 270 | # exit() 271 | 272 | 273 | # if global_att: 274 | # adj_joint = np.ones([25, 25]) 275 | # hyper-Level (interpolation) adjacent matrix, NOT used, replaced by generate_denser_adj 276 | # i_pair_1 = np.array( 277 | # [3, 31, 31, 2, 2, 32, 32, 20, 20, 42, 42, 8, 8, 37, 37, 9, 9, 44, 44, 10, 10, 38, 38, 11, 11, 278 | # 39, 39, 23, 10, 40, 40, 24, 20, 41, 41, 4, 4, 33, 33, 5, 5, 43, 43, 6, 6, 34, 34, 7, 7, 35, 279 | # 35, 21, 6, 36, 36, 22, 280 | # 20, 45, 45, 1, 1, 30, 30, 0, 0, 46, 46, 12, 12, 26, 26, 13, 13, 48, 48, 14, 14, 27, 27, 15, 281 | # 0, 47, 47, 16, 16, 28, 28, 17, 17, 49, 49, 18, 18, 29, 29, 19]) 282 | # # miss 25 283 | # i_pair_1 = i_pair_1.tolist() 284 | # for i in range(len(i_pair_1)): 285 | # if i_pair_1[i] > 24: 286 | # i_pair_1[i] -= 1 287 | # i_pair_1 = np.array(i_pair_1) 288 | # i_pair_2 = np.array( 289 | # [31, 3, 2, 31, 32, 2, 20, 32, 42, 20, 8, 42, 37, 8, 9, 37, 44, 9, 10, 44, 38, 10, 11, 38, 39, 290 | # 11, 23, 39, 40, 10, 24, 40, 41, 20, 4, 41, 33, 4, 5, 33, 43, 5, 6, 43, 34, 6, 7, 34, 35, 7, 291 | # 21, 35, 36, 6, 22, 36, 292 | # 45, 20, 1, 45, 30, 1, 0, 30, 46, 0, 12, 46, 26, 12, 13, 26, 48, 13, 14, 48, 27, 14, 15, 27, 293 | # 47, 0, 16, 47, 28, 16, 17, 28, 49, 17, 18, 49, 29, 18, 19, 29]) 294 | # i_pair_2 = i_pair_2.tolist() 295 | # for i in range(len(i_pair_2)): 296 | # if i_pair_2[i] > 24: 297 | # i_pair_2[i] -= 1 298 | # i_pair_2 = np.array(i_pair_2) 299 | # # print(i_pair_1.shape, i_pair_2.shape) 300 | # con_matrix = np.ones([96]) 301 | # adj_interp = scipy.sparse.coo_matrix((con_matrix, (i_pair_1, i_pair_2)), 302 | # shape=(49, 49)).toarray() 303 | adj_interp = generate_denser_adj(adj_joint) 304 | elif dataset == 'CASIA_B': 305 | # Joint-Level adjacent matrix 306 | j_pair_1 = np.array([0, 1, 1, 2, 2, 3, 3, 4, 1, 5, 5, 6, 6, 7, 1, 8, 8, 9, 9, 10, 1, 11, 11, 12, 12, 13]) 307 | j_pair_2 = np.array([1, 0, 2, 1, 3, 2, 4, 3, 5, 1, 6, 5, 7, 6, 8, 1, 9, 8, 10, 9, 11, 1, 12, 11, 13, 12]) 308 | con_matrix = np.ones([26]) 309 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 310 | # if global_att: 311 | # adj_joint = np.ones([14, 14]) 312 | adj_interp = generate_denser_adj(adj_joint) 313 | else: 314 | # Joint-Level adjacent matrix 315 | j_pair_1 = np.array([3, 2, 2, 8, 8, 9, 10, 9, 11, 10, 4, 2, 4, 5, 5, 6, 6, 7, 1, 2, 1, 0, 16, 0, 316 | 12, 0, 16, 17, 12, 13, 17, 18, 19, 18, 13, 14, 14, 15]) 317 | j_pair_2 = np.array([2, 3, 8, 2, 9, 8, 9, 10, 10, 11, 2, 4, 5, 4, 6, 5, 7, 6, 2, 1, 0, 1, 0, 16, 318 | 0, 12, 17, 16, 13, 12, 18, 17, 18, 19, 14, 13, 15, 14]) 319 | con_matrix = np.ones([38]) 320 | adj_joint = scipy.sparse.coo_matrix((con_matrix, (j_pair_1, j_pair_2)), shape=(nb_nodes, nb_nodes)).toarray() 321 | # if global_att: 322 | # adj_joint = np.ones([20, 20]) 323 | adj_interp = generate_denser_adj(adj_joint) 324 | # compute Laplacian matrix, use the k smallest non-trivial eigen-vectors 325 | k = enc_k 326 | L = normalized_laplacian(adj_joint) 327 | # only real 328 | EigVal, EigVec = np.linalg.eig(L) 329 | idx = EigVal.argsort() 330 | EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 331 | # print(EigVal.shape, EigVec.shape) 332 | # exit() 333 | pos_enc_ori = EigVec[:, 1:k + 1] 334 | 335 | if graph or L3: 336 | # Part-Level adjacent matrix 337 | p_pair_1 = np.array([5, 6, 5, 8, 6, 7, 8, 9, 5, 4, 4, 2, 4, 0, 2, 3, 1, 0]) 338 | p_pair_2 = np.array([6, 5, 8, 5, 7, 6, 9, 8, 4, 5, 2, 4, 0, 4, 3, 2, 0, 1]) 339 | con_matrix = np.ones([18]) 340 | adj_part = scipy.sparse.coo_matrix((con_matrix, (p_pair_1, p_pair_2)), shape=(10, 10)).toarray() 341 | 342 | # Body-Level adjacent matrix 343 | b_pair_1 = np.array([2, 3, 2, 4, 2, 1, 2, 0]) 344 | b_pair_2 = np.array([3, 2, 4, 2, 1, 2, 0, 2]) 345 | con_matrix = np.ones([8]) 346 | adj_body = scipy.sparse.coo_matrix((con_matrix, (b_pair_1, b_pair_2)), shape=(5, 5)).toarray() 347 | 348 | # Hyper-Body-Level adjacent matrix 349 | # h_b_pair_1 = np.array([0, 1, 2, 1]) 350 | # h_b_pair_2 = np.array([1, 0, 1, 2]) 351 | # con_matrix = np.ones([4]) 352 | # adj_hyper_body = scipy.sparse.coo_matrix((con_matrix, (h_b_pair_1, h_b_pair_2)), shape=(3, 3)).toarray() 353 | 354 | # compute Laplacian matrix for part-level and body-level, use the k (1/2 node number) smallest non-trivial eigen-vectors 355 | # part-level 356 | k = 9 357 | L = normalized_laplacian(adj_part) 358 | # only real 359 | EigVal, EigVec = np.linalg.eig(L) 360 | idx = EigVal.argsort() 361 | EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 362 | pos_enc_part = EigVec[:, 1:k + 1] 363 | 364 | # body-level 365 | k = 4 366 | L = normalized_laplacian(adj_body) 367 | # only real 368 | EigVal, EigVec = np.linalg.eig(L) 369 | idx = EigVal.argsort() 370 | EigVal, EigVec = EigVal[idx], np.real(EigVec[:, idx]) 371 | pos_enc_body = EigVec[:, 1:k + 1] 372 | 373 | if not L3: 374 | # if dataset != 'KS20': 375 | X_train_In = generate_denser_graph_data(X_train_J, adj_joint, nb_nodes) 376 | X_test_In = generate_denser_graph_data(X_test_J, adj_joint, nb_nodes) 377 | 378 | if global_att: 379 | if dataset == 'KS20': 380 | node_num = 25 381 | elif dataset == 'CASIA_B': 382 | node_num = 14 383 | else: 384 | node_num = 20 385 | adj_joint = np.ones([node_num, node_num]) 386 | adj_interp = np.ones([node_num * 2 - 1, node_num * 2 - 1]) 387 | adj_part = np.ones([10, 10]) 388 | adj_body = np.ones([5, 5]) 389 | 390 | if dataset == 'IAS': 391 | nb_classes = 11 392 | elif dataset == 'KGBD': 393 | nb_classes = 164 394 | elif dataset == 'BIWI': 395 | nb_classes = 50 396 | elif dataset == 'KS20': 397 | nb_classes = 20 398 | elif dataset == 'CASIA_B': 399 | nb_classes = 124 400 | elif dataset == 'KinectREID': 401 | nb_classes = 71 402 | 403 | adj_joint = adj_joint[np.newaxis] 404 | biases_joint = adj_to_bias(adj_joint, [nb_nodes], nhood=nhood) 405 | 406 | if graph or L3: 407 | adj_part = adj_part[np.newaxis] 408 | biases_part = adj_to_bias(adj_part, [10], nhood=1) 409 | 410 | adj_body = adj_body[np.newaxis] 411 | biases_body = adj_to_bias(adj_body, [5], nhood=1) 412 | 413 | # adj_hyper_body = adj_hyper_body[np.newaxis] 414 | # biases_hyper_body = adj_to_bias(adj_hyper_body, [3], nhood=1) 415 | # 416 | # adj_interp = adj_interp[np.newaxis] 417 | # biases_interp = adj_to_bias(adj_interp, [nb_nodes*2-1], nhood=1) 418 | 419 | if not D: 420 | if graph: 421 | return X_train_J, X_train_P, X_train_B, X_train_H_B, X_train_In, y_train, X_test_J, X_test_P, X_test_B, X_test_H_B, X_test_In, y_test, \ 422 | adj_joint, biases_joint, adj_part, biases_part, adj_body, biases_body, adj_hyper_body, biases_hyper_body, adj_interp, biases_interp, nb_classes 423 | else: 424 | if not L3: 425 | return X_train_J, 0, 0, 0, 0, y_train, X_test_J, 0, 0, 0, 0, y_test, \ 426 | adj_joint, biases_joint, 0, 0, 0, 0, 0, 0, 0, 0, nb_classes 427 | else: 428 | return X_train_J, X_train_P, X_train_B, 0, 0, y_train, X_test_J, X_test_P, X_test_B, 0, 0, y_test, \ 429 | adj_joint, 0, pos_enc_ori, adj_part, 0, pos_enc_part, adj_body, 0, pos_enc_body, 0, 0, 0, 0, nb_classes 430 | else: 431 | return X_train_J, X_train_P, X_train_B, X_train_H_B, X_train_In, y_train, D_13_train, D_16_train,\ 432 | X_test_J, X_test_P, X_test_B, X_test_H_B, X_test_In, y_test, D_13_test, D_16_test, \ 433 | adj_joint, biases_joint, adj_part, biases_part, adj_body, biases_body, adj_hyper_body, biases_hyper_body, adj_interp, biases_interp, nb_classes 434 | """ 435 | Generate part-level skeleton graphs. 436 | """ 437 | 438 | def reduce2part(X, joint_num=20): 439 | if joint_num == 25: 440 | left_leg_up = [12, 13] 441 | left_leg_down = [14, 15] 442 | right_leg_up = [16, 17] 443 | right_leg_down = [18, 19] 444 | torso = [0, 1] 445 | head = [2, 3, 20] 446 | left_arm_up = [4, 5] 447 | left_arm_down = [6, 7, 21, 22] 448 | right_arm_up = [8, 9] 449 | right_arm_down = [10, 11, 23, 24] 450 | elif joint_num == 20: 451 | left_leg_up = [12, 13] 452 | left_leg_down = [14, 15] 453 | right_leg_up = [16, 17] 454 | right_leg_down = [18, 19] 455 | torso = [0, 1] 456 | head = [2, 3] 457 | left_arm_up = [4, 5] 458 | left_arm_down = [6, 7] 459 | right_arm_up = [8, 9] 460 | right_arm_down = [10, 11] 461 | elif joint_num == 14: 462 | left_leg_up = [11] 463 | left_leg_down = [12, 13] 464 | right_leg_up = [8] 465 | right_leg_down = [9, 10] 466 | torso = [1] 467 | head = [0] 468 | left_arm_up = [5] 469 | left_arm_down = [6, 7] 470 | right_arm_up = [2] 471 | right_arm_down = [3, 4] 472 | 473 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 474 | x_leftlegup = np.mean(X[:, :, left_leg_up, :], axis=2) 475 | x_leftlegdown = np.mean(X[:, :, left_leg_down, :], axis=2) 476 | x_rightlegup = np.mean(X[:, :, right_leg_up, :], axis=2) 477 | x_rightlegdown = np.mean(X[:, :, right_leg_down, :], axis=2) 478 | x_head = np.mean(X[:, :, head, :], axis=2) 479 | x_leftarmup = np.mean(X[:, :, left_arm_up, :], axis=2) 480 | x_leftarmdown = np.mean(X[:, :, left_arm_down, :], axis=2) 481 | x_rightarmup = np.mean(X[:, :, right_arm_up, :], axis=2) 482 | x_rightarmdown = np.mean(X[:, :, right_arm_down, :], axis=2) 483 | X_part = np.concatenate((x_leftlegup, x_leftlegdown, x_rightlegup, x_rightlegdown, x_torso, x_head, x_leftarmup, 484 | x_leftarmdown, x_rightarmup, x_rightarmdown), axis=-1) \ 485 | .reshape([X.shape[0], X.shape[1], 10, 3]) 486 | return X_part 487 | 488 | """ 489 | Generate body-level skeleton graphs. 490 | """ 491 | 492 | def reduce2body(X, joint_num=20): 493 | if joint_num == 25: 494 | left_leg = [12, 13, 14, 15] 495 | right_leg = [16, 17, 18, 19] 496 | torso = [0, 1, 2, 3, 20] 497 | left_arm = [4, 5, 6, 7, 21, 22] 498 | right_arm = [8, 9, 10, 11, 23, 24] 499 | elif joint_num == 20: 500 | left_leg = [12, 13, 14, 15] 501 | right_leg = [16, 17, 18, 19] 502 | torso = [0, 1, 2, 3] 503 | left_arm = [4, 5, 6, 7] 504 | right_arm = [8, 9, 10, 11] 505 | elif joint_num == 14: 506 | left_leg = [11, 12, 13] 507 | right_leg = [8, 9, 10] 508 | torso = [0, 1] 509 | left_arm = [5, 6, 7] 510 | right_arm = [2, 3, 4] 511 | 512 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 513 | x_leftleg = np.mean(X[:, :, left_leg, :], axis=2) 514 | x_rightleg = np.mean(X[:, :, right_leg, :], axis=2) 515 | x_leftarm = np.mean(X[:, :, left_arm, :], axis=2) 516 | x_rightarm = np.mean(X[:, :, right_arm, :], axis=2) 517 | X_body = np.concatenate((x_leftleg, x_rightleg, x_torso, x_leftarm, x_rightarm), axis=-1)\ 518 | .reshape([X.shape[0], X.shape[1], 5, 3]) 519 | return X_body 520 | 521 | """ 522 | Generate hyper-body-level skeleton graphs. 523 | """ 524 | 525 | def reduce2h_body(X, joint_num=20): 526 | if joint_num == 25: 527 | # left_leg = [12, 13, 14, 15] 528 | # right_leg = [16, 17, 18, 19] 529 | # torso = [0, 1, 2, 3, 20] 530 | # left_arm = [4, 5, 6, 7, 21, 22] 531 | # right_arm = [8, 9, 10, 11, 23, 24] 532 | upper = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 20, 21, 22, 23, 24] 533 | middle = [0] 534 | lower = [12, 13, 14, 15, 16, 17, 18, 19] 535 | elif joint_num == 20: 536 | # left_leg = [12, 13, 14, 15] 537 | # right_leg = [16, 17, 18, 19] 538 | # torso = [0, 1, 2, 3] 539 | # left_arm = [4, 5, 6, 7] 540 | # right_arm = [8, 9, 10, 11] 541 | upper = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 542 | middle = [0] 543 | lower = [12, 13, 14, 15, 16, 17, 18, 19] 544 | elif joint_num == 14: 545 | # left_leg = [11, 12, 13] 546 | # right_leg = [8, 9, 10] 547 | # torso = [0, 1] 548 | # left_arm = [5, 6, 7] 549 | # right_arm = [2, 3, 4] 550 | upper = [0, 1, 2, 3, 4, 5, 6, 7] 551 | middle = [8, 11] 552 | lower = [9, 10, 12, 13] 553 | 554 | x_upper = np.mean(X[:, :, upper, :], axis=2) # [N * T, V=1] 555 | x_middle = np.mean(X[:, :, middle, :], axis=2) 556 | x_lower = np.mean(X[:, :, lower, :], axis=2) 557 | X_body = np.concatenate((x_upper, x_middle, x_lower), axis=-1)\ 558 | .reshape([X.shape[0], X.shape[1], 3, 3]) 559 | return X_body 560 | 561 | 562 | """ 563 | Linear interpolation 564 | """ 565 | 566 | def interpolation(X, joint_num=20): 567 | if joint_num == 25: 568 | left_leg_up = [12, 13] 569 | left_leg_down = [14, 15] 570 | right_leg_up = [16, 17] 571 | right_leg_down = [18, 19] 572 | torso = [0, 1] 573 | head_1 = [2, 3] 574 | head_2 = [2, 20] 575 | left_arm_up = [4, 5] 576 | left_arm_down_1 = [6, 7] 577 | left_arm_down_2 = [7, 21] 578 | left_arm_down_3 = [6, 22] 579 | right_arm_up = [8, 9] 580 | right_arm_down_1 = [10, 11] 581 | right_arm_down_2 = [11, 23] 582 | right_arm_down_3 = [10, 24] 583 | shoulder_1 = [4, 20] 584 | shoulder_2 = [8, 20] 585 | elbow_1 = [5, 6] 586 | elbow_2 = [9, 10] 587 | spine_mm = [20, 1] 588 | hip_1 = [0, 12] 589 | hip_2 = [0, 16] 590 | knee_1 = [13, 14] 591 | knee_2 = [17, 18] 592 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 593 | x_leftlegup = np.mean(X[:, :, left_leg_up, :], axis=2) 594 | x_leftlegdown = np.mean(X[:, :, left_leg_down, :], axis=2) 595 | x_rightlegup = np.mean(X[:, :, right_leg_up, :], axis=2) 596 | x_rightlegdown = np.mean(X[:, :, right_leg_down, :], axis=2) 597 | x_head_1 = np.mean(X[:, :, head_1, :], axis=2) 598 | x_head_2 = np.mean(X[:, :, head_2, :], axis=2) 599 | x_leftarmup = np.mean(X[:, :, left_arm_up, :], axis=2) 600 | x_leftarmdown_1 = np.mean(X[:, :, left_arm_down_1, :], axis=2) 601 | x_leftarmdown_2 = np.mean(X[:, :, left_arm_down_2, :], axis=2) 602 | x_leftarmdown_3 = np.mean(X[:, :, left_arm_down_3, :], axis=2) 603 | x_rightarmup = np.mean(X[:, :, right_arm_up, :], axis=2) 604 | x_rightarmdown_1 = np.mean(X[:, :, right_arm_down_1, :], axis=2) 605 | x_rightarmdown_2 = np.mean(X[:, :, right_arm_down_2, :], axis=2) 606 | x_rightarmdown_3 = np.mean(X[:, :, right_arm_down_3, :], axis=2) 607 | shoulder_1 = np.mean(X[:, :, shoulder_1, :], axis=2) 608 | shoulder_2 = np.mean(X[:, :, shoulder_2, :], axis=2) 609 | elbow_1 = np.mean(X[:, :, elbow_1, :], axis=2) 610 | elbow_2 = np.mean(X[:, :, elbow_2, :], axis=2) 611 | spine_mm = np.mean(X[:, :, spine_mm, :], axis=2) 612 | hip_1 = np.mean(X[:, :, hip_1, :], axis=2) 613 | hip_2 = np.mean(X[:, :, hip_2, :], axis=2) 614 | knee_1 = np.mean(X[:, :, knee_1, :], axis=2) 615 | knee_2 = np.mean(X[:, :, knee_2, :], axis=2) 616 | X_part = np.concatenate((x_leftlegup, x_leftlegdown, x_rightlegup, 617 | x_rightlegdown, x_torso, x_head_1, x_head_2, x_leftarmup, 618 | x_leftarmdown_1, x_leftarmdown_2, x_leftarmdown_3, 619 | x_rightarmup, x_rightarmdown_1, x_rightarmdown_2, x_rightarmdown_3, 620 | shoulder_1, shoulder_2, elbow_1, elbow_2, spine_mm, 621 | hip_1, hip_2, knee_1, knee_2), axis=-1) \ 622 | .reshape([X.shape[0], X.shape[1], 24, 3]) 623 | # 25+24 624 | X_interp = np.concatenate((X, X_part), axis=-2) 625 | # Changed 626 | elif joint_num == 20: 627 | # left_leg_up = [12, 13] 628 | # left_leg_down = [14, 15] 629 | # right_leg_up = [16, 17] 630 | # right_leg_down = [18, 19] 631 | # torso = [0, 1] 632 | # head = [2, 3] 633 | # left_arm_up = [4, 5] 634 | # left_arm_down = [6, 7] 635 | # right_arm_up = [8, 9] 636 | # right_arm_down = [10, 11] 637 | # 638 | left_leg_up = [12, 13] 639 | left_leg_down = [14, 15] 640 | right_leg_up = [16, 17] 641 | right_leg_down = [18, 19] 642 | torso = [0, 1] 643 | head_1 = [2, 3] 644 | # head_2 = [2, 20] 645 | left_arm_up = [4, 5] 646 | left_arm_down_1 = [6, 7] 647 | # left_arm_down_2 = [7, 21] 648 | # left_arm_down_3 = [6, 22] 649 | right_arm_up = [8, 9] 650 | right_arm_down_1 = [10, 11] 651 | # right_arm_down_2 = [11, 23] 652 | # right_arm_down_3 = [10, 24] 653 | # shoulder_1 = [4, 20] 654 | # shoulder_2 = [8, 20] 655 | shoulder_1 = [4, 2] 656 | shoulder_2 = [8, 2] 657 | elbow_1 = [5, 6] 658 | elbow_2 = [9, 10] 659 | # spine_mm = [20, 1] 660 | spine_mm = [2, 1] 661 | hip_1 = [0, 12] 662 | hip_2 = [0, 16] 663 | knee_1 = [13, 14] 664 | knee_2 = [17, 18] 665 | 666 | x_torso = np.mean(X[:, :, torso, :], axis=2) # [N * T, V=1] 667 | x_leftlegup = np.mean(X[:, :, left_leg_up, :], axis=2) 668 | x_leftlegdown = np.mean(X[:, :, left_leg_down, :], axis=2) 669 | x_rightlegup = np.mean(X[:, :, right_leg_up, :], axis=2) 670 | x_rightlegdown = np.mean(X[:, :, right_leg_down, :], axis=2) 671 | x_head_1 = np.mean(X[:, :, head_1, :], axis=2) 672 | # x_head_2 = np.mean(X[:, :, head_2, :], axis=2) 673 | x_leftarmup = np.mean(X[:, :, left_arm_up, :], axis=2) 674 | x_leftarmdown_1 = np.mean(X[:, :, left_arm_down_1, :], axis=2) 675 | # x_leftarmdown_2 = np.mean(X[:, :, left_arm_down_2, :], axis=2) 676 | # x_leftarmdown_3 = np.mean(X[:, :, left_arm_down_3, :], axis=2) 677 | x_rightarmup = np.mean(X[:, :, right_arm_up, :], axis=2) 678 | x_rightarmdown_1 = np.mean(X[:, :, right_arm_down_1, :], axis=2) 679 | # x_rightarmdown_2 = np.mean(X[:, :, right_arm_down_2, :], axis=2) 680 | # x_rightarmdown_3 = np.mean(X[:, :, right_arm_down_3, :], axis=2) 681 | shoulder_1 = np.mean(X[:, :, shoulder_1, :], axis=2) 682 | shoulder_2 = np.mean(X[:, :, shoulder_2, :], axis=2) 683 | elbow_1 = np.mean(X[:, :, elbow_1, :], axis=2) 684 | elbow_2 = np.mean(X[:, :, elbow_2, :], axis=2) 685 | spine_mm = np.mean(X[:, :, spine_mm, :], axis=2) 686 | hip_1 = np.mean(X[:, :, hip_1, :], axis=2) 687 | hip_2 = np.mean(X[:, :, hip_2, :], axis=2) 688 | knee_1 = np.mean(X[:, :, knee_1, :], axis=2) 689 | knee_2 = np.mean(X[:, :, knee_2, :], axis=2) 690 | X_part = np.concatenate((x_leftlegup, x_leftlegdown, x_rightlegup, 691 | x_rightlegdown, x_torso, x_head_1, x_leftarmup, 692 | x_leftarmdown_1, 693 | x_rightarmup, x_rightarmdown_1, 694 | shoulder_1, shoulder_2, elbow_1, elbow_2, spine_mm, 695 | hip_1, hip_2, knee_1, knee_2), axis=-1) \ 696 | .reshape([X.shape[0], X.shape[1], 19, 3]) 697 | # 25+24 698 | X_interp = np.concatenate((X, X_part), axis=-2) 699 | return X_interp 700 | 701 | def generate_denser_graph_data(X, adj, joint_num=20): 702 | adj_temp = copy.deepcopy(adj) 703 | adj_temp = adj_temp.tolist() 704 | node_num = len(adj_temp) 705 | cnt = node_num 706 | for i in range(node_num): 707 | for j in range(node_num): 708 | if adj_temp[i][j] == 1: 709 | adj_temp[i][j] = adj_temp[j][i] = 0 710 | new_node = np.mean(X[:, :, [i, j], :], axis=2) 711 | # print(new_node.shape) 712 | if cnt == node_num: 713 | X_interp = new_node 714 | else: 715 | X_interp = np.concatenate((X_interp, new_node), axis=-1) 716 | # print(X_interp.shape) 717 | # print(i, j) 718 | # print(i, j, cnt) 719 | cnt += 1 720 | # print(X_interp.shape) 721 | X_interp = np.reshape(X_interp, [X.shape[0], X.shape[1], node_num-1, 3]) 722 | X_interp = np.concatenate((X, X_interp), axis=-2) 723 | return X_interp 724 | 725 | """ 726 | Calculate normalized area under curves. 727 | """ 728 | def cal_nAUC(scores, labels): 729 | scores = np.array(scores) 730 | labels = np.array(labels) 731 | # Compute micro-average ROC curve and ROC area 732 | fpr, tpr, thresholds = roc_curve(labels.ravel(), scores.ravel()) 733 | roc_auc = auc(fpr, tpr) 734 | return roc_auc 735 | 736 | """ 737 | Generate training data with evenly distributed classes. 738 | """ 739 | # replaced by random shuffling 740 | def class_samp_gen(X, y, ids_, batch_size): 741 | class_num = len(ids_.keys()) 742 | ids_ = sorted(ids_.items(), key=lambda item: item[0]) 743 | cnt = 0 744 | all_batch_X = [] 745 | all_batch_y = [] 746 | total = y.shape[0] 747 | batch_num = total // batch_size * 2 748 | class_in_bacth = class_num 749 | batch_per_class = batch_size // class_in_bacth 750 | class_cnt = class_in_bacth 751 | # print(total, batch_num, batch_per_class) 752 | for i in range(batch_num): 753 | batch_X = [] 754 | batch_y = [] 755 | for k, v in ids_[class_cnt-class_in_bacth:class_cnt]: 756 | # print(k, len(v)) 757 | # cnt += len(v) 758 | if len(v[batch_per_class*i:batch_per_class*(i+1)]) < batch_per_class: 759 | rand_ind = np.random.choice(len(v), batch_per_class) 760 | v_array = np.array(v) 761 | samp_per_class = v_array[rand_ind].tolist() 762 | batch_X.extend(samp_per_class) 763 | else: 764 | batch_X.extend(v[batch_per_class*i:batch_per_class*(i+1)]) 765 | batch_y.extend(batch_per_class * [k]) 766 | if class_cnt + class_in_bacth > class_num and class_cnt <= class_num: 767 | class_cnt = class_num 768 | else: 769 | class_cnt = class_cnt + class_in_bacth 770 | # print(len(batch_X)) 771 | # exit() 772 | all_batch_X.extend(batch_X) 773 | all_batch_y.extend(batch_y) 774 | # print(len(all_batch_X), len(all_batch_y)) 775 | X_train = X[all_batch_X] 776 | y_train = np.array(all_batch_y) 777 | return X_train, y_train 778 | 779 | """ 780 | Prepare adjacency matrix by expanding up to a given neighbourhood. 781 | This will insert loops on every node. 782 | Finally, the matrix is converted to bias vectors. 783 | Expected shape: [graph, nodes, nodes] 784 | """ 785 | 786 | 787 | def adj_to_bias(adj, sizes, nhood=1): 788 | nb_graphs = adj.shape[0] 789 | mt = np.empty(adj.shape) 790 | for g in range(nb_graphs): 791 | mt[g] = np.eye(adj.shape[1]) 792 | for _ in range(nhood): 793 | mt[g] = np.matmul(mt[g], (adj[g] + np.eye(adj.shape[1]))) 794 | for i in range(sizes[g]): 795 | for j in range(sizes[g]): 796 | if mt[g][i][j] > 0.0: 797 | mt[g][i][j] = 1.0 798 | return -1e9 * (1.0 - mt) 799 | 800 | 801 | ############################################### 802 | # This section of code adapted from tkipf/gcn # 803 | ############################################### 804 | 805 | def parse_index_file(filename): 806 | """Parse index file.""" 807 | index = [] 808 | for line in open(filename): 809 | index.append(int(line.strip())) 810 | return index 811 | 812 | 813 | def sample_mask(idx, l): 814 | """Create mask.""" 815 | mask = np.zeros(l) 816 | mask[idx] = 1 817 | return np.array(mask, dtype=np.bool) 818 | 819 | 820 | def load_data(dataset_str): # {'pubmed', 'citeseer', 'cora'} 821 | """Load data.""" 822 | names = ['x', 'y', 'tx', 'ty', 'allx', 'ally', 'graph'] 823 | objects = [] 824 | for i in range(len(names)): 825 | with open("data/ind.{}.{}".format(dataset_str, names[i]), 'rb') as f: 826 | if sys.version_info > (3, 0): 827 | objects.append(pkl.load(f, encoding='latin1')) 828 | else: 829 | objects.append(pkl.load(f)) 830 | 831 | x, y, tx, ty, allx, ally, graph = tuple(objects) 832 | test_idx_reorder = parse_index_file("data/ind.{}.test.index".format(dataset_str)) 833 | test_idx_range = np.sort(test_idx_reorder) 834 | 835 | if dataset_str == 'citeseer': 836 | # Fix citeseer dataset (there are some isolated nodes in the graph) 837 | # Find isolated nodes, add them as zero-vecs into the right position 838 | test_idx_range_full = range(min(test_idx_reorder), max(test_idx_reorder) + 1) 839 | tx_extended = sp.lil_matrix((len(test_idx_range_full), x.shape[1])) 840 | tx_extended[test_idx_range - min(test_idx_range), :] = tx 841 | tx = tx_extended 842 | ty_extended = np.zeros((len(test_idx_range_full), y.shape[1])) 843 | ty_extended[test_idx_range - min(test_idx_range), :] = ty 844 | ty = ty_extended 845 | 846 | features = sp.vstack((allx, tx)).tolil() 847 | features[test_idx_reorder, :] = features[test_idx_range, :] 848 | adj = nx.adjacency_matrix(nx.from_dict_of_lists(graph)) 849 | 850 | labels = np.vstack((ally, ty)) 851 | labels[test_idx_reorder, :] = labels[test_idx_range, :] 852 | 853 | idx_test = test_idx_range.tolist() 854 | idx_train = range(len(y)) 855 | idx_val = range(len(y), len(y) + 500) 856 | 857 | train_mask = sample_mask(idx_train, labels.shape[0]) 858 | val_mask = sample_mask(idx_val, labels.shape[0]) 859 | test_mask = sample_mask(idx_test, labels.shape[0]) 860 | 861 | y_train = np.zeros(labels.shape) 862 | y_val = np.zeros(labels.shape) 863 | y_test = np.zeros(labels.shape) 864 | y_train[train_mask, :] = labels[train_mask, :] 865 | y_val[val_mask, :] = labels[val_mask, :] 866 | y_test[test_mask, :] = labels[test_mask, :] 867 | 868 | # print(adj.shape) 869 | # print(features.shape) 870 | 871 | return adj, features, y_train, y_val, y_test, train_mask, val_mask, test_mask 872 | 873 | 874 | def load_random_data(size): 875 | adj = sp.random(size, size, density=0.002) # density similar to cora 876 | features = sp.random(size, 1000, density=0.015) 877 | int_labels = np.random.randint(7, size=(size)) 878 | labels = np.zeros((size, 7)) # Nx7 879 | labels[np.arange(size), int_labels] = 1 880 | 881 | train_mask = np.zeros((size,)).astype(bool) 882 | train_mask[np.arange(size)[0:int(size / 2)]] = 1 883 | 884 | val_mask = np.zeros((size,)).astype(bool) 885 | val_mask[np.arange(size)[int(size / 2):]] = 1 886 | 887 | test_mask = np.zeros((size,)).astype(bool) 888 | test_mask[np.arange(size)[int(size / 2):]] = 1 889 | 890 | y_train = np.zeros(labels.shape) 891 | y_val = np.zeros(labels.shape) 892 | y_test = np.zeros(labels.shape) 893 | y_train[train_mask, :] = labels[train_mask, :] 894 | y_val[val_mask, :] = labels[val_mask, :] 895 | y_test[test_mask, :] = labels[test_mask, :] 896 | 897 | # sparse NxN, sparse NxF, norm NxC, ..., norm Nx1, ... 898 | return adj, features, y_train, y_val, y_test, train_mask, val_mask, test_mask 899 | 900 | 901 | def sparse_to_tuple(sparse_mx): 902 | """Convert sparse matrix to tuple representation.""" 903 | 904 | def to_tuple(mx): 905 | if not sp.isspmatrix_coo(mx): 906 | mx = mx.tocoo() 907 | coords = np.vstack((mx.row, mx.col)).transpose() 908 | values = mx.data 909 | shape = mx.shape 910 | return coords, values, shape 911 | 912 | if isinstance(sparse_mx, list): 913 | for i in range(len(sparse_mx)): 914 | sparse_mx[i] = to_tuple(sparse_mx[i]) 915 | else: 916 | sparse_mx = to_tuple(sparse_mx) 917 | 918 | return sparse_mx 919 | 920 | 921 | def standardize_data(f, train_mask): 922 | """Standardize feature matrix and convert to tuple representation""" 923 | # standardize data 924 | f = f.todense() 925 | mu = f[train_mask == True, :].mean(axis=0) 926 | sigma = f[train_mask == True, :].std(axis=0) 927 | f = f[:, np.squeeze(np.array(sigma > 0))] 928 | mu = f[train_mask == True, :].mean(axis=0) 929 | sigma = f[train_mask == True, :].std(axis=0) 930 | f = (f - mu) / sigma 931 | return f 932 | 933 | 934 | def preprocess_features(features): 935 | """Row-normalize feature matrix and convert to tuple representation""" 936 | rowsum = np.array(features.sum(1)) 937 | r_inv = np.power(rowsum, -1).flatten() 938 | r_inv[np.isinf(r_inv)] = 0. 939 | r_mat_inv = sp.diags(r_inv) 940 | features = r_mat_inv.dot(features) 941 | return features.todense(), sparse_to_tuple(features) 942 | 943 | 944 | def normalize_adj(adj): 945 | """Symmetrically normalize adjacency matrix.""" 946 | adj = sp.coo_matrix(adj) 947 | rowsum = np.array(adj.sum(1)) 948 | d_inv_sqrt = np.power(rowsum, -0.5).flatten() 949 | d_inv_sqrt[np.isinf(d_inv_sqrt)] = 0. 950 | d_mat_inv_sqrt = sp.diags(d_inv_sqrt) 951 | return adj.dot(d_mat_inv_sqrt).transpose().dot(d_mat_inv_sqrt).tocoo() 952 | 953 | 954 | def preprocess_adj(adj): 955 | """Preprocessing of adjacency matrix for simple GCN model and conversion to tuple representation.""" 956 | adj_normalized = normalize_adj(adj + sp.eye(adj.shape[0])) 957 | return sparse_to_tuple(adj_normalized) 958 | 959 | 960 | def preprocess_adj_bias(adj): 961 | num_nodes = adj.shape[0] 962 | adj = adj + sp.eye(num_nodes) # self-loop 963 | adj[adj > 0.0] = 1.0 964 | if not sp.isspmatrix_coo(adj): 965 | adj = adj.tocoo() 966 | adj = adj.astype(np.float32) 967 | indices = np.vstack( 968 | (adj.col, adj.row)).transpose() # This is where I made a mistake, I used (adj.row, adj.col) instead 969 | # return tf.SparseTensor(indices=indices, values=adj.data, dense_shape=adj.shape) 970 | return indices, adj.data, adj.shape 971 | -------------------------------------------------------------------------------- /Data-process.py: -------------------------------------------------------------------------------- 1 | import os 2 | import matplotlib as mpl 3 | mpl.use('Agg') 4 | import matplotlib.pyplot as plt 5 | import pandas as pd 6 | import numpy as np 7 | from sklearn import preprocessing 8 | import copy 9 | clip_num = 10 10 | overlap = 20 11 | time_steps = 20 12 | 13 | def preprocess_IAS(type, clip_num=10): 14 | cnt = 0 15 | p_txt = [] 16 | p = [] 17 | if type == 'A': 18 | name = 'TestingA' 19 | elif type == 'B': 20 | name = 'TestingB' 21 | elif type == 'Train': 22 | name = 'Training' 23 | p_gal = [] 24 | p_gal_txt = [] 25 | print('(IAS - %s) txt -> npy' % name) 26 | all_person_folder = os.listdir('./IAS/' + name) 27 | for a, b, c in os.walk('./IAS/' + name): 28 | if len(b) == 0: 29 | txt_t = [] 30 | for file in c: 31 | if 'skel.txt' in file: 32 | txt_t.append(a + '/' + file) 33 | txt_t.sort() 34 | # print(len(txt_t)) 35 | if type == 'Train': 36 | # for train random half 37 | # perm = np.random.permutation(len(txt_t)) 38 | # txt_t = np.array(txt_t)[perm].tolist() 39 | # p_txt.append(txt_t[:len(txt_t)//2]) 40 | p_txt.append(txt_t) 41 | p.append(cnt) 42 | # for gallery 43 | # p_gal_txt.append(txt_t[len(txt_t) // 2:]) 44 | # p_gal.append(cnt) 45 | else: 46 | p_txt.append(txt_t) 47 | p.append(cnt) 48 | cnt += 1 49 | assert len(p_txt) == len(p) 50 | X = [] 51 | Y = [] 52 | Z = [] 53 | f_num = 0 54 | abnormal_num = 0 55 | normal_num = 0 56 | for index, txts_dir in enumerate(p_txt): 57 | # print(len(txts_dir)) 58 | # print(index) 59 | # print(txts_dir) 60 | p_x = [] 61 | p_y = [] 62 | p_z = [] 63 | for txt in txts_dir: 64 | track_state = True 65 | with open(txt, 'r') as f: 66 | lines = f.readlines() 67 | label_t = [] 68 | x_t = [] 69 | y_t = [] 70 | z_t = [] 71 | for line in lines: 72 | temp = line.split(',') 73 | TrackingState = float(temp[6]) 74 | QualityFlage = float(temp[7]) 75 | if TrackingState == 0: 76 | track_state = False 77 | break 78 | label_t.append(float(temp[0])) 79 | x_t.append(float(temp[1])) 80 | y_t.append(float(temp[2])) 81 | z_t.append(float(temp[3])) 82 | if not track_state: 83 | abnormal_num += 1 84 | continue 85 | else: 86 | normal_num += 1 87 | p_x.append(x_t) 88 | p_y.append(y_t) 89 | p_z.append(z_t) 90 | # print(len(p_x)) 91 | assert len(p_x) == len(p_y) and len(p_y) == len(p_z) 92 | # print(len(p_x[clip_num:-clip_num])) 93 | if type != 'Train': 94 | f_num += len(p_x[clip_num:-clip_num]) 95 | X.append(p_x[clip_num:-clip_num]) 96 | Y.append(p_y[clip_num:-clip_num]) 97 | Z.append(p_z[clip_num:-clip_num]) 98 | else: 99 | f_num += len(p_x[clip_num:]) 100 | X.append(p_x[clip_num:]) 101 | Y.append(p_y[clip_num:]) 102 | Z.append(p_z[clip_num:]) 103 | 104 | # print() 105 | # print(str(normal_num / len(set(p)))) 106 | # print(str(f_num / len(set(p)))) 107 | # print(abnormal_num) 108 | # print(normal_num) 109 | np.save('./IAS/' + name + '_label.npy', p) 110 | np.save('./IAS/' + name + '_x.npy', X) 111 | np.save('./IAS/' + name + '_y.npy', Y) 112 | np.save('./IAS/' + name + '_z.npy', Z) 113 | # print(X) 114 | # print(np.array(X).shape) 115 | if type == 'Train': 116 | X = [] 117 | Y = [] 118 | Z = [] 119 | f_num = 0 120 | abnormal_num = 0 121 | normal_num = 0 122 | for index, txts_dir in enumerate(p_gal_txt): 123 | # print(index) 124 | # print(txts_dir) 125 | p_x = [] 126 | p_y = [] 127 | p_z = [] 128 | for txt in txts_dir: 129 | track_state = True 130 | with open(txt, 'r') as f: 131 | lines = f.readlines() 132 | label_t = [] 133 | x_t = [] 134 | y_t = [] 135 | z_t = [] 136 | for line in lines: 137 | temp = line.split(',') 138 | TrackingState = float(temp[6]) 139 | QualityFlage = float(temp[7]) 140 | if TrackingState == 0: 141 | track_state = False 142 | break 143 | label_t.append(float(temp[0])) 144 | x_t.append(float(temp[1])) 145 | y_t.append(float(temp[2])) 146 | z_t.append(float(temp[3])) 147 | if not track_state: 148 | abnormal_num += 1 149 | continue 150 | else: 151 | normal_num += 1 152 | p_x.append(x_t) 153 | p_y.append(y_t) 154 | p_z.append(z_t) 155 | assert len(p_x) == len(p_y) and len(p_y) == len(p_z) 156 | # print(len(p_x[clip_num:-clip_num])) 157 | # f_num += len(p_x[clip_num:-clip_num]) 158 | X.append(p_x[:-clip_num]) 159 | # print(len(X), len(X[0])) 160 | Y.append(p_y[:-clip_num]) 161 | Z.append(p_z[:-clip_num]) 162 | 163 | # print() 164 | # print(str(normal_num / len(set(p)))) 165 | # print(str(f_num / len(set(p)))) 166 | # print(abnormal_num) 167 | # print(normal_num) 168 | np.save('./IAS/Gallery_label.npy', p_gal) 169 | np.save('./IAS/Gallery_x.npy', X) 170 | np.save('./IAS/Gallery_y.npy', Y) 171 | np.save('./IAS/Gallery_z.npy', Z) 172 | # print(np.array(X).shape) 173 | 174 | 175 | def preprocess_BIWI(type, clip_num=10): 176 | cnt = 0 177 | p = [] 178 | p_txt = [] 179 | if type == 'W': 180 | name = 'Testing/Walking' 181 | elif type == 'S': 182 | name = 'Testing/Still' 183 | elif type == 'Train': 184 | name = 'Training' 185 | print('(BIWI - %s) txt -> npy' % name) 186 | for a, b, c in os.walk('./BIWI/' + name): 187 | if len(b) == 0: 188 | txt_t = [] 189 | for file in c: 190 | if 'skel.txt' in file: 191 | txt_t.append(a + '/' + file) 192 | txt_t.sort() 193 | p_txt.append(txt_t) 194 | p_num = int(a.split('/')[-1]) 195 | # print(p_num) 196 | p.append(p_num) 197 | cnt += 1 198 | assert len(p_txt) == len(p) 199 | X = [] 200 | Y = [] 201 | Z = [] 202 | f_num = 0 203 | abnormal_num = 0 204 | normal_num = 0 205 | for index, txts_dir in enumerate(p_txt): 206 | # print(index) 207 | # print(txts_dir) 208 | p_x = [] 209 | p_y = [] 210 | p_z = [] 211 | for txt in txts_dir: 212 | track_state = True 213 | with open(txt, 'r') as f: 214 | lines = f.readlines() 215 | label_t = [] 216 | x_t = [] 217 | y_t = [] 218 | z_t = [] 219 | for index, line in enumerate(lines[:20]): 220 | temp = line.split(',') 221 | TrackingState = float(temp[6]) 222 | QualityFlage = float(temp[7]) 223 | if TrackingState == 0 and index < 20: 224 | track_state = False 225 | break 226 | label_t.append(float(temp[0])) 227 | x_t.append(float(temp[1])) 228 | y_t.append(float(temp[2])) 229 | z_t.append(float(temp[3])) 230 | # if QualityFlage > 0: 231 | # print(line) 232 | if not track_state: 233 | abnormal_num += 1 234 | continue 235 | else: 236 | normal_num += 1 237 | p_x.append(x_t) 238 | p_y.append(y_t) 239 | p_z.append(z_t) 240 | assert len(p_x) == len(p_y) and len(p_y) == len(p_z) 241 | # print(len(p_x[clip_num:-clip_num])) 242 | f_num += len(p_x[clip_num:-clip_num]) 243 | X.append(p_x[clip_num:-clip_num]) 244 | Y.append(p_y[clip_num:-clip_num]) 245 | Z.append(p_z[clip_num:-clip_num]) 246 | # print(p) 247 | # print() 248 | # print(str(normal_num / len(set(p)))) 249 | # print(str(f_num / len(set(p)))) 250 | # print(abnormal_num) 251 | # print(normal_num) 252 | if type == 'W': 253 | np.save('./BIWI/Walking_label.npy', p) 254 | np.save('./BIWI/Walking_x.npy', X) 255 | np.save('./BIWI/Walking_y.npy', Y) 256 | np.save('./BIWI/Walking_z.npy', Z) 257 | elif type == 'S': 258 | np.save('./BIWI/Still_label.npy', p) 259 | np.save('./BIWI/Still_x.npy', X) 260 | np.save('./BIWI/Still_y.npy', Y) 261 | np.save('./BIWI/Still_z.npy', Z) 262 | else: 263 | np.save('./BIWI/Train_label.npy', p) 264 | np.save('./BIWI/Train_x.npy', X) 265 | np.save('./BIWI/Train_y.npy', Y) 266 | np.save('./BIWI/Train_z.npy', Z) 267 | 268 | 269 | 270 | # preprocess KGBD dataset 271 | def process_dataset_KGBD(save_dir, fr=6): 272 | global overlap, time_steps 273 | try: 274 | os.mkdir('Datasets/KGBD/') 275 | except: 276 | pass 277 | save_dir = 'Datasets/KGBD/' + save_dir 278 | try: 279 | os.mkdir(save_dir) 280 | except: 281 | pass 282 | overlap = fr 283 | time_steps = fr 284 | a = 'KGBD/kinect gait raw dataset/' 285 | p_dir = os.listdir(a) 286 | label = [] 287 | X = [] 288 | Y = [] 289 | Z = [] 290 | gal_label = [] 291 | gal_X = [] 292 | gal_Y = [] 293 | gal_Z = [] 294 | pro_label = [] 295 | pro_X = [] 296 | pro_Y = [] 297 | pro_Z = [] 298 | for p in p_dir: 299 | if 'Person' not in p: 300 | continue 301 | # print(p) 302 | p_num = int(p.split('Person')[1]) 303 | data_dir = a + p 304 | fold_dir = os.listdir(data_dir) 305 | fold_dir = sorted(fold_dir) 306 | if p_num == 15: 307 | # only 3 txt, for train (1) + probe (1) + gallery (1) 308 | rand_seq_nums = np.random.choice(len(fold_dir), 2, replace=False) 309 | gal_seq_num = rand_seq_nums[:1] 310 | pro_seq_num = rand_seq_nums[1] 311 | print('Person ID: %d | Gallery Seqs: %d | Probe Seq: %d | Total Seq: %d' % ( 312 | p_num, gal_seq_num[0], pro_seq_num, len(fold_dir))) 313 | else: 314 | rand_seq_nums = np.random.choice(len(fold_dir), 3, replace=False) 315 | gal_seq_num = rand_seq_nums[:2] 316 | pro_seq_num = rand_seq_nums[2] 317 | print('Person ID: %d | Gallery Seqs: %d, %d| Probe Seq: %d | Total Seq: %d' % (p_num, gal_seq_num[0], gal_seq_num[1], pro_seq_num, len(fold_dir))) 318 | for seq_num, txt_ind in enumerate(fold_dir): 319 | with open(a + p + '/' + txt_ind, 'r') as f: 320 | lines = f.readlines() 321 | x = [] 322 | y = [] 323 | z = [] 324 | x_t = [] 325 | y_t = [] 326 | z_t = [] 327 | cnt = 0 328 | for line in lines: 329 | coors = line.split(';')[1:] 330 | x_t.append(float(coors[0])) 331 | y_t.append(float(coors[1])) 332 | z_t.append(float(coors[2])) 333 | cnt += 1 334 | if cnt % 20 == 0: 335 | x.append(x_t) 336 | y.append(y_t) 337 | z.append(z_t) 338 | x_t = [] 339 | y_t = [] 340 | z_t = [] 341 | if seq_num not in rand_seq_nums: 342 | X.append(x[clip_num:-clip_num]) 343 | Y.append(y[clip_num:-clip_num]) 344 | Z.append(z[clip_num:-clip_num]) 345 | label.append(p_num) 346 | elif seq_num in gal_seq_num: 347 | gal_X.append(x[clip_num:-clip_num]) 348 | gal_Y.append(y[clip_num:-clip_num]) 349 | gal_Z.append(z[clip_num:-clip_num]) 350 | gal_label.append(p_num) 351 | elif seq_num == pro_seq_num: 352 | pro_X.append(x[clip_num:-clip_num]) 353 | pro_Y.append(y[clip_num:-clip_num]) 354 | pro_Z.append(z[clip_num:-clip_num]) 355 | pro_label.append(p_num) 356 | 357 | def get_npy_data(save_dir, X, Y ,Z, label, type): 358 | x_source = [] 359 | y_source = [] 360 | z_source = [] 361 | # x_target = [] 362 | # y_target = [] 363 | # z_target = [] 364 | frames_cnt = 0 365 | ids = {} 366 | frame_id = [] 367 | for s in range(0, 1): 368 | for index, xx_row in enumerate(X): 369 | # for frame in x_row: 370 | # for i in range(1, 25): 371 | # frame[i] -= frame[0] 372 | x_row = copy.deepcopy(xx_row[s:]) 373 | if type=='XX': 374 | t = [x_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 375 | range((len(x_row) - overlap) // (overlap // 2) + 1)] 376 | else: 377 | t = [x_row[(i * overlap): (i * overlap + overlap)] for i in 378 | range(len(x_row) // overlap)] 379 | person = label[index] 380 | if person not in ids.keys(): 381 | ids[person] = [] 382 | if len(t) > 0: 383 | ids[person].extend([i + frames_cnt for i in range(len(t))]) 384 | frame_id.extend([person for i in range(len(t))]) 385 | frames_cnt += len(t) 386 | # rev_t = copy.deepcopy(t) 387 | # for k in rev_t: 388 | # k.reverse() 389 | if len(t) > 0: 390 | t = sum(t, []) 391 | x_source.extend(t) 392 | # rev_t = sum(rev_t, []) 393 | # x_target.extend(rev_t) 394 | for index, yy_row in enumerate(Y): 395 | # for frame in y_row: 396 | # for i in range(1, 25): 397 | # frame[i] -= frame[0] 398 | y_row = yy_row[s:] 399 | if type == 'XX': 400 | t = [y_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 401 | range((len(y_row) - overlap) // (overlap // 2) + 1)] 402 | else: 403 | t = [y_row[(i * overlap): (i * overlap + overlap)] for i in 404 | range(len(y_row) // overlap)] 405 | # rev_t = copy.deepcopy(t) 406 | # for k in rev_t: 407 | # k.reverse() 408 | if len(t) > 0: 409 | t = sum(t, []) 410 | y_source.extend(t) 411 | # rev_t = sum(rev_t, []) 412 | # y_target.extend(rev_t) 413 | for index, zz_row in enumerate(Z): 414 | # for frame in z_row: 415 | # for i in range(1, 25): 416 | # frame[i] -= frame[0] 417 | z_row = zz_row[s:] 418 | if type == 'XX': 419 | t = [z_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 420 | range((len(z_row) - overlap) // (overlap // 2) + 1)] 421 | else: 422 | t = [z_row[(i * overlap): (i * overlap + overlap)] for i in 423 | range(len(z_row) // overlap)] 424 | # rev_t = copy.deepcopy(t) 425 | # for k in rev_t: 426 | # k.reverse() 427 | if len(t) > 0: 428 | t = sum(t, []) 429 | z_source.extend(t) 430 | # rev_t = sum(rev_t, []) 431 | # z_target.extend(rev_t) 432 | x_source = np.array(x_source) 433 | y_source = np.array(y_source) 434 | z_source = np.array(z_source) 435 | # x_target = np.array(x_target) 436 | # y_target = np.array(y_target) 437 | # z_target = np.array(z_target) 438 | # x_data -= x_data[:, 0] 439 | # y_data -= y_data[:, 0] 440 | # z_data -= z_data[:, 0] 441 | assert len(x) == len(y) and len(y) == len(z) 442 | # for i in range(1, 165): 443 | # if i not in ids.keys(): 444 | # print('lack: ', i) 445 | if type == 'Train': 446 | save_dir += 'train_npy_data/' 447 | try: 448 | os.mkdir(save_dir) 449 | except: 450 | pass 451 | # permutation = np.random.permutation(x_source.shape[0]) 452 | # x_source = x_source[permutation,] 453 | # y_source = y_source[permutation,] 454 | # z_source = z_source[permutation,] 455 | # x_target = x_target[permutation,] 456 | # y_target = y_target[permutation,] 457 | # z_target = z_target[permutation,] 458 | # ids = ids[permutation,] 459 | # frame_id = frame_id[permutation,] 460 | np.save(save_dir + 'source_x_KGBD_' + str(fr) + '.npy', x_source) 461 | np.save(save_dir + 'source_y_KGBD_' + str(fr) + '.npy', y_source) 462 | np.save(save_dir + 'source_z_KGBD_' + str(fr) + '.npy', z_source) 463 | # np.save(save_dir + 'target_x_KGBD_' + str(fr) + '.npy', x_target) 464 | # np.save(save_dir + 'target_y_KGBD_' + str(fr) + '.npy', y_target) 465 | # np.save(save_dir + 'target_z_KGBD_' + str(fr) + '.npy', z_target) 466 | np.save(save_dir + 'ids_KGBD_' + str(fr) + '.npy', ids) 467 | np.save(save_dir + 'frame_id_KGBD_' + str(fr) + '.npy', frame_id) 468 | else: 469 | save_dir += 'test_npy_data/' 470 | try: 471 | os.mkdir(save_dir) 472 | except: 473 | pass 474 | save_dir += type + '/' 475 | try: 476 | os.mkdir(save_dir) 477 | except: 478 | pass 479 | # permutation = np.random.permutation(x_source.shape[0]) 480 | # x_source = x_source[permutation,] 481 | # y_source = y_source[permutation,] 482 | # z_source = z_source[permutation,] 483 | # x_target = x_target[permutation,] 484 | # y_target = y_target[permutation,] 485 | # z_target = z_target[permutation,] 486 | # ids = ids[permutation,] 487 | # frame_id = frame_id[permutation,] 488 | np.save(save_dir + 't_source_x_KGBD_' + str(fr) + '.npy', x_source) 489 | np.save(save_dir + 't_source_y_KGBD_' + str(fr) + '.npy', y_source) 490 | np.save(save_dir + 't_source_z_KGBD_' + str(fr) + '.npy', z_source) 491 | # np.save(save_dir + 't_target_x_KGBD_' + str(fr) + '.npy', x_target) 492 | # np.save(save_dir + 't_target_y_KGBD_' + str(fr) + '.npy', y_target) 493 | # np.save(save_dir + 't_target_z_KGBD_' + str(fr) + '.npy', z_target) 494 | np.save(save_dir + 'ids_KGBD_' + str(fr) + '.npy', ids) 495 | np.save(save_dir + 'frame_id_KGBD_' + str(fr) + '.npy', frame_id) 496 | 497 | get_npy_data(save_dir, X, Y, Z, label, type='Train') 498 | get_npy_data(save_dir, gal_X, gal_Y, gal_Z, gal_label, type='gallery') 499 | get_npy_data(save_dir, pro_X, pro_Y, pro_Z, pro_label, type='probe') 500 | 501 | 502 | # preprocess IAS dataset 503 | def process_dataset_IAS(save_dir, fr=6): 504 | preprocess_IAS('Train') 505 | preprocess_IAS('A') 506 | preprocess_IAS('B') 507 | try: 508 | os.mkdir('Datasets/IAS/') 509 | except: 510 | pass 511 | save_dir = 'Datasets/IAS/' + save_dir 512 | try: 513 | os.mkdir(save_dir) 514 | except: 515 | pass 516 | global overlap, time_steps 517 | overlap = fr 518 | time_steps = fr 519 | X = np.load('./IAS/Training_x.npy').tolist() 520 | Y = np.load('./IAS/Training_y.npy').tolist() 521 | Z = np.load('./IAS/Training_z.npy').tolist() 522 | label = np.load('./IAS/Training_label.npy').tolist() 523 | 524 | # gal_X = np.load('./IAS/Gallery_x.npy').tolist() 525 | # gal_Y = np.load('./IAS/Gallery_y.npy').tolist() 526 | # gal_Z = np.load('./IAS/Gallery_z.npy').tolist() 527 | # gal_label = np.load('./IAS/Gallery_label.npy').tolist() 528 | 529 | A_X = np.load('./IAS/TestingA_x.npy').tolist() 530 | A_Y = np.load('./IAS/TestingA_y.npy').tolist() 531 | A_Z = np.load('./IAS/TestingA_z.npy').tolist() 532 | A_label = np.load('./IAS/TestingA_label.npy').tolist() 533 | 534 | B_X = np.load('./IAS/TestingB_x.npy').tolist() 535 | B_Y = np.load('./IAS/TestingB_y.npy').tolist() 536 | B_Z = np.load('./IAS/TestingB_z.npy').tolist() 537 | B_label = np.load('./IAS/TestingB_label.npy').tolist() 538 | 539 | # print(len(X), len(X[0])) 540 | # exit() 541 | # X = X.tolist() 542 | # Y = Y.tolist() 543 | # Z = Z.tolist() 544 | # label = label.tolist() 545 | def get_npy_data(save_dir, X, Y ,Z, label, type): 546 | print('Process %s data' % type) 547 | x_source = [] 548 | y_source = [] 549 | z_source = [] 550 | # x_target = [] 551 | # y_target = [] 552 | # z_target = [] 553 | frames_cnt = 0 554 | ids = {} 555 | frame_id = [] 556 | for s in range(0, time_steps): 557 | for index, xx_row in enumerate(X): 558 | # for frame in x_row: 559 | # for i in range(1, 25): 560 | # frame[i] -= frame[0] 561 | x_row = copy.deepcopy(xx_row[s:]) 562 | # print(type, 'x_row', x_row) 563 | if type=='Train': 564 | t = [x_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 565 | range((len(x_row) - overlap) // (overlap // 2) + 1)] 566 | elif s == 0: 567 | t = [x_row[(i * overlap): (i * overlap + overlap)] for i in 568 | range(len(x_row) // overlap)] 569 | else: 570 | continue 571 | person = label[index] 572 | if person not in ids.keys(): 573 | ids[person] = [] 574 | if len(t) > 0: 575 | ids[person].extend([i + frames_cnt for i in range(len(t))]) 576 | frame_id.extend([person for i in range(len(t))]) 577 | frames_cnt += len(t) 578 | # rev_t = copy.deepcopy(t) 579 | # for k in rev_t: 580 | # k.reverse() 581 | # print('ori', t) 582 | if len(t) > 0: 583 | t = sum(t, []) 584 | x_source.extend(t) 585 | # rev_t = sum(rev_t, []) 586 | # x_target.extend(rev_t) 587 | for index, yy_row in enumerate(Y): 588 | # for frame in y_row: 589 | # for i in range(1, 25): 590 | # frame[i] -= frame[0] 591 | y_row = yy_row[s:] 592 | if type == 'Train': 593 | t = [y_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 594 | range((len(y_row) - overlap) // (overlap // 2) + 1)] 595 | elif s==0: 596 | t = [y_row[(i * overlap): (i * overlap + overlap)] for i in 597 | range(len(y_row) // overlap)] 598 | else: 599 | continue 600 | # rev_t = copy.deepcopy(t) 601 | # for k in rev_t: 602 | # k.reverse() 603 | if len(t) > 0: 604 | t = sum(t, []) 605 | y_source.extend(t) 606 | # rev_t = sum(rev_t, []) 607 | # y_target.extend(rev_t) 608 | for index, zz_row in enumerate(Z): 609 | # for frame in z_row: 610 | # for i in range(1, 25): 611 | # frame[i] -= frame[0] 612 | z_row = zz_row[s:] 613 | if type == 'Train': 614 | t = [z_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 615 | range((len(z_row) - overlap) // (overlap // 2) + 1)] 616 | elif s == 0: 617 | t = [z_row[(i * overlap): (i * overlap + overlap)] for i in 618 | range(len(z_row) // overlap)] 619 | else: 620 | continue 621 | # rev_t = copy.deepcopy(t) 622 | # for k in rev_t: 623 | # k.reverse() 624 | if len(t) > 0: 625 | t = sum(t, []) 626 | z_source.extend(t) 627 | # rev_t = sum(rev_t, []) 628 | # z_target.extend(rev_t) 629 | x_source = np.array(x_source) 630 | y_source = np.array(y_source) 631 | z_source = np.array(z_source) 632 | # x_target = np.array(x_target) 633 | # y_target = np.array(y_target) 634 | # z_target = np.array(z_target) 635 | # x_data -= x_data[:, 0] 636 | # y_data -= y_data[:, 0] 637 | # z_data -= z_data[:, 0] 638 | # assert len(x) == len(y) and len(y) == len(z) 639 | # for i in range(0, 11): 640 | # if i not in ids.keys(): 641 | # print('lack: ', type, i) 642 | if type == 'Train': 643 | save_dir += 'train_npy_data/' 644 | try: 645 | os.mkdir(save_dir) 646 | except: 647 | pass 648 | # permutation = np.random.permutation(x_source.shape[0]) 649 | # x_source = x_source[permutation,] 650 | # y_source = y_source[permutation,] 651 | # z_source = z_source[permutation,] 652 | # x_target = x_target[permutation,] 653 | # y_target = y_target[permutation,] 654 | # z_target = z_target[permutation,] 655 | # ids = ids[permutation,] 656 | # frame_id = frame_id[permutation,] 657 | np.save(save_dir + 'source_x_IAS_' + str(fr) + '.npy', x_source) 658 | np.save(save_dir + 'source_y_IAS_' + str(fr) + '.npy', y_source) 659 | np.save(save_dir + 'source_z_IAS_' + str(fr) + '.npy', z_source) 660 | # np.save(save_dir + 'target_x_IAS_' + str(fr) + '.npy', x_target) 661 | # np.save(save_dir + 'target_y_IAS_' + str(fr) + '.npy', y_target) 662 | # np.save(save_dir + 'target_z_IAS_' + str(fr) + '.npy', z_target) 663 | np.save(save_dir + 'ids_IAS_' + str(fr) + '.npy', ids) 664 | np.save(save_dir + 'frame_id_IAS_' + str(fr) + '.npy', frame_id) 665 | else: 666 | save_dir += 'test_npy_data/' 667 | try: 668 | os.mkdir(save_dir) 669 | except: 670 | pass 671 | save_dir += type + '/' 672 | try: 673 | os.mkdir(save_dir) 674 | except: 675 | pass 676 | # permutation = np.random.permutation(x_source.shape[0]) 677 | # x_source = x_source[permutation,] 678 | # y_source = y_source[permutation,] 679 | # z_source = z_source[permutation,] 680 | # x_target = x_target[permutation,] 681 | # y_target = y_target[permutation,] 682 | # z_target = z_target[permutation,] 683 | # ids = ids[permutation,] 684 | # frame_id = frame_id[permutation,] 685 | np.save(save_dir + 't_source_x_IAS_' + str(fr) + '.npy', x_source) 686 | np.save(save_dir + 't_source_y_IAS_' + str(fr) + '.npy', y_source) 687 | np.save(save_dir + 't_source_z_IAS_' + str(fr) + '.npy', z_source) 688 | # np.save(save_dir + 't_target_x_IAS-A_' + str(fr) + '.npy', x_target) 689 | # np.save(save_dir + 't_target_y_IAS-A_' + str(fr) + '.npy', y_target) 690 | # np.save(save_dir + 't_target_z_IAS-A_' + str(fr) + '.npy', z_target) 691 | np.save(save_dir + 'ids_IAS_' + str(fr) + '.npy', ids) 692 | np.save(save_dir + 'frame_id_IAS_' + str(fr) + '.npy', frame_id) 693 | 694 | get_npy_data(save_dir, X, Y, Z, label, type='Train') 695 | # get_npy_data(save_dir, gal_X, gal_Y, gal_Z, gal_label, type='gallery') 696 | get_npy_data(save_dir, A_X, A_Y, A_Z, A_label, type='A') 697 | get_npy_data(save_dir, B_X, B_Y, B_Z, B_label, type='B') 698 | 699 | 700 | 701 | # preprocess BIWI dataset 702 | def process_dataset_BIWI(save_dir, fr=6): 703 | preprocess_BIWI('Train') 704 | preprocess_BIWI('W') 705 | preprocess_BIWI('S') 706 | try: 707 | os.mkdir('Datasets/BIWI/') 708 | except: 709 | pass 710 | save_dir = 'Datasets/BIWI/' + save_dir 711 | try: 712 | os.mkdir(save_dir) 713 | except: 714 | pass 715 | global overlap, time_steps 716 | overlap = fr 717 | time_steps = fr 718 | X = np.load('./BIWI/Train_x.npy') 719 | Y = np.load('./BIWI/Train_y.npy') 720 | Z = np.load('./BIWI/Train_z.npy') 721 | label = np.load('./BIWI/Train_label.npy') 722 | 723 | W_X = np.load('./BIWI/Walking_x.npy').tolist() 724 | W_Y = np.load('./BIWI/Walking_y.npy').tolist() 725 | W_Z = np.load('./BIWI/Walking_z.npy').tolist() 726 | W_label = np.load('./BIWI/Walking_label.npy').tolist() 727 | 728 | S_X = np.load('./BIWI/Still_x.npy').tolist() 729 | S_Y = np.load('./BIWI/Still_y.npy').tolist() 730 | S_Z = np.load('./BIWI/Still_z.npy').tolist() 731 | S_label = np.load('./BIWI/Still_label.npy').tolist() 732 | 733 | 734 | def get_npy_data(save_dir, X, Y ,Z, label, type): 735 | print('Process %s data' % type) 736 | x_source = [] 737 | y_source = [] 738 | z_source = [] 739 | # x_target = [] 740 | # y_target = [] 741 | # z_target = [] 742 | frames_cnt = 0 743 | ids = {} 744 | frame_id = [] 745 | for s in range(0, time_steps): 746 | for index, xx_row in enumerate(X): 747 | # for frame in x_row: 748 | # for i in range(1, 25): 749 | # frame[i] -= frame[0] 750 | x_row = copy.deepcopy(xx_row[s:]) 751 | if type=='Train': 752 | t = [x_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 753 | range((len(x_row) - overlap) // (overlap // 2) + 1)] 754 | elif s==0: 755 | t = [x_row[(i * overlap): (i * overlap + overlap)] for i in 756 | range(len(x_row) // overlap)] 757 | else: 758 | continue 759 | person = label[index] 760 | if person not in ids.keys(): 761 | ids[person] = [] 762 | if len(t) > 0: 763 | ids[person].extend([i + frames_cnt for i in range(len(t))]) 764 | frame_id.extend([person for i in range(len(t))]) 765 | frames_cnt += len(t) 766 | # rev_t = copy.deepcopy(t) 767 | # for k in rev_t: 768 | # k.reverse() 769 | if len(t) > 0: 770 | t = sum(t, []) 771 | x_source.extend(t) 772 | # rev_t = sum(rev_t, []) 773 | # x_target.extend(rev_t) 774 | for index, yy_row in enumerate(Y): 775 | # for frame in y_row: 776 | # for i in range(1, 25): 777 | # frame[i] -= frame[0] 778 | y_row = yy_row[s:] 779 | if type == 'Train': 780 | t = [y_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 781 | range((len(y_row) - overlap) // (overlap // 2) + 1)] 782 | elif s==0: 783 | t = [y_row[(i * overlap): (i * overlap + overlap)] for i in 784 | range(len(y_row) // overlap)] 785 | else: 786 | continue 787 | # rev_t = copy.deepcopy(t) 788 | # for k in rev_t: 789 | # k.reverse() 790 | if len(t) > 0: 791 | t = sum(t, []) 792 | y_source.extend(t) 793 | # rev_t = sum(rev_t, []) 794 | # y_target.extend(rev_t) 795 | for index, zz_row in enumerate(Z): 796 | # for frame in z_row: 797 | # for i in range(1, 25): 798 | # frame[i] -= frame[0] 799 | z_row = zz_row[s:] 800 | if type == 'Train': 801 | t = [z_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 802 | range((len(z_row) - overlap) // (overlap // 2) + 1)] 803 | elif s==0: 804 | t = [z_row[(i * overlap): (i * overlap + overlap)] for i in 805 | range(len(z_row) // overlap)] 806 | else: 807 | continue 808 | # rev_t = copy.deepcopy(t) 809 | # for k in rev_t: 810 | # k.reverse() 811 | if len(t) > 0: 812 | t = sum(t, []) 813 | z_source.extend(t) 814 | # rev_t = sum(rev_t, []) 815 | # z_target.extend(rev_t) 816 | x_source = np.array(x_source) 817 | y_source = np.array(y_source) 818 | z_source = np.array(z_source) 819 | # x_target = np.array(x_target) 820 | # y_target = np.array(y_target) 821 | # z_target = np.array(z_target) 822 | # x_data -= x_data[:, 0] 823 | # y_data -= y_data[:, 0] 824 | # z_data -= z_data[:, 0] 825 | # assert len(x) == len(y) and len(y) == len(z) 826 | if type == 'Train': 827 | save_dir += 'train_npy_data/' 828 | try: 829 | os.mkdir(save_dir) 830 | except: 831 | pass 832 | # permutation = np.random.permutation(x_source.shape[0]) 833 | # x_source = x_source[permutation,] 834 | # y_source = y_source[permutation,] 835 | # z_source = z_source[permutation,] 836 | # x_target = x_target[permutation,] 837 | # y_target = y_target[permutation,] 838 | # z_target = z_target[permutation,] 839 | # ids = ids[permutation,] 840 | # frame_id = frame_id[permutation,] 841 | np.save(save_dir + 'source_x_BIWI_' + str(fr) + '.npy', x_source) 842 | np.save(save_dir + 'source_y_BIWI_' + str(fr) + '.npy', y_source) 843 | np.save(save_dir + 'source_z_BIWI_' + str(fr) + '.npy', z_source) 844 | # np.save(save_dir + 'target_x_IAS_' + str(fr) + '.npy', x_target) 845 | # np.save(save_dir + 'target_y_IAS_' + str(fr) + '.npy', y_target) 846 | # np.save(save_dir + 'target_z_IAS_' + str(fr) + '.npy', z_target) 847 | np.save(save_dir + 'ids_BIWI_' + str(fr) + '.npy', ids) 848 | np.save(save_dir + 'frame_id_BIWI_' + str(fr) + '.npy', frame_id) 849 | else: 850 | save_dir += 'test_npy_data/' 851 | try: 852 | os.mkdir(save_dir) 853 | except: 854 | pass 855 | save_dir += type + '/' 856 | try: 857 | os.mkdir(save_dir) 858 | except: 859 | pass 860 | # permutation = np.random.permutation(x_source.shape[0]) 861 | # x_source = x_source[permutation,] 862 | # y_source = y_source[permutation,] 863 | # z_source = z_source[permutation,] 864 | # x_target = x_target[permutation,] 865 | # y_target = y_target[permutation,] 866 | # z_target = z_target[permutation,] 867 | # ids = ids[permutation,] 868 | # frame_id = frame_id[permutation,] 869 | np.save(save_dir + 't_source_x_BIWI_' + str(fr) + '.npy', x_source) 870 | np.save(save_dir + 't_source_y_BIWI_' + str(fr) + '.npy', y_source) 871 | np.save(save_dir + 't_source_z_BIWI_' + str(fr) + '.npy', z_source) 872 | # np.save(save_dir + 't_target_x_IAS-A_' + str(fr) + '.npy', x_target) 873 | # np.save(save_dir + 't_target_y_IAS-A_' + str(fr) + '.npy', y_target) 874 | # np.save(save_dir + 't_target_z_IAS-A_' + str(fr) + '.npy', z_target) 875 | np.save(save_dir + 'ids_BIWI_' + str(fr) + '.npy', ids) 876 | np.save(save_dir + 'frame_id_BIWI_' + str(fr) + '.npy', frame_id) 877 | 878 | get_npy_data(save_dir, X, Y, Z, label, type='Train') 879 | get_npy_data(save_dir, W_X, W_Y, W_Z, W_label, type='Walking') 880 | get_npy_data(save_dir, S_X, S_Y, S_Z, S_label, type='Still') 881 | 882 | 883 | # preprocess KS20 dataset 884 | def process_dataset_KS20(save_dir, fr=10): 885 | global overlap, time_steps 886 | try: 887 | os.mkdir('Datasets/KS20/') 888 | except: 889 | pass 890 | save_dir = 'Datasets/KS20/' + save_dir 891 | try: 892 | os.mkdir(save_dir) 893 | except: 894 | pass 895 | overlap = fr 896 | time_steps = fr 897 | a = 'KS20/' 898 | label = [] 899 | X = [] 900 | Y = [] 901 | Z = [] 902 | label_gallery = [] 903 | X_gallery = [] 904 | Y_gallery = [] 905 | Z_gallery = [] 906 | label_probe = [] 907 | X_probe = [] 908 | Y_probe = [] 909 | Z_probe = [] 910 | views = ['frontal', 'left_diagonal', 'left_lateral', 'right_diagonal', 'right_lateral'] 911 | settings = ['gc1', 'gc2', 'gc3'] 912 | # bodies = ['1','2','3','4','7','8','9','11','12','13', 913 | # '14','15','16', '17', '19', '20', '21', '22', '23', '24'] 914 | # corresponeding testing views 915 | gallery_views = [] 916 | probe_views = [] 917 | skeleton_num = 0 918 | for i in range(20): 919 | gallery_num, probe_num = np.random.choice(3, 2, replace=False) 920 | # random_num = np.random.randint(0, 3) 921 | # test_views.append(settings[random_num]) 922 | gallery_views.append(settings[gallery_num]) 923 | probe_views.append(settings[probe_num]) 924 | for view in views: 925 | p_dir = os.listdir(a + view) 926 | current_num = -1 927 | p_dir = sorted(p_dir) 928 | for p in p_dir: 929 | # if 'ren.py' in p: 930 | # continue 931 | p_num = int(p.split('_')[1].split('body')[1]) 932 | if 1 <= p_num <= 4: 933 | p_num -= 1 934 | elif 7 <= p_num <= 9: 935 | p_num -= 3 936 | elif 11 <= p_num <= 17: 937 | p_num -= 4 938 | elif 19 <= p_num <= 24: 939 | p_num -= 5 940 | print('Process ID: %d | Gallery Seq: %s | Probe Seq: %s | Train Seq: Others' % (p_num, gallery_views[current_num], probe_views[current_num])) 941 | current_setting = p.split('_')[0] 942 | # 6 not in train or probe 943 | # if p_num == 6: 944 | # print() 945 | # print(current_setting) 946 | if current_num != p_num: 947 | # print(current_num, p_num, current_setting) 948 | if current_num != -1: 949 | if current_setting == gallery_views[current_num]: 950 | # print('gallery', current_setting) 951 | X_gallery.append(x) 952 | Y_gallery.append(y) 953 | Z_gallery.append(z) 954 | label_gallery.append(current_num) 955 | elif current_setting == probe_views[current_num]: 956 | # print('probe', current_setting) 957 | X_probe.append(x) 958 | Y_probe.append(y) 959 | Z_probe.append(z) 960 | label_probe.append(current_num) 961 | else: 962 | # print('train', current_setting) 963 | X.append(x) 964 | Y.append(y) 965 | Z.append(z) 966 | label.append(current_num) 967 | current_num = p_num 968 | x = [] 969 | y = [] 970 | z = [] 971 | timestamp = int(p.split('_')[2]) 972 | with open(a + view + '/' + p, 'r') as f: 973 | skeleton_num += 1 974 | lines = f.readlines() 975 | cnt = 0 976 | x_t = [] 977 | y_t = [] 978 | z_t = [] 979 | for i in range(25): 980 | coor_x, coor_y, coor_z = \ 981 | float(lines[i].split(',')[1]), float(lines[i].split(',')[2]), float(lines[i].split(',')[3]) 982 | x_t.append(coor_x) 983 | y_t.append(coor_y) 984 | z_t.append(coor_z) 985 | cnt += 1 986 | x.append(x_t) 987 | y.append(y_t) 988 | z.append(z_t) 989 | # print(skeleton_num) 990 | # exit(1) 991 | def get_npy_data(save_dir, X, Y ,Z, label, type): 992 | print('Process %s data' % type) 993 | x_source = [] 994 | y_source = [] 995 | z_source = [] 996 | # x_target = [] 997 | # y_target = [] 998 | # z_target = [] 999 | frames_cnt = 0 1000 | ids = {} 1001 | frame_id = [] 1002 | for s in range(0, time_steps): 1003 | for index, xx_row in enumerate(X): 1004 | # for frame in x_row: 1005 | # for i in range(1, 25): 1006 | # frame[i] -= frame[0] 1007 | x_row = copy.deepcopy(xx_row[s:]) 1008 | if type=='Train': 1009 | t = [x_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 1010 | range((len(x_row) - overlap) // (overlap // 2) + 1)] 1011 | elif s==0: 1012 | t = [x_row[(i * overlap): (i * overlap + overlap)] for i in 1013 | range(len(x_row) // overlap)] 1014 | else: 1015 | continue 1016 | person = label[index] 1017 | if person not in ids.keys(): 1018 | ids[person] = [] 1019 | if len(t) > 0: 1020 | ids[person].extend([i + frames_cnt for i in range(len(t))]) 1021 | frame_id.extend([person for i in range(len(t))]) 1022 | frames_cnt += len(t) 1023 | # rev_t = copy.deepcopy(t) 1024 | # for k in rev_t: 1025 | # k.reverse() 1026 | if len(t) > 0: 1027 | t = sum(t, []) 1028 | x_source.extend(t) 1029 | # rev_t = sum(rev_t, []) 1030 | # x_target.extend(rev_t) 1031 | for index, yy_row in enumerate(Y): 1032 | # for frame in y_row: 1033 | # for i in range(1, 25): 1034 | # frame[i] -= frame[0] 1035 | y_row = yy_row[s:] 1036 | if type == 'Train': 1037 | t = [y_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 1038 | range((len(y_row) - overlap) // (overlap // 2) + 1)] 1039 | elif s==0: 1040 | t = [y_row[(i * overlap): (i * overlap + overlap)] for i in 1041 | range(len(y_row) // overlap)] 1042 | else: 1043 | continue 1044 | # rev_t = copy.deepcopy(t) 1045 | # for k in rev_t: 1046 | # k.reverse() 1047 | if len(t) > 0: 1048 | t = sum(t, []) 1049 | y_source.extend(t) 1050 | # rev_t = sum(rev_t, []) 1051 | # y_target.extend(rev_t) 1052 | for index, zz_row in enumerate(Z): 1053 | # for frame in z_row: 1054 | # for i in range(1, 25): 1055 | # frame[i] -= frame[0] 1056 | z_row = zz_row[s:] 1057 | if type == 'Train': 1058 | t = [z_row[(i * overlap // 2): (i * overlap // 2 + overlap)] for i in 1059 | range((len(z_row) - overlap) // (overlap // 2) + 1)] 1060 | elif s==0: 1061 | t = [z_row[(i * overlap): (i * overlap + overlap)] for i in 1062 | range(len(z_row) // overlap)] 1063 | else: 1064 | continue 1065 | # rev_t = copy.deepcopy(t) 1066 | # for k in rev_t: 1067 | # k.reverse() 1068 | if len(t) > 0: 1069 | t = sum(t, []) 1070 | z_source.extend(t) 1071 | # rev_t = sum(rev_t, []) 1072 | # z_target.extend(rev_t) 1073 | x_source = np.array(x_source) 1074 | y_source = np.array(y_source) 1075 | z_source = np.array(z_source) 1076 | # x_target = np.array(x_target) 1077 | # y_target = np.array(y_target) 1078 | # z_target = np.array(z_target) 1079 | # x_data -= x_data[:, 0] 1080 | # y_data -= y_data[:, 0] 1081 | # z_data -= z_data[:, 0] 1082 | # assert len(x) == len(y) and len(y) == len(z) 1083 | # for i in range(0, 20): 1084 | # if i not in ids.keys(): 1085 | # print('lack: ', type, i) 1086 | if type == 'Train': 1087 | save_dir += 'train_npy_data/' 1088 | try: 1089 | os.mkdir(save_dir) 1090 | except: 1091 | pass 1092 | # permutation = np.random.permutation(x_source.shape[0]) 1093 | # x_source = x_source[permutation,] 1094 | # y_source = y_source[permutation,] 1095 | # z_source = z_source[permutation,] 1096 | # x_target = x_target[permutation,] 1097 | # y_target = y_target[permutation,] 1098 | # z_target = z_target[permutation,] 1099 | # ids = ids[permutation,] 1100 | # frame_id = frame_id[permutation,] 1101 | np.save(save_dir + 'source_x_KS20_' + str(fr) + '.npy', x_source) 1102 | np.save(save_dir + 'source_y_KS20_' + str(fr) + '.npy', y_source) 1103 | np.save(save_dir + 'source_z_KS20_' + str(fr) + '.npy', z_source) 1104 | # np.save(save_dir + 'target_x_IAS_' + str(fr) + '.npy', x_target) 1105 | # np.save(save_dir + 'target_y_IAS_' + str(fr) + '.npy', y_target) 1106 | # np.save(save_dir + 'target_z_IAS_' + str(fr) + '.npy', z_target) 1107 | np.save(save_dir + 'ids_KS20_' + str(fr) + '.npy', ids) 1108 | np.save(save_dir + 'frame_id_KS20_' + str(fr) + '.npy', frame_id) 1109 | else: 1110 | save_dir += 'test_npy_data/' 1111 | try: 1112 | os.mkdir(save_dir) 1113 | except: 1114 | pass 1115 | save_dir += type + '/' 1116 | try: 1117 | os.mkdir(save_dir) 1118 | except: 1119 | pass 1120 | # permutation = np.random.permutation(x_source.shape[0]) 1121 | # x_source = x_source[permutation,] 1122 | # y_source = y_source[permutation,] 1123 | # z_source = z_source[permutation,] 1124 | # x_target = x_target[permutation,] 1125 | # y_target = y_target[permutation,] 1126 | # z_target = z_target[permutation,] 1127 | # ids = ids[permutation,] 1128 | # frame_id = frame_id[permutation,] 1129 | np.save(save_dir + 't_source_x_KS20_' + str(fr) + '.npy', x_source) 1130 | np.save(save_dir + 't_source_y_KS20_' + str(fr) + '.npy', y_source) 1131 | np.save(save_dir + 't_source_z_KS20_' + str(fr) + '.npy', z_source) 1132 | # np.save(save_dir + 't_target_x_IAS-A_' + str(fr) + '.npy', x_target) 1133 | # np.save(save_dir + 't_target_y_IAS-A_' + str(fr) + '.npy', y_target) 1134 | # np.save(save_dir + 't_target_z_IAS-A_' + str(fr) + '.npy', z_target) 1135 | np.save(save_dir + 'ids_KS20_' + str(fr) + '.npy', ids) 1136 | np.save(save_dir + 'frame_id_KS20_' + str(fr) + '.npy', frame_id) 1137 | 1138 | get_npy_data(save_dir, X, Y, Z, label, type='Train') 1139 | get_npy_data(save_dir, X_gallery, Y_gallery, Z_gallery, label_gallery, type='gallery') 1140 | get_npy_data(save_dir, X_probe, Y_probe, Z_probe, label_probe, type='probe') 1141 | 1142 | 1143 | if __name__ == '__main__': 1144 | import sys 1145 | f = sys.argv[1] 1146 | f = int(f) 1147 | try: 1148 | os.mkdir('Datasets') 1149 | except: 1150 | pass 1151 | process_dataset_KGBD(str(f) + '/', f) 1152 | process_dataset_IAS(str(f) + '/', f) 1153 | process_dataset_BIWI(str(f) + '/', f) 1154 | process_dataset_KS20(str(f) + '/', f) -------------------------------------------------------------------------------- /MoCos.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | import torch 4 | import os, sys 5 | from utils import process_SG as process 6 | from tensorflow.python.layers.core import Dense 7 | from sklearn.preprocessing import label_binarize 8 | from sklearn.cluster import DBSCAN 9 | import collections 10 | from sklearn.metrics import average_precision_score 11 | from sklearn import metrics as mr 12 | import gc 13 | import copy 14 | 15 | 16 | dataset = '' 17 | probe = '' 18 | os.environ['CUDA_VISIBLE_DEVICES'] = '0' 19 | os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" 20 | config = tf.ConfigProto() 21 | config.gpu_options.allow_growth = True 22 | 23 | nb_nodes = 20 24 | ft_size = 3 # originial node feature dimension (D) 25 | time_step = 6 # sequence length (f) 26 | 27 | # training params 28 | batch_size = 256 29 | nb_epochs = 100000 30 | patience = 100 # patience for early stopping 31 | 32 | 33 | tf.app.flags.DEFINE_string('dataset', 'KS20', "Dataset: IAS, KS20, BIWI, CASIA-B or KGBD") 34 | tf.app.flags.DEFINE_string('length', '6', "4, 6, 8 or 10") # sequence length (f) 35 | tf.app.flags.DEFINE_string('lr', '0.00035', "learning rate") 36 | tf.app.flags.DEFINE_string('probe', 'probe', 37 | "for testing probe") # "probe" (for KGBD/KS20), "A", "B" (for IAS), "Walking", "Still" (for BIWI) 38 | tf.app.flags.DEFINE_string('gpu', '0', "GPU number") 39 | tf.app.flags.DEFINE_string('probe_type', '', "probe.gallery") # probe and gallery setting for CASIA-B 40 | tf.app.flags.DEFINE_string('patience', '150', "epochs for early stopping") 41 | tf.app.flags.DEFINE_string('mode', 'Train', "Training (Train) or Evaluation (Eval)") 42 | tf.app.flags.DEFINE_string('save_flag', '0', 43 | "") # save model metrics (top-1, top-5. top-10, mAP, CSP loss, mACT, mRCL) 44 | tf.app.flags.DEFINE_string('save_model', '0', "") # save best model 45 | tf.app.flags.DEFINE_string('batch_size', '256', "") 46 | tf.app.flags.DEFINE_string('model_size', '0', "") # output model size and computational complexity 47 | 48 | tf.app.flags.DEFINE_string('H', '128', "") # embedding size for node representations 49 | tf.app.flags.DEFINE_string('n_heads', '8', "") # number of Full-Relation (FR) heads 50 | tf.app.flags.DEFINE_string('L_transformer', '2', "") # number of MGT layers 51 | tf.app.flags.DEFINE_string('fusion_lambda', '0.5', "") # fusion coefficient for fusing sub-tracklet-level and sub-skeleton-level CSP 52 | tf.app.flags.DEFINE_string('t_1', '0.1', "") # global temperatures t1 53 | tf.app.flags.DEFINE_string('t_2', '10', "") # global temperatures t2 54 | tf.app.flags.DEFINE_string('pos_enc', '1', "") # positional encoding or not 55 | tf.app.flags.DEFINE_string('enc_k', '10', "") # first K eigenvectors for positional encoding 56 | tf.app.flags.DEFINE_string('rand_flip', '1', "") # random flipping strategy 57 | tf.app.flags.DEFINE_string('prob_t', '0.0', "") # probability for masking temporal combinatoiral (skeleton graphs) 58 | tf.app.flags.DEFINE_string('prob_s', '0.5', "") # probability for masking spatial combinatoiral features (body-joint nodes) 59 | tf.app.flags.DEFINE_string('motif_all', '1', "") 60 | 61 | FLAGS = tf.app.flags.FLAGS 62 | 63 | # check parameters 64 | if FLAGS.dataset not in ['IAS', 'KGBD', 'KS20', 'BIWI', 'CASIA_B']: 65 | raise Exception('Dataset must be IAS, KGBD, KS20, BIWI or CASIA B.') 66 | if FLAGS.dataset == 'CASIA_B': 67 | FLAGS.length = '40' 68 | if FLAGS.length not in ['40', '50', '60']: 69 | raise Exception('Length number must be 40, 50 or 60') 70 | else: 71 | if FLAGS.length not in ['4', '6', '8', '10']: 72 | raise Exception('Length number must be 4, 6, 8 or 10') 73 | if FLAGS.mode not in ['Train', 'Eval']: 74 | raise Exception('Mode must be Train or Eval.') 75 | 76 | os.environ["CUDA_VISIBLE_DEVICES"] = FLAGS.gpu 77 | dataset = FLAGS.dataset 78 | probe = FLAGS.probe 79 | # optimal paramters in MoCos 80 | if dataset == 'KGBD': 81 | FLAGS.lr = '0.00035' 82 | FLAGS.rand_flip = '0' 83 | FLAGS.prob_s = '0.5' 84 | FLAGS.prob_t = '0.25' 85 | FLAGS.fusion_lambda = '0.9' 86 | elif dataset == 'CASIA_B': 87 | FLAGS.lr = '0.00035' 88 | FLAGS.rand_flip = '0' 89 | if FLAGS.patience == '150': 90 | FLAGS.patience = '100' 91 | FLAGS.fusion_lambda = '1.0' 92 | else: 93 | FLAGS.lr = '0.00035' 94 | if dataset == 'IAS': 95 | FLAGS.rand_flip = '0' 96 | if probe == 'A': 97 | FLAGS.prob_s = '0.5' 98 | FLAGS.prob_t = '0.1' 99 | FLAGS.fusion_lambda = '0.75' 100 | elif probe == 'B': 101 | FLAGS.prob_s = '0.25' 102 | FLAGS.prob_t = '0.25' 103 | FLAGS.fusion_lambda = '0.75' 104 | elif dataset == 'BIWI': 105 | if probe == 'Walking': 106 | FLAGS.prob_s = '0.25' 107 | FLAGS.prob_t = '0.25' 108 | FLAGS.fusion_lambda = '0.9' 109 | elif probe == 'Still': 110 | FLAGS.prob_s = '0.25' 111 | FLAGS.prob_t = '0.25' 112 | FLAGS.fusion_lambda = '0.25' 113 | elif dataset == 'KS20': 114 | FLAGS.prob_s = '0.25' 115 | FLAGS.prob_t = '0.25' 116 | FLAGS.fusion_lambda = '0.9' 117 | 118 | time_step = int(FLAGS.length) 119 | probe = FLAGS.probe 120 | patience = int(FLAGS.patience) 121 | batch_size = int(FLAGS.batch_size) 122 | 123 | # not used 124 | global_att = False 125 | nhood = 1 126 | residual = False 127 | nonlinearity = tf.nn.elu 128 | 129 | pre_dir = 'ReID_Models/' 130 | # Customize the [directory] to save models with different hyper-parameters 131 | change = '' 132 | 133 | if FLAGS.probe_type != '': 134 | change += '_CME' 135 | 136 | 137 | try: 138 | os.mkdir(pre_dir) 139 | except: 140 | pass 141 | 142 | if dataset == 'KS20': 143 | nb_nodes = 25 144 | 145 | if dataset == 'CASIA_B': 146 | nb_nodes = 14 147 | 148 | if FLAGS.dataset == 'CASIA_B': 149 | FLAGS.length = '40' 150 | 151 | 152 | print('----- Model hyperparams -----') 153 | print('f (sequence length): ' + str(time_step)) 154 | print('H (embedding size): ' + FLAGS.H) 155 | print('MGT Layers: ' + FLAGS.L_transformer) 156 | print('heads: ' + FLAGS.n_heads) 157 | print('MoCos fusion lambda: ' + FLAGS.fusion_lambda) 158 | print('t1: ' + FLAGS.t_1) 159 | print('t2: ' + FLAGS.t_2) 160 | 161 | print('batch_size: ' + str(batch_size)) 162 | print('lr: ' + str(FLAGS.lr)) 163 | 164 | print('patience: ' + FLAGS.patience) 165 | print('Mode: ' + FLAGS.mode) 166 | 167 | print('p_s: ' + FLAGS.prob_s) 168 | print('p_t: ' + FLAGS.prob_t) 169 | print('lambda: ' + FLAGS.fusion_lambda) 170 | 171 | if dataset == 'CASIA_B': 172 | # 1-order 173 | adj_1 = np.zeros([14, 14]) 174 | adj_1.astype(np.int8) 175 | adj_1[0, 1] = adj_1[1, 0] = 1 176 | adj_1[1, 2] = adj_1[2, 1] = 1 177 | adj_1[2, 3] = adj_1[3, 2] = 1 178 | adj_1[3, 4] = adj_1[4, 3] = 1 179 | adj_1[1, 5] = adj_1[5, 1] = 1 180 | adj_1[5, 6] = adj_1[6, 5] = 1 181 | adj_1[6, 7] = adj_1[7, 6] = 1 182 | adj_1[1, 8] = adj_1[8, 1] = 1 183 | adj_1[8, 9] = adj_1[9, 8] = 1 184 | adj_1[9, 10] = adj_1[10, 9] = 1 185 | adj_1[1, 11] = adj_1[11, 1] = 1 186 | adj_1[11, 12] = adj_1[12, 11] = 1 187 | adj_1[12, 13] = adj_1[13, 12] = 1 188 | 189 | # 2-order 190 | adj_2 = copy.deepcopy(adj_1) 191 | adj_2[0, [2, 5, 8, 11]] = 1 192 | adj_2[1, [3, 6, 9, 12]] = 1 193 | adj_2[2, [0, 5, 4, 8, 11]] = 1 194 | adj_2[3, 1] = 1 195 | adj_2[4, 2] = 1 196 | adj_2[5, [0, 2, 8, 11, 7]] = 1 197 | adj_2[6, 1] = 1 198 | adj_2[7, 5] = 1 199 | adj_2[8, [2, 0, 5, 11, 10]] = 1 200 | adj_2[9, 1] = 1 201 | adj_2[10, 8] = 1 202 | adj_2[11, [0, 2, 5, 8, 13]] = 1 203 | adj_2[12, 1] = 1 204 | adj_2[13, 11] = 1 205 | 206 | # 3-order 207 | adj_3 = copy.deepcopy(adj_2) 208 | adj_3[0, [3, 6, 9, 12]] = 1 209 | adj_3[1, [4, 7, 10, 13]] = 1 210 | adj_3[2, [6, 9, 12]] = 1 211 | adj_3[3, [0, 5, 8, 11]] = 1 212 | adj_3[4, 1] = 1 213 | adj_3[5, [3, 9, 12]] = 1 214 | adj_3[6, [0, 2, 8, 11]] = 1 215 | adj_3[7, [1]] = 1 216 | adj_3[8, [3, 12, 6]] = 1 217 | adj_3[9, [0, 2, 11, 5]] = 1 218 | adj_3[10, [1]] = 1 219 | adj_3[11, [3, 9, 6]] = 1 220 | adj_3[12, [0, 2, 5, 8]] = 1 221 | adj_3[13, [1]] = 1 222 | 223 | # hand + arm 224 | adj_4 = np.zeros([14, 14]) 225 | adj_4[2, [3, 4]] = 1 226 | adj_4[3, [2, 4]] = 1 227 | adj_4[4, [2, 3]] = 1 228 | for i in [2, 3, 4]: 229 | for j in [5, 6, 7, 8, 9, 10, 11, 12, 13]: 230 | adj_4[i, j] = 1 231 | adj_4[5, [6, 7]] = 1 232 | adj_4[6, [5, 7]] = 1 233 | adj_4[7, [5, 6]] = 1 234 | for i in [5, 6, 7]: 235 | for j in [2, 3, 4, 8, 9, 10, 11, 12, 13]: 236 | adj_4[i, j] = 1 237 | 238 | # foot + leg 239 | adj_5 = np.zeros([14, 14]) 240 | adj_5[8, [9, 10]] = 1 241 | adj_5[9, [8, 10]] = 1 242 | adj_5[10, [8, 9]] = 1 243 | for i in [8, 9, 10]: 244 | for j in [2, 3, 4, 5, 6, 7, 11, 12, 13]: 245 | adj_5[i, j] = 1 246 | adj_5[11, [12, 13]] = 1 247 | adj_5[12, [11, 13]] = 1 248 | adj_5[13, [11, 12]] = 1 249 | for i in [11, 12, 13]: 250 | for j in [2, 3, 4, 5, 6, 7, 8, 9, 10]: 251 | adj_5[i, j] = 1 252 | 253 | elif dataset != 'KS20': 254 | # 1-order 255 | adj_1 = [[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0,], 256 | [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 257 | [0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 258 | [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 259 | [0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 260 | [0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 261 | [0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 262 | [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 263 | [0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 264 | [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 265 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 266 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 267 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,], 268 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0,], 269 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0,], 270 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,], 271 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,], 272 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0,], 273 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1,], 274 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,]] 275 | 276 | # 2-order 277 | adj_2 = [[0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0,], 278 | [1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0,], 279 | [1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 280 | [0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 281 | [0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 282 | [0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 283 | [0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 284 | [0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 285 | [0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 286 | [0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 287 | [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 288 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 289 | [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0,], 290 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0,], 291 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0,], 292 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,], 293 | [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0,], 294 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1,], 295 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1,], 296 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0,]] 297 | 298 | # 3-order 299 | adj_3 = [[0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0,], 300 | [1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0,], 301 | [1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0,], 302 | [1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 303 | [1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 304 | [0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 305 | [0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 306 | [0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 307 | [1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 308 | [0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 309 | [0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 310 | [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 311 | [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0,], 312 | [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0,], 313 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0,], 314 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0,], 315 | [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1,], 316 | [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1,], 317 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1,], 318 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,]] 319 | 320 | # hand + arm 321 | adj_4 = np.zeros([20, 20]) 322 | adj_4[8, [9, 10, 11]] = 1 323 | adj_4[9, [8, 10, 11]] = 1 324 | adj_4[10, [9, 8, 11]] = 1 325 | adj_4[11, [9, 10, 8]] = 1 326 | for i in [8, 9, 10, 11]: 327 | for j in [4, 5, 6, 7, 16, 17, 18, 19, 12, 13, 14, 15]: 328 | adj_4[i, j] = 1 329 | adj_4[4, [5, 6, 7]] = 1 330 | adj_4[5, [4, 6, 7]] = 1 331 | adj_4[6, [5, 4, 7]] = 1 332 | adj_4[7, [5, 6, 4]] = 1 333 | # # 334 | for i in [4, 5, 6, 7]: 335 | for j in [8, 9, 10, 11, 16, 17, 18, 19, 12, 13, 14, 15]: 336 | adj_4[i, j] = 1 337 | 338 | # foot + leg 339 | adj_5 = np.zeros([20, 20]) 340 | adj_5[16, [17, 18, 19]] = 1 341 | adj_5[17, [16, 18, 19]] = 1 342 | adj_5[18, [16, 17, 19]] = 1 343 | adj_5[19, [16, 18, 17]] = 1 344 | for i in [16, 17, 18, 19]: 345 | for j in [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]: 346 | adj_5[i, j] = 1 347 | adj_5[12, [13, 14, 15]] = 1 348 | adj_5[13, [12, 14, 15]] = 1 349 | adj_5[14, [12, 13, 15]] = 1 350 | adj_5[15, [12, 13, 14]] = 1 351 | for i in [12, 13, 14, 15]: 352 | for j in [4, 5, 6, 7, 8, 9, 10, 11, 16, 17, 18, 19]: 353 | adj_5[i, j] = 1 354 | else: 355 | # 1-order 356 | adj_1 = [[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,], 357 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,], 358 | [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,], 359 | [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 360 | [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,], 361 | [0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 362 | [0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,], 363 | [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,], 364 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,], 365 | [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 366 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,], 367 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,], 368 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 369 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 370 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 371 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 372 | [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,], 373 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0,], 374 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0,], 375 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,], 376 | [0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], # 20 377 | [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 378 | [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 379 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,], 380 | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,],] 381 | 382 | # 2-order 383 | adj_2 = copy.deepcopy(adj_1) 384 | adj_2 = np.array(adj_2) 385 | adj_2[0, [1, 20, 12, 13, 16, 17]] = 1 386 | adj_2[1, [20, 2, 4, 8, 0, 12, 16]] = 1 387 | adj_2[2, [3, 20, 4, 8, 1]] = 1 388 | adj_2[3, [2, 20]] = 1 389 | adj_2[4, [5, 6, 20, 2, 8, 1]] = 1 390 | adj_2[5, [4, 20, 6, 7, 22]] = 1 391 | adj_2[6, [7, 21, 5, 4]] = 1 392 | adj_2[7, [21, 6, 22,5]] = 1 393 | adj_2[8, [20, 2, 4, 1, 9, 10]] = 1 394 | adj_2[9, [8, 20, 10, 11, 24]] = 1 395 | adj_2[10, [9, 8, 11, 23]] = 1 396 | adj_2[11, [10, 9, 24]] = 1 397 | adj_2[12, [0, 1, 16, 13, 14]] = 1 398 | adj_2[13, [14, 15, 12, 0]] = 1 399 | adj_2[14, [13, 12]] = 1 400 | adj_2[15, [14, 13]] = 1 401 | adj_2[16, [0, 1, 12, 17, 18]] = 1 402 | adj_2[17, [16, 0, 18, 19]] = 1 403 | adj_2[18, [17, 16]] = 1 404 | adj_2[19, [18, 17]] = 1 405 | adj_2[20, [4, 5, 2, 3, 8, 9, 1, 0]] = 1 406 | adj_2[21, [7, 6]] = 1 407 | adj_2[22, [6, 7, 5]] = 1 408 | adj_2[23, [11, 10]] = 1 409 | adj_2[24, [10, 9, 11]] = 1 410 | 411 | 412 | # 3-order 413 | adj_3 = copy.deepcopy(adj_1) 414 | adj_3 = np.array(adj_3) 415 | adj_3[0, [1, 20, 2, 4, 8, 12, 13, 14, 16, 17, 18]] = 1 416 | adj_3[1, [20, 2, 3, 4, 5, 8, 9, 0, 12, 13, 16, 17]] = 1 417 | adj_3[2, [3, 20, 1, 4, 5, 8, 9, 1, 0]] = 1 418 | adj_3[3, [2, 20, 4, 8, 1]] = 1 419 | adj_3[4, [5, 6, 7, 22, 20, 2, 3, 8, 9, 1, 0]] = 1 420 | adj_3[5, [4, 20, 2, 1, 8, 6, 7, 21, 22]] = 1 421 | adj_3[6, [7, 21, 5, 4, 20]] = 1 422 | adj_3[7, [21, 6, 22, 5, 4]] = 1 423 | adj_3[8, [20, 2, 3, 4, 5, 1, 0, 9, 10, 11, 24]] = 1 424 | adj_3[9, [8, 20, 2, 4, 1, 10, 11, 23, 24]] = 1 425 | adj_3[10, [9, 8, 20, 11, 23]] = 1 426 | adj_3[11, [10, 9, 8, 24]] = 1 427 | adj_3[12, [0, 1, 20, 16, 17, 13, 14, 15]] = 1 428 | adj_3[13, [14, 15, 12, 0, 1, 16]] = 1 429 | adj_3[14, [13, 12, 0]] = 1 430 | adj_3[15, [14, 13, 12]] = 1 431 | adj_3[16, [0, 1, 20, 12, 13, 17, 18, 19]] = 1 432 | adj_3[17, [16, 0, 1, 12, 18, 19]] = 1 433 | adj_3[18, [17, 16, 0]] = 1 434 | adj_3[19, [18, 17, 16]] = 1 435 | adj_3[20, [4, 5, 6, 2, 3, 8, 9, 10, 1, 0, 12, 16]] = 1 436 | adj_3[21, [7, 6, 5, 22]] = 1 437 | adj_3[22, [6, 7, 21, 5, 4]] = 1 438 | adj_3[23, [11, 10, 9, 24]] = 1 439 | adj_3[24, [10, 9, 8, 11, 23]] = 1 440 | 441 | # hand + arm 442 | adj_4 = np.zeros([25, 25]) 443 | adj_4[8, [9, 10, 11, 23, 24]] = 1 444 | adj_4[9, [8, 10, 11, 23, 24]] = 1 445 | adj_4[10, [8, 9, 11, 23, 24]] = 1 446 | adj_4[11, [8, 9, 10, 23, 24]] = 1 447 | adj_4[23, [8, 9, 10, 11, 24]] = 1 448 | adj_4[24, [8, 9, 10, 11, 23]] = 1 449 | for i in [8, 9, 10, 11, 23, 24]: 450 | for j in [4, 5, 6, 7, 21, 22, 16, 17, 18, 19, 12, 13, 14, 15]: 451 | adj_4[i, j] = 1 452 | adj_4[4, [5, 6, 7, 21, 22]] = 1 453 | adj_4[5, [4, 6, 7, 21, 22]] = 1 454 | adj_4[6, [4, 5, 7, 21, 22]] = 1 455 | adj_4[7, [4, 5, 6, 21, 22]] = 1 456 | adj_4[21, [4, 5, 6, 7, 22]] = 1 457 | adj_4[22, [4, 5, 6, 7, 21]] = 1 458 | for i in [4, 5, 6, 7, 21, 22]: 459 | for j in [8, 9, 10, 11, 23, 24, 16, 17, 18, 19, 12, 13, 14, 15]: 460 | adj_4[i, j] = 1 461 | 462 | # foot + leg 463 | adj_5 = np.zeros([25, 25]) 464 | adj_5[16, [17, 18, 19]] = 1 465 | adj_5[17, [16, 18, 19]] = 1 466 | adj_5[18, [16, 17, 19]] = 1 467 | adj_5[19, [16, 18, 17]] = 1 468 | for i in [16, 17, 18, 19]: 469 | for j in [4, 5, 6, 7, 21, 22, 8, 9, 10, 11, 23, 24, 12, 13, 14, 15]: 470 | adj_5[i, j] = 1 471 | adj_5[12, [13, 14, 15]] = 1 472 | adj_5[13, [12, 14, 15]] = 1 473 | adj_5[14, [12, 13, 15]] = 1 474 | adj_5[15, [12, 13, 14]] = 1 475 | for i in [12, 13, 14, 15]: 476 | for j in [4, 5, 6, 7, 21, 22, 8, 9, 10, 11, 23, 24, 16, 17, 18, 19]: 477 | adj_5[i, j] = 1 478 | 479 | if FLAGS.mode == 'Train': 480 | print('----- Dataset Information -----') 481 | print('Dataset: ' + dataset) 482 | if dataset == 'CASIA_B': 483 | print('Probe.Gallery: ', FLAGS.probe_type.split('.')[0], FLAGS.probe_type.split('.')[1]) 484 | else: 485 | print('Probe: ' + FLAGS.probe) 486 | 487 | if FLAGS.probe_type == '': 488 | X_train_J, _, _, _, _, y_train, X_test_J, _, _, _, _, y_test, \ 489 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 490 | process.gen_train_data(dataset=dataset, split=probe, time_step=time_step, 491 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, batch_size=batch_size,) 492 | del _ 493 | gc.collect() 494 | 495 | else: 496 | from utils import process_cme_SG as process 497 | 498 | X_train_J, _, _, _, _, y_train, X_test_J, _, _, _, _, y_test, \ 499 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 500 | process.gen_train_data(dataset=dataset, split=probe, time_step=time_step, 501 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, batch_size=batch_size, 502 | PG_type=FLAGS.probe_type.split('.')[0]) 503 | print('## [Probe].[Gallery]', FLAGS.probe_type) 504 | del _ 505 | gc.collect() 506 | 507 | all_ftr_size = int(FLAGS.H) 508 | loaded_graph = tf.Graph() 509 | joint_num = X_train_J.shape[2] 510 | cluster_epochs = 15000 511 | display = 20 512 | k = int(FLAGS.enc_k) 513 | 514 | change += '_MoCos_Formal_f_' + FLAGS.length + '_prob_s_' + FLAGS.prob_s + '_prob_t_' + FLAGS.prob_t + '_lambda_' + FLAGS.fusion_lambda 515 | 516 | if FLAGS.mode == 'Train': 517 | loaded_graph = tf.Graph() 518 | with loaded_graph.as_default(): 519 | with tf.name_scope('Input'): 520 | J_in = tf.placeholder(dtype=tf.float32, shape=(batch_size * time_step, joint_num, ft_size)) 521 | L_eig = tf.placeholder(dtype=tf.float32, shape=(joint_num, k)) 522 | train_flag = tf.placeholder(dtype=tf.bool, shape=()) 523 | pseudo_lab_1 = tf.placeholder(dtype=tf.int32, shape=(batch_size,)) 524 | seq_cluster_ftr_1 = tf.placeholder(dtype=tf.float32, shape=(None, all_ftr_size)) 525 | pseudo_lab_2 = tf.placeholder(dtype=tf.int32, shape=(batch_size,)) 526 | seq_cluster_ftr_2 = tf.placeholder(dtype=tf.float32, shape=(None, all_ftr_size)) 527 | seq_mask = tf.placeholder(dtype=tf.float32, shape=(time_step,)) 528 | seq_mask_2 = tf.placeholder(dtype=tf.float32, shape=(time_step,)) 529 | node_mask = tf.placeholder(dtype=tf.float32, shape=(joint_num,)) 530 | 531 | gt_class_ftr = tf.placeholder(dtype=tf.float32, shape=(nb_classes, all_ftr_size)) 532 | gt_lab = tf.placeholder(dtype=tf.int32, shape=(batch_size,)) 533 | 534 | with tf.name_scope("MoCos"), tf.variable_scope("MoCos", reuse=tf.AUTO_REUSE): 535 | inputs = tf.reshape(J_in, [time_step * batch_size * joint_num, 3]) 536 | outputs = inputs 537 | outputs = tf.layers.dense(outputs, int(FLAGS.H), activation=tf.nn.relu) 538 | s_rep = outputs 539 | s_rep = tf.layers.dense(s_rep, int(FLAGS.H), activation=None) 540 | s_rep = tf.reshape(s_rep, [-1]) 541 | optimizer = tf.train.AdamOptimizer(learning_rate=float(FLAGS.lr)) 542 | seq_ftr = tf.reshape(s_rep, [batch_size, time_step, joint_num, -1]) 543 | pos_enc = tf.tile(tf.reshape(L_eig, [1, 1, joint_num, k]), [batch_size, time_step, 1, 1]) 544 | pos_enc = tf.layers.dense(pos_enc, int(FLAGS.H), activation=None) 545 | ori_ftr = seq_ftr 546 | if FLAGS.pos_enc == '1': 547 | seq_ftr = seq_ftr + pos_enc 548 | H = int(FLAGS.H) 549 | W_head = lambda: tf.Variable(tf.random_normal([H, H // int(FLAGS.n_heads)])) 550 | 551 | h = seq_ftr 552 | for l in range(int(FLAGS.L_transformer)): 553 | for i in range(int(FLAGS.n_heads)): 554 | W_Q = tf.Variable(initial_value=W_head) 555 | W_K = tf.Variable(initial_value=W_head) 556 | W_V = tf.Variable(initial_value=W_head) 557 | Q_h = tf.matmul(h, W_Q) 558 | K_h = tf.matmul(h, W_K) 559 | K_h = tf.transpose(K_h, perm=[0, 1, 3, 2]) 560 | motif_indice = [] 561 | if i == 1: 562 | att_scores = tf.nn.softmax( 563 | tf.clip_by_value(tf.matmul(Q_h, K_h) * adj_1 / np.sqrt(H // int(FLAGS.n_heads)), -5, 5)) 564 | if i == 2: 565 | att_scores = tf.nn.softmax( 566 | tf.clip_by_value(tf.matmul(Q_h, K_h) * adj_2 / np.sqrt(H // int(FLAGS.n_heads)), -5, 5)) 567 | if i == 3: 568 | att_scores = tf.nn.softmax( 569 | tf.clip_by_value(tf.matmul(Q_h, K_h) * adj_3 / np.sqrt(H // int(FLAGS.n_heads)), -5, 5)) 570 | if i == 4: 571 | att_scores = tf.nn.softmax( 572 | tf.clip_by_value(tf.matmul(Q_h, K_h) * adj_4 / np.sqrt(H // int(FLAGS.n_heads)), -5, 5)) 573 | if i == 5: 574 | att_scores = tf.nn.softmax( 575 | tf.clip_by_value(tf.matmul(Q_h, K_h) * adj_5 / np.sqrt(H // int(FLAGS.n_heads)), -5, 5)) 576 | if i not in [1, 2, 3, 4, 5]: 577 | # numerical stability to clamp [-5, 5] 578 | att_scores = tf.nn.softmax( 579 | tf.clip_by_value(tf.matmul(Q_h, K_h) / np.sqrt(H // int(FLAGS.n_heads)), -5, 5)) 580 | V_h = tf.matmul(h, W_V) 581 | att_scores = tf.tile(tf.reshape(att_scores, [batch_size, time_step, joint_num, joint_num, 1]), 582 | [1, 1, 1, 1, H // int(FLAGS.n_heads)]) 583 | aggr_features = tf.reduce_sum( 584 | att_scores * tf.reshape(V_h, [batch_size, time_step, 1, joint_num, H // int(FLAGS.n_heads)]), 585 | axis=-2) 586 | if i == 0: 587 | concat_features = aggr_features 588 | else: 589 | concat_features = tf.concat([concat_features, aggr_features], axis=-1) 590 | h = concat_features 591 | 592 | h = tf.layers.dropout(h, rate=0.5, training=train_flag) 593 | h = tf.layers.dense(h, H, activation=None) 594 | h_res1 = seq_ftr 595 | h = h_res1 + h 596 | h = tf.layers.batch_normalization(h, training=train_flag) 597 | h_res2 = h 598 | h = tf.layers.dense(h, H * 2, activation=tf.nn.relu) 599 | h = tf.layers.dropout(h, rate=0.5, training=train_flag) 600 | h = tf.layers.dense(h, H, activation=None) 601 | h = h_res2 + h 602 | h = tf.layers.batch_normalization(h, training=train_flag) 603 | spatial_h = h 604 | 605 | # sub-skeleton-level CSP in MoCos 606 | def CSP_ske(t, labels, all_ftr, cluster_ftr): 607 | W_head = lambda: tf.Variable(tf.random_normal([H, H])) 608 | head_num = 1 609 | for i in range(head_num): 610 | f_1 = tf.Variable(initial_value=W_head) 611 | f_2 = tf.Variable(initial_value=W_head) 612 | all_ftr_trans = tf.matmul(all_ftr, f_1) 613 | cluster_ftr_trans = tf.matmul(cluster_ftr, f_2) 614 | logits = tf.matmul(all_ftr_trans, tf.transpose(cluster_ftr_trans)) / t 615 | label_frames = tf.reshape(tf.tile(tf.reshape(labels, [-1, 1]), [1, time_step]), [-1]) 616 | label_frames = tf.reshape(label_frames, [batch_size, time_step]) 617 | loss = tf.reduce_mean(tf.reduce_sum(tf.nn.sparse_softmax_cross_entropy_with_logits(labels=label_frames, 618 | logits=logits), axis=-1)) 619 | return loss 620 | 621 | ori_spatial_h = spatial_h 622 | spatial_h = tf.reduce_mean(spatial_h, axis=-2) 623 | h2 = spatial_h 624 | C_seq = seq_ftr = h2 625 | 626 | seq_ftr = tf.reduce_mean(seq_ftr, axis=1) 627 | seq_ftr = tf.reshape(seq_ftr, [batch_size, -1]) 628 | 629 | mask_seq_1 = tf.boolean_mask(C_seq, tf.reshape(seq_mask, [-1]), axis=1) 630 | mask_seq_1 = tf.reduce_mean(mask_seq_1, axis=1) 631 | 632 | 633 | # sub-tracklet-level CSP in MoCos 634 | def CSP_str(t, pseudo_lab, all_ftr, cluster_ftr): 635 | all_ftr = tf.nn.l2_normalize(all_ftr, axis=-1) 636 | cluster_ftr = tf.nn.l2_normalize(cluster_ftr, axis=-1) 637 | output = tf.matmul(all_ftr, tf.transpose(cluster_ftr)) 638 | output /= t 639 | loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(labels=pseudo_lab, logits=output)) 640 | return loss 641 | 642 | mask_G = tf.boolean_mask(ori_spatial_h, tf.reshape(node_mask, [-1]), axis=-2) 643 | G_h = tf.reduce_mean(mask_G, axis=-2) 644 | # temporal masking in CSP 645 | G_mask_seq = tf.boolean_mask(G_h, tf.reshape(seq_mask, [-1]), axis=1) 646 | G_h_seq = tf.reduce_mean(G_mask_seq, axis=-2) 647 | 648 | STr_CSP_loss = CSP_str(float(FLAGS.t_1), gt_lab, G_h_seq, gt_class_ftr) 649 | SSk_CSP_loss = CSP_ske(float(FLAGS.t_2), gt_lab, G_h, gt_class_ftr) 650 | CSP_loss = (1 - float(FLAGS.fusion_lambda)) * SSk_CSP_loss + float(FLAGS.fusion_lambda) * STr_CSP_loss 651 | train_op = optimizer.minimize(CSP_loss) 652 | 653 | saver = tf.train.Saver() 654 | init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer()) 655 | 656 | with tf.Session(config=config) as sess: 657 | sess.run(init_op) 658 | if FLAGS.model_size == '1': 659 | # compute model size (M) and computational complexity (GFLOPs) 660 | def stats_graph(graph): 661 | flops = tf.profiler.profile(graph, options=tf.profiler.ProfileOptionBuilder.float_operation()) 662 | params = tf.profiler.profile(graph, 663 | options=tf.profiler.ProfileOptionBuilder.trainable_variables_parameter()) 664 | print('FLOPs: {} GFLOPS; Trainable params: {} M'.format(flops.total_float_ops / 1e9, 665 | params.total_parameters / 1e6)) 666 | stats_graph(loaded_graph) 667 | print('prob_s:', FLAGS.prob_s) 668 | print('prob_t:', FLAGS.prob_t) 669 | print('fusion_lambda:', FLAGS.fusion_lambda) 670 | exit() 671 | 672 | mask_rand_save = [] 673 | mask_rand_save_2 = [] 674 | node_mask_save = [] 675 | cur_epoch = 0 676 | 677 | def train_loader(X_train_J, y_train): 678 | global mask_rand_save, node_mask_save, mask_rand_save_2 679 | 680 | # temporal masking 681 | mask_rand_save = [] 682 | mask_rand_save_2 = [] 683 | # spatial masking 684 | node_mask_save = [] 685 | tr_step = 0 686 | tr_size = X_train_J.shape[0] 687 | train_labels_all = [] 688 | train_features_all = [] 689 | while tr_step * batch_size < tr_size: 690 | if (tr_step + 1) * batch_size > tr_size: 691 | break 692 | X_input_J = X_train_J[tr_step * batch_size:(tr_step + 1) * batch_size] 693 | X_input_J = X_input_J.reshape([-1, joint_num, 3]) 694 | labels = y_train[tr_step * batch_size:(tr_step + 1) * batch_size] 695 | prob = np.random.uniform(0, 1, [time_step, ]) 696 | mask_rand_1 = prob >= float(FLAGS.prob_t) 697 | # ensure not all zeros 698 | while np.mean(mask_rand_1) == 0: 699 | prob = np.random.uniform(0, 1, [time_step, ]) 700 | mask_rand_1 = prob >= float(FLAGS.prob_t) 701 | mask_rand_1 = np.reshape(mask_rand_1, [time_step]) 702 | mask_rand_save.append(mask_rand_1.tolist()) 703 | prob = np.random.uniform(0, 1, [time_step, ]) 704 | mask_rand_2 = prob >= float(FLAGS.prob_t) 705 | while np.mean(mask_rand_2) == 0: 706 | prob = np.random.uniform(0, 1, [time_step, ]) 707 | mask_rand_2 = prob >= float(FLAGS.prob_t) 708 | mask_rand_2 = np.reshape(mask_rand_2, [time_step]) 709 | mask_rand_save_2.append(mask_rand_2.tolist()) 710 | prob = np.random.uniform(0, 1, [joint_num, ]) 711 | node_mask_rand = prob >= float(FLAGS.prob_s) 712 | # ensure not all zeros 713 | while np.mean(node_mask_rand) == 0: 714 | prob = np.random.uniform(0, 1, [joint_num, ]) 715 | node_mask_rand = prob >= float(FLAGS.prob_s) 716 | node_mask_save.append(node_mask_rand.tolist()) 717 | 718 | [all_features] = sess.run([seq_ftr], 719 | feed_dict={ 720 | J_in: X_input_J, 721 | seq_mask: mask_rand_1, 722 | L_eig: pos_enc_ori, 723 | train_flag: False, 724 | node_mask: node_mask_rand, 725 | }) 726 | train_features_all.extend(all_features.tolist()) 727 | train_labels_all.extend(labels.tolist()) 728 | tr_step += 1 729 | 730 | train_features_all = np.array(train_features_all).astype(np.float32) 731 | train_features_all = torch.from_numpy(train_features_all) 732 | return train_features_all, train_labels_all 733 | 734 | def gal_loader(X_train_J, y_train): 735 | tr_step = 0 736 | tr_size = X_train_J.shape[0] 737 | gal_logits_all = [] 738 | gal_labels_all = [] 739 | gal_features_all = [] 740 | embed_1_all = [] 741 | embed_2_all = [] 742 | 743 | while tr_step * batch_size < tr_size: 744 | if (tr_step + 1) * batch_size > tr_size: 745 | break 746 | X_input_J = X_train_J[tr_step * batch_size:(tr_step + 1) * batch_size] 747 | X_input_J = X_input_J.reshape([-1, joint_num, 3]) 748 | labels = y_train[tr_step * batch_size:(tr_step + 1) * batch_size] 749 | [Seq_features] = sess.run([seq_ftr], 750 | feed_dict={ 751 | J_in: X_input_J, 752 | L_eig: pos_enc_ori, 753 | train_flag: False 754 | }) 755 | gal_features_all.extend(Seq_features.tolist()) 756 | gal_labels_all.extend(labels.tolist()) 757 | tr_step += 1 758 | return gal_features_all, gal_labels_all, embed_1_all, embed_2_all 759 | 760 | def evaluation(): 761 | vl_step = 0 762 | vl_size = X_test_J.shape[0] 763 | pro_labels_all = [] 764 | pro_features_all = [] 765 | while vl_step * batch_size < vl_size: 766 | if (vl_step + 1) * batch_size > vl_size: 767 | break 768 | X_input_J = X_test_J[vl_step * batch_size:(vl_step + 1) * batch_size] 769 | X_input_J = X_input_J.reshape([-1, joint_num, 3]) 770 | labels = y_test[vl_step * batch_size:(vl_step + 1) * batch_size] 771 | [Seq_features] = sess.run([seq_ftr], 772 | feed_dict={ 773 | J_in: X_input_J, 774 | L_eig: pos_enc_ori, 775 | train_flag: False 776 | }) 777 | pro_labels_all.extend(labels.tolist()) 778 | pro_features_all.extend(Seq_features.tolist()) 779 | vl_step += 1 780 | X = np.array(gal_features_all) 781 | y = np.array(gal_labels_all) 782 | t_X = np.array(pro_features_all) 783 | t_y = np.array(pro_labels_all) 784 | t_y = np.argmax(t_y, axis=-1) 785 | y = np.argmax(y, axis=-1) 786 | 787 | def mean_ap(distmat, query_ids=None, gallery_ids=None, 788 | query_cams=None, gallery_cams=None): 789 | # distmat = to_numpy(distmat) 790 | m, n = distmat.shape 791 | # Fill up default values 792 | if query_ids is None: 793 | query_ids = np.arange(m) 794 | if gallery_ids is None: 795 | gallery_ids = np.arange(n) 796 | if query_cams is None: 797 | query_cams = np.zeros(m).astype(np.int32) 798 | if gallery_cams is None: 799 | gallery_cams = np.ones(n).astype(np.int32) 800 | # Ensure numpy array 801 | query_ids = np.asarray(query_ids) 802 | gallery_ids = np.asarray(gallery_ids) 803 | query_cams = np.asarray(query_cams) 804 | gallery_cams = np.asarray(gallery_cams) 805 | # Sort and find correct matches 806 | indices = np.argsort(distmat, axis=1) 807 | matches = (gallery_ids[indices] == query_ids[:, np.newaxis]) 808 | # Compute AP for each query 809 | aps = [] 810 | if (FLAGS.probe_type == 'nm.nm' or FLAGS.probe_type == 'cl.cl' or FLAGS.probe_type == 'bg.bg'): 811 | for i in range(1, m): 812 | valid = ((gallery_ids[indices[i]] != query_ids[i]) | 813 | (gallery_cams[indices[i]] != query_cams[i])) 814 | 815 | y_true = matches[i, valid] 816 | y_score = -distmat[i][indices[i]][valid] 817 | # discard nan 818 | y_score[np.isnan(y_score)] = 0 819 | if not np.any(y_true): continue 820 | aps.append(average_precision_score(y_true, y_score)) 821 | else: 822 | for i in range(m): 823 | valid = ((gallery_ids[indices[i]] != query_ids[i]) | 824 | (gallery_cams[indices[i]] != query_cams[i])) 825 | y_true = matches[i, valid] 826 | y_score = -distmat[i][indices[i]][valid] 827 | if not np.any(y_true): continue 828 | aps.append(average_precision_score(y_true, y_score)) 829 | if len(aps) == 0: 830 | raise RuntimeError("No valid query") 831 | return np.mean(aps) 832 | 833 | def metrics(X, y, t_X, t_y): 834 | # compute Euclidean distance 835 | if dataset != 'CASIA_B': 836 | a, b = torch.from_numpy(t_X), torch.from_numpy(X) 837 | m, n = a.size(0), b.size(0) 838 | a = a.view(m, -1) 839 | b = b.view(n, -1) 840 | dist_m = torch.pow(a, 2).sum(dim=1, keepdim=True).expand(m, n) + \ 841 | torch.pow(b, 2).sum(dim=1, keepdim=True).expand(n, m).t() 842 | dist_m.addmm_(1, -2, a, b.t()) 843 | # 1e-12 844 | dist_m = (dist_m.clamp(min=1e-12)).sqrt() 845 | mAP = mean_ap(distmat=dist_m.numpy(), query_ids=t_y, gallery_ids=y) 846 | _, dist_sort = dist_m.sort(1) 847 | dist_sort = dist_sort.numpy() 848 | else: 849 | X = np.array(X) 850 | t_X = np.array(t_X) 851 | dist_m = [(np.linalg.norm(X - i, axis=1)).tolist() for i in t_X] 852 | dist_m = np.array(dist_m) 853 | mAP = mean_ap(distmat=dist_m, query_ids=t_y, gallery_ids=y) 854 | dist_sort = [np.argsort(np.linalg.norm(X - i, axis=1)).tolist() for i in t_X] 855 | dist_sort = np.array(dist_sort) 856 | 857 | top_1 = top_5 = top_10 = 0 858 | probe_num = dist_sort.shape[0] 859 | if (FLAGS.probe_type == 'nm.nm' or 860 | FLAGS.probe_type == 'cl.cl' or FLAGS.probe_type == 'bg.bg'): 861 | for i in range(probe_num): 862 | if t_y[i] in y[dist_sort[i, 1:2]]: 863 | top_1 += 1 864 | if t_y[i] in y[dist_sort[i, 1:6]]: 865 | top_5 += 1 866 | if t_y[i] in y[dist_sort[i, 1:11]]: 867 | top_10 += 1 868 | else: 869 | for i in range(probe_num): 870 | # print(dist_sort[i, :10]) 871 | if t_y[i] in y[dist_sort[i, :1]]: 872 | top_1 += 1 873 | if t_y[i] in y[dist_sort[i, :5]]: 874 | top_5 += 1 875 | if t_y[i] in y[dist_sort[i, :10]]: 876 | top_10 += 1 877 | return mAP, top_1 / probe_num, top_5 / probe_num, top_10 / probe_num 878 | 879 | mAP, top_1, top_5, top_10 = metrics(X, y, t_X, t_y) 880 | del X, y, t_X, t_y, pro_labels_all, pro_features_all 881 | gc.collect() 882 | return mAP, top_1, top_5, top_10 883 | 884 | max_acc_1 = 0 885 | max_acc_2 = 0 886 | top_5_max = 0 887 | top_10_max = 0 888 | cur_patience = 0 889 | top_1s = [] 890 | top_5s = [] 891 | top_10s = [] 892 | mAPs = [] 893 | STr_CSP_losses = [] 894 | SSk_CSP_losses = [] 895 | CSP_losses = [] 896 | 897 | mACT = [] 898 | mRCL = [] 899 | 900 | if dataset == 'KGBD' or dataset == 'KS20': 901 | # if FLAGS.level == 'J': 902 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 903 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 904 | process.gen_train_data(dataset=dataset, split='gallery', time_step=time_step, 905 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 906 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 907 | 908 | elif dataset == 'BIWI': 909 | if probe == 'Walking': 910 | # if FLAGS.level == 'J': 911 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 912 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 913 | process.gen_train_data(dataset=dataset, split='Still', time_step=time_step, 914 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 915 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 916 | else: 917 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 918 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 919 | process.gen_train_data(dataset=dataset, split='Walking', time_step=time_step, 920 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 921 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 922 | elif dataset == 'IAS': 923 | if probe == 'A': 924 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 925 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 926 | process.gen_train_data(dataset=dataset, split='B', time_step=time_step, 927 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 928 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 929 | else: 930 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 931 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 932 | process.gen_train_data(dataset=dataset, split='A', time_step=time_step, 933 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 934 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 935 | 936 | elif dataset == 'CASIA_B': 937 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 938 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 939 | process.gen_train_data(dataset=dataset, split=probe, time_step=time_step, 940 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 941 | batch_size=batch_size, PG_type=FLAGS.probe_type.split('.')[1]) 942 | del _ 943 | gc.collect() 944 | 945 | all_strs = '' 946 | simp_strs = '' 947 | for epoch in range(cluster_epochs): 948 | cur_epoch = epoch 949 | train_features_all, train_labels_all = train_loader(X_train_J, y_train) 950 | gal_features_all, gal_labels_all, gal_embed_1_all, gal_embed_2_all = gal_loader(X_gal_J, y_gal) 951 | ori_train_labels = copy.deepcopy(train_labels_all) 952 | if FLAGS.save_flag == '1': 953 | # Compute mean intra-class tightness (mACT) and mean inter-class tightness (mRCL) 954 | # see "Skeleton Prototype Contrastive Learning with Multi-level Graph Relation Modeling 955 | # for Unsupervised Person Re-Identification" (Rao et al.) for details of above metrics 956 | train_features_all = train_features_all.numpy() 957 | labels = np.argmax(np.array(train_labels_all), axis=-1) 958 | label_t = set(labels.tolist()) 959 | y = np.array(labels) 960 | X = np.array(train_features_all) 961 | sorted_indices = np.argsort(y, axis=0) 962 | sort_y = y[sorted_indices] 963 | sort_X = X[sorted_indices] 964 | all_class_ftrs = {} 965 | class_start_indices = {} 966 | class_end_indices = {} 967 | pre_label = sort_y[0] 968 | class_start_indices[pre_label] = 0 969 | for i, label in enumerate(sort_y): 970 | if sort_y[i] not in all_class_ftrs.keys(): 971 | all_class_ftrs[sort_y[i]] = [sort_X[i]] 972 | else: 973 | all_class_ftrs[sort_y[i]].append(sort_X[i]) 974 | if label != pre_label: 975 | class_start_indices[label] = class_end_indices[pre_label] = i 976 | pre_label = label 977 | if i == len(sort_y) - 1: 978 | class_end_indices[label] = i 979 | center_ftrs = [] 980 | for label, class_ftrs in all_class_ftrs.items(): 981 | class_ftrs = np.array(class_ftrs) 982 | center_ftr = np.mean(class_ftrs, axis=0) 983 | center_ftrs.append(center_ftr) 984 | center_ftrs = np.array(center_ftrs) 985 | 986 | a, b = torch.from_numpy(sort_X), torch.from_numpy(center_ftrs) 987 | 988 | a_norm = a / a.norm(dim=1)[:, None] 989 | b_norm = b / b.norm(dim=1)[:, None] 990 | dist_m = 1 - torch.mm(a_norm, b_norm.t()) 991 | dist_m = dist_m.numpy() 992 | 993 | prototype_dis_m = np.zeros([nb_classes, nb_classes]) 994 | for i in range(nb_classes): 995 | prototype_dis_m[i, :] = np.mean(dist_m[class_start_indices[i]:class_end_indices[i], :], axis=0) 996 | 997 | intra_class_dis = np.mean(prototype_dis_m.diagonal()) 998 | sum_distance = np.reshape(np.sum(prototype_dis_m, axis=-1), [nb_classes, ]) 999 | average_distance = np.sum(sum_distance) / (nb_classes * nb_classes) 1000 | 1001 | a = b = torch.from_numpy(center_ftrs) 1002 | a_norm = a / a.norm(dim=1)[:, None] 1003 | b_norm = b / b.norm(dim=1)[:, None] 1004 | dist_m = 1 - torch.mm(a_norm, b_norm.t()) 1005 | dist_m = dist_m.numpy() 1006 | inter_class_dis = np.mean(dist_m) 1007 | 1008 | mACT.append(average_distance / intra_class_dis) 1009 | mRCL.append(inter_class_dis / average_distance) 1010 | print('mACT: ', average_distance / intra_class_dis, 'mRCL: ', inter_class_dis / average_distance) 1011 | train_features_all = torch.from_numpy(train_features_all) 1012 | mAP, top_1, top_5, top_10 = evaluation() 1013 | top_1s.append(top_1) 1014 | top_5s.append(top_5) 1015 | top_10s.append(top_10) 1016 | mAPs.append(mAP) 1017 | 1018 | cur_patience += 1 1019 | if epoch > 0 and top_1 > max_acc_2: 1020 | max_acc_1 = mAP 1021 | max_acc_2 = top_1 1022 | top_5_max = top_5 1023 | top_10_max = top_10 1024 | 1025 | try: 1026 | best_cluster_info_1[0] = num_cluster 1027 | best_cluster_info_1[1] = outlier_num 1028 | except: 1029 | pass 1030 | cur_patience = 0 1031 | 1032 | if FLAGS.mode == 'Train': 1033 | if FLAGS.dataset != 'CASIA_B': 1034 | checkpt_file = pre_dir + dataset + '/' + probe + change + '/' + 'best.ckpt' 1035 | elif FLAGS.dataset == 'CASIA_B': 1036 | checkpt_file = pre_dir + dataset + '/' + probe + change + '/' + FLAGS.probe_type + '_best.ckpt' 1037 | print(checkpt_file) 1038 | if FLAGS.save_model == '1': 1039 | saver.save(sess, checkpt_file) 1040 | if epoch > 0: 1041 | if dataset == 'CASIA_B': 1042 | print( 1043 | '[MoCos - Probe Evaluation] %s - %s | mAP: %.4f (%.4f) | Top-1: %.4f (%.4f) | Top-5: %.4f (%.4f) | Top-10: %.4f (%.4f) | ' % ( 1044 | FLAGS.dataset, FLAGS.probe_type, mAP, max_acc_1, top_1, max_acc_2, top_5, top_5_max, top_10, top_10_max)) 1045 | else: 1046 | print( 1047 | '[MoCos - Probe Evaluation] %s - %s | mAP: %.4f (%.4f) | Top-1: %.4f (%.4f) | Top-5: %.4f (%.4f) | Top-10: %.4f (%.4f) | ' % ( 1048 | FLAGS.dataset, FLAGS.probe, mAP, max_acc_1, top_1, max_acc_2, top_5, top_5_max, top_10, top_10_max)) 1049 | 1050 | print( 1051 | 'Max: %.4f-%.4f-%.4f-%.4f' % (max_acc_1, max_acc_2, top_5_max, top_10_max)) 1052 | if cur_patience == patience: 1053 | print(simp_strs) 1054 | break 1055 | 1056 | def generate_cluster_features(labels, features): 1057 | centers = collections.defaultdict(list) 1058 | for i, label in enumerate(labels): 1059 | if label == -1: 1060 | continue 1061 | centers[labels[i]].append(features[i]) 1062 | 1063 | centers = [ 1064 | torch.stack(centers[idx], dim=0).mean(0) for idx in sorted(centers.keys()) 1065 | ] 1066 | centers = torch.stack(centers, dim=0) 1067 | return centers 1068 | 1069 | ori_train_labels = np.array(ori_train_labels) 1070 | ori_train_labels = np.argmax(ori_train_labels, axis=-1) 1071 | y_true = ori_train_labels 1072 | gt_class_features = generate_cluster_features(ori_train_labels, train_features_all) 1073 | X_train_J_new = X_train_J 1074 | tr_step = 0 1075 | tr_size = X_train_J_new.shape[0] 1076 | 1077 | mask_rand_save = np.array(mask_rand_save) 1078 | node_mask_save = np.array(node_mask_save) 1079 | batch_STr_CSP_loss = [] 1080 | batch_SSk_CSP_loss = [] 1081 | batch_CSP_loss = [] 1082 | 1083 | while tr_step * batch_size < tr_size: 1084 | if (tr_step + 1) * batch_size > tr_size: 1085 | break 1086 | X_input_J = X_train_J_new[tr_step * batch_size:(tr_step + 1) * batch_size] 1087 | X_input_J = X_input_J.reshape([-1, joint_num, 3]) 1088 | mask_rand = mask_rand_save[tr_step:(tr_step + 1)] 1089 | mask_rand = np.reshape(mask_rand, [time_step]) 1090 | mask_rand_2 = mask_rand_save_2[tr_step:(tr_step + 1)] 1091 | mask_rand_2 = np.reshape(mask_rand_2, [time_step]) 1092 | 1093 | gt_labels = y_true[tr_step * batch_size:(tr_step + 1) * batch_size] 1094 | node_mask_rand = node_mask_save[tr_step:(tr_step + 1)] 1095 | node_mask_rand = np.reshape(node_mask_rand, [joint_num]) 1096 | if FLAGS.rand_flip == '1': 1097 | sign_flip = np.random.random(pos_enc_ori.shape[1]) 1098 | sign_flip[sign_flip >= 0.5] = 1.0 1099 | sign_flip[sign_flip < 0.5] = -1.0 1100 | pos_enc_ori_rand = pos_enc_ori * sign_flip 1101 | _, CSP_loss_, Seq_features, STr_CSP_loss_, SSk_CSP_loss_, = sess.run( 1102 | [train_op, CSP_loss, seq_ftr, STr_CSP_loss, SSk_CSP_loss], 1103 | feed_dict={ 1104 | J_in: X_input_J, 1105 | seq_mask: mask_rand, 1106 | seq_mask_2: mask_rand_2, 1107 | node_mask: node_mask_rand, 1108 | L_eig: pos_enc_ori_rand, 1109 | gt_lab: gt_labels, 1110 | gt_class_ftr: gt_class_features, 1111 | train_flag: True 1112 | }) 1113 | elif FLAGS.rand_flip == '0': 1114 | _, CSP_loss_, Seq_features, STr_CSP_loss_, SSk_CSP_loss_, = sess.run( 1115 | [train_op, CSP_loss, seq_ftr, STr_CSP_loss, SSk_CSP_loss], 1116 | feed_dict={ 1117 | J_in: X_input_J, 1118 | seq_mask: mask_rand, 1119 | seq_mask_2: mask_rand_2, 1120 | node_mask: node_mask_rand, 1121 | L_eig: pos_enc_ori, 1122 | gt_lab: gt_labels, 1123 | gt_class_ftr: gt_class_features, 1124 | train_flag: True 1125 | }) 1126 | if FLAGS.save_flag == '1': 1127 | batch_CSP_loss.append(CSP_loss_) 1128 | batch_STr_CSP_loss.append(STr_CSP_loss_) 1129 | batch_SSk_CSP_loss.append(SSk_CSP_loss_) 1130 | 1131 | if tr_step % display == 0: 1132 | print( 1133 | '[%s] Batch num: %d | CSP Loss: %.5f | SSk-CSP Loss: %.5f | STr-CSP Loss: %.5f |' % 1134 | (str(epoch), tr_step, CSP_loss_, SSk_CSP_loss_, STr_CSP_loss_,)) 1135 | tr_step += 1 1136 | if FLAGS.save_flag == '1': 1137 | CSP_losses.append(np.mean(batch_CSP_loss)) 1138 | STr_CSP_losses.append(np.mean(batch_STr_CSP_loss)) 1139 | SSk_CSP_losses.append(np.mean(batch_SSk_CSP_loss)) 1140 | 1141 | if FLAGS.save_flag == '1': 1142 | try: 1143 | os.mkdir(pre_dir + dataset + '/' + probe + change + '/') 1144 | except: 1145 | pass 1146 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'top_1s.npy', top_1s) 1147 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'top_5s.npy', top_5s) 1148 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'top_10s.npy', top_10s) 1149 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'mAPs.npy', mAPs) 1150 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'STr_CSP_loss.npy', STr_CSP_losses) 1151 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'SSk_CSP_loss.npy', SSk_CSP_losses) 1152 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'CSP_loss.npy', CSP_losses) 1153 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'mACT.npy', mACT) 1154 | np.save(pre_dir + dataset + '/' + probe + change + '/' + 'mRCL.npy', mRCL) 1155 | 1156 | sess.close() 1157 | 1158 | elif FLAGS.mode == 'Eval': 1159 | checkpt_file = pre_dir + FLAGS.dataset + '/' + FLAGS.probe + change + '/best.ckpt' 1160 | 1161 | with tf.Session(graph=loaded_graph, config=config) as sess: 1162 | loader = tf.train.import_meta_graph(checkpt_file + '.meta') 1163 | 1164 | J_in = loaded_graph.get_tensor_by_name("Input/Placeholder:0") 1165 | L_eig = loaded_graph.get_tensor_by_name("Input/Placeholder_1:0") 1166 | train_flag = loaded_graph.get_tensor_by_name("Input/Placeholder_2:0") 1167 | seq_ftr = loaded_graph.get_tensor_by_name("MoCos/MoCos/Reshape_36:0") 1168 | 1169 | init_op = tf.global_variables_initializer() 1170 | sess.run(init_op) 1171 | loader.restore(sess, checkpt_file) 1172 | saver = tf.train.Saver() 1173 | mask_rand_save = [] 1174 | node_mask_save = [] 1175 | 1176 | def gal_loader(X_train_J, y_train): 1177 | tr_step = 0 1178 | tr_size = X_train_J.shape[0] 1179 | gal_logits_all = [] 1180 | gal_labels_all = [] 1181 | gal_features_all = [] 1182 | embed_1_all = [] 1183 | embed_2_all = [] 1184 | 1185 | while tr_step * batch_size < tr_size: 1186 | if (tr_step + 1) * batch_size > tr_size: 1187 | break 1188 | X_input_J = X_train_J[tr_step * batch_size:(tr_step + 1) * batch_size] 1189 | X_input_J = X_input_J.reshape([-1, joint_num, 3]) 1190 | labels = y_train[tr_step * batch_size:(tr_step + 1) * batch_size] 1191 | [Seq_features] = sess.run([seq_ftr], 1192 | feed_dict={ 1193 | J_in: X_input_J, 1194 | L_eig: pos_enc_ori, 1195 | train_flag: False 1196 | }) 1197 | gal_features_all.extend(Seq_features.tolist()) 1198 | gal_labels_all.extend(labels.tolist()) 1199 | tr_step += 1 1200 | return gal_features_all, gal_labels_all, embed_1_all, embed_2_all 1201 | 1202 | def evaluation(): 1203 | vl_step = 0 1204 | vl_size = X_test_J.shape[0] 1205 | pro_labels_all = [] 1206 | pro_features_all = [] 1207 | while vl_step * batch_size < vl_size: 1208 | if (vl_step + 1) * batch_size > vl_size: 1209 | break 1210 | X_input_J = X_test_J[vl_step * batch_size:(vl_step + 1) * batch_size] 1211 | X_input_J = X_input_J.reshape([-1, joint_num, 3]) 1212 | labels = y_test[vl_step * batch_size:(vl_step + 1) * batch_size] 1213 | [Seq_features] = sess.run([seq_ftr], 1214 | feed_dict={ 1215 | J_in: X_input_J, 1216 | L_eig: pos_enc_ori, 1217 | train_flag: False 1218 | }) 1219 | pro_labels_all.extend(labels.tolist()) 1220 | pro_features_all.extend(Seq_features.tolist()) 1221 | vl_step += 1 1222 | X = np.array(gal_features_all) 1223 | y = np.array(gal_labels_all) 1224 | t_X = np.array(pro_features_all) 1225 | t_y = np.array(pro_labels_all) 1226 | t_y = np.argmax(t_y, axis=-1) 1227 | y = np.argmax(y, axis=-1) 1228 | 1229 | def mean_ap(distmat, query_ids=None, gallery_ids=None, 1230 | query_cams=None, gallery_cams=None): 1231 | # distmat = to_numpy(distmat) 1232 | m, n = distmat.shape 1233 | # Fill up default values 1234 | if query_ids is None: 1235 | query_ids = np.arange(m) 1236 | if gallery_ids is None: 1237 | gallery_ids = np.arange(n) 1238 | if query_cams is None: 1239 | query_cams = np.zeros(m).astype(np.int32) 1240 | if gallery_cams is None: 1241 | gallery_cams = np.ones(n).astype(np.int32) 1242 | # Ensure numpy array 1243 | query_ids = np.asarray(query_ids) 1244 | gallery_ids = np.asarray(gallery_ids) 1245 | query_cams = np.asarray(query_cams) 1246 | gallery_cams = np.asarray(gallery_cams) 1247 | # Sort and find correct matches 1248 | indices = np.argsort(distmat, axis=1) 1249 | matches = (gallery_ids[indices] == query_ids[:, np.newaxis]) 1250 | # Compute AP for each query 1251 | aps = [] 1252 | if (FLAGS.probe_type == 'nm.nm' or FLAGS.probe_type == 'cl.cl' or FLAGS.probe_type == 'bg.bg'): 1253 | for i in range(1, m): 1254 | valid = ((gallery_ids[indices[i]] != query_ids[i]) | 1255 | (gallery_cams[indices[i]] != query_cams[i])) 1256 | 1257 | y_true = matches[i, valid] 1258 | y_score = -distmat[i][indices[i]][valid] 1259 | # discard nan 1260 | y_score[np.isnan(y_score)] = 0 1261 | if not np.any(y_true): continue 1262 | aps.append(average_precision_score(y_true, y_score)) 1263 | else: 1264 | for i in range(m): 1265 | valid = ((gallery_ids[indices[i]] != query_ids[i]) | 1266 | (gallery_cams[indices[i]] != query_cams[i])) 1267 | y_true = matches[i, valid] 1268 | y_score = -distmat[i][indices[i]][valid] 1269 | if not np.any(y_true): continue 1270 | aps.append(average_precision_score(y_true, y_score)) 1271 | if len(aps) == 0: 1272 | raise RuntimeError("No valid query") 1273 | return np.mean(aps) 1274 | 1275 | def metrics(X, y, t_X, t_y): 1276 | # compute Euclidean distance 1277 | if dataset != 'CASIA_B': 1278 | a, b = torch.from_numpy(t_X), torch.from_numpy(X) 1279 | m, n = a.size(0), b.size(0) 1280 | a = a.view(m, -1) 1281 | b = b.view(n, -1) 1282 | dist_m = torch.pow(a, 2).sum(dim=1, keepdim=True).expand(m, n) + \ 1283 | torch.pow(b, 2).sum(dim=1, keepdim=True).expand(n, m).t() 1284 | dist_m.addmm_(1, -2, a, b.t()) 1285 | # 1e-12 1286 | dist_m = (dist_m.clamp(min=1e-12)).sqrt() 1287 | mAP = mean_ap(distmat=dist_m.numpy(), query_ids=t_y, gallery_ids=y) 1288 | _, dist_sort = dist_m.sort(1) 1289 | dist_sort = dist_sort.numpy() 1290 | else: 1291 | X = np.array(X) 1292 | t_X = np.array(t_X) 1293 | dist_m = [(np.linalg.norm(X - i, axis=1)).tolist() for i in t_X] 1294 | dist_m = np.array(dist_m) 1295 | mAP = mean_ap(distmat=dist_m, query_ids=t_y, gallery_ids=y) 1296 | dist_sort = [np.argsort(np.linalg.norm(X - i, axis=1)).tolist() for i in t_X] 1297 | dist_sort = np.array(dist_sort) 1298 | 1299 | top_1 = top_5 = top_10 = 0 1300 | probe_num = dist_sort.shape[0] 1301 | if (FLAGS.probe_type == 'nm.nm' or 1302 | FLAGS.probe_type == 'cl.cl' or FLAGS.probe_type == 'bg.bg'): 1303 | for i in range(probe_num): 1304 | if t_y[i] in y[dist_sort[i, 1:2]]: 1305 | top_1 += 1 1306 | if t_y[i] in y[dist_sort[i, 1:6]]: 1307 | top_5 += 1 1308 | if t_y[i] in y[dist_sort[i, 1:11]]: 1309 | top_10 += 1 1310 | else: 1311 | for i in range(probe_num): 1312 | # print(dist_sort[i, :10]) 1313 | if t_y[i] in y[dist_sort[i, :1]]: 1314 | top_1 += 1 1315 | if t_y[i] in y[dist_sort[i, :5]]: 1316 | top_5 += 1 1317 | if t_y[i] in y[dist_sort[i, :10]]: 1318 | top_10 += 1 1319 | return mAP, top_1 / probe_num, top_5 / probe_num, top_10 / probe_num 1320 | 1321 | mAP, top_1, top_5, top_10 = metrics(X, y, t_X, t_y) 1322 | del X, y, t_X, t_y, pro_labels_all, pro_features_all 1323 | gc.collect() 1324 | return mAP, top_1, top_5, top_10 1325 | 1326 | 1327 | if dataset == 'KGBD' or dataset == 'KS20': 1328 | # if FLAGS.level == 'J': 1329 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 1330 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 1331 | process.gen_train_data(dataset=dataset, split='gallery', time_step=time_step, 1332 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 1333 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 1334 | 1335 | elif dataset == 'BIWI': 1336 | if probe == 'Walking': 1337 | # if FLAGS.level == 'J': 1338 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 1339 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 1340 | process.gen_train_data(dataset=dataset, split='Still', time_step=time_step, 1341 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 1342 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 1343 | 1344 | else: 1345 | # if FLAGS.level == 'J': 1346 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 1347 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 1348 | process.gen_train_data(dataset=dataset, split='Walking', time_step=time_step, 1349 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 1350 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 1351 | 1352 | elif dataset == 'IAS': 1353 | if probe == 'A': 1354 | # if FLAGS.level == 'J': 1355 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 1356 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 1357 | process.gen_train_data(dataset=dataset, split='B', time_step=time_step, 1358 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 1359 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 1360 | 1361 | else: 1362 | # if FLAGS.level == 'J': 1363 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 1364 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 1365 | process.gen_train_data(dataset=dataset, split='A', time_step=time_step, 1366 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 1367 | batch_size=batch_size, enc_k=int(FLAGS.enc_k)) 1368 | 1369 | elif dataset == 'CASIA_B': 1370 | X_train_J, _, _, _, _, y_train, X_gal_J, _, _, _, _, y_gal, \ 1371 | adj_J, _, pos_enc_ori, _, _, _, _, _, _, _, _, _, _, nb_classes = \ 1372 | process.gen_train_data(dataset=dataset, split=probe, time_step=time_step, 1373 | nb_nodes=nb_nodes, nhood=nhood, global_att=global_att, 1374 | batch_size=batch_size, PG_type=FLAGS.probe_type.split('.')[1]) 1375 | del _ 1376 | gc.collect() 1377 | 1378 | mAP_max = top_1_max = top_5_max = top_10_max = 0 1379 | 1380 | gal_features_all, gal_labels_all, gal_embed_1_all, gal_embed_2_all = gal_loader(X_gal_J, y_gal) 1381 | mAP, top_1, top_5, top_10 = evaluation() 1382 | print( 1383 | '[Evaluation on %s - %s] mAP: %.4f | R1: %.4f - R5: %.4f - R10: %.4f |' % 1384 | (FLAGS.dataset, FLAGS.probe, mAP, top_1, top_5, top_10,)) 1385 | sess.close() 1386 | exit() 1387 | 1388 | print('----- Model hyperparams -----') 1389 | print('f (sequence length): ' + str(time_step)) 1390 | print('H (embedding size): ' + FLAGS.H) 1391 | print('MGT Layers: ' + FLAGS.L_transformer) 1392 | print('heads: ' + FLAGS.n_heads) 1393 | print('MoCos fusion lambda: ' + FLAGS.fusion_lambda) 1394 | print('t1: ' + FLAGS.t_1) 1395 | print('t2: ' + FLAGS.t_2) 1396 | 1397 | print('batch_size: ' + str(batch_size)) 1398 | print('lr: ' + str(FLAGS.lr)) 1399 | 1400 | print('patience: ' + FLAGS.patience) 1401 | print('Mode: ' + FLAGS.mode) 1402 | 1403 | print('p_s: ' + FLAGS.prob_s) 1404 | print('p_t: ' + FLAGS.prob_t) 1405 | print('lambda: ' + FLAGS.fusion_lambda) 1406 | 1407 | if FLAGS.mode == 'Train': 1408 | print('----- Dataset Information -----') 1409 | print('Dataset: ' + dataset) 1410 | if dataset == 'CASIA_B': 1411 | print('Probe.Gallery: ', FLAGS.probe_type.split('.')[0], FLAGS.probe_type.split('.')[1]) 1412 | else: 1413 | print('Probe: ' + FLAGS.probe) 1414 | --------------------------------------------------------------------------------