├── .gitignore ├── README.md ├── requirements.txt ├── run.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled/cached Python 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # Temporaries 7 | .ipynb_checkpoints/ 8 | *.swp 9 | 10 | # Outputs 11 | *.csv 12 | *.svg 13 | *.eps 14 | *.out 15 | *.mat 16 | *.bmp 17 | *.png 18 | *.npy 19 | 20 | .idea 21 | *.DS_Store 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PanNuke Evaluation Metrics 2 | 3 | This repository calculates metrics on the PanNuke dataset, as reported in:
4 | 5 | **"PanNuke Dataset Extension, Insights and Baselines"**
6 | 7 | The PanNuke dataset can be downloaded [here](https://warwick.ac.uk/fac/sci/dcs/research/tia/data/pannuke).
8 | In the repository, the metrics that are calculated are:
9 | 10 | - **Binary PQ (bPQ)**: Assumes all nuclei belong to same class and reports the average PQ across tissue types.
11 | - **Multi-Class PQ (mPQ)**: Reports the average PQ across the classes and tissue types.
12 | - **Neoplastic PQ**: Reports the PQ for the neoplastic class on all tissues.
13 | - **Non-Neoplastic PQ**: Reports the PQ for the non-neoplastic class on all tissues.
14 | - **Inflammatory PQ**: Reports the PQ for the inflammatory class on all tissues.
15 | - **Connective PQ**: Reports the PQ for the connective class on all tissues.
16 | - **Dead PQ**: Reports the PQ for the dead class on all tissues.
17 | 18 | 19 | For detection based metrics, we used [this function](https://github.com/vqdang/hover_net/blob/master/src/compute_stats.py#L13). 20 | 21 | ## Set up envrionment 22 | 23 | ``` 24 | conda create --name pannuke python=3.6 25 | conda activate pannuke 26 | pip install -r requirements.txt 27 | ``` 28 | 29 | ## Running the Code 30 | 31 | Usage: 32 | ``` 33 | """run. 34 | 35 | Usage: 36 | run.py --true_path= --pred_path= --save_path= 37 | run.py (-h | --help) 38 | run.py --version 39 | ``` 40 | 41 | Options: 42 | ``` 43 | -h --help Show this string. 44 | --version Show version. 45 | --true_path= Root path to where the ground-truth is saved. 46 | --pred_path= Root path to where the predictions are saved. 47 | --save_path= Path where the prediction CSV files will be saved 48 | ``` 49 | 50 | Before running the code, ground truth and predictions must be saved in the following structure:
51 | 52 | - True Masks: 53 | - `/masks.npy` 54 | - True Tissue Types: 55 | - `/types.npy` 56 | - Prediction Masks: 57 | - `/masks.npy` 58 | 59 | Here, prediction masks are saved in the same format as the true masks. i.e a single `Nx256x256xC` array, where `N` is the number of test images in that specific fold and `C` is the number of positive classes. The ordering of the channels from index `0` to `4` is `neoplastic`, `inflammatory`, `connective tissue`, `dead` and `non-neoplastic epithelial`. 60 | 61 | ## Citation 62 | 63 | If using this code, please cite:
64 | 65 | ``` 66 | @inproceedings{gamper2019pannuke, 67 | title={PanNuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification}, 68 | author={Gamper, Jevgenij and Koohbanani, Navid Alemi and Benet, Ksenija and Khuram, Ali and Rajpoot, Nasir}, 69 | booktitle={European Congress on Digital Pathology}, 70 | pages={11--19}, 71 | year={2019}, 72 | organization={Springer} 73 | } 74 | ``` 75 | ``` 76 | @article{gamper2020pannuke, 77 | title={PanNuke Dataset Extension, Insights and Baselines}, 78 | author={Gamper, Jevgenij and Koohbanani, Navid Alemi and Graham, Simon and Jahanifar, Mostafa and Khurram, Syed Ali and Azam, Ayesha and Hewitt, Katherine and Rajpoot, Nasir}, 79 | journal={arXiv preprint arXiv:2003.10778}, 80 | year={2020} 81 | } 82 | ``` 83 | 84 | 85 | 86 | 87 | 88 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | docopt 2 | numpy==1.15.4 3 | pandas 4 | scipy 5 | -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | """run. 2 | 3 | Usage: 4 | run.py --true_path= --pred_path= --save_path= 5 | run.py (-h | --help) 6 | run.py --version 7 | 8 | Options: 9 | -h --help Show this string. 10 | --version Show version. 11 | --true_path= Root path to where the ground-truth is saved. 12 | --pred_path= Root path to where the predictions are saved. 13 | --save_path= Path where the prediction CSV files will be saved 14 | """ 15 | 16 | 17 | import docopt 18 | import os 19 | import numpy as np 20 | import pandas as pd 21 | from utils import get_fast_pq, remap_label, binarize 22 | 23 | tissue_types = [ 24 | 'Adrenal_gland', 25 | 'Bile-duct', 26 | 'Bladder', 27 | 'Breast', 28 | 'Cervix', 29 | 'Colon', 30 | 'Esophagus', 31 | 'HeadNeck', 32 | 'Kidney', 33 | 'Liver', 34 | 'Lung', 35 | 'Ovarian', 36 | 'Pancreatic', 37 | 'Prostate', 38 | 'Skin', 39 | 'Stomach', 40 | 'Testis', 41 | 'Thyroid', 42 | 'Uterus' 43 | ] 44 | 45 | def main(args): 46 | """ 47 | This function returns the statistics reported on the PanNuke dataset, reported in the paper below: 48 | 49 | Gamper, Jevgenij, Navid Alemi Koohbanani, Simon Graham, Mostafa Jahanifar, Syed Ali Khurram, 50 | Ayesha Azam, Katherine Hewitt, and Nasir Rajpoot. 51 | "PanNuke Dataset Extension, Insights and Baselines." arXiv preprint arXiv:2003.10778 (2020). 52 | 53 | Args: 54 | Root path to the ground-truth 55 | Root path to the predictions 56 | Path where results will be saved 57 | 58 | Output: 59 | Terminal output of bPQ and mPQ results for each class and across tissues 60 | Saved CSV files for bPQ and mPQ results for each class and across tissues 61 | """ 62 | 63 | true_root = args['--true_path'] 64 | pred_root = args['--pred_path'] 65 | save_path = args['--save_path'] 66 | 67 | if not os.path.exists(save_path): 68 | os.mkdir(save_path) 69 | 70 | true_path = os.path.join(true_root,'masks.npy') # path to the GT for a specific split 71 | pred_path = os.path.join(pred_root, 'masks.npy') # path to the predictions for a specific split 72 | types_path = os.path.join(true_root,'types.npy') # path to the nuclei types 73 | 74 | # load the data 75 | true = np.load(true_path) 76 | pred = np.load(pred_path) 77 | types = np.load(types_path) 78 | 79 | mPQ_all = [] 80 | bPQ_all = [] 81 | 82 | # loop over the images 83 | for i in range(true.shape[0]): 84 | pq = [] 85 | pred_bin = binarize(pred[i,:,:,:5]) 86 | true_bin = binarize(true[i,:,:,:5]) 87 | 88 | if len(np.unique(true_bin)) == 1: 89 | pq_bin = np.nan # if ground truth is empty for that class, skip from calculation 90 | else: 91 | [_, _, pq_bin], _ = get_fast_pq(true_bin, pred_bin) # compute PQ 92 | 93 | # loop over the classes 94 | for j in range(5): 95 | pred_tmp = pred[i,:,:,j] 96 | pred_tmp = pred_tmp.astype('int32') 97 | true_tmp = true[i,:,:,j] 98 | true_tmp = true_tmp.astype('int32') 99 | pred_tmp = remap_label(pred_tmp) 100 | true_tmp = remap_label(true_tmp) 101 | 102 | if len(np.unique(true_tmp)) == 1: 103 | pq_tmp = np.nan # if ground truth is empty for that class, skip from calculation 104 | else: 105 | [_, _, pq_tmp] , _ = get_fast_pq(true_tmp, pred_tmp) # compute PQ 106 | 107 | pq.append(pq_tmp) 108 | 109 | mPQ_all.append(pq) 110 | bPQ_all.append([pq_bin]) 111 | 112 | # using np.nanmean skips values with nan from the mean calculation 113 | mPQ_each_image = [np.nanmean(pq) for pq in mPQ_all] 114 | bPQ_each_image = [np.nanmean(pq_bin) for pq_bin in bPQ_all] 115 | 116 | # class metric 117 | neo_PQ = np.nanmean([pq[0] for pq in mPQ_all]) 118 | inflam_PQ = np.nanmean([pq[1] for pq in mPQ_all]) 119 | conn_PQ = np.nanmean([pq[2] for pq in mPQ_all]) 120 | dead_PQ = np.nanmean([pq[3] for pq in mPQ_all]) 121 | nonneo_PQ = np.nanmean([pq[4] for pq in mPQ_all]) 122 | 123 | # Print for each class 124 | print('Printing calculated metrics on a single split') 125 | print('-'*40) 126 | print('Neoplastic PQ: {}'.format(neo_PQ)) 127 | print('Inflammatory PQ: {}'.format(inflam_PQ)) 128 | print('Connective PQ: {}'.format(conn_PQ)) 129 | print('Dead PQ: {}'.format(dead_PQ)) 130 | print('Non-Neoplastic PQ: {}'.format(nonneo_PQ)) 131 | print('-' * 40) 132 | 133 | # Save per-class metrics as a csv file 134 | for_dataframe = {'Class Name': ['Neoplastic', 'Inflam', 'Connective', 'Dead', 'Non-Neoplastic'], 135 | 'PQ': [neo_PQ, conn_PQ, conn_PQ, dead_PQ, nonneo_PQ]} 136 | df = pd.DataFrame(for_dataframe, columns=['Tissue name', 'PQ']) 137 | df.to_csv(save_path + '/class_stats.csv') 138 | 139 | # Print for each tissue 140 | all_tissue_mPQ = [] 141 | all_tissue_bPQ = [] 142 | for tissue_name in tissue_types: 143 | indices = [i for i, x in enumerate(types) if x == tissue_name] 144 | tissue_PQ = [mPQ_each_image[i] for i in indices] 145 | print('{} PQ: {} '.format(tissue_name, np.nanmean(tissue_PQ))) 146 | tissue_PQ_bin = [bPQ_each_image[i] for i in indices] 147 | print('{} PQ binary: {} '.format(tissue_name, np.nanmean(tissue_PQ_bin))) 148 | all_tissue_mPQ.append(np.nanmean(tissue_PQ)) 149 | all_tissue_bPQ.append(np.nanmean(tissue_PQ_bin)) 150 | 151 | # Save per-tissue metrics as a csv file 152 | for_dataframe = {'Tissue name': tissue_types + ['mean'], 153 | 'PQ': all_tissue_mPQ + [np.nanmean(all_tissue_mPQ)] , 'PQ bin': all_tissue_bPQ + [np.nanmean(all_tissue_bPQ)]} 154 | df = pd.DataFrame(for_dataframe, columns=['Tissue name', 'PQ', 'PQ bin']) 155 | df.to_csv(save_path + '/tissue_stats.csv') 156 | 157 | # Show overall metrics - mPQ is average PQ over the classes and the tissues, bPQ is average binary PQ over the tissues 158 | print('-' * 40) 159 | print('Average mPQ:{}'.format(np.nanmean(all_tissue_mPQ))) 160 | print('Average bPQ:{}'.format(np.nanmean(all_tissue_bPQ))) 161 | 162 | ##### 163 | if __name__ == '__main__': 164 | args = docopt.docopt(__doc__, version='PanNuke Evaluation v1.0') 165 | main(args) 166 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy.optimize import linear_sum_assignment 3 | 4 | #### 5 | def get_fast_pq(true, pred, match_iou=0.5): 6 | """ 7 | `match_iou` is the IoU threshold level to determine the pairing between 8 | GT instances `p` and prediction instances `g`. `p` and `g` is a pair 9 | if IoU > `match_iou`. However, pair of `p` and `g` must be unique 10 | (1 prediction instance to 1 GT instance mapping). 11 | 12 | If `match_iou` < 0.5, Munkres assignment (solving minimum weight matching 13 | in bipartite graphs) is caculated to find the maximal amount of unique pairing. 14 | 15 | If `match_iou` >= 0.5, all IoU(p,g) > 0.5 pairing is proven to be unique and 16 | the number of pairs is also maximal. 17 | 18 | Fast computation requires instance IDs are in contiguous orderding 19 | i.e [1, 2, 3, 4] not [2, 3, 6, 10]. Please call `remap_label` beforehand 20 | and `by_size` flag has no effect on the result. 21 | 22 | Returns: 23 | [dq, sq, pq]: measurement statistic 24 | 25 | [paired_true, paired_pred, unpaired_true, unpaired_pred]: 26 | pairing information to perform measurement 27 | 28 | """ 29 | assert match_iou >= 0.0, "Cant' be negative" 30 | 31 | true = np.copy(true) 32 | pred = np.copy(pred) 33 | true_id_list = list(np.unique(true)) 34 | pred_id_list = list(np.unique(pred)) 35 | 36 | true_masks = [None,] 37 | for t in true_id_list[1:]: 38 | t_mask = np.array(true == t, np.uint8) 39 | true_masks.append(t_mask) 40 | 41 | pred_masks = [None,] 42 | for p in pred_id_list[1:]: 43 | p_mask = np.array(pred == p, np.uint8) 44 | pred_masks.append(p_mask) 45 | 46 | # prefill with value 47 | pairwise_iou = np.zeros([len(true_id_list) -1, 48 | len(pred_id_list) -1], dtype=np.float64) 49 | 50 | # caching pairwise iou 51 | for true_id in true_id_list[1:]: # 0-th is background 52 | t_mask = true_masks[true_id] 53 | pred_true_overlap = pred[t_mask > 0] 54 | pred_true_overlap_id = np.unique(pred_true_overlap) 55 | pred_true_overlap_id = list(pred_true_overlap_id) 56 | for pred_id in pred_true_overlap_id: 57 | if pred_id == 0: # ignore 58 | continue # overlaping background 59 | p_mask = pred_masks[pred_id] 60 | total = (t_mask + p_mask).sum() 61 | inter = (t_mask * p_mask).sum() 62 | iou = inter / (total - inter) 63 | pairwise_iou[true_id-1, pred_id-1] = iou 64 | # 65 | if match_iou >= 0.5: 66 | paired_iou = pairwise_iou[pairwise_iou > match_iou] 67 | pairwise_iou[pairwise_iou <= match_iou] = 0.0 68 | paired_true, paired_pred = np.nonzero(pairwise_iou) 69 | paired_iou = pairwise_iou[paired_true, paired_pred] 70 | paired_true += 1 # index is instance id - 1 71 | paired_pred += 1 # hence return back to original 72 | else: # * Exhaustive maximal unique pairing 73 | #### Munkres pairing with scipy library 74 | # the algorithm return (row indices, matched column indices) 75 | # if there is multiple same cost in a row, index of first occurence 76 | # is return, thus the unique pairing is ensure 77 | # inverse pair to get high IoU as minimum 78 | paired_true, paired_pred = linear_sum_assignment(-pairwise_iou) 79 | ### extract the paired cost and remove invalid pair 80 | paired_iou = pairwise_iou[paired_true, paired_pred] 81 | 82 | # now select those above threshold level 83 | # paired with iou = 0.0 i.e no intersection => FP or FN 84 | paired_true = list(paired_true[paired_iou > match_iou] + 1) 85 | paired_pred = list(paired_pred[paired_iou > match_iou] + 1) 86 | paired_iou = paired_iou[paired_iou > match_iou] 87 | 88 | # get the actual FP and FN 89 | unpaired_true = [idx for idx in true_id_list[1:] if idx not in paired_true] 90 | unpaired_pred = [idx for idx in pred_id_list[1:] if idx not in paired_pred] 91 | # print(paired_iou.shape, paired_true.shape, len(unpaired_true), len(unpaired_pred)) 92 | 93 | # 94 | tp = len(paired_true) 95 | fp = len(unpaired_pred) 96 | fn = len(unpaired_true) 97 | # get the F1-score i.e DQ 98 | dq = tp / (tp + 0.5 * fp + 0.5 * fn) 99 | # get the SQ, no paired has 0 iou so not impact 100 | sq = paired_iou.sum() / (tp + 1.0e-6) 101 | 102 | return [dq, sq, dq * sq], [paired_true, paired_pred, unpaired_true, unpaired_pred] 103 | ##### 104 | 105 | def remap_label(pred, by_size=False): 106 | """ 107 | Rename all instance id so that the id is contiguous i.e [0, 1, 2, 3] 108 | not [0, 2, 4, 6]. The ordering of instances (which one comes first) 109 | is preserved unless by_size=True, then the instances will be reordered 110 | so that bigger nucler has smaller ID 111 | 112 | Args: 113 | pred : the 2d array contain instances where each instances is marked 114 | by non-zero integer 115 | by_size : renaming with larger nuclei has smaller id (on-top) 116 | """ 117 | pred_id = list(np.unique(pred)) 118 | pred_id.remove(0) 119 | if len(pred_id) == 0: 120 | return pred # no label 121 | if by_size: 122 | pred_size = [] 123 | for inst_id in pred_id: 124 | size = (pred == inst_id).sum() 125 | pred_size.append(size) 126 | # sort the id by size in descending order 127 | pair_list = zip(pred_id, pred_size) 128 | pair_list = sorted(pair_list, key=lambda x: x[1], reverse=True) 129 | pred_id, pred_size = zip(*pair_list) 130 | 131 | new_pred = np.zeros(pred.shape, np.int32) 132 | for idx, inst_id in enumerate(pred_id): 133 | new_pred[pred == inst_id] = idx + 1 134 | return new_pred 135 | #### 136 | 137 | 138 | def binarize(x): 139 | ''' 140 | convert multichannel (multiclass) instance segmetation tensor 141 | to binary instance segmentation (bg and nuclei), 142 | 143 | :param x: B*B*C (for PanNuke 256*256*5 ) 144 | :return: Instance segmentation 145 | ''' 146 | out = np.zeros([x.shape[0], x.shape[1]]) 147 | count = 1 148 | for i in range(x.shape[2]): 149 | x_ch = x[:,:,i] 150 | unique_vals = np.unique(x_ch) 151 | unique_vals = unique_vals.tolist() 152 | unique_vals.remove(0) 153 | for j in unique_vals: 154 | x_tmp = x_ch == j 155 | x_tmp_c = 1- x_tmp 156 | out *= x_tmp_c 157 | out += count*x_tmp 158 | count += 1 159 | out = out.astype('int32') 160 | return out 161 | #### 162 | 163 | def get_tissue_idx(tissue_indices, idx): 164 | for i in range(len(tissue_indices)): 165 | if tissue_indices[i].count(idx) == 1: 166 | tiss_idx = i 167 | return tiss_idx --------------------------------------------------------------------------------