├── .gitignore ├── LICENSE ├── README.md ├── data ├── calib.txt └── list50.txt ├── docs └── int7Inference.png ├── example └── vgg16 │ ├── run.sh │ ├── run_caffe2ncnn.sh │ ├── run_infer_shape.sh │ ├── run_scale_fine_tuning.sh │ ├── run_scale_quantation.sh │ └── run_validation.sh ├── model └── vgg16 │ └── net_file_upgrade.sh ├── python_ncnn ├── Makefile ├── README.md ├── ncnn.cpp ├── ncnn.h └── ncnn2python.cpp ├── requirements.txt └── tools ├── caffe_quanttable_e2e.py ├── hist_tool.py ├── infer_shape.py ├── ncnn_val_caffe.py └── scale_fine_tuning.py /.gitignore: -------------------------------------------------------------------------------- 1 | # pickle file of result 2 | *.pickle 3 | *.pkl 4 | 5 | # log file 6 | *.log 7 | 8 | # model file 9 | *.param 10 | *.bin 11 | *.prototxt 12 | *.caffemodel 13 | 14 | # build object 15 | *.o 16 | *.so 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | Copyright (C) 2020 DeepGlint. All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EasyQuant: Post-training Quantization via Scale Optimization 2 | 3 | EasyQuant(EQ) is an efficient and simple post-training quantization method via effectively optimizing the scales of weights and activations. Our paper is available on [arXiv](https://arxiv.org/pdf/2006.16669.pdf) 4 | 5 | ## Requirements 6 | 7 | [eq-ncnn](https://github.com/deepglint/eq-ncnn) 8 | 9 | [caffe](https://github.com/BVLC/caffe) 10 | 11 | ``` 12 | pip install -r requirements.txt 13 | ``` 14 | 15 | ## Updates: 16 | * 06/25/2020: We have released EasyQuant.pdf paper and eq-ncnn. 17 | * 06/24/2020: We have released VGG16 example. 18 | 19 | ## Data Preparation 20 | First, for ImageNet1k classification task, please download [ImageNet2012](http://image-net.org/download). We random sampled 3000 calibration images from ImageNet val set to `data/calib.txt` for KLD quantization and select 50 samples from calib to `data/list50.txt` for EasyQuant scale finetuning. Then we will evaluate quantified models on val set. 21 | 22 | ## How to Run 23 | 24 | 1. Get caffe and ncnn ready 25 | 2. Build python\_ncnn 26 | ``` 27 | # 28 | cd python_ncnn 29 | # modify ncnn build path in Makefile 30 | make -j8 31 | ``` 32 | 33 | 3. Run VGG16 example 34 | 35 | ``` 36 | cd .. 37 | sh example/vgg16/run.sh 38 | ``` 39 | 40 | This following 6 steps will be performed in run.sh. 41 | ``` 42 | # download vgg16 model and upgrade proto and caffemodel files 43 | sh model/vgg16/net_file_upgrade.sh 44 | 45 | # generate scale of weight and activation use quantation tools 46 | sh example/vgg16/run_scale_quantation.sh 47 | 48 | # get ncnn param bin from scale table 49 | sh example/vgg16/run_caffe2ncnn.sh 50 | 51 | # infer layer blob shape 52 | sh example/vgg16/run_infer_shape.sh 53 | 54 | # run scale fine tuning 55 | sh example/vgg16/run_scale_fine_tuning.sh 56 | 57 | # run validtion on imagenet val 58 | sh example/vgg16/run_validation.sh 59 | ``` 60 | 61 | ## Results 62 | 63 | ### Validation Results 64 | #### 1. Classification on ImageNet2012 validation dataset for different convolutional models in context of both INT8 and INT7 post-training quantization. 65 | | Models | FP32 | INT8-TRT | INT8-EQ | INT7-TRT | INT7-EQ | 66 | | :----: | :--: | :------: | :-----: | :------: | :-----: | 67 | | SqueezeNetV1.1 | 56.56 | 56.24 | **56.28** | 54.88 | **56.08** | 68 | | MobileNetV1 | 69.33 | 68.74 | **68.84** | 66.97 | **68.26** | 69 | | VGG16 | 70.97 | 70.95 | **70.97** | 70.92 | **70.96** | 70 | | ResNet50 | 75.20 | 75.04 | **75.13** | 72.78 | **75.04** | 71 | 72 | #### 2. Object detection on VOC2007 task for SSD models with bachbone SqueezeNet and MobileNet V1. 73 | 74 | | Models | FP32 | INT8-TRT | INT8-EQ | INT7-TRT | INT7-EQ | 75 | | :----: | :--: | :------: | :-----: | :------: | :-----: | 76 | | SqueezeNet-SSD | 62.00 | 61.45 | **62.05** | 60.01 | **61.62** | 77 | | MobileNet-SSD | 72.04 | 69.79 | **71.39** | 63.88 | **68.79** | 78 | 79 | #### 3. Verification performance for InsightFace model MobileFaceNet on 7 most common validation dataset 80 | | Models | FP32 | INT8-TRT | INT8-EQ | INT7-TRT | INT7-EQ | 81 | | :----: | :--: | :------: | :-----: | :------: | :-----: | 82 | | lfw | 99.45 | 99.36 | **99.48** | 99.28 | **99.36** | 83 | | agedb\_30 | 95.78 | 95.23 | **95.38** | 95.03 | **95.73** | 84 | | calfw | 95.05 | 94.76 | **94.88** | **94.75** | 94.68 | 85 | | cfp\_ff | 99.50 | 99.50 | **99.61** | 99.44 | **99.60** | 86 | | cfp\_fp | 89.77 | 89.17 | **90.04** | 88.47 | **89.87** | 87 | | cplfw | 86.45 | 85.58 | **86.03** | 85.91 | **86.76** | 88 | | vgg2\_fp | 90.64 | 89.70 | **90.50** | 89.64 | **90.44** | 89 | 90 | #### 4. Compare our method with more complex QAT(quantization aware trainin) approach in 8 bit width. 91 | | Methods | MobileNetV1-FP32 | MobileNetV1-INT8 | ResNet50-FP32 | ResNet50-INT8 | 92 | | :-----: | :--------------: | :--------------: | :-----------: | :-----------: | 93 | | EQ | 69.33 | 68.84 | 75.20 | **75.13** | 94 | | QAT | 70.90 | **70.70** | 75.20 | 75.00 | 95 | 96 | ### Speed Tests 97 | 98 | INT7 Post-training Inference VS INT8 on real devices. 99 | We implement our efficient designs on INT7 post-training inference which well be released in [eq-ncnn](https://github.com/deepglint/eq-ncnn) 100 | 101 | 102 | #### 1. The latency (ms) performance on RK3399, whose inside is a 1.5 GHz 64-bit Quad-core ARM Cortex-A53. \#k means k threads. 103 | 104 | | Models | TRT-INT8(\#1) | EQ-INT7(\#1) | TRT-INT8(\#4) | EQ-INT7(\#4) | 105 | | :----: | :-----------: | :----------: | :-----------: | :----------: | 106 | | SqueezeNetV1.1 | 180 | **120** | 66 | **44** | 107 | | MobileNetV1 | 234 | **189** | 65 | **57** | 108 | | VGG16 | 3326 | **2873** | 1423 | **1252** | 109 | | ResNet50 | 1264 | **993** | 415 | **300** | 110 | 111 | #### 2. The latency (ms) performance on RK3399, which inside is a 1.8 GHz 64-bit Dual-core ARM Cortex-A72. \#k means k threads. 112 | | Models | TRT-INT8(\#1) | EQ-INT7(\#1) | TRT-INT8(\#2) | EQ-INT7(\#2) | 113 | | :----: | :-----------: | :----------: | :-----------: | :----------: | 114 | | SqueezeNetV1.1 | 79 | **57** | 54 | **37** | 115 | | MobileNetV1 | 105 | **84** | 56 | **46** | 116 | | VGG16 | 1659 | **1385** | 1034 | **849** | 117 | | ResNet50 | 559 | **463** | 338 | **262** | 118 | ## Contributing 119 | 120 | PRs accepted. 121 | 122 | ## License and Citation 123 | 124 | BSD3 © DeepGlint 125 | ``` 126 | @inproceedings{easyquant, 127 | title={EasyQuant: Post-training Quantization via Scale Optimization}, 128 | author={Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Debing Zhang, Ying Fu}, 129 | year={2020} 130 | } 131 | ``` 132 | -------------------------------------------------------------------------------- /data/calib.txt: -------------------------------------------------------------------------------- 1 | n01440764_11652.JPEG 2 | n01440764_490.JPEG 3 | n01440764_9929.JPEG 4 | n01443537_16652.JPEG 5 | n01443537_17231.JPEG 6 | n01443537_3645.JPEG 7 | n01484850_11174.JPEG 8 | n01484850_20658.JPEG 9 | n01484850_3533.JPEG 10 | n01491361_2343.JPEG 11 | n01491361_6464.JPEG 12 | n01491361_9400.JPEG 13 | n01494475_16238.JPEG 14 | n01494475_24635.JPEG 15 | n01494475_902.JPEG 16 | n01496331_12448.JPEG 17 | n01496331_30855.JPEG 18 | n01496331_3115.JPEG 19 | n01498041_11621.JPEG 20 | n01498041_13389.JPEG 21 | n01498041_8215.JPEG 22 | n01514668_13306.JPEG 23 | n01514668_18120.JPEG 24 | n01514668_22604.JPEG 25 | n01514859_10560.JPEG 26 | n01514859_3162.JPEG 27 | n01514859_658.JPEG 28 | n01518878_1010.JPEG 29 | n01518878_15882.JPEG 30 | n01518878_3294.JPEG 31 | n01530575_1395.JPEG 32 | n01530575_5879.JPEG 33 | n01530575_8253.JPEG 34 | n01531178_1043.JPEG 35 | n01531178_14534.JPEG 36 | n01531178_5019.JPEG 37 | n01532829_17905.JPEG 38 | n01532829_21122.JPEG 39 | n01532829_5736.JPEG 40 | n01534433_13443.JPEG 41 | n01534433_31597.JPEG 42 | n01534433_925.JPEG 43 | n01537544_209.JPEG 44 | n01537544_4328.JPEG 45 | n01537544_861.JPEG 46 | n01558993_11007.JPEG 47 | n01558993_2954.JPEG 48 | n01558993_5586.JPEG 49 | n01560419_16484.JPEG 50 | n01560419_2447.JPEG 51 | n01560419_6431.JPEG 52 | n01580077_10569.JPEG 53 | n01580077_3498.JPEG 54 | n01580077_5633.JPEG 55 | n01582220_11409.JPEG 56 | n01582220_11503.JPEG 57 | n01582220_4942.JPEG 58 | n01592084_3006.JPEG 59 | n01592084_3677.JPEG 60 | n01592084_8563.JPEG 61 | n01601694_10160.JPEG 62 | n01601694_15606.JPEG 63 | n01601694_8231.JPEG 64 | n01608432_3145.JPEG 65 | n01608432_45765.JPEG 66 | n01608432_8207.JPEG 67 | n01614925_38010.JPEG 68 | n01614925_551.JPEG 69 | n01614925_5742.JPEG 70 | n01616318_10973.JPEG 71 | n01616318_1867.JPEG 72 | n01616318_5775.JPEG 73 | n01622779_6023.JPEG 74 | n01622779_8227.JPEG 75 | n01622779_950.JPEG 76 | n01629819_18647.JPEG 77 | n01629819_6479.JPEG 78 | n01629819_7557.JPEG 79 | n01630670_5281.JPEG 80 | n01630670_6290.JPEG 81 | n01630670_6475.JPEG 82 | n01631663_2528.JPEG 83 | n01631663_5339.JPEG 84 | n01631663_8275.JPEG 85 | n01632458_1196.JPEG 86 | n01632458_5721.JPEG 87 | n01632458_6070.JPEG 88 | n01632777_14469.JPEG 89 | n01632777_15476.JPEG 90 | n01632777_20240.JPEG 91 | n01641577_12194.JPEG 92 | n01641577_12300.JPEG 93 | n01641577_1963.JPEG 94 | n01644373_1466.JPEG 95 | n01644373_2644.JPEG 96 | n01644373_5278.JPEG 97 | n01644900_13830.JPEG 98 | n01644900_19601.JPEG 99 | n01644900_931.JPEG 100 | n01664065_10088.JPEG 101 | n01664065_239.JPEG 102 | n01664065_8624.JPEG 103 | n01665541_11100.JPEG 104 | n01665541_13446.JPEG 105 | n01665541_1958.JPEG 106 | n01667114_2286.JPEG 107 | n01667114_3424.JPEG 108 | n01667114_934.JPEG 109 | n01667778_1516.JPEG 110 | n01667778_5486.JPEG 111 | n01667778_8931.JPEG 112 | n01669191_12283.JPEG 113 | n01669191_1365.JPEG 114 | n01669191_685.JPEG 115 | n01675722_1931.JPEG 116 | n01675722_2294.JPEG 117 | n01675722_5447.JPEG 118 | n01677366_14621.JPEG 119 | n01677366_16217.JPEG 120 | n01677366_40615.JPEG 121 | n01682714_10604.JPEG 122 | n01682714_14273.JPEG 123 | n01682714_356.JPEG 124 | n01685808_4839.JPEG 125 | n01685808_5668.JPEG 126 | n01685808_681.JPEG 127 | n01687978_13703.JPEG 128 | n01687978_656.JPEG 129 | n01687978_6763.JPEG 130 | n01688243_2238.JPEG 131 | n01688243_256.JPEG 132 | n01688243_458.JPEG 133 | n01689811_10967.JPEG 134 | n01689811_2575.JPEG 135 | n01689811_9870.JPEG 136 | n01692333_10873.JPEG 137 | n01692333_5568.JPEG 138 | n01692333_6245.JPEG 139 | n01693334_10047.JPEG 140 | n01693334_13005.JPEG 141 | n01693334_13796.JPEG 142 | n01694178_3789.JPEG 143 | n01694178_6088.JPEG 144 | n01694178_9698.JPEG 145 | n01695060_16811.JPEG 146 | n01695060_26010.JPEG 147 | n01695060_34344.JPEG 148 | n01697457_2206.JPEG 149 | n01697457_6386.JPEG 150 | n01697457_8263.JPEG 151 | n01698640_5547.JPEG 152 | n01698640_7594.JPEG 153 | n01698640_7692.JPEG 154 | n01704323_4947.JPEG 155 | n01704323_635.JPEG 156 | n01704323_684.JPEG 157 | n01728572_139.JPEG 158 | n01728572_15256.JPEG 159 | n01728572_3290.JPEG 160 | n01728920_14271.JPEG 161 | n01728920_1523.JPEG 162 | n01728920_2418.JPEG 163 | n01729322_13584.JPEG 164 | n01729322_173.JPEG 165 | n01729322_7786.JPEG 166 | n01729977_10434.JPEG 167 | n01729977_3424.JPEG 168 | n01729977_7967.JPEG 169 | n01734418_11151.JPEG 170 | n01734418_13064.JPEG 171 | n01734418_1434.JPEG 172 | n01735189_12893.JPEG 173 | n01735189_2977.JPEG 174 | n01735189_3987.JPEG 175 | n01737021_1112.JPEG 176 | n01737021_12591.JPEG 177 | n01737021_227.JPEG 178 | n01739381_2398.JPEG 179 | n01739381_290.JPEG 180 | n01739381_4047.JPEG 181 | n01740131_14657.JPEG 182 | n01740131_15700.JPEG 183 | n01740131_8938.JPEG 184 | n01742172_10903.JPEG 185 | n01742172_20878.JPEG 186 | n01742172_2159.JPEG 187 | n01744401_11966.JPEG 188 | n01744401_14745.JPEG 189 | n01744401_4086.JPEG 190 | n01748264_12012.JPEG 191 | n01748264_17162.JPEG 192 | n01748264_21591.JPEG 193 | n01749939_3261.JPEG 194 | n01749939_5849.JPEG 195 | n01749939_8699.JPEG 196 | n01751748_11544.JPEG 197 | n01751748_13214.JPEG 198 | n01751748_6471.JPEG 199 | n01753488_14295.JPEG 200 | n01753488_3334.JPEG 201 | n01753488_7383.JPEG 202 | n01755581_1441.JPEG 203 | n01755581_5246.JPEG 204 | n01755581_9229.JPEG 205 | n01756291_13003.JPEG 206 | n01756291_24291.JPEG 207 | n01756291_7947.JPEG 208 | n01768244_3676.JPEG 209 | n01768244_3897.JPEG 210 | n01768244_460.JPEG 211 | n01770081_14727.JPEG 212 | n01770081_2005.JPEG 213 | n01770081_7269.JPEG 214 | n01770393_12995.JPEG 215 | n01770393_14926.JPEG 216 | n01770393_6241.JPEG 217 | n01773157_1441.JPEG 218 | n01773157_436.JPEG 219 | n01773157_9970.JPEG 220 | n01773549_1018.JPEG 221 | n01773549_3088.JPEG 222 | n01773549_5251.JPEG 223 | n01773797_1357.JPEG 224 | n01773797_5054.JPEG 225 | n01773797_6487.JPEG 226 | n01774384_13098.JPEG 227 | n01774384_142.JPEG 228 | n01774384_1456.JPEG 229 | n01774750_5319.JPEG 230 | n01774750_7553.JPEG 231 | n01774750_9390.JPEG 232 | n01775062_3190.JPEG 233 | n01775062_3636.JPEG 234 | n01775062_4751.JPEG 235 | n01776313_18095.JPEG 236 | n01776313_2223.JPEG 237 | n01776313_25661.JPEG 238 | n01784675_15984.JPEG 239 | n01784675_23232.JPEG 240 | n01784675_3440.JPEG 241 | n01795545_2149.JPEG 242 | n01795545_4868.JPEG 243 | n01795545_6892.JPEG 244 | n01796340_13033.JPEG 245 | n01796340_26194.JPEG 246 | n01796340_2790.JPEG 247 | n01797886_11482.JPEG 248 | n01797886_1395.JPEG 249 | n01797886_21178.JPEG 250 | n01798484_13514.JPEG 251 | n01798484_3820.JPEG 252 | n01798484_8267.JPEG 253 | n01806143_1250.JPEG 254 | n01806143_14008.JPEG 255 | n01806143_7699.JPEG 256 | n01806567_10549.JPEG 257 | n01806567_14331.JPEG 258 | n01806567_8177.JPEG 259 | n01807496_11945.JPEG 260 | n01807496_5064.JPEG 261 | n01807496_7760.JPEG 262 | n01817953_1285.JPEG 263 | n01817953_3726.JPEG 264 | n01817953_8652.JPEG 265 | n01818515_1287.JPEG 266 | n01818515_14338.JPEG 267 | n01818515_23394.JPEG 268 | n01819313_13191.JPEG 269 | n01819313_14030.JPEG 270 | n01819313_4536.JPEG 271 | n01820546_4637.JPEG 272 | n01820546_6681.JPEG 273 | n01820546_6979.JPEG 274 | n01824575_548.JPEG 275 | n01824575_5930.JPEG 276 | n01824575_9167.JPEG 277 | n01828970_10788.JPEG 278 | n01828970_2438.JPEG 279 | n01828970_5842.JPEG 280 | n01829413_2603.JPEG 281 | n01829413_2654.JPEG 282 | n01829413_7402.JPEG 283 | n01833805_10797.JPEG 284 | n01833805_22826.JPEG 285 | n01833805_5975.JPEG 286 | n01843065_1427.JPEG 287 | n01843065_2018.JPEG 288 | n01843065_2496.JPEG 289 | n01843383_2048.JPEG 290 | n01843383_2860.JPEG 291 | n01843383_6218.JPEG 292 | n01847000_12191.JPEG 293 | n01847000_14982.JPEG 294 | n01847000_5263.JPEG 295 | n01855032_2445.JPEG 296 | n01855032_2510.JPEG 297 | n01855032_2838.JPEG 298 | n01855672_13294.JPEG 299 | n01855672_8896.JPEG 300 | n01855672_9425.JPEG 301 | n01860187_1819.JPEG 302 | n01860187_1848.JPEG 303 | n01860187_8368.JPEG 304 | n01871265_3832.JPEG 305 | n01871265_3925.JPEG 306 | n01871265_7225.JPEG 307 | n01872401_2045.JPEG 308 | n01872401_3592.JPEG 309 | n01872401_7806.JPEG 310 | n01873310_10088.JPEG 311 | n01873310_56073.JPEG 312 | n01873310_7798.JPEG 313 | n01877812_10973.JPEG 314 | n01877812_1515.JPEG 315 | n01877812_26132.JPEG 316 | n01882714_10498.JPEG 317 | n01882714_11289.JPEG 318 | n01882714_8076.JPEG 319 | n01883070_14189.JPEG 320 | n01883070_1871.JPEG 321 | n01883070_26842.JPEG 322 | n01910747_14844.JPEG 323 | n01910747_2025.JPEG 324 | n01910747_3289.JPEG 325 | n01914609_6600.JPEG 326 | n01914609_6975.JPEG 327 | n01914609_8208.JPEG 328 | n01917289_1557.JPEG 329 | n01917289_3507.JPEG 330 | n01917289_4830.JPEG 331 | n01924916_14341.JPEG 332 | n01924916_2823.JPEG 333 | n01924916_7909.JPEG 334 | n01930112_14880.JPEG 335 | n01930112_6360.JPEG 336 | n01930112_6608.JPEG 337 | n01943899_22992.JPEG 338 | n01943899_2595.JPEG 339 | n01943899_7081.JPEG 340 | n01944390_17299.JPEG 341 | n01944390_17422.JPEG 342 | n01944390_8040.JPEG 343 | n01945685_10091.JPEG 344 | n01945685_1452.JPEG 345 | n01945685_2884.JPEG 346 | n01950731_14149.JPEG 347 | n01950731_22621.JPEG 348 | n01950731_5553.JPEG 349 | n01955084_8319.JPEG 350 | n01955084_8821.JPEG 351 | n01955084_9239.JPEG 352 | n01968897_11727.JPEG 353 | n01968897_17387.JPEG 354 | n01968897_5834.JPEG 355 | n01978287_2369.JPEG 356 | n01978287_3248.JPEG 357 | n01978287_3604.JPEG 358 | n01978455_239.JPEG 359 | n01978455_5999.JPEG 360 | n01978455_8185.JPEG 361 | n01980166_3083.JPEG 362 | n01980166_5166.JPEG 363 | n01980166_8364.JPEG 364 | n01981276_20276.JPEG 365 | n01981276_2341.JPEG 366 | n01981276_284.JPEG 367 | n01983481_12382.JPEG 368 | n01983481_15542.JPEG 369 | n01983481_421.JPEG 370 | n01984695_16008.JPEG 371 | n01984695_42223.JPEG 372 | n01984695_45009.JPEG 373 | n01985128_26026.JPEG 374 | n01985128_26546.JPEG 375 | n01985128_8456.JPEG 376 | n01986214_15915.JPEG 377 | n01986214_20179.JPEG 378 | n01986214_8426.JPEG 379 | n01990800_1710.JPEG 380 | n01990800_202.JPEG 381 | n01990800_3615.JPEG 382 | n02002556_1086.JPEG 383 | n02002556_15443.JPEG 384 | n02002556_9778.JPEG 385 | n02002724_141.JPEG 386 | n02002724_3844.JPEG 387 | n02002724_750.JPEG 388 | n02006656_4274.JPEG 389 | n02006656_5834.JPEG 390 | n02006656_7462.JPEG 391 | n02007558_10025.JPEG 392 | n02007558_10199.JPEG 393 | n02007558_20724.JPEG 394 | n02009229_2412.JPEG 395 | n02009229_4375.JPEG 396 | n02009229_6881.JPEG 397 | n02009912_10231.JPEG 398 | n02009912_37210.JPEG 399 | n02009912_7430.JPEG 400 | n02011460_1106.JPEG 401 | n02011460_1642.JPEG 402 | n02011460_2807.JPEG 403 | n02012849_106.JPEG 404 | n02012849_2960.JPEG 405 | n02012849_4455.JPEG 406 | n02013706_16807.JPEG 407 | n02013706_17388.JPEG 408 | n02013706_454.JPEG 409 | n02017213_11589.JPEG 410 | n02017213_5154.JPEG 411 | n02017213_9755.JPEG 412 | n02018207_11518.JPEG 413 | n02018207_1721.JPEG 414 | n02018207_8707.JPEG 415 | n02018795_17315.JPEG 416 | n02018795_5265.JPEG 417 | n02018795_6012.JPEG 418 | n02025239_1227.JPEG 419 | n02025239_323.JPEG 420 | n02025239_677.JPEG 421 | n02027492_10180.JPEG 422 | n02027492_1374.JPEG 423 | n02027492_3007.JPEG 424 | n02028035_18615.JPEG 425 | n02028035_7020.JPEG 426 | n02028035_7447.JPEG 427 | n02033041_1316.JPEG 428 | n02033041_362.JPEG 429 | n02033041_4531.JPEG 430 | n02037110_10147.JPEG 431 | n02037110_32120.JPEG 432 | n02037110_4430.JPEG 433 | n02051845_13495.JPEG 434 | n02051845_35164.JPEG 435 | n02051845_3974.JPEG 436 | n02056570_8526.JPEG 437 | n02056570_8531.JPEG 438 | n02056570_9541.JPEG 439 | n02058221_17044.JPEG 440 | n02058221_2293.JPEG 441 | n02058221_411.JPEG 442 | n02066245_7389.JPEG 443 | n02066245_8259.JPEG 444 | n02066245_8772.JPEG 445 | n02071294_29833.JPEG 446 | n02071294_42684.JPEG 447 | n02071294_43651.JPEG 448 | n02074367_2287.JPEG 449 | n02074367_502.JPEG 450 | n02074367_920.JPEG 451 | n02077923_11868.JPEG 452 | n02077923_347.JPEG 453 | n02077923_981.JPEG 454 | n02085620_1729.JPEG 455 | n02085620_3844.JPEG 456 | n02085620_6659.JPEG 457 | n02085782_1835.JPEG 458 | n02085782_2537.JPEG 459 | n02085782_7557.JPEG 460 | n02085936_2636.JPEG 461 | n02085936_4590.JPEG 462 | n02085936_4784.JPEG 463 | n02086079_17324.JPEG 464 | n02086079_3417.JPEG 465 | n02086079_41215.JPEG 466 | n02086240_574.JPEG 467 | n02086240_6898.JPEG 468 | n02086240_7474.JPEG 469 | n02086646_6052.JPEG 470 | n02086646_7498.JPEG 471 | n02086646_9025.JPEG 472 | n02086910_4075.JPEG 473 | n02086910_4726.JPEG 474 | n02086910_8603.JPEG 475 | n02087046_10404.JPEG 476 | n02087046_333.JPEG 477 | n02087046_4816.JPEG 478 | n02087394_14708.JPEG 479 | n02087394_2319.JPEG 480 | n02087394_9834.JPEG 481 | n02088094_1232.JPEG 482 | n02088094_544.JPEG 483 | n02088094_9895.JPEG 484 | n02088238_13373.JPEG 485 | n02088238_2073.JPEG 486 | n02088238_5635.JPEG 487 | n02088364_14157.JPEG 488 | n02088364_1492.JPEG 489 | n02088364_7747.JPEG 490 | n02088466_11599.JPEG 491 | n02088466_2787.JPEG 492 | n02088466_6901.JPEG 493 | n02088632_213.JPEG 494 | n02088632_416.JPEG 495 | n02088632_8001.JPEG 496 | n02089078_2471.JPEG 497 | n02089078_6079.JPEG 498 | n02089078_8149.JPEG 499 | n02089867_1348.JPEG 500 | n02089867_6737.JPEG 501 | n02089867_8806.JPEG 502 | n02089973_2698.JPEG 503 | n02089973_364.JPEG 504 | n02089973_639.JPEG 505 | n02090379_10808.JPEG 506 | n02090379_3721.JPEG 507 | n02090379_931.JPEG 508 | n02090622_1167.JPEG 509 | n02090622_2435.JPEG 510 | n02090622_24.JPEG 511 | n02090721_2548.JPEG 512 | n02090721_428.JPEG 513 | n02090721_6184.JPEG 514 | n02091032_1360.JPEG 515 | n02091032_378.JPEG 516 | n02091032_4100.JPEG 517 | n02091134_1100.JPEG 518 | n02091134_18664.JPEG 519 | n02091134_9729.JPEG 520 | n02091244_2688.JPEG 521 | n02091244_3087.JPEG 522 | n02091244_7601.JPEG 523 | n02091467_138.JPEG 524 | n02091467_3408.JPEG 525 | n02091467_4560.JPEG 526 | n02091635_5290.JPEG 527 | n02091635_5574.JPEG 528 | n02091635_699.JPEG 529 | n02091831_1592.JPEG 530 | n02091831_3093.JPEG 531 | n02091831_9062.JPEG 532 | n02092002_198.JPEG 533 | n02092002_5058.JPEG 534 | n02092002_929.JPEG 535 | n02092339_347.JPEG 536 | n02092339_6869.JPEG 537 | n02092339_7697.JPEG 538 | n02093256_1090.JPEG 539 | n02093256_13506.JPEG 540 | n02093256_555.JPEG 541 | n02093428_1248.JPEG 542 | n02093428_17998.JPEG 543 | n02093428_795.JPEG 544 | n02093647_2245.JPEG 545 | n02093647_4135.JPEG 546 | n02093647_4425.JPEG 547 | n02093754_1197.JPEG 548 | n02093754_6927.JPEG 549 | n02093754_7958.JPEG 550 | n02093859_2437.JPEG 551 | n02093859_4038.JPEG 552 | n02093859_5301.JPEG 553 | n02093991_1555.JPEG 554 | n02093991_4675.JPEG 555 | n02093991_7672.JPEG 556 | n02094114_2923.JPEG 557 | n02094114_4590.JPEG 558 | n02094114_8131.JPEG 559 | n02094258_1124.JPEG 560 | n02094258_1281.JPEG 561 | n02094258_1719.JPEG 562 | n02094433_1770.JPEG 563 | n02094433_5114.JPEG 564 | n02094433_8057.JPEG 565 | n02095314_1950.JPEG 566 | n02095314_5543.JPEG 567 | n02095314_6325.JPEG 568 | n02095570_6169.JPEG 569 | n02095570_7225.JPEG 570 | n02095570_9169.JPEG 571 | n02095889_1412.JPEG 572 | n02095889_14321.JPEG 573 | n02095889_4676.JPEG 574 | n02096051_4966.JPEG 575 | n02096051_6510.JPEG 576 | n02096051_8422.JPEG 577 | n02096177_4914.JPEG 578 | n02096177_6298.JPEG 579 | n02096177_9490.JPEG 580 | n02096294_2857.JPEG 581 | n02096294_4278.JPEG 582 | n02096294_8699.JPEG 583 | n02096437_2093.JPEG 584 | n02096437_3507.JPEG 585 | n02096437_725.JPEG 586 | n02096585_1056.JPEG 587 | n02096585_1285.JPEG 588 | n02096585_9171.JPEG 589 | n02097047_2289.JPEG 590 | n02097047_3088.JPEG 591 | n02097047_4563.JPEG 592 | n02097130_2969.JPEG 593 | n02097130_5455.JPEG 594 | n02097130_603.JPEG 595 | n02097209_1136.JPEG 596 | n02097209_3045.JPEG 597 | n02097209_5964.JPEG 598 | n02097298_10984.JPEG 599 | n02097298_2215.JPEG 600 | n02097298_4752.JPEG 601 | n02097474_1744.JPEG 602 | n02097474_3478.JPEG 603 | n02097474_4228.JPEG 604 | n02097658_1469.JPEG 605 | n02097658_4987.JPEG 606 | n02097658_7351.JPEG 607 | n02098105_118.JPEG 608 | n02098105_3531.JPEG 609 | n02098105_6024.JPEG 610 | n02098286_1160.JPEG 611 | n02098286_5006.JPEG 612 | n02098286_5735.JPEG 613 | n02098413_2569.JPEG 614 | n02098413_369.JPEG 615 | n02098413_638.JPEG 616 | n02099267_16449.JPEG 617 | n02099267_3472.JPEG 618 | n02099267_3934.JPEG 619 | n02099429_398.JPEG 620 | n02099429_4266.JPEG 621 | n02099429_5293.JPEG 622 | n02099601_10612.JPEG 623 | n02099601_2169.JPEG 624 | n02099601_2783.JPEG 625 | n02099712_2101.JPEG 626 | n02099712_218.JPEG 627 | n02099712_2400.JPEG 628 | n02099849_5948.JPEG 629 | n02099849_7642.JPEG 630 | n02099849_7727.JPEG 631 | n02100236_6168.JPEG 632 | n02100236_7637.JPEG 633 | n02100236_8057.JPEG 634 | n02100583_12253.JPEG 635 | n02100583_13445.JPEG 636 | n02100583_5452.JPEG 637 | n02100735_5358.JPEG 638 | n02100735_5919.JPEG 639 | n02100735_5988.JPEG 640 | n02100877_12577.JPEG 641 | n02100877_4699.JPEG 642 | n02100877_5963.JPEG 643 | n02101006_11240.JPEG 644 | n02101006_5356.JPEG 645 | n02101006_8355.JPEG 646 | n02101388_20966.JPEG 647 | n02101388_3576.JPEG 648 | n02101388_5756.JPEG 649 | n02101556_12321.JPEG 650 | n02101556_15218.JPEG 651 | n02101556_2360.JPEG 652 | n02102040_1076.JPEG 653 | n02102040_3162.JPEG 654 | n02102040_6298.JPEG 655 | n02102177_2511.JPEG 656 | n02102177_3100.JPEG 657 | n02102177_3289.JPEG 658 | n02102318_1716.JPEG 659 | n02102318_4668.JPEG 660 | n02102318_7870.JPEG 661 | n02102480_2182.JPEG 662 | n02102480_2419.JPEG 663 | n02102480_346.JPEG 664 | n02102973_1445.JPEG 665 | n02102973_5838.JPEG 666 | n02102973_6551.JPEG 667 | n02104029_7246.JPEG 668 | n02104029_8154.JPEG 669 | n02104029_831.JPEG 670 | n02104365_10794.JPEG 671 | n02104365_1721.JPEG 672 | n02104365_9639.JPEG 673 | n02105056_11870.JPEG 674 | n02105056_15670.JPEG 675 | n02105056_5436.JPEG 676 | n02105162_3235.JPEG 677 | n02105162_5462.JPEG 678 | n02105162_9951.JPEG 679 | n02105251_17922.JPEG 680 | n02105251_1900.JPEG 681 | n02105251_5911.JPEG 682 | n02105412_3483.JPEG 683 | n02105412_4432.JPEG 684 | n02105412_476.JPEG 685 | n02105505_3967.JPEG 686 | n02105505_4114.JPEG 687 | n02105505_6557.JPEG 688 | n02105641_4321.JPEG 689 | n02105641_6071.JPEG 690 | n02105641_7193.JPEG 691 | n02105855_15760.JPEG 692 | n02105855_19431.JPEG 693 | n02105855_9099.JPEG 694 | n02106030_17578.JPEG 695 | n02106030_2937.JPEG 696 | n02106030_4790.JPEG 697 | n02106166_14823.JPEG 698 | n02106166_17037.JPEG 699 | n02106166_182.JPEG 700 | n02106382_6092.JPEG 701 | n02106382_6596.JPEG 702 | n02106382_7455.JPEG 703 | n02106550_12151.JPEG 704 | n02106550_4492.JPEG 705 | n02106550_5063.JPEG 706 | n02106662_10761.JPEG 707 | n02106662_36926.JPEG 708 | n02106662_5308.JPEG 709 | n02107142_11838.JPEG 710 | n02107142_46943.JPEG 711 | n02107142_53128.JPEG 712 | n02107312_2509.JPEG 713 | n02107312_4015.JPEG 714 | n02107312_4363.JPEG 715 | n02107574_1820.JPEG 716 | n02107574_4945.JPEG 717 | n02107574_4963.JPEG 718 | n02107683_3357.JPEG 719 | n02107683_355.JPEG 720 | n02107683_3560.JPEG 721 | n02107908_10748.JPEG 722 | n02107908_11618.JPEG 723 | n02107908_5821.JPEG 724 | n02108000_2212.JPEG 725 | n02108000_4071.JPEG 726 | n02108000_661.JPEG 727 | n02108089_11271.JPEG 728 | n02108089_46330.JPEG 729 | n02108089_727.JPEG 730 | n02108422_12370.JPEG 731 | n02108422_3576.JPEG 732 | n02108422_802.JPEG 733 | n02108551_11956.JPEG 734 | n02108551_502.JPEG 735 | n02108551_5560.JPEG 736 | n02108915_3464.JPEG 737 | n02108915_4878.JPEG 738 | n02108915_6177.JPEG 739 | n02109047_1449.JPEG 740 | n02109047_18614.JPEG 741 | n02109047_2851.JPEG 742 | n02109525_3855.JPEG 743 | n02109525_69456.JPEG 744 | n02109525_74755.JPEG 745 | n02109961_21790.JPEG 746 | n02109961_37796.JPEG 747 | n02109961_43564.JPEG 748 | n02110063_18068.JPEG 749 | n02110063_6276.JPEG 750 | n02110063_8947.JPEG 751 | n02110185_14437.JPEG 752 | n02110185_14738.JPEG 753 | n02110185_9481.JPEG 754 | n02110341_19310.JPEG 755 | n02110341_5667.JPEG 756 | n02110341_6256.JPEG 757 | n02110627_13453.JPEG 758 | n02110627_3246.JPEG 759 | n02110627_7323.JPEG 760 | n02110806_11654.JPEG 761 | n02110806_1347.JPEG 762 | n02110806_23653.JPEG 763 | n02110958_12860.JPEG 764 | n02110958_16452.JPEG 765 | n02110958_724.JPEG 766 | n02111129_1094.JPEG 767 | n02111129_6772.JPEG 768 | n02111129_8044.JPEG 769 | n02111277_21026.JPEG 770 | n02111277_3653.JPEG 771 | n02111277_4520.JPEG 772 | n02111500_1850.JPEG 773 | n02111500_3898.JPEG 774 | n02111500_8880.JPEG 775 | n02111889_10472.JPEG 776 | n02111889_2119.JPEG 777 | n02111889_9153.JPEG 778 | n02112018_11787.JPEG 779 | n02112018_12199.JPEG 780 | n02112018_5888.JPEG 781 | n02112137_10325.JPEG 782 | n02112137_16511.JPEG 783 | n02112137_4072.JPEG 784 | n02112350_1376.JPEG 785 | n02112350_4332.JPEG 786 | n02112350_9778.JPEG 787 | n02112706_2070.JPEG 788 | n02112706_2927.JPEG 789 | n02112706_4376.JPEG 790 | n02113023_6444.JPEG 791 | n02113023_6624.JPEG 792 | n02113023_994.JPEG 793 | n02113186_11741.JPEG 794 | n02113186_6741.JPEG 795 | n02113186_987.JPEG 796 | n02113624_1929.JPEG 797 | n02113624_3103.JPEG 798 | n02113624_3944.JPEG 799 | n02113712_18400.JPEG 800 | n02113712_2543.JPEG 801 | n02113712_87.JPEG 802 | n02113799_1552.JPEG 803 | n02113799_572.JPEG 804 | n02113799_721.JPEG 805 | n02113978_2399.JPEG 806 | n02113978_2977.JPEG 807 | n02113978_3123.JPEG 808 | n02114367_15502.JPEG 809 | n02114367_17849.JPEG 810 | n02114367_3962.JPEG 811 | n02114548_10590.JPEG 812 | n02114548_18591.JPEG 813 | n02114548_8962.JPEG 814 | n02114712_1835.JPEG 815 | n02114712_18839.JPEG 816 | n02114712_4294.JPEG 817 | n02114855_14794.JPEG 818 | n02114855_20790.JPEG 819 | n02114855_9859.JPEG 820 | n02115641_15186.JPEG 821 | n02115641_6940.JPEG 822 | n02115641_7387.JPEG 823 | n02115913_8259.JPEG 824 | n02115913_903.JPEG 825 | n02115913_9116.JPEG 826 | n02116738_15207.JPEG 827 | n02116738_22432.JPEG 828 | n02116738_7860.JPEG 829 | n02117135_11835.JPEG 830 | n02117135_5458.JPEG 831 | n02117135_9146.JPEG 832 | n02119022_13634.JPEG 833 | n02119022_1876.JPEG 834 | n02119022_4224.JPEG 835 | n02119789_1012.JPEG 836 | n02119789_15866.JPEG 837 | n02119789_6109.JPEG 838 | n02120079_10698.JPEG 839 | n02120079_11173.JPEG 840 | n02120079_13677.JPEG 841 | n02120505_18456.JPEG 842 | n02120505_3402.JPEG 843 | n02120505_7858.JPEG 844 | n02123045_12635.JPEG 845 | n02123045_13192.JPEG 846 | n02123045_627.JPEG 847 | n02123159_3289.JPEG 848 | n02123159_903.JPEG 849 | n02123159_9697.JPEG 850 | n02123394_2053.JPEG 851 | n02123394_383.JPEG 852 | n02123394_8848.JPEG 853 | n02123597_3686.JPEG 854 | n02123597_7373.JPEG 855 | n02123597_9505.JPEG 856 | n02124075_12672.JPEG 857 | n02124075_208.JPEG 858 | n02124075_5424.JPEG 859 | n02125311_18341.JPEG 860 | n02125311_43096.JPEG 861 | n02125311_46231.JPEG 862 | n02127052_10529.JPEG 863 | n02127052_10546.JPEG 864 | n02127052_4625.JPEG 865 | n02128385_1015.JPEG 866 | n02128385_10710.JPEG 867 | n02128385_15941.JPEG 868 | n02128757_11216.JPEG 869 | n02128757_2305.JPEG 870 | n02128757_3448.JPEG 871 | n02128925_40688.JPEG 872 | n02128925_66204.JPEG 873 | n02128925_9223.JPEG 874 | n02129165_12418.JPEG 875 | n02129165_5769.JPEG 876 | n02129165_8350.JPEG 877 | n02129604_14163.JPEG 878 | n02129604_19875.JPEG 879 | n02129604_9700.JPEG 880 | n02130308_2182.JPEG 881 | n02130308_3684.JPEG 882 | n02130308_8733.JPEG 883 | n02132136_20369.JPEG 884 | n02132136_21402.JPEG 885 | n02132136_2331.JPEG 886 | n02133161_1807.JPEG 887 | n02133161_2566.JPEG 888 | n02133161_6316.JPEG 889 | n02134084_27458.JPEG 890 | n02134084_5226.JPEG 891 | n02134084_6887.JPEG 892 | n02134418_1860.JPEG 893 | n02134418_434.JPEG 894 | n02134418_4959.JPEG 895 | n02137549_10277.JPEG 896 | n02137549_8630.JPEG 897 | n02137549_968.JPEG 898 | n02138441_10137.JPEG 899 | n02138441_2519.JPEG 900 | n02138441_6856.JPEG 901 | n02165105_2413.JPEG 902 | n02165105_3106.JPEG 903 | n02165105_5337.JPEG 904 | n02165456_2242.JPEG 905 | n02165456_4769.JPEG 906 | n02165456_645.JPEG 907 | n02167151_1318.JPEG 908 | n02167151_295.JPEG 909 | n02167151_3790.JPEG 910 | n02168699_5671.JPEG 911 | n02168699_5864.JPEG 912 | n02168699_6579.JPEG 913 | n02169497_2484.JPEG 914 | n02169497_250.JPEG 915 | n02169497_2843.JPEG 916 | n02172182_2625.JPEG 917 | n02172182_605.JPEG 918 | n02172182_7455.JPEG 919 | n02174001_405.JPEG 920 | n02174001_7941.JPEG 921 | n02174001_901.JPEG 922 | n02177972_12534.JPEG 923 | n02177972_1736.JPEG 924 | n02177972_5608.JPEG 925 | n02190166_11739.JPEG 926 | n02190166_3342.JPEG 927 | n02190166_66.JPEG 928 | n02206856_1406.JPEG 929 | n02206856_5018.JPEG 930 | n02206856_7149.JPEG 931 | n02219486_23567.JPEG 932 | n02219486_23894.JPEG 933 | n02219486_438.JPEG 934 | n02226429_12510.JPEG 935 | n02226429_22026.JPEG 936 | n02226429_6924.JPEG 937 | n02229544_1751.JPEG 938 | n02229544_2648.JPEG 939 | n02229544_4920.JPEG 940 | n02231487_5057.JPEG 941 | n02231487_829.JPEG 942 | n02231487_9451.JPEG 943 | n02233338_15741.JPEG 944 | n02233338_16771.JPEG 945 | n02233338_8687.JPEG 946 | n02236044_13304.JPEG 947 | n02236044_17349.JPEG 948 | n02236044_1747.JPEG 949 | n02256656_16377.JPEG 950 | n02256656_3843.JPEG 951 | n02256656_629.JPEG 952 | n02259212_3638.JPEG 953 | n02259212_5530.JPEG 954 | n02259212_6780.JPEG 955 | n02264363_1072.JPEG 956 | n02264363_11956.JPEG 957 | n02264363_19826.JPEG 958 | n02268443_40139.JPEG 959 | n02268443_41007.JPEG 960 | n02268443_5666.JPEG 961 | n02268853_2385.JPEG 962 | n02268853_5301.JPEG 963 | n02268853_898.JPEG 964 | n02276258_1184.JPEG 965 | n02276258_36421.JPEG 966 | n02276258_7396.JPEG 967 | n02277742_20939.JPEG 968 | n02277742_3365.JPEG 969 | n02277742_4722.JPEG 970 | n02279972_22215.JPEG 971 | n02279972_25594.JPEG 972 | n02279972_654.JPEG 973 | n02280649_2519.JPEG 974 | n02280649_6967.JPEG 975 | n02280649_7234.JPEG 976 | n02281406_4718.JPEG 977 | n02281406_5997.JPEG 978 | n02281406_6119.JPEG 979 | n02281787_4280.JPEG 980 | n02281787_5505.JPEG 981 | n02281787_5790.JPEG 982 | n02317335_21062.JPEG 983 | n02317335_22015.JPEG 984 | n02317335_6253.JPEG 985 | n02319095_1145.JPEG 986 | n02319095_4508.JPEG 987 | n02319095_8564.JPEG 988 | n02321529_6330.JPEG 989 | n02321529_9137.JPEG 990 | n02321529_981.JPEG 991 | n02325366_4702.JPEG 992 | n02325366_5096.JPEG 993 | n02325366_8097.JPEG 994 | n02326432_25996.JPEG 995 | n02326432_8143.JPEG 996 | n02326432_9053.JPEG 997 | n02328150_13701.JPEG 998 | n02328150_15092.JPEG 999 | n02328150_1589.JPEG 1000 | n02342885_2527.JPEG 1001 | n02342885_522.JPEG 1002 | n02342885_888.JPEG 1003 | n02346627_10917.JPEG 1004 | n02346627_1228.JPEG 1005 | n02346627_8389.JPEG 1006 | n02356798_318.JPEG 1007 | n02356798_3233.JPEG 1008 | n02356798_3832.JPEG 1009 | n02361337_1184.JPEG 1010 | n02361337_7850.JPEG 1011 | n02361337_9502.JPEG 1012 | n02363005_16248.JPEG 1013 | n02363005_26297.JPEG 1014 | n02363005_5591.JPEG 1015 | n02364673_13242.JPEG 1016 | n02364673_395.JPEG 1017 | n02364673_4288.JPEG 1018 | n02389026_1079.JPEG 1019 | n02389026_18619.JPEG 1020 | n02389026_4849.JPEG 1021 | n02391049_5982.JPEG 1022 | n02391049_860.JPEG 1023 | n02391049_907.JPEG 1024 | n02395406_20306.JPEG 1025 | n02395406_32520.JPEG 1026 | n02395406_39805.JPEG 1027 | n02396427_10010.JPEG 1028 | n02396427_10767.JPEG 1029 | n02396427_15699.JPEG 1030 | n02397096_1111.JPEG 1031 | n02397096_1217.JPEG 1032 | n02397096_4992.JPEG 1033 | n02398521_16091.JPEG 1034 | n02398521_1823.JPEG 1035 | n02398521_8863.JPEG 1036 | n02403003_1410.JPEG 1037 | n02403003_2215.JPEG 1038 | n02403003_7207.JPEG 1039 | n02408429_240.JPEG 1040 | n02408429_8850.JPEG 1041 | n02408429_926.JPEG 1042 | n02410509_1017.JPEG 1043 | n02410509_8161.JPEG 1044 | n02410509_8210.JPEG 1045 | n02412080_13473.JPEG 1046 | n02412080_15004.JPEG 1047 | n02412080_19859.JPEG 1048 | n02415577_1666.JPEG 1049 | n02415577_3763.JPEG 1050 | n02415577_6990.JPEG 1051 | n02417914_4235.JPEG 1052 | n02417914_5887.JPEG 1053 | n02417914_6307.JPEG 1054 | n02422106_1480.JPEG 1055 | n02422106_157.JPEG 1056 | n02422106_4080.JPEG 1057 | n02422699_114.JPEG 1058 | n02422699_27002.JPEG 1059 | n02422699_686.JPEG 1060 | n02423022_1701.JPEG 1061 | n02423022_413.JPEG 1062 | n02423022_4755.JPEG 1063 | n02437312_1188.JPEG 1064 | n02437312_35483.JPEG 1065 | n02437312_7271.JPEG 1066 | n02437616_120.JPEG 1067 | n02437616_2004.JPEG 1068 | n02437616_5109.JPEG 1069 | n02441942_16068.JPEG 1070 | n02441942_7427.JPEG 1071 | n02441942_8442.JPEG 1072 | n02442845_19700.JPEG 1073 | n02442845_25714.JPEG 1074 | n02442845_3331.JPEG 1075 | n02443114_28399.JPEG 1076 | n02443114_3859.JPEG 1077 | n02443114_5161.JPEG 1078 | n02443484_12589.JPEG 1079 | n02443484_6460.JPEG 1080 | n02443484_9604.JPEG 1081 | n02444819_21834.JPEG 1082 | n02444819_6178.JPEG 1083 | n02444819_6674.JPEG 1084 | n02445715_13240.JPEG 1085 | n02445715_15693.JPEG 1086 | n02445715_8539.JPEG 1087 | n02447366_1435.JPEG 1088 | n02447366_2745.JPEG 1089 | n02447366_282.JPEG 1090 | n02454379_12357.JPEG 1091 | n02454379_12608.JPEG 1092 | n02454379_15849.JPEG 1093 | n02457408_16426.JPEG 1094 | n02457408_5260.JPEG 1095 | n02457408_7226.JPEG 1096 | n02480495_10836.JPEG 1097 | n02480495_2576.JPEG 1098 | n02480495_7611.JPEG 1099 | n02480855_11932.JPEG 1100 | n02480855_13720.JPEG 1101 | n02480855_20883.JPEG 1102 | n02481823_11374.JPEG 1103 | n02481823_3153.JPEG 1104 | n02481823_4360.JPEG 1105 | n02483362_17686.JPEG 1106 | n02483362_5650.JPEG 1107 | n02483362_9730.JPEG 1108 | n02483708_3453.JPEG 1109 | n02483708_5366.JPEG 1110 | n02483708_6905.JPEG 1111 | n02484975_1559.JPEG 1112 | n02484975_476.JPEG 1113 | n02484975_9546.JPEG 1114 | n02486261_14667.JPEG 1115 | n02486261_20403.JPEG 1116 | n02486261_5591.JPEG 1117 | n02486410_2355.JPEG 1118 | n02486410_4303.JPEG 1119 | n02486410_8821.JPEG 1120 | n02487347_20328.JPEG 1121 | n02487347_2174.JPEG 1122 | n02487347_2687.JPEG 1123 | n02488291_1456.JPEG 1124 | n02488291_2333.JPEG 1125 | n02488291_3540.JPEG 1126 | n02488702_195.JPEG 1127 | n02488702_5852.JPEG 1128 | n02488702_682.JPEG 1129 | n02489166_10466.JPEG 1130 | n02489166_2021.JPEG 1131 | n02489166_8274.JPEG 1132 | n02490219_5374.JPEG 1133 | n02490219_7767.JPEG 1134 | n02490219_844.JPEG 1135 | n02492035_11307.JPEG 1136 | n02492035_202.JPEG 1137 | n02492035_2656.JPEG 1138 | n02492660_1158.JPEG 1139 | n02492660_4925.JPEG 1140 | n02492660_8192.JPEG 1141 | n02493509_10578.JPEG 1142 | n02493509_12892.JPEG 1143 | n02493509_2136.JPEG 1144 | n02493793_10029.JPEG 1145 | n02493793_23015.JPEG 1146 | n02493793_6777.JPEG 1147 | n02494079_14399.JPEG 1148 | n02494079_1445.JPEG 1149 | n02494079_948.JPEG 1150 | n02497673_1112.JPEG 1151 | n02497673_1458.JPEG 1152 | n02497673_601.JPEG 1153 | n02500267_2924.JPEG 1154 | n02500267_3334.JPEG 1155 | n02500267_6903.JPEG 1156 | n02504013_6615.JPEG 1157 | n02504013_6999.JPEG 1158 | n02504013_8298.JPEG 1159 | n02504458_3728.JPEG 1160 | n02504458_3897.JPEG 1161 | n02504458_6999.JPEG 1162 | n02509815_13315.JPEG 1163 | n02509815_2614.JPEG 1164 | n02509815_6113.JPEG 1165 | n02510455_1096.JPEG 1166 | n02510455_27191.JPEG 1167 | n02510455_4362.JPEG 1168 | n02514041_3300.JPEG 1169 | n02514041_8000.JPEG 1170 | n02514041_9289.JPEG 1171 | n02526121_27046.JPEG 1172 | n02526121_6274.JPEG 1173 | n02526121_6842.JPEG 1174 | n02536864_25848.JPEG 1175 | n02536864_32765.JPEG 1176 | n02536864_5376.JPEG 1177 | n02606052_19583.JPEG 1178 | n02606052_20424.JPEG 1179 | n02606052_2133.JPEG 1180 | n02607072_3790.JPEG 1181 | n02607072_4183.JPEG 1182 | n02607072_9219.JPEG 1183 | n02640242_34173.JPEG 1184 | n02640242_37448.JPEG 1185 | n02640242_48127.JPEG 1186 | n02641379_10043.JPEG 1187 | n02641379_17548.JPEG 1188 | n02641379_8500.JPEG 1189 | n02643566_2138.JPEG 1190 | n02643566_2816.JPEG 1191 | n02643566_2959.JPEG 1192 | n02655020_12757.JPEG 1193 | n02655020_22877.JPEG 1194 | n02655020_32.JPEG 1195 | n02666196_13484.JPEG 1196 | n02666196_15347.JPEG 1197 | n02666196_7062.JPEG 1198 | n02667093_5733.JPEG 1199 | n02667093_623.JPEG 1200 | n02667093_9198.JPEG 1201 | n02669723_12175.JPEG 1202 | n02669723_5467.JPEG 1203 | n02669723_7122.JPEG 1204 | n02672831_1690.JPEG 1205 | n02672831_31333.JPEG 1206 | n02672831_3141.JPEG 1207 | n02676566_10175.JPEG 1208 | n02676566_4679.JPEG 1209 | n02676566_7102.JPEG 1210 | n02687172_28493.JPEG 1211 | n02687172_38911.JPEG 1212 | n02687172_90545.JPEG 1213 | n02690373_1161.JPEG 1214 | n02690373_13604.JPEG 1215 | n02690373_2865.JPEG 1216 | n02692877_10167.JPEG 1217 | n02692877_12884.JPEG 1218 | n02692877_13065.JPEG 1219 | n02699494_13248.JPEG 1220 | n02699494_14332.JPEG 1221 | n02699494_5673.JPEG 1222 | n02701002_10847.JPEG 1223 | n02701002_14340.JPEG 1224 | n02701002_9074.JPEG 1225 | n02704792_30010.JPEG 1226 | n02704792_32451.JPEG 1227 | n02704792_9160.JPEG 1228 | n02708093_1803.JPEG 1229 | n02708093_2030.JPEG 1230 | n02708093_6592.JPEG 1231 | n02727426_22478.JPEG 1232 | n02727426_23214.JPEG 1233 | n02727426_32869.JPEG 1234 | n02730930_2343.JPEG 1235 | n02730930_24483.JPEG 1236 | n02730930_7053.JPEG 1237 | n02747177_14219.JPEG 1238 | n02747177_16068.JPEG 1239 | n02747177_50740.JPEG 1240 | n02749479_5668.JPEG 1241 | n02749479_5880.JPEG 1242 | n02749479_928.JPEG 1243 | n02769748_32058.JPEG 1244 | n02769748_3727.JPEG 1245 | n02769748_40056.JPEG 1246 | n02776631_20771.JPEG 1247 | n02776631_30447.JPEG 1248 | n02776631_4294.JPEG 1249 | n02777292_17105.JPEG 1250 | n02777292_1753.JPEG 1251 | n02777292_4063.JPEG 1252 | n02782093_226.JPEG 1253 | n02782093_3574.JPEG 1254 | n02782093_983.JPEG 1255 | n02783161_31820.JPEG 1256 | n02783161_44224.JPEG 1257 | n02783161_9811.JPEG 1258 | n02786058_3230.JPEG 1259 | n02786058_8751.JPEG 1260 | n02786058_8943.JPEG 1261 | n02787622_15319.JPEG 1262 | n02787622_334.JPEG 1263 | n02787622_5570.JPEG 1264 | n02788148_25094.JPEG 1265 | n02788148_26828.JPEG 1266 | n02788148_94020.JPEG 1267 | n02790996_10952.JPEG 1268 | n02790996_1712.JPEG 1269 | n02790996_8212.JPEG 1270 | n02791124_1719.JPEG 1271 | n02791124_2954.JPEG 1272 | n02791124_4121.JPEG 1273 | n02791270_12409.JPEG 1274 | n02791270_21894.JPEG 1275 | n02791270_21995.JPEG 1276 | n02793495_10621.JPEG 1277 | n02793495_10965.JPEG 1278 | n02793495_8913.JPEG 1279 | n02794156_12031.JPEG 1280 | n02794156_13767.JPEG 1281 | n02794156_9582.JPEG 1282 | n02795169_11443.JPEG 1283 | n02795169_23212.JPEG 1284 | n02795169_7299.JPEG 1285 | n02797295_10194.JPEG 1286 | n02797295_1608.JPEG 1287 | n02797295_8526.JPEG 1288 | n02799071_23511.JPEG 1289 | n02799071_36345.JPEG 1290 | n02799071_7512.JPEG 1291 | n02802426_11271.JPEG 1292 | n02802426_13858.JPEG 1293 | n02802426_8031.JPEG 1294 | n02804414_2386.JPEG 1295 | n02804414_2968.JPEG 1296 | n02804414_6224.JPEG 1297 | n02804610_16469.JPEG 1298 | n02804610_1783.JPEG 1299 | n02804610_28053.JPEG 1300 | n02807133_16871.JPEG 1301 | n02807133_7461.JPEG 1302 | n02807133_8772.JPEG 1303 | n02808304_16677.JPEG 1304 | n02808304_7533.JPEG 1305 | n02808304_8698.JPEG 1306 | n02808440_17828.JPEG 1307 | n02808440_3389.JPEG 1308 | n02808440_48787.JPEG 1309 | n02814533_40471.JPEG 1310 | n02814533_81090.JPEG 1311 | n02814533_94804.JPEG 1312 | n02814860_10136.JPEG 1313 | n02814860_32132.JPEG 1314 | n02814860_34603.JPEG 1315 | n02815834_2112.JPEG 1316 | n02815834_2544.JPEG 1317 | n02815834_7809.JPEG 1318 | n02817516_16702.JPEG 1319 | n02817516_16758.JPEG 1320 | n02817516_23696.JPEG 1321 | n02823428_3003.JPEG 1322 | n02823428_5953.JPEG 1323 | n02823428_680.JPEG 1324 | n02823750_11547.JPEG 1325 | n02823750_15123.JPEG 1326 | n02823750_9367.JPEG 1327 | n02825657_1672.JPEG 1328 | n02825657_500.JPEG 1329 | n02825657_9234.JPEG 1330 | n02834397_18910.JPEG 1331 | n02834397_5722.JPEG 1332 | n02834397_9490.JPEG 1333 | n02835271_19907.JPEG 1334 | n02835271_2987.JPEG 1335 | n02835271_5908.JPEG 1336 | n02837789_12468.JPEG 1337 | n02837789_22643.JPEG 1338 | n02837789_913.JPEG 1339 | n02840245_10542.JPEG 1340 | n02840245_2336.JPEG 1341 | n02840245_3583.JPEG 1342 | n02841315_12598.JPEG 1343 | n02841315_31223.JPEG 1344 | n02841315_4089.JPEG 1345 | n02843684_15082.JPEG 1346 | n02843684_38904.JPEG 1347 | n02843684_9423.JPEG 1348 | n02859443_327.JPEG 1349 | n02859443_432.JPEG 1350 | n02859443_6807.JPEG 1351 | n02860847_12653.JPEG 1352 | n02860847_28876.JPEG 1353 | n02860847_8339.JPEG 1354 | n02865351_32695.JPEG 1355 | n02865351_3906.JPEG 1356 | n02865351_5929.JPEG 1357 | n02869837_10170.JPEG 1358 | n02869837_18069.JPEG 1359 | n02869837_3669.JPEG 1360 | n02870880_6566.JPEG 1361 | n02870880_7484.JPEG 1362 | n02870880_9669.JPEG 1363 | n02871525_22188.JPEG 1364 | n02871525_25274.JPEG 1365 | n02871525_3104.JPEG 1366 | n02877765_1690.JPEG 1367 | n02877765_3189.JPEG 1368 | n02877765_9641.JPEG 1369 | n02879718_1291.JPEG 1370 | n02879718_22375.JPEG 1371 | n02879718_24570.JPEG 1372 | n02883205_25226.JPEG 1373 | n02883205_2588.JPEG 1374 | n02883205_62884.JPEG 1375 | n02892201_30678.JPEG 1376 | n02892201_42316.JPEG 1377 | n02892201_53214.JPEG 1378 | n02892767_13894.JPEG 1379 | n02892767_18696.JPEG 1380 | n02892767_25762.JPEG 1381 | n02894605_15820.JPEG 1382 | n02894605_78567.JPEG 1383 | n02894605_79325.JPEG 1384 | n02895154_35263.JPEG 1385 | n02895154_38519.JPEG 1386 | n02895154_960.JPEG 1387 | n02906734_1281.JPEG 1388 | n02906734_18629.JPEG 1389 | n02906734_5537.JPEG 1390 | n02909870_2373.JPEG 1391 | n02909870_51840.JPEG 1392 | n02909870_7691.JPEG 1393 | n02910353_13117.JPEG 1394 | n02910353_2557.JPEG 1395 | n02910353_2955.JPEG 1396 | n02916936_137.JPEG 1397 | n02916936_2381.JPEG 1398 | n02916936_577.JPEG 1399 | n02917067_13607.JPEG 1400 | n02917067_16569.JPEG 1401 | n02917067_8537.JPEG 1402 | n02927161_10763.JPEG 1403 | n02927161_11674.JPEG 1404 | n02927161_15346.JPEG 1405 | n02930766_26211.JPEG 1406 | n02930766_30532.JPEG 1407 | n02930766_524.JPEG 1408 | n02939185_17984.JPEG 1409 | n02939185_6974.JPEG 1410 | n02939185_9576.JPEG 1411 | n02948072_11390.JPEG 1412 | n02948072_16217.JPEG 1413 | n02948072_33397.JPEG 1414 | n02950826_10475.JPEG 1415 | n02950826_3342.JPEG 1416 | n02950826_35052.JPEG 1417 | n02951358_11369.JPEG 1418 | n02951358_12027.JPEG 1419 | n02951358_4496.JPEG 1420 | n02951585_6236.JPEG 1421 | n02951585_7208.JPEG 1422 | n02951585_7293.JPEG 1423 | n02963159_4250.JPEG 1424 | n02963159_46575.JPEG 1425 | n02963159_5590.JPEG 1426 | n02965783_1839.JPEG 1427 | n02965783_3653.JPEG 1428 | n02965783_6871.JPEG 1429 | n02966193_16657.JPEG 1430 | n02966193_2618.JPEG 1431 | n02966193_41762.JPEG 1432 | n02966687_17407.JPEG 1433 | n02966687_1779.JPEG 1434 | n02966687_8627.JPEG 1435 | n02971356_14439.JPEG 1436 | n02971356_26666.JPEG 1437 | n02971356_34193.JPEG 1438 | n02974003_2929.JPEG 1439 | n02974003_4921.JPEG 1440 | n02974003_807.JPEG 1441 | n02977058_19962.JPEG 1442 | n02977058_22323.JPEG 1443 | n02977058_37350.JPEG 1444 | n02978881_15331.JPEG 1445 | n02978881_26869.JPEG 1446 | n02978881_8240.JPEG 1447 | n02979186_26456.JPEG 1448 | n02979186_27347.JPEG 1449 | n02979186_6056.JPEG 1450 | n02980441_2551.JPEG 1451 | n02980441_3837.JPEG 1452 | n02980441_8243.JPEG 1453 | n02981792_16843.JPEG 1454 | n02981792_22632.JPEG 1455 | n02981792_9529.JPEG 1456 | n02988304_2352.JPEG 1457 | n02988304_7165.JPEG 1458 | n02988304_765.JPEG 1459 | n02992211_43117.JPEG 1460 | n02992211_6874.JPEG 1461 | n02992211_9974.JPEG 1462 | n02992529_18876.JPEG 1463 | n02992529_49787.JPEG 1464 | n02992529_57172.JPEG 1465 | n02999410_10144.JPEG 1466 | n02999410_16665.JPEG 1467 | n02999410_55449.JPEG 1468 | n03000134_1648.JPEG 1469 | n03000134_4902.JPEG 1470 | n03000134_5078.JPEG 1471 | n03000247_24733.JPEG 1472 | n03000247_33355.JPEG 1473 | n03000247_59819.JPEG 1474 | n03000684_13343.JPEG 1475 | n03000684_3361.JPEG 1476 | n03000684_4401.JPEG 1477 | n03014705_14319.JPEG 1478 | n03014705_14997.JPEG 1479 | n03014705_16037.JPEG 1480 | n03016953_13641.JPEG 1481 | n03016953_31622.JPEG 1482 | n03016953_6419.JPEG 1483 | n03017168_14929.JPEG 1484 | n03017168_48526.JPEG 1485 | n03017168_77960.JPEG 1486 | n03018349_2277.JPEG 1487 | n03018349_2883.JPEG 1488 | n03018349_9661.JPEG 1489 | n03026506_1714.JPEG 1490 | n03026506_2829.JPEG 1491 | n03026506_5134.JPEG 1492 | n03028079_15102.JPEG 1493 | n03028079_16118.JPEG 1494 | n03028079_9094.JPEG 1495 | n03032252_15271.JPEG 1496 | n03032252_3311.JPEG 1497 | n03032252_42192.JPEG 1498 | n03041632_24286.JPEG 1499 | n03041632_5128.JPEG 1500 | n03041632_99875.JPEG 1501 | n03042490_1673.JPEG 1502 | n03042490_3910.JPEG 1503 | n03042490_794.JPEG 1504 | n03045698_2159.JPEG 1505 | n03045698_5106.JPEG 1506 | n03045698_7092.JPEG 1507 | n03047690_18490.JPEG 1508 | n03047690_6527.JPEG 1509 | n03047690_7405.JPEG 1510 | n03062245_15067.JPEG 1511 | n03062245_1827.JPEG 1512 | n03062245_3574.JPEG 1513 | n03063599_2823.JPEG 1514 | n03063599_3938.JPEG 1515 | n03063599_4173.JPEG 1516 | n03063689_13462.JPEG 1517 | n03063689_13757.JPEG 1518 | n03063689_21985.JPEG 1519 | n03065424_38850.JPEG 1520 | n03065424_7506.JPEG 1521 | n03065424_9632.JPEG 1522 | n03075370_1317.JPEG 1523 | n03075370_17989.JPEG 1524 | n03075370_8253.JPEG 1525 | n03085013_19303.JPEG 1526 | n03085013_20216.JPEG 1527 | n03085013_5430.JPEG 1528 | n03089624_22409.JPEG 1529 | n03089624_23814.JPEG 1530 | n03089624_8590.JPEG 1531 | n03095699_21024.JPEG 1532 | n03095699_4024.JPEG 1533 | n03095699_5301.JPEG 1534 | n03100240_1390.JPEG 1535 | n03100240_633.JPEG 1536 | n03100240_7553.JPEG 1537 | n03109150_28682.JPEG 1538 | n03109150_721.JPEG 1539 | n03109150_8133.JPEG 1540 | n03110669_113648.JPEG 1541 | n03110669_2100.JPEG 1542 | n03110669_38824.JPEG 1543 | n03124043_1193.JPEG 1544 | n03124043_2468.JPEG 1545 | n03124043_8592.JPEG 1546 | n03124170_6989.JPEG 1547 | n03124170_7976.JPEG 1548 | n03124170_8418.JPEG 1549 | n03125729_21724.JPEG 1550 | n03125729_6131.JPEG 1551 | n03125729_63.JPEG 1552 | n03126707_21630.JPEG 1553 | n03126707_22527.JPEG 1554 | n03126707_24370.JPEG 1555 | n03127747_10543.JPEG 1556 | n03127747_13526.JPEG 1557 | n03127747_1999.JPEG 1558 | n03127925_13948.JPEG 1559 | n03127925_3009.JPEG 1560 | n03127925_923.JPEG 1561 | n03131574_27141.JPEG 1562 | n03131574_8063.JPEG 1563 | n03131574_9319.JPEG 1564 | n03133878_12259.JPEG 1565 | n03133878_3996.JPEG 1566 | n03133878_5966.JPEG 1567 | n03134739_1021.JPEG 1568 | n03134739_3000.JPEG 1569 | n03134739_7335.JPEG 1570 | n03141823_18571.JPEG 1571 | n03141823_3782.JPEG 1572 | n03141823_7225.JPEG 1573 | n03146219_10554.JPEG 1574 | n03146219_17182.JPEG 1575 | n03146219_22611.JPEG 1576 | n03160309_111039.JPEG 1577 | n03160309_20302.JPEG 1578 | n03160309_26667.JPEG 1579 | n03179701_5361.JPEG 1580 | n03179701_7730.JPEG 1581 | n03179701_7758.JPEG 1582 | n03180011_6860.JPEG 1583 | n03180011_7904.JPEG 1584 | n03180011_8640.JPEG 1585 | n03187595_10383.JPEG 1586 | n03187595_7407.JPEG 1587 | n03187595_756.JPEG 1588 | n03188531_22218.JPEG 1589 | n03188531_32474.JPEG 1590 | n03188531_3889.JPEG 1591 | n03196217_2611.JPEG 1592 | n03196217_382.JPEG 1593 | n03196217_4464.JPEG 1594 | n03197337_4780.JPEG 1595 | n03197337_5505.JPEG 1596 | n03197337_6479.JPEG 1597 | n03201208_24590.JPEG 1598 | n03201208_28578.JPEG 1599 | n03201208_30788.JPEG 1600 | n03207743_10711.JPEG 1601 | n03207743_12550.JPEG 1602 | n03207743_308.JPEG 1603 | n03207941_16426.JPEG 1604 | n03207941_25852.JPEG 1605 | n03207941_26260.JPEG 1606 | n03208938_29322.JPEG 1607 | n03208938_5064.JPEG 1608 | n03208938_6760.JPEG 1609 | n03216828_11449.JPEG 1610 | n03216828_16165.JPEG 1611 | n03216828_73924.JPEG 1612 | n03218198_4996.JPEG 1613 | n03218198_8892.JPEG 1614 | n03218198_957.JPEG 1615 | n03220513_14139.JPEG 1616 | n03220513_44388.JPEG 1617 | n03220513_7924.JPEG 1618 | n03223299_39459.JPEG 1619 | n03223299_4881.JPEG 1620 | n03223299_9805.JPEG 1621 | n03240683_11334.JPEG 1622 | n03240683_14011.JPEG 1623 | n03240683_15063.JPEG 1624 | n03249569_30248.JPEG 1625 | n03249569_39262.JPEG 1626 | n03249569_9307.JPEG 1627 | n03250847_14883.JPEG 1628 | n03250847_2350.JPEG 1629 | n03250847_2754.JPEG 1630 | n03255030_10065.JPEG 1631 | n03255030_1024.JPEG 1632 | n03255030_874.JPEG 1633 | n03259280_1374.JPEG 1634 | n03259280_2659.JPEG 1635 | n03259280_5579.JPEG 1636 | n03271574_24954.JPEG 1637 | n03271574_3003.JPEG 1638 | n03271574_3773.JPEG 1639 | n03272010_13813.JPEG 1640 | n03272010_6456.JPEG 1641 | n03272010_6954.JPEG 1642 | n03272562_11181.JPEG 1643 | n03272562_21535.JPEG 1644 | n03272562_7637.JPEG 1645 | n03290653_11379.JPEG 1646 | n03290653_4882.JPEG 1647 | n03290653_9770.JPEG 1648 | n03291819_29825.JPEG 1649 | n03291819_4319.JPEG 1650 | n03291819_8351.JPEG 1651 | n03297495_272.JPEG 1652 | n03297495_4716.JPEG 1653 | n03297495_7116.JPEG 1654 | n03314780_10337.JPEG 1655 | n03314780_6258.JPEG 1656 | n03314780_7775.JPEG 1657 | n03325584_12180.JPEG 1658 | n03325584_40759.JPEG 1659 | n03325584_9594.JPEG 1660 | n03337140_10860.JPEG 1661 | n03337140_32664.JPEG 1662 | n03337140_3633.JPEG 1663 | n03344393_11308.JPEG 1664 | n03344393_6467.JPEG 1665 | n03344393_6773.JPEG 1666 | n03345487_16803.JPEG 1667 | n03345487_25497.JPEG 1668 | n03345487_8490.JPEG 1669 | n03347037_18783.JPEG 1670 | n03347037_24599.JPEG 1671 | n03347037_5221.JPEG 1672 | n03355925_12490.JPEG 1673 | n03355925_252.JPEG 1674 | n03355925_408.JPEG 1675 | n03372029_23568.JPEG 1676 | n03372029_34144.JPEG 1677 | n03372029_44012.JPEG 1678 | n03376595_1233.JPEG 1679 | n03376595_3974.JPEG 1680 | n03376595_407.JPEG 1681 | n03379051_11254.JPEG 1682 | n03379051_335.JPEG 1683 | n03379051_795.JPEG 1684 | n03384352_2644.JPEG 1685 | n03384352_418.JPEG 1686 | n03384352_8072.JPEG 1687 | n03388043_12565.JPEG 1688 | n03388043_16140.JPEG 1689 | n03388043_4692.JPEG 1690 | n03388183_1397.JPEG 1691 | n03388183_7066.JPEG 1692 | n03388183_72.JPEG 1693 | n03388549_13735.JPEG 1694 | n03388549_563.JPEG 1695 | n03388549_5705.JPEG 1696 | n03393912_1645.JPEG 1697 | n03393912_555.JPEG 1698 | n03393912_814.JPEG 1699 | n03394916_39786.JPEG 1700 | n03394916_42175.JPEG 1701 | n03394916_47211.JPEG 1702 | n03400231_11311.JPEG 1703 | n03400231_1579.JPEG 1704 | n03400231_24586.JPEG 1705 | n03404251_4770.JPEG 1706 | n03404251_6124.JPEG 1707 | n03404251_9636.JPEG 1708 | n03417042_25940.JPEG 1709 | n03417042_28762.JPEG 1710 | n03417042_4407.JPEG 1711 | n03424325_15559.JPEG 1712 | n03424325_21291.JPEG 1713 | n03424325_23218.JPEG 1714 | n03425413_21766.JPEG 1715 | n03425413_28113.JPEG 1716 | n03425413_9117.JPEG 1717 | n03443371_11379.JPEG 1718 | n03443371_18015.JPEG 1719 | n03443371_3413.JPEG 1720 | n03444034_10921.JPEG 1721 | n03444034_5395.JPEG 1722 | n03444034_7225.JPEG 1723 | n03445777_1866.JPEG 1724 | n03445777_4588.JPEG 1725 | n03445777_8289.JPEG 1726 | n03445924_16171.JPEG 1727 | n03445924_1649.JPEG 1728 | n03445924_9542.JPEG 1729 | n03447447_1533.JPEG 1730 | n03447447_2722.JPEG 1731 | n03447447_5905.JPEG 1732 | n03447721_22924.JPEG 1733 | n03447721_25199.JPEG 1734 | n03447721_51466.JPEG 1735 | n03450230_14929.JPEG 1736 | n03450230_22624.JPEG 1737 | n03450230_6440.JPEG 1738 | n03452741_12531.JPEG 1739 | n03452741_7350.JPEG 1740 | n03452741_8820.JPEG 1741 | n03457902_10806.JPEG 1742 | n03457902_15926.JPEG 1743 | n03457902_8190.JPEG 1744 | n03459775_11076.JPEG 1745 | n03459775_15888.JPEG 1746 | n03459775_3642.JPEG 1747 | n03461385_2860.JPEG 1748 | n03461385_40466.JPEG 1749 | n03461385_54651.JPEG 1750 | n03467068_15187.JPEG 1751 | n03467068_17697.JPEG 1752 | n03467068_7474.JPEG 1753 | n03476684_13404.JPEG 1754 | n03476684_14577.JPEG 1755 | n03476684_14648.JPEG 1756 | n03476991_1334.JPEG 1757 | n03476991_24263.JPEG 1758 | n03476991_39386.JPEG 1759 | n03478589_1571.JPEG 1760 | n03478589_15795.JPEG 1761 | n03478589_7136.JPEG 1762 | n03481172_14265.JPEG 1763 | n03481172_17430.JPEG 1764 | n03481172_8758.JPEG 1765 | n03482405_10532.JPEG 1766 | n03482405_27608.JPEG 1767 | n03482405_4043.JPEG 1768 | n03483316_1219.JPEG 1769 | n03483316_16812.JPEG 1770 | n03483316_2236.JPEG 1771 | n03485407_1038.JPEG 1772 | n03485407_15658.JPEG 1773 | n03485407_4667.JPEG 1774 | n03485794_18191.JPEG 1775 | n03485794_30950.JPEG 1776 | n03485794_70820.JPEG 1777 | n03492542_24149.JPEG 1778 | n03492542_2805.JPEG 1779 | n03492542_46289.JPEG 1780 | n03494278_35610.JPEG 1781 | n03494278_38369.JPEG 1782 | n03494278_39315.JPEG 1783 | n03495258_11598.JPEG 1784 | n03495258_18143.JPEG 1785 | n03495258_23090.JPEG 1786 | n03496892_15476.JPEG 1787 | n03496892_23785.JPEG 1788 | n03496892_27093.JPEG 1789 | n03498962_11632.JPEG 1790 | n03498962_4727.JPEG 1791 | n03498962_5998.JPEG 1792 | n03527444_17842.JPEG 1793 | n03527444_18269.JPEG 1794 | n03527444_1977.JPEG 1795 | n03529860_5059.JPEG 1796 | n03529860_5208.JPEG 1797 | n03529860_5520.JPEG 1798 | n03530642_11928.JPEG 1799 | n03530642_23289.JPEG 1800 | n03530642_33679.JPEG 1801 | n03532672_22309.JPEG 1802 | n03532672_33366.JPEG 1803 | n03532672_8419.JPEG 1804 | n03534580_19060.JPEG 1805 | n03534580_2011.JPEG 1806 | n03534580_32948.JPEG 1807 | n03535780_27565.JPEG 1808 | n03535780_31030.JPEG 1809 | n03535780_48037.JPEG 1810 | n03538406_2984.JPEG 1811 | n03538406_3346.JPEG 1812 | n03538406_9621.JPEG 1813 | n03544143_10186.JPEG 1814 | n03544143_6174.JPEG 1815 | n03544143_7992.JPEG 1816 | n03584254_3184.JPEG 1817 | n03584254_3662.JPEG 1818 | n03584254_7059.JPEG 1819 | n03584829_23825.JPEG 1820 | n03584829_29898.JPEG 1821 | n03584829_36836.JPEG 1822 | n03590841_2695.JPEG 1823 | n03590841_2828.JPEG 1824 | n03590841_9901.JPEG 1825 | n03594734_13644.JPEG 1826 | n03594734_34418.JPEG 1827 | n03594734_67977.JPEG 1828 | n03594945_12330.JPEG 1829 | n03594945_2143.JPEG 1830 | n03594945_35828.JPEG 1831 | n03595614_20379.JPEG 1832 | n03595614_23559.JPEG 1833 | n03595614_8106.JPEG 1834 | n03598930_10711.JPEG 1835 | n03598930_15280.JPEG 1836 | n03598930_3476.JPEG 1837 | n03599486_12732.JPEG 1838 | n03599486_13100.JPEG 1839 | n03599486_6086.JPEG 1840 | n03602883_13584.JPEG 1841 | n03602883_19656.JPEG 1842 | n03602883_7963.JPEG 1843 | n03617480_5541.JPEG 1844 | n03617480_5765.JPEG 1845 | n03617480_7760.JPEG 1846 | n03623198_13943.JPEG 1847 | n03623198_3104.JPEG 1848 | n03623198_3598.JPEG 1849 | n03627232_11263.JPEG 1850 | n03627232_29804.JPEG 1851 | n03627232_5004.JPEG 1852 | n03630383_2087.JPEG 1853 | n03630383_4130.JPEG 1854 | n03630383_5516.JPEG 1855 | n03633091_1266.JPEG 1856 | n03633091_12816.JPEG 1857 | n03633091_14950.JPEG 1858 | n03637318_1893.JPEG 1859 | n03637318_19794.JPEG 1860 | n03637318_20830.JPEG 1861 | n03642806_1716.JPEG 1862 | n03642806_1957.JPEG 1863 | n03642806_57.JPEG 1864 | n03649909_21339.JPEG 1865 | n03649909_4184.JPEG 1866 | n03649909_6954.JPEG 1867 | n03657121_16250.JPEG 1868 | n03657121_24927.JPEG 1869 | n03657121_8361.JPEG 1870 | n03658185_10091.JPEG 1871 | n03658185_18670.JPEG 1872 | n03658185_7264.JPEG 1873 | n03661043_10101.JPEG 1874 | n03661043_10587.JPEG 1875 | n03661043_19538.JPEG 1876 | n03662601_12226.JPEG 1877 | n03662601_22870.JPEG 1878 | n03662601_3753.JPEG 1879 | n03666591_32873.JPEG 1880 | n03666591_35854.JPEG 1881 | n03666591_38505.JPEG 1882 | n03670208_13439.JPEG 1883 | n03670208_14846.JPEG 1884 | n03670208_14906.JPEG 1885 | n03673027_11328.JPEG 1886 | n03673027_15080.JPEG 1887 | n03673027_8010.JPEG 1888 | n03676483_12719.JPEG 1889 | n03676483_2985.JPEG 1890 | n03676483_9359.JPEG 1891 | n03680355_2551.JPEG 1892 | n03680355_4042.JPEG 1893 | n03680355_4058.JPEG 1894 | n03690938_11061.JPEG 1895 | n03690938_808.JPEG 1896 | n03690938_9305.JPEG 1897 | n03691459_22521.JPEG 1898 | n03691459_23888.JPEG 1899 | n03691459_570.JPEG 1900 | n03692522_20864.JPEG 1901 | n03692522_28439.JPEG 1902 | n03692522_3615.JPEG 1903 | n03697007_2034.JPEG 1904 | n03697007_23925.JPEG 1905 | n03697007_2914.JPEG 1906 | n03706229_105.JPEG 1907 | n03706229_2855.JPEG 1908 | n03706229_8442.JPEG 1909 | n03709823_24255.JPEG 1910 | n03709823_25525.JPEG 1911 | n03709823_8375.JPEG 1912 | n03710193_10764.JPEG 1913 | n03710193_5703.JPEG 1914 | n03710193_6843.JPEG 1915 | n03710637_6708.JPEG 1916 | n03710637_6757.JPEG 1917 | n03710637_757.JPEG 1918 | n03710721_3812.JPEG 1919 | n03710721_4910.JPEG 1920 | n03710721_8513.JPEG 1921 | n03717622_10735.JPEG 1922 | n03717622_2957.JPEG 1923 | n03717622_9835.JPEG 1924 | n03720891_13887.JPEG 1925 | n03720891_149.JPEG 1926 | n03720891_2982.JPEG 1927 | n03721384_17353.JPEG 1928 | n03721384_34889.JPEG 1929 | n03721384_3704.JPEG 1930 | n03724870_2088.JPEG 1931 | n03724870_934.JPEG 1932 | n03724870_9871.JPEG 1933 | n03729826_10874.JPEG 1934 | n03729826_26201.JPEG 1935 | n03729826_27596.JPEG 1936 | n03733131_15151.JPEG 1937 | n03733131_1857.JPEG 1938 | n03733131_2333.JPEG 1939 | n03733281_12695.JPEG 1940 | n03733281_13628.JPEG 1941 | n03733281_13753.JPEG 1942 | n03733805_13176.JPEG 1943 | n03733805_25652.JPEG 1944 | n03733805_9921.JPEG 1945 | n03742115_12598.JPEG 1946 | n03742115_2382.JPEG 1947 | n03742115_6861.JPEG 1948 | n03743016_13944.JPEG 1949 | n03743016_21571.JPEG 1950 | n03743016_5530.JPEG 1951 | n03759954_12381.JPEG 1952 | n03759954_19272.JPEG 1953 | n03759954_25065.JPEG 1954 | n03761084_15185.JPEG 1955 | n03761084_1978.JPEG 1956 | n03761084_9066.JPEG 1957 | n03763968_1854.JPEG 1958 | n03763968_1858.JPEG 1959 | n03763968_5758.JPEG 1960 | n03764736_11998.JPEG 1961 | n03764736_152.JPEG 1962 | n03764736_21552.JPEG 1963 | n03769881_3284.JPEG 1964 | n03769881_8009.JPEG 1965 | n03769881_8269.JPEG 1966 | n03770439_10091.JPEG 1967 | n03770439_16128.JPEG 1968 | n03770439_2305.JPEG 1969 | n03770679_16607.JPEG 1970 | n03770679_25459.JPEG 1971 | n03770679_8538.JPEG 1972 | n03773504_1868.JPEG 1973 | n03773504_2784.JPEG 1974 | n03773504_8783.JPEG 1975 | n03775071_1023.JPEG 1976 | n03775071_13971.JPEG 1977 | n03775071_9544.JPEG 1978 | n03775546_11081.JPEG 1979 | n03775546_3108.JPEG 1980 | n03775546_8362.JPEG 1981 | n03776460_245.JPEG 1982 | n03776460_29968.JPEG 1983 | n03776460_6340.JPEG 1984 | n03777568_11073.JPEG 1985 | n03777568_3143.JPEG 1986 | n03777568_5650.JPEG 1987 | n03777754_28089.JPEG 1988 | n03777754_3584.JPEG 1989 | n03777754_4385.JPEG 1990 | n03781244_112513.JPEG 1991 | n03781244_19180.JPEG 1992 | n03781244_67251.JPEG 1993 | n03782006_11411.JPEG 1994 | n03782006_42578.JPEG 1995 | n03782006_606.JPEG 1996 | n03785016_15892.JPEG 1997 | n03785016_19695.JPEG 1998 | n03785016_970.JPEG 1999 | n03786901_10748.JPEG 2000 | n03786901_3009.JPEG 2001 | n03786901_5503.JPEG 2002 | n03787032_3700.JPEG 2003 | n03787032_5084.JPEG 2004 | n03787032_779.JPEG 2005 | n03788195_34055.JPEG 2006 | n03788195_8186.JPEG 2007 | n03788195_8261.JPEG 2008 | n03788365_20364.JPEG 2009 | n03788365_6961.JPEG 2010 | n03788365_9778.JPEG 2011 | n03791053_15778.JPEG 2012 | n03791053_3784.JPEG 2013 | n03791053_7929.JPEG 2014 | n03792782_11000.JPEG 2015 | n03792782_55401.JPEG 2016 | n03792782_8080.JPEG 2017 | n03792972_2235.JPEG 2018 | n03792972_7834.JPEG 2019 | n03792972_829.JPEG 2020 | n03793489_3363.JPEG 2021 | n03793489_54445.JPEG 2022 | n03793489_6900.JPEG 2023 | n03794056_10225.JPEG 2024 | n03794056_10784.JPEG 2025 | n03794056_2240.JPEG 2026 | n03796401_16046.JPEG 2027 | n03796401_6526.JPEG 2028 | n03796401_7817.JPEG 2029 | n03803284_11827.JPEG 2030 | n03803284_12453.JPEG 2031 | n03803284_18177.JPEG 2032 | n03804744_10768.JPEG 2033 | n03804744_30467.JPEG 2034 | n03804744_35413.JPEG 2035 | n03814639_6425.JPEG 2036 | n03814639_6900.JPEG 2037 | n03814639_763.JPEG 2038 | n03814906_32423.JPEG 2039 | n03814906_35683.JPEG 2040 | n03814906_3690.JPEG 2041 | n03825788_1139.JPEG 2042 | n03825788_16216.JPEG 2043 | n03825788_35254.JPEG 2044 | n03832673_2668.JPEG 2045 | n03832673_2766.JPEG 2046 | n03832673_456.JPEG 2047 | n03837869_3225.JPEG 2048 | n03837869_4388.JPEG 2049 | n03837869_9014.JPEG 2050 | n03838899_26458.JPEG 2051 | n03838899_498.JPEG 2052 | n03838899_6918.JPEG 2053 | n03840681_10007.JPEG 2054 | n03840681_17270.JPEG 2055 | n03840681_29902.JPEG 2056 | n03841143_13351.JPEG 2057 | n03841143_1510.JPEG 2058 | n03841143_8003.JPEG 2059 | n03843555_10157.JPEG 2060 | n03843555_1596.JPEG 2061 | n03843555_60.JPEG 2062 | n03854065_38692.JPEG 2063 | n03854065_4628.JPEG 2064 | n03854065_6095.JPEG 2065 | n03857828_22423.JPEG 2066 | n03857828_38049.JPEG 2067 | n03857828_8543.JPEG 2068 | n03866082_10137.JPEG 2069 | n03866082_3971.JPEG 2070 | n03866082_522.JPEG 2071 | n03868242_1294.JPEG 2072 | n03868242_7901.JPEG 2073 | n03868242_8289.JPEG 2074 | n03868863_2247.JPEG 2075 | n03868863_2483.JPEG 2076 | n03868863_7616.JPEG 2077 | n03871628_11242.JPEG 2078 | n03871628_5232.JPEG 2079 | n03871628_5409.JPEG 2080 | n03873416_16858.JPEG 2081 | n03873416_20910.JPEG 2082 | n03873416_2903.JPEG 2083 | n03874293_1436.JPEG 2084 | n03874293_15601.JPEG 2085 | n03874293_2183.JPEG 2086 | n03874599_1420.JPEG 2087 | n03874599_8751.JPEG 2088 | n03874599_8944.JPEG 2089 | n03876231_10647.JPEG 2090 | n03876231_17851.JPEG 2091 | n03876231_8796.JPEG 2092 | n03877472_25419.JPEG 2093 | n03877472_56235.JPEG 2094 | n03877472_935.JPEG 2095 | n03877845_1753.JPEG 2096 | n03877845_20673.JPEG 2097 | n03877845_9511.JPEG 2098 | n03884397_21355.JPEG 2099 | n03884397_22920.JPEG 2100 | n03884397_25709.JPEG 2101 | n03887697_779.JPEG 2102 | n03887697_7914.JPEG 2103 | n03887697_8747.JPEG 2104 | n03888257_14853.JPEG 2105 | n03888257_18360.JPEG 2106 | n03888257_37513.JPEG 2107 | n03888605_12182.JPEG 2108 | n03888605_1244.JPEG 2109 | n03888605_2674.JPEG 2110 | n03891251_2168.JPEG 2111 | n03891251_322.JPEG 2112 | n03891251_5869.JPEG 2113 | n03891332_1745.JPEG 2114 | n03891332_3029.JPEG 2115 | n03891332_4700.JPEG 2116 | n03895866_22460.JPEG 2117 | n03895866_49245.JPEG 2118 | n03895866_58080.JPEG 2119 | n03899768_11905.JPEG 2120 | n03899768_31992.JPEG 2121 | n03899768_37315.JPEG 2122 | n03902125_15238.JPEG 2123 | n03902125_2888.JPEG 2124 | n03902125_5106.JPEG 2125 | n03903868_51335.JPEG 2126 | n03903868_52569.JPEG 2127 | n03903868_8420.JPEG 2128 | n03908618_12183.JPEG 2129 | n03908618_867.JPEG 2130 | n03908618_9824.JPEG 2131 | n03908714_3763.JPEG 2132 | n03908714_4954.JPEG 2133 | n03908714_6725.JPEG 2134 | n03916031_12567.JPEG 2135 | n03916031_2432.JPEG 2136 | n03916031_54543.JPEG 2137 | n03920288_1857.JPEG 2138 | n03920288_2397.JPEG 2139 | n03920288_3272.JPEG 2140 | n03924679_12192.JPEG 2141 | n03924679_13844.JPEG 2142 | n03924679_15917.JPEG 2143 | n03929660_20947.JPEG 2144 | n03929660_26525.JPEG 2145 | n03929660_5009.JPEG 2146 | n03929855_1015.JPEG 2147 | n03929855_1075.JPEG 2148 | n03929855_3690.JPEG 2149 | n03930313_11531.JPEG 2150 | n03930313_2909.JPEG 2151 | n03930313_6933.JPEG 2152 | n03930630_11996.JPEG 2153 | n03930630_2444.JPEG 2154 | n03930630_708.JPEG 2155 | n03933933_12369.JPEG 2156 | n03933933_21446.JPEG 2157 | n03933933_2920.JPEG 2158 | n03935335_29647.JPEG 2159 | n03935335_4341.JPEG 2160 | n03935335_6711.JPEG 2161 | n03937543_1326.JPEG 2162 | n03937543_1719.JPEG 2163 | n03937543_7230.JPEG 2164 | n03938244_10811.JPEG 2165 | n03938244_2980.JPEG 2166 | n03938244_4612.JPEG 2167 | n03942813_14517.JPEG 2168 | n03942813_4177.JPEG 2169 | n03942813_6935.JPEG 2170 | n03944341_26420.JPEG 2171 | n03944341_8037.JPEG 2172 | n03944341_8746.JPEG 2173 | n03947888_1629.JPEG 2174 | n03947888_48712.JPEG 2175 | n03947888_5305.JPEG 2176 | n03950228_18454.JPEG 2177 | n03950228_3133.JPEG 2178 | n03950228_6752.JPEG 2179 | n03954731_19279.JPEG 2180 | n03954731_25797.JPEG 2181 | n03954731_8145.JPEG 2182 | n03956157_12236.JPEG 2183 | n03956157_14495.JPEG 2184 | n03956157_32106.JPEG 2185 | n03958227_12841.JPEG 2186 | n03958227_3025.JPEG 2187 | n03958227_6427.JPEG 2188 | n03961711_1466.JPEG 2189 | n03961711_2989.JPEG 2190 | n03961711_7125.JPEG 2191 | n03967562_23207.JPEG 2192 | n03967562_27963.JPEG 2193 | n03967562_35411.JPEG 2194 | n03970156_10084.JPEG 2195 | n03970156_22085.JPEG 2196 | n03970156_9214.JPEG 2197 | n03976467_4181.JPEG 2198 | n03976467_5725.JPEG 2199 | n03976467_7239.JPEG 2200 | n03976657_12018.JPEG 2201 | n03976657_3599.JPEG 2202 | n03976657_41923.JPEG 2203 | n03977966_36479.JPEG 2204 | n03977966_37672.JPEG 2205 | n03977966_48003.JPEG 2206 | n03980874_10576.JPEG 2207 | n03980874_13020.JPEG 2208 | n03980874_7403.JPEG 2209 | n03982430_20759.JPEG 2210 | n03982430_31992.JPEG 2211 | n03982430_38194.JPEG 2212 | n03983396_12563.JPEG 2213 | n03983396_1584.JPEG 2214 | n03983396_8294.JPEG 2215 | n03991062_1088.JPEG 2216 | n03991062_5942.JPEG 2217 | n03991062_6699.JPEG 2218 | n03992509_11021.JPEG 2219 | n03992509_2753.JPEG 2220 | n03992509_748.JPEG 2221 | n03995372_16008.JPEG 2222 | n03995372_16226.JPEG 2223 | n03995372_6386.JPEG 2224 | n03998194_25579.JPEG 2225 | n03998194_28798.JPEG 2226 | n03998194_9461.JPEG 2227 | n04004767_10231.JPEG 2228 | n04004767_10757.JPEG 2229 | n04004767_12405.JPEG 2230 | n04005630_102964.JPEG 2231 | n04005630_111958.JPEG 2232 | n04005630_53099.JPEG 2233 | n04008634_216.JPEG 2234 | n04008634_22865.JPEG 2235 | n04008634_5061.JPEG 2236 | n04009552_17489.JPEG 2237 | n04009552_227.JPEG 2238 | n04009552_8350.JPEG 2239 | n04019541_39856.JPEG 2240 | n04019541_61986.JPEG 2241 | n04019541_62542.JPEG 2242 | n04023962_43526.JPEG 2243 | n04023962_8015.JPEG 2244 | n04023962_9159.JPEG 2245 | n04026417_14712.JPEG 2246 | n04026417_15020.JPEG 2247 | n04026417_21449.JPEG 2248 | n04033901_16542.JPEG 2249 | n04033901_16753.JPEG 2250 | n04033901_17674.JPEG 2251 | n04033995_1213.JPEG 2252 | n04033995_20707.JPEG 2253 | n04033995_5630.JPEG 2254 | n04037443_18678.JPEG 2255 | n04037443_21205.JPEG 2256 | n04037443_9519.JPEG 2257 | n04039381_18582.JPEG 2258 | n04039381_20264.JPEG 2259 | n04039381_85.JPEG 2260 | n04040759_10565.JPEG 2261 | n04040759_12029.JPEG 2262 | n04040759_8198.JPEG 2263 | n04041544_40100.JPEG 2264 | n04041544_42620.JPEG 2265 | n04041544_9021.JPEG 2266 | n04044716_11473.JPEG 2267 | n04044716_13329.JPEG 2268 | n04044716_1891.JPEG 2269 | n04049303_3747.JPEG 2270 | n04049303_4243.JPEG 2271 | n04049303_8109.JPEG 2272 | n04065272_1031.JPEG 2273 | n04065272_16050.JPEG 2274 | n04065272_2288.JPEG 2275 | n04067472_7477.JPEG 2276 | n04067472_7708.JPEG 2277 | n04067472_8543.JPEG 2278 | n04069434_334.JPEG 2279 | n04069434_4985.JPEG 2280 | n04069434_656.JPEG 2281 | n04070727_26186.JPEG 2282 | n04070727_40833.JPEG 2283 | n04070727_45355.JPEG 2284 | n04074963_13692.JPEG 2285 | n04074963_38897.JPEG 2286 | n04074963_6554.JPEG 2287 | n04081281_13114.JPEG 2288 | n04081281_29686.JPEG 2289 | n04081281_39518.JPEG 2290 | n04086273_20829.JPEG 2291 | n04086273_26442.JPEG 2292 | n04086273_27801.JPEG 2293 | n04090263_2051.JPEG 2294 | n04090263_2880.JPEG 2295 | n04090263_8613.JPEG 2296 | n04099969_12218.JPEG 2297 | n04099969_15641.JPEG 2298 | n04099969_5698.JPEG 2299 | n04111531_23340.JPEG 2300 | n04111531_3902.JPEG 2301 | n04111531_7843.JPEG 2302 | n04116512_11486.JPEG 2303 | n04116512_15926.JPEG 2304 | n04116512_7283.JPEG 2305 | n04118538_1384.JPEG 2306 | n04118538_3607.JPEG 2307 | n04118538_8369.JPEG 2308 | n04118776_13055.JPEG 2309 | n04118776_40085.JPEG 2310 | n04118776_42696.JPEG 2311 | n04120489_2158.JPEG 2312 | n04120489_4748.JPEG 2313 | n04120489_8999.JPEG 2314 | n04125021_10077.JPEG 2315 | n04125021_1395.JPEG 2316 | n04125021_8601.JPEG 2317 | n04127249_10548.JPEG 2318 | n04127249_1643.JPEG 2319 | n04127249_808.JPEG 2320 | n04131690_12697.JPEG 2321 | n04131690_3365.JPEG 2322 | n04131690_3828.JPEG 2323 | n04133789_11550.JPEG 2324 | n04133789_3863.JPEG 2325 | n04133789_605.JPEG 2326 | n04136333_15090.JPEG 2327 | n04136333_2041.JPEG 2328 | n04136333_22346.JPEG 2329 | n04141076_12287.JPEG 2330 | n04141076_17573.JPEG 2331 | n04141076_5697.JPEG 2332 | n04141327_1428.JPEG 2333 | n04141327_6803.JPEG 2334 | n04141327_7180.JPEG 2335 | n04141975_10659.JPEG 2336 | n04141975_24993.JPEG 2337 | n04141975_28535.JPEG 2338 | n04146614_12868.JPEG 2339 | n04146614_12888.JPEG 2340 | n04146614_6470.JPEG 2341 | n04147183_11625.JPEG 2342 | n04147183_22854.JPEG 2343 | n04147183_3083.JPEG 2344 | n04149813_217.JPEG 2345 | n04149813_2570.JPEG 2346 | n04149813_7136.JPEG 2347 | n04152593_18706.JPEG 2348 | n04152593_21480.JPEG 2349 | n04152593_4490.JPEG 2350 | n04153751_24406.JPEG 2351 | n04153751_24854.JPEG 2352 | n04153751_3714.JPEG 2353 | n04154565_10692.JPEG 2354 | n04154565_13022.JPEG 2355 | n04154565_24912.JPEG 2356 | n04162706_20611.JPEG 2357 | n04162706_29628.JPEG 2358 | n04162706_35208.JPEG 2359 | n04179913_14773.JPEG 2360 | n04179913_4418.JPEG 2361 | n04179913_8437.JPEG 2362 | n04192698_1502.JPEG 2363 | n04192698_43013.JPEG 2364 | n04192698_9236.JPEG 2365 | n04200800_14970.JPEG 2366 | n04200800_37337.JPEG 2367 | n04200800_953.JPEG 2368 | n04201297_14197.JPEG 2369 | n04201297_6607.JPEG 2370 | n04201297_8147.JPEG 2371 | n04204238_5352.JPEG 2372 | n04204238_7586.JPEG 2373 | n04204238_8425.JPEG 2374 | n04204347_1271.JPEG 2375 | n04204347_3899.JPEG 2376 | n04204347_6068.JPEG 2377 | n04208210_21836.JPEG 2378 | n04208210_5525.JPEG 2379 | n04208210_5787.JPEG 2380 | n04209133_10606.JPEG 2381 | n04209133_2421.JPEG 2382 | n04209133_4248.JPEG 2383 | n04209239_14768.JPEG 2384 | n04209239_4865.JPEG 2385 | n04209239_8335.JPEG 2386 | n04228054_26395.JPEG 2387 | n04228054_33011.JPEG 2388 | n04228054_8116.JPEG 2389 | n04229816_6143.JPEG 2390 | n04229816_7346.JPEG 2391 | n04229816_8768.JPEG 2392 | n04235860_14985.JPEG 2393 | n04235860_1548.JPEG 2394 | n04235860_6002.JPEG 2395 | n04238763_1053.JPEG 2396 | n04238763_26268.JPEG 2397 | n04238763_8641.JPEG 2398 | n04239074_13005.JPEG 2399 | n04239074_1357.JPEG 2400 | n04239074_24294.JPEG 2401 | n04243546_10804.JPEG 2402 | n04243546_29485.JPEG 2403 | n04243546_4231.JPEG 2404 | n04251144_2095.JPEG 2405 | n04251144_22655.JPEG 2406 | n04251144_3454.JPEG 2407 | n04252077_2282.JPEG 2408 | n04252077_2849.JPEG 2409 | n04252077_857.JPEG 2410 | n04252225_18547.JPEG 2411 | n04252225_19174.JPEG 2412 | n04252225_6219.JPEG 2413 | n04254120_1668.JPEG 2414 | n04254120_2459.JPEG 2415 | n04254120_855.JPEG 2416 | n04254680_1597.JPEG 2417 | n04254680_5329.JPEG 2418 | n04254680_5982.JPEG 2419 | n04254777_16295.JPEG 2420 | n04254777_8023.JPEG 2421 | n04254777_8942.JPEG 2422 | n04258138_16804.JPEG 2423 | n04258138_4926.JPEG 2424 | n04258138_7671.JPEG 2425 | n04259630_15622.JPEG 2426 | n04259630_20030.JPEG 2427 | n04259630_2725.JPEG 2428 | n04263257_3484.JPEG 2429 | n04263257_3623.JPEG 2430 | n04263257_5639.JPEG 2431 | n04264628_2759.JPEG 2432 | n04264628_659.JPEG 2433 | n04264628_690.JPEG 2434 | n04265275_13197.JPEG 2435 | n04265275_2946.JPEG 2436 | n04265275_6130.JPEG 2437 | n04266014_22830.JPEG 2438 | n04266014_474.JPEG 2439 | n04266014_796.JPEG 2440 | n04270147_10343.JPEG 2441 | n04270147_13600.JPEG 2442 | n04270147_7762.JPEG 2443 | n04273569_13136.JPEG 2444 | n04273569_3209.JPEG 2445 | n04273569_32512.JPEG 2446 | n04275548_2099.JPEG 2447 | n04275548_2908.JPEG 2448 | n04275548_3655.JPEG 2449 | n04277352_10477.JPEG 2450 | n04277352_35807.JPEG 2451 | n04277352_8509.JPEG 2452 | n04285008_4930.JPEG 2453 | n04285008_4999.JPEG 2454 | n04285008_7951.JPEG 2455 | n04286575_16113.JPEG 2456 | n04286575_3656.JPEG 2457 | n04286575_40079.JPEG 2458 | n04296562_18833.JPEG 2459 | n04296562_22286.JPEG 2460 | n04296562_55241.JPEG 2461 | n04310018_13857.JPEG 2462 | n04310018_6647.JPEG 2463 | n04310018_7928.JPEG 2464 | n04311004_2261.JPEG 2465 | n04311004_4116.JPEG 2466 | n04311004_6274.JPEG 2467 | n04311174_5468.JPEG 2468 | n04311174_6481.JPEG 2469 | n04311174_858.JPEG 2470 | n04317175_11104.JPEG 2471 | n04317175_2121.JPEG 2472 | n04317175_2130.JPEG 2473 | n04325704_11628.JPEG 2474 | n04325704_847.JPEG 2475 | n04325704_855.JPEG 2476 | n04326547_1078.JPEG 2477 | n04326547_14024.JPEG 2478 | n04326547_2805.JPEG 2479 | n04328186_2704.JPEG 2480 | n04328186_45143.JPEG 2481 | n04328186_5532.JPEG 2482 | n04330267_16382.JPEG 2483 | n04330267_1669.JPEG 2484 | n04330267_18725.JPEG 2485 | n04332243_11649.JPEG 2486 | n04332243_12709.JPEG 2487 | n04332243_4551.JPEG 2488 | n04335435_15438.JPEG 2489 | n04335435_24796.JPEG 2490 | n04335435_2636.JPEG 2491 | n04336792_10345.JPEG 2492 | n04336792_110.JPEG 2493 | n04336792_8194.JPEG 2494 | n04344873_12292.JPEG 2495 | n04344873_4071.JPEG 2496 | n04344873_8118.JPEG 2497 | n04346328_1290.JPEG 2498 | n04346328_15499.JPEG 2499 | n04346328_8218.JPEG 2500 | n04347754_13053.JPEG 2501 | n04347754_17713.JPEG 2502 | n04347754_20786.JPEG 2503 | n04350905_13565.JPEG 2504 | n04350905_20210.JPEG 2505 | n04350905_2720.JPEG 2506 | n04355338_10020.JPEG 2507 | n04355338_4117.JPEG 2508 | n04355338_5564.JPEG 2509 | n04355933_1589.JPEG 2510 | n04355933_3359.JPEG 2511 | n04355933_7788.JPEG 2512 | n04356056_39246.JPEG 2513 | n04356056_5070.JPEG 2514 | n04356056_7446.JPEG 2515 | n04357314_12693.JPEG 2516 | n04357314_15413.JPEG 2517 | n04357314_34770.JPEG 2518 | n04366367_16650.JPEG 2519 | n04366367_3081.JPEG 2520 | n04366367_6784.JPEG 2521 | n04367480_32775.JPEG 2522 | n04367480_34409.JPEG 2523 | n04367480_6014.JPEG 2524 | n04370456_1099.JPEG 2525 | n04370456_28401.JPEG 2526 | n04370456_9208.JPEG 2527 | n04371430_10419.JPEG 2528 | n04371430_13069.JPEG 2529 | n04371430_266.JPEG 2530 | n04371774_12919.JPEG 2531 | n04371774_14005.JPEG 2532 | n04371774_8883.JPEG 2533 | n04372370_33949.JPEG 2534 | n04372370_34864.JPEG 2535 | n04372370_4415.JPEG 2536 | n04376876_10087.JPEG 2537 | n04376876_7537.JPEG 2538 | n04376876_8601.JPEG 2539 | n04380533_1836.JPEG 2540 | n04380533_28002.JPEG 2541 | n04380533_6062.JPEG 2542 | n04389033_1304.JPEG 2543 | n04389033_2035.JPEG 2544 | n04389033_8822.JPEG 2545 | n04392985_10422.JPEG 2546 | n04392985_2645.JPEG 2547 | n04392985_3999.JPEG 2548 | n04398044_11870.JPEG 2549 | n04398044_4927.JPEG 2550 | n04398044_6052.JPEG 2551 | n04399382_13253.JPEG 2552 | n04399382_24863.JPEG 2553 | n04399382_5877.JPEG 2554 | n04404412_17522.JPEG 2555 | n04404412_23553.JPEG 2556 | n04404412_23829.JPEG 2557 | n04409515_16390.JPEG 2558 | n04409515_4483.JPEG 2559 | n04409515_8479.JPEG 2560 | n04417672_11359.JPEG 2561 | n04417672_26.JPEG 2562 | n04417672_8856.JPEG 2563 | n04418357_4297.JPEG 2564 | n04418357_4683.JPEG 2565 | n04418357_6504.JPEG 2566 | n04423845_10717.JPEG 2567 | n04423845_44208.JPEG 2568 | n04423845_6705.JPEG 2569 | n04428191_14876.JPEG 2570 | n04428191_27601.JPEG 2571 | n04428191_4745.JPEG 2572 | n04429376_18296.JPEG 2573 | n04429376_24091.JPEG 2574 | n04429376_5431.JPEG 2575 | n04435653_10153.JPEG 2576 | n04435653_11053.JPEG 2577 | n04435653_5650.JPEG 2578 | n04442312_11829.JPEG 2579 | n04442312_2008.JPEG 2580 | n04442312_7955.JPEG 2581 | n04443257_17638.JPEG 2582 | n04443257_23089.JPEG 2583 | n04443257_8561.JPEG 2584 | n04447861_1403.JPEG 2585 | n04447861_2608.JPEG 2586 | n04447861_6467.JPEG 2587 | n04456115_20254.JPEG 2588 | n04456115_29278.JPEG 2589 | n04456115_46773.JPEG 2590 | n04458633_17371.JPEG 2591 | n04458633_18920.JPEG 2592 | n04458633_3676.JPEG 2593 | n04461696_4557.JPEG 2594 | n04461696_4933.JPEG 2595 | n04461696_6999.JPEG 2596 | n04462240_16171.JPEG 2597 | n04462240_739.JPEG 2598 | n04462240_9342.JPEG 2599 | n04465501_2027.JPEG 2600 | n04465501_2042.JPEG 2601 | n04465501_7681.JPEG 2602 | n04467665_4989.JPEG 2603 | n04467665_67081.JPEG 2604 | n04467665_7451.JPEG 2605 | n04476259_12258.JPEG 2606 | n04476259_18166.JPEG 2607 | n04476259_19099.JPEG 2608 | n04479046_24318.JPEG 2609 | n04479046_2786.JPEG 2610 | n04479046_972.JPEG 2611 | n04482393_12542.JPEG 2612 | n04482393_12738.JPEG 2613 | n04482393_26430.JPEG 2614 | n04483307_10882.JPEG 2615 | n04483307_3231.JPEG 2616 | n04483307_6238.JPEG 2617 | n04485082_2008.JPEG 2618 | n04485082_39458.JPEG 2619 | n04485082_4948.JPEG 2620 | n04486054_1089.JPEG 2621 | n04486054_4249.JPEG 2622 | n04486054_4625.JPEG 2623 | n04487081_19492.JPEG 2624 | n04487081_40389.JPEG 2625 | n04487081_5046.JPEG 2626 | n04487394_17830.JPEG 2627 | n04487394_2093.JPEG 2628 | n04487394_38617.JPEG 2629 | n04493381_4316.JPEG 2630 | n04493381_69421.JPEG 2631 | n04493381_7355.JPEG 2632 | n04501370_16836.JPEG 2633 | n04501370_23272.JPEG 2634 | n04501370_3547.JPEG 2635 | n04505470_2886.JPEG 2636 | n04505470_3309.JPEG 2637 | n04505470_612.JPEG 2638 | n04507155_1604.JPEG 2639 | n04507155_9711.JPEG 2640 | n04507155_9932.JPEG 2641 | n04509417_4178.JPEG 2642 | n04509417_5328.JPEG 2643 | n04509417_5356.JPEG 2644 | n04515003_10783.JPEG 2645 | n04515003_23763.JPEG 2646 | n04515003_3294.JPEG 2647 | n04517823_25970.JPEG 2648 | n04517823_31159.JPEG 2649 | n04517823_7213.JPEG 2650 | n04522168_17756.JPEG 2651 | n04522168_8712.JPEG 2652 | n04522168_9957.JPEG 2653 | n04523525_16135.JPEG 2654 | n04523525_42908.JPEG 2655 | n04523525_5221.JPEG 2656 | n04525038_4024.JPEG 2657 | n04525038_46544.JPEG 2658 | n04525038_8647.JPEG 2659 | n04525305_10127.JPEG 2660 | n04525305_10815.JPEG 2661 | n04525305_8371.JPEG 2662 | n04532106_11801.JPEG 2663 | n04532106_12672.JPEG 2664 | n04532106_17401.JPEG 2665 | n04532670_17750.JPEG 2666 | n04532670_23110.JPEG 2667 | n04532670_7630.JPEG 2668 | n04536866_1605.JPEG 2669 | n04536866_37248.JPEG 2670 | n04536866_39549.JPEG 2671 | n04540053_1808.JPEG 2672 | n04540053_2468.JPEG 2673 | n04540053_2545.JPEG 2674 | n04542943_1102.JPEG 2675 | n04542943_6442.JPEG 2676 | n04542943_832.JPEG 2677 | n04548280_21.JPEG 2678 | n04548280_5721.JPEG 2679 | n04548280_6115.JPEG 2680 | n04548362_14984.JPEG 2681 | n04548362_18140.JPEG 2682 | n04548362_49031.JPEG 2683 | n04550184_5566.JPEG 2684 | n04550184_67178.JPEG 2685 | n04550184_8135.JPEG 2686 | n04552348_20020.JPEG 2687 | n04552348_23726.JPEG 2688 | n04552348_25563.JPEG 2689 | n04553703_1668.JPEG 2690 | n04553703_21296.JPEG 2691 | n04553703_5200.JPEG 2692 | n04554684_12721.JPEG 2693 | n04554684_452.JPEG 2694 | n04554684_9164.JPEG 2695 | n04557648_20590.JPEG 2696 | n04557648_2063.JPEG 2697 | n04557648_376.JPEG 2698 | n04560804_1835.JPEG 2699 | n04560804_2205.JPEG 2700 | n04560804_6654.JPEG 2701 | n04562935_10914.JPEG 2702 | n04562935_3449.JPEG 2703 | n04562935_6940.JPEG 2704 | n04579145_1546.JPEG 2705 | n04579145_6697.JPEG 2706 | n04579145_7128.JPEG 2707 | n04579432_11493.JPEG 2708 | n04579432_3265.JPEG 2709 | n04579432_34682.JPEG 2710 | n04584207_4008.JPEG 2711 | n04584207_5870.JPEG 2712 | n04584207_6806.JPEG 2713 | n04589890_1750.JPEG 2714 | n04589890_218.JPEG 2715 | n04589890_503.JPEG 2716 | n04590129_17533.JPEG 2717 | n04590129_2112.JPEG 2718 | n04590129_22042.JPEG 2719 | n04591157_3259.JPEG 2720 | n04591157_3512.JPEG 2721 | n04591157_5804.JPEG 2722 | n04591713_2010.JPEG 2723 | n04591713_2534.JPEG 2724 | n04591713_3484.JPEG 2725 | n04592741_13607.JPEG 2726 | n04592741_13802.JPEG 2727 | n04592741_24885.JPEG 2728 | n04596742_13438.JPEG 2729 | n04596742_3682.JPEG 2730 | n04596742_42399.JPEG 2731 | n04597913_5850.JPEG 2732 | n04597913_8721.JPEG 2733 | n04597913_9953.JPEG 2734 | n04599235_19073.JPEG 2735 | n04599235_25152.JPEG 2736 | n04599235_3961.JPEG 2737 | n04604644_17940.JPEG 2738 | n04604644_37324.JPEG 2739 | n04604644_4035.JPEG 2740 | n04606251_554.JPEG 2741 | n04606251_9737.JPEG 2742 | n04606251_9981.JPEG 2743 | n04612504_4440.JPEG 2744 | n04612504_5575.JPEG 2745 | n04612504_6937.JPEG 2746 | n04613696_125.JPEG 2747 | n04613696_617.JPEG 2748 | n04613696_642.JPEG 2749 | n06359193_13297.JPEG 2750 | n06359193_38643.JPEG 2751 | n06359193_91738.JPEG 2752 | n06596364_10323.JPEG 2753 | n06596364_2289.JPEG 2754 | n06596364_3569.JPEG 2755 | n06785654_13482.JPEG 2756 | n06785654_17482.JPEG 2757 | n06785654_29332.JPEG 2758 | n06794110_1713.JPEG 2759 | n06794110_2416.JPEG 2760 | n06794110_2797.JPEG 2761 | n06874185_116404.JPEG 2762 | n06874185_20178.JPEG 2763 | n06874185_26729.JPEG 2764 | n07248320_12004.JPEG 2765 | n07248320_24630.JPEG 2766 | n07248320_3026.JPEG 2767 | n07565083_2354.JPEG 2768 | n07565083_3060.JPEG 2769 | n07565083_6367.JPEG 2770 | n07579787_2530.JPEG 2771 | n07579787_3934.JPEG 2772 | n07579787_52418.JPEG 2773 | n07583066_1042.JPEG 2774 | n07583066_5386.JPEG 2775 | n07583066_7031.JPEG 2776 | n07584110_10550.JPEG 2777 | n07584110_5969.JPEG 2778 | n07584110_8679.JPEG 2779 | n07590611_3359.JPEG 2780 | n07590611_6530.JPEG 2781 | n07590611_7416.JPEG 2782 | n07613480_14779.JPEG 2783 | n07613480_17589.JPEG 2784 | n07613480_3859.JPEG 2785 | n07614500_41258.JPEG 2786 | n07614500_42305.JPEG 2787 | n07614500_61219.JPEG 2788 | n07615774_12282.JPEG 2789 | n07615774_13922.JPEG 2790 | n07615774_35494.JPEG 2791 | n07684084_1013.JPEG 2792 | n07684084_769.JPEG 2793 | n07684084_9179.JPEG 2794 | n07693725_5781.JPEG 2795 | n07693725_6635.JPEG 2796 | n07693725_6639.JPEG 2797 | n07695742_11620.JPEG 2798 | n07695742_14206.JPEG 2799 | n07695742_9118.JPEG 2800 | n07697313_2234.JPEG 2801 | n07697313_2417.JPEG 2802 | n07697313_7468.JPEG 2803 | n07697537_23415.JPEG 2804 | n07697537_31506.JPEG 2805 | n07697537_3776.JPEG 2806 | n07711569_2316.JPEG 2807 | n07711569_3669.JPEG 2808 | n07711569_4183.JPEG 2809 | n07714571_15007.JPEG 2810 | n07714571_2692.JPEG 2811 | n07714571_3321.JPEG 2812 | n07714990_11933.JPEG 2813 | n07714990_277.JPEG 2814 | n07714990_5000.JPEG 2815 | n07715103_11446.JPEG 2816 | n07715103_14703.JPEG 2817 | n07715103_17163.JPEG 2818 | n07716358_16925.JPEG 2819 | n07716358_6499.JPEG 2820 | n07716358_9112.JPEG 2821 | n07716906_1851.JPEG 2822 | n07716906_5029.JPEG 2823 | n07716906_5547.JPEG 2824 | n07717410_11221.JPEG 2825 | n07717410_2362.JPEG 2826 | n07717410_5916.JPEG 2827 | n07717556_1926.JPEG 2828 | n07717556_3128.JPEG 2829 | n07717556_594.JPEG 2830 | n07718472_11091.JPEG 2831 | n07718472_17646.JPEG 2832 | n07718472_379.JPEG 2833 | n07718747_17834.JPEG 2834 | n07718747_19295.JPEG 2835 | n07718747_4526.JPEG 2836 | n07720875_12911.JPEG 2837 | n07720875_14590.JPEG 2838 | n07720875_7031.JPEG 2839 | n07730033_4438.JPEG 2840 | n07730033_8737.JPEG 2841 | n07730033_8819.JPEG 2842 | n07734744_1527.JPEG 2843 | n07734744_2158.JPEG 2844 | n07734744_2678.JPEG 2845 | n07742313_19329.JPEG 2846 | n07742313_20689.JPEG 2847 | n07742313_648.JPEG 2848 | n07745940_12439.JPEG 2849 | n07745940_13403.JPEG 2850 | n07745940_13890.JPEG 2851 | n07747607_13236.JPEG 2852 | n07747607_13248.JPEG 2853 | n07747607_4399.JPEG 2854 | n07749582_11845.JPEG 2855 | n07749582_25729.JPEG 2856 | n07749582_5104.JPEG 2857 | n07753113_2141.JPEG 2858 | n07753113_34759.JPEG 2859 | n07753113_36208.JPEG 2860 | n07753275_19418.JPEG 2861 | n07753275_21925.JPEG 2862 | n07753275_8050.JPEG 2863 | n07753592_10118.JPEG 2864 | n07753592_22116.JPEG 2865 | n07753592_8527.JPEG 2866 | n07754684_10687.JPEG 2867 | n07754684_26830.JPEG 2868 | n07754684_7306.JPEG 2869 | n07760859_1825.JPEG 2870 | n07760859_7123.JPEG 2871 | n07760859_8325.JPEG 2872 | n07768694_27882.JPEG 2873 | n07768694_3360.JPEG 2874 | n07768694_7379.JPEG 2875 | n07802026_13834.JPEG 2876 | n07802026_18150.JPEG 2877 | n07802026_9349.JPEG 2878 | n07831146_2642.JPEG 2879 | n07831146_3115.JPEG 2880 | n07831146_8691.JPEG 2881 | n07836838_16378.JPEG 2882 | n07836838_1812.JPEG 2883 | n07836838_7292.JPEG 2884 | n07860988_22104.JPEG 2885 | n07860988_30029.JPEG 2886 | n07860988_33855.JPEG 2887 | n07871810_24220.JPEG 2888 | n07871810_4697.JPEG 2889 | n07871810_870.JPEG 2890 | n07873807_1920.JPEG 2891 | n07873807_5314.JPEG 2892 | n07873807_8243.JPEG 2893 | n07875152_1351.JPEG 2894 | n07875152_2190.JPEG 2895 | n07875152_3216.JPEG 2896 | n07880968_4073.JPEG 2897 | n07880968_5463.JPEG 2898 | n07880968_7618.JPEG 2899 | n07892512_19590.JPEG 2900 | n07892512_4812.JPEG 2901 | n07892512_876.JPEG 2902 | n07920052_13931.JPEG 2903 | n07920052_16359.JPEG 2904 | n07920052_6857.JPEG 2905 | n07930864_15445.JPEG 2906 | n07930864_19722.JPEG 2907 | n07930864_30153.JPEG 2908 | n07932039_2589.JPEG 2909 | n07932039_3494.JPEG 2910 | n07932039_7473.JPEG 2911 | n09193705_13608.JPEG 2912 | n09193705_880.JPEG 2913 | n09193705_9265.JPEG 2914 | n09229709_2111.JPEG 2915 | n09229709_23747.JPEG 2916 | n09229709_34008.JPEG 2917 | n09246464_17444.JPEG 2918 | n09246464_21608.JPEG 2919 | n09246464_43206.JPEG 2920 | n09256479_1094.JPEG 2921 | n09256479_13107.JPEG 2922 | n09256479_8080.JPEG 2923 | n09288635_14030.JPEG 2924 | n09288635_3523.JPEG 2925 | n09288635_4095.JPEG 2926 | n09332890_1196.JPEG 2927 | n09332890_2017.JPEG 2928 | n09332890_8081.JPEG 2929 | n09399592_16483.JPEG 2930 | n09399592_61233.JPEG 2931 | n09399592_62877.JPEG 2932 | n09421951_13324.JPEG 2933 | n09421951_20816.JPEG 2934 | n09421951_7074.JPEG 2935 | n09428293_12910.JPEG 2936 | n09428293_29993.JPEG 2937 | n09428293_32054.JPEG 2938 | n09468604_16715.JPEG 2939 | n09468604_19501.JPEG 2940 | n09468604_36406.JPEG 2941 | n09472597_26616.JPEG 2942 | n09472597_40331.JPEG 2943 | n09472597_4563.JPEG 2944 | n09835506_10139.JPEG 2945 | n09835506_4778.JPEG 2946 | n09835506_6051.JPEG 2947 | n10148035_15676.JPEG 2948 | n10148035_17703.JPEG 2949 | n10148035_4450.JPEG 2950 | n10565667_12741.JPEG 2951 | n10565667_23178.JPEG 2952 | n10565667_5636.JPEG 2953 | n11879895_10857.JPEG 2954 | n11879895_3555.JPEG 2955 | n11879895_9406.JPEG 2956 | n11939491_11967.JPEG 2957 | n11939491_17005.JPEG 2958 | n11939491_6422.JPEG 2959 | n12057211_4535.JPEG 2960 | n12057211_8395.JPEG 2961 | n12057211_9140.JPEG 2962 | n12144580_1668.JPEG 2963 | n12144580_19998.JPEG 2964 | n12144580_30601.JPEG 2965 | n12267677_13565.JPEG 2966 | n12267677_3132.JPEG 2967 | n12267677_7119.JPEG 2968 | n12620546_17383.JPEG 2969 | n12620546_52118.JPEG 2970 | n12620546_6582.JPEG 2971 | n12768682_11900.JPEG 2972 | n12768682_35676.JPEG 2973 | n12768682_9193.JPEG 2974 | n12985857_5120.JPEG 2975 | n12985857_7692.JPEG 2976 | n12985857_8241.JPEG 2977 | n12998815_18230.JPEG 2978 | n12998815_4793.JPEG 2979 | n12998815_9119.JPEG 2980 | n13037406_4553.JPEG 2981 | n13037406_5864.JPEG 2982 | n13037406_8206.JPEG 2983 | n13040303_10301.JPEG 2984 | n13040303_1640.JPEG 2985 | n13040303_484.JPEG 2986 | n13044778_10141.JPEG 2987 | n13044778_1612.JPEG 2988 | n13044778_1839.JPEG 2989 | n13052670_4678.JPEG 2990 | n13052670_7581.JPEG 2991 | n13052670_8877.JPEG 2992 | n13054560_13365.JPEG 2993 | n13054560_13747.JPEG 2994 | n13054560_3194.JPEG 2995 | n13133613_17314.JPEG 2996 | n13133613_28648.JPEG 2997 | n13133613_5858.JPEG 2998 | n15075141_11800.JPEG 2999 | n15075141_20918.JPEG 3000 | n15075141_2463.JPEG 3001 | -------------------------------------------------------------------------------- /data/list50.txt: -------------------------------------------------------------------------------- 1 | n03249569_39262.JPEG 2 | n01685808_5668.JPEG 3 | n07754684_10687.JPEG 4 | n04277352_10477.JPEG 5 | n02097474_1744.JPEG 6 | n02791124_4121.JPEG 7 | n03255030_1024.JPEG 8 | n07920052_13931.JPEG 9 | n04592741_13802.JPEG 10 | n02106030_17578.JPEG 11 | n02104029_831.JPEG 12 | n03124170_7976.JPEG 13 | n04099969_12218.JPEG 14 | n07718747_17834.JPEG 15 | n02088466_11599.JPEG 16 | n01978455_239.JPEG 17 | n04040759_8198.JPEG 18 | n03803284_11827.JPEG 19 | n02951358_11369.JPEG 20 | n02814533_40471.JPEG 21 | n01871265_3925.JPEG 22 | n13044778_1839.JPEG 23 | n01440764_490.JPEG 24 | n02361337_7850.JPEG 25 | n03337140_32664.JPEG 26 | n02423022_4755.JPEG 27 | n03590841_2695.JPEG 28 | n03742115_2382.JPEG 29 | n07930864_30153.JPEG 30 | n03063689_21985.JPEG 31 | n02102177_3100.JPEG 32 | n04033995_1213.JPEG 33 | n04033901_16542.JPEG 34 | n02092002_198.JPEG 35 | n07753113_36208.JPEG 36 | n02437616_5109.JPEG 37 | n02814533_94804.JPEG 38 | n01860187_1848.JPEG 39 | n04154565_24912.JPEG 40 | n06596364_2289.JPEG 41 | n04370456_28401.JPEG 42 | n03884397_22920.JPEG 43 | n04192698_43013.JPEG 44 | n02111500_1850.JPEG 45 | n03658185_7264.JPEG 46 | n02786058_3230.JPEG 47 | n02096051_4966.JPEG 48 | n02342885_888.JPEG 49 | n03095699_5301.JPEG 50 | n01496331_12448.JPEG 51 | -------------------------------------------------------------------------------- /docs/int7Inference.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/deepglint/EasyQuant/64cba1695c06f4a6b3f770c8290b5376e28ef02d/docs/int7Inference.png -------------------------------------------------------------------------------- /example/vgg16/run.sh: -------------------------------------------------------------------------------- 1 | # download vgg16 model and upgrade proto and caffemodel files 2 | sh model/vgg16/net_file_upgrade.sh 3 | 4 | # generate scale of weight and activation use quantation tools 5 | sh example/vgg16/run_scale_quantation.sh 6 | 7 | # get ncnn param bin from scale table 8 | sh example/vgg16/run_caffe2ncnn.sh 9 | 10 | # infer layer blob shape 11 | sh example/vgg16/run_infer_shape.sh 12 | 13 | # run scale fine tuning 14 | sh example/vgg16/run_scale_fine_tuning.sh 15 | 16 | # run validtion on imagenet val 17 | sh example/vgg16/run_validation.sh 18 | -------------------------------------------------------------------------------- /example/vgg16/run_caffe2ncnn.sh: -------------------------------------------------------------------------------- 1 | caffe2ncnn model/vgg16/VGG_ILSVRC_16_layers_deploy_new.prototxt model/vgg16/VGG_ILSVRC_16_layers_new.caffemodel model/vgg16/VGG.param model/vgg16/VGG.bin 256 model/vgg16/vgg16.88.scale.table 2 | -------------------------------------------------------------------------------- /example/vgg16/run_infer_shape.sh: -------------------------------------------------------------------------------- 1 | python tools/infer_shape.py --proto model/vgg16/VGG_ILSVRC_16_layers_deploy_new.prototxt --model model/vgg16/VGG_ILSVRC_16_layers_new.caffemodel --save model/vgg16/layerDims.pickle 2 | -------------------------------------------------------------------------------- /example/vgg16/run_scale_fine_tuning.sh: -------------------------------------------------------------------------------- 1 | python tools/scale_fine_tuning.py --proto model/vgg16/VGG_ILSVRC_16_layers_deploy_new.prototxt --model model/vgg16/VGG_ILSVRC_16_layers_new.caffemodel --param model/vgg16/VGG.param --bin model/vgg16/VGG.bin --table scale_table/vgg16.88.scale.table --layerDims model/vgg16/layerDims.pickle --search-log model/vgg16/vgg16_int88_search.log 2 | -------------------------------------------------------------------------------- /example/vgg16/run_scale_quantation.sh: -------------------------------------------------------------------------------- 1 | python tools/caffe_quanttable_e2e.py --proto=model/vgg16/VGG_ILSVRC_16_layers_deploy_new.prototxt --model=model/vgg16/VGG_ILSVRC_16_layers_new.caffemodel --mean 103.94 116.78 123.68 --norm=1.0 --images=data/calib --output=model/vgg16/vgg16.88.scale.table --gpu 1 --threshold 102400 --histgram True --cv2 True 2 | -------------------------------------------------------------------------------- /example/vgg16/run_validation.sh: -------------------------------------------------------------------------------- 1 | python tools/ncnn_val_caffe.py --param_path model/vgg16/VGG.param --bin_path model/vgg16/VGG.bin --platform ncnn --num_workers 16 2 | -------------------------------------------------------------------------------- /model/vgg16/net_file_upgrade.sh: -------------------------------------------------------------------------------- 1 | # download vgg16 model files from model pool 2 | wget -P model/vgg16/ https://gist.githubusercontent.com/ksimonyan/211839e770f7b538e2d8/raw/ded9363bd93ec0c770134f4e387d8aaaaa2407ce/VGG_ILSVRC_16_layers_deploy.prototxt 3 | wget -P model/vgg16/ http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_16_layers.caffemodel 4 | 5 | # upgrade proto and bin model file 6 | caffe/build/tools/upgrade_net_proto_text model/vgg16/VGG_ILSVRC_16_layers_deploy.prototxt model/vgg16/VGG_ILSVRC_16_layers_deploy_new.prototxt 7 | caffe/build/tools/upgrade_net_proto_binary model/vgg16/VGG_ILSVRC_16_layers.caffemodel model/vgg16/VGG_ILSVRC_16_layers_new.caffemodel 8 | -------------------------------------------------------------------------------- /python_ncnn/Makefile: -------------------------------------------------------------------------------- 1 | BOOST_INCLUDE_PATH=/usr/include 2 | BOOST_LIB_PATH=/usr/lib/x86_64-linux-gnu 3 | NCNN_INCLUDE_PATH=/train/trainset/1/docker_run/ncnn/build/install/include/ncnn 4 | NCNN_LIB_PATH=/train/trainset/1/docker_run/ncnn/build/install/lib 5 | PYTHON_LIB_PATH=/usr/lib/x86_64-linux-gnu 6 | PYTHON_INCLUDE_PATH=/usr/include/python2.7 7 | NUMPY_INCLUDE_PATH=/usr/local/lib/python2.7/dist-packages/numpy/core/include 8 | 9 | all: 10 | g++ -fPIC -fpermissive -I ${BOOST_INCLUDE_PATH} -I ${PYTHON_INCLUDE_PATH} -I ${NUMPY_INCLUDE_PATH} -I ${NCNN_INCLUDE_PATH} -c ncnn.cpp 11 | g++ -fPIC -fpermissive -I ${BOOST_INCLUDE_PATH} -I ${PYTHON_INCLUDE_PATH} -I ${NUMPY_INCLUDE_PATH} -I ${NCNN_INCLUDE_PATH} -c ncnn2python.cpp 12 | g++ -shared -o ncnn.so ncnn.o ncnn2python.o -L ${BOOST_LIB_PATH} -lboost_python -L ${PYTHON_LIB_PATH} -lpython2.7 -L ${NCNN_LIB_PATH} -lncnn -lgomp 13 | cp ncnn.so /usr/local/lib/python2.7/dist-packages/ 14 | 15 | 16 | clean: 17 | rm ncnn.so ncnn.o ncnn2python.o 18 | -------------------------------------------------------------------------------- /python_ncnn/README.md: -------------------------------------------------------------------------------- 1 | ## 使用步骤 2 | 3 | 1. 安装依赖包 4 | 5 | ``` 6 | sudo apt-get install libboost-python-dev 7 | ``` 8 | 9 | 2. 修改 Makefile 开头的变量路径； 10 | 11 | 3. 执行 make 命令编译 ncnn.so； 12 | 13 | 4. 把 ncnn.so 放到 site-packages 目录下( /home/zyy/anaconda3/lib/python3.7/site-packages )，在 python 环境下 import ncnn 进行测试，不报错即加载成功。 14 | -------------------------------------------------------------------------------- /python_ncnn/ncnn.cpp: -------------------------------------------------------------------------------- 1 | // BSD 3-Clause License 2 | // 3 | // DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | // Copyright (C) 2020 DeepGlint. All rights reserved. 5 | // 6 | // Redistribution and use in source and binary forms, with or without 7 | // modification, are permitted provided that the following conditions are met: 8 | // 9 | // * Redistributions of source code must retain the above copyright notice, this 10 | // list of conditions and the following disclaimer. 11 | // 12 | // * Redistributions in binary form must reproduce the above copyright notice, 13 | // this list of conditions and the following disclaimer in the documentation 14 | // and/or other materials provided with the distribution. 15 | // 16 | // * Neither the name of the copyright holder nor the names of its 17 | // contributors may be used to endorse or promote products derived from 18 | // this software without specific prior written permission. 19 | // 20 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | // SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | // CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | // OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | #include 32 | #include 33 | 34 | #include 35 | #include 36 | #include 37 | 38 | #include 39 | #include 40 | #include 41 | #include 42 | 43 | #include "ncnn.h" 44 | 45 | static ncnn::UnlockedPoolAllocator g_blob_pool_allocator; 46 | static ncnn::PoolAllocator g_workspace_pool_allocator; 47 | 48 | 49 | Net::Net() 50 | { 51 | g_blob_pool_allocator.set_size_compare_ratio(0.0f); 52 | g_workspace_pool_allocator.set_size_compare_ratio(0.5f); 53 | ncnn::Option opt; 54 | opt.lightmode = true; 55 | opt.num_threads = 1; 56 | opt.blob_allocator = &g_blob_pool_allocator; 57 | opt.workspace_allocator = &g_workspace_pool_allocator; 58 | 59 | // github recent release of ncnn 60 | net.opt = opt; 61 | // if use rq-ncnn opt set using this api 62 | // ncnn::set_default_option(opt); 63 | 64 | ncnn::set_cpu_powersave(1); 65 | ncnn::set_omp_dynamic(0); 66 | ncnn::set_omp_num_threads(opt.num_threads); 67 | } 68 | Net::~Net() 69 | { 70 | 71 | } 72 | int Net::load_param(const char * paramPath) 73 | { 74 | return net.load_param(paramPath); 75 | } 76 | 77 | int Net::load_model(const char * modelPath) 78 | { 79 | return net.load_model(modelPath); 80 | } 81 | 82 | void Net::setInputBlobName(string name) 83 | { 84 | inputBlobNmae = name; 85 | 86 | } 87 | void Net::setOutputBlobName(string name) 88 | { 89 | outputBlobName = name; 90 | } 91 | 92 | int Net::inference(object & input_object,object & output_object,int inputHeight,int inputWidth) 93 | { 94 | 95 | PyArrayObject* input_data_arr = reinterpret_cast(input_object.ptr()); 96 | float * input = static_cast(PyArray_DATA(input_data_arr)); 97 | 98 | PyArrayObject* output_data_arr = reinterpret_cast(output_object.ptr()); 99 | float * output = static_cast(PyArray_DATA(output_data_arr)); 100 | 101 | ncnn::Mat in = ncnn::Mat(inputWidth,inputHeight,3,4u); 102 | 103 | memcpy(in.channel(2),input,sizeof(float)*inputWidth*inputHeight); 104 | memcpy(in.channel(1),input+inputWidth*inputHeight,sizeof(float)*inputWidth*inputHeight); 105 | memcpy(in.channel(0),input+2*inputWidth*inputHeight,sizeof(float)*inputWidth*inputHeight); 106 | 107 | ncnn::Extractor ex = net.create_extractor(); 108 | ex.set_light_mode(true); 109 | // ex.set_num_threads(4); 110 | ex.input(inputBlobNmae.c_str(), in); 111 | ncnn::Mat out; 112 | ex.extract(outputBlobName.c_str(),out); 113 | 114 | for(int i = 0;i(input_object.ptr()); 159 | float * input = static_cast(PyArray_DATA(input_data_arr)); 160 | 161 | ncnn::Mat in = ncnn::Mat(inputWidth,inputHeight,3,4u); 162 | 163 | memcpy(in.channel(2),input,sizeof(float)*inputWidth*inputHeight); 164 | memcpy(in.channel(1),input+inputWidth*inputHeight,sizeof(float)*inputWidth*inputHeight); 165 | memcpy(in.channel(0),input+2*inputWidth*inputHeight,sizeof(float)*inputWidth*inputHeight); 166 | 167 | ncnn::Extractor ex = net.create_extractor(); 168 | ex.set_light_mode(true); 169 | // ex.set_num_threads(4); 170 | 171 | 172 | //输出各个层的结果到文件 173 | for(int layer_index = 1;layer_index<184;layer_index++) 174 | { 175 | char layerName[128] = {'\0'}; 176 | sprintf(layerName, "ConvNd_%d", layer_index); 177 | ex.input(inputBlobNmae.c_str(), in); 178 | ncnn::Mat out; 179 | ex.extract(layerName,out); 180 | extract_feature_blob_f32_debug("debug",layerName,out); 181 | } 182 | 183 | 184 | return 0; 185 | 186 | } 187 | /* 188 | * Extract the blob feature map 189 | */ 190 | void extract_feature_blob_f32_debug(const char* comment, const char* layer_name, const ncnn::Mat& blob) 191 | { 192 | char file_path_output[128] = {'\0'}; 193 | char file_dir[128] = {'\0'}; 194 | 195 | FILE *pFile = NULL; 196 | 197 | std::string name = layer_name; 198 | 199 | sprintf(file_dir, "./output/"); 200 | mkdir(file_dir, 0777); 201 | 202 | sprintf(file_path_output, "./output/%s_%s_blob_data.txt", name.c_str(), comment); 203 | 204 | pFile = fopen(file_path_output,"w"); 205 | if(pFile == NULL) 206 | { 207 | printf("open file error!\n"); 208 | } 209 | 210 | int channel_num = blob.c; 211 | 212 | //save top feature maps 213 | for(int k = 0; k < channel_num; k++) 214 | { 215 | fprintf(pFile, "blob channel %d:\n", k); 216 | 217 | //float *data = top_blob.data + top_blob.cstep*k; 218 | const float *data = blob.channel(k); 219 | for(int i = 0; i < blob.h; i++) 220 | { 221 | for(int j = 0; j < blob.w; j++) 222 | { 223 | fprintf(pFile, "%s%8.6f ", (data[j]<0)?"":" ", data[j]); 224 | } 225 | fprintf(pFile, "\n"); 226 | data += blob.w; 227 | } 228 | fprintf(pFile, "\n"); 229 | } 230 | 231 | //close file 232 | fclose(pFile); 233 | pFile = NULL; 234 | } 235 | -------------------------------------------------------------------------------- /python_ncnn/ncnn.h: -------------------------------------------------------------------------------- 1 | // BSD 3-Clause License 2 | // 3 | // DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | // Copyright (C) 2020 DeepGlint. All rights reserved. 5 | // 6 | // Redistribution and use in source and binary forms, with or without 7 | // modification, are permitted provided that the following conditions are met: 8 | // 9 | // * Redistributions of source code must retain the above copyright notice, this 10 | // list of conditions and the following disclaimer. 11 | // 12 | // * Redistributions in binary form must reproduce the above copyright notice, 13 | // this list of conditions and the following disclaimer in the documentation 14 | // and/or other materials provided with the distribution. 15 | // 16 | // * Neither the name of the copyright holder nor the names of its 17 | // contributors may be used to endorse or promote products derived from 18 | // this software without specific prior written permission. 19 | // 20 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | // SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | // CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | // OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | #include "net.h" 32 | #include "cpu.h" 33 | #include "benchmark.h" 34 | #include 35 | #include 36 | #include 37 | 38 | using namespace boost::python; 39 | namespace bp = boost::python; 40 | using namespace std; 41 | 42 | 43 | class Net{ 44 | public: 45 | Net(); 46 | ~Net(); 47 | int load_param(const char * paramPath); 48 | int load_model(const char * modelPath); 49 | void setInputBlobName(string name); 50 | void setOutputBlobName(string name); 51 | int inference(object & input_object,object & output_object,int inputHeight,int inputWidth); 52 | int inference_debug_writeOutputBlob2File(object & input_object,object & output_object,int inputHeight,int inputWidth); 53 | private: 54 | ncnn::Net net; 55 | string inputBlobNmae; 56 | string outputBlobName; 57 | 58 | }; -------------------------------------------------------------------------------- /python_ncnn/ncnn2python.cpp: -------------------------------------------------------------------------------- 1 | // BSD 3-Clause License 2 | // 3 | // DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | // Copyright (C) 2020 DeepGlint. All rights reserved. 5 | // 6 | // Redistribution and use in source and binary forms, with or without 7 | // modification, are permitted provided that the following conditions are met: 8 | // 9 | // * Redistributions of source code must retain the above copyright notice, this 10 | // list of conditions and the following disclaimer. 11 | // 12 | // * Redistributions in binary form must reproduce the above copyright notice, 13 | // this list of conditions and the following disclaimer in the documentation 14 | // and/or other materials provided with the distribution. 15 | // 16 | // * Neither the name of the copyright holder nor the names of its 17 | // contributors may be used to endorse or promote products derived from 18 | // this software without specific prior written permission. 19 | // 20 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | // AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | // FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | // SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | // CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | // OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | #include "ncnn.h" 32 | 33 | BOOST_PYTHON_MODULE(ncnn) 34 | { 35 | class_("net",init<>()) 36 | .def("load_param",&Net::load_param) 37 | .def("load_model",&Net::load_model) 38 | .def("setInputBlobName",&Net::setInputBlobName) 39 | .def("setOutputBlobName",&Net::setOutputBlobName) 40 | .def("inference",&Net::inference) 41 | .def("inference_debug_writeOutputBlob2File",&Net::inference_debug_writeOutputBlob2File) 42 | ; 43 | } -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | easydict==1.9 2 | graphviz==0.8.4 3 | leveldb==0.194 4 | matplotlib==2.2.4 5 | numpy==1.16.3 6 | Pillow==6.0.0 7 | protobuf==2.6.1 8 | python-gflags==3.1.2 9 | scikit-image==0.14.2 10 | scikit-learn==0.20.4 11 | scipy==0.17.0 12 | six==1.12.0 13 | tqdm==4.36.1 14 | urllib3==1.25.6 15 | -------------------------------------------------------------------------------- /tools/caffe_quanttable_e2e.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # BSD 3-Clause License 3 | # 4 | # DeepGlint is pleased to support the open source community by making EasyQuant available. 5 | # Copyright (C) 2020 DeepGlint. All rights reserved. 6 | # 7 | # Redistribution and use in source and binary forms, with or without 8 | # modification, are permitted provided that the following conditions are met: 9 | # 10 | # * Redistributions of source code must retain the above copyright notice, this 11 | # list of conditions and the following disclaimer. 12 | # 13 | # * Redistributions in binary form must reproduce the above copyright notice, 14 | # this list of conditions and the following disclaimer in the documentation 15 | # and/or other materials provided with the distribution. 16 | # 17 | # * Neither the name of the copyright holder nor the names of its 18 | # contributors may be used to endorse or promote products derived from 19 | # this software without specific prior written permission. 20 | # 21 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 22 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 23 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 24 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 25 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 26 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 27 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 28 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 29 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 30 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 31 | 32 | # Modified from https://github.com/BUG1989/caffe-int8-convert-tools 33 | 34 | # BUG1989 is pleased to support the open source community by supporting ncnn available. 35 | # 36 | # Copyright (C) 2019 BUG1989. All rights reserved. 37 | # 38 | # Licensed under the BSD 3-Clause License (the "License"); you may not use this file except 39 | # in compliance with the License. You may obtain a copy of the License at 40 | # 41 | # https://opensource.org/licenses/BSD-3-Clause 42 | # 43 | # Unless required by applicable law or agreed to in writing, software distributed 44 | # under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 45 | # CONDITIONS OF ANY KIND, either express or implied. See the License for the 46 | # specific language governing permissions and limitations under the License. 47 | 48 | 49 | """ 50 | Quantization module for generating the calibration tables will be used by 51 | quantized (INT8) models from FP32 models.with bucket split,[k, k, cin, cout] 52 | cut into "cout" buckets. 53 | This tool is based on Caffe Framework. 54 | """ 55 | from __future__ import division 56 | from __future__ import print_function 57 | import argparse 58 | import numpy as np 59 | import math, copy 60 | import matplotlib.pyplot as plt 61 | import sys,os 62 | import caffe 63 | import caffe.proto.caffe_pb2 as caffe_pb2 64 | import time 65 | import datetime 66 | from google.protobuf import text_format 67 | from scipy import stats 68 | import cv2 69 | 70 | # np.set_printoptions(threshold='nan') 71 | np.set_printoptions(suppress=True) 72 | 73 | def parse_args(): 74 | parser = argparse.ArgumentParser( 75 | description='find the pretrained caffe models int8 quantize scale value') 76 | parser.add_argument('--proto', dest='proto', 77 | help="path to deploy prototxt.", type=str) 78 | parser.add_argument('--model', dest='model', 79 | help='path to pretrained weights', type=str) 80 | parser.add_argument('--mean', dest='mean', 81 | help='value of mean', type=float, nargs=3) 82 | parser.add_argument('--norm', dest='norm', 83 | help='value of normalize', type=float, nargs=1, default=1.0) 84 | parser.add_argument('--images', dest='images', 85 | help='path to calibration images', type=str) 86 | parser.add_argument('--output', dest='output', 87 | help='path to output calibration table file', type=str, default='calibration-dev.table') 88 | parser.add_argument('--group', dest='group', 89 | help='enable the group scale', type=int, default=1) 90 | parser.add_argument('--gpu', dest='gpu', 91 | help='use gpu to forward', type=int, default=0) 92 | parser.add_argument('--threshold', dest='threshold', 93 | help='the threshold of activations', type=float, default=float('inf')) 94 | parser.add_argument('--histgram', dest='histgram', 95 | help='whether to generate activation histograms', type=bool, default=False) 96 | parser.add_argument('--cv2', dest='cv2',help='whether use opencv read image', type=bool, default=False) 97 | args = parser.parse_args() 98 | return args, parser 99 | 100 | 101 | global args, parser 102 | args, parser = parse_args() 103 | 104 | 105 | # global params 106 | QUANTIZE_NUM = 127# 7bit 107 | QUANTIZE_WINOGRAND_NUM = 127 # 6bit 108 | STATISTIC = 1 109 | INTERVAL_NUM = 2001 110 | 111 | # ugly global params 112 | quantize_layer_lists = [] 113 | 114 | def image_processing(image, image_size, mean_value): 115 | w = image.shape[1] 116 | h = image.shape[0] 117 | m = min(w, h) 118 | ratio = 256.0 / m 119 | new_w, new_h = int(ratio * w), int(ratio * h) 120 | image = cv2.resize(image, (new_w, new_h)) 121 | image = image.astype(np.float32) 122 | top = (new_w - image_size)//2 123 | left = (new_h - image_size)//2 124 | image = image[left:left+image_size, top:top+image_size] 125 | image = image - mean_value 126 | image = image.transpose(2, 0, 1) 127 | 128 | return image # bgr, chw, normalized 129 | 130 | class QuantizeLayer: 131 | def __init__(self, name, blob_name, group_num): 132 | self.name = name 133 | self.blob_name = blob_name 134 | self.group_num = group_num 135 | self.weight_scale = np.zeros(group_num) 136 | self.blob_max = 0.0 137 | self.blob_distubution_interval = 0.0 138 | self.blob_distubution = np.zeros(INTERVAL_NUM) 139 | self.blob_distubution_edges= np.zeros(INTERVAL_NUM) 140 | self.blob_threshold = 0 141 | self.blob_scale = 1.0 142 | self.group_zero = np.zeros(group_num) 143 | self.pc= True 144 | 145 | def quantize_weight(self, weight_data, flag): 146 | # spilt the weight data by cout num 147 | blob_group_data = np.array_split(weight_data, self.group_num) 148 | #add by diwu 149 | glob_group_max= np.max(weight_data) 150 | glob_group_min= np.min(weight_data) 151 | glob_group_threshold = max(abs(glob_group_max), abs(glob_group_min)) 152 | for i, group_data in enumerate(blob_group_data): 153 | #per channel quant 154 | if self.pc: 155 | max_val = np.max(group_data) 156 | min_val = np.min(group_data) 157 | threshold = max(abs(max_val), abs(min_val)) 158 | if threshold < 0.0001: 159 | self.weight_scale[i] = 0 160 | self.group_zero[i] = 1 161 | else: 162 | if(flag == True): 163 | self.weight_scale[i] = QUANTIZE_WINOGRAND_NUM / threshold 164 | else: 165 | self.weight_scale[i] = QUANTIZE_NUM / threshold 166 | print("%-20s group : %-5d max_val : %-10f scale_val : %-10f" % (self.name + "_param0", i, threshold, self.weight_scale[i])) 167 | else: 168 | if glob_group_threshold < 0.0001: 169 | self.weight_scale[i] = 0 170 | self.group_zero[i] = 1 171 | else: 172 | if(flag == True): 173 | self.weight_scale[i] = QUANTIZE_WINOGRAND_NUM / glob_group_threshold 174 | else: 175 | self.weight_scale[i] = QUANTIZE_NUM / glob_group_threshold 176 | print("%-20s group : %-5d max_val : %-10f scale_val : %-10f" % (self.name + "_param0", i, glob_group_threshold, self.weight_scale[i])) 177 | 178 | def initial_blob_max(self, blob_data): 179 | # get the max value of blob 180 | max_val = np.max(blob_data) 181 | min_val = np.min(blob_data) 182 | self.blob_max = max(self.blob_max, max(abs(max_val), abs(min_val))) 183 | # Avoid unusually large activation by clip blob_max with threshold 184 | self.th= min(self.blob_max, args.threshold) 185 | 186 | def initial_blob_distubution_interval(self): 187 | self.blob_distubution_interval = STATISTIC * self.th / INTERVAL_NUM 188 | print("%-20s max_val : %-10.8f distribution_intervals : %-10.8f" % (self.name, self.blob_max, self.blob_distubution_interval)) 189 | 190 | def initial_histograms(self, blob_data): 191 | # collect histogram of every group channel blob 192 | th= self.th 193 | # Truncate the boundary of the active hist graph, 194 | # so the number exceeding the boundary value will not fall into statistics. 195 | # add by diwu 196 | hist, hist_edge = np.histogram(blob_data, bins=INTERVAL_NUM, range=(0, th)) 197 | self.blob_distubution_edges = hist_edge 198 | self.blob_distubution += hist 199 | 200 | def quantize_blob(self): 201 | # calculate threshold 202 | distribution = np.array(self.blob_distubution) 203 | # pick threshold which minimizes KL divergence 204 | threshold_bin = threshold_distribution(distribution) 205 | self.blob_threshold = threshold_bin 206 | threshold = (threshold_bin + 0.5) * self.blob_distubution_interval 207 | # get the activation calibration value 208 | self.blob_scale = QUANTIZE_NUM / threshold 209 | #self.blob_scale = np.max(self.blob_scale,1) #add by diwu 210 | print("%-20s bin : %-8d threshold : %-10f interval : %-10f scale : %-10f" % (self.name, threshold_bin, threshold, self.blob_distubution_interval, self.blob_scale)) 211 | 212 | 213 | def _smooth_distribution(p, eps=0.0001): 214 | """Given a discrete distribution (may have not been normalized to 1), 215 | smooth it by replacing zeros with eps multiplied by a scaling factor and taking the 216 | corresponding amount off the non-zero values. 217 | Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf 218 | """ 219 | is_zeros = (p == 0).astype(np.float32) 220 | is_nonzeros = (p != 0).astype(np.float32) 221 | n_zeros = is_zeros.sum() 222 | n_nonzeros = p.size - n_zeros 223 | if not n_nonzeros: 224 | raise ValueError('The discrete probability distribution is malformed. All entries are 0.') 225 | eps1 = eps * float(n_zeros) / float(n_nonzeros) 226 | assert eps1 < 1.0, 'n_zeros=%d, n_nonzeros=%d, eps1=%f' % (n_zeros, n_nonzeros, eps1) 227 | hist = p.astype(np.float32) 228 | hist += eps * is_zeros + (-eps1) * is_nonzeros 229 | assert (hist <= 0).sum() == 0 230 | return hist 231 | 232 | 233 | def threshold_distribution(distribution, target_bin=128): 234 | """ 235 | Return the best threshold value. 236 | Ref: https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py 237 | Args: 238 | distribution: list, activations has been processed by histogram and normalize,size is 2048 239 | target_bin: int, the num of bin that is used by quantize, Int8 default value is 128 240 | Returns: 241 | target_threshold: int, num of bin with the minimum KL 242 | """ 243 | distribution = distribution[1:] 244 | length = distribution.size 245 | threshold_sum = sum(distribution[target_bin:]) 246 | kl_divergence = np.zeros(length - target_bin) 247 | 248 | for threshold in range(target_bin, length): 249 | sliced_nd_hist = copy.deepcopy(distribution[:threshold]) 250 | 251 | # generate reference distribution p 252 | p = sliced_nd_hist.copy() 253 | p[threshold-1] += threshold_sum 254 | threshold_sum = threshold_sum - distribution[threshold] 255 | 256 | # is_nonzeros[k] indicates whether hist[k] is nonzero 257 | is_nonzeros = (p != 0).astype(np.int64) 258 | # 259 | quantized_bins = np.zeros(target_bin, dtype=np.int64) 260 | # calculate how many bins should be merged to generate quantized distribution q 261 | num_merged_bins = sliced_nd_hist.size // target_bin 262 | 263 | # merge hist into num_quantized_bins bins 264 | for j in range(target_bin): 265 | start = j * num_merged_bins 266 | stop = start + num_merged_bins 267 | quantized_bins[j] = sliced_nd_hist[start:stop].sum() 268 | quantized_bins[-1] += sliced_nd_hist[target_bin * num_merged_bins:].sum() 269 | 270 | # expand quantized_bins into p.size bins 271 | q = np.zeros(sliced_nd_hist.size, dtype=np.float64) 272 | for j in range(target_bin): 273 | start = j * num_merged_bins 274 | if j == target_bin - 1: 275 | stop = -1 276 | else: 277 | stop = start + num_merged_bins 278 | norm = is_nonzeros[start:stop].sum() 279 | if norm != 0: 280 | q[start:stop] = float(quantized_bins[j]) / float(norm) 281 | q[p == 0] = 0 282 | p = _smooth_distribution(p) # with some bugs, need to fix 283 | q = _smooth_distribution(q) 284 | p[p == 0] = 0.0001 285 | q[q == 0] = 0.0001 286 | 287 | # calculate kl_divergence between q and p 288 | kl_divergence[threshold - target_bin] = stats.entropy(p, q) 289 | 290 | min_kl_divergence = np.argmin(kl_divergence) 291 | threshold_value = min_kl_divergence + target_bin 292 | 293 | return threshold_value 294 | 295 | 296 | def net_forward(net, image_path, transformer=None, image_size=224, mean_value=[103.939, 116.779, 123.68]): 297 | """ 298 | network inference and statistics the cost time 299 | Args: 300 | net: the instance of Caffe inference 301 | image_path: a image need to be inference 302 | transformer: caffe io transformar 303 | image_size: image shape of blob data 304 | mean_value: mean value for normalization 305 | Returns: 306 | none 307 | """ 308 | if args.cv2: 309 | # load image 310 | image = cv2.imread(image_path) 311 | image = image_processing(image, image_size, mean_value) 312 | net.blobs['data'].reshape(1, 3, image_size, image_size) 313 | net.blobs['data'].data[...] = np.array([image], dtype=np.float32) 314 | else: 315 | # load image 316 | im = caffe.io.load_image(image_path) 317 | nh, nw = 224, 224 318 | h, w, _ = im.shape 319 | if h < w: 320 | off = int((w - h) / 2) 321 | im = im[:, off:off + h] 322 | else: 323 | off = int((h - w) / 2) 324 | im = im[off:off + h, :] 325 | im = caffe.io.resize_image(im, [nh, nw]) 326 | # transformer.preprocess the image 327 | net.blobs['data'].data[...] = transformer.preprocess('data', im) 328 | 329 | # net forward 330 | output = net.forward() 331 | 332 | 333 | def file_name(file_dir): 334 | """ 335 | Find the all file path with the directory 336 | Args: 337 | file_dir: The source file directory 338 | Returns: 339 | files_path: all the file path into a list 340 | """ 341 | files_path = [] 342 | 343 | for root, dir, files in os.walk(file_dir): 344 | for name in files: 345 | file_path = root + "/" + name 346 | print(file_path) 347 | files_path.append(file_path) 348 | 349 | return files_path 350 | 351 | 352 | def network_prepare(net, mean, norm): 353 | """ 354 | instance the prepare process param of caffe network inference 355 | Args: 356 | net: the instance of Caffe inference 357 | mean: the value of mean 358 | norm: the value of normalize 359 | Returns: 360 | none 361 | """ 362 | print("Network initial") 363 | 364 | img_mean = np.array(mean, dtype=np.float32) 365 | 366 | # initial transformer 367 | transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape}) 368 | # convert hwc to cwh 369 | transformer.set_transpose('data', (2,0,1)) 370 | # convert RGB -> BGR 371 | transformer.set_channel_swap('data', (2,1,0)) 372 | # resize image data from [0,1] to [0,255] 373 | transformer.set_raw_scale('data', 255) 374 | # load meanfile 375 | transformer.set_mean('data', img_mean) 376 | # normalize 377 | transformer.set_input_scale('data', norm) 378 | 379 | return transformer 380 | 381 | 382 | def weight_quantize(net, net_file, group_on, winograd=False): 383 | """ 384 | CaffeModel convolution weight blob Int8 quantize 385 | Args: 386 | net: the instance of Caffe inference 387 | net_file: deploy caffe prototxt 388 | Returns: 389 | none 390 | """ 391 | print("\nQuantize the kernel weight:") 392 | 393 | # parse the net param from deploy prototxt 394 | params = caffe_pb2.NetParameter() 395 | with open(net_file) as f: 396 | text_format.Merge(f.read(), params) 397 | 398 | for i, layer in enumerate(params.layer): 399 | # find the convolution layers to get out the weight_scale 400 | if(layer.type == "Convolution" or layer.type == "ConvolutionDepthwise"): 401 | weight_blob = net.params[layer.name][0].data 402 | # initial the instance of QuantizeLayer Class lists,you can use enable group quantize to generate int8 scale for each group layer.convolution_param.group 403 | if (group_on == 1): 404 | quanitze_layer = QuantizeLayer(layer.name, layer.bottom[0], layer.convolution_param.num_output) 405 | else: 406 | quanitze_layer = QuantizeLayer(layer.name, layer.bottom[0], 1) 407 | 408 | if not winograd: 409 | # quantize the weight value using QUANTIZE_WINOGRAND_NUM for all layers 410 | quanitze_layer.quantize_weight(weight_blob, True) 411 | else: 412 | # quantize the weight value using 6bit for conv3x3s1 layer to winograd F(4,3) 413 | if(layer.type == "Convolution" and layer.convolution_param.kernel_size[0] == 3 and ((len(layer.convolution_param.stride) == 0) or layer.convolution_param.stride[0] == 1)): 414 | if(layer.convolution_param.group != layer.convolution_param.num_output): 415 | quanitze_layer.quantize_weight(weight_blob, True) 416 | else: 417 | quanitze_layer.quantize_weight(weight_blob, False) 418 | # quantize the weight value using 8bit for another conv layers 419 | else: 420 | quanitze_layer.quantize_weight(weight_blob, False) 421 | 422 | # add the quantize_layer into the save list 423 | quantize_layer_lists.append(quanitze_layer) 424 | 425 | return None 426 | 427 | 428 | def activation_quantize(net, transformer, images_files): 429 | """ 430 | Activation Int8 quantize, optimaize threshold selection with KL divergence, 431 | given a dataset, find the optimal threshold for quantizing it. 432 | Ref: http://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf 433 | Args: 434 | net: the instance of Caffe inference 435 | transformer: 436 | images_files: calibration dataset 437 | Returns: 438 | none 439 | """ 440 | print("\nQuantize the Activation:") 441 | # run float32 inference on calibration dataset to find the activations range 442 | for i , image in enumerate(images_files): 443 | # inference 444 | net_forward(net, image, transformer) 445 | # find max threshold 446 | for layer in quantize_layer_lists: 447 | blob = net.blobs[layer.blob_name].data[0].flatten() 448 | layer.initial_blob_max(blob) 449 | if i % 100 == 0: 450 | print("loop stage 1 : %d/%d" % (i, len(images_files))) 451 | 452 | # calculate statistic blob scope and interval distribution 453 | for layer in quantize_layer_lists: 454 | layer.initial_blob_distubution_interval() 455 | 456 | # for each layers 457 | # collect histograms of activations 458 | print("\nCollect histograms of activations:") 459 | for i, image in enumerate(images_files): 460 | net_forward(net, image, transformer) 461 | for layer in quantize_layer_lists: 462 | blob = net.blobs[layer.blob_name].data[0].flatten() 463 | blob= blob[blob > 0] 464 | layer.initial_histograms(blob) 465 | if i % 100 == 0: 466 | print("loop stage 2 : %d/%d" % (i, len(images_files))) 467 | 468 | # calculate threshold with KL divergence 469 | if args.histgram: 470 | from collections import OrderedDict 471 | quant_hist= OrderedDict() 472 | for layer in quantize_layer_lists: 473 | layer.quantize_blob() 474 | if args.histgram: 475 | quant_hist[layer.name]= (layer.blob_max,layer.blob_distubution,layer.blob_distubution_edges,QUANTIZE_NUM/layer.blob_scale) 476 | 477 | if args.histgram: 478 | import pickle 479 | with open('histgram.pkl','wb') as f: 480 | pickle.dump(quant_hist,f) 481 | print('save histograms success! use plot script to generate graphs') 482 | 483 | return None 484 | 485 | 486 | def save_calibration_file(calibration_path): 487 | calibration_file = open(calibration_path, 'w') 488 | # save temp 489 | save_temp = [] 490 | # save weight scale 491 | for layer in quantize_layer_lists: 492 | save_string = layer.name + "_param_0" 493 | for i in range(layer.group_num): 494 | save_string = save_string + " " + str(layer.weight_scale[i]) 495 | save_temp.append(save_string) 496 | 497 | # save bottom blob scales 498 | for layer in quantize_layer_lists: 499 | save_string = layer.name + " " + str(layer.blob_scale) 500 | save_temp.append(save_string) 501 | 502 | # save into txt file 503 | for data in save_temp: 504 | calibration_file.write(data + "\n") 505 | 506 | calibration_file.close() 507 | 508 | # save calibration logs 509 | save_temp_log = [] 510 | calibration_file_log = open(calibration_path + ".log", 'w') 511 | for layer in quantize_layer_lists: 512 | save_string = layer.name + ": value range 0 - " + str(layer.blob_max) \ 513 | + ", interval " + str(layer.blob_distubution_interval) \ 514 | + ", interval num " + str(INTERVAL_NUM) \ 515 | + ", threshold num " + str(layer.blob_threshold) + "\n" \ 516 | + str(layer.blob_distubution.astype(dtype=np.int64)) 517 | save_temp_log.append(save_string) 518 | 519 | # save into txt file 520 | for data in save_temp_log: 521 | calibration_file_log.write(data + "\n") 522 | 523 | 524 | def usage_info(): 525 | """ 526 | usage info 527 | """ 528 | print("Input params is illegal...╮(╯3╰)╭") 529 | print("try it again:\n python caffe-int8-scale-tools-dev.py -h") 530 | 531 | 532 | def main(): 533 | """ 534 | main function 535 | """ 536 | 537 | # time start 538 | 539 | time_start = datetime.datetime.now() 540 | 541 | print(args) 542 | 543 | if args.proto == None or args.model == None or args.mean == None or args.images == None: 544 | usage_info() 545 | return None 546 | 547 | # deploy caffe prototxt path 548 | net_file = args.proto 549 | 550 | # trained caffemodel path 551 | caffe_model = args.model 552 | 553 | # mean value 554 | mean = args.mean 555 | 556 | # norm value 557 | norm = 1.0 558 | if args.norm != 1.0: 559 | norm = args.norm[0] 560 | 561 | # calibration dataset 562 | images_path = args.images 563 | 564 | # the output calibration file 565 | calibration_path = args.output 566 | 567 | # enable the group scale 568 | group_on = args.group 569 | 570 | # default use CPU to forwark 571 | 572 | if args.gpu != 0: 573 | caffe.set_mode_gpu() 574 | caffe.set_device(0) 575 | 576 | # initial caffe net and the forword model(GPU or CPU) 577 | net = caffe.Net(net_file,caffe_model,caffe.TEST) 578 | 579 | # prepare the cnn network 580 | transformer = network_prepare(net, mean, norm) 581 | 582 | # get the calibration datasets images files path 583 | images_files = file_name(images_path) 584 | 585 | # quanitze kernel weight of the caffemodel to find it's calibration table 586 | weight_quantize(net, net_file, group_on) 587 | 588 | # quantize activation value of the caffemodel to find it's calibration table 589 | activation_quantize(net, transformer, images_files) 590 | 591 | # save the calibration tables,best wish for your INT8 inference have low accuracy loss :) 592 | save_calibration_file(calibration_path) 593 | 594 | # time end 595 | time_end = datetime.datetime.now() 596 | 597 | print("\nCaffe Int8 Calibration table create success, it's cost %s, best wish for your INT8 inference has a low accuracy loss...\(^▽^)/...2333..." % (time_end - time_start)) 598 | 599 | if __name__ == "__main__": 600 | main() 601 | -------------------------------------------------------------------------------- /tools/hist_tool.py: -------------------------------------------------------------------------------- 1 | # BSD 3-Clause License 2 | # 3 | # DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | # Copyright (C) 2020 DeepGlint. All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are met: 8 | # 9 | # * Redistributions of source code must retain the above copyright notice, this 10 | # list of conditions and the following disclaimer. 11 | # 12 | # * Redistributions in binary form must reproduce the above copyright notice, 13 | # this list of conditions and the following disclaimer in the documentation 14 | # and/or other materials provided with the distribution. 15 | # 16 | # * Neither the name of the copyright holder nor the names of its 17 | # contributors may be used to endorse or promote products derived from 18 | # this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | import pickle 32 | import matplotlib 33 | matplotlib.use('Agg') 34 | import matplotlib.pyplot as plt 35 | import numpy as np 36 | import sys 37 | import glob 38 | 39 | ################################################## 40 | #quant_hist[layer][0]: max_data 41 | #quant_hist[layer][1]: hist 42 | #quant_hist[layer][2]: hist_edges 43 | #quant_hist[layer][3]: threshold 44 | ################################################# 45 | 46 | quant_hist= pickle.load(open('./histgram.pkl','rb')) 47 | i= 0 48 | for layer in quant_hist.keys(): 49 | layer_name= layer 50 | max_data= quant_hist[layer][0] 51 | hist,hist_edges= quant_hist[layer][1],quant_hist[layer][2] 52 | hist_x= (hist_edges[0:-1]+hist_edges[1:])/2 53 | fig = plt.figure() 54 | hist_x= np.log2(hist_x+1) 55 | hist= np.log2(hist+1) 56 | plt.bar(hist_x,hist) 57 | plt.vlines(np.log2(quant_hist[layer][3]),0,np.max(hist),'r') 58 | plt.title(layer_name+' max_data:'+str(max_data)) 59 | plt.xlabel('log2 transform') 60 | plt.ylabel('log2 transform') 61 | #plt.show() 62 | fig.savefig('./hist_figures/'+layer_name+'.jpg') 63 | i +=1 64 | print(i) 65 | -------------------------------------------------------------------------------- /tools/infer_shape.py: -------------------------------------------------------------------------------- 1 | # BSD 3-Clause License 2 | # 3 | # DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | # Copyright (C) 2020 DeepGlint. All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are met: 8 | # 9 | # * Redistributions of source code must retain the above copyright notice, this 10 | # list of conditions and the following disclaimer. 11 | # 12 | # * Redistributions in binary form must reproduce the above copyright notice, 13 | # this list of conditions and the following disclaimer in the documentation 14 | # and/or other materials provided with the distribution. 15 | # 16 | # * Neither the name of the copyright holder nor the names of its 17 | # contributors may be used to endorse or promote products derived from 18 | # this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | import argparse 32 | import pickle 33 | import caffe 34 | import caffe.proto.caffe_pb2 as caffe_pb2 35 | import cv2 36 | import numpy as np 37 | import shutil 38 | from google.protobuf import text_format 39 | import scipy.spatial.distance as dis 40 | import sys 41 | from collections import OrderedDict 42 | import numpy as np 43 | from functools import reduce 44 | import os 45 | import re 46 | ############### end ################################### 47 | def parse_args(): 48 | parser = argparse.ArgumentParser( 49 | description='find the pretrained caffe models int8 quantize scale value') 50 | parser.add_argument('--proto', dest='proto', 51 | help="path to deploy prototxt.", type=str) 52 | 53 | parser.add_argument('--model', dest='model', 54 | help='path to pretrained weights', type=str) 55 | 56 | parser.add_argument('--save', dest='save', 57 | help='path to saved shape pkl file', type=str, default='layerDims.pickle') 58 | 59 | args = parser.parse_args() 60 | return args, parser 61 | 62 | 63 | global args, parser 64 | args, parser = parse_args() 65 | proto = args.proto 66 | model = args.model 67 | 68 | beginLayerIndex = 1 69 | endLayerIndex = 110 70 | 71 | def layerToOutputName(): 72 | namePat = re.compile(r'\s+?name:\s+?"(.*)"') 73 | topPat = re.compile(r'\s+?top:\s+?"(.*)"') 74 | res = {} 75 | with open(args.proto) as file: 76 | name = None 77 | top = None 78 | for line in file.readlines(): 79 | if re.match(namePat, line): 80 | name = re.match(namePat, line).group(1) 81 | if re.match(topPat, line): 82 | top = re.match(topPat, line).group(1) 83 | res[name] = top 84 | return res 85 | 86 | def findEachLayerDim(caffe_model, net_file): 87 | layer2OutputName = layerToOutputName() 88 | res = OrderedDict() 89 | with open(net_file, 'r') as fin: 90 | with open('temp.prototxt', 'w') as fout: 91 | for line in fin.readlines(): 92 | fout.write(line.replace('ReLU6', 'ReLU')) 93 | 94 | net = caffe.Net('temp.prototxt', caffe_model, caffe.TEST) 95 | 96 | img = np.random.random((224, 224, 3)) 97 | img = img.transpose(2, 0, 1) 98 | 99 | net.blobs['data'].data[...] = img 100 | 101 | output = net.forward() 102 | 103 | params = caffe_pb2.NetParameter() 104 | with open(net_file) as f: 105 | text_format.Merge(f.read(), params) 106 | print(net.blobs.keys()) 107 | for i, layer in enumerate(params.layer): 108 | print(layer.name) 109 | if layer.name in layer2OutputName.keys() and layer2OutputName[layer.name] in net.blobs.keys(): 110 | res[layer.name] = net.blobs[layer2OutputName[layer.name]].data[0].shape 111 | return res 112 | 113 | 114 | def main(): 115 | res = findEachLayerDim(args.model, args.proto) 116 | for k in res: 117 | print(k, res[k]) 118 | import os 119 | os.remove('temp.prototxt') 120 | with open(args.save, 'w') as file: 121 | pickle.dump(res, file) 122 | 123 | if __name__ == '__main__': 124 | main() 125 | -------------------------------------------------------------------------------- /tools/ncnn_val_caffe.py: -------------------------------------------------------------------------------- 1 | # BSD 3-Clause License 2 | # 3 | # DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | # Copyright (C) 2020 DeepGlint. All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are met: 8 | # 9 | # * Redistributions of source code must retain the above copyright notice, this 10 | # list of conditions and the following disclaimer. 11 | # 12 | # * Redistributions in binary form must reproduce the above copyright notice, 13 | # this list of conditions and the following disclaimer in the documentation 14 | # and/or other materials provided with the distribution. 15 | # 16 | # * Neither the name of the copyright holder nor the names of its 17 | # contributors may be used to endorse or promote products derived from 18 | # this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | #!/usr/bin/env python 32 | # -*- coding=utf8 -*- 33 | """ 34 | # Author: xiaozhang 35 | # Created Time : 2019-05-10 36 | # File Name: generateResult.py 37 | # Description: 38 | """ 39 | import os 40 | import cv2 41 | import numpy as np 42 | import ncnn 43 | from tqdm import tqdm 44 | import multiprocessing 45 | import argparse 46 | import scipy.spatial.distance as dis 47 | import glob 48 | 49 | # Hyper-params 50 | parser = argparse.ArgumentParser(description='PyTorch RTPose Testining') 51 | parser.add_argument('--param_path', default='../models/resnetFaceRec_no_bn_sq.param', type=str, 52 | help='path to where load param file') 53 | parser.add_argument('--bin_path', default='../models/resnetFaceRec_no_bn_sq.bin', type=str, 54 | help='path to where load bin file') 55 | parser.add_argument('--platform', default='ncnn', type=str, 56 | help='path to where load bin file') 57 | parser.add_argument('--num_workers', default=48, type=int, 58 | help='path to where load bin file') 59 | args = parser.parse_args() 60 | 61 | def image_processing(image, image_size, mean_value, std=[1.0, 1.0, 1.0]): 62 | w = image.shape[1] 63 | h = image.shape[0] 64 | m = min(w, h) 65 | ratio = 256.0 / m 66 | new_w, new_h = int(ratio * w), int(ratio * h) 67 | image = cv2.resize(image, (new_w, new_h)) 68 | image = image.astype(np.float32) 69 | top = (new_w - image_size)//2 70 | left = (new_h - image_size)//2 71 | image = image[left:left+image_size, top:top+image_size] 72 | image = image.transpose(2, 0, 1) 73 | image[0, :, :] = (image[0, :, :] - mean_value[0]) / std[0] # b 74 | image[1, :, :] = (image[1, :, :] - mean_value[1]) / std[1] # g 75 | image[2, :, :] = (image[2, :, :] - mean_value[2]) / std[2] # r 76 | 77 | return image # bgr, chw, normalized 78 | 79 | def threadFuncCaffe(proto, caffemodel, imgs, gpuid, image_size=224, mean_value=[104., 117., 123.], std=[1.0, 1.0, 1.0]): 80 | import caffe 81 | caffe.set_device(gpuid) 82 | caffe.set_mode_gpu() 83 | net = caffe.Net(proto, caffemodel, caffe.TEST) 84 | res= {} 85 | for idx in tqdm(range(len(imgs))): 86 | imagepath= imgs[idx] 87 | image = cv2.imread(imagepath) 88 | if image is None: 89 | continue 90 | image = image_processing(image, image_size, mean_value, std) 91 | 92 | net.blobs['data'].reshape(1, 3, image_size, image_size) 93 | net.blobs['data'].data[...] = np.array([image], dtype=np.float32) 94 | out = net.forward() 95 | prob = out['prob'] 96 | prob = np.squeeze(prob) 97 | ind = np.argsort(-prob)[0:5] 98 | res[imagepath]= ind 99 | return res 100 | 101 | def threadFunc(param_path,bin_path,imgs, gpuid, image_size=224, mean_value=[104., 117., 123.], std=[1., 1., 1.]): 102 | net = ncnn.net() 103 | net.load_param(param_path) 104 | net.load_model(bin_path) 105 | net.setInputBlobName("data") 106 | net.setOutputBlobName("prob") 107 | res= {} 108 | for idx in tqdm(range(len(imgs))): 109 | imagepath= imgs[idx] 110 | image = cv2.imread(imagepath) 111 | if image is None: 112 | continue 113 | image = image_processing(image, image_size, mean_value, std) 114 | 115 | image = image[::-1] 116 | image = image.reshape((image_size * image_size * 3,)) 117 | result = np.zeros((1000,)) 118 | result = result.astype(np.float32) 119 | net.inference(image,result, image_size, image_size) 120 | result = result.astype(np.float32) 121 | result = result.reshape(1000) 122 | ind = np.argsort(-result)[0:5] 123 | res[imagepath]= ind 124 | return res 125 | 126 | 127 | def main(): 128 | import glob 129 | imgs= glob.glob('data/val/*.JPEG')[:50000] 130 | fp32_param = args.param_path 131 | fp32_bin= args.bin_path 132 | fp32_res=[] 133 | image_size = 224 134 | mean_value = [103.939, 116.779, 123.63] 135 | std = [1., 1., 1.] 136 | 137 | res= [] 138 | processNum = args.num_workers # 48 139 | processList=[] 140 | pool = multiprocessing.Pool(processes=processNum) 141 | for i in range(processNum): 142 | minIndex = int(i*len(imgs)/processNum) 143 | maxIndex = np.min([int((i+1)*len(imgs)/processNum),len(imgs)]) 144 | if args.platform == 'ncnn': 145 | fp32_res.append(pool.apply_async(threadFunc, (fp32_param,fp32_bin,imgs[minIndex:maxIndex], i%8, image_size, mean_value, std))) 146 | else: 147 | fp32_res.append(pool.apply_async(threadFuncCaffe, (fp32_param,fp32_bin,imgs[minIndex:maxIndex], i%4, image_size, mean_value, std))) 148 | 149 | pool.close() 150 | pool.join() 151 | 152 | fp32_res_dict={} 153 | for i in fp32_res: 154 | fp32_res_dict.update(i.get()) 155 | 156 | import pickle 157 | print(len(fp32_res_dict)) 158 | with open('res_val_int88.pkl','wb') as f: 159 | pickle.dump(fp32_res_dict,f) 160 | 161 | gt = np.loadtxt('val1.txt') 162 | f = open('res_ncnn_int88.txt','w') 163 | for k,v in fp32_res_dict.items(): 164 | f.write(str(k)+' '+str(v[0])+' '+str(v[1])+' '+str(v[2])+' '+str(v[3])+' '+str(v[4])+'\n') 165 | f.close() 166 | key = list(fp32_res_dict.keys()) 167 | key.sort() 168 | top5 =0 169 | top1 =0 170 | for i in range(len(fp32_res_dict)): 171 | l = int(os.path.basename(key[i])[:-5].split('_')[-1])-1 172 | #print (key[i]+'\n') 173 | #print (gt[l],res[key[i]]) 174 | if gt[l] in fp32_res_dict[key[i]]: 175 | top5+=1. 176 | if gt[l] == fp32_res_dict[key[i]][0]: 177 | top1+=1. 178 | print (top5) 179 | print (top1) 180 | acc_top5 = top5/len(fp32_res_dict) 181 | acc_top1 = top1/len(fp32_res_dict) 182 | print (acc_top5) 183 | print (acc_top1) 184 | 185 | if __name__ == '__main__': 186 | main() 187 | -------------------------------------------------------------------------------- /tools/scale_fine_tuning.py: -------------------------------------------------------------------------------- 1 | # BSD 3-Clause License 2 | # 3 | # DeepGlint is pleased to support the open source community by making EasyQuant available. 4 | # Copyright (C) 2020 DeepGlint. All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are met: 8 | # 9 | # * Redistributions of source code must retain the above copyright notice, this 10 | # list of conditions and the following disclaimer. 11 | # 12 | # * Redistributions in binary form must reproduce the above copyright notice, 13 | # this list of conditions and the following disclaimer in the documentation 14 | # and/or other materials provided with the distribution. 15 | # 16 | # * Neither the name of the copyright holder nor the names of its 17 | # contributors may be used to endorse or promote products derived from 18 | # this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | import argparse 32 | import pickle 33 | # import caffe 34 | # import caffe.proto.caffe_pb2 as caffe_pb2 35 | import cv2 36 | import numpy as np 37 | import ncnn 38 | import re 39 | import shutil 40 | import scipy.spatial.distance as dis 41 | import sys 42 | from collections import OrderedDict 43 | import numpy as np 44 | from functools import reduce 45 | import os 46 | import multiprocessing 47 | import copy 48 | import datetime 49 | 50 | 51 | 52 | ############### end ################################### 53 | def parse_args(): 54 | parser = argparse.ArgumentParser( 55 | description='find the pretrained caffe models int8 quantize scale value') 56 | parser.add_argument('--proto', dest='proto', 57 | help="path to deploy prototxt.", type=str) 58 | 59 | parser.add_argument('--model', dest='model', 60 | help='path to pretrained weights', type=str) 61 | 62 | parser.add_argument('--param', dest='param', 63 | help="path to deploy prototxt.", type=str) 64 | 65 | parser.add_argument('--bin', dest='bin', 66 | help='path to pretrained weights', type=str) 67 | parser.add_argument('--table', dest='table', help='path to scale table', type=str) 68 | 69 | parser.add_argument('--layerDims', dest='layerDims', help='file stored layerdims', type=str) 70 | 71 | parser.add_argument('--search-log', dest='search_log', default='log.log', 72 | help='log of search', type=str) 73 | args = parser.parse_args() 74 | return args, parser 75 | 76 | def layerToOurputName(): 77 | namePat = re.compile(r'\s+?name:\s+?"(.*)"') 78 | topPat = re.compile(r'\s+?top:\s+?"(.*)"') 79 | res = {} 80 | with open(args.proto) as file: 81 | name = None 82 | top = None 83 | for line in file.readlines(): 84 | if re.match(namePat, line): 85 | name = re.match(namePat, line).group(1) 86 | if re.match(topPat, line): 87 | top = re.match(topPat, line).group(1) 88 | res[name] = top 89 | return res 90 | 91 | 92 | 93 | 94 | 95 | # def findEachLayerDim(caffe_model, net_file): 96 | # 97 | # res = OrderedDict() 98 | # with open(net_file, 'r') as fin: 99 | # with open('temp.prototxt', 'w') as fout: 100 | # for line in fin.readlines(): 101 | # fout.write(line.replace('ReLU6', 'ReLU')) 102 | # 103 | # net = caffe.Net('temp.prototxt', caffe_model, caffe.TEST) 104 | # 105 | # img = np.random.random((224, 224, 3)) 106 | # img = img.transpose(2, 0, 1) 107 | # 108 | # net.blobs['data'].data[...] = img 109 | # 110 | # output = net.forward() 111 | # 112 | # params = caffe_pb2.NetParameter() 113 | # with open(net_file) as f: 114 | # text_format.Merge(f.read(), params) 115 | # print(net.blobs.keys()) 116 | # for i, layer in enumerate(params.layer): 117 | # net_layer_name = layer.name.replace('conv', 'bn') 118 | # if net_layer_name in net.blobs.keys(): 119 | # res[layer.name] = net.blobs[net_layer_name].data[0].shape 120 | # os.remove('temp.prototxt') 121 | # return res 122 | 123 | def inferenceThread(net_fp32, net_int8, image_net, dim, layer, relu= False, image_size=224): 124 | 125 | 126 | net_fp32 = ncnn.net() 127 | net_fp32.load_param(param) 128 | net_fp32.load_model(bins) 129 | net_fp32.setInputBlobName("data") 130 | if relu: 131 | relu_name = layer.replace('conv', 'relu') 132 | out_blob_name = layer2OutputName[relu_name] + '_' + relu_name 133 | else: 134 | out_blob_name = layer2OutputName[layer] 135 | net_fp32.setOutputBlobName(out_blob_name) 136 | 137 | net_int8 = ncnn.net() 138 | net_int8.load_param('modified.param') 139 | net_int8.load_model("modified.bin") 140 | net_int8.setInputBlobName("data") 141 | net_int8.setOutputBlobName(out_blob_name) 142 | 143 | result_fp32 = np.zeros(dim).astype(np.float32) 144 | result_int8 = np.zeros(dim).astype(np.float32) 145 | 146 | net_int8.inference(image_net, result_int8, image_size, image_size) 147 | net_fp32.inference(image_net, result_fp32, image_size, image_size) 148 | 149 | result_fp32 = result_fp32.reshape((dim[0], -1)) 150 | result_int8 = result_int8.reshape((dim[0], -1)) 151 | cosSimilarity_allChannel = np.zeros(dim[0]) 152 | for i in range(dim[0]): 153 | cosSimilarity_allChannel[i] = cosSimilatity(result_fp32[i], result_int8[i]) 154 | 155 | return cosSimilarity_allChannel 156 | 157 | def inferenceThread_PerLayer(image_net, dim, layer, relu= False, image_size=224): 158 | 159 | 160 | net_fp32 = ncnn.net() 161 | net_fp32.load_param(param) 162 | net_fp32.load_model(bins) 163 | net_fp32.setInputBlobName("data") 164 | if relu: 165 | relu_name = layer.replace('conv', 'relu') 166 | out_blob_name = layer2OutputName[relu_name] + '_' + relu_name 167 | else: 168 | out_blob_name = layer2OutputName[layer] 169 | net_fp32.setOutputBlobName(out_blob_name) 170 | 171 | net_int8 = ncnn.net() 172 | net_int8.load_param('modified.param') 173 | net_int8.load_model("modified.bin") 174 | net_int8.setInputBlobName("data") 175 | net_int8.setOutputBlobName(out_blob_name) 176 | 177 | result_fp32 = np.zeros(dim).astype(np.float32) 178 | result_int8 = np.zeros(dim).astype(np.float32) 179 | 180 | net_fp32.inference(image_net, result_fp32, image_size, image_size) 181 | net_int8.inference(image_net, result_int8, image_size, image_size) 182 | 183 | result_fp32 = result_fp32.reshape(-1) 184 | result_int8 = result_int8.reshape(-1) 185 | 186 | 187 | return cosSimilatity(result_fp32, result_int8) 188 | 189 | def image_processing(image, image_size, mean_value): 190 | w = image.shape[1] 191 | h = image.shape[0] 192 | m = min(w, h) 193 | ratio = 256.0 / m 194 | new_w, new_h = int(ratio * w), int(ratio * h) 195 | image = cv2.resize(image, (new_w, new_h)) 196 | image = image.astype(np.float32) 197 | top = (new_w - image_size)//2 198 | left = (new_h - image_size)//2 199 | image = image[left:left+image_size, top:top+image_size] 200 | image = image.transpose(2, 0, 1) 201 | image[0, :, :] = image[0, :, :] - mean_value[0] 202 | image[1, :, :] = image[1, :, :] - mean_value[1] 203 | image[2, :, :] = image[2, :, :] - mean_value[2] 204 | 205 | return image # bgr, chw, normalized 206 | 207 | 208 | def calcChannelSimilarity(net_fp32, net_int8, imagenames, dim, layer): 209 | channelSimilarity = np.zeros(dim[0]) 210 | pool = multiprocessing.Pool(processes=len(imagenames)) 211 | poolThreads = [] 212 | image_size = 224 213 | mean_value = [103.939, 116.779, 123.63] 214 | for imagename in imagenames: 215 | image = cv2.imread(imagename) 216 | image = image_processing(image, image_size, mean_value) 217 | image = image[::-1] 218 | image_net = image.reshape((image_size * image_size * 3,)) 219 | 220 | if 'fc' in layer or 'branch1' in layer or 'branch2c' in layer: 221 | has_relu = False 222 | else: 223 | has_relu = True 224 | poolThreads.append(pool.apply_async(inferenceThread, (net_fp32, net_int8, image_net, dim, layer, has_relu, image_size,))) 225 | # channelSimilarity += inferenceThread(net_fp32, net_int8, image_net, dim, layer, has_relu, image_size,) 226 | #print('Threads start...') 227 | pool.close() 228 | pool.join() 229 | 230 | for thread in poolThreads: 231 | channelSimilarity += thread.get() 232 | 233 | return channelSimilarity / len(imagenames) 234 | 235 | 236 | def calcLayerSimilarity(imagenames, dim, layer): 237 | layerSimilarity = 0. 238 | pool = multiprocessing.Pool(processes=len(imagenames)) 239 | poolThreads = [] 240 | image_size = 224 241 | mean_value = [103.939, 116.779, 123.63] 242 | for imagename in imagenames: 243 | image = cv2.imread(imagename) 244 | image = image_processing(image, image_size, mean_value) 245 | image = image[::-1] 246 | image_net = image.reshape((image_size * image_size* 3,)) 247 | if 'fc' in layer or 'branch1' in layer or 'branch2c' in layer: 248 | has_relu = False 249 | else: 250 | has_relu = True 251 | poolThreads.append(pool.apply_async(inferenceThread_PerLayer, (image_net, dim, layer, has_relu, image_size,))) 252 | # layerSimilarity += inferenceThread_PerLayer(image_net, dim, layer, has_relu, image_size,) 253 | #print('Threads start...') 254 | pool.close() 255 | pool.join() 256 | 257 | for thread in poolThreads: 258 | layerSimilarity += thread.get() 259 | 260 | return layerSimilarity / len(imagenames) 261 | 262 | def cosSimilatity(x, y): 263 | if np.sum(np.abs(x)) == 0 or np.sum(np.abs(y)) == 0: 264 | x = np.add(x, 1e-5) 265 | y = np.add(y, 1e-5) 266 | return 1 - dis.cosine(x, y) 267 | 268 | 269 | ###################### end ############################## 270 | 271 | 272 | def modifyTable_forLayer(origin, modified, layer, channels, scales): 273 | fin = open(origin, 'r') 274 | fout = open(modified, 'w') 275 | modified = False 276 | for line in fin.readlines(): 277 | if len(line.split()) < 5: 278 | fout.write(line) 279 | else: 280 | splits = line.split() 281 | layer_name = splits[0][:-8] 282 | if layer_name == layer: 283 | assert len(channels) == len(scales) 284 | for i in range(len(channels)): 285 | splits[channels[i] + 1] = str(scales[i]) 286 | fout.write(' '.join(splits)) 287 | fout.write('\n') 288 | modified = True 289 | else: 290 | fout.write(line) 291 | fin.close() 292 | fout.close() 293 | if not modified: 294 | print("Not modified for layer: " + str(layer)) 295 | exit(0) 296 | 297 | def modifyActivationScale(origin, modified, layer, scale): 298 | fin = open(origin, 'r') 299 | fout = open(modified, 'w') 300 | modified = False 301 | for line in fin.readlines(): 302 | if len(line.split()) > 5 or len(line.split()) < 2: 303 | fout.write(line) 304 | else: 305 | splits = line.split() 306 | layer_name = splits[0] 307 | if layer_name == layer: 308 | splits[1] = str(scale) 309 | fout.write(' '.join(splits)) 310 | fout.write('\n') 311 | modified = True 312 | else: 313 | fout.write(line) 314 | fin.close() 315 | fout.close() 316 | if not modified: 317 | print("Not modified for layer: " + str(layer)) 318 | exit(0) 319 | 320 | def weight_fine_tuning(log): 321 | layers = collections.OrderedDict() 322 | 323 | with open('original.table', 'r') as f: 324 | for line in f.readlines(): 325 | if len(line.split()) < 5: 326 | continue 327 | layer_name = line.split()[0][:-8] 328 | layer_scale = list(map(float, line.split()[1:])) 329 | if not layer_name in layers: 330 | layers[layer_name] = layer_scale 331 | 332 | print("weight scale fine tuning...") 333 | start_time = datetime.datetime.now() 334 | for layer in layers.keys(): 335 | log.write('layer: {}\n'.format(layer)) 336 | log.flush() 337 | print('layer {}: searching...'.format(layer)) 338 | 339 | os.system('caffe2ncnn ' + proto + ' ' + model + ' modified.param modified.bin 0 original.table') 340 | 341 | res = calcChannelSimilarity(None, None, imageNames, layerDims[layer], layer) 342 | problem_channels = [] 343 | channels_scale = [] 344 | channels_cosval = [] 345 | 346 | for channel in range(len(res)): 347 | problem_channels.append(channel) 348 | scales = [layers[layer][channel]] 349 | #print(layer) 350 | if 'proj' in layer: 351 | scales.extend(np.linspace(1, layers[layer][channel] * 1., 100)[1:-1]) 352 | else: 353 | scales.extend(np.linspace(1, layers[layer][channel] * 1., 100)[1:-1]) 354 | channels_scale.append(scales) 355 | channels_cosval.append([res[channel]]) 356 | if len(problem_channels) == 0: 357 | continue 358 | assert len(problem_channels) == len(channels_scale) == len(channels_cosval) 359 | channels_scale = np.array(channels_scale) 360 | for i in range(1, len(channels_scale[0])): 361 | need_to_modify_scale = channels_scale[:, i] 362 | modifyTable_forLayer('original.table', 'modified.table', layer, problem_channels, need_to_modify_scale) 363 | os.system('caffe2ncnn ' + proto + ' ' + model + ' modified.param modified.bin 0 modified.table') 364 | 365 | cadidate_res = calcChannelSimilarity(None, None, imageNames, layerDims[layer], layer) 366 | for j in range(len(problem_channels)): 367 | channels_cosval[j].append(cadidate_res[problem_channels[j]]) 368 | best_scale = [] 369 | best_cos = [] 370 | for k in range(len(channels_cosval)): 371 | index = np.argmax(channels_cosval[k]) 372 | best_scale.append(channels_scale[k][index]) 373 | best_cos.append(np.max(channels_cosval[k])) 374 | modifyTable_forLayer('original.table', 'modified.table', layer, problem_channels, best_scale) 375 | log.write('fixed channels\' index is: ') 376 | for each in problem_channels: 377 | log.write(str(each) + ' ') 378 | log.write('\n') 379 | for i in range(len(problem_channels)): 380 | log.write('channel ' + str(problem_channels[i]) + '\n') 381 | for scale in channels_scale[i]: 382 | log.write(str(scale) + ' ') 383 | log.write('\n') 384 | for val in channels_cosval[i]: 385 | log.write(str(val) + ' ') 386 | log.write('\n') 387 | log.write('choose scale value is: ' + str(best_scale[i]) + '| cos_val: ' + str(best_cos[i]) + '\n') 388 | shutil.copyfile('modified.table', 'original.table') 389 | 390 | end_time = datetime.datetime.now() 391 | print(start_time) 392 | print(end_time) 393 | print((end_time - start_time).seconds) 394 | 395 | def activation_fine_tuning(log): 396 | print('activation scale fine tuning') 397 | start_time = datetime.datetime.now() 398 | 399 | layers = collections.OrderedDict() 400 | 401 | with open('original.table', 'r') as f: 402 | for line in f.readlines(): 403 | if len(line.split()) < 2 or len(line.split()) > 5: 404 | continue 405 | layer_name = line.split()[0] 406 | layer_scale = float(line.split()[1]) 407 | if not layer_name in layers: 408 | layers[layer_name] = layer_scale 409 | 410 | for layer in layers.keys(): 411 | 412 | log.write('layer: ' + layer + '\n') 413 | log.flush() 414 | print('layer ' + layer + ": searching...") 415 | 416 | os.system('caffe2ncnn ' + proto + ' ' + model + ' modified.param modified.bin 0 original.table') 417 | 418 | res = calcLayerSimilarity(imageNames, layerDims[layer], layer) 419 | 420 | layer_cosval = [] 421 | 422 | layer_scales = [layers[layer]] 423 | layer_scales.extend(np.linspace(layers[layer] * 0.8, layers[layer] * 1.0, 200)[1:-1]) 424 | layer_cosval.append(res) 425 | 426 | for i in range(1, len(layer_scales)): 427 | 428 | need_to_modify_scale = layer_scales[i] 429 | modifyActivationScale('original.table', 'modified.table', layer, need_to_modify_scale) 430 | os.system('caffe2ncnn ' + proto + ' ' + model + ' modified.param modified.bin 0 modified.table') 431 | 432 | cadidate_res = calcLayerSimilarity(imageNames, layerDims[layer], layer) 433 | layer_cosval.append(cadidate_res) 434 | 435 | for k in range(len(layer_cosval)): 436 | index = np.argmax(layer_cosval) 437 | best_scale = layer_scales[index] 438 | best_cos = (np.max(layer_cosval)) 439 | modifyActivationScale('original.table', 'modified.table', layer, best_scale) 440 | 441 | log.write('\n') 442 | 443 | for scale in layer_scales: 444 | log.write(str(scale) + ' ') 445 | log.write('\n') 446 | for val in layer_cosval: 447 | log.write(str(val) + ' ') 448 | log.write('\n') 449 | log.write('choose scale value is: ' + str(best_scale) + '| cos_val: ' + str(best_cos) + '\n') 450 | 451 | shutil.copyfile('modified.table', 'original.table') 452 | 453 | end_time = datetime.datetime.now() 454 | print(start_time) 455 | print(end_time) 456 | print((end_time - start_time).seconds) 457 | 458 | 459 | 460 | 461 | global args, parser 462 | args, parser = parse_args() 463 | proto = args.proto 464 | model = args.model 465 | param = args.param 466 | bins = args.bin 467 | 468 | scale_table = args.table 469 | layerDimsFile = args.layerDims 470 | file = open(layerDimsFile, 'rb') 471 | layerDims = pickle.load(file) 472 | file.close() 473 | layer2OutputName = layerToOurputName() 474 | 475 | 476 | 477 | beginLayerIndex = 1 478 | endLayerIndex = 110 479 | 480 | imageRoot = 'data/calib/' 481 | f = open("data/list50.txt", 'r') 482 | lines = f.readlines() 483 | imageNames = [] 484 | for line in lines: 485 | imageName = str(line).strip() 486 | if len(imageName) < 1: 487 | continue 488 | imageNames.append(imageRoot + imageName) 489 | 490 | 491 | shutil.copyfile(scale_table, 'original.table') 492 | shutil.copyfile('original.table', 'modified.table') 493 | import collections 494 | 495 | log = open(args.search_log, 'w') 496 | weight_fine_tuning(log) 497 | activation_fine_tuning(log) 498 | 499 | os.system('caffe2ncnn ' + proto + ' ' + model + ' modified.param modified.bin 0 original.table') 500 | log.close() 501 | 502 | 503 | 504 | --------------------------------------------------------------------------------