├── .gitignore ├── README.md ├── files ├── dusan_stosic_intro_to_mixed_precision_training.pdf ├── guilin_liu_mixed_precision_for_inpainting.pdf ├── karan_sapra_mixed_precision_for_semantic_seg.pdf ├── mingyuliu_nvidia_mixed_precision_for_GauGAN.pdf ├── pavlo_molchanov_mixed_precision_for_pruning.pdf └── tingchunw_nvidia_mixed_precision_for_pix2pixHD.pdf ├── imgs ├── ArunMallya.jpg ├── CarlCase.jpg ├── GuilinLiu.jpg ├── KaranSapra.jpg ├── MingYuLiu.jpg ├── PauliusM.jpeg ├── PavloMolchanov.jpg ├── TingChunWang.jpg ├── favicon.ico ├── map.png └── placeholder.png └── index.html /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # iccv2019-mixed-precision-tutorial 2 | -------------------------------------------------------------------------------- /files/dusan_stosic_intro_to_mixed_precision_training.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/files/dusan_stosic_intro_to_mixed_precision_training.pdf -------------------------------------------------------------------------------- /files/guilin_liu_mixed_precision_for_inpainting.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/files/guilin_liu_mixed_precision_for_inpainting.pdf -------------------------------------------------------------------------------- /files/karan_sapra_mixed_precision_for_semantic_seg.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/files/karan_sapra_mixed_precision_for_semantic_seg.pdf -------------------------------------------------------------------------------- /files/mingyuliu_nvidia_mixed_precision_for_GauGAN.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/files/mingyuliu_nvidia_mixed_precision_for_GauGAN.pdf -------------------------------------------------------------------------------- /files/pavlo_molchanov_mixed_precision_for_pruning.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/files/pavlo_molchanov_mixed_precision_for_pruning.pdf -------------------------------------------------------------------------------- /files/tingchunw_nvidia_mixed_precision_for_pix2pixHD.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/files/tingchunw_nvidia_mixed_precision_for_pix2pixHD.pdf -------------------------------------------------------------------------------- /imgs/ArunMallya.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/ArunMallya.jpg -------------------------------------------------------------------------------- /imgs/CarlCase.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/CarlCase.jpg -------------------------------------------------------------------------------- /imgs/GuilinLiu.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/GuilinLiu.jpg -------------------------------------------------------------------------------- /imgs/KaranSapra.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/KaranSapra.jpg -------------------------------------------------------------------------------- /imgs/MingYuLiu.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/MingYuLiu.jpg -------------------------------------------------------------------------------- /imgs/PauliusM.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/PauliusM.jpeg -------------------------------------------------------------------------------- /imgs/PavloMolchanov.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/PavloMolchanov.jpg -------------------------------------------------------------------------------- /imgs/TingChunWang.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/TingChunWang.jpg -------------------------------------------------------------------------------- /imgs/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/favicon.ico -------------------------------------------------------------------------------- /imgs/map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/map.png -------------------------------------------------------------------------------- /imgs/placeholder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NVlabs/iccv2019-mixed-precision-tutorial/aecd4c190534d7646309b94ac1e40f23b256bfd5/imgs/placeholder.png -------------------------------------------------------------------------------- /index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 13 | 14 | 136 | 137 | AMP Tutorial 138 | 139 | 140 | 141 | 142 |

143 | ICCV 2019 Tutorial on
144 | Accelerating Computer Vision with Mixed Precision

145 | 146 | Time and Location:
147 | Saturday, Nov 2, Half Day AM
148 | Room E5 149 |

150 |

151 | 152 |
153 |
Update
154 |

155 | 156 | A newer version of this tutorial held at ECCV 2020 can be found here 157 | 158 |
159 |

160 |
161 |
162 | 163 |
164 |
Overview
165 |

166 | New levels of accuracy in computer vision, from image recognition and detection, to generating images with 167 | GANs, have been achieved by increasing the size of trained models. Fast turn-around times while iterating on 168 | the design of such models would greatly improve the rate of progress in this new era of computer vision. 169 |
170 |

171 | 172 |

173 | This tutorial will describe techniques that utilize half-precision floating point representations to 174 | allow deep learning practitioners to accelerate the training of large deep networks while also reducing 175 | memory requirements.
176 |

177 | 178 |

179 | The talks and sessions below will provide a deep-dive into available software packages that enable easy 180 | conversion of 181 | models to mixed precision training, practical application examples and tricks of the trade (mixed precision 182 | arithmetic, loss scaling, etc.), as well as considerations relevant to training many popular models in 183 | commonly used deep learning frameworks including PyTorch 184 | and TensorFlow. 185 |

186 | 187 | 188 |
189 |
190 | 191 | 192 |
193 |
Schedule
194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 232 | 233 | 234 | 235 | 238 | 239 | 240 | 241 | 242 | 243 | 246 | 247 | 248 | 249 | 250 | 251 | 254 | 255 | 256 | 257 | 258 | 259 | 262 | 263 | 264 | 265 | 266 | 267 | 270 | 271 | 272 | 273 | 274 |
TimeTitleSpeakerAffil.
08:45 - 08:50WelcomeArun MallyaNVIDIA
08:50 - 09:30 211 | Introduction to Mixed Precision Training with PyTorch and TensorFlow 212 | Dusan StosicNVIDIA
09:30 - 10:00Mixed Precision Training and Inference at Scale at AlibabaMengdi WangAlibaba
10:00 - 11:00Coffee Break
230 | Accelerating Various Tasks with Mixed Precision 231 |
11:00 - 11:20 236 | Network Pruning 237 | Pavlo MolchanovNVIDIA
11:20 - 11:35 244 | Semantic Segmentation 245 | Karan SapraNVIDIA
11:35 - 11:50 252 | Image Processing 253 | Guilin LiuNVIDIA
11:50 - 12:10 260 | Image Synthesis 261 | Ting-Chun WangNVIDIA
12:10 - 12:30 268 | Generative Adversarial Networks (GANs) 269 | Ming-Yu LiuNVIDIA
275 |
276 |
277 | 278 |
279 |
Useful Links
280 |
281 |
282 | NVIDIA Tensor Cores for Mixed Precision: 283 | Discover, Learn, Test, and Implement 284 | / 285 | Optimizing 286 | for Tensor Cores 287 |

288 | 289 | High-Performance Sample Code for Various Applications: 290 | PyTorch and TensorFlow 291 |

292 | 293 | NVIDIA Automatic Mixed Precision (AMP): 294 | Training ImageNet in 295 | PyTorch / 296 | Introduction / 297 | Documentation / 298 | Github 299 |

300 | 301 | NVIDIA Data Loading Library (DALI) for faster data loading: 302 | Introduction / 303 | Documentation / 305 | Github 306 |
307 |
308 |
309 | 310 |
311 |
AMP Model Zoo
312 |

313 | Below are code repositories which have used AMP to train their models. Please refer to their respective 314 | READMEs 315 | to activate AMP.
316 | These repos can serve as a guide for using AMP in your project. 317 |

318 | 319 | 320 | 321 | 322 | 324 | 325 | 326 | 327 | 329 | 330 | 331 | 332 | 335 | 336 | 337 | 338 | 341 | 342 | 343 | 344 | 346 | 347 | 348 | 349 | 352 | 353 | 354 | 355 | 358 | 359 | 360 | 361 | 364 | 365 | 366 | 367 | 370 | 371 | 372 | 373 | 376 | 377 | 378 |
ImageNet Classificationhttps://github.com/NVIDIA/apex/tree/master/examples/imagenet
Semantic Segmentationhttps://github.com/NVIDIA/semantic-segmentation
Mask R-CNNhttps://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/MaskRCNN 334 |
SSD Detectionhttps://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD 340 |
Vid2Vidhttps://github.com/NVIDIA/vid2vid 345 |
Pix2PixHDhttps://github.com/NVIDIA/pix2pixHD 351 |
Image Inpaintinghttps://github.com/NVIDIA/partialconv#mixed-precision-training-with-amp-for-image-inpainting 357 |
Language Modelinghttps://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling/ 363 |
All TensorFlow linkshttps://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow 369 |
All PyTorch linkshttps://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch 375 |
379 |
380 |
381 | 382 |
383 |
Organizers
384 |
385 | 386 | 387 | 388 | 389 | 390 | 391 | 400 | 401 | 402 | 403 | 404 | 416 | 417 | 418 | 419 | 420 | 428 | 429 | 430 | 431 | 432 | 443 | 444 | 445 | 446 | 447 | 455 | 456 | 457 | 458 | 459 | 470 | 471 | 472 | 473 | 474 | 485 | 486 | 487 | 488 | 489 | 506 | 507 | 508 |
Arun Mallya 392 | Arun Mallya is a Research 393 | Scientist at NVIDIA Research. He obtained his Ph.D. from the University of 394 | Illinois at Urbana-Champaign in 2018, with a focus on performing multiple tasks efficiently with 395 | a single deep network. He holds a B.Tech. in Computer Science and Engineering from the Indian 396 | Institute of Technology - Kharagpur (2012), an MS in Computer Science from the University of 397 | Illinois at Urbana-Champaign (2014). His current interests are on making neural networks faster, 398 | better, and more compact. 399 |
Paulius Micikevicius 405 | Paulius Micikevicius works in compute architecture at NVIDIA on accelerating DNN training 406 | and 407 | inference through reduced arithmetic and other optimization techniques. In his previous roles at 408 | NVIDIA Paulius focused on parallelizing various compute workloads for GPUs, including DL, 409 | computer vision, scientific, and engineering applications. He also worked worked on developer 410 | education, connecting chip architecture details to software optimization techniques, resulting 411 | some of the highest rated and attended GPU Technology Conference talks. Paulius has also worked 412 | on perception while at a self-driving car company Zoox and as an assistant professor of Computer 413 | Science at Armstrong Atlantic State University. He holds a PhD in Computer Science from the 414 | University of Central Florida. 415 |
Carl Case 421 | Carl Case is a senior architect in compute architecture at NVIDIA, where he works on 422 | reduced-precision arithmetic for training deep neural networks. His focus is optimizing the 423 | entire stack of deep learning training - from hardware to high-level software - to accelerate 424 | the pace of AI development. Previously, he worked as a machine learning researcher on Deep 425 | Speech and its successor speech recognition systems at Baidu's Silicon Valley AI Lab. He holds 426 | bachelor's and master's degrees in computer science from Stanford University. 427 |
Pavlo Molchanov 433 | Pavlo Molchanov obtained PhD from Tampere University of Technology, Finland in the area of 434 | signal processing in 2014. His dissertation was focused on designing automatic target 435 | recognition systems for radars. Since 2015 he is with the Learning and Perception Research team 436 | at NVIDIA, currently holding a senior research scientist position. His research is focused on 437 | methods for neural network acceleration, and designing novel systems for human-computer 438 | interaction and human understanding. For network acceleration, he is interested in neural 439 | network pruning methods and conditional inference. For human understanding, he is working on 440 | landmark estimation, gesture recognition, hand pose estimation. He received the EuRAD best paper 441 | award in 2011 and EuRAD young engineer award in 2013. 442 |
Karan Sapra 448 | Karan Sapra 449 | is a research scientist at NVIDIA in Santa Clara, US. He obtained his PhD. 450 | from Clemson University in 2018. During his PhD studies, he was an intern at Oak Ridge National 451 | Lab in 2015. His research interest lies in using deep learning for computer vision and computer 452 | graphics. During his PhD. he has also worked on various other research areas such as Peer2peer 453 | (P2P) networks, computer networking and security, and high performance computing (HPC) 454 |
Guilin Liu 460 | Guilin Liu is a senior 461 | research 462 | scientist at NVIDIA in Santa Clara, US. He obtained 463 | his Ph.D. from George Mason University in 2017. During his P.h.D study, he as an intern at Adobe 464 | Research in 2016. He received his B.E. from Wuhan University in 2012. His research interest lies 465 | in the intersection among deep learning, computer vision and graphics. His recent research 466 | interest focus on using deep learning for image processing and estimating physical properties 467 | from images. His works have been published at ICCV, CVPR, ECCV, NeurIPS, IROS etc and featured 468 | in some mainstream media outlets including Fortune, Yahoo Finance, VentureBeat etc. 469 |
Ting-Chun Wang 475 | Ting-Chun Wang is a research 476 | scientist at NVIDIA in Santa Clara, US. He obtained his Ph.D. 477 | from 478 | University of California, Berkeley, department of EECS, advised by Professor Ravi Ramamoorthi 479 | and Alexei A. Efros. He received his B.E from National Taiwan University. He is a recipient of 480 | the Berkeley Fellowship. His research interests include computer vision, machine learning and 481 | computer graphics, particularly the intersections of all three. His recent research focus is on 482 | using generative adversarial models to synthesize realistic images and videos, with applications 483 | to rendering, visual manipulations and beyond. 484 |
Ming-Yu Liu 490 | Ming-Yu Liu is a principal research 491 | scientist at NVIDIA Research. Before joining NVIDIA in 492 | 2016, 493 | he 494 | was a principal research scientist at Mitsubishi Electric Research Labs (MERL). He earned his 495 | Ph.D. 496 | from the Department of Electrical and Computer Engineering at the University of Maryland College 497 | Park in 2012. He is a recipient of the R&D 100 Award by R&D Magazine for his robotic bin picking 498 | system in 2014. His semantic image synthesis paper and scene understanding paper are in the best 499 | paper finalist in the 2019 CVPR and 2015 RSS conferences, respectively. In SIGGRAPH 2019, he won 500 | the 501 | Best in Show Award and Audience Choice Award in the Real Time Live show for his image synthesis 502 | work. His research focus is on generative image modeling. His research goal is to enable 503 | machines 504 | human-like imagination capability. 505 |
509 |
510 | 511 |







512 | 513 | 514 | 515 | --------------------------------------------------------------------------------